Repository navigation
Reduce / simplify ceremony to access ProjectedRasterTile in python #147
Description
I commented in a PR about python TileUDT on the following code block there.
# rf_local_add(t, 0) is to force lazy eval; accessing tile.tile is to get at the actual Tile type vs PRT struct
df = self.spark.read.rastersource(self.img_uri).withColumn('tile2', rf_local_add_int(col('tile.tile'), 0))
# downstream we collect df in the python driver
The problem is that on the python side this rastersource tile column is actually a struct, so the user has to overcome the friction of use tile.tile to get the tile type.
The second challenge is the lazy evaluation. In the above snip if i did not add 0, I'd see the following error in the deserialize. Obviously we need to do some null checking but better would be a mechanism for the python side to request the evaluation of the cell values. Preferably this would happen without the user having to worry if the under the hood implementation is lazy.
/usr/local/rasterframes/pyrasterframes.zip/pyrasterframes/rf_types.py in deserialize(self, datum)
416 cell_data_bytes = datum.cell_data.cells
417 try:
--> 418 as_numpy = np.frombuffer(cell_data_bytes, dtype=cell_type.to_numpy_dtype())
419 reshaped = as_numpy.reshape((rows, cols))
420 t = Tile(reshaped, cell_type)
AttributeError: 'NoneType' object has no attribute '__buffer__'
Metadata
Metadata
Assignees
Labels
No labels
Activity
metasim commentedon Jun 17, 2019
@vpipkt
WRT the first issue, you should be able to do
rf_local_add_int(col('tile'), 0), as all the columnar functions are supposed to handle the PRT -> Tile conversion automatically.metasim commentedon Jun 17, 2019
WRT the second issue, I'd indeed like this to be automatic. I think adding a function to "reify" the tile in the JVM in this case should be straight forward. My hope was to inject a "fetch" instruction through the Catalyst optimizer, but I ran into some problems in the tree matching code.
vpipkt commentedon Jun 18, 2019
Yes I am able to do the
rf_local_add(tile)I just don't want to :-Dmetasim commentedon Jun 18, 2019
Related: #128