Repository navigation
Can't load saved model: ImportError: No module named 'astraea' #83
Description
I can save a model to s3 but can't load it with the following:
from pyspark.ml import PipelineModel
model.save("s3://activemapper/test.model")
model_load = PipelineModel.load("s3://activemapper/test.model")
Documentation indicates this is the way to load and use past models: https://stackoverflow.com/questions/50587744/how-to-save-the-model-after-doing-pipeline-fit
The error is
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-15-a0eb67c190e1> in <module>()
----> 1 model_load = PipelineModel.load("s3://activemapper/test.model")
/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/util.py in load(cls, path)
255 def load(cls, path):
256 """Reads an ML instance from the input path, a shortcut of `read().load(path)`."""
--> 257 return cls.read().load(path)
258
259
/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/util.py in load(self, path)
199 raise NotImplementedError("This Java ML type cannot be loaded into Python currently: %r"
200 % self._clazz)
--> 201 return self._clazz._from_java(java_obj)
202
203 def context(self, sqlContext):
/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/pipeline.py in _from_java(cls, java_stage)
230 """
231 # Load information from java_stage to the instance.
--> 232 py_stages = [JavaParams._from_java(s) for s in java_stage.stages()]
233 # Create a new instance of this stage.
234 py_stage = cls(py_stages)
/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/pipeline.py in <listcomp>(.0)
230 """
231 # Load information from java_stage to the instance.
--> 232 py_stages = [JavaParams._from_java(s) for s in java_stage.stages()]
233 # Create a new instance of this stage.
234 py_stage = cls(py_stages)
/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/wrapper.py in _from_java(java_stage)
200 stage_name = java_stage.getClass().getName().replace("org.apache.spark", "pyspark")
201 # Generate a default new instance from the stage_name class.
--> 202 py_type = __get_class(stage_name)
203 if issubclass(py_type, JavaParams):
204 # Load information from java_stage to the instance.
/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/wrapper.py in __get_class(clazz)
194 parts = clazz.split('.')
195 module = ".".join(parts[:-1])
--> 196 m = __import__(module)
197 for comp in parts[1:]:
198 m = getattr(m, comp)
ImportError: No module named 'astraea'
pyspark raises this error higher in the chain: NotImplementedError("This Java ML type cannot be loaded into Python currently
Any help is appreciated!
Activity
metasim commentedon Oct 15, 2018
@rbavery Sorry for the delay in responding to this. Can you confirm that before you tried to load the model back in that you've called the
withRasterFrames()method on theSparkSession? My initial guess is that it's behaving as if the JVM library isn't loaded.Are you willing to share the JSON component of the saved workflow?
rbavery commentedon Oct 15, 2018
Hi @metasim thanks for the reply. I definitely called .withRasterFrames, here is what that section looks like
Attached is the saved workflow
model_pipeline.zip
metasim commentedon Oct 18, 2018
@rbavery Yeh, the saved workflow looks fine. Have you tried loading it in
localmode? I'm wondering if it's a YARN deployment thing.... the requisitejarand/orzipnot getting delivered properly.I'm inclined to want to ask @jbouffard on Gitter if he has seen this error before, as he's the person that seems to have done the most with pyrasterframes outside of my organization (where we may have inadvertently made assumptions about runtime environment).
rbavery commentedon Oct 18, 2018
@metasim haven't tried local mode, would I set it like this, keeping everything else constant?
Thanks for the help, I appreciate it.
rbavery commentedon Oct 22, 2018
@metasim Heads up I got a slightly different error when trying the above, just switching
master="yarn"tomaster=''local"metasim commentedon Oct 22, 2018
@rbavery I'm sorry this continues to be a problem. Would you be able to run the following before you try to load the model and confirm that there's a
rasterframesjar file in the classpath?If not, I'd need to know more about the difference between your training vs. scoring environment, to determine why RasterFrames was in the classspath in the former case, but not the latter.
rbavery commentedon Oct 26, 2018
Looks like there is not. I ran the following
and got
all of the following code I've posted has been tested in a jupyter notebook that has been configured by @jpolcho to run rasterframes and geopyspark. We use the notebook to try out new code and we use terraform to provision the cluster and run the spark submit command, which contains this argument
--packages", "io.astraea:pyrasterframes:0.7.3-GT2,org.apache.hadoop:hadoo p-aws:2.7.3,org.apache.logging.log4j:log4j-core:2.11.1", could this argument in the spark submit be relevant and if so how would I ensure it gets accounted for in the notebookenvironment when I set up theconfobject? Let me know what info you think might be helpful about the training and scoring environments and I can provide. Thanks for your help!metasim commentedon Oct 29, 2018
@rbavery TBH I'm not up on how GeoPySpark is set up and configured. For RasterFrames to work, you'll need the
rasterframes_2.11andrasterframes-datasource_2.11jars in the classpath too. So from here it looks like those libraries aren't being deployed correctly. Maybe ping @jpolcho on the RasterFrames gitter channel?jpolchlo commentedon Oct 30, 2018
@metasim I set up the environment for @rbavery by manually building a development version of RF directly in the cluster setup. This was accomplished by a call to
sbt pyrasterframes/spPublishLocal. The resulting jar is loaded into the spark environment using a--packagesargument. We use the custom version mentioned above, so it definitely isn't pulling from some remote maven, and it is loading correctly and is usable. The trouble only occurs when spark ML models are being loaded. I suspect that the functionality inrasterframes-datasourceisn't getting loaded in, but it's puzzling that the model saves just fine this way. Any idea what class isn't getting loaded to make that model import fail?rbavery commentedon Oct 30, 2018
When I added
io.astraea:rasterframes-datasource_2.11:0.7.0-RC4to
I still get
Thanks for your help @jpolchlo and @metasim !
metasim commentedon Oct 31, 2018
@jpolchlo Thanks for your help!
@rbavery
TwoThree things come to mind:org.locationtech.rasterframesgroup ID:org.locationtech.rasterframes:rasterframes_2.11:0.7.1rasterframes_2.11artifact to the--packageslist (you only havepyrasterframesandrasterframes-datasource? I would have thought the transitive dependency loading would have picked it up, but it can't hurt to be more explicit.jpolchlo commentedon Nov 9, 2018
Ran the above test. This was on EMR with Spark and Hive loaded. Started
pysparkwith the following command:Ran the two statements
Encountered the same error:
This is especially frustrating in that if
pysparkis replaced withspark-shell, thenimport astraea.spark.rasterframes.ml.NoDataFilterworks without complaint. Clearly the jar artifacts contain the necessary classes. In fact,PipelineModel.load(...)works just fine on the Scala side. Attemptingspark._jvm.astraea.spark.rasterframes.ml.NoDataFilter.load(...)from pyspark produces the same error as on the scala side, so the artifacts are available via the py4j gateway.Getting to be at a loss for solutions here. The environment seems to be properly configured, but for some reason, inaccessible from python in whatever way spark needs it to be.
Thoughts, @metasim?
metasim commentedon Nov 11, 2018
@jpolchlo @rbavery I got a little further with these scripts (but not total success):
https://gist.github.com/metasim/c4d1225124fcf53cac2d04330ee3f411
However, you can run this with success:
which says to me that the classes are on the classpath.
I'm not sure what else to do at the point, other than find others who have written ML transformers in Scala and been able to save/restore from Python. I'm wondering if you either have to have mirror classes in Python for each Scala class, or if there is some additional metadata required to instruct python how to reconstitute the Pipeline.
I'm sorry this has been a long road to what might be a dead end. If the goal is just to score a dataset, perhaps doing it in Scala isn't such a heavy lift.
cc: @bguseman
jpolchlo commentedon Feb 8, 2019
It appears that for Spark 2.2.1, the
PipelineModelloader relies on there being a python wrapper class in a python module of the same name as the java class that is listed in the model metadata (in this case, the python classastraea.spark.rasterframes.ml.NoDataFiltermust exist). It appears that the default Python readers in spark 2.3.2 might be smart enough to construct an appropriate wrapper class for the underlying RF java pipeline object, but this remains untested.I believe we forked off a version of RF before spark 2.3 compatibility was established, and then built new functionality on top of that. Looks like the solution might be to rebase the changes we made onto a newer version of RF and bump up the EMR version to one that has spark 2.3 or newer.
metasim commentedon Feb 11, 2019
@jpolchlo Thanks for the update.... super helpful! Is there some functionality from your fork you want rolled in?
jpolchlo commentedon Feb 11, 2019
We have some very rough code in our fork that establishes some pseudo-focal operations (via overbuffered tiles). See https://github.com/geotrellis/rasterframes/tree/feature/focal-operations. This is a model for how to provide the functionality, but not an implementation I'd really stand behind. Real GT
BufferedTiles should be promoted for that purpose, but that interface in GT isn't what I'd characterize as "battle-hardened". Might be a good opportunity to get some work in to geotrellis/geotrellis-contrib to make sure that proper buffering of tiles is built in to the new RasterSource interface.metasim commentedon Feb 11, 2019
@jpolchlo We definitely need buffered tiles in RF, so maybe that will be a good topic of discussion for when I visit!