Repository navigation
Description
Hello everyone,
Thanks a lot for all your hard work developing this tool, it has been very helpful to me in my work.
I’ll like to report a problem with the method “download_size” in the Class “Proyect.py” of “ost” directory, in my query I have a series of Sentinel-1 images that have less than 1 Gb of information, and the method cannot transform the str to a flout because mb is not deleted from the str, so it shot an error (or that's what I understand) and I’m not able to know the complete download size of my search.
Terminal error:
Traceback (most recent call last):
File "path/ost_prueba_01/ost_download_test01.py", line 120, in
ost_s1.download_size(ost_s1.refine_inventory())
File "path/envs/virtualenv/lib/python3.6/site-packages/ost/Project.py", line 330, in download_size
self.inventory["size"].str.replace(" GB", "").astype("float32").sum()
File "path/envs/virtualenv/lib/python3.6/site-packages/pandas/core/generic.py", line 5546, in astype
new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors,)
File "path/envs/virtualenv/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 595, in astype
return self.apply("astype", dtype=dtype, copy=copy, errors=errors)
File "path/envs/virtualenv/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 406, in apply
applied = getattr(b, f)(**kwargs)
File "path/envs/virtualenv/lib/python3.6/site-packages/pandas/core/internals/blocks.py", line 595, in astype
values = astype_nansafe(vals1d, dtype, copy=True)
File "path/envs/virtualenv/lib/python3.6/site-packages/pandas/core/dtypes/cast.py", line 995, in astype_nansafe
return arr.astype(dtype, copy=True)
ValueError: could not convert string to float: '887.4 MB'
Script of "download_size":
def download_size(self, inventory_df=None):
"""Function to get the total size of all products when extracted in GB
:param inventory_df:
:return:
"""
if inventory_df is None:
download_size = (
self.inventory["size"].str.replace(" GB", "").astype("float32").sum()
)
else:
download_size = (
inventory_df["size"].str.replace(" GB", "").astype("float32").sum()
)logger.info(f"There are about {download_size} GB need to be downloaded.")
I’m not a programmer so I’m not able to give more information about the issue.
Thanks a lot.
Alexis A.
Activity
KBodolai commentedon Apr 24, 2022
Hey Alexis,
that output is enough, we just need to make it a bit more flexible, so it would replace anything at the end. I'd be happy to help you do a pull request, or to work on it myself in the coming weeks. Happy to point you in the right direction if y ou're feeling brave :)
Quick fix would be to add another
.replace(" MB", "")after the one for GB, but perhaps something more flexible using, for instance, regex would be clearer.Cheers,
Kristian.
alexisahedo commentedon Apr 26, 2022
Hello dear KBodolai,
Thanks a lot for the advice and for the good disposition to work into it in the future, I’ll give it a shot and try to make something with the sort of “if else” to round the images with Mb suffix to 1, so that the final sum be the closest as possible; I’ll also check out RegEx and try to apply it if it’s not too time consuming.
It will be very important for me because I’ll star to download soon a hole year of Sentinel-1 images for a national project and the space is a big deal.
Saludos.
Alexis A.
BuddyVolly commentedon Apr 28, 2022
Dear Alexis,
I am implementing the fix suggested by KBodolai, so it won't throw an error.
However the calculation of the full download size will be wrong, as it will assume those MBs as GBs.
I'll let you know, when a new version containing this fix is available.
Saludos, BV
alexisahedo commentedon Apr 28, 2022
Dear @BuddyVolly,
As always, I’m really thankful in all the work and effort everyone invests in this code, it really helps me a lot.
As @KBodolai suggested I tried really hard to implement a more flexible option using RegEx and came with a possible solution to transform MB to GB and make the proper sum. I’ll star a pull request (my first time ever, so hopefully I do it correctly); I’m going to add the mentioned script modification next:
Thanks both of you for your kind help.
Saludos cordiales.
Alexis A.
BuddyVolly commentedon Apr 28, 2022
Hi Alexis, no need for a PR. I think it can be simplified, but that piece of code already helps me
BuddyVolly commentedon Apr 28, 2022
can you check if this works for you?I only have GBs in my inventory
alexisahedo commentedon Apr 28, 2022
I'm sorry, I already made a PL, should I close it?.
When I tried to compile my working script it showed me the next error:
12rambau commentedon Apr 29, 2022
that's normal you're trying to divide a "str" by 1024. The result of the
re.subis a string so it needs to be casted tointfirst. Also why usingre.subinsted ofstr.replace?KBodolai commentedon Apr 29, 2022
yes, my idea for regex was avoiding the if / else statement by just pattern matching to the string bit, but hadn't thought about the scale factor, so it ends up not providing that much of an advantage, in which case just the string method is probably best, sorry for the confusion!