Repository navigation
Error getting BOM file from poetry #448
Description
I have a project in which I use poetry, and I want to be able to obtain the BOM file with the poetry dependencies.
When I run the following command: cyclonedx-py --poetry --output bom.xml
I get the following error:
Traceback (most recent call last):
File "C:\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\jcmencia\PycharmProjects\QA-Pruebas\py_requests\venv\Scripts\cyclonedx-py.exe\__main__.py", line 7, in <module>
File "C:\Users\jcmencia\PycharmProjects\QA-Pruebas\py_requests\venv\lib\site-packages\cyclonedx_py\client.py", line 307, in main
CycloneDxCmd(args).execute()
File "C:\Users\jcmencia\PycharmProjects\QA-Pruebas\py_requests\venv\lib\site-packages\cyclonedx_py\client.py", line 147, in execute
output = self.get_output()
File "C:\Users\jcmencia\PycharmProjects\QA-Pruebas\py_requests\venv\lib\site-packages\cyclonedx_py\client.py", line 84, in get_output
parser = self._get_input_parser()
File "C:\Users\jcmencia\PycharmProjects\QA-Pruebas\py_requests\venv\lib\site-packages\cyclonedx_py\client.py", line 282, in _get_input_parser
input_data = input_data_fh.read()
File "C:\Python310\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 6935: character maps to <undefined>
The poetry.lock file is in UTF-8 format and with LF line endings and I use Windows with PyCharm
If I edit in the client.py file the following line:
self._arguments.input_source = open(os.path.join(current_directory, 'poetry.lock'), 'r')And I add the encoding parameter:
self._arguments.input_source = open(os.path.join(current_directory, 'poetry.lock'), 'r', encoding="utf-8")The command works perfectly for me.
Is there any configuration I can do to make it use UTF-8 encoding instead of opting for cp1252? Or any solution for this problem? Everything I have tried has not worked.
Thanks in advance!
Activity
[-]Error getting file from poetry[/-][+]Error getting BOM file from poetry[/+]JCHacking commentedon Nov 30, 2022
Could a parameter be added to the command like --utf-8 to add that extra parameter to open?
So if my application requires not to use PYTHONUTF8 I can use this library.
jkowalleck commentedon Dec 1, 2022
my idea to the solution:
i would prefer to have char-detect, rather than hardcoding the char-encoding or having a user to decide.
Need to think about it and need to prepare reproducible envs for testing, first.
JCHacking commentedon Oct 12, 2023
Hi @jkowalleck
I have opened a pull request #596 with the solution to this (Autodetecting characters). I have tested it on my Windows system and it works correctly so I think it is solved.
Regards!!!
jkowalleck commentedon Oct 12, 2023
oh, great.
I will see if I can provide a CT/CI env to reproduce the issue, so that i can use it as a regression test env
jkowalleck commentedon Oct 12, 2023
re #448 (comment)
re #448 (comment)
@JCHacking could you help me reproducing the issue?
maybe provide the input file that caused your issue?
JCHacking commentedon Oct 12, 2023
@jkowalleck
The problem is not in the file. I will try to explain it as best I can.
The Python open function seems to work differently on Windows systems than on Unix systems, by modifying the default encoding with which you open it.
What this causes is that in Windows systems when you try to open the poetry.lock file you will find a character that does not exist in Windows-1252.
The applied solution skips this encoding, since when opening it in byte mode it does not have to encode it.
Then when we evaluate the content and determine the encoding, is when we pass it to string (In python 3 the strings already work in UTF-8, so it will always be in the same format).
Now, to be able to put a test of this scenario there is a problem, and is that each character that there is in Windows-1252 exists in UTF-8, so you can not put a file with X character "rare" and make it fail without this solution, because the container of the test (I understand that they are made with github actions and that probably is a linux container) will open the file as UTF-8.
In this scenario I can think of 2 possible solutions:
If you need me to investigate to apply the second solution I would have no problem (the first one would cost me more because I don't know how github actions work, since I haven't worked with it yet).
jkowalleck commentedon Oct 19, 2023
Discussion:
The more I read about the issue, the more I think it is a bad idea to use dynamic encoding detection for files that are written by machines and read by machines.
This would include:
Pipfile.lock,poetry.lockThis would not include
requirements.txtSince we must handle one of the files file, then why not also handle the others.
Your approach appears reasonable, but might get optimized: only char-detect where needed.
JCHacking commentedon Oct 19, 2023
In this way, by validating the encoding of each file we could also allow the client to modify one of the locks manually and without its editor realizing it, change the encoding.
This is rare but it could happen. Thus, we also allow that any poetry or library decides the encoding of its files, since as we auto-detect it, it would not cause any problems with those changes.
I think it will help us avoid having to make absurd changes due to external factors.
jkowalleck commentedon Oct 19, 2023
this is no intended use case of lock files. If they modify it, they need to keep the standard encoding.
We do not want to find excuses for any implementation. Rather than implementing as use cases appear.
for
requirement.txtit often happens that they are written manually. Here, I totally see your point.fix: input file encoding (#596)