Repository navigation
UnicodeDecodeError when read() special characters - non-UTF-8 env [fixed] #14
Description
Hi Ken,
I've encountered an error while crawling the products from Lazada. The error is due to the "R" symbol inside a certain product name: Windows 10 Home (64-bit) NVIDIA®
Attaching the python codes below:
crawlProduct.txt
TagUI stopped once it hit the error, how can I handle the error so that it will continue to crawl the next product?
Best regards,
Yang Liu
Activity
kensoh commentedon Jun 22, 2019
Hi Yang Liu, this may be a Python / Pandas question instead of TagUI for Python question. When I tried running below (stripping away the pandas code), it works even with ® in the output.
modified code without pandas
output
kensoh commentedon Jun 22, 2019
Adding on, have tried your original code and run without errors. Also, forming a dataframe this way in below code I'm also able to run without issues. Can you try and let me know where does the error happen and what is the error message?
If error happens within Python can use normal try-catch. If error happens within TagUI, will need to know more replication details and example of error to see where is the part to dig further.
[-]Error handling when encountered undefined character[/-][+]Error handling when encountered undefined character - need replication / error details[/+][-]Error handling when encountered undefined character - need replication / error details[/-][+]Error handling when encountered undefined character - need eg of error[/+]kensoh commentedon Jun 22, 2019
Above was with Jupyter notebook. Also wasn't able to replicate using your original script from running via terminal using python3 command.
Until you can share the error details and how to replicate the environment / error, some other ideas you can try is using
# -*- coding: utf-8 -*-at top of your script.yangliu8912 commentedon Jun 23, 2019
Hi Ken,
Thank you so much for your reply. I've tried taking out the pandas code but I still encounter the same error. Please kindly see the error details below:
I've also tried using # -- coding: utf-8 -- at the first line of my script but it did not work.
Best regards,
Yang Liu
kensoh commentedon Jun 24, 2019
Thanks Yang Liu, for the error details! 👍 The difference in our env could be btw Anaconda installed Python vs direct installation, or maybe Windows vs my laptop (macOS).
But it has to be resolved within TagUI for Python. Have an idea where to tackle this, will test and try to deploy a new package version that can fix this and remain Python 2/3 compatible.
#14 - explicit unicode utf-8 encoding
kensoh commentedon Jun 24, 2019
Hi Yang Liu, can you try
pip install tagui --upgrade, restart Jupyter notebook kernel and try again? I was not able to replicate it in my environment but suspect the cause to be related to this.TagUI writes in UTF-8 but your laptop preferred encoding may be something else, causing reading special characters from TagUI to result in error. This v1.4 version explicitly use UTF-8 for both Python 2 and 3 environments, regardless of user's local preferred encoding.
Internal reference - made a commit with following comments
[-]Error handling when encountered undefined character - need eg of error[/-][+]Error handling when encountered special characters - non-UTF-8 env [fixed][/+]convexset commentedon Jun 24, 2019
The problem can be solved practically by:
convexset commentedon Jun 24, 2019
You just lose the characters that cannot be decoded.
kensoh commentedon Jun 25, 2019
Cool stuff, thanks @convexset ! This workaround is not implementable for now because at this stage, need to address new known gaps where possible. Losing characters or data without error feedback to user can lead to situations where some users run marathon jobs for more than 10 hours only to find out much later they missed some data and need to re-run. I'll check back with you again later on to see if you can commit a PR for edge cases that can't have addressable solutions.
[-]Error handling when encountered special characters - non-UTF-8 env [fixed][/-][+]UnicodeDecodeError when encounter special characters - non-UTF-8 env [fixed][/+][-]UnicodeDecodeError when encounter special characters - non-UTF-8 env [fixed][/-][+]UnicodeDecodeError when read() special characters - non-UTF-8 env [fixed][/+]yangliu8912 commentedon Jul 1, 2019
Hi Ken,
Sorry for the late reply as I was overseas and just came back to Singapore. Thank you for the fix, I've tested that it solves my issue : )
Best regards,
Yang Liu
kensoh commentedon Jul 1, 2019
Hi Yang Liu, no problem at all and welcome back to sunny island paradise :) Thanks for raising this and providing the replication details. This is an important issue to fix before the package gets used by more people, because it involves losing data and is a fundamental problem that has to be fixed.
If you encounter more edge cases issues let me know, and I see whether can put in something to address the problem. When nothing more can be done, then I'll invite @convexset to commit his solution to drop characters on unrecoverable data read error, and with a warning message.
wwwfzt commentedon Aug 20, 2022
still have the error when running the sample.py My env is a chinese windows 11 with python3.8.1
[RPA][ERROR] - 'utf-8' codec can't decode bytes in position 82-83: invalid continuation byte
wwwfzt commentedon Aug 20, 2022
It occurs in r.init()
kensoh commentedon Aug 20, 2022
Looks like there could be some characters that are not in UTF-8 format. Could you check if another computer or your friend can run this? And compare your PCs to see what setting to set to enable UTF-8. This package has been used by many Chinese users, so there should be a solution.