Skip to content

UnicodeDecodeError when work with MPD 0.19.x #49

Description

@handsomegui

I have lots of audio files in Chinese in my muisc_directory.

I use python-mpd to interact with MPD.
It works well with MPD 0.17.4, but after i upgrade to MPD 0.19.7, UnicodeDecodeError occurs.

With below code to show you how it happened:

handler = mpd.MPDClient(use_unicode=True)
handler.timeout = 10
handler.connect(host, port)

Below error triggered during execution: (While invoking: handler.playlistinfo() )

  File "C:/Users/gui/.code/xplayer/xplayer.py", line 532, in init_ui
    plsInfo = handler.playlistinfo()
  File "C:\Users\gui\.py2\lib\site-packages\mpd.py", line 600, in decorator
    return wrapper(self, name, args, bound_decorator(self, returnValue))
  File "C:\Users\gui\.py2\lib\site-packages\mpd.py", line 238, in _execute
    return retval()
  File "C:\Users\gui\.py2\lib\site-packages\mpd.py", line 595, in decorator
    return function(self, *args, **kwargs)
  File "C:\Users\gui\.py2\lib\site-packages\mpd.py", line 406, in _fetch_songs
    return self._fetch_objects(["file"])
  File "C:\Users\gui\.py2\lib\site-packages\mpd.py", line 394, in _fetch_objects
    return self._wrap_iterator(self._read_objects(delimiters))
  File "C:\Users\gui\.py2\lib\site-packages\mpd.py", line 358, in _wrap_iterator
    return list(iterator)
  File "C:\Users\gui\.py2\lib\site-packages\mpd.py", line 316, in _read_objects
    for key, value in self._read_pairs():
  File "C:\Users\gui\.py2\lib\site-packages\mpd.py", line 298, in _read_pairs
    pair = self._read_pair(separator)
  File "C:\Users\gui\.py2\lib\site-packages\mpd.py", line 286, in _read_pair
    line = self._read_line()
  File "C:\Users\gui\.py2\lib\site-packages\mpd.py", line 268, in _read_line
    line = decode_str(line)
  File "C:\Users\gui\.py2\lib\site-packages\mpd.py", line 32, in <lambda>
    decode_str = lambda s: s.decode("utf-8")
  File "C:\Users\gui\.py2\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xd0 in position 12: invalid continuation byte

Activity

  1. handsomegui commented on Dec 18, 2014

    @handsomegui
    Author

    Correction:
    I use python-mpd2 (0.5.4) to interact with MPD 0.19.7

  2. Mic92 commented on Dec 18, 2014

    @Mic92
    Owner

    Can you test at which version of mpd the code begins to break? 0.17.4 to 0.19.7 is more than a year of development. If you found the offending version, can you capture the network traffic from mpd (port 6600) for both the working and not working version (with Wireshark for example)?

    Or send me an example file name with broken encoding.

  3. handsomegui commented on Dec 19, 2014

    @handsomegui
    Author

    Please try the following names(In HKSCS):

    石崗.mp3
    赤鱲角.mp3
    

    The special part is that i use MPD on Windows platform (Window 7).
    By download the latest snapshot version at:
    http://www.musicpd.org/download/win32/snapshot/

    And i also test on the OSX platform with pretty much the same files by using python-mpd2 0.5.4 & MPD 0.19.7, all good without error reports.
    So i now think that maybe a bug of the upstream MPD specifically for the Windows port/version.

    Why 0.17.4 to 0.19.7?
    Because the last binary version for Windows platform is 0.17.4.
    So it will be difficult to test the version in the middle.

    I'm not sure it is a bug of MPD or python-mpd2, so i'm looking help from here.
    Thanks.

  4. multani commented on Dec 19, 2014

    @multani
    Collaborator

    You can try to setup a building environment and use git bisect between the 2 versions of mpd that you have been testing to see where the fault is coming from.
    I often read songs with Chinese characters and I never had a problem, even with latest mpd (on Linux).

    On December 19, 2014 6:23:49 PM GMT+01:00, Gui Zhenwei notifications@github.com wrote:

    Please try the following names(In HKSCS):

    石崗.mp3
    赤鱲角.mp3
    

    The special part is that i use MPD on Windows platform (Window 7).
    By download the latest snapshot version at:
    http://www.musicpd.org/download/win32/snapshot/

    And i also test on the OSX platform with pretty much the same files by
    using python-mpd2 0.5.4 & MPD 0.19.7, all good without error reports.
    So i now think that maybe a bug of the upstream MPD specifically for
    the Windows port/version.

    Why 0.17.4 to 0.19.7?
    Because the last binary version for Windows platform is 0.17.4.
    So it will be difficult to test the version in the middle.

    I'm not sure it is a bug of MPD or python-mpd2, so i'm looking help
    from here.
    Thanks.


    Reply to this email directly or view it on GitHub:
    #49 (comment)

  5. Mic92 commented on Dec 20, 2014

    @Mic92
    Owner

    @handsomegui If you cannot compile mpd on windows, could you open this bug at the mpd bugtracker? If you provide a failing python script and sample data, they can easily use git bisect to track down this bug.

  6. handsomegui commented on Dec 21, 2014

    @handsomegui
    Author

    @Mic92 @multani Sure. Will do that. Thanks for the suggestion. Please close this thread as need.

  7. handsomegui commented on Dec 28, 2014

    @handsomegui
    Author

    @Mic92 @multani

    I've reported this issue to the bug tracker of MPD:
    http://bugs.musicpd.org/view.php?id=4259

  8. handsomegui commented on Dec 28, 2014

    @handsomegui
    Author

    No problem for running command with mpc client:

    gui@WIN7 /C/mpd_files
    $ mpc playlist
    赤鱲角2.mp3
    

    But will reports error with python-mpd2 (0.5.4) client.

  9. handsomegui commented on Dec 28, 2014

    @handsomegui
    Author

    As a side note:
    For the same characters, no problem on Linux & OSX platforms.
    It is Windows platform specifically.

  10. handsomegui commented on Dec 28, 2014

    @handsomegui
    Author
  11. handsomegui commented on Dec 30, 2014

    @handsomegui
    Author

    @Mic92 @multani
    The core developer of MPD cirrus wrote this on the bugtracker:

    Give me raw data from the wire. I don't know the Python code you're using, and I don't want to waste time on learning about it. All I know is MPD.
    

    Could you guys explain to me what is the row data from the wire? And how can i get that then post to him? Thanks in advance.

  12. multani commented on Dec 30, 2014

    @multani
    Collaborator

    The raw data from the wire refers to the data which is sent on the
    socket when the client is talking to the server.

    There's nothing in python-mpd2 to display the exact data sent to the
    server, but you can hack into MPDClient._write_line() and probably the
    "raw data" is the line argument passed to this method, with the
    additional \n added.
    You can change it to something like this to get what you want:

        def _write_line(self, line):
            line = "%s\n" % line
            print("Raw data: %s" % repr(line))
            self._wfile.write(line)
            self._wfile.flush()
    
    

    You can do about the same on the "receive" side, look up the _rfile
    attribute to see where the data is being read from.

  13. handsomegui commented on Jan 6, 2015

    @handsomegui
    Author

    @multani
    Thank you for the help. I will try that later.

    For current phase, i am confused by the following tests.
    I'm still not sure it is a bug of MPD or bug of python-mpd2.

    See the mpc.exe part, which works well as a client, under the same condition.
    It prints out non-ascii characters (Traditional Chinese in HKSCS) correctly after issuing commands.

    But under the same condition, python-mpd2 will report UnicodeDecodeError.
    Could someone tell me more on this?

    Server Side
    I downloaded mpd.exe from: http://www.musicpd.org/download/win32/snapshot/
    Then execute commands in git bash command prompt:

    $ mpd.exe --version
    Music Player Daemon 0.20 (v0.19.6-209-ge22df43-dirty)
    ...
    
    $ ls
    mpd.conf  mpd.exe  mpd.exe.sig
    
    $ mpd.exe -v mpd.conf
    config_file: loading file mpd.conf
    Jan 06 20:24 : vorbis: Xiph.Org libVorbis 1.3.4
    Jan 06 20:24 : opus: libopus 1.1
    Jan 06 20:24 : db: Database corrupted
    Jan 06 20:24 : curl: version 7.39.0
    Jan 06 20:24 : ffmpeg: Input plugin 'ffmpeg' is unavailable: No protocol
    
    

    Client Side by mpc.exe
    I downloaded mpc.exe from: http://www.musicpd.org/download/mpc/0/mpc-0.20-win32.zip
    then:

    $ mpc.exe ls
    赤鱲角2.mp3
    
    $ mpc.exe playlist
    赤鱲角2.mp3
    
    
    

    Client Side by Python-MPD2
    Executing commands in IPython 2.3.1 with python-mpd2 0.5.4:

    In [1]: import mpd
    
    In [2]: c = mpd.MPDClient(use_unicode=True)
    
    In [3]: c.connect('localhost', 6600)
    
    In [4]: c.mpd_version
    Out[4]: '0.19.0'
    
    In [5]: c.status()
    Out[5]:
    {u'consume': u'0',
     u'mixrampdb': u'0.000000',
     u'playlist': u'2',
     u'playlistlength': u'1',
     u'random': u'0',
     u'repeat': u'0',
     u'single': u'0',
     u'state': u'stop',
     u'volume': u'-1'}
    
    In [6]: c.playlistinfo()
    ---------------------------------------------------------------------------
    UnicodeDecodeError                        Traceback (most recent call last)
    <ipython-input-6-d775747f0040> in <module>()
    ----> 1 c.playlistinfo()
    
    c:\Python27\lib\site-packages\mpd.pyc in decorator(self, *args)
        598 def newFunction(wrapper, name, returnValue):
        599     def decorator(self, *args):
    --> 600         return wrapper(self, name, args, bound_decorator(self, returnValue))
        601     return decorator
        602
    
    c:\Python27\lib\site-packages\mpd.pyc in _execute(self, command, args, retval)
        236             self._write_command(command, args)
        237             if isinstance(retval, Callable):
    --> 238                 return retval()
        239             return retval
        240
    
    c:\Python27\lib\site-packages\mpd.pyc in decorator(*args, **kwargs)
        593
        594     def decorator(*args, **kwargs):
    --> 595         return function(self, *args, **kwargs)
        596     return decorator
        597
    
    c:\Python27\lib\site-packages\mpd.pyc in _fetch_songs(self)
        404
        405     def _fetch_songs(self):
    --> 406         return self._fetch_objects(["file"])
        407
        408     def _fetch_mounts(self):
    
    c:\Python27\lib\site-packages\mpd.pyc in _fetch_objects(self, delimiters)
        392
        393     def _fetch_objects(self, delimiters):
    --> 394         return self._wrap_iterator(self._read_objects(delimiters))
        395
        396     def _fetch_changes(self):
    
    c:\Python27\lib\site-packages\mpd.pyc in _wrap_iterator(self, iterator)
        356     def _wrap_iterator(self, iterator):
        357         if not self.iterate:
    --> 358             return list(iterator)
        359         self._iterating = True
        360         return self._iterator_wrapper(iterator)
    
    c:\Python27\lib\site-packages\mpd.pyc in _read_objects(self, delimiters)
        314     def _read_objects(self, delimiters=[]):
        315         obj = {}
    --> 316         for key, value in self._read_pairs():
        317             key = key.lower()
        318             if obj:
    
    c:\Python27\lib\site-packages\mpd.pyc in _read_pairs(self, separator)
        293
        294     def _read_pairs(self, separator=": "):
    --> 295         pair = self._read_pair(separator)
        296         while pair:
        297             yield pair
    
    c:\Python27\lib\site-packages\mpd.pyc in _read_pair(self, separator)
        284
        285     def _read_pair(self, separator):
    --> 286         line = self._read_line()
        287         if line is None:
        288             return
    
    c:\Python27\lib\site-packages\mpd.pyc in _read_line(self)
        266         line = self._rfile.readline()
        267         if self.use_unicode:
    --> 268             line = decode_str(line)
        269         if not line.endswith("\n"):
        270             self.disconnect()
    
    c:\Python27\lib\site-packages\mpd.pyc in <lambda>(s)
         30 IS_PYTHON2 = sys.version_info < (3, 0)
         31 if IS_PYTHON2:
    ---> 32     decode_str = lambda s: s.decode("utf-8")
         33     encode_str = lambda s: s if type(s) == str else (unicode(s)).encode("utf-8")
         34 else:
    
    c:\Python27\lib\encodings\utf_8.pyc in decode(input, errors)
         14
         15 def decode(input, errors='strict'):
    ---> 16     return codecs.utf_8_decode(input, errors, True)
         17
         18 class IncrementalEncoder(codecs.IncrementalEncoder):
    
    UnicodeDecodeError: 'utf8' codec can't decode byte 0xb3 in position 6: invalid start byte
    
    In [7]:
    
  14. multani commented on Jan 6, 2015

    @multani
    Collaborator

    @handsomegui I created a branch debug-utf8 on https://github.com/multani/python-mpd2/

    Can you run the same test as you did and post here what it displays before the traceback (something like Unable to decode data ...)

    Without this at least, or better yet, the raw data from the wire as asked before, I don't think we will be able to solve the problem.

  15. Mic92 commented on Jan 6, 2015

    @Mic92
    Owner

    I expect that mpc does not handle utf-8 at all and just treat the output of mpd as series of bytes and your terminal does the right job interpreting bytes correct. I added however decoding/encoding of utf-8 strings from bytes natively in the library, because otherwise every user of the library would have to handle this on his own. The error thrown by s.decode("utf-8") indicates that encoding of MPD is not valid utf-8 (http://en.wikipedia.org/wiki/UTF-8#Invalid_byte_sequences). I am not an expert on encoding, but NTFS on Windows uses UTF-16 instead of UTF-8. Probably there is a conversion bug when reading from the filesystem.

  16. handsomegui commented on Jan 6, 2015

    @handsomegui
    Author

    @multani @Mic92 Thank you very much for the help.

    @multani I cloned your debug-utf8 branch from the url your provided.
    Here is the information i got:

    Welcome to Git (version 1.9.4-preview20140929)
    
    
    Run 'git help git' to display the help index.
    Run 'git help <command>' to display help for specific commands.
    
    gui@WIN7 /C/mpd_files
    $ ipython
    Python 2.7.9 (default, Dec 10 2014, 12:28:03) [MSC v.1500 64 bit (AMD64)]
    Type "copyright", "credits" or "license" for more information.
    
    IPython 2.3.1 -- An enhanced Interactive Python.
    ?         -> Introduction and overview of IPython's features.
    %quickref -> Quick reference.
    help      -> Python's own help system.
    object?   -> Details about 'object', use 'object??' for extra details.
    
    In [1]: import mpd
    
    In [2]: mpd.VERSION
    Out[2]: (0, 5, 4)
    
    In [3]: c = mpd.MPDClient(use_unicode=True)
    
    In [4]: c.connect('localhost', 6600)
    
    In [5]: c.mpd_version
    Out[5]: '0.19.0'
    
    In [6]: c.status()
    Out[6]:
    {u'consume': u'0',
     u'mixrampdb': u'0.000000',
     u'playlist': u'1',
     u'playlistlength': u'0',
     u'random': u'0',
     u'repeat': u'0',
     u'single': u'0',
     u'state': u'stop',
     u'volume': u'-1'}
    
    In [7]: c.playlistinfo()
    Out[7]: []
    
    In [8]: c.listall()
    Unable to decode data 'file: \xb3\xe0\xf7v\xbd\xc72.mp3\n': 'utf8' codec can't decode byte 0xb3 in position 6: invalid s
    tart byte
    ---------------------------------------------------------------------------
    UnicodeDecodeError                        Traceback (most recent call last)
    <ipython-input-8-7362a016e585> in <module>()
    ----> 1 c.listall()
    
    c:\Python27\lib\site-packages\python_mpd2-0.5.4-py2.7.egg\mpd.pyc in decorator(self, *args)
        602 def newFunction(wrapper, name, returnValue):
        603     def decorator(self, *args):
    --> 604         return wrapper(self, name, args, bound_decorator(self, returnValue))
        605     return decorator
        606
    
    c:\Python27\lib\site-packages\python_mpd2-0.5.4-py2.7.egg\mpd.pyc in _execute(self, command, args, retval)
        236             self._write_command(command, args)
        237             if isinstance(retval, Callable):
    --> 238                 return retval()
        239             return retval
        240
    
    c:\Python27\lib\site-packages\python_mpd2-0.5.4-py2.7.egg\mpd.pyc in decorator(*args, **kwargs)
        597
        598     def decorator(*args, **kwargs):
    --> 599         return function(self, *args, **kwargs)
        600     return decorator
        601
    
    c:\Python27\lib\site-packages\python_mpd2-0.5.4-py2.7.egg\mpd.pyc in _fetch_database(self)
        420
        421     def _fetch_database(self):
    --> 422         return self._fetch_objects(["file", "directory", "playlist"])
        423
        424     def _fetch_messages(self):
    
    c:\Python27\lib\site-packages\python_mpd2-0.5.4-py2.7.egg\mpd.pyc in _fetch_objects(self, delimiters)
        396
        397     def _fetch_objects(self, delimiters):
    --> 398         return self._wrap_iterator(self._read_objects(delimiters))
        399
        400     def _fetch_changes(self):
    
    c:\Python27\lib\site-packages\python_mpd2-0.5.4-py2.7.egg\mpd.pyc in _wrap_iterator(self, iterator)
        360     def _wrap_iterator(self, iterator):
        361         if not self.iterate:
    --> 362             return list(iterator)
        363         self._iterating = True
        364         return self._iterator_wrapper(iterator)
    
    c:\Python27\lib\site-packages\python_mpd2-0.5.4-py2.7.egg\mpd.pyc in _read_objects(self, delimiters)
        318     def _read_objects(self, delimiters=[]):
        319         obj = {}
    --> 320         for key, value in self._read_pairs():
        321             key = key.lower()
        322             if obj:
    
    c:\Python27\lib\site-packages\python_mpd2-0.5.4-py2.7.egg\mpd.pyc in _read_pairs(self, separator)
        297
        298     def _read_pairs(self, separator=": "):
    --> 299         pair = self._read_pair(separator)
        300         while pair:
        301             yield pair
    
    c:\Python27\lib\site-packages\python_mpd2-0.5.4-py2.7.egg\mpd.pyc in _read_pair(self, separator)
        288
        289     def _read_pair(self, separator):
    --> 290         line = self._read_line()
        291         if line is None:
        292             return
    
    c:\Python27\lib\site-packages\python_mpd2-0.5.4-py2.7.egg\mpd.pyc in _read_line(self)
        267         if self.use_unicode:
        268             try:
    --> 269                 line = decode_str(line)
        270             except UnicodeDecodeError as exc:
        271                 print("Unable to decode data %r: %s" % (line, exc))
    
    c:\Python27\lib\site-packages\python_mpd2-0.5.4-py2.7.egg\mpd.pyc in <lambda>(s)
         30 IS_PYTHON2 = sys.version_info < (3, 0)
         31 if IS_PYTHON2:
    ---> 32     decode_str = lambda s: s.decode("utf-8")
         33     encode_str = lambda s: s if type(s) == str else (unicode(s)).encode("utf-8")
         34 else:
    
    c:\Python27\lib\encodings\utf_8.pyc in decode(input, errors)
         14
         15 def decode(input, errors='strict'):
    ---> 16     return codecs.utf_8_decode(input, errors, True)
         17
         18 class IncrementalEncoder(codecs.IncrementalEncoder):
    
    UnicodeDecodeError: 'utf8' codec can't decode byte 0xb3 in position 6: invalid start byte
    
    In [9]:
    
  17. handsomegui commented on Jan 6, 2015

    @handsomegui
    Author

    The error part is:

    Unable to decode data 'file: \xb3\xe0\xf7v\xbd\xc72.mp3\n': 'utf8' codec can't decode byte 0xb3 in position 6: invalid start byte
    

    And by print \xb3\xe0\xf7v\xbd\xc72.mp3 in python it will be: 赤鱲角2.mp3

    Which is in Traditional Chinese. (More precisely, in HKSCS)

    About HKSCS:
    http://www.microsoft.com/hk/hkscs/
    http://me.abelcheung.org/articles/research/what-is-cp951/

  18. handsomegui commented on Jan 6, 2015

    @handsomegui
    Author

    By trying some other file name in Simplified Chinese (Not in Traditional Chinese & HKSCS).
    I got this instead (a bit different than the previous report):

    
    In [6]: c.listall()
    Unable to decode data 'file: \xd6\xd2\xd3\xda\xb0\xae\xc7\xe9.mp3\n': 'utf8' codec can't decode byte 0xd6 in position 6:
     invalid continuation byte
    ---------------------------------------------------------------------------
    UnicodeDecodeError                        Traceback (most recent call last)
    <ipython-input-6-7362a016e585> in <module>()
    ----> 1 c.listall()
    
    c:\Python27\lib\site-packages\python_mpd2-0.5.4-py2.7.egg\mpd.pyc in decorator(self, *args)
        602 def newFunction(wrapper, name, returnValue):
        603     def decorator(self, *args):
    --> 604         return wrapper(self, name, args, bound_decorator(self, returnValue))
        605     return decorator
        606
    
    c:\Python27\lib\site-packages\python_mpd2-0.5.4-py2.7.egg\mpd.pyc in _execute(self, command, args, retval)
        236             self._write_command(command, args)
        237             if isinstance(retval, Callable):
    --> 238                 return retval()
        239             return retval
        240
    
    c:\Python27\lib\site-packages\python_mpd2-0.5.4-py2.7.egg\mpd.pyc in decorator(*args, **kwargs)
        597
        598     def decorator(*args, **kwargs):
    --> 599         return function(self, *args, **kwargs)
        600     return decorator
        601
    
    c:\Python27\lib\site-packages\python_mpd2-0.5.4-py2.7.egg\mpd.pyc in _fetch_database(self)
        420
        421     def _fetch_database(self):
    --> 422         return self._fetch_objects(["file", "directory", "playlist"])
        423
        424     def _fetch_messages(self):
    
    c:\Python27\lib\site-packages\python_mpd2-0.5.4-py2.7.egg\mpd.pyc in _fetch_objects(self, delimiters)
        396
        397     def _fetch_objects(self, delimiters):
    --> 398         return self._wrap_iterator(self._read_objects(delimiters))
        399
        400     def _fetch_changes(self):
    
    c:\Python27\lib\site-packages\python_mpd2-0.5.4-py2.7.egg\mpd.pyc in _wrap_iterator(self, iterator)
        360     def _wrap_iterator(self, iterator):
        361         if not self.iterate:
    --> 362             return list(iterator)
        363         self._iterating = True
        364         return self._iterator_wrapper(iterator)
    
    c:\Python27\lib\site-packages\python_mpd2-0.5.4-py2.7.egg\mpd.pyc in _read_objects(self, delimiters)
        318     def _read_objects(self, delimiters=[]):
        319         obj = {}
    --> 320         for key, value in self._read_pairs():
        321             key = key.lower()
        322             if obj:
    
    c:\Python27\lib\site-packages\python_mpd2-0.5.4-py2.7.egg\mpd.pyc in _read_pairs(self, separator)
        297
        298     def _read_pairs(self, separator=": "):
    --> 299         pair = self._read_pair(separator)
        300         while pair:
        301             yield pair
    
    c:\Python27\lib\site-packages\python_mpd2-0.5.4-py2.7.egg\mpd.pyc in _read_pair(self, separator)
        288
        289     def _read_pair(self, separator):
    --> 290         line = self._read_line()
        291         if line is None:
        292             return
    
    c:\Python27\lib\site-packages\python_mpd2-0.5.4-py2.7.egg\mpd.pyc in _read_line(self)
        267         if self.use_unicode:
        268             try:
    --> 269                 line = decode_str(line)
        270             except UnicodeDecodeError as exc:
        271                 print("Unable to decode data %r: %s" % (line, exc))
    
    c:\Python27\lib\site-packages\python_mpd2-0.5.4-py2.7.egg\mpd.pyc in <lambda>(s)
         30 IS_PYTHON2 = sys.version_info < (3, 0)
         31 if IS_PYTHON2:
    ---> 32     decode_str = lambda s: s.decode("utf-8")
         33     encode_str = lambda s: s if type(s) == str else (unicode(s)).encode("utf-8")
         34 else:
    
    c:\Python27\lib\encodings\utf_8.pyc in decode(input, errors)
         14
         15 def decode(input, errors='strict'):
    ---> 16     return codecs.utf_8_decode(input, errors, True)
         17
         18 class IncrementalEncoder(codecs.IncrementalEncoder):
    
    UnicodeDecodeError: 'utf8' codec can't decode byte 0xd6 in position 6: invalid continuation byte
    
    In [7]:
    
  19. multani commented on Jan 6, 2015

    @multani
    Collaborator

    So, it seems MPD sends these files with the hkscs encoding, then?:

    Python 2.7.9 (default, Dec 12 2014, 06:07:10) 
    [GCC 4.9.2] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> x = b'file: \xb3\xe0\xf7v\xbd\xc72.mp3\n'
    >>> x.decode('utf-8')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
        return codecs.utf_8_decode(input, errors, True)
    UnicodeDecodeError: 'utf8' codec can't decode byte 0xb3 in position 6: invalid start byte
    >>> x.decode('utf-16')
    u'\u6966\u656c\u203a\ue0b3\u76f7\uc7bd\u2e32\u706d\u0a33'
    >>> print x.decode('utf-16')
    楦敬›盷잽⸲灭ਲ਼
    >>> print x.decode('hkscs')
    file: 喪饔褒2.mp3
    
    >>> x = b'file: \xd6\xd2\xd3\xda\xb0\xae\xc7\xe9.mp3\n'
    >>> print x.decode('hkscs')
    file: 笳衾乾ロ.mp3
    
    

    What if you connect and your MPD server using something like telnet, or nc and executes listall "", can you send here what it returns (skipping some lines if there are too many lines).

  20. handsomegui commented on Jan 6, 2015

    @handsomegui
    Author

    @multani Thank you for your patience. I will do the telnet test later tomorrow, sleeping time for my timezone:)

    For the HKSCS character file name(赤鱲角2.mp3),
    I followed your test, with a little changes, my output is:

    Python 2.7.9 (default, Dec 10 2014, 12:28:03) [MSC v.1500 64 bit (AMD64)] on win32
    Type "copyright", "credits" or "license()" for more information.
    >>> x = b'file: \xb3\xe0\xf7v\xbd\xc72.mp3\n'
    >>> type(x)
    <type 'str'>
    >>> print x
    file: 赤鱲角2.mp3
    
    >>> x.decode('utf-8')
    
    Traceback (most recent call last):
      File "<pyshell#3>", line 1, in <module>
        x.decode('utf-8')
      File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode
        return codecs.utf_8_decode(input, errors, True)
    UnicodeDecodeError: 'utf8' codec can't decode byte 0xb3 in position 6: invalid start byte
    >>> import sys
    >>> sys.getfilesystemencoding()
    'mbcs'
    >>> fs_encoding = sys.getfilesystemencoding()
    >>> x.decode(fs_encoding)
    u'file: \u8d64\u9c72\u89d22.mp3\n'
    >>> print x.decode(fs_encoding)
    file: 赤鱲角2.mp3
    
    

    From the above output, i saw print as string or print as unicode is all good. (Except decode as utf-8)

    If decode as utf-16, and decode as hkscs, my output will be the same as yours.
    No error, but wrong display(results). (The correct output should be: file: 赤鱲角2.mp3 as above.)

  21. handsomegui commented on Jan 6, 2015

    @handsomegui
    Author

    The second part for the Simplified Chinese file name, NOT hkscs characters.

    >>> x = b'file: \xd6\xd2\xd3\xda\xb0\xae\xc7\xe9.mp3\n'
    >>> type(x)
    <type 'str'>
    >>> print x
    file: 忠于爱情.mp3
    
    >>> print x.decode(sys.getfilesystemencoding())
    file: 忠于爱情.mp3
    
    >>> sys.getfilesystemencoding()
    'mbcs'
    >>> print x.decode('hkscs')    # The result will be wrong.
    file: 笳衾乾ロ.mp3
    
  22. handsomegui commented on Jan 6, 2015

    @handsomegui
    Author

    @Mic92 @multani

    OMG !!!
    I just found if i connect MPD without using use_unicode=True, the result will be correct !!!

    Python 2.7.9 (default, Dec 10 2014, 12:28:03) [MSC v.1500 64 bit (AMD64)] on win32
    Type "copyright", "credits" or "license()" for more information.
    >>> import mpd
    >>> mpd.VERSION
    (0, 5, 4)
    >>> c = mpd.MPDClient()    # SEE THIS LINE. WITHOUT: use_unicode=True
    >>> c.connect('localhost', 6600)
    >>> c.mpd_version
    '0.19.0'
    >>> c.status()
    {'playlistlength': '0', 'playlist': '1', 'repeat': '0', 'consume': '0', 'mixrampdb': '0.000000', 'random': '0', 'state': 'stop', 'volume': '-1', 'single': '0'}
    >>> c.listall()
    [{'file': '\xb3\xe0\xf7v\xbd\xc72.mp3'}, {'file': '\xd6\xd2\xd3\xda\xb0\xae\xc7\xe9.mp3'}]
    >>> for i in c.listall():
        print i['file']
    
    
    赤鱲角2.mp3
    忠于爱情.mp3
    >>> 
    

    So what does this mean? Data from MPD can NOT be unicode?

  23. Mic92 commented on Jan 6, 2015

    @Mic92
    Owner

    If you remove the unicode-option, the library will not attempt to decode the byte strings as unicode ("string" instead of u"string"). Removing this option break your player, if you want to operate on text like substrings, sorting, string length, as it operate on byte level instead of character level. Actually this is a bug in MPD because they claim to always use UTF-8 as of this document: http://www.musicpd.org/doc/protocol/request_syntax.html Actually this behaviour should be pretty easy fixable by the MPD devs

    By the way, is your filesystem_charset option in mpd.conf correct?

  24. handsomegui commented on Mar 5, 2015

    @handsomegui
    Author

    @Mic92 @multani

    The bug has been fixed by the upstream MPD core developer TODAY.

    Reference info:
    Bug reports here: http://bugs.musicpd.org/view.php?id=4311
    Fixed binary for Windows: http://www.musicpd.org/download/win32/snapshot/
    The above binary was uploaded at 2015-03-05.

    Thanks to you guys for all the nice help.

  25. changed the title [-]UnicodeDecodeError when work with MPD 0.19.7[/-] [+]UnicodeDecodeError when work with MPD 0.19.x[/+] on Mar 5, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions