python-memcached return wrong value after crash??

I'm getting the following issue on trying to run this command on IPython. I see python-memcached return wrong value after crash.

Environment

MacOS
Python 3.7.4
IPython 7.7.0
python-memcached 1.59

Test scenario:
Run IPython and enter the following commands

bash-3.2$ ipython
Python 3.7.4 (default, Jul  9 2019, 18:13:23)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.7.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from memcache import Client as Memcache
   ...: CACHE = Memcache(['127.0.0.1:11211'], socket_timeout=0.5)

In [2]: exit
bash-3.2$ clear
bash-3.2$ ipython
Python 3.7.4 (default, Jul  9 2019, 18:13:23)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.7.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from memcache import Client as Memcache
In [2]: CACHE = Memcache(['127.0.0.1:11211'], socket_timeout=0.5)
In [3]: CACHE.flush_all()
In [4]: for i in range (1, 11):
   ...:     print(CACHE.set('sam-%s' % i, i))
   ...:
True
True
True
True
True
True
True
True
True
True

In [5]: for i in range (1, 11):
   ...:     print('sam-%s = %s' % (i, CACHE.get('sam-%s' % i)))
   ...:
sam-1 = 1
sam-2 = 2
sam-3 = 3
sam-4 = 4
sam-5 = 5
sam-6 = 6
sam-7 = 7
sam-8 = 8
sam-9 = 9
sam-10 = 10

In [6]: for i in range(1, 1000):
   ...:     # Run for a while, press Ctrl+C to raise crash!
   ...:     print(CACHE.set('interrupt-%s' % i, i))
   ...:
True
True
True
True
True
True
^C---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
<ipython-input-6-b57eaab76114> in <module>
      1 for i in range(1, 1000):
      2     # Run for a while, press Ctrl+C to raise crash!
----> 3     print(CACHE.set('interrupt-%s' % i, i))
      4

/usr/local/lib/python3.7/site-packages/python_memcached-1.59-py3.7.egg/memcache.py in set(self, key, val, time, min_compress_len, noreply)
    725         send the reply.
    726         '''
--> 727         return self._set("set", key, val, time, min_compress_len, noreply)
    728
    729     def cas(self, key, val, time=0, min_compress_len=0, noreply=False):

/usr/local/lib/python3.7/site-packages/python_memcached-1.59-py3.7.egg/memcache.py in _set(self, cmd, key, val, time, min_compress_len, noreply)
   1050
   1051         try:
-> 1052             return _unsafe_set()
   1053         except _ConnectionDeadError:
   1054             # retry once

/usr/local/lib/python3.7/site-packages/python_memcached-1.59-py3.7.egg/memcache.py in _unsafe_set()
   1042                 if noreply:
   1043                     return True
-> 1044                 return server.expect(b"STORED", raise_exception=True) == b"STORED"
   1045             except socket.error as msg:
   1046                 if isinstance(msg, tuple):

/usr/local/lib/python3.7/site-packages/python_memcached-1.59-py3.7.egg/memcache.py in expect(self, text, raise_exception)
   1461
   1462     def expect(self, text, raise_exception=False):
-> 1463         line = self.readline(raise_exception)
   1464         if self.debug and line != text:
   1465             if six.PY3:

/usr/local/lib/python3.7/site-packages/python_memcached-1.59-py3.7.egg/memcache.py in readline(self, raise_exception)
   1447             if index >= 0:
   1448                 break
-> 1449             data = recv(4096)
   1450             if not data:
   1451                 # connection close, let's kill it and raise

KeyboardInterrupt:

In [7]: # python-memcached return wrong value!!!
In [8]: for i in range (1, 11):
   ...:     print('sam-%s = %s' % (i, CACHE.get('sam-%s' % i)))
   ...:
sam-1 = None
sam-2 = 1
sam-3 = 2
sam-4 = 3
sam-5 = 4
sam-6 = 5
sam-7 = 6
sam-8 = 7
sam-9 = 8
sam-10 = 9

In [9]: # python-memcached return wrong value!!!
In [10]: for i in range (1, 11):
    ...:     print('sam-%s = %s' % (i, CACHE.get('sam-%s' % i)))
    ...:
sam-1 = 10
sam-2 = 1
sam-3 = 2
sam-4 = 3
sam-5 = 4
sam-6 = 5
sam-7 = 6
sam-8 = 7
sam-9 = 8
sam-10 = 9

I see that after crash python-memcached still keep connection to Memcached but function get() set() have something wrong.

Has anyone run into the same problem? Is this a bug of python-memcached?

I ran into the same problem, and it's a problem with python-memcached. I made a python script based on your ipython commands that recreates the bug on most runs.


import time

from multiprocessing import Process
from memcache import Client as Memcache


CACHE = Memcache(['127.0.0.1:11211'], socket_timeout=0.5)

# Set some values in the cache
print("Setting!")
for i in range (1, 11):
    CACHE.set('sam-%s' % i, i)
    print('set: sam-%s = %s' % (i,i))

# Get's all the set values to ensure they are saved
print("Getting!")
for i in range (1, 11):
    print('get: sam-%s: %s' % (i,CACHE.get('sam-%s' % i)))


def worker():
  print('Function started')
  for i in range(1000):
      CACHE.set('interrupt-%s' % i, i)
  print('Function finished')

# Start a new process and kill it while it's setting values in the cache
p = Process(target=worker)
p.start()
time.sleep(0.1)
p.terminate()
print("process killed")

# Get the values again
print("Getting!")
for i in range (1, 11):
    print('get: sam-%s: %s' % (i,CACHE.get('sam-%s' % i)))

The output should look like this:

Setting!
set: sam-1 = 1
set: sam-2 = 2
set: sam-3 = 3
set: sam-4 = 4
set: sam-5 = 5
set: sam-6 = 6
set: sam-7 = 7
set: sam-8 = 8
set: sam-9 = 9
set: sam-10 = 10
Getting!
get: sam-1: 1
get: sam-2: 2
get: sam-3: 3
get: sam-4: 4
get: sam-5: 5
get: sam-6: 6
get: sam-7: 7
get: sam-8: 8
get: sam-9: 9
get: sam-10: 10
Function started
process killed
Getting!
get: sam-1: None
get: sam-2: 1
get: sam-3: 2
get: sam-4: 3
get: sam-5: 4
get: sam-6: 5
get: sam-7: 6
get: sam-8: 7
get: sam-9: 8
get: sam-10: 9

It looks like the problem arises when there are 2 or more processes that uses the same Memcache client object, and one of them crashes while setting a value into the cache.
After the crash, the other process with the same Memcached object starts returning wrong key/value pairs.

I fixed it by making sure each process initiates it's own Memcached object and made a lookup based on the process id.

_global_caches = {}
def _get_cache():
    # get current process pid
    pid = current_process().pid

    # See if we have a memcache object with this pid
    if pid not in _global_caches:
        _global_caches[pid] = Memcache(['127.0.0.1:11211'])

    return _global_caches[pid]


# Set some values in the cache
print("Setting!")
for i in range (1, 11):
    _get_cache().set('sam-%s' % i, i)
    print('set: sam-%s = %s' % (i,i))

# Get's all the set values to ensure they are saved
print("Getting!")
for i in range (1, 11):
    print('get: sam-%s: %s' % (i,_get_cache().get('sam-%s' % i)))


def worker():
  print('Function started')
  for i in range(1000):
      _get_cache().set('interrupt-%s' % i, i)
  print('Function finished')

# Start a new process that we kill while it's setting values in the 
p = Process(target=worker)
p.start()
time.sleep(0.1)
p.terminate()
print("process killed")

print("Getting!")
for i in range (1, 11):
    print('get: sam-%s: %s' % (i,_get_cache().get('sam-%s' % i)))

This should output the more expected

Setting!
set: sam-1 = 1
set: sam-2 = 2
set: sam-3 = 3
set: sam-4 = 4
set: sam-5 = 5
set: sam-6 = 6
set: sam-7 = 7
set: sam-8 = 8
set: sam-9 = 9
set: sam-10 = 10
Getting!
get: sam-1: 1
get: sam-2: 2
get: sam-3: 3
get: sam-4: 4
get: sam-5: 5
get: sam-6: 6
get: sam-7: 7
get: sam-8: 8
get: sam-9: 9
get: sam-10: 10
Function started
process killed
Getting!
get: sam-1: 1
get: sam-2: 2
get: sam-3: 3
get: sam-4: 4
get: sam-5: 5
get: sam-6: 6
get: sam-7: 7
get: sam-8: 8
get: sam-9: 9
get: sam-10: 10

Just note that this solution uses more sockets for the memcache, as it creates one for each process

Yes, the problem arises when there are more processes that uses the same Memcache client object. I switched to pymemcache library. It has not the same problem.

Navigation Menu

Repository navigation

python-memcached return wrong value after crash?? #166

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

python-memcached return wrong value after crash?? #166

Description

Activity

KPassov commented on Nov 1, 2019

samsonle1809 commented on Nov 2, 2019

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions