| Finding locking deadlocks in python |
I re-factored some code today, and in the process managed to create a lock deadlock for myself. In the end it turned out to be an exception was being thrown when a lock was held, and adding a try / finally resolved the real underlying problem. However, in the process I ended up writing this little helper that I am sure will be useful in the future.
import gflags
import thread
import threading
import traceback
import logging
...
FLAGS = gflags.FLAGS
gflags.DEFINE_boolean('dumplocks', False,
'If true, dumps information about lock activity')
...
class LockHelper(object):
"""A wrapper which makes it easier to see what locks are doing."""
lock = thread.allocate_lock()
def acquire(self):
if FLAGS.dumplocks:
logging.info('%s acquiring lock' % threading.currentThread().getName())
for s in traceback.extract_stack():
logging.info(' Trace %s:%s [%s] %s' % s)
self.lock.acquire()
def release(self):
if FLAGS.dumplocks:
logging.info('%s releasing lock' % threading.currentThread().getName())
for s in traceback.extract_stack():
logging.info(' Trace %s:%s [%s] %s' % s)
self.lock.release()
Now I can just use this helper in the place of thread.allocate_lock() when I want to see what is happening with locking. It saved me a lot of staring at random code today.
posted at: 15:46 | path: /python | permanent link to this entry
-
#1
Chris
In case things like this ever happen again to you,
Locks in Python implement the context manager protocol: http://www.python.org/dev/peps/pep-0343/ -- so when you use locks, you can shove the code that requires a lock to be held inside a 'with' clause, and have the lock get automatically acquired on entry and released on exit (even with an exception)
Really really worth looking at.
-
#2
Michael Still
That does look cool. It looks to me like I'd still need to write a class which wraps the lock though?
-
#4
Chris
Locks themselves (as provided by threading) already are context-managed, so if you're using the locks as provided, then you get the necessary behaviour for free.
If you want to write your own context managers, you either can implement the __enter__ and __exit__ methods in a class that you want to context manage, or you can use the contextmanager decorator that is provided in contextlib (part of the Python > 2.5 standard lib).
contextmanager is great -- you just write a generator function that yields a single result, and the decorator implements the context manager protocol so that you can use it with 'with'.
-
#7
lorg
While I agree that logging is a very good way to go about solving deadlocks and just about any other bug, there is also another tool that can help you.
When you use a Python debugger such as WinPDB (http://winpdb.org/) you can pause the program when already inside the deadlock and see on which actual calls your code is blocked.
In that sense deadlocks are easier than other bugs, because they do you the favor of stopping at the right place. Just try not to get into deadlocks that involve the GIL or the GTK lock. You really don't want to go there :)