Django, ZeroMQ and Celery: multiprocessing gotcha
ZeroMQ can be a great tool, however, there a few things to keep in mind, especially if you are using it in conjunction with Celery (not as a message broker, I assume you’re using something else like RabbitMQ for this).
It’s not very well documented, but during launch, Celery uses multiprocessing to start the Python workers. On Unix-based systems, multiprocessing uses the fork() system call to spawn new processes. Forking copies the parent process into a new process, which causes an issue with open file descriptors (including sockets). This would also be a problem for Django’s database connection, but Celery will reset that by default.
So, if you are using ZeroMQ inside a Celery task (for logging for example), you need to force the worker to manually close and re-open the ZeroMQ context. You can easily do this using the built-in Celery signal worker_process_init. The following helper module takes care of this:
from celery import signals import zmq import logging log = logging.getLogger("app." + __name__) ZMQ_SOCKET_LINGER = 100 context = zmq.Context() context.linger = ZMQ_SOCKET_LINGER def reset_zmq_context(**kwargs): log.debug("Resetting ZMQ Context") reset_context() signals.worker_process_init.connect(reset_zmq_context) def get_context(): global context if context.closed: context = zmq.Context() context.linger = ZMQ_SOCKET_LINGER return context def reset_context(): global context context.term() context = zmq.Context() context.linger = ZMQ_SOCKET_LINGER