Dick Brouwer

Django, ZeroMQ and Celery: multiprocessing gotcha

ZeroMQ can be a great tool, however, there a few things to keep in mind, especially if you are using it in conjunction with Celery (not as a message broker, I assume you’re using something else like RabbitMQ for this).

It’s not very well documented, but during launch, Celery uses multiprocessing to start the Python workers. On Unix-based systems, multiprocessing uses the fork() system call to spawn new processes. Forking copies the parent process into a new process, which causes an issue with open file descriptors (including sockets). This would also be a problem for Django’s database connection, but Celery will reset that by default.

So, if you are using ZeroMQ inside a Celery task (for logging for example), you need to force the worker to manually close and re-open the ZeroMQ context. You can easily do this using the built-in Celery signal worker_process_init. The following helper module takes care of this:

from celery import signals
import zmq

import logging
log = logging.getLogger("app." + __name__)

ZMQ_SOCKET_LINGER = 100

context = zmq.Context()
context.linger = ZMQ_SOCKET_LINGER


def reset_zmq_context(**kwargs):
    log.debug("Resetting ZMQ Context")
    reset_context()
signals.worker_process_init.connect(reset_zmq_context)


def get_context():
    global context
    if context.closed:
        context = zmq.Context()
        context.linger = ZMQ_SOCKET_LINGER
    return context


def reset_context():
    global context
    context.term()
    context = zmq.Context()
    context.linger = ZMQ_SOCKET_LINGER