Kombu "timed out" bug: connection hangs forever in Kombu 4.1.0

25 January 2018

Kombu connects to RabbitMQ. However in Kombu 4.1.0 there is a bug where it waits forever if the broker is down rather than timing out. This affects Celery too. There is a fix in trunk but at time of writing there is no new release of Kombu which contains this fix. A workaround is to downgrade to Kombu 4.0.2.

# kombu==4.1.0
>>> app.control.ping()
# hangs forever

# kombu==4.0.2
>>> app.control.ping()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/venv/lib/python2.7/site-packages/celery/app/control.py", line 235, in ping
    timeout=timeout, **kwargs)
  File "/opt/venv/lib/python2.7/site-packages/celery/app/control.py", line 436, in broadcast
    limit, callback, channel=channel,
  File "/opt/venv/lib/python2.7/site-packages/kombu/pidbox.py", line 304, in _broadcast
    chan = channel or self.connection.default_channel
  File "/opt/venv/lib/python2.7/site-packages/kombu/connection.py", line 819, in default_channel
    self.connection
  File "/opt/venv/lib/python2.7/site-packages/kombu/connection.py", line 802, in connection
    self._connection = self._establish_connection()
  File "/opt/venv/lib/python2.7/site-packages/kombu/connection.py", line 757, in _establish_connection
    conn = self.transport.establish_connection()
  File "/opt/venv/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
    conn.connect()
  File "/opt/venv/lib/python2.7/site-packages/amqp/connection.py", line 302, in connect
    self.drain_events(timeout=self.connect_timeout)
  File "/opt/venv/lib/python2.7/site-packages/amqp/connection.py", line 485, in drain_events
    while not self.blocking_read(timeout):
  File "/opt/venv/lib/python2.7/site-packages/amqp/connection.py", line 490, in blocking_read
    frame = self.transport.read_frame()
  File "/opt/venv/lib/python2.7/site-packages/amqp/transport.py", line 240, in read_frame
    frame_header = read(7, True)
  File "/opt/venv/lib/python2.7/site-packages/amqp/transport.py", line 415, in _read
    s = recv(n - len(rbuf))
socket.timeout: timed out
>>>