gevent built-in function getaddrinfo failed with error

Problem

Under heavy load using gevent, I see this:

Traceback (most recent call last):
 File "/usr/local/lib/python2.7/dist-packages/gevent/threadpool.py", line 207, in _worker
   value = func(*args, **kwargs)
error: [Errno 11] Resource temporarily unavailable
(<ThreadPool at 0x7fe468930dd0 0/5/10>, <built-in function getaddrinfo>) failed with error

Solution

There’s no good solution out there. Actually, it is easier to solve than expected. You only have to change the gevent’s DNS resolver.

In the doc, they didn’t clearly state the difference between the resolvers. However, if you take a closer look, in the change log

On Linux, however, the c-ares based resolver is probably a better choice.

and in ares resolver description

However, because it does not use threads, it may scale better for applications that make many lookups.

There you go. The solution is to simply set the environment variable:

GEVENT_RESOLVER=ares

The Next Problem

After fixing the gevent DNS resolver, I still occasionally see this:

IOError: [Errno 24] Too many open files: '/proc/79689/stat'

Solution

Obviously the process has reached its “number of open files” limit. Check the limit using ulimit -n (for hard limit use ulimit -Hn, for soft limit use ulimit -Sn) and change it like this:

ulimit -n 2048

To change the hard limit, e.g. ulimit -Hn 2048, you may need to use sudo.

To persist the changes, change the config file at /etc/security/limits.conf. Add a line inside the file:

user    soft    nofile  2048

Substitute user with your username and change soft to hard if you want to change the hard limit.

If you use supervisor, modify the minfds config.