Under heavy load using gevent, I see this:
Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/gevent/threadpool.py", line 207, in _worker value = func(*args, **kwargs) error: [Errno 11] Resource temporarily unavailable (<ThreadPool at 0x7fe468930dd0 0/5/10>, <built-in function getaddrinfo>) failed with error
There’s no good solution out there. Actually, it is easier to solve than expected. You only have to change the gevent’s DNS resolver.
In the doc, they didn’t clearly state the difference between the resolvers. However, if you take a closer look, in the change log
On Linux, however, the c-ares based resolver is probably a better choice.
and in ares resolver description
However, because it does not use threads, it may scale better for applications that make many lookups.
There you go. The solution is to simply set the environment variable:
The Next Problem
After fixing the gevent DNS resolver, I still occasionally see this:
IOError: [Errno 24] Too many open files: '/proc/79689/stat'
Obviously the process has reached its “number of open files” limit. Check the limit using
ulimit -n (for hard limit use
ulimit -Hn, for soft limit use
ulimit -Sn) and change it like this:
ulimit -n 2048
To change the hard limit, e.g.
ulimit -Hn 2048, you may need to use
To persist the changes, change the config file at
/etc/security/limits.conf. Add a line inside the file:
user soft nofile 2048
user with your username and change
hard if you want to change the hard limit.
If you use supervisor, modify the