Random Notes on Scaling

1. parallel-ssh has a -I option to feed in a shell script file such that quotes escaping is not needed.
2. On Ubuntu 14.04, the HAProxy may have 2 pids after sudo service haproxy restart. I suspect one of them is a zombie since the new open file limit does not apply. The safer way is to killall haproxy then sudo service haproxy start.
3. HAProxy has a maxconn globally and maxconn for each frontend. The default for the latter is 2000. Make sure there is a maxconn for the frontend.
4. Performance schema is your friend in MySQL. e.g. To debug tmp disk table usage, use SELECT * FROM performance_schema.events_statements_summary_by_digest order by SUM_CREATED_TMP_DISK_TABLES desc limit 10\G and you’ll find the problematic query.
5. Pipe sleep to another command can terminate a command which would otherwise require manual intervention. e.g. sleep 1 | openssl s_client -connect imap.gmail.com:993 -crlf | grep 'Gimap ready'
6. In SQLAlchemy, commit inside a query loop is a bad idea. It requires refreshing all of the objects in session. The whole loop will be O(N^2).
7. Sometimes eager loading is not the best idea in SQLAlchemy when a simple count is possible. Eager loading will load the discriminator column for polymorphic classes which may prevent the use of a covering index.