Python & MySQL Interrupted System Call & pyinstrument

Background

My Python application cannot connect to MySQL. The error message looks like this:

Can’t connect to MySQL server on ‘127.0.0.1’ (4)

Error 4 means Interrupted system call.

I am using mysqlclient, the C wrapper MySQL connector. The error happens on both MySQL 5.6 and 5.7. It can be reproduced consistently. It seems that PyMySQL doesn’t have this problem. Also I am using gevent but it is not much related in this case.

Investigation

Google search yields very few results regarding this error which means I have to gather enough info for debugging by myself.

tcpdump & Wireshark

tcpdump -vv gives this:

a.51894 > b.mysql: Flags [S], cksum 0x143b (incorrect -> 0x07a7), seq 2054845572, win 29200, options [mss 1460,sackOK,TS val 1209248649 ecr 0,nop,wscale 7], length 0
b.mysql > a.51894: Flags [S.], cksum 0x1803 (correct), seq 109868999, ack 2054845573, win 28960, options [mss 1418,sackOK,TS val 1942748813 ecr 1209248649,nop,wscale 7], length 0
a.51894 > b.mysql: Flags [R], cksum 0xd533 (correct), seq 2054845573, win 0, length 0

The IPs are removed and replaced by a (application) and b (MySQL server).

First two lines look fine like a normal TCP handshake (SYN and SYN-ACK). The 3rd line is problematic where the application is sending a RST instead of an ACK.

The dump can be loaded into wireshark for easier inspection but in this case it is clear enough that the application is doing something wrong.

strace

Let’s trace some system calls and signals.

strace gives this:

--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
rt_sigreturn()                          = -1 EINTR (Interrupted system call)
clock_gettime(CLOCK_MONOTONIC, {191047, 182973829}) = 0
setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0, 1000}}, {it_interval={0, 0}, it_value={0, 0}}) = 0
clock_gettime(CLOCK_MONOTONIC, {191047, 183269214}) = 0
epoll_wait(12, 2c77020, 64, 59743)      = -1 EINTR (Interrupted system call)
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
rt_sigreturn()                          = -1 EINTR (Interrupted system call)
clock_gettime(CLOCK_MONOTONIC, {191047, 184632773}) = 0
setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0, 1000}}, {it_interval={0, 0}, it_value={0, 0}}) = 0
clock_gettime(CLOCK_MONOTONIC, {191047, 184979382}) = 0
epoll_wait(12, 2c77020, 64, 59743)      = -1 EINTR (Interrupted system call)
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
rt_sigreturn()                          = -1 EINTR (Interrupted system call)
clock_gettime(CLOCK_MONOTONIC, {191047, 186375363}) = 0
setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0, 1000}}, {it_interval={0, 0}, it_value={0, 0}}) = 0
clock_gettime(CLOCK_MONOTONIC, {191047, 186695691}) = 0
epoll_wait(12, 2c77020, 64, 59743)      = -1 EINTR (Interrupted system call)

The connect call looks fine (it is not shown above), but something is spamming SIGALRM in the application and there’s that Interrupted system call.

Source of SIGALRM & pyinstrument

I have to tear the code apart to find the source of this regular SIGALRM. Turns out it is from pyinstrument, a Python profiler and I am using the 0.12 version.

There’s a signal mode described in the README:

On Mac/Linux/Unix, pyinstrument can run in ‘signal’ mode. This uses OS-provided signals to interrupt the process every 1ms and record the stack. It gives much lower overhead (and thus accurate) readings than the standard Python sys.setprofile style profilers. However, this can only profile the main thread.

In the known issues section, it says:

Some system calls can fail with IOError when being profiled in signal mode. If this happens to you, your only option is to run in setprofile mode, by passing –setprofile to the command-line interface or use_signal=False to the Python API.

Here’s also a related GitHub issue.

Solution

The solution is clear. I have to either use pyinstrument with use_signal=False or use the latest version of it where it doesn’t use signals anymore.

What’s new in v2.0.0

Pyinstrument refactored to use a new profiling mode. Rather than using signals, pyintrument uses a new statistical profiler built on PyEval_SetProfile. This means no more main thread restriction, no more IO errors when using Pyinstrument, and no more ‘setprofile’ mode!