Subtle evil of close_fds parameter in subprocess.Popen

In python newly created sub-process inherits file descriptors from parent process and these descriptors are left open – at least this was default till python ver. 3.3. subprocces.Popen constructor has parameter close_fds (defaults to False on python ver. 2.7), which can say if to close inherited FDs or not.  Leaving them open FDs for child process can lead to many problems as explained here and here.

I was also hit by this problem in very strange manner –  when working on btclient I experienced strange behaviour in HTTP client (based on urllib2), when a sub-process was launched HTTP requests behaved slightly differently – which caused remote server to return many 503 errors.  This was very weird, and it was driving me mad, until I tracked it down to the issue of inheriting opened file descriptors in sub-process.

I still wonder how exactly can sub-process ( it was basically any program – even something as simple as bash -c "sleep 1000" – it was just enough to inherit opened FDs) influence behaviour of socket ( on linux platform) and thus behaviour of HTTP  protocol. I’d be glad if somebody more profound in linux  can explain me exact mechanism behind this.

Leave a Reply

Your email address will not be published. Required fields are marked *