mkdtemp spontaneously stopped working reliably last night

[saporta98]

I have multiple production workloads running on multiple servers. All using Python 2.7.12. This morning I arrived at work to find lots of errors in my logs. The tracebacks all include this:

File "/usr/lib/python2.7/tempfile.py", line 331, in mkdtemp
dir = gettempdir()
File "/usr/lib/python2.7/tempfile.py", line 275, in gettempdir
tempdir = _get_default_tempdir()
File "/usr/lib/python2.7/tempfile.py", line 217, in _get_default_tempdir
("No usable temporary directory found in %s" % dirlist))
IOError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/']

We are not out of disk space. This is happening on lots of different servers.

We did not deploy any new code -- it was a Sunday evening. The first occurrence of the error was a few minutes after 0000 GMT this morning. We do not have automatic patching or updates enabled. Could there be something in the Python tempfile module that started failing as of a certain date/time?? Anyone else having a similar problem?

---

Well, my bad. We *are* out of disk space. Just not on the servers I was examining. The problem was on some back-end servers that process jobs sent by the other servers. The Python tempfile module is fine.

[nilamo]

Thanks for letting us know what happened :)