Where is the memory leak in this Flask Code?

[Morkus]

I'm having an issue where every time I run a method in a Flask service, the memory grows. Not sure why.

This looks to be the problem method:

def save_figure_to_bytes():
    buf = io.BytesIO()
    plt.savefig(buf, format="png")
    plt.close()
    encoded_str = str(base64.b64encode(buf.getvalue()), "UTF-8")
    buf.close()
    return encoded_str

Below is the output from the python memory_profiler. Every time I run this code, there's a positive "Increment" noted in the line plt.savefig(buf, format="png"). As noted in the method, I close the buffer and the matplotlib plot, "plt", before returning so how does the memory keep growing?

Not sure where the memory leak is and would appreciate knowing what I'm doing wrong here. Eventually, after a few runs, the code crashes with a memory error.

Thanks.

Line # Mem usage Increment Occurences Line Contents
============================================================
41 7248.5859 MiB 7248.5859 MiB 1 @profile(precision=4)
42 def save_figure_to_bytes():
45 7248.5859 MiB 0.0000 MiB 1 buf = io.BytesIO()
46 7780.2031 MiB 531.6172 MiB 1 plt.savefig(buf, format="png")
47 7780.2031 MiB 0.0000 MiB 1 plt.close()
50 7780.2031 MiB 0.0000 MiB 1 encoded_str = str(base64.b64encode(buf.getvalue()), "UTF-8")
51 7780.2031 MiB 0.0000 MiB 1 buf.close()
52 7780.2031 MiB 0.0000 MiB 1 return encoded_str

Here is the requirements.txt file's contents for a listing of dependencies:

appdirs==1.4.4
appnope==0.1.2
argon2-cffi==20.1.0
async-generator==1.10
attrs==20.3.0
backcall==0.2.0
bleach==3.3.0
certifi==2020.12.5
cffi==1.14.5
click==7.1.2
cycler==0.10.0
decorator==5.0.7
defusedxml==0.7.1
distlib==0.3.1
entrypoints==0.3
filelock==3.0.12
Flask==1.1.4
ipykernel==5.5.3
ipython==7.23.0
ipython-genutils==0.2.0
ipywidgets==7.6.3
itsdangerous==1.1.0
jedi==0.18.0
Jinja2==2.11.3
joblib==1.0.1
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==6.1.12
jupyter-console==6.4.0
jupyter-core==4.7.1
jupyterlab-pygments==0.1.2
jupyterlab-widgets==1.0.0
kiwisolver==1.3.1
MarkupSafe==2.0.1
matplotlib==3.4.2
matplotlib-inline==0.1.2
mistune==0.8.4
nbclient==0.5.3
nbconvert==6.0.7
nbformat==5.1.3
nest-asyncio==1.5.1
notebook==6.3.0
numpy==1.20.3
packaging==20.9
pandas==1.2.4
pandocfilters==1.4.3
parso==0.8.2
pexpect==4.8.0
pickleshare==0.7.5
Pillow==8.2.0
praat-parselmouth==0.4.0
prometheus-client==0.10.1
prompt-toolkit==3.0.18
ptyprocess==0.7.0
pycparser==2.20
Pygments==2.8.1
pyparsing==2.4.7
pyrsistent==0.17.3
python-dateutil==2.8.1
pytz==2021.1
pyzmq==22.0.3
qtconsole==5.0.3
QtPy==1.9.0
scikit-learn==0.24.2
scipy==1.6.3
seaborn==0.11.1
Send2Trash==1.5.0
six==1.15.0
terminado==0.9.4
testpath==0.4.4
threadpoolctl==2.1.0
tornado==6.1
traitlets==5.0.5
virtualenv==20.4.6
wcwidth==0.2.5
webencodings==0.5.1
Werkzeug==1.0.1
widgetsnbextension==3.5.1

Here's a working sample of the code itself:

import base64
import urllib
import numpy as np
import parselmouth as parselmouth
from flask import Flask, make_response
import tempfile
from flask import request
import matplotlib.pyplot as plt
from urllib.parse import unquote
import io
import os
from memory_profiler import profile

application = Flask(__name__)


@profile(precision=4)
def get_data_and_write_temp_file():
    plt.close()
    # strip off url encoding
    file_data = urllib.parse.unquote(request.data).replace("file_data=", "")
    # strip off base64
    file_data_no_base64 = base64.standard_b64decode(file_data)
    with tempfile.TemporaryDirectory() as td:
        f_name = os.path.join(td, 'test')
        with open(f_name, 'wb') as fh:
            fh.write(file_data_no_base64)
            fh.seek(0)
            value_to_return = parselmouth.Sound(f_name)  
            fh.close()
        return value_to_return


@profile(precision=4)
def save_figure_to_bytes():
    plt.savefig(buf, format="png")
    plt.close()
    encoded_str = str(base64.b64encode(buf.getvalue()), "UTF-8")
    buf.close()
    return encoded_str


@profile(precision=4)
@application.route('/get_amplitude', methods=['POST', 'GET'])
def get_amplitude():
    snd = get_data_and_write_temp_file()
    # do the figure actions to create plot.
    plt.figure()
    plt.plot(snd.xs(), snd.values.T)
    plt.xlim([snd.xmin, snd.xmax])
    plt.xlabel("time [s]")
    plt.ylabel("amplitude")

    encoded_str = save_figure_to_bytes()
    return encoded_str


if __name__ == '__main__':
    application.run(host="0.0.0.1")

----

After spending 3 solid days trying everything I can think of, there's no difference in any of my attempts.

I've tried "gunicorn" on Mac, "Waitress" on Windows. Also tried running the app directly from the IDE.

Same result each time.

Thanks in advance for any help or suggestions.

- m

[Oliver]

Matplotlib has a documented memory leak when used with web services like Flask.

The workaround I saw suggested was to use a "figure", not a "plt", but when I tried to do that, it did not fix the memory leak.

Hopefully others here will chime in with some ideas.

Good luck!

Oliver