![]() |
Where is the memory leak in this Flask Code? - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: General Coding Help (https://python-forum.io/forum-8.html) +--- Thread: Where is the memory leak in this Flask Code? (/thread-34006.html) |
Where is the memory leak in this Flask Code? - Morkus - Jun-17-2021 I'm having an issue where every time I run a method in a Flask service, the memory grows. Not sure why. This looks to be the problem method: def save_figure_to_bytes(): buf = io.BytesIO() plt.savefig(buf, format="png") plt.close() encoded_str = str(base64.b64encode(buf.getvalue()), "UTF-8") buf.close() return encoded_strBelow is the output from the python memory_profiler. Every time I run this code, there's a positive "Increment" noted in the line plt.savefig(buf, format="png"). As noted in the method, I close the buffer and the matplotlib plot, "plt", before returning so how does the memory keep growing? Not sure where the memory leak is and would appreciate knowing what I'm doing wrong here. Eventually, after a few runs, the code crashes with a memory error. Thanks. Line # Mem usage Increment Occurences Line Contents ============================================================ 41 7248.5859 MiB 7248.5859 MiB 1 @profile(precision=4) 42 def save_figure_to_bytes(): 45 7248.5859 MiB 0.0000 MiB 1 buf = io.BytesIO() 46 7780.2031 MiB 531.6172 MiB 1 plt.savefig(buf, format="png") 47 7780.2031 MiB 0.0000 MiB 1 plt.close() 50 7780.2031 MiB 0.0000 MiB 1 encoded_str = str(base64.b64encode(buf.getvalue()), "UTF-8") 51 7780.2031 MiB 0.0000 MiB 1 buf.close() 52 7780.2031 MiB 0.0000 MiB 1 return encoded_str Here is the requirements.txt file's contents for a listing of dependencies: appdirs==1.4.4 appnope==0.1.2 argon2-cffi==20.1.0 async-generator==1.10 attrs==20.3.0 backcall==0.2.0 bleach==3.3.0 certifi==2020.12.5 cffi==1.14.5 click==7.1.2 cycler==0.10.0 decorator==5.0.7 defusedxml==0.7.1 distlib==0.3.1 entrypoints==0.3 filelock==3.0.12 Flask==1.1.4 ipykernel==5.5.3 ipython==7.23.0 ipython-genutils==0.2.0 ipywidgets==7.6.3 itsdangerous==1.1.0 jedi==0.18.0 Jinja2==2.11.3 joblib==1.0.1 jsonschema==3.2.0 jupyter==1.0.0 jupyter-client==6.1.12 jupyter-console==6.4.0 jupyter-core==4.7.1 jupyterlab-pygments==0.1.2 jupyterlab-widgets==1.0.0 kiwisolver==1.3.1 MarkupSafe==2.0.1 matplotlib==3.4.2 matplotlib-inline==0.1.2 mistune==0.8.4 nbclient==0.5.3 nbconvert==6.0.7 nbformat==5.1.3 nest-asyncio==1.5.1 notebook==6.3.0 numpy==1.20.3 packaging==20.9 pandas==1.2.4 pandocfilters==1.4.3 parso==0.8.2 pexpect==4.8.0 pickleshare==0.7.5 Pillow==8.2.0 praat-parselmouth==0.4.0 prometheus-client==0.10.1 prompt-toolkit==3.0.18 ptyprocess==0.7.0 pycparser==2.20 Pygments==2.8.1 pyparsing==2.4.7 pyrsistent==0.17.3 python-dateutil==2.8.1 pytz==2021.1 pyzmq==22.0.3 qtconsole==5.0.3 QtPy==1.9.0 scikit-learn==0.24.2 scipy==1.6.3 seaborn==0.11.1 Send2Trash==1.5.0 six==1.15.0 terminado==0.9.4 testpath==0.4.4 threadpoolctl==2.1.0 tornado==6.1 traitlets==5.0.5 virtualenv==20.4.6 wcwidth==0.2.5 webencodings==0.5.1 Werkzeug==1.0.1 widgetsnbextension==3.5.1 Here's a working sample of the code itself: import base64 import urllib import numpy as np import parselmouth as parselmouth from flask import Flask, make_response import tempfile from flask import request import matplotlib.pyplot as plt from urllib.parse import unquote import io import os from memory_profiler import profile application = Flask(__name__) @profile(precision=4) def get_data_and_write_temp_file(): plt.close() # strip off url encoding file_data = urllib.parse.unquote(request.data).replace("file_data=", "") # strip off base64 file_data_no_base64 = base64.standard_b64decode(file_data) with tempfile.TemporaryDirectory() as td: f_name = os.path.join(td, 'test') with open(f_name, 'wb') as fh: fh.write(file_data_no_base64) fh.seek(0) value_to_return = parselmouth.Sound(f_name) fh.close() return value_to_return @profile(precision=4) def save_figure_to_bytes(): plt.savefig(buf, format="png") plt.close() encoded_str = str(base64.b64encode(buf.getvalue()), "UTF-8") buf.close() return encoded_str @profile(precision=4) @application.route('/get_amplitude', methods=['POST', 'GET']) def get_amplitude(): snd = get_data_and_write_temp_file() # do the figure actions to create plot. plt.figure() plt.plot(snd.xs(), snd.values.T) plt.xlim([snd.xmin, snd.xmax]) plt.xlabel("time [s]") plt.ylabel("amplitude") encoded_str = save_figure_to_bytes() return encoded_str if __name__ == '__main__': application.run(host="0.0.0.1")---- After spending 3 solid days trying everything I can think of, there's no difference in any of my attempts. I've tried "gunicorn" on Mac, "Waitress" on Windows. Also tried running the app directly from the IDE. Same result each time. Thanks in advance for any help or suggestions. - m RE: Where is the memory leak in this Flask Code? - Oliver - Jun-18-2021 Matplotlib has a documented memory leak when used with web services like Flask. The workaround I saw suggested was to use a "figure", not a "plt", but when I tried to do that, it did not fix the memory leak. Hopefully others here will chime in with some ideas. Good luck! Oliver |