Python Forum
upload big file in Django with process bar and i get error : MemoryError
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
upload big file in Django with process bar and i get error : MemoryError
#1
i create a web page in Django for upload big file (any file) with process bar

i used IIS and django version 2.2.2
views.py :
def upload_file(request):
if request.method == 'POST':
    form = UploadFileForm(request.POST, request.FILES)
    if form.is_valid():
        handle_uploaded_file(request.FILES['file'])
        return HttpResponse('/success/url/')
else:
    form = UploadFileForm()
return render(request, 'Demo/home.html', {'form': form})


def handle_uploaded_file(f):
 namefile = f.name
 with open(namefile, 'wb+') as destination:
    print(f.chunks())
    for chunk in f.chunks():
        destination.write(chunk)
forms.py
from django import forms


class UploadFileForm(forms.Form):
 title = forms.CharField(max_length=50)
 file = forms.FileField()
upload.js :
   $(document).ready(function () {

    $('form').on('submit', function (event) {

        event.preventDefault();

        var formData = new FormData($('form')[0]);

        $.ajax({
            xhr: function () {
                var xhr = new window.XMLHttpRequest();

                xhr.upload.addEventListener('progress', function (e) {

                    if (e.lengthComputable) {

                        console.log('Bytes Loaded: ' + e.loaded);
                        console.log('Total Size: ' + e.total);
                        console.log('Percentage Uploaded: ' + (e.loaded / e.total))

                        var percent = Math.round((e.loaded / e.total) * 100);

                        $('#progressBar').attr('aria-valuenow', percent).css('width', percent + '%').text(percent + '%');

                    }

                });

                return xhr;
            },
            type: 'POST',
            url: '/',
            data: formData,
            processData: false,
            contentType: false,
            success: function () {
                alert('File uploaded!');
            }
        });

    });

});
html:
<!DOCTYPE html>
<html lang="en">

<head>
    {% load static %}
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <title>Document</title>
    <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js"></script>
    <script src="{% static 'Demo/upload.js' %}"></script>

    <!-- Latest compiled and minified CSS -->
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css"
        integrity="sha384-1q8mTJOASx8j1Au+a5WDVnPi2lkFfwwEAa8hDDdjZlpLegxhjVME1fgjWPGmkzs7" crossorigin="anonymous">

    <!-- Optional theme -->
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap-theme.min.css"
        integrity="sha384-fLW2N01lMqjakBkx3l/M9EahuwpSfeNvV63J5ezn3uZzapT0u7EYsXMjQV+0En5r" crossorigin="anonymous">

    <!-- Latest compiled and minified JavaScript -->
    <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap.min.js"
        integrity="sha384-0mSbJDEHialfmuBBQP6A4Qrprq5OVfW37PRR3j5ELqxss1yVqOtnepnHVP9aJ7xS" crossorigin="anonymous">
    </script>
</head>

<body>


    <div>
        <form action="" method="post" enctype="multipart/form-data">
            {% csrf_token %}
            {{form.as_p}}
            <input type="submit" value="ok">
        </form>

        <div class="progress">
            <div id="progressBar" class="progress-bar" role="progressbar" aria-valuenow="0" aria-valuemin="0"
                aria-valuemax="100" style="width: 0%;">
                0%
            </div>
        </div>
    </div>
</body>

</html>
i get this error :
Quote:Internal Server Error: / Traceback (most recent call last): File "C:\Python37\lib\site-packages\django\core\handlers\exception.py", line 34, in inner response = get_response(request) File "C:\Python37\lib\site-packages\django\core\handlers\base.py", line 106, in _get_response response = middleware_method(request, callback, callback_args, callback_kwargs) File "C:\Python37\lib\site-packages\django\middleware\csrf.py", line 295, in process_view request_csrf_token = request.POST.get('csrfmiddlewaretoken', '') File "C:\Python37\lib\site-packages\django\core\handlers\wsgi.py", line 110, in _get_post self._load_post_and_files() File "C:\Python37\lib\site-packages\django\http\request.py", line 314, in _load_post_and_files self._post, self._files = self.parse_file_upload(self.META, data) File "C:\Python37\lib\site-packages\django\http\request.py", line 274, in parse_file_upload return parser.parse() File "C:\Python37\lib\site-packages\django\http\multipartparser.py", line 254, in parse chunk = handler.receive_data_chunk(chunk, counters[i]) File "C:\Python37\lib\site-packages\django\core\files\uploadhandler.py", line 174, in receive_data_chunk self.file.write(raw_data) MemoryError

in settings.py i add:
Quote:FILE_UPLOAD_MAX_MEMORY_SIZE = 5368709120

DATA_UPLOAD_MAX_MEMORY_SIZE = 5368709120

what is my mistake .. is better way for upload big file with process bar in django
Reply
#2
I'm not familiar with Django.

Maybe the print(f.chunks()) causes the MemoryError.
Normally it should not, because the print function does not consume generators.

In the documentation they write, that the method UploadedFile.chunks() should used always.
It the method returns a generator, which is consumed by the for-loop.
The for-loop itself should not cause this issue. For each iteration you get a new chunk
and the old chunk object is garbage collected.

Maybe you overwrite at some other place your settings. Does it work with small uploads?
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply
#3
Hi tanks for your answer

I don't know what is problems it is work for small file upload 50 mb but for 700 mb i tested my code and give me error "memory error "
Reply
#4
As DeaD_EyE sad, "print" function can be the reason of troubles, especially in Windows (you use Windows as I guess seeing IIS).
I've run the same code on Windows 10 Pro core I7 64GB, And Linux virtual machine.

import timeit
print(timeit.timeit("print(1)", number=1000))
Now, look to results:

Windows:
0.2165976

Linux:
0.0051541

First of all rid of print. It gives you terrible overhead in any case.
Second, try your code on the usually recommended environment: Linux + uwsgi + Nginx. I can't except, that IIS is the trouble maker.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Python django view error ZeeKolachi 1 233 Mar-18-2024, 03:14 PM
Last Post: Sowmya
Photo After using models.Model on my class Im getting 'ImproperlyConfigured' error Django khavro 1 2,150 Apr-05-2021, 03:11 PM
Last Post: SheeppOSU
  Error - ManyToMany Fields in Django rob25111 0 1,503 Jan-17-2021, 04:58 PM
Last Post: rob25111
  creating an exe file for a python django project Sanjish 0 2,586 Dec-27-2020, 07:33 AM
Last Post: Sanjish
  Upload big file on the server HTTP protocol Timych 1 2,376 May-15-2020, 07:12 AM
Last Post: snippsat
  [Django] css file is not applying to the page SheeppOSU 1 3,029 May-09-2020, 06:54 AM
Last Post: menator01
  fix error upload image in python abdlwafitahiri 1 2,319 Jan-05-2020, 08:49 AM
Last Post: Larz60+
  Django Python Vscode Error Please Help Smhbugra 3 2,662 Jun-30-2019, 10:54 AM
Last Post: noisefloor
  Django: How to automatically substitute a variable in the admin page at Django 1.11? m0ntecr1st0 3 3,246 Jun-30-2019, 12:21 AM
Last Post: scidam
  Django Connection Error erfanakbari1 1 2,543 Mar-21-2019, 08:09 AM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020