Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 upload big file in Django with process bar and i get error : MemoryError
#1
i create a web page in Django for upload big file (any file) with process bar

i used IIS and django version 2.2.2
views.py :
def upload_file(request):
if request.method == 'POST':
    form = UploadFileForm(request.POST, request.FILES)
    if form.is_valid():
        handle_uploaded_file(request.FILES['file'])
        return HttpResponse('/success/url/')
else:
    form = UploadFileForm()
return render(request, 'Demo/home.html', {'form': form})


def handle_uploaded_file(f):
 namefile = f.name
 with open(namefile, 'wb+') as destination:
    print(f.chunks())
    for chunk in f.chunks():
        destination.write(chunk)
forms.py
from django import forms


class UploadFileForm(forms.Form):
 title = forms.CharField(max_length=50)
 file = forms.FileField()
upload.js :
   $(document).ready(function () {

    $('form').on('submit', function (event) {

        event.preventDefault();

        var formData = new FormData($('form')[0]);

        $.ajax({
            xhr: function () {
                var xhr = new window.XMLHttpRequest();

                xhr.upload.addEventListener('progress', function (e) {

                    if (e.lengthComputable) {

                        console.log('Bytes Loaded: ' + e.loaded);
                        console.log('Total Size: ' + e.total);
                        console.log('Percentage Uploaded: ' + (e.loaded / e.total))

                        var percent = Math.round((e.loaded / e.total) * 100);

                        $('#progressBar').attr('aria-valuenow', percent).css('width', percent + '%').text(percent + '%');

                    }

                });

                return xhr;
            },
            type: 'POST',
            url: '/',
            data: formData,
            processData: false,
            contentType: false,
            success: function () {
                alert('File uploaded!');
            }
        });

    });

});
html:
<!DOCTYPE html>
<html lang="en">

<head>
    {% load static %}
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <title>Document</title>
    <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js"></script>
    <script src="{% static 'Demo/upload.js' %}"></script>

    <!-- Latest compiled and minified CSS -->
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css"
        integrity="sha384-1q8mTJOASx8j1Au+a5WDVnPi2lkFfwwEAa8hDDdjZlpLegxhjVME1fgjWPGmkzs7" crossorigin="anonymous">

    <!-- Optional theme -->
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap-theme.min.css"
        integrity="sha384-fLW2N01lMqjakBkx3l/M9EahuwpSfeNvV63J5ezn3uZzapT0u7EYsXMjQV+0En5r" crossorigin="anonymous">

    <!-- Latest compiled and minified JavaScript -->
    <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap.min.js"
        integrity="sha384-0mSbJDEHialfmuBBQP6A4Qrprq5OVfW37PRR3j5ELqxss1yVqOtnepnHVP9aJ7xS" crossorigin="anonymous">
    </script>
</head>

<body>


    <div>
        <form action="" method="post" enctype="multipart/form-data">
            {% csrf_token %}
            {{form.as_p}}
            <input type="submit" value="ok">
        </form>

        <div class="progress">
            <div id="progressBar" class="progress-bar" role="progressbar" aria-valuenow="0" aria-valuemin="0"
                aria-valuemax="100" style="width: 0%;">
                0%
            </div>
        </div>
    </div>
</body>

</html>
i get this error :
Quote:Internal Server Error: / Traceback (most recent call last): File "C:\Python37\lib\site-packages\django\core\handlers\exception.py", line 34, in inner response = get_response(request) File "C:\Python37\lib\site-packages\django\core\handlers\base.py", line 106, in _get_response response = middleware_method(request, callback, callback_args, callback_kwargs) File "C:\Python37\lib\site-packages\django\middleware\csrf.py", line 295, in process_view request_csrf_token = request.POST.get('csrfmiddlewaretoken', '') File "C:\Python37\lib\site-packages\django\core\handlers\wsgi.py", line 110, in _get_post self._load_post_and_files() File "C:\Python37\lib\site-packages\django\http\request.py", line 314, in _load_post_and_files self._post, self._files = self.parse_file_upload(self.META, data) File "C:\Python37\lib\site-packages\django\http\request.py", line 274, in parse_file_upload return parser.parse() File "C:\Python37\lib\site-packages\django\http\multipartparser.py", line 254, in parse chunk = handler.receive_data_chunk(chunk, counters[i]) File "C:\Python37\lib\site-packages\django\core\files\uploadhandler.py", line 174, in receive_data_chunk self.file.write(raw_data) MemoryError

in settings.py i add:
Quote:FILE_UPLOAD_MAX_MEMORY_SIZE = 5368709120

DATA_UPLOAD_MAX_MEMORY_SIZE = 5368709120

what is my mistake .. is better way for upload big file with process bar in django
Quote
#2
I'm not familiar with Django.

Maybe the print(f.chunks()) causes the MemoryError.
Normally it should not, because the print function does not consume generators.

In the documentation they write, that the method UploadedFile.chunks() should used always.
It the method returns a generator, which is consumed by the for-loop.
The for-loop itself should not cause this issue. For each iteration you get a new chunk
and the old chunk object is garbage collected.

Maybe you overwrite at some other place your settings. Does it work with small uploads?
My code examples are always for Python >=3.6.0
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Quote
#3
Hi tanks for your answer

I don't know what is problems it is work for small file upload 50 mb but for 700 mb i tested my code and give me error "memory error "
Quote
#4
As DeaD_EyE sad, "print" function can be the reason of troubles, especially in Windows (you use Windows as I guess seeing IIS).
I've run the same code on Windows 10 Pro core I7 64GB, And Linux virtual machine.

import timeit
print(timeit.timeit("print(1)", number=1000))
Now, look to results:

Windows:
0.2165976

Linux:
0.0051541

First of all rid of print. It gives you terrible overhead in any case.
Second, try your code on the usually recommended environment: Linux + uwsgi + Nginx. I can't except, that IIS is the trouble maker.
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Django Python Vscode Error Please Help Smhbugra 3 290 Jun-30-2019, 10:54 AM
Last Post: noisefloor
  Django: How to automatically substitute a variable in the admin page at Django 1.11? m0ntecr1st0 3 216 Jun-30-2019, 12:21 AM
Last Post: scidam
  Django Connection Error erfanakbari1 1 232 Mar-21-2019, 08:09 AM
Last Post: Larz60+
  Error when running .py file but not when entering exact same code in CLI Broadsworde 3 348 Dec-11-2018, 09:06 AM
Last Post: Broadsworde
  upload folder in server ftp falloff 3 545 Nov-02-2018, 09:46 PM
Last Post: Gribouillis
  How to upload images after first paragraph ends instead of uploading them at the end? SamLearnsPython 0 381 Sep-15-2018, 06:53 AM
Last Post: SamLearnsPython
  Error while creating new Creating Models on Django aniyanetworks 2 1,108 Jul-13-2018, 12:57 PM
Last Post: aniyanetworks
  django-geoposition and unknown column error? PrateekG 3 929 Jun-26-2018, 11:56 AM
Last Post: PrateekG
  Open Sqlite database file directly in django sourabhjaiswal92 0 679 May-22-2018, 05:55 AM
Last Post: sourabhjaiswal92
  python largesize(6GB) file upload fails nithya_g 2 1,039 Jan-29-2018, 09:11 PM
Last Post: micseydel

Forum Jump:


Users browsing this thread: 1 Guest(s)