Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 upload big file in Django with process bar and i get error : MemoryError
#1
i create a web page in Django for upload big file (any file) with process bar

i used IIS and django version 2.2.2
views.py :
def upload_file(request):
if request.method == 'POST':
    form = UploadFileForm(request.POST, request.FILES)
    if form.is_valid():
        handle_uploaded_file(request.FILES['file'])
        return HttpResponse('/success/url/')
else:
    form = UploadFileForm()
return render(request, 'Demo/home.html', {'form': form})


def handle_uploaded_file(f):
 namefile = f.name
 with open(namefile, 'wb+') as destination:
    print(f.chunks())
    for chunk in f.chunks():
        destination.write(chunk)
forms.py
from django import forms


class UploadFileForm(forms.Form):
 title = forms.CharField(max_length=50)
 file = forms.FileField()
upload.js :
   $(document).ready(function () {

    $('form').on('submit', function (event) {

        event.preventDefault();

        var formData = new FormData($('form')[0]);

        $.ajax({
            xhr: function () {
                var xhr = new window.XMLHttpRequest();

                xhr.upload.addEventListener('progress', function (e) {

                    if (e.lengthComputable) {

                        console.log('Bytes Loaded: ' + e.loaded);
                        console.log('Total Size: ' + e.total);
                        console.log('Percentage Uploaded: ' + (e.loaded / e.total))

                        var percent = Math.round((e.loaded / e.total) * 100);

                        $('#progressBar').attr('aria-valuenow', percent).css('width', percent + '%').text(percent + '%');

                    }

                });

                return xhr;
            },
            type: 'POST',
            url: '/',
            data: formData,
            processData: false,
            contentType: false,
            success: function () {
                alert('File uploaded!');
            }
        });

    });

});
html:
<!DOCTYPE html>
<html lang="en">

<head>
    {% load static %}
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <title>Document</title>
    <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js"></script>
    <script src="{% static 'Demo/upload.js' %}"></script>

    <!-- Latest compiled and minified CSS -->
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css"
        integrity="sha384-1q8mTJOASx8j1Au+a5WDVnPi2lkFfwwEAa8hDDdjZlpLegxhjVME1fgjWPGmkzs7" crossorigin="anonymous">

    <!-- Optional theme -->
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap-theme.min.css"
        integrity="sha384-fLW2N01lMqjakBkx3l/M9EahuwpSfeNvV63J5ezn3uZzapT0u7EYsXMjQV+0En5r" crossorigin="anonymous">

    <!-- Latest compiled and minified JavaScript -->
    <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap.min.js"
        integrity="sha384-0mSbJDEHialfmuBBQP6A4Qrprq5OVfW37PRR3j5ELqxss1yVqOtnepnHVP9aJ7xS" crossorigin="anonymous">
    </script>
</head>

<body>


    <div>
        <form action="" method="post" enctype="multipart/form-data">
            {% csrf_token %}
            {{form.as_p}}
            <input type="submit" value="ok">
        </form>

        <div class="progress">
            <div id="progressBar" class="progress-bar" role="progressbar" aria-valuenow="0" aria-valuemin="0"
                aria-valuemax="100" style="width: 0%;">
                0%
            </div>
        </div>
    </div>
</body>

</html>
i get this error :
Quote:Internal Server Error: / Traceback (most recent call last): File "C:\Python37\lib\site-packages\django\core\handlers\exception.py", line 34, in inner response = get_response(request) File "C:\Python37\lib\site-packages\django\core\handlers\base.py", line 106, in _get_response response = middleware_method(request, callback, callback_args, callback_kwargs) File "C:\Python37\lib\site-packages\django\middleware\csrf.py", line 295, in process_view request_csrf_token = request.POST.get('csrfmiddlewaretoken', '') File "C:\Python37\lib\site-packages\django\core\handlers\wsgi.py", line 110, in _get_post self._load_post_and_files() File "C:\Python37\lib\site-packages\django\http\request.py", line 314, in _load_post_and_files self._post, self._files = self.parse_file_upload(self.META, data) File "C:\Python37\lib\site-packages\django\http\request.py", line 274, in parse_file_upload return parser.parse() File "C:\Python37\lib\site-packages\django\http\multipartparser.py", line 254, in parse chunk = handler.receive_data_chunk(chunk, counters[i]) File "C:\Python37\lib\site-packages\django\core\files\uploadhandler.py", line 174, in receive_data_chunk self.file.write(raw_data) MemoryError

in settings.py i add:
Quote:FILE_UPLOAD_MAX_MEMORY_SIZE = 5368709120

DATA_UPLOAD_MAX_MEMORY_SIZE = 5368709120

what is my mistake .. is better way for upload big file with process bar in django
Quote
#2
I'm not familiar with Django.

Maybe the print(f.chunks()) causes the MemoryError.
Normally it should not, because the print function does not consume generators.

In the documentation they write, that the method UploadedFile.chunks() should used always.
It the method returns a generator, which is consumed by the for-loop.
The for-loop itself should not cause this issue. For each iteration you get a new chunk
and the old chunk object is garbage collected.

Maybe you overwrite at some other place your settings. Does it work with small uploads?
My code examples are always for Python >=3.6.0
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Quote
#3
Hi tanks for your answer

I don't know what is problems it is work for small file upload 50 mb but for 700 mb i tested my code and give me error "memory error "
Quote
#4
As DeaD_EyE sad, "print" function can be the reason of troubles, especially in Windows (you use Windows as I guess seeing IIS).
I've run the same code on Windows 10 Pro core I7 64GB, And Linux virtual machine.

import timeit
print(timeit.timeit("print(1)", number=1000))
Now, look to results:

Windows:
0.2165976

Linux:
0.0051541

First of all rid of print. It gives you terrible overhead in any case.
Second, try your code on the usually recommended environment: Linux + uwsgi + Nginx. I can't except, that IIS is the trouble maker.
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Django Python Vscode Error Please Help Smhbugra 3 423 Jun-30-2019, 10:54 AM
Last Post: noisefloor
  Django Connection Error erfanakbari1 1 344 Mar-21-2019, 08:09 AM
Last Post: Larz60+
  django-geoposition and unknown column error? PrateekG 3 1,096 Jun-26-2018, 11:56 AM
Last Post: PrateekG
  Open Sqlite database file directly in django sourabhjaiswal92 0 780 May-22-2018, 05:55 AM
Last Post: sourabhjaiswal92
  python largesize(6GB) file upload fails nithya_g 2 1,240 Jan-29-2018, 09:11 PM
Last Post: micseydel
  file upload from windows10 machine using send_key in selenium fails nithya_g 3 1,526 Jan-12-2018, 09:32 PM
Last Post: metulburr
  How do I make a Django model from a tab-delimited data file? newbietostuff 0 719 Jan-09-2018, 02:44 AM
Last Post: newbietostuff
  Django serve list-file foralobo 4 1,404 Oct-03-2017, 03:48 PM
Last Post: nilamo
  read text file using python and display its output to html using django amit 0 11,611 Jul-23-2017, 06:14 AM
Last Post: amit

Forum Jump:


Users browsing this thread: 1 Guest(s)