Python Forum
Scraping a dynamic data-table in python through AJAX request - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html)
+--- Thread: Scraping a dynamic data-table in python through AJAX request (/thread-25077.html)



Scraping a dynamic data-table in python through AJAX request - filozofo - Mar-18-2020

Ref to the stackoverflow thread:
I would like to implement a demo backend script in python in order to scrape the list of document paths through the URL. If I try to handle through the basic request with urllib.request, it does not return the content of the datatable. It is just returned an empty content as below. Hence, the content of the data table might be captured with the AJAX request in this scenario. The example code which is taken from homepage of the actual website (Actual Page). I think main issue here might be token side. How can I handle the python side in order to capture the content of the dynamic datatable?

The code I wrote on python side (Current python code) can not handle this request and returns "The page has expired due to inactivity."

Any suggestions, directions ?

Current python code:

from requests import Session
session = Session()
session.head("mainpage")
response = session.post(
    url= 'apiurl/getviewdatatable',
    data= {
        'name' : 'name',
        'documentPath' : 'documentPath',
    }           
)

print(response.text)
Actual Page: mainpage

<script>
        $(document).ready(function () {
            $('#posts').DataTable({
                "processing": true,
                "serverSide": true,
                "ajax": {
                    "url": "apiurl/getviewdatatable",
                    "dataType": "json",
                    "type": "POST",
                    "data": {
                        _token: "5uC8cmrJRYnn2gB4TCU9EFj0wjgA8BtmqqO64Uk1",
                    }
                },
                "columns": [
                                            {
                        "data": "name"
                    },
                                        {
                        "data": "documentPath"
                    },

                ],
                'createdRow': function (row, data, index) {
                    //
                    // if the second column cell is blank apply special formatting
                    //
                    $(row).addClass('table-row');

                },
                "language": {
                    "url": "//cdn.datatables.net/plug-ins/9dcbecd42ad/i18n/Turkish.json"
                },
                "rowReorder":{
                    selector:'td:nth-child(2)'
                },
            });
        });
</script>



RE: Scraping a dynamic data-table in python through AJAX request - kashcode - Aug-14-2020

In python part you also need get token and when you have token use it in request like you use it in javascript part.