Python - Scrapy Javascript Pagination (next_page)

[stranac]

The best way to do things like this is often to figure out what requests are being made using your browser's developer tools, and simply recreate those.
For example, clicking the next button shows this request:


I tried playing with some parameters, changing a few and omitting them, and also found out you can get all the results using a single request.
All that's left to do now is replace start_urls with start_requests() yielding a custom request, and you get all the items:

def start_requests(self):
    yield scrapy.FormRequest(
        url='https://www.siriocenter.gr/Proionta/PartialAntallaktika',
        formdata={
            'modelStr': json.dumps({
                'pageSize': 1000,
            }),
        },
    )