Python Forum

Hello. I need to scrape following page:

https://www.semanticscholar.org/paper/BE...1594e0e992

I don't know how to get data which is loaded with ajax when page numbers are clicked. I know how to retrieve data from single page.

from bs4 import BeautifulSoup as BS
import requests

r = requests.get("https://www.semanticscholar.org/paper/BERT%3A-Pre-training-of-Deep-Bidirectional-for-Devlin-Chang/df2b0e26d0599ce3e70df8a9da02e51594e0e992")
html = BS(r.content, 'html.parser')

index = 0
for el in html.select(".citation-list__citations > .cl-paper-row"):
    title = el.select('a')
    
    index += 1
    if index == 11:
        break

    print(title[0].text)

Please help.