May-29-2022, 10:32 PM
Hello. I need to scrape following page:
https://www.semanticscholar.org/paper/BE...1594e0e992
I don't know how to get data which is loaded with ajax when page numbers are clicked. I know how to retrieve data from single page.
https://www.semanticscholar.org/paper/BE...1594e0e992
I don't know how to get data which is loaded with ajax when page numbers are clicked. I know how to retrieve data from single page.
from bs4 import BeautifulSoup as BS import requests r = requests.get("https://www.semanticscholar.org/paper/BERT%3A-Pre-training-of-Deep-Bidirectional-for-Devlin-Chang/df2b0e26d0599ce3e70df8a9da02e51594e0e992") html = BS(r.content, 'html.parser') index = 0 for el in html.select(".citation-list__citations > .cl-paper-row"): title = el.select('a') index += 1 if index == 11: break print(title[0].text)Please help.