+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html)
+--- Thread: Retrieve data from ajax (/thread-37343.html)
Retrieve data from ajax - pthon3 - May-29-2022
Hello. I need to scrape following page:
https://www.semanticscholar.org/paper/BERT%3A-Pre-training-of-Deep-Bidirectional-for-Devlin-Chang/df2b0e26d0599ce3e70df8a9da02e51594e0e992
I don't know how to get data which is loaded with ajax when page numbers are clicked. I know how to retrieve data from single page.
from bs4 import BeautifulSoup as BS
import requests
r = requests.get("https://www.semanticscholar.org/paper/BERT%3A-Pre-training-of-Deep-Bidirectional-for-Devlin-Chang/df2b0e26d0599ce3e70df8a9da02e51594e0e992")
html = BS(r.content, 'html.parser')
index = 0
for el in html.select(".citation-list__citations > .cl-paper-row"):
title = el.select('a')
index += 1
if index == 11:
break
print(title[0].text)Please help.