Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Retrieve data from ajax
#1
Hello. I need to scrape following page:

https://www.semanticscholar.org/paper/BE...1594e0e992

I don't know how to get data which is loaded with ajax when page numbers are clicked. I know how to retrieve data from single page.

from bs4 import BeautifulSoup as BS
import requests

r = requests.get("https://www.semanticscholar.org/paper/BERT%3A-Pre-training-of-Deep-Bidirectional-for-Devlin-Chang/df2b0e26d0599ce3e70df8a9da02e51594e0e992")
html = BS(r.content, 'html.parser')

index = 0
for el in html.select(".citation-list__citations > .cl-paper-row"):
    title = el.select('a')
    
    index += 1
    if index == 11:
        break

    print(title[0].text)
Please help.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  POST Reply to Ajax/jQuery (mostly an HTML question) gw1500se 5 2,686 Nov-18-2021, 02:44 PM
Last Post: gw1500se
  Scraping a dynamic data-table in python through AJAX request filozofo 1 3,887 Aug-14-2020, 10:13 AM
Last Post: kashcode
  Flask SelectField using Ajax noah_f 0 2,560 Mar-23-2020, 04:03 PM
Last Post: noah_f
  how retrieve database save multiple data in web Django 2.1 taomihiranga 0 2,807 Jul-30-2019, 04:58 PM
Last Post: taomihiranga
  Ajax and Selenium ankitjindalbti 3 3,339 May-31-2019, 03:40 PM
Last Post: ankitjindalbti
  Django - Retrieve form data justantest 0 2,847 May-23-2019, 11:47 AM
Last Post: justantest
  How to retrieve data from site ROHK 2 2,436 Mar-01-2019, 12:26 PM
Last Post: ROHK
  AJAX: readyState not as expected pleriche 2 3,413 Aug-08-2017, 08:55 AM
Last Post: pleriche

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020