Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Pagination
#1
Hello there,
I am working on my first Webscraper and have a problem with the pagination of the Website i want to crawl.

This is my code so far:
import bs4 as bs
import urllib.request
import pprint
import time

source = urllib.request.urlopen(

    'https://radiobochum.radiosparbox.de/'

).read()
soup = bs.BeautifulSoup(source, 'lxml')


Anbieter = []
for anbieter in soup.select('h2.artist'):
    Anbieter.append(str(anbieter.text[1:]))

Anzahl = []
for anzahlTi in soup.select('span.stock'):
    Anzahl.append(str(anzahlTi.text))

westfunk = {}
for f, b in zip(Anbieter, Anzahl):
    westfunk[f] = b

print(time.strftime("Statistiken der Westfunk am: " "%d.%m.%Y %H:%M:%S""\n"))
print("__________________________________________________________________________")

for anbieter_out, anzahl_out in westfunk.items():
    print(anbieter_out + ":" + anzahl_out)
 HERE
I know it's not the best but i hope you can help me with my problem.

Thank you so much in advance,
greetings from Germany. Smile
Reply
#2
You can deal with Pagination to some extent using just Beautiful soup, by following links, but if any JavaScript is involved, you will probably fail. A better solution is to use selinium, see: http://seleniumhome.blogspot.com/2013/07...using.html
Reply
#3
Try looking at the developers' pages. They have blocks or articles in which they give explanations
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  BeautifulSoup pagination using href rhat398 1 2,355 Jun-30-2021, 10:55 AM
Last Post: snippsat
  Python beautifulsoup pagination error The61 5 3,402 Apr-09-2020, 09:17 PM
Last Post: Larz60+
  Scrapy Javascript Pagination (next_page) nazmulfinance 2 2,987 Nov-18-2019, 01:01 AM
Last Post: nazmulfinance
  pagination for non standarded pages zarize 12 5,898 Sep-02-2019, 12:35 PM
Last Post: zarize
  Python - Scrapy Javascript Pagination (next_page) Baggelhsk95 3 9,919 Oct-08-2018, 01:20 PM
Last Post: stranac
  Filtering and pagination garynobles 0 34,346 Jun-14-2018, 08:11 PM
Last Post: garynobles
  BeautifulSoup and pagination. Mike Ru 1 7,850 Sep-22-2017, 10:15 AM
Last Post: snippsat

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020