Python Forum
Code stops after 20min+ with no output
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Code stops after 20min+ with no output
#1
Hi,

I'm trying to get the 'src' from 500 profile pictures on Transfermarkt. I've managed to get the URL:s for every player in a list, but when I'm iterating through them to get the 'src' for their profile picture, the code just goes on and on, for 20min+ and then stops, without any output.

List of players

Please help.

from bs4 import BeautifulSoup
import requests
import pandas as pd


playerID = []
playerImgSrc = []


result = []

for page in range(1, 21):

    r = requests.get("https://www.transfermarkt.com/spieler-statistik/wertvollstespieler/marktwertetop?land_id=0&ausrichtung=alle&spielerposition_id=alle&altersklasse=alle&jahrgang=0&kontinent_id=0&plus=1",
        params= {"page": page},
        headers= {"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0"}
    )
    soup = BeautifulSoup(r.content, "html.parser")

    links = soup.select('a.spielprofil_tooltip')
    
    for i in range(len(links)):
        playerID.append(links[i].get('id'))

    playerProfile = ["https://www.transfermarkt.com/josh-maja/profil/spieler/" + x for x in playerID]

    for p in playerProfile:
        html = requests.get(p).text
        soup = BeautifulSoup(html, "html.parser")
        
        link = soup.select('div.dataBild')

    for i in range(len(link)):
        playerImgSrc.append(link[i].get('src'))
print(playerImgSrc)
Reply
#2
line 33:
for i in range(len(link)):
shouldn't this be:
for i in range(len(links)):

or if not, lines 33-34 need to be indented
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  AttributeError: 'NoneType' object in a parser - stops it apollo 4 3,963 May-28-2021, 02:13 PM
Last Post: Daring_T
  BeautifulSoup: 6k records - but stops after parsing 20 lines apollo 0 1,787 May-10-2021, 05:08 PM
Last Post: apollo

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020