+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html)
+--- Thread: Unexpected output (/thread-25561.html)
Unexpected output - JacobK - Apr-03-2020
Hi!
I'm having problems with my code. I've managed to scrape a player ID for 500 football players, which I then want to add to a URL and save in a list. It should look something like this: https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/342228
The problem is that my loop adds the same player ID, 500 times, and creating 500 identical URL:s. What am I doing wrong?
from bs4 import BeautifulSoup
import requests
import pandas as pd
playerID = []
playerImage = []
playerProfile = []
result = []
for page in range(1, 21):
r = requests.get("https://www.transfermarkt.com/spieler-statistik/wertvollstespieler/marktwertetop? land_id=0&ausrichtung=alle&spielerposition_id=alle&altersklasse=alle&jahrgang=0&kontinent_id=0&plus=1",
params= {"page": page},
headers= {"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0"}
)
soup = BeautifulSoup(r.content, "html.parser")
links = soup.select('a.spielprofil_tooltip')
for i in range(len(links)):
playerID.append(links[i].get('id'))
for j in playerID[i]:
playerProfile.append("https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/" + playerID[i])
print(playerProfile)Output:'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789', 'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/131789']RE: Unexpected output - ndc85430 - Apr-03-2020
Lines 23-24: are you using the right loop variable?
RE: Unexpected output - JacobK - Apr-03-2020
(Apr-03-2020, 07:36 AM)ndc85430 Wrote: Lines 23-24: are you using the right loop variable?
Should it be i again? Sorry, I'm bad at this, could you help me out?
RE: Unexpected output - snippsat - Apr-03-2020
Try this.
from bs4 import BeautifulSoup
import requests
import pandas as pd
playerID = []
playerImage = []
playerProfile = []
result = []
for page in range(1, 3):
r = requests.get(
"https://www.transfermarkt.com/spieler-statistik/wertvollstespieler/marktwertetop? land_id=0&ausrichtung=alle&spielerposition_id=alle&altersklasse=alle&jahrgang=0&kontinent_id=0&plus=1",
params={"page": page},
headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0"},
)
soup = BeautifulSoup(r.content, "html.parser")
links = soup.select("a.spielprofil_tooltip")
for tag in links:
tag_id = tag.get("id")
playerID.append(tag_id)
playerProfile.append(f'https://www.transfermarkt.com/chaher-idjihadi/profil/spieler/{tag_id}')