Python Forum

Hello there,

I'm a newbie and still learning python and I have some problem with my scraping code. Namely, I would like to scrap data from 273 pages of the same website . My code works when I want to scrap one page, but when I add more pages to code it scrapes only from the last one. There is code below:

import requests
import bs4
from bs4 import BeautifulSoup
headers = {
    'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36'
}
page = 1
while page !=274:
    url=f"https://www.kmotorshop.com/en/article-list/list/0/0/tree-shop%7C258?page={page}&itemsPerPage=50&style=1"
    page = page + 1
r = requests.get(url,{'headers':headers})
soup = bs4.BeautifulSoup(r.text,'html.parser')
i = 0
while i != 50:
    print (soup.findAll('div',{'class': 'product-line__heading'})[i].find('a').text)
    i+=1

Can someone help me to connect all pages and scrape them all together?

In the while loop, the variable url is overwritten each time it loops leaving it with only the last value,
When using variables in loops you could either interact with the current item ieurl inside of the loop or use something that can contain multiple items like a list