I want to loop a url for scrape all products info from each pages but two parameters of this url automatically changes when it go to next page. This two parameters changes "2&qid=1552585481" and "&ref=sr_pg_2" such as when I go to 3rd page it will be "&qid=1552585493' and "&ref=sr_pg_3". Below my code given:
import requests from bs4 import BeautifulSoup url = 'https://www.amazon.com/s?k=%2Fwomen-dress%2Fs%3Fk%3Dwomen+dress&qid=1552587323&ref=sr_pg_1' #page_url_setup for page in range(10): print('---', page, '---') r = requests.get(url + str(page)) soup = BeautifulSoup(r.content, "html.parser") for link in soup.find_all('a',{'class':'ebayui-pagination__control','rel':'next'}): print("<a href='>%s'>%s</a>" % (link.get("href"), link.text))