Python Forum

Full Version: struggling with loop/webscrape
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Hi guys,

i made variable:

page = input("Paste link:         ") + str("&LISTpg=")
and then another one:

for i in range(1, (int(max_pages)+1)):
    page = page + str(i)
    r = requests.get(page, headers=headers, params=params)
    content = (r.text)
    soup = BeautifulSoup(content, 'html.parser')
My question is... is it possible to somehow add value so it would be

www.example.com&LISTpg=1
www.example.com&LISTpg=2
www.example.com&LISTpg=3

instead of
www.example.com&LISTpg=1
www.example.com&LISTpg=12
www.example.com&LISTpg=123
www.example.com&LISTpg=1234
Maybe something along these lines:

>>> page = 'https://mysecretsite.com'
>>> for i in range(1, 4):
...     print(f'{page}#secret_anchor{i}')
... 
https://mysecretsite.com#secret_anchor1
https://mysecretsite.com#secret_anchor2
https://mysecretsite.com#secret_anchor3
Thanks but it doesn't work :(

what i am doing wrong?

for i in range(1, (int(max_pages)+1)):
    page = f'{page}#&LISTpg={i}'
    r = requests.get(page, headers=headers, params=params)
    content = (r.text)
    soup = BeautifulSoup(content, 'html.parser')
i get page + &LISTpg

"&LISTpg" is added to the end of sentence each time :|
(Sep-26-2019, 02:28 PM)zarize Wrote: [ -> ]My question is... is it possible to somehow add value so it would be

www.example.com&LISTpg=1
www.example.com&LISTpg=2
www.example.com&LISTpg=3

instead of
www.example.com&LISTpg=1
www.example.com&LISTpg=12
www.example.com&LISTpg=123
www.example.com&LISTpg=1234

Your initial question was about correct numbering and LISTpg= was expected and desired result.

'Doesn't work' is not correct description of the situation. It does work.

If you need something else, rephrase your question.
Thanks guys, i tried a bit and now i figured out what was wrong in my code :)

My code was adding "LISTpg" because i was trying to set variable from page again
As you can see i was using 'page' for both variables, that's why it wasn't working as intended.
page = 'https://mysecretsite.com'
for i in range(1, 4):
    page = f'{page}&LISTpg={i}'
That is correct version :P
page = 'https://mysecretsite.com'
for i in range(1, 4):
    page2 = f'{page}&LISTpg={i}'