Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
working with lxml and requests
#23
(Apr-19-2018, 05:43 PM)gentoobob Wrote: Let me ask you, will I have to have a separate "soup" variable for each URL or can I cram all the pages into one variable?
You pass url to Requests and BS in same loop.
Example:
start = 1
stop = 5
for page in range(start, stop):
    url = 'https://10.10.10.0/vmrest/users?rowsPerPage=2000&pageNumber={}'.format(page)
    url_get = requests.get(url)
    soup = BeautifulSoup(url_get.content, 'lxml')
    foo = soup.find('do scraping')
    # save foo
Quote:or can I cram all the pages into one variable?
Not variable but a data structure like eg list,you can collect urls.
from pprint import pprint

urls = []
start = 1
stop = 5
for page in range(start, stop):
    url = 'https://10.10.10.0/vmrest/users?rowsPerPage=2000&pageNumber={}'.format(page)
    urls.append(url)

pprint(urls)
Output:
['https://10.10.10.0/vmrest/users?rowsPerPage=2000&pageNumber=1', 'https://10.10.10.0/vmrest/users?rowsPerPage=2000&pageNumber=2', 'https://10.10.10.0/vmrest/users?rowsPerPage=2000&pageNumber=3', 'https://10.10.10.0/vmrest/users?rowsPerPage=2000&pageNumber=4']
Reply


Messages In This Thread
working with lxml and requests - by gentoobob - Apr-18-2018, 02:47 PM
RE: working with lxml and requests - by nilamo - Apr-18-2018, 03:04 PM
RE: working with lxml and requests - by gentoobob - Apr-18-2018, 04:43 PM
RE: working with lxml and requests - by Larz60+ - Apr-18-2018, 05:07 PM
RE: working with lxml and requests - by gentoobob - Apr-18-2018, 05:16 PM
RE: working with lxml and requests - by nilamo - Apr-18-2018, 05:52 PM
RE: working with lxml and requests - by gentoobob - Apr-18-2018, 06:16 PM
RE: working with lxml and requests - by nilamo - Apr-18-2018, 06:53 PM
RE: working with lxml and requests - by gentoobob - Apr-18-2018, 07:30 PM
RE: working with lxml and requests - by nilamo - Apr-18-2018, 07:35 PM
RE: working with lxml and requests - by snippsat - Apr-18-2018, 08:08 PM
RE: working with lxml and requests - by nilamo - Apr-18-2018, 08:15 PM
RE: working with lxml and requests - by gentoobob - Apr-18-2018, 08:21 PM
RE: working with lxml and requests - by snippsat - Apr-18-2018, 08:27 PM
RE: working with lxml and requests - by snippsat - Apr-18-2018, 10:48 PM
RE: working with lxml and requests - by gentoobob - Apr-19-2018, 01:08 PM
RE: working with lxml and requests - by snippsat - Apr-19-2018, 01:36 PM
RE: working with lxml and requests - by gentoobob - Apr-19-2018, 01:39 PM
RE: working with lxml and requests - by nilamo - Apr-19-2018, 02:44 PM
RE: working with lxml and requests - by gentoobob - Apr-19-2018, 04:41 PM
RE: working with lxml and requests - by snippsat - Apr-19-2018, 05:17 PM
RE: working with lxml and requests - by gentoobob - Apr-19-2018, 05:43 PM
RE: working with lxml and requests - by snippsat - Apr-19-2018, 06:49 PM
RE: working with lxml and requests - by gentoobob - Apr-19-2018, 06:54 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  POST requests - different requests return the same response Default_001 3 1,998 Mar-10-2022, 11:26 PM
Last Post: Default_001
  requests module is not working varsh 3 3,880 Sep-10-2020, 03:53 PM
Last Post: buran
  Flask, Posgresql - Multiple requests are not working bmaganti 5 2,840 Feb-20-2020, 03:02 PM
Last Post: bmaganti
  [Help]xpath is not working with lxml mr_byte31 3 6,330 Jul-22-2018, 04:10 PM
Last Post: stranac

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020