Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Break out of nested loops
#3
I would refactor like this:
import requests
from bs4 import BeautifulSoup  # Or could import bs4 as bs

HEADERS = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
    'Accept-Language': 'en-US,en;q=0.9',
    'Referer': 'https://google.com',
    'DNT': '1',
}
BASE_URL = 'https://quotes.toscrape.com/page/{}/'
 
def find_tag(tag, url, headers):
    '''Iterator to get text for matching tags from url'''
    base_session = requests.Session()
    page = 1
    while True:
        page_session = base_session.get(url.format(page), headers=headers)  # Does not need to have own function
        if page_session.status_code != 200:
            continue  # avoid walking code off the page with multiple levels of indentation

        soup = BeautifulSoup(page_session.text, 'lxml')  # Avoid embedding package version info in code.
        for name in soup.select(tag):
            yield name.text 
 
        if not(soup.select('li.next')):
            break   # Found last page.  All done

        page += 1
 
authors = [author for author in find_tag('.author', BASE_URL, HEADERS)]
for author in sorted(set(authors)):
    print(author)
I don't have beautiful soup or requests installed, so this is completely untested. It may also miss important steps and this may not be the right way to extract tags. I just refactored your code.
Reply


Messages In This Thread
Break out of nested loops - by muzikman - Sep-15-2021, 06:02 PM
RE: Break out of nested loops - by muzikman - Sep-16-2021, 02:34 PM
RE: Break out of nested loops - by deanhystad - Sep-16-2021, 03:37 PM
RE: Break out of nested loops - by muzikman - Sep-16-2021, 05:53 PM
RE: Break out of nested loops - by muzikman - Sep-16-2021, 06:00 PM
RE: Break out of nested loops - by deanhystad - Sep-16-2021, 07:31 PM
RE: Break out of nested loops - by muzikman - Sep-17-2021, 03:06 PM
RE: Break out of nested loops - by muzikman - Sep-17-2021, 04:18 PM
RE: Break out of nested loops - by deanhystad - Sep-17-2021, 05:34 PM
RE: Break out of nested loops - by muzikman - Sep-17-2021, 05:48 PM
RE: Break out of nested loops - by deanhystad - Sep-17-2021, 07:31 PM
RE: Break out of nested loops - by muzikman - Sep-18-2021, 12:59 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  for loops break when I call the list I'm looping through Radical 4 1,995 Sep-18-2023, 07:52 AM
Last Post: buran
  reduce nested for-loops Phaze90 11 5,204 Mar-16-2023, 06:28 PM
Last Post: ndc85430
  Nested for loops: Iterating over columns of a DataFrame to plot on subplots dm222 0 2,977 Aug-19-2022, 11:07 AM
Last Post: dm222
  Nested for loops - help with iterating a variable outside of the main loop dm222 4 2,985 Aug-17-2022, 10:17 PM
Last Post: deanhystad
  breaking out of nested loops Skaperen 3 2,018 Jul-18-2022, 12:59 AM
Last Post: Skaperen
  How to break out of nested loops pace 11 7,740 Mar-03-2021, 06:25 PM
Last Post: pace
  Nested for Loops sammay 1 12,769 Jan-09-2021, 06:48 PM
Last Post: deanhystad
  How to make this function general to create binary numbers? (many nested for loops) dospina 4 6,327 Jun-24-2020, 04:05 AM
Last Post: deanhystad
  Conditionals, while loops, continue, break (PyBite 102) Drone4four 2 4,138 Jun-04-2020, 12:08 PM
Last Post: Drone4four
  Python beginner - nested while loops mikebarden 1 2,520 Jun-01-2020, 01:04 PM
Last Post: DPaul

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020