Python Forum
getting links from webpage and store it into an array
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
getting links from webpage and store it into an array
#2
Is there a question? It appears that problem is related to missing module and not actual code?

Some observations nevertheless.

Is there particular need to ignore function naming convention set in PEP8 ("Function names should be lowercase, with words separated by underscores as necessary to improve readability.")?

Is there need to use re? Every time I see re in context of web parsing it reminds me this legendary StackOverflow answer.

My take (without re) would be something along those lines:

import requests
import bs4 as bs

url = 'https://www.fdic.gov/resources/resolutions/bank-failures/failed-bank-list/'
response = requests.get(url)
soup = bs.BeautifulSoup(response.text, 'lxml')
table = soup.find('table')

links = []
for row in table.find_all('tr'):
    for link in row.find_all('a'):
        links.append(f'{url}{link.get("href")}')
Just for fun of irritating others (and future yourself if you return to this code say, in couple of weeks) much of it can be condensed into two rows:

table = bs.BeautifulSoup(requests.get(url).text, 'lxml').find('table')
links = [f'{url}{link.get("href")}' for row in table.find_all('tr') for link in row.find_all('a')]


In real life scenario I would probably write generator function instead of constructing list just for single iteration.
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Reply


Messages In This Thread
RE: getting links from webpage and store it into an array - by perfringo - May-22-2021, 03:35 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  How to extract links from grid located on webpage Pavel_47 5 1,473 Aug-04-2023, 12:43 PM
Last Post: Gaurav_Kumar
  webscrapping links and then enter those links to scrape data kirito85 2 3,225 Jun-13-2019, 02:23 AM
Last Post: kirito85

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020