Python Forum
Unable to fetch product url using BeautifulSoup with Python3.6
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Unable to fetch product url using BeautifulSoup with Python3.6
#4
please post enough code to run without work, it forces us to improvise, perhaps causing differnet results:

My attempt:
from bs4 import BeautifulSoup
import requests


def get_soup(url):
    soup = None
    try:
        response = requests.get(url, headers=headers, timeout=timeout)
        if response.status_code == 200:
            html = response.content
            soup = BeautifulSoup(html, "html.parser")
    except Exception as exc:
        print("error::", str(exc))
    finally:
        return soup
 
def get_category_urls(url):
    soup = get_soup(url)
    cat_urls = []
    try:
        categories = soup.find('div', attrs={'id': 'menu_oc'})
        if categories is not None:
            for c in categories.findAll('a'):
                if c['href'] is not None:
                    cat_urls.append(c['href'])
    except Exception as exc:
        print("error..", str(exc))
    finally:
        print("category urls::", cat_urls)
        return cat_urls

def main():
    url = 'http://www.infantree.net/shop/'
    soup = get_soup(url)

if __name__ == '__main__':
    main()
Error:
error:: name 'headers' is not defined
Reply


Messages In This Thread
RE: Issue while srapping product urls in python 3.6? - by Larz60+ - Jun-05-2018, 12:50 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Unable to convert browser generated xml to parse in BeautifulSoup Nik1811 0 354 Mar-22-2024, 01:37 PM
Last Post: Nik1811
  All product links to products on a website MarionStorm 0 1,114 Jun-02-2022, 11:17 PM
Last Post: MarionStorm
  Help with python3 (BeautifulSoup) freaknez 1 3,042 Sep-14-2018, 09:50 PM
Last Post: Larz60+
  My Django 2.0.6 logging is not working while product merging PrateekG 0 2,195 Jul-26-2018, 02:24 PM
Last Post: PrateekG
  Need help to get product details using BeautifulSoup+Python3.6! PrateekG 2 2,915 Jun-27-2018, 08:52 AM
Last Post: PrateekG
  How to fetch latitude,longitude from location and save them separately in db(Django2) PrateekG 0 2,670 Jun-21-2018, 04:40 AM
Last Post: PrateekG
  Getting 'list index out of range' while fetching product details using BeautifulSoup? PrateekG 8 8,259 Jun-06-2018, 12:15 PM
Last Post: snippsat
  Not able to fetch data from a webpage sumandas89 3 4,795 Dec-21-2017, 08:30 AM
Last Post: sumandas89
  How do I fetch values from db to Select Options using Flask? progShubham 2 17,793 Jul-25-2017, 05:52 PM
Last Post: nilamo

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020