Python Forum
Getting 'NoneType' object has no attribute 'find' error when WebScraping with BS
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Getting 'NoneType' object has no attribute 'find' error when WebScraping with BS
#1
Hello,
please help python experts. I want to iterate through multiple pages for products info but when loop gets to the second page of products it throws me 'NoneType' object has no attribute 'find' error. Everything looks find to me though.

from urllib.request import urlopen as uReq
from urllib.request import Request
import requests
from bs4 import BeautifulSoup as soup
pagenumbers = ["1", "2", "3", "4", "5", "6", "7", "8", "9","10","11","12","13","14","15","16","17","18","19","20","21","22","23","24","25","26","27","28","29","30"]

url = "https://www.costway.ca"
first_r = requests.get(url)
first_soup = soup(first_r.content,'lxml')
allpage_urls = first_soup.find('div',{'class':'category-content shop-by-catagory'}).find_all('a')
#print(allpage_urls)
for link in allpage_urls:
    originalurl = "https://www.costway.ca" + link['href'] 
    print(originalurl)
    for page in pagenumbers:
        Cat_page_url = originalurl+"?p="+page
        print(Cat_page_url)
        third_r = requests.get(Cat_page_url)
        third_soup = soup(third_r.content,'lxml')
        names=third_soup.find('div',{'class':'pro_datu'}).find_all('a')
        for y in names:
            product_url = y['href']
            print(product_url)
            fourth_r = requests.get(product_url)
            fourth_soup = soup(fourth_r.content,'lxml')
            product_names=fourth_soup.find('div',{'class':'prod_nx'}).find('div',{'class':'orat'}).find('span').find_next_sibling('span')
            Product_number = product_names.text
            print(Product_number)  
https://www.costway.ca/set-of-2-adjustable-pu-leather-backless-bar-stools.html
Item No: 78435109
https://www.costway.ca/29-set-of-2-saddle-nailhead-kitchen-counter-chair.html
Item No: 13678425
https://www.costway.ca/furniture/chairs.html?p=2
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-199-4b0149b6ae67> in <module>
     20             fourth_r = requests.get(product_url)
     21             fourth_soup = soup(fourth_r.content,'lxml')
---> 22             product_names=fourth_soup.find('div',{'class':'prod_nx'}).find('div',{'class':'orat'}).find('span').find_next_sibling('span')
     23             Product_number = product_names.text
     24             print(Product_number)

AttributeError: 'NoneType' object has no attribute 'find'
Thank you very much in advance for looking into it!
Reply


Messages In This Thread
Getting 'NoneType' object has no attribute 'find' error when WebScraping with BS - by Franky77 - Aug-17-2021, 02:48 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
Bug TypeError: 'NoneType' object is not subscriptable TheLummen 4 773 Nov-27-2023, 11:34 AM
Last Post: TheLummen
  Error: audioio has no attribute 'AudioOut' netwrok 3 687 Oct-22-2023, 05:53 PM
Last Post: netwrok
  TypeError: 'NoneType' object is not callable akbarza 4 1,083 Aug-24-2023, 05:14 PM
Last Post: snippsat
  AttributeError: '_tkinter.tkapp' object has no attribute 'username' Konstantin23 4 1,817 Aug-04-2023, 12:41 PM
Last Post: Konstantin23
  Need help with 'str' object is not callable error. Fare 4 885 Jul-23-2023, 02:25 PM
Last Post: Fare
  Python: Regex is not good for re.search (AttributeError: 'NoneType' object has no att Melcu54 9 1,576 Jun-28-2023, 11:13 AM
Last Post: Melcu54
  Python: AttributeError: 'PageObject' object has no attribute 'extract_images' Melcu54 2 4,008 Jun-18-2023, 07:47 PM
Last Post: Melcu54
  Object attribute behavior different in 2 scripts db042190 1 781 Jun-14-2023, 12:37 PM
Last Post: deanhystad
  cx_oracle Error - AttributeError: 'function' object has no attribute 'cursor' birajdarmm 1 2,456 Apr-15-2023, 05:17 PM
Last Post: deanhystad
  Pandas AttributeError: 'DataFrame' object has no attribute 'concat' Sameer33 5 5,810 Feb-17-2023, 06:01 PM
Last Post: Sameer33

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020