Python Forum
Getting 'NoneType' object has no attribute 'find' error when WebScraping with BS
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Getting 'NoneType' object has no attribute 'find' error when WebScraping with BS
#1
Hello,
please help python experts. I want to iterate through multiple pages for products info but when loop gets to the second page of products it throws me 'NoneType' object has no attribute 'find' error. Everything looks find to me though.

from urllib.request import urlopen as uReq
from urllib.request import Request
import requests
from bs4 import BeautifulSoup as soup
pagenumbers = ["1", "2", "3", "4", "5", "6", "7", "8", "9","10","11","12","13","14","15","16","17","18","19","20","21","22","23","24","25","26","27","28","29","30"]

url = "https://www.costway.ca"
first_r = requests.get(url)
first_soup = soup(first_r.content,'lxml')
allpage_urls = first_soup.find('div',{'class':'category-content shop-by-catagory'}).find_all('a')
#print(allpage_urls)
for link in allpage_urls:
    originalurl = "https://www.costway.ca" + link['href'] 
    print(originalurl)
    for page in pagenumbers:
        Cat_page_url = originalurl+"?p="+page
        print(Cat_page_url)
        third_r = requests.get(Cat_page_url)
        third_soup = soup(third_r.content,'lxml')
        names=third_soup.find('div',{'class':'pro_datu'}).find_all('a')
        for y in names:
            product_url = y['href']
            print(product_url)
            fourth_r = requests.get(product_url)
            fourth_soup = soup(fourth_r.content,'lxml')
            product_names=fourth_soup.find('div',{'class':'prod_nx'}).find('div',{'class':'orat'}).find('span').find_next_sibling('span')
            Product_number = product_names.text
            print(Product_number)  
https://www.costway.ca/set-of-2-adjustable-pu-leather-backless-bar-stools.html
Item No: 78435109
https://www.costway.ca/29-set-of-2-saddle-nailhead-kitchen-counter-chair.html
Item No: 13678425
https://www.costway.ca/furniture/chairs.html?p=2
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-199-4b0149b6ae67> in <module>
     20             fourth_r = requests.get(product_url)
     21             fourth_soup = soup(fourth_r.content,'lxml')
---> 22             product_names=fourth_soup.find('div',{'class':'prod_nx'}).find('div',{'class':'orat'}).find('span').find_next_sibling('span')
     23             Product_number = product_names.text
     24             print(Product_number)

AttributeError: 'NoneType' object has no attribute 'find'
Thank you very much in advance for looking into it!
Reply
#2
You need to post the entire traceback if you want help, as it gives useful information about the problem (e.g. the line on which the exception was raised). What have you done to debug the problem, though? Something is clearly None so you need to work backwards from there and work out why.
Reply
#3
(Aug-17-2021, 03:42 PM)ndc85430 Wrote: You need to post the entire traceback if you want help, as it gives useful information about the problem (e.g. the line on which the exception was raised). What have you done to debug the problem, though? Something is clearly None so you need to work backwards from there and work out why.


thank you for looking into this ndc85430.

Here is the entire error message from the query:

https://www.costway.ca/set-of-2-adjustable-pu-leather-backless-bar-stools.html
Item No: 78435109
https://www.costway.ca/29-set-of-2-saddle-nailhead-kitchen-counter-chair.html
Item No: 13678425
https://www.costway.ca/furniture/chairs.html?p=2
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-199-4b0149b6ae67> in <module>
     20             fourth_r = requests.get(product_url)
     21             fourth_soup = soup(fourth_r.content,'lxml')
---> 22             product_names=fourth_soup.find('div',{'class':'prod_nx'}).find('div',{'class':'orat'}).find('span').find_next_sibling('span')
     23             Product_number = product_names.text
     24             print(Product_number)

AttributeError: 'NoneType' object has no attribute 'find'
Well, I understand that error message indicates that there is none in the page when loop gets to the next page. Though when I run the query only for that next page result is successful. That is why I am puzzled here.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  AttributeError: 'list' object has no attribute 'values' ilknurg 4 185 Jan-19-2022, 08:33 AM
Last Post: menator01
  AttributeError: 'NoneType' object has no attribute 'group' MaartenRo 3 228 Jan-02-2022, 09:53 AM
Last Post: snippsat
  AttributeError: 'NoneType' object has no attribute 'group' MaartenRo 3 193 Jan-01-2022, 04:16 PM
Last Post: MaartenRo
  AttributeError: 'Message' object has no attribute 'split' helpme1 3 2,205 Dec-14-2021, 07:20 AM
Last Post: alexaminar
  How to access parent object attribute Pavel_47 2 409 Nov-19-2021, 09:36 PM
Last Post: deanhystad
  what if __getattr__() can't find an attribute, either? Skaperen 1 406 Oct-08-2021, 08:43 PM
Last Post: Yoriz
  TypeError: 'NoneType' object is not subscriptable syafiq14 2 2,089 Aug-26-2021, 06:18 AM
Last Post: alexaminar
  Attribute Error received not understood (Please Help) crocolicious 5 1,017 Jun-19-2021, 08:45 PM
Last Post: crocolicious
  AttributeError: 'DataFrame' object has no attribute 'Articles' Anldra12 5 2,709 Jun-17-2021, 04:54 PM
Last Post: Larz60+
  AttributeError: 'Node' object has no attribute 'insert' Anldra12 4 2,190 May-11-2021, 10:12 AM
Last Post: Anldra12

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020