Getting 'NoneType' object has no attribute 'find' error when WebScraping with BS - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: General Coding Help (https://python-forum.io/forum-8.html) +--- Thread: Getting 'NoneType' object has no attribute 'find' error when WebScraping with BS (/thread-34646.html) |
Getting 'NoneType' object has no attribute 'find' error when WebScraping with BS - Franky77 - Aug-17-2021 Hello, please help python experts. I want to iterate through multiple pages for products info but when loop gets to the second page of products it throws me 'NoneType' object has no attribute 'find' error. Everything looks find to me though. from urllib.request import urlopen as uReq from urllib.request import Request import requests from bs4 import BeautifulSoup as soup pagenumbers = ["1", "2", "3", "4", "5", "6", "7", "8", "9","10","11","12","13","14","15","16","17","18","19","20","21","22","23","24","25","26","27","28","29","30"] url = "https://www.costway.ca" first_r = requests.get(url) first_soup = soup(first_r.content,'lxml') allpage_urls = first_soup.find('div',{'class':'category-content shop-by-catagory'}).find_all('a') #print(allpage_urls) for link in allpage_urls: originalurl = "https://www.costway.ca" + link['href'] print(originalurl) for page in pagenumbers: Cat_page_url = originalurl+"?p="+page print(Cat_page_url) third_r = requests.get(Cat_page_url) third_soup = soup(third_r.content,'lxml') names=third_soup.find('div',{'class':'pro_datu'}).find_all('a') for y in names: product_url = y['href'] print(product_url) fourth_r = requests.get(product_url) fourth_soup = soup(fourth_r.content,'lxml') product_names=fourth_soup.find('div',{'class':'prod_nx'}).find('div',{'class':'orat'}).find('span').find_next_sibling('span') Product_number = product_names.text print(Product_number) https://www.costway.ca/set-of-2-adjustable-pu-leather-backless-bar-stools.html Item No: 78435109 https://www.costway.ca/29-set-of-2-saddle-nailhead-kitchen-counter-chair.html Item No: 13678425 https://www.costway.ca/furniture/chairs.html?p=2 --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-199-4b0149b6ae67> in <module> 20 fourth_r = requests.get(product_url) 21 fourth_soup = soup(fourth_r.content,'lxml') ---> 22 product_names=fourth_soup.find('div',{'class':'prod_nx'}).find('div',{'class':'orat'}).find('span').find_next_sibling('span') 23 Product_number = product_names.text 24 print(Product_number) AttributeError: 'NoneType' object has no attribute 'find'Thank you very much in advance for looking into it! RE: Getting 'NoneType' object has no attribute 'find' error when WebScraping with BS - ndc85430 - Aug-17-2021 You need to post the entire traceback if you want help, as it gives useful information about the problem (e.g. the line on which the exception was raised). What have you done to debug the problem, though? Something is clearly None so you need to work backwards from there and work out why.
RE: Getting 'NoneType' object has no attribute 'find' error when WebScraping with BS - Franky77 - Aug-17-2021 (Aug-17-2021, 03:42 PM)ndc85430 Wrote: You need to post the entire traceback if you want help, as it gives useful information about the problem (e.g. the line on which the exception was raised). What have you done to debug the problem, though? Something is clearly thank you for looking into this ndc85430. Here is the entire error message from the query: https://www.costway.ca/set-of-2-adjustable-pu-leather-backless-bar-stools.html Item No: 78435109 https://www.costway.ca/29-set-of-2-saddle-nailhead-kitchen-counter-chair.html Item No: 13678425 https://www.costway.ca/furniture/chairs.html?p=2 --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-199-4b0149b6ae67> in <module> 20 fourth_r = requests.get(product_url) 21 fourth_soup = soup(fourth_r.content,'lxml') ---> 22 product_names=fourth_soup.find('div',{'class':'prod_nx'}).find('div',{'class':'orat'}).find('span').find_next_sibling('span') 23 Product_number = product_names.text 24 print(Product_number) AttributeError: 'NoneType' object has no attribute 'find'Well, I understand that error message indicates that there is none in the page when loop gets to the next page. Though when I run the query only for that next page result is successful. That is why I am puzzled here. |