Couldn't Continue to Iterate The List in Python 3 - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html) +--- Thread: Couldn't Continue to Iterate The List in Python 3 (/thread-4172.html) |
Couldn't Continue to Iterate The List in Python 3 - OliverBird - Jul-27-2017 After several searching and modifies, I still couldn't figure it out. It stops at the 19th in the list. This is the error. Please help me resolve it. line 22 in <module> parent_td=[td for td in possible_tds if 'Product' in td.text][0] Index Error: List index out of range from urllib.request import urlopen as uReq from urllib.request import Request from bs4 import BeautifulSoup as soup import re ListName = "list" OpenList = open(ListName,"r") n=int(OpenList.readline()) for num in range(1,n+1): print(num) theList=OpenList.readline() AccessCME=Request(theList,headers={"User-Agent":"Mozilla/5.0"}) CMEPage=uReq(AccessCME).read() page_soup=soup(CMEPage,"html.parser") cme=page_soup.find("div",{"class":"cmeProduct section"}) FuturesContracts=cme.span.text.strip() possible_tds=page_soup.find_all('td',attrs={'class':"prodSpecAtribute"}) parent_td=[td for td in possible_tds if 'Product' in td.text][0] target = parent_td.fetchNextSiblings('td')[0].text.strip() first_take=re.sub('CME Globex:\s', '', target) BaseSymbol=re.sub(r'CME ClearPort:.*', '', first_take) print(FuturesContracts, BaseSymbol)The link is the list file: https://drive.google.com/file/d/0BzzXkoIWuMAHWEhaQktrT3BKdG8/view?usp=sharing Let me know if I need to clarify anything else. Improvements/suggestions are welcome. RE: Couldn't Continue to Iterate The List in Python 3 - nilamo - Jul-27-2017 Quote: line 22 in <module> parent_td=[td for td in possible_tds if 'Product' in td.text][0] So possible_tds doesn't have any elements, or at least none of them have "Product" in the text. Try print(possible_tds) and see what's actually there.
RE: Couldn't Continue to Iterate The List in Python 3 - ichabod801 - Jul-27-2017 Look at the source for link #19. None of the td tags with class prodSpecAttribute have Product in their text. So the list comprehension in line 22 comes back empty, and there is no 0 index, so you get the error. RE: Couldn't Continue to Iterate The List in Python 3 - buran - Jul-27-2017 http://www.cmegroup.com/trading/metals/base/copper_contract_specifications.html has no Product in any td.text, so parent_id is empty list and when you try to get element with index 0 you get this error. There are a lot of possible improvements. EDIT: Sorry for double/triple post :-) |