Python Forum
webscrapping lists to dataframe - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/Forum-Python-Coding)
+--- Forum: Web Scraping & Web Development (https://python-forum.io/Forum-Web-Scraping-Web-Development)
+--- Thread: webscrapping lists to dataframe (/Thread-webscrapping-lists-to-dataframe)



webscrapping lists to dataframe - kirito85 - Jun-10-2019

Hi i am doing some webscrapping and is stuck at the follow codes below:

For some reason the record_lost is not inserted into the dataframe. Unable to debug this portion and would appreciate any kind help on this.

record_list = [list(item) for item in list(zip(url_list, title_list, description_list))]
df = pd.DataFrame(data=record_list,columns=['URL','Title', 'Description'])




RE: webscrapping lists to dataframe - snippsat - Jun-10-2019

Post a sample of output record_list.


RE: webscrapping lists to dataframe - perfringo - Jun-10-2019

Isn't there needless conversions in creating record_list?

>>> a = [1, 2, 3]                                                              
>>> b = 'abc'                                                                  
>>> c = [10, 20, 30] 
>>> for row in zip(a, b, c): 
....    print(row) 
....                                                                           
(1, 'a', 10)
(2, 'b', 20)
(3, 'c', 30)
>>> for row in list(zip(a, b, c)): 
....    print(row) 
....                                                                           
(1, 'a', 10)
(2, 'b', 20)
(3, 'c', 30)
>>> [list(item) for item in list(zip(a, b, c))]                                
[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]]
>>> list(zip(a, b, c))                                                         
[(1, 'a', 10), (2, 'b', 20), (3, 'c', 30)]
DataFrame data source can be iterable (documentation: data : ndarray (structured or homogeneous), Iterable, dict, or DataFrame) therefore even converting to list is not necessary:

>>> df = pd.DataFrame(zip(a, b, c), columns = ('First', 'Second', 'Third'))
>>> df
   First Second  Third
0      1      a     10
1      2      b     20
2      3      c     30



RE: webscrapping lists to dataframe - kirito85 - Jun-10-2019

Hi snippsat, thanks for your quick reply. Actually i am reusing code i found and editing it. I found out why the lists not working cus my webscrapping didnt pull the values successfully into the lists and they are empty which is why they didnt work. Once i fixed that it is working now. I will reread your dataframe advice again to understand it.