Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
webscrapping lists to dataframe
#1
Hi i am doing some webscrapping and is stuck at the follow codes below:

For some reason the record_lost is not inserted into the dataframe. Unable to debug this portion and would appreciate any kind help on this.

record_list = [list(item) for item in list(zip(url_list, title_list, description_list))]
df = pd.DataFrame(data=record_list,columns=['URL','Title', 'Description'])
Reply
#2
Post a sample of output record_list.
Reply
#3
Isn't there needless conversions in creating record_list?

>>> a = [1, 2, 3]                                                              
>>> b = 'abc'                                                                  
>>> c = [10, 20, 30] 
>>> for row in zip(a, b, c): 
....    print(row) 
....                                                                           
(1, 'a', 10)
(2, 'b', 20)
(3, 'c', 30)
>>> for row in list(zip(a, b, c)): 
....    print(row) 
....                                                                           
(1, 'a', 10)
(2, 'b', 20)
(3, 'c', 30)
>>> [list(item) for item in list(zip(a, b, c))]                                
[[1, 'a', 10], [2, 'b', 20], [3, 'c', 30]]
>>> list(zip(a, b, c))                                                         
[(1, 'a', 10), (2, 'b', 20), (3, 'c', 30)]
DataFrame data source can be iterable (documentation: data : ndarray (structured or homogeneous), Iterable, dict, or DataFrame) therefore even converting to list is not necessary:

>>> df = pd.DataFrame(zip(a, b, c), columns = ('First', 'Second', 'Third'))
>>> df
   First Second  Third
0      1      a     10
1      2      b     20
2      3      c     30
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Reply
#4
Hi snippsat, thanks for your quick reply. Actually i am reusing code i found and editing it. I found out why the lists not working cus my webscrapping didnt pull the values successfully into the lists and they are empty which is why they didnt work. Once i fixed that it is working now. I will reread your dataframe advice again to understand it.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Webscrapping sport betting websites KoinKoin 3 5,336 Nov-08-2023, 03:00 PM
Last Post: LoriBrown
  webscrapping links from pandas dataframe Wolverin 2 2,233 Aug-28-2023, 12:07 PM
Last Post: Gaurav_Kumar
  Webscrapping of Images that requires Authentication junos4350 1 1,945 Jun-08-2020, 08:32 AM
Last Post: alekson
  webscrapping links and then enter those links to scrape data kirito85 2 3,143 Jun-13-2019, 02:23 AM
Last Post: kirito85

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020