Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
facebook friends crawler
#11
hey guys, so i'm still working on the loading friends part of the program but i'm almost there (i tried a while loop with a definite nr of loops, i tried the page length trick and i wanna try the tag search thing; i think i'll go with the tag search but i'm not sure yet how big of an impact will it have on performance).

i've been googling lately about databases and how to store friends and connections and i found out something about graph databases and whatever but i have no idea how to implement that into my code.
i'm thinking i should make a class with a name and id subclasses (i'll probably wanna add some more subclasses later) and then store each instance of the class into a db file but i don't know how to even start thinking about doing that.

you guys have any ideas where i might get started with storing the output of the function into a db? (preferrably a graph db where you don't have to make a list for every person out there with their friends).
also, any idea if i need a db client? i heard about mysql and sqlite but i don't know if they can make the type of db i'm looking for (i searched on youtube but all i could find was graph as in, a line chart for functions).
also, is it a good idea of putting the output of the crawler into class instances?

thanks
Reply
#12
hey guys, it's been a while since my last update.
i've managed to get the crawler to reliably scan people's friends lists, even people with ~5000 friends in a reasonable timeframe (~30 s).
now i'm working on the graph part of the function.
i'm trying to store all the people and ids into a database and make it so that every person's friends list is connected to the other people's lists.
problem is that i can't seem to figure out how to do that.

i thought about a couple of methods but they range from useless af to i have no idea if this is even possible.
i'm also using the sqlite3 module for python and i'm using the sqlite3 db browser to view the results.

1st method and most intuitive and least efficient is: create one table for every person.
it goes like this:
  • say we have people A, B, C, D and they're all friends with each other.
  • create table A
  • scan person A
  • find people B, C, D
  • create tables B, C, D
  • add A to tables B, C, D
  • add B, C, D to table A
  • then move to person B and repeat the process without having to add person A again
with this method you'll have nr tables = nr or people on the network and you need to have some sort of link between tables so that when you try to look for a common link between people the program goes through all the linked tables.
you'll also have a lot of repeat entries in each table (person A appears 3 times, once in each table).

2nd method i thought about is slightly more elegant but it's more or less the same thing.
  • have 1 master table with all the people A, B, C, D: table I
  • have another table with same people; table II
  • have each entry in table Ii reference entries in table I based on person A's friends list
with this method python or sqlite (or both) need to be able to identify lists of links between each entry pairs (table II > table I) and based on that list search through that person's friends list.
Ex.: we have 3 people this time: A, B, C.
and B is friends with A and D but A and D are only friends with B but not with one another.
you want to tell python to find a link between A and C.
first scan A and find B.
scan C and find B.
show B as successful result.

3rd method i thought is method 2 but even more trimmed down.
  • create 1 table with everybody
  • create links between every entry in the table based on their friends list
this is the most efficient one i could think if but i still have to be able to create links or references between entries in the table.

so, my question, how do i, is it even possible to have links or references between 1 entry and multiple other entries in sqlite3? :D
thanks guys
Reply
#13
(Sep-02-2017, 01:36 PM)edithegodfather Wrote: hey guys, it's been a while since my last update. i've managed to get the crawler to reliably scan people's friends lists, even people with ~5000 friends in a reasonable timeframe (~30 s). now i'm working on the graph part of the function. i'm trying to store all the people and ids into a database and make it so that every person's friends list is connected to the other people's lists. problem is that i can't seem to figure out how to do that. i thought about a couple of methods but they range from useless af to i have no idea if this is even possible. i'm also using the sqlite3 module for python and i'm using the sqlite3 db browser to view the results. 1st method and most intuitive and least efficient is: create one table for every person. it goes like this:
  • say we have people A, B, C, D and they're all friends with each other.
  • create table A
  • scan person A
  • find people B, C, D
  • create tables B, C, D
  • add A to tables B, C, D
  • add B, C, D to table A
  • then move to person B and repeat the process without having to add person A again
with this method you'll have nr tables = nr or people on the network and you need to have some sort of link between tables so that when you try to look for a common link between people the program goes through all the linked tables. you'll also have a lot of repeat entries in each table (person A appears 3 times, once in each table). 2nd method i thought about is slightly more elegant but it's more or less the same thing.
  • have 1 master table with all the people A, B, C, D: table I
  • have another table with same people; table II
  • have each entry in table Ii reference entries in table I based on person A's friends list
with this method python or sqlite (or both) need to be able to identify lists of links between each entry pairs (table II > table I) and based on that list search through that person's friends list. Ex.: we have 3 people this time: A, B, C. and B is friends with A and D but A and D are only friends with B but not with one another. you want to tell python to find a link between A and C. first scan A and find B. scan C and find B. show B as successful result. 3rd method i thought is method 2 but even more trimmed down.
  • create 1 table with everybody
  • create links between every entry in the table based on their friends list
this is the most efficient one i could think if but i still have to be able to create links or references between entries in the table. so, my question, how do i, is it even possible to have links or references between 1 entry and multiple other entries in sqlite3? :D thanks guys
Hi there. I'm finding the way to crawl facebook friends either. Can u share with me the code( or maybe the way ) u do it concretely?
Thanks!
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Automating Facebook Posts mieciow 2 1,335 Aug-07-2023, 12:34 PM
Last Post: Gaurav_Kumar
  Web scrapping login facebook credentials kosmas9 0 1,912 Aug-17-2020, 01:33 PM
Last Post: kosmas9
  How to click facebook message button JanelleGuthrie 2 2,372 May-14-2020, 06:02 PM
Last Post: Larz60+
  Web Crawler help Mr_Mafia 2 1,847 Apr-04-2020, 07:20 PM
Last Post: Mr_Mafia
  Web Crawler help takaa 39 26,860 Apr-26-2019, 12:14 PM
Last Post: stateitreal
  Application like simpler facebook wall in python framework seidman 1 3,274 Mar-11-2018, 05:07 PM
Last Post: wavic
  Need help with Django 2.0 + Facebook SDK avtrrr 4 4,537 Jan-31-2018, 08:36 AM
Last Post: Larz60+
  I need request POST for Facebook in My Profile Kalet 4 4,246 Sep-27-2017, 05:53 PM
Last Post: Kalet
  Facebook AutoPost badpandahere 6 4,893 Sep-12-2017, 11:20 AM
Last Post: badpandahere
  facebook scraping metulburr 3 8,249 Jun-02-2017, 01:00 AM
Last Post: metulburr

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020