|
Search Results
|
| Post |
Author |
Forum |
Replies |
Views |
Posted
[asc]
|
| |
|
Thread: facebook friends crawler
Post: RE: facebook friends crawler
| hey guys, it's been a while since my last update.
i've managed to get the crawler to reliably scan people's friends lists, even people with ~5000 friends in a reasonable timeframe (~30 s).
now i'm wor... |
|
edithegodfather |
Web Scraping & Web Development |
12 |
21,040 |
Sep-02-2017, 01:36 PM |
| |
|
Thread: facebook friends crawler
Post: RE: facebook friends crawler
| hey guys, so i'm still working on the loading friends part of the program but i'm almost there (i tried a while loop with a definite nr of loops, i tried the page length trick and i wanna try the tag ... |
|
edithegodfather |
Web Scraping & Web Development |
12 |
21,040 |
Aug-27-2017, 02:07 AM |
| |
|
Thread: facebook friends crawler
Post: RE: facebook friends crawler
| hey guys
so i've successfully managed to load the friends page and make it grab all the friends' names and ids but problem is that the window only shows 20 friends until you scroll down when it loads... |
|
edithegodfather |
Web Scraping & Web Development |
12 |
21,040 |
Aug-24-2017, 12:37 AM |
| |
|
Thread: facebook friends crawler
Post: RE: facebook friends crawler
| hey man, you're not gonna believe how ridiculously easy this was, like, embarrasingly easy. xD
browser = webdriver.Chrome(CHROMEDRIVER LOCATION)
def facebook():
# here goes the login part of the... |
|
edithegodfather |
Web Scraping & Web Development |
12 |
21,040 |
Aug-21-2017, 07:45 PM |
| |
|
Thread: facebook friends crawler
Post: RE: facebook friends crawler
| hmm, you're right. the html tag has a <class="_6-6"> tag that stays consistent throughout.
i'll try and use the html tag instead of the css and see what happens.
say, can i at least make chrome... |
|
edithegodfather |
Web Scraping & Web Development |
12 |
21,040 |
Aug-21-2017, 03:24 AM |
| |
|
Thread: facebook friends crawler
Post: RE: facebook friends crawler
| hey, thanks a lot.
i played a bit with webdriver and chromedriver and all i can say is: 'facebook api what?'. :)))
i setup a fb acc. and made my way to a user's home page but i keep getting stuck at ... |
|
edithegodfather |
Web Scraping & Web Development |
12 |
21,040 |
Aug-21-2017, 02:01 AM |
| |
|
Thread: facebook friends crawler
Post: facebook friends crawler
| hey guys,
so recently i started getting into python again and i was thinking about taking on a bigger challenge: a facebook friends list crawler.
i've done a crawler before using 'requests' and 'beau... |
|
edithegodfather |
Web Scraping & Web Development |
12 |
21,040 |
Aug-20-2017, 08:50 PM |
| |
|
Thread: How to store python output into a file?
Post: RE: How to store python output into a file?
| @Larz60+: that, is scary.
@wavic: what i'm trying to do is print the program output; it already goes in there and grabs all that info and prints it out in the console; i'm trying to store the output i... |
|
edithegodfather |
Web Scraping & Web Development |
5 |
7,737 |
Jan-19-2017, 09:40 PM |
| |
|
Thread: How to store python output into a file?
Post: How to store python output into a file?
| hey guys,
i've recently finished a web crawler app for an ads website that prints the ad title, ad link and the number of views each ad has.
the crawler uses chromedriver which is an app that execute... |
|
edithegodfather |
Web Scraping & Web Development |
5 |
7,737 |
Jan-18-2017, 06:14 PM |
| |
|
Thread: web crawler that retrieves data not stored in source code
Post: RE: web crawler that retrieves data not stored in ...
| hey guys, i just finished the crawler project for the romanian ads website, here's the source code for anyone who wants to play with it; you guys helped me a great deal with it so i feel it belongs to... |
|
edithegodfather |
Web Scraping & Web Development |
14 |
11,324 |
Jan-14-2017, 01:01 AM |
| |
|
Thread: web crawler that retrieves data not stored in source code
Post: RE: web crawler that retrieves data not stored in ...
| hey, thanks for the reply, sorry i couldn't get back to you sooner, i started work again and i don't have that much time on my hands anymore.
also sorry about the code tag thing, i kept noticing it wa... |
|
edithegodfather |
Web Scraping & Web Development |
14 |
11,324 |
Jan-11-2017, 01:28 PM |
| |
|
Thread: web crawler that retrieves data not stored in source code
Post: RE: web crawler that retrieves data not stored in ...
| hey guys, i was wondering if you can help me out with something on this crawler.
i'm almost done with it but i'm trying to figure out a way of filtering out some of the results it prints out when it ... |
|
edithegodfather |
Web Scraping & Web Development |
14 |
11,324 |
Jan-10-2017, 05:58 AM |
| |
|
Thread: web crawler that retrieves data not stored in source code
Post: RE: web crawler that retrieves data not stored in ...
| yeah, i checked the page source and it works just fine; i don't think there's gonna be much variation in there, unless they change the whole layout of the website but in that case it's not just gonna ... |
|
edithegodfather |
Web Scraping & Web Development |
14 |
11,324 |
Jan-07-2017, 08:11 PM |
| |
|
Thread: web crawler that retrieves data not stored in source code
Post: RE: web crawler that retrieves data not stored in ...
| ok, so i managed make the crawler pick the links and the titles now all i'm missing is the view count.
now for the view count there are 2 types of ads:
the ones that have been reposted automatically:... |
|
edithegodfather |
Web Scraping & Web Development |
14 |
11,324 |
Jan-07-2017, 12:44 AM |
| |
|
Thread: web crawler that retrieves data not stored in source code
Post: RE: web crawler that retrieves data not stored in ...
| no problem, i'll keep an eye out for it in case i miss out any values.
i'm gonna take that code now and put it together with the rest of it so it picks up the view nr from every link and prints it out... |
|
edithegodfather |
Web Scraping & Web Development |
14 |
11,324 |
Jan-06-2017, 03:07 PM |
| |
|
Thread: web crawler that retrieves data not stored in source code
Post: RE: web crawler that retrieves data not stored in ...
| hey, thanks a lot, that actually worked :D
i did make some changes though to adapt the code to my system:
from bs4 import BeautifulSoup
from selenium import webdriver
import time
import os
url = 'htt... |
|
edithegodfather |
Web Scraping & Web Development |
14 |
11,324 |
Jan-05-2017, 08:40 PM |
| |
|
Thread: web crawler that retrieves data not stored in source code
Post: RE: web crawler that retrieves data not stored in ...
| sure, the website is publi24.ro
it doesn't really have an english version but what you're looking for is:
publi24.ro/anunturi/locuri-de-munca/bucuresti/
or with page increment
publi24.ro/anunturi/loc... |
|
edithegodfather |
Web Scraping & Web Development |
14 |
11,324 |
Jan-05-2017, 01:24 AM |
| |
|
Thread: web crawler that retrieves data not stored in source code
Post: web crawler that retrieves data not stored in sour...
| hey guys,
so i'm working right now on a small program that can help me gather ads from an ad website in my country (i'm gonna post some screenshots here; i'm not sure if i can post the website link b... |
|
edithegodfather |
Web Scraping & Web Development |
14 |
11,324 |
Jan-05-2017, 12:09 AM |
