Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Crawling tweets with scrapy
#1
Hello,
I want to crawl all tweets of a specific user because the twitter api does not allow this behavior.
I thinks twitter loads the tweets if you scroll down with a javascript. How can I locate it and than execute it with scrapy?
Reply
#2
(1) You'll probably be in violation of their ToS by attempting this.
(2) scrapy, from my understanding, doesn't do Javascript at all. You can make it do what the Javascript does by either **shivers** reading the Javascript or by using your the developer tools in your browser to see what it does and build something out yourself.

Once you give the code a try, feel free to post back with questions (and don't forget to include the code).
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Unicode letters in crawling page DMDoniz 5 2,651 Oct-31-2020, 07:03 AM
Last Post: buran
  Web Crawler's Crawling Ability samlee916 3 2,695 Aug-10-2020, 12:50 PM
Last Post: abusalim
  Web Scraping and crawling venkataramakrishna 1 1,814 Jan-25-2020, 06:07 PM
Last Post: Larz60+
  Trying to automate tweets without Twitter API HowardHarkness 10 6,308 Jan-07-2019, 01:34 AM
Last Post: HowardHarkness
  Good book on Web scraping and crawling Surya 2 5,091 May-08-2017, 02:01 AM
Last Post: Larz60+
  Scrapy-cut: Advanced Cookiecutter Scrapy Templating scriptso 2 4,608 Feb-02-2017, 07:57 PM
Last Post: scriptso

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020