Python Forum

Full Version: Crawling tweets with scrapy
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Hello,
I want to crawl all tweets of a specific user because the twitter api does not allow this behavior.
I thinks twitter loads the tweets if you scroll down with a javascript. How can I locate it and than execute it with scrapy?
(1) You'll probably be in violation of their ToS by attempting this.
(2) scrapy, from my understanding, doesn't do Javascript at all. You can make it do what the Javascript does by either **shivers** reading the Javascript or by using your the developer tools in your browser to see what it does and build something out yourself.

Once you give the code a try, feel free to post back with questions (and don't forget to include the code).