+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html)
+--- Thread: Crawling tweets with scrapy (/thread-1561.html)
Crawling tweets with scrapy - R3turnz - Jan-12-2017
Hello,
I want to crawl all tweets of a specific user because the twitter api does not allow this behavior.
I thinks twitter loads the tweets if you scroll down with a javascript. How can I locate it and than execute it with scrapy?
RE: Crawling tweets with scrapy - micseydel - Jan-16-2017
(1) You'll probably be in violation of their ToS by attempting this.
(2) scrapy, from my understanding, doesn't do Javascript at all. You can make it do what the Javascript does by either **shivers** reading the Javascript or by using your the developer tools in your browser to see what it does and build something out yourself.
Once you give the code a try, feel free to post back with questions (and don't forget to include the code).