(May-21-2019, 11:09 PM)Truman Wrote: I installed scrapy too but get this errorYou should follow tutorial,to make sure that work first.
(May-21-2019, 11:09 PM)Truman Wrote: Also wondering should I use scrapy at all. This stuff can be done with requests and bs4 in less lines.The user case can be very different,Scrapy is the fully fledged Spider library.
Which mean that it eg can follow a
start_url
and extract data from eg 10 to millions of pages.start_urls = ['http://alibaba.com/']alibaba has a lot of product pages,can eg use Srapy to find product and price of product of choice.
It will start and go trough 1000's of pages,and can save data get back in json,csv...ect
Writing the same code with Requests and BS would be a lot of work.