Jan-18-2017, 06:14 PM
hey guys,
i've recently finished a web crawler app for an ads website that prints the ad title, ad link and the number of views each ad has.
the crawler uses chromedriver which is an app that executes javascript on a website to extract information off of it as opposed to getting it out of the source code with beautifulsoup, data that's not stored in the source code otherwise.
after the app prints all that it goes to the next ad an so on.
my question is: i'm trying to store all the output into a file that i can access and have a bird's eye view of and i'm trying to figure out the best way to go about it.
i thought initially of excel because i've been getting ads off of the website manually anyway and stored them into an excel for a while now but i'm having trouble finding the best module to work with.
i looked into openpyxl but i can't seem to find any tutorials on it to help me learn how to use it.
then i looked into pandas because it comes up a lot but all the videos are aimed at data processing which doesn't really apply to my case.
i also tried xlwt but it's kinda old now and it doesn't handle .xlsx files.
and i'm kinda running out of ideas and i though i'd ask you guys about it, maybe you know a good excel module out there that also has tutorials or if there's any better way of storing data other than excel.
thanks
i've recently finished a web crawler app for an ads website that prints the ad title, ad link and the number of views each ad has.
the crawler uses chromedriver which is an app that executes javascript on a website to extract information off of it as opposed to getting it out of the source code with beautifulsoup, data that's not stored in the source code otherwise.
after the app prints all that it goes to the next ad an so on.
my question is: i'm trying to store all the output into a file that i can access and have a bird's eye view of and i'm trying to figure out the best way to go about it.
i thought initially of excel because i've been getting ads off of the website manually anyway and stored them into an excel for a while now but i'm having trouble finding the best module to work with.
i looked into openpyxl but i can't seem to find any tutorials on it to help me learn how to use it.
then i looked into pandas because it comes up a lot but all the videos are aimed at data processing which doesn't really apply to my case.
i also tried xlwt but it's kinda old now and it doesn't handle .xlsx files.
and i'm kinda running out of ideas and i though i'd ask you guys about it, maybe you know a good excel module out there that also has tutorials or if there's any better way of storing data other than excel.
thanks
