Python Forum

Hello All,

I am looking for recommendation for a good start in building my first Python scraper script.
Actually I am not even sure if I should use scrapy, beautiful soup or else.
I have watched a few dozen of tutorial of scrapy, beautiful soup and others, but none were showing what I am really looking for.

I am a big fan of travel, all inclusives deal and many countries.

I am looking at many suppliers but always need to go back every day, sometime every hour to refresh the search.

The idea is to create a python script to automate this search and push the data in a spreadsheet.

The main issue I am facing is the multiple options on each suppliers, drop down for example.
From: , TO: , destination: , Date:, duration (Length) , Hotels: , Star rating... Then I press search...

I was able to create a script for simple search where we have only 1 option with 1 URL which covers the search and push it to .csv file, but this is different with the drop down options and multiple choices/ criteria in the selection.

https://voyagesarabais.com is the place where I look normally, but there are other url where we have to select multiple options prior to get the price and list of location/resorts.

Any recommendations?
Module, package to use for this mini-project.

Thanks,
Sylvain

you need a web crawler and scraper.

The web crawler looks at all or a filtered list of sites to determine if they are suitable for scraping
based on the subject 'Travel' (an indexer)
They can both be in the same module, but it makes sense to separate into two separate modules.

Scrapy is a common: https://scrapy.org/ web crawler/scraper - I haven't used it as I prefer to write my own, but it is popular.

Here's a site to help with scrapy: https://www.datacamp.com/community/tutor...apy-python

You can write your own as well
find packages here: https://pypi.org/search/?q=%22web+crawler%22&o=
for blogs, google 'web crawler python 3'

web scraper - Suggest tutorials on this forum
web scraping part 1
web scraping part 2

Thanks for the quick reply Larz60+ !

I will definitively look into it.

Thanks for your directions.
Regards,
Sylvain