Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Need logic on how to scrap 100K URLs
#1
Hi,
I request you to explain to me the logic of how to proceed with the requirement.

My requirement is-

I have a website say http://www.example.com and once I log in, I have to search for a product. Then the website returns me things like -
  1. The demand for the product
  2. The supply for the product
  3. Medium sales for the product
  4. Maximum sales for the product

In another line, it gives me the' total number of this product'.

In another line, it gives me the most important information which is -
'Other related products'

These related products are like 'product name 123', 'product name 236, 'product name 483', etc. Once you click on all these related product, it will have a similar page with the same type of information like -
  1. The demand for the product
  2. The supply for the product
  3. Medium sales for the product
  4. Maximum sales for the product
In another line, it gives me the' total number of this product'.

In another line, it gives me the most important information which is -
'Other related products' etc and then some process has to be followed with each product.

What can be a python script logic which reads one URL and get all the information of that URL like -
demand, supply, medium sales, maximum sales, the total number of the product.

Then, it should click on all the related products one by one and extract all this information. so, it will open a chain of URLs as each product will have some related products and each related product has its own related project.

In this way, one URL will simultaneously open 100K URLs in the browser. So, to summarize, how I can proceed with the logic to extract information from around 100K URLs. The information which I want is -

  1. demand
  2. supply
  3. medium sales
  4. maximum sales
  5. total number of products on sales
Quote
#2
you can start here, doesn't take long, and you'll learn a lot about the basics

web scraping part 1
web scraping part 2
Quote
#3
(Jun-29-2020, 08:28 AM)Larz60+ Wrote: you can start here, doesn't take long, and you'll learn a lot about the basics

web scraping part 1
web scraping part 2

Thanks for the Reply. I am going through the posts which you shared but my requirement is completely different.

The posts do not cover any similar logic. I guess I have to learn advanced stuff and then proceed.

What do you suggest?
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Web scrap multiple pages anilacem_302 3 201 Jul-01-2020, 07:50 PM
Last Post: mlieqo
  Scrap a dynamic span hefaz 0 725 Mar-07-2020, 02:56 PM
Last Post: hefaz
  scrap by defining 3 functions zarize 0 298 Feb-18-2020, 03:55 PM
Last Post: zarize
  Skipping anti-scrap zarize 0 351 Jan-17-2020, 11:51 AM
Last Post: zarize
  page impossible to scrap? :O zarize 2 993 Oct-03-2019, 02:44 PM
Last Post: zarize
  Scrap a value from website harsush 1 410 Aug-29-2019, 01:57 PM
Last Post: snippsat
  Scrap text out of td table from URLS Gochix2020 4 1,401 Aug-03-2019, 02:56 AM
Last Post: Larz60+
  scrap macrotrends mr_byte31 7 2,336 Aug-02-2019, 12:02 AM
Last Post: mr_byte31
  Scrap arbitrage odds -help Gochix2020 3 694 Jul-31-2019, 10:45 AM
Last Post: Gochix2020
  i am trying to web scrap this .asp nufan0000 1 545 May-30-2019, 02:27 AM
Last Post: Larz60+

Forum Jump:


Users browsing this thread: 1 Guest(s)