Python Forum
Why is grequests.map() so slow?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Why is grequests.map() so slow?
#1
So. I am getting links from a website. Storing them in array and then iterating over them.

rs = (grequests.get(d) for d in data)
responses = grequests.map(rs)
But the grequests.map() is sooooo slow. Any idea what can cause this? Or I am not using it right. Or maybe there is better library or a way to send async request to links to check if they work.

I need to make this program fast because there is approx 70 links per page.

And also is there a way to get response code with grequests?
Reply
#2
UPDATE!

After few tests looks like the problem is not in grequests.map() function.

Looks like the thing is somewhere else.

So. When I hard code lot of links in array grequests.map() finishes it in 10 sec.

What is slowing it down is finding a tags in website and appending them to array. That process takes long time.

I do it like this

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from bs4 import BeautifulSoup

urls = []

options = Options()
options.headless = True
driver = webdriver.Chrome(options=options)

driver.get('https://www.website.com')

results = driver.find_elements_by_tag_name('a')

for result in results:
    req = result.get_attribute("href")
    urls.append(req)
    print(req)
Is there efficient/faster way to do this?

I am using webdriver because data on website is dynamic. Created with js.
Reply


Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020