Why is grequests.map() so slow?

[M1ck0]

So. I am getting links from a website. Storing them in array and then iterating over them.

1 2	rs = (grequests.get(d) for d in data) responses = grequests.map(rs)

But the grequests.map() is sooooo slow. Any idea what can cause this? Or I am not using it right. Or maybe there is better library or a way to send async request to links to check if they work.

I need to make this program fast because there is approx 70 links per page.

And also is there a way to get response code with grequests?

[M1ck0]

UPDATE!

After few tests looks like the problem is not in grequests.map() function.

Looks like the thing is somewhere else.

So. When I hard code lot of links in array grequests.map() finishes it in 10 sec.

What is slowing it down is finding a tags in website and appending them to array. That process takes long time.

I do it like this

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

from selenium import webdriver from selenium.webdriver.chrome.options import Options from bs4 import BeautifulSoup   urls = []   options = Options() options.headless = True driver = webdriver.Chrome(options=options)   driver.get('https://www.website.com')   results = driver.find_elements_by_tag_name('a')   for result in results:     req = result.get_attribute("href")     urls.append(req)     print(req)

Is there efficient/faster way to do this?

I am using webdriver because data on website is dynamic. Created with js.