Python Forum

So. I am getting links from a website. Storing them in array and then iterating over them.

rs = (grequests.get(d) for d in data)
responses = grequests.map(rs)

But the grequests.map() is sooooo slow. Any idea what can cause this? Or I am not using it right. Or maybe there is better library or a way to send async request to links to check if they work.

I need to make this program fast because there is approx 70 links per page.

And also is there a way to get response code with grequests?

UPDATE!

After few tests looks like the problem is not in grequests.map() function.

Looks like the thing is somewhere else.

So. When I hard code lot of links in array grequests.map() finishes it in 10 sec.

What is slowing it down is finding a tags in website and appending them to array. That process takes long time.

I do it like this

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from bs4 import BeautifulSoup

urls = []

options = Options()
options.headless = True
driver = webdriver.Chrome(options=options)

driver.get('https://www.website.com')

results = driver.find_elements_by_tag_name('a')

for result in results:
    req = result.get_attribute("href")
    urls.append(req)
    print(req)

Is there efficient/faster way to do this?

I am using webdriver because data on website is dynamic. Created with js.