UPDATE!
After few tests looks like the problem is not in grequests.map() function.
Looks like the thing is somewhere else.
So. When I hard code lot of links in array grequests.map() finishes it in 10 sec.
What is slowing it down is finding a tags in website and appending them to array. That process takes long time.
I do it like this
I am using webdriver because data on website is dynamic. Created with js.
After few tests looks like the problem is not in grequests.map() function.
Looks like the thing is somewhere else.
So. When I hard code lot of links in array grequests.map() finishes it in 10 sec.
What is slowing it down is finding a tags in website and appending them to array. That process takes long time.
I do it like this
from selenium import webdriver from selenium.webdriver.chrome.options import Options from bs4 import BeautifulSoup urls = [] options = Options() options.headless = True driver = webdriver.Chrome(options=options) driver.get('https://www.website.com') results = driver.find_elements_by_tag_name('a') for result in results: req = result.get_attribute("href") urls.append(req) print(req)Is there efficient/faster way to do this?
I am using webdriver because data on website is dynamic. Created with js.