Python Forum
Why is grequests.map() so slow?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Why is grequests.map() so slow?
#2
UPDATE!

After few tests looks like the problem is not in grequests.map() function.

Looks like the thing is somewhere else.

So. When I hard code lot of links in array grequests.map() finishes it in 10 sec.

What is slowing it down is finding a tags in website and appending them to array. That process takes long time.

I do it like this

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from bs4 import BeautifulSoup

urls = []

options = Options()
options.headless = True
driver = webdriver.Chrome(options=options)

driver.get('https://www.website.com')

results = driver.find_elements_by_tag_name('a')

for result in results:
    req = result.get_attribute("href")
    urls.append(req)
    print(req)
Is there efficient/faster way to do this?

I am using webdriver because data on website is dynamic. Created with js.
Reply


Messages In This Thread
Why is grequests.map() so slow? - by M1ck0 - Jul-21-2019, 08:08 AM
RE: Why is grequests.map() so slow? - by M1ck0 - Jul-21-2019, 09:38 AM

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020