Is it possible to add a delay right after a request.get()

[alan_a]

Building a scraper can be challenging, especially when dealing with dynamic websites that have loading delays or use AJAX to load content. Given the "processing" activity you described, it sounds like the website might be using some form of lazy loading or client-side rendering. When considering mobile app development in ... it's essential to be aware of such intricacies, as they can affect the user experience and the efficiency of data retrieval.

To handle this in your scraper, you might need to use a tool like Selenium or Puppeteer, which allows for browser automation. These tools can mimic real user interactions, like waiting for a page to load fully before scraping the content.

Here's a basic approach using Selenium:

`python
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Initialize the browser
driver = webdriver.Chrome()

# Navigate to the website
driver.get('URL_OF_THE_WEBSITE')

# Wait for the "processing" activity to complete
wait = WebDriverWait(driver, 10) # wait for up to 10 seconds
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, 'CSS_SELECTOR_OF_AN_ELEMENT_YOU_WANT_TO_WAIT_FOR')))

# Now, you can scrape the content
content = driver.page_source

# Don't forget to close the browser once done
driver.quit()
`

Remember to replace 'URL_OF_THE_WEBSITE' with the actual URL and 'CSS_SELECTOR_OF_AN_ELEMENT_YOU_WANT_TO_WAIT_FOR' with a CSS selector of an element you know will be present after the "processing" activity.

Also, when building or using scrapers, always ensure you're respecting the website's robots.txt file and terms of service. Some sites might have restrictions against scraping, and you wouldn't want to inadvertently violate any terms.

Larz60+ write Aug-30-2023, 03:27 AM:
spam content removed
Also:
Please post all code, output and errors (it it's entirety) between their respective tags. Refer to BBCode help topic on how to post. Use the "Preview Post" button to make sure the code is presented as you expect before hitting the "Post Reply/Thread" button.