Jul-02-2024, 04:43 AM
Hey guys new to python and to this forum. Right now I'm working on a project where I'm using selenium to grab some data and then have FastAPI serve this data back. However when i make a request to my endpoint: "reviews" I get returned "null". I checked the info on the web scraper side where it holds the list of data before returning it and it has data. I will place the code below. I removed some code to make it more readable. and i commented where i'm having the problem. Variables i'm having trouble with is called "info" and is at the bottom of the Scrapper class. Thanks so much for any help.
Scraper Class Below:
Scraper Class Below:
from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.common.keys import Keys from selenium.webdriver.support.wait import WebDriverWait from selenium.webdriver.support import expected_conditions as EC import time import json class Scraper: def diff_review_page(self): data_container = WebDriverWait(self.driver, 10).until(EC.visibility_of_element_located((By.XPATH, '//div[@class="kKWzSd"]'))) num_of_scrolls = 0 while True: current_reviews = len(self.driver.find_elements(By.CSS_SELECTOR, ".bwb7ce")) self.driver.execute_script('arguments[0].scrollTop = arguments[0].scrollHeight', data_container) time.sleep(4) # wait for the reviews to load new_num_loaded_reviews = len(self.driver.find_elements(By.CSS_SELECTOR, ".bwb7ce")) if current_reviews == new_num_loaded_reviews: break total_reviews = self.driver.find_elements(By.CSS_SELECTOR, ".bwb7ce") time.sleep(2) info = [] review_txt = open('reviewfile.txt', 'w') for review in total_reviews: name = review.find_element(By.CLASS_NAME, 'Vpc5Fe').text stars = review.find_element(By.CLASS_NAME, 'dHX2k') star_rating = stars.get_attribute('aria-label') # Retrieve the aria-label attribute for the stars #Loop was breaking since some reviews didnt have comments so i placed this to keep the loop going. try: comment = review.find_element(By.CLASS_NAME, "OA1nbd").text except: comment = '' date_of_review = review.find_element(By.CLASS_NAME, "y3Ibjb").text info.append({ 'name': name, 'rating': star_rating, 'comment': comment, 'date_of_review': date_of_review }) self.driver.quit() #THIS IS WHERE IM HAVING THE PROBLEM BELOW ## if i print info i get all my reviews. But when i return it like below it comes back as null return {"reviews": info}API Script below:
@app.get("/review/{review_url:path}") async def reviewUrl(review_url: str): scrape = Scraper() reviews_json = scrape.start(review_url) return reviews_json #Returns Null