Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
FastAPI returns null
#1
Hey guys new to python and to this forum. Right now I'm working on a project where I'm using selenium to grab some data and then have FastAPI serve this data back. However when i make a request to my endpoint: "reviews" I get returned "null". I checked the info on the web scraper side where it holds the list of data before returning it and it has data. I will place the code below. I removed some code to make it more readable. and i commented where i'm having the problem. Variables i'm having trouble with is called "info" and is at the bottom of the Scrapper class. Thanks so much for any help.

Scraper Class Below:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.wait import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC
import time
import json


class Scraper:
    def diff_review_page(self):
        data_container = WebDriverWait(self.driver, 10).until(EC.visibility_of_element_located((By.XPATH, '//div[@class="kKWzSd"]')))
        num_of_scrolls = 0
            
        while True:
            current_reviews = len(self.driver.find_elements(By.CSS_SELECTOR, ".bwb7ce"))
            self.driver.execute_script('arguments[0].scrollTop = arguments[0].scrollHeight', data_container)
            time.sleep(4) # wait for the reviews to load
            
            new_num_loaded_reviews = len(self.driver.find_elements(By.CSS_SELECTOR, ".bwb7ce"))
            if current_reviews == new_num_loaded_reviews: break
            
        total_reviews = self.driver.find_elements(By.CSS_SELECTOR, ".bwb7ce")
        
        time.sleep(2)
        info = []
        review_txt = open('reviewfile.txt', 'w')
        for review in total_reviews:
            name = review.find_element(By.CLASS_NAME, 'Vpc5Fe').text
            stars = review.find_element(By.CLASS_NAME, 'dHX2k')
            star_rating = stars.get_attribute('aria-label')  # Retrieve the aria-label attribute for the stars
            
            #Loop was breaking since some reviews didnt have comments so i placed this to keep the loop going.
            try:
                comment = review.find_element(By.CLASS_NAME, "OA1nbd").text
            except:
                comment = ''
            date_of_review = review.find_element(By.CLASS_NAME, "y3Ibjb").text

            info.append({
                'name': name,
                'rating': star_rating,
                'comment': comment,
                'date_of_review': date_of_review
            })
        self.driver.quit()
       #THIS IS WHERE IM HAVING THE PROBLEM BELOW
       ## if i print info i get all my reviews. But when i return it like below it comes back as null
        return {"reviews": info}
API Script below:

@app.get("/review/{review_url:path}")
async def reviewUrl(review_url: str):
    scrape = Scraper()

    reviews_json = scrape.start(review_url)
    return reviews_json #Returns Null
Reply
#2
Your snippet cannot be run as posted.
Where is the web page being extracted?
where's app?
Reply
#3
Nevermind i just figured it out. Thanks for the help though!
Reply


Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020