Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 [split] [Help] Keep getting a 'TypeError' from Django and BeautifulSoup
#1
Hi. I am also new to Django. I want to do the same as you scrape using Django as a web application. the code right now working fine using python3 Jupyter notebook.

import requests
import json
import time
from bs4 import BeautifulSoup as soup

def getLi(text):
    for i in x: 
        if i[0] == "i" and i[1:].isdigit():
            return (i.replace("i",""))
            
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'}

count = 0
link = input("Please enter url to scrape ")
x = link.split("-")
d = soup(requests.get(link).text, 'html.parser')
results = list(map(int, filter(None, [i.text for i in d.find_all('button', {'class':'next-pagination-item'})])))
print (results)
for i in range(min(results), max(results)+1):
    count+=1
    time.sleep(10)  #0.5 * random.random()
    url = ('https://my.lazada.com.my/pdp/review/getReviewList?'
        'itemId='+str(getLi(x))+'&pageSize=5&filter=0&sort=0&pageNo='+str(count))
    print(url)
    req = requests.get(url,headers=headers)
    data = req.json()
    print(count)
    for i in data['model']['items']:
    
        reviewContent = i['reviewContent']
        
        print(reviewContent)
        print('------------------------------------------')
where should I put this code? models.py right? what should I put at views.py and templates folder that contain HTML. Huh Huh
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  BeautifulSoup: Error while extracting a value from an HTML table kawasso 3 100 Yesterday, 01:13 AM
Last Post: kawasso
  get function returns None from Beautifulsoup object DeanAseraf1 3 152 Aug-03-2019, 04:00 PM
Last Post: DeanAseraf1
  TypeError: missing 1 required positional argument (word counter Django app) Drone4four 2 212 Jul-11-2019, 09:34 PM
Last Post: Drone4four
  Django: How to automatically substitute a variable in the admin page at Django 1.11? m0ntecr1st0 3 227 Jun-30-2019, 12:21 AM
Last Post: scidam
  Beautifulsoup Scraping PolskaYBZ 3 361 Jun-22-2019, 10:05 AM
Last Post: PolskaYBZ
  BeautifulSoup Installed but not Found in Atom wakegate 4 280 Jun-14-2019, 05:54 PM
Last Post: snippsat
  beautifulsoup error rudolphyaber 7 671 May-26-2019, 02:12 PM
Last Post: heiner55
  Looping with Beautifulsoup CaptainCsaba 8 591 Jan-23-2019, 12:38 PM
Last Post: buran
  Mechanize and BeautifulSoup read not correct hours vaeVictis 5 661 Jan-15-2019, 01:27 PM
Last Post: metulburr
  TypeError: __init__() IMuriel 7 883 Jan-09-2019, 09:48 PM
Last Post: nilamo

Forum Jump:


Users browsing this thread: 1 Guest(s)