Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 [split] [Help] Keep getting a 'TypeError' from Django and BeautifulSoup
#1
Hi. I am also new to Django. I want to do the same as you scrape using Django as a web application. the code right now working fine using python3 Jupyter notebook.

import requests
import json
import time
from bs4 import BeautifulSoup as soup

def getLi(text):
    for i in x: 
        if i[0] == "i" and i[1:].isdigit():
            return (i.replace("i",""))
            
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'}

count = 0
link = input("Please enter url to scrape ")
x = link.split("-")
d = soup(requests.get(link).text, 'html.parser')
results = list(map(int, filter(None, [i.text for i in d.find_all('button', {'class':'next-pagination-item'})])))
print (results)
for i in range(min(results), max(results)+1):
    count+=1
    time.sleep(10)  #0.5 * random.random()
    url = ('https://my.lazada.com.my/pdp/review/getReviewList?'
        'itemId='+str(getLi(x))+'&pageSize=5&filter=0&sort=0&pageNo='+str(count))
    print(url)
    req = requests.get(url,headers=headers)
    data = req.json()
    print(count)
    for i in data['model']['items']:
    
        reviewContent = i['reviewContent']
        
        print(reviewContent)
        print('------------------------------------------')
where should I put this code? models.py right? what should I put at views.py and templates folder that contain HTML. Huh Huh
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  get function returns None from Beautifulsoup object DeanAseraf1 3 126 Aug-03-2019, 04:00 PM
Last Post: DeanAseraf1
  TypeError: missing 1 required positional argument (word counter Django app) Drone4four 2 191 Jul-11-2019, 09:34 PM
Last Post: Drone4four
  Django: How to automatically substitute a variable in the admin page at Django 1.11? m0ntecr1st0 3 214 Jun-30-2019, 12:21 AM
Last Post: scidam
  Beautifulsoup Scraping PolskaYBZ 3 335 Jun-22-2019, 10:05 AM
Last Post: PolskaYBZ
  BeautifulSoup Installed but not Found in Atom wakegate 4 259 Jun-14-2019, 05:54 PM
Last Post: snippsat
  beautifulsoup error rudolphyaber 7 642 May-26-2019, 02:12 PM
Last Post: heiner55
  Looping with Beautifulsoup CaptainCsaba 8 563 Jan-23-2019, 12:38 PM
Last Post: buran
  Mechanize and BeautifulSoup read not correct hours vaeVictis 5 643 Jan-15-2019, 01:27 PM
Last Post: metulburr
  TypeError: __init__() IMuriel 7 835 Jan-09-2019, 09:48 PM
Last Post: nilamo
  BeautifulSoup n levels of nested xml elements fatwalletguy 1 672 Nov-08-2018, 12:23 AM
Last Post: Larz60+

Forum Jump:


Users browsing this thread: 1 Guest(s)