Nov-15-2018, 07:38 AM
Hi. I am also new to Django. I want to do the same as you scrape using Django as a web application. the code right now working fine using python3 Jupyter notebook.
import requests import json import time from bs4 import BeautifulSoup as soup def getLi(text): for i in x: if i[0] == "i" and i[1:].isdigit(): return (i.replace("i","")) headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'} count = 0 link = input("Please enter url to scrape ") x = link.split("-") d = soup(requests.get(link).text, 'html.parser') results = list(map(int, filter(None, [i.text for i in d.find_all('button', {'class':'next-pagination-item'})]))) print (results) for i in range(min(results), max(results)+1): count+=1 time.sleep(10) #0.5 * random.random() url = ('https://my.lazada.com.my/pdp/review/getReviewList?' 'itemId='+str(getLi(x))+'&pageSize=5&filter=0&sort=0&pageNo='+str(count)) print(url) req = requests.get(url,headers=headers) data = req.json() print(count) for i in data['model']['items']: reviewContent = i['reviewContent'] print(reviewContent) print('------------------------------------------')where should I put this code? models.py right? what should I put at views.py and templates folder that contain HTML.

