[Help] Keep getting a 'TypeError' from Django and BeautifulSoup - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html) +--- Thread: [Help] Keep getting a 'TypeError' from Django and BeautifulSoup (/thread-10786.html) |
[Help] Keep getting a 'TypeError' from Django and BeautifulSoup - Plant_Boy - Jun-06-2018 So, I'm using BeautifulSoup to try and read urls from a website. The code I'm using so far is something along the lines of: File: appName \ models.py ~~~ from django.db import models from bs4 import BeautifulSoup from urllib import request class BBCHeaders(models.Model): url = "http://www.bbc.co.uk/news" # URL for test reasons content = request.urlopen(url).read() # Open url soup = BeautifulSoup(content, "html.parser") # Grab the page details try: # I am unsure about 'try' function, using it to 'try' a value (I may be using it wrong?) for element in soup.body.find_all('nav'): # Search for all elements that are of the category 'nav' for navigation tree for link in element.find_all('a', text = True): # Find all the links within the navigation tree for strLink in link.find_all('span'): # Check for heading strings in navigation tree if strLink != None: # Filter out all results where span is 'None' # print(link.span.string) # Print the heading of the link # print(link.get('href')) # Print the URL padding for link nBBC = models.CharField(link.string) # Save the headers into SQL urlBBC = models.CharField(link.get('href')) # Save links into SQL except(TypeError): # Exception of TypeError pass # Do Nothing ~~~The code keeps returning a type error. Typing it into the python shell and using the print instead of the nBBC saving it posts teh results which are the headings followed by the link padding to guide you to the destination. The script works in a Python 3.6 shell however Django doesn't like it and I suspect I have my *try* function in the wrong location or I might have to refine my search criteria? Has anyone with better experience encountered a *NoneType* error? How did you get around it? RE: [Help] Keep getting a 'TypeError' from Django and BeautifulSoup - Larz60+ - Jun-06-2018 Quote:The code keeps returning a type errorShow the full error traceback (the complete unmodified error message). RE: [Help] Keep getting a 'TypeError' from Django and BeautifulSoup - Plant_Boy - Jun-06-2018 Thank you for your help! I am still really new when it comes to programming in Python. As I gather it just doesn't like the *NoneType* The complete error
RE: [Help] Keep getting a 'TypeError' from Django and BeautifulSoup - Larz60+ - Jun-06-2018 The error is on the class definition (line 5 above) it is stating that models.Model has no value. RE: [Help] Keep getting a 'TypeError' from Django and BeautifulSoup - Plant_Boy - Jun-06-2018 Hmm, So here I should give it a value in the exception case? RE: [Help] Keep getting a 'TypeError' from Django and BeautifulSoup - Larz60+ - Jun-06-2018 No, it's saying that the call to instantiate BBCHeaders class is like: myclass = BBCHeaders(None)and you need to pass a valid model RE: [Help] Keep getting a 'TypeError' from Django and BeautifulSoup - Plant_Boy - Jun-13-2018 Okay, so I've written a script that grabs the BBC navigation headers and saves them into a CSV. I was thinking from that I can write them into a database but maybe I have to do that from an outside process. ie, write a new file outside of models.py and reference it... That is to say write write my model elements so models.py class BBCHEaders(models.Model): nBBC = models.CharField(max_length=200)And then have a new module something like: storedNav.py from .models import BBCHeaders.nBBC def writeBBC with open('C:\\...app\\Headings\\BBCHeadings.csv') as f: freader = csv.DictReader(f, delimiter=',') for r in freader: nBBC.create(r['Heading'])I am wondering if this would write to the nBBC model database and then I can refer to it with the views.py. views.py from .storedNav import writeBBC from .models import BBCHeaders class BBCView(generic.ListView): writeBBC() model = BBCHeaders template_name = 'news/BBCNews.html' RE: [Help] Keep getting a 'TypeError' from Django and BeautifulSoup - Larz60+ - Jun-13-2018 what you need to do is make sure you are passing the correct values to BBCHEaders and everything else should fall into place. |