Hello guest, if you read this it means you are not registered. Click here to register in a few simple steps, you will enjoy all features of our Forum.
Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
[Help] Keep getting a 'TypeError' from Django and BeautifulSoup
#1
So, I'm using BeautifulSoup to try and read urls from a website. The code I'm using so far is something along the lines of:

File: appName \ models.py

~~~
from django.db import models
from bs4 import BeautifulSoup
from urllib import request

class BBCHeaders(models.Model):

    url = "http://www.bbc.co.uk/news"                          # URL for test reasons
    content = request.urlopen(url).read()                      # Open url
    soup = BeautifulSoup(content, "html.parser")	           # Grab the page details
    
    try:                                                       # I am unsure about 'try' function, using it to 'try' a value (I may be using it wrong?)
        for element in soup.body.find_all('nav'):              # Search for all elements that are of the category 'nav' for navigation tree
            for link in element.find_all('a', text = True):    # Find all the links within the navigation tree
                for strLink in link.find_all('span'):          # Check for heading strings in navigation tree
                    if strLink != None:                        # Filter out all results where span is 'None'
                        # print(link.span.string)              # Print the heading of the link
                        # print(link.get('href'))              # Print the URL padding for link
                        nBBC = models.CharField(link.string)   # Save the headers into SQL
                        urlBBC = models.CharField(link.get('href'))    # Save links into SQL

    except(TypeError):                                         # Exception of TypeError
        pass                                                   # Do Nothing
~~~
The code keeps returning a type error. Typing it into the python shell and using the print instead of the nBBC saving it posts teh results which are the headings followed by the link padding to guide you to the destination. The script works in a Python 3.6 shell however Django doesn't like it and I suspect I have my *try* function in the wrong location or I might have to refine my search criteria?

Has anyone with better experience encountered a *NoneType* error? How did you get around it?
Quote
#2
Quote:The code keeps returning a type error
Show the full error traceback (the complete unmodified error message).
Quote
#3
Thank you for your help! I am still really new when it comes to programming in Python.

As I gather it just doesn't like the *NoneType*

The complete error

Error:
Traceback (most recent call last): File "manage.py", line 15, in (module) execute_from_command_line(sys.argv) File "C:\...\python36-32\...\django\core\management\__init__.py", line 371, in execute_from_command_line utility.execute() File "C:\...\python36-32\...\django\core\management\__init__.py", line 347, in execute django.setup() File "C:\...\python36-32\...\django\__init__.py", line 24, in setup apps.populate(settings.INSTALLED_APPS) File "C:\...\python36-32\...\django\apps\registry.py", line 112, in populate app_config.import_models() File "C:\...\python36-32\...\django\apps\config.py", line 198, in import_models self.models_module = import_module(models_module_name) File "C:\...\python36-32\...\importlib\__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package level) File "<frozen importlib._bootstrap>", line 994, in _gcd_import File "<frozen importlib._bootstrap>", line 971, in _find_and_load File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 665, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 678, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with__frames_removed File "C:\siteName\siteName\appName\models.py", line 28, in <module> class BBCHeaders(models.Model): File "C:\...\python36-32\...\django\db\models\base.py", line 152, in __new__ new_class.add_to_class(obj_name, obj) File "C:\...\python36-32\...\django\db\models\base.py", line 315, in add_to_class value.contribute_to_class(cls, name) TypeError: 'NoneType' object is not callable
Quote
#4
The error is on the class definition (line 5 above) it is stating that models.Model has no value.
Quote
#5
Hmm,

So here I should give it a value in the exception case?
Quote
#6
No, it's saying that the call to instantiate BBCHeaders class
is like:
myclass = BBCHeaders(None)
and you need to pass a valid model
Quote
#7
Okay, so I've written a script that grabs the BBC navigation headers and saves them into a CSV. I was thinking from that I can write them into a database but maybe I have to do that from an outside process. ie, write a new file outside of models.py and reference it...

That is to say write write my model elements so

models.py
class BBCHEaders(models.Model):
   nBBC = models.CharField(max_length=200)
And then have a new module something like:

storedNav.py
from .models import BBCHeaders.nBBC

def writeBBC
    with open('C:\\...app\\Headings\\BBCHeadings.csv') as f:
        freader = csv.DictReader(f, delimiter=',')
            for r in freader:
                nBBC.create(r['Heading'])
I am wondering if this would write to the nBBC model database and then I can refer to it with the views.py.

views.py
from .storedNav import writeBBC
from .models import BBCHeaders

class BBCView(generic.ListView):
    writeBBC()
    model = BBCHeaders
    template_name = 'news/BBCNews.html'

Quote
#8
what you need to do is make sure you are passing the correct values to BBCHEaders and everything else should fall into place.
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  How to create dynamic webscraper in Django using BeautifulSoup Prince_Bhatia 1 340 Jan-26-2018, 02:07 PM
Last Post: frostbite

Forum Jump:


Users browsing this thread: 1 Guest(s)