Python Forum
Scraping data saving to DB error with Cursor
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Scraping data saving to DB error with Cursor
#1
So ive been working on a scrapying project and up until now everything has been working great and as expected. Now that we have successfully scrapped the website we wanted, we wanted to port it over to a database to store the data instead writing to a csv file.

My attempt has failed due to a cursor error within my scrapy pipelines.py file.

here is my code: (Below the code is the error im getting)
    def _init_(self):
        try:
            self.mariadb_connection = mariadb.connect(user='test', password='test', database='test')
            self.cursor = self.mariadb_connection.cursor()
            
        except mariadb.Error as e:
            raise e
    
#    def close_spider(self, spider):
#        ## clean up when spider is closed
#        self.client.close()
        
    def process_item(self, item, spider):
        try:
            [b]self.cursor.execute[/b]( "INSERT INTO test (style,brand,description,price,compurl,reviewcount,reviewrating,model,packsize,spechandle,specbladelength,specoveralllength,specweight) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)", 
            (item['style'],
            item['brand'],                    
            item['description'],
            item['price'],
            item['compurl'],
            item['reviewcount'],
            item['reviewrating'],
            item['model'],
            item['spechandle'],
            item['specbladelength'],
            item['specoveralllength'],
            item['specweight'],
            item['packsize'])) 
            self.conn.commit()
            
        except mariadb.Error as e:
            print("Error %d: %s" % (e.args[0], e.args[1]))
            return item
AttributeError: 'DsgDbPipeline' object has no attribute 'cursor'

I went ahead and moved the connection lines around and got it to work..
Basically move the connection and cursor down into the process_item and that worked..
Reply
#2
The reason DsgDbPipeline object has no attribute cursor is because the init did not happen because
def _init_(self):
should have double underscores
def __init__(self):
Reply
#3
so should i re-enable the init properly formatted and run it that way from now on?
Reply
#4
Yes, if you fix the init it will work as originally intended.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  How to scraping data from dinamic site sergio21124444 2 646 Nov-08-2023, 12:43 PM
Last Post: sergio21124444
  I am scraping a web page but got an Error Sarmad54 3 1,417 Mar-02-2023, 08:20 PM
Last Post: Sarmad54
  What is the error of not being able to pull data in this code? i didn't see an error? TestPerson 2 1,183 Sep-30-2022, 02:36 PM
Last Post: DeaD_EyE
  Scraping data from table into existing dataframe vincer58 1 1,960 Jan-09-2022, 05:15 PM
Last Post: vincer58
  Web scraping data Mike_Eddy 2 2,489 Jul-03-2021, 05:49 PM
Last Post: Mike_Eddy
  Scraping lender data from Ren Ren Dai website using Python. I will pay for that 200$ Hafedh_2021 1 2,724 May-18-2021, 08:41 PM
Last Post: snippsat
  Scraping a page with log in data (security, proxies) iamaghost 0 2,103 Mar-27-2021, 02:56 PM
Last Post: iamaghost
  Scraping Data from Singapore Turf Club singaporeman 2 2,359 Dec-15-2020, 01:28 PM
Last Post: MrBitPythoner
Thumbs Up Issue facing while scraping the data from different websites in single script. Balamani 1 2,076 Oct-20-2020, 09:56 AM
Last Post: Larz60+
  error in code web scraping alexisbrunaux 5 3,729 Aug-19-2020, 02:31 AM
Last Post: alexisbrunaux

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020