Python Forum
Fastest Way of Writing/Reading Data
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Fastest Way of Writing/Reading Data
#1
I have inherited a script that scrapes data from the web every few seconds and stores it in a mysql database. It then deletes data that is more than 30 seconds old. This means there is a maximum of 5-6 records in the database. Each record consists of 2 fields (scrape time and the data).

I don't know why mysql has been used to store the data but it seems unnecessarily cumbersome (although I am a beginner).

Is my thinking correct? If so, what would be the quickest way to write/read/clear data in these circumstances?
Reply
#2
you can use a flat file for storage, open using mode 'w+', ('a+' if you wish to keep old contents) and after each write make a call to flush()

example -- something like this (won't run as written, but syntax is there):
import os
from pathlib import Path

# following assures starting directory same as script.
os.chdir(os.path.abspath(os.path.dirname(__file__)))

# create ref to script directory
homepath = Path('.')

# Here, create a subdirectory to hold output data (move this to wherever you wish)
# if paths exists, do nothing. If not create them
OutDatapath = homepath / 'data_out'
OutDatapath.mkdir(exist_ok=True)

outfile = OutDatapath / 'MyData.txt' # Rename this as you wish

with outfile.open('w+') as fp:
    # ... your code here
    fp.write(scraped_data)
    fp.flush()
since the file is flushed each time written, it can be opened for read by another program, and get all data up to the last write.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Help with WebSocket reading data from anoter function korenron 0 184 Sep-19-2021, 11:08 AM
Last Post: korenron
  Reading data to python: turn into list or dataframe hhchenfx 2 706 Jun-01-2021, 10:28 AM
Last Post: Larz60+
  Reading data from mysql. stsxbel 2 656 May-23-2021, 06:56 PM
Last Post: stsxbel
Smile Set 'Time' format cell when writing data to excel and not 'custom' limors 3 1,407 Mar-29-2021, 09:36 PM
Last Post: Larz60+
  reading canbus data as hex korenron 9 1,702 Dec-30-2020, 01:52 PM
Last Post: korenron
  Reading Serial data Moris526 6 1,321 Dec-26-2020, 04:04 PM
Last Post: Moris526
  wrong data reading on uart fahri 6 1,142 Sep-29-2020, 03:07 PM
Last Post: Larz60+
  Fastest Method for Querying SQL Server with Python Pandas BuJayBelvin 7 1,583 Aug-02-2020, 06:21 PM
Last Post: jefsummers
  writing data to a csv-file apollo 1 1,107 Jul-03-2020, 02:28 PM
Last Post: DeaD_EyE
  Reading serial data and saving to a file Mohan 1 2,863 May-25-2020, 04:18 PM
Last Post: pyzyx3qwerty

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020