Python Forum
Fastest Way of Writing/Reading Data
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Fastest Way of Writing/Reading Data
#1
I have inherited a script that scrapes data from the web every few seconds and stores it in a mysql database. It then deletes data that is more than 30 seconds old. This means there is a maximum of 5-6 records in the database. Each record consists of 2 fields (scrape time and the data).

I don't know why mysql has been used to store the data but it seems unnecessarily cumbersome (although I am a beginner).

Is my thinking correct? If so, what would be the quickest way to write/read/clear data in these circumstances?
Reply
#2
you can use a flat file for storage, open using mode 'w+', ('a+' if you wish to keep old contents) and after each write make a call to flush()

example -- something like this (won't run as written, but syntax is there):
import os
from pathlib import Path

# following assures starting directory same as script.
os.chdir(os.path.abspath(os.path.dirname(__file__)))

# create ref to script directory
homepath = Path('.')

# Here, create a subdirectory to hold output data (move this to wherever you wish)
# if paths exists, do nothing. If not create them
OutDatapath = homepath / 'data_out'
OutDatapath.mkdir(exist_ok=True)

outfile = OutDatapath / 'MyData.txt' # Rename this as you wish

with outfile.open('w+') as fp:
    # ... your code here
    fp.write(scraped_data)
    fp.flush()
since the file is flushed each time written, it can be opened for read by another program, and get all data up to the last write.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Fastest way tkinter Quatrixouuu 2 349 Feb-19-2024, 07:20 AM
Last Post: Danishhafeez
  What is the fastest way to get all the frames from a video file? glorsh66 3 992 May-26-2023, 04:41 AM
Last Post: Gribouillis
  Issue in writing sql data into csv for decimal value to scientific notation mg24 8 2,874 Dec-06-2022, 11:09 AM
Last Post: mg24
  Reading All The RAW Data Inside a PDF NBAComputerMan 4 1,275 Nov-30-2022, 10:54 PM
Last Post: Larz60+
  Create a function for writing to SQL data to csv mg24 4 1,111 Oct-01-2022, 04:30 AM
Last Post: mg24
  Reading Data from JSON tpolim008 2 1,031 Sep-27-2022, 06:34 PM
Last Post: Larz60+
  Need Help writing data into Excel format ajitnayak87 8 2,438 Feb-04-2022, 03:00 AM
Last Post: Jeff_t
  Help reading data from serial RS485 korenron 8 13,596 Nov-14-2021, 06:49 AM
Last Post: korenron
  Help with WebSocket reading data from anoter function korenron 0 1,300 Sep-19-2021, 11:08 AM
Last Post: korenron
  Reading data to python: turn into list or dataframe hhchenfx 2 5,279 Jun-01-2021, 10:28 AM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020