Python Forum
Copy and append data from .dat file
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Copy and append data from .dat file
#11
Great, thanks alot for your help, much appreciated Smile

I just tried as a test to remove the first data line in the sourcefile and close the running script and started again after some minutes, as soon as i started the script, an error is shown.
I cant run the script now.

Traceback (most recent call last):
  File "F:\Software\python definitief\copieer bron naar copie alleen nieuw data deze werkt.py", line 16, in <module>
    outfile.write(content)
ValueError: I/O operation on closed file.
The script i ran was this:

import time
import schedule    
starttime=time.time()



Input_file: "C:\\Campbellsci\\LoggerNet\\CR1000_Table1.dat"
Output_file: "C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat"

while True:
    """ Not to append duplicate data to output file"""
    existingLines = set(line.strip() for line in open("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat"))
    outfile = open("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat", "a+")
    for content in open("C:\\Campbellsci\\LoggerNet\\CR1000_Table1.dat", "r"):
     if content.strip() not in existingLines: # to void duplicate lines
        outfile.write(content)
        existingLines.add(content)
        outfile.close()

time.sleep(60 - time.time() % 60)

I removed the "outfile.close()" from the script, now it runs again.
Dont know if its working ok now...

I noticed double data figures and after a restart of the script (at 21:12:00) its showing data already shown, see a piece of output below:

"2020-01-17 21:08:00",2067,12.08,262,15.37,17.65,6.376,6.478,6.267,71.8867,71.96,71.82,1018.672,1018.724,1018.662,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.682,1.776,1.581,4.294,4.375,4.189,6.376,6.478,959,966,952,0,0,0,0,0,3.948,3.155
"2020-01-17 21:08:00",2067,12.08,262,15.37,17.65,6.376,6.478,6.267,71.8867,71.96,71.82,1018.672,1018.724,1018.662,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.682,1.776,1.581,4.294,4.375,4.189,6.376,6.478,959,966,952,0,0,0,0,0,3.948,3.155
"2020-01-17 21:09:00",2068,8.16,263.7,26.31,12.47,6.36,6.444,6.25,71.96032,72.01,71.91,1018.679,1018.724,1018.662,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.68,1.761,1.579,4.285,4.358,4.179,6.36,6.444,958,964,951,0,0,0,0,0,4.822,3.757
"2020-01-17 21:09:00",2068,8.16,263.7,26.31,12.47,6.36,6.444,6.25,71.96032,72.01,71.91,1018.679,1018.724,1018.662,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.68,1.761,1.579,4.285,4.358,4.179,6.36,6.444,958,964,951,0,0,0,0,0,4.822,3.757
"2020-01-17 21:10:00",2069,9.31,264.8,24.91,14,6.366,6.461,6.284,72.02,72.05,71.98,1018.684,1018.724,1018.662,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.697,1.794,1.613,4.295,4.374,4.213,6.366,6.461,958,965,953,0,0,0,0,0,4.497,3.601
"2020-01-17 21:11:00",2070,9.09,256.9,12.86,13.59,6.37,6.444,6.318,72.05154,72.09,72.03,1018.699,1018.724,1018.662,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.707,1.788,1.656,4.302,4.362,4.251,6.37,6.444,959,964,955,0,0,0,0,0,4.582,3.614
"2020-01-17 21:12:00",2071,9.87,259.8,23,14.7,6.371,6.461,6.258,72.08898,72.15,72.04,1018.698,1018.724,1018.662,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.716,1.804,1.61,4.305,4.378,4.198,6.371,6.461,959,965,951,0,0,0,0,0,4.378,3.516
"2020-01-17 21:10:00",2069,9.31,264.8,24.91,14,6.366,6.461,6.284,72.02,72.05,71.98,1018.684,1018.724,1018.662,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.697,1.794,1.613,4.295,4.374,4.213,6.366,6.461,958,965,953,0,0,0,0,0,4.497,3.601
"2020-01-17 21:11:00",2070,9.09,256.9,12.86,13.59,6.37,6.444,6.318,72.05154,72.09,72.03,1018.699,1018.724,1018.662,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.707,1.788,1.656,4.302,4.362,4.251,6.37,6.444,959,964,955,0,0,0,0,0,4.582,3.614
"2020-01-17 21:12:00",2071,9.87,259.8,23,14.7,6.371,6.461,6.258,72.08898,72.15,72.04,1018.698,1018.724,1018.662,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.716,1.804,1.61,4.305,4.378,4.198,6.371,6.461,959,965,951,0,0,0,0,0,4.378,3.516
"2020-01-17 21:13:00",2072,9.46,253.7,17.5,12.95,6.35,6.444,6.258,72.16513,72.23,72.11,1018.71,1018.724,1018.662,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.711,1.804,1.622,4.291,4.365,4.203,6.35,6.444,957,964,951,0,0,0,0,0,4.439,3.746
"2020-01-17 21:13:00",2072,9.46,253.7,17.5,12.95,6.35,6.444,6.258,72.16513,72.23,72.11,1018.71,1018.724,1018.662,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.711,1.804,1.622,4.291,4.365,4.203,6.35,6.444,957,964,951,0,0,0,0,0,4.439,3.746
Reply
#12
No you must close the file. But you changed the indentation. It should be:
import time
import schedule    
starttime=time.time()
 
 
 
#Input_file: "C:\\Campbellsci\\LoggerNet\\CR1000_Table1.dat"
#Output_file: "C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat"
 
while True:
    """ Not to append duplicate data to output file"""
    existingLines = set(line.strip() for line in open("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat"))
    outfile = open("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat", "a+")
    for content in open("C:\\Campbellsci\\LoggerNet\\CR1000_Table1.dat", "r"):
        if content.strip() not in existingLines: # to void duplicate lines
            outfile.write(content)
            existingLines.add(content)
    outfile.close()
 
    time.sleep(60 - time.time() % 60)
Reply
#13
Hi,

Thanks for your input.
Its running better now, i didnt know the size of space is that sensitive.
Only thing i notice is that the output files data is one minute behind the source file data.
Could this be caused by the fact that the source file is updating every 00:01 minutes, the same as the output file?
Can i change that so both files are updating simultanious every minute?
The source file is always updating on the minute, every 00:01:00, which i cant change.

When i stop the script for some minutes and start again at lets say 22:41:21, the output is including data from 22:41:00.
Data values are ok, correspond with record number.
Reply
#14
Hi,

Ive made a change and added the schedule function.
Now the output is spot on on the minute like the source.

Thanks.

import time
import schedule    
#starttime=time.time()
  
def task():  
  
#Input_file: "C:\\Campbellsci\\LoggerNet\\CR1000_Table1.dat"
#Output_file: "C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat"
  
 while True:
    """ Not to append duplicate data to output file"""
    existingLines = set(line.strip() for line in open("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat"))
    outfile = open("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat", "a+")
    for content in open("C:\\Campbellsci\\LoggerNet\\CR1000_Table1.dat", "r"):
        if content.strip() not in existingLines: # to void duplicate lines
            outfile.write(content)
            existingLines.add(content)
    outfile.close()
  
schedule.every().minute.at(":01").do(task)
while True:
    schedule.run_pending()
    time.sleep(1)
refresh()
Reply
#15
Hi,

I was thinking how i can accomplish to have an extra output file in the form of CSV in the code above, working with the same source.
With the same options to output as the other, but in Excel file CSV so i can use that data directly into Excel.

Thanks.
Reply
#16
I managed to add some code below to have a seperate csv file.

import time
import schedule    
import csv

#starttime=time.time()
  
def task():  
  
#Input_file: "C:\\Campbellsci\\LoggerNet\\CR1000_Table1.dat"
#Output_file: "C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat"
  
 while True:
    """ Not to append duplicate data to output file"""
    existingLines = set(line.strip() for line in open("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat"))
    outfile = open("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat", "a+")
    for content in open("C:\\Campbellsci\\LoggerNet\\CR1000_Table1.dat", "r"):
        if content.strip() not in existingLines: # to void duplicate lines
            outfile.write(content)
            existingLines.add(content)
    outfile.close()

    with open("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat", "r") as f:
        with open("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.csv", "a+") as f1:
            for line in f:
                f1.write(line)

    
schedule.every().minute.at(":01").do(task)
while True:
    schedule.run_pending()
    time.sleep(1)
refresh()
The CSV file is getting the data but after that, i get a error:

Traceback (most recent call last):
  File "F:\Software\python definitief\copieer bron naar copie alleen nieuw data deze werkt op de minuut precies kopie.py", line 31, in <module>
    schedule.run_pending()
  File "C:\Python\lib\site-packages\schedule\__init__.py", line 563, in run_pending
    default_scheduler.run_pending()
  File "C:\Python\lib\site-packages\schedule\__init__.py", line 94, in run_pending
    self._run_job(job)
  File "C:\Python\lib\site-packages\schedule\__init__.py", line 147, in _run_job
    ret = job.run()
  File "C:\Python\lib\site-packages\schedule\__init__.py", line 466, in run
    ret = self.job_func()
  File "F:\Software\python definitief\copieer bron naar copie alleen nieuw data deze werkt op de minuut precies kopie.py", line 23, in task
    with open("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.csv", "a+") as f1:
PermissionError: [Errno 13] Permission denied: 'C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.csv'
>>> 
And the csv file isnt displayed the comma seperated data in Excel in columns.
How can i achieve that?

Thanks Smile
Reply
#17
You don't show what the line numbers are in run, run_job and run_pending that are involved with the error "PermissionError: [Errno 13] Permission denied". But my best bet is that you had the file open with Excel.
The csv file does not show the columns in Excel? I think Excel expects a semicolon as list separator instead of a comma. This is normal behaviour if you chose "Dutch" in your language settings. Search in your Windows settings for "lijstscheidingsteken". An other way is to execute "data to columns" in Excel.
Reply
#18
Hi,

Thanks for your reply, ill have a look.
Can i use the output csv file from the script as a live source for Excel?

Thanks.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Copy Paste excel files based on the first letters of the file name Viento 2 346 Feb-07-2024, 12:24 PM
Last Post: Viento
  How to copy work sheet data one workbook to other? sayyedkamran 2 645 Nov-03-2023, 09:10 AM
Last Post: Larz60+
  Copy data from Excel and paste into Discord (Midjourney) Joe_Wright 4 1,923 Jun-06-2023, 05:49 PM
Last Post: rajeshgk
Question How to append integers from file to list? Milan 8 1,358 Mar-11-2023, 10:59 PM
Last Post: DeaD_EyE
  is it possible to copy image from email and place into excel file? cubangt 3 1,212 Nov-30-2022, 05:11 PM
Last Post: snippsat
  Please help me [copy and paste file from src to dst] midomarc 2 961 Nov-24-2022, 10:13 PM
Last Post: midomarc
  read a text file, find all integers, append to list oldtrafford 12 3,367 Aug-11-2022, 08:23 AM
Last Post: Pedroski55
  How to modify python script to append data on file using sql server 2019? ahmedbarbary 1 1,175 Aug-03-2022, 06:03 AM
Last Post: Pedroski55
  Append data to Yaml section tbaror 0 6,886 Feb-09-2022, 06:56 PM
Last Post: tbaror
  Copy column from one existing excel file to another file mkujawsk 0 5,485 Apr-14-2021, 06:33 PM
Last Post: mkujawsk

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020