Copy and append data from .dat file

[Makada]

Great, thanks alot for your help, much appreciated

---

I just tried as a test to remove the first data line in the sourcefile and close the running script and started again after some minutes, as soon as i started the script, an error is shown.
I cant run the script now.

Traceback (most recent call last):
  File "F:\Software\python definitief\copieer bron naar copie alleen nieuw data deze werkt.py", line 16, in <module>
    outfile.write(content)
ValueError: I/O operation on closed file.

The script i ran was this:

import time
import schedule    
starttime=time.time()



Input_file: "C:\\Campbellsci\\LoggerNet\\CR1000_Table1.dat"
Output_file: "C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat"

while True:
    """ Not to append duplicate data to output file"""
    existingLines = set(line.strip() for line in open("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat"))
    outfile = open("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat", "a+")
    for content in open("C:\\Campbellsci\\LoggerNet\\CR1000_Table1.dat", "r"):
     if content.strip() not in existingLines: # to void duplicate lines
        outfile.write(content)
        existingLines.add(content)
        outfile.close()

time.sleep(60 - time.time() % 60)

---

I removed the "outfile.close()" from the script, now it runs again.
Dont know if its working ok now...

---

I noticed double data figures and after a restart of the script (at 21:12:00) its showing data already shown, see a piece of output below:

"2020-01-17 21:08:00",2067,12.08,262,15.37,17.65,6.376,6.478,6.267,71.8867,71.96,71.82,1018.672,1018.724,1018.662,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.682,1.776,1.581,4.294,4.375,4.189,6.376,6.478,959,966,952,0,0,0,0,0,3.948,3.155
"2020-01-17 21:08:00",2067,12.08,262,15.37,17.65,6.376,6.478,6.267,71.8867,71.96,71.82,1018.672,1018.724,1018.662,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.682,1.776,1.581,4.294,4.375,4.189,6.376,6.478,959,966,952,0,0,0,0,0,3.948,3.155
"2020-01-17 21:09:00",2068,8.16,263.7,26.31,12.47,6.36,6.444,6.25,71.96032,72.01,71.91,1018.679,1018.724,1018.662,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.68,1.761,1.579,4.285,4.358,4.179,6.36,6.444,958,964,951,0,0,0,0,0,4.822,3.757
"2020-01-17 21:09:00",2068,8.16,263.7,26.31,12.47,6.36,6.444,6.25,71.96032,72.01,71.91,1018.679,1018.724,1018.662,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.68,1.761,1.579,4.285,4.358,4.179,6.36,6.444,958,964,951,0,0,0,0,0,4.822,3.757
"2020-01-17 21:10:00",2069,9.31,264.8,24.91,14,6.366,6.461,6.284,72.02,72.05,71.98,1018.684,1018.724,1018.662,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.697,1.794,1.613,4.295,4.374,4.213,6.366,6.461,958,965,953,0,0,0,0,0,4.497,3.601
"2020-01-17 21:11:00",2070,9.09,256.9,12.86,13.59,6.37,6.444,6.318,72.05154,72.09,72.03,1018.699,1018.724,1018.662,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.707,1.788,1.656,4.302,4.362,4.251,6.37,6.444,959,964,955,0,0,0,0,0,4.582,3.614
"2020-01-17 21:12:00",2071,9.87,259.8,23,14.7,6.371,6.461,6.258,72.08898,72.15,72.04,1018.698,1018.724,1018.662,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.716,1.804,1.61,4.305,4.378,4.198,6.371,6.461,959,965,951,0,0,0,0,0,4.378,3.516
"2020-01-17 21:10:00",2069,9.31,264.8,24.91,14,6.366,6.461,6.284,72.02,72.05,71.98,1018.684,1018.724,1018.662,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.697,1.794,1.613,4.295,4.374,4.213,6.366,6.461,958,965,953,0,0,0,0,0,4.497,3.601
"2020-01-17 21:11:00",2070,9.09,256.9,12.86,13.59,6.37,6.444,6.318,72.05154,72.09,72.03,1018.699,1018.724,1018.662,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.707,1.788,1.656,4.302,4.362,4.251,6.37,6.444,959,964,955,0,0,0,0,0,4.582,3.614
"2020-01-17 21:12:00",2071,9.87,259.8,23,14.7,6.371,6.461,6.258,72.08898,72.15,72.04,1018.698,1018.724,1018.662,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.716,1.804,1.61,4.305,4.378,4.198,6.371,6.461,959,965,951,0,0,0,0,0,4.378,3.516
"2020-01-17 21:13:00",2072,9.46,253.7,17.5,12.95,6.35,6.444,6.258,72.16513,72.23,72.11,1018.71,1018.724,1018.662,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.711,1.804,1.622,4.291,4.365,4.203,6.35,6.444,957,964,951,0,0,0,0,0,4.439,3.746
"2020-01-17 21:13:00",2072,9.46,253.7,17.5,12.95,6.35,6.444,6.258,72.16513,72.23,72.11,1018.71,1018.724,1018.662,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.711,1.804,1.622,4.291,4.365,4.203,6.35,6.444,957,964,951,0,0,0,0,0,4.439,3.746

[ibreeden]

No you must close the file. But you changed the indentation. It should be:

import time
import schedule    
starttime=time.time()
 
 
 
#Input_file: "C:\\Campbellsci\\LoggerNet\\CR1000_Table1.dat"
#Output_file: "C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat"
 
while True:
    """ Not to append duplicate data to output file"""
    existingLines = set(line.strip() for line in open("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat"))
    outfile = open("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat", "a+")
    for content in open("C:\\Campbellsci\\LoggerNet\\CR1000_Table1.dat", "r"):
        if content.strip() not in existingLines: # to void duplicate lines
            outfile.write(content)
            existingLines.add(content)
    outfile.close()
 
    time.sleep(60 - time.time() % 60)

[Makada]

Hi,

Thanks for your input.
Its running better now, i didnt know the size of space is that sensitive.
Only thing i notice is that the output files data is one minute behind the source file data.
Could this be caused by the fact that the source file is updating every 00:01 minutes, the same as the output file?
Can i change that so both files are updating simultanious every minute?
The source file is always updating on the minute, every 00:01:00, which i cant change.

When i stop the script for some minutes and start again at lets say 22:41:21, the output is including data from 22:41:00.
Data values are ok, correspond with record number.

[Makada]

Hi,

Ive made a change and added the schedule function.
Now the output is spot on on the minute like the source.

Thanks.

import time
import schedule    
#starttime=time.time()
  
def task():  
  
#Input_file: "C:\\Campbellsci\\LoggerNet\\CR1000_Table1.dat"
#Output_file: "C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat"
  
 while True:
    """ Not to append duplicate data to output file"""
    existingLines = set(line.strip() for line in open("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat"))
    outfile = open("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat", "a+")
    for content in open("C:\\Campbellsci\\LoggerNet\\CR1000_Table1.dat", "r"):
        if content.strip() not in existingLines: # to void duplicate lines
            outfile.write(content)
            existingLines.add(content)
    outfile.close()
  
schedule.every().minute.at(":01").do(task)
while True:
    schedule.run_pending()
    time.sleep(1)
refresh()

[Makada]

Hi,

I was thinking how i can accomplish to have an extra output file in the form of CSV in the code above, working with the same source.
With the same options to output as the other, but in Excel file CSV so i can use that data directly into Excel.

Thanks.

[Makada]

I managed to add some code below to have a seperate csv file.

import time
import schedule    
import csv

#starttime=time.time()
  
def task():  
  
#Input_file: "C:\\Campbellsci\\LoggerNet\\CR1000_Table1.dat"
#Output_file: "C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat"
  
 while True:
    """ Not to append duplicate data to output file"""
    existingLines = set(line.strip() for line in open("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat"))
    outfile = open("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat", "a+")
    for content in open("C:\\Campbellsci\\LoggerNet\\CR1000_Table1.dat", "r"):
        if content.strip() not in existingLines: # to void duplicate lines
            outfile.write(content)
            existingLines.add(content)
    outfile.close()

    with open("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.dat", "r") as f:
        with open("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.csv", "a+") as f1:
            for line in f:
                f1.write(line)

    
schedule.every().minute.at(":01").do(task)
while True:
    schedule.run_pending()
    time.sleep(1)
refresh()

The CSV file is getting the data but after that, i get a error:

Traceback (most recent call last):
  File "F:\Software\python definitief\copieer bron naar copie alleen nieuw data deze werkt op de minuut precies kopie.py", line 31, in <module>
    schedule.run_pending()
  File "C:\Python\lib\site-packages\schedule\__init__.py", line 563, in run_pending
    default_scheduler.run_pending()
  File "C:\Python\lib\site-packages\schedule\__init__.py", line 94, in run_pending
    self._run_job(job)
  File "C:\Python\lib\site-packages\schedule\__init__.py", line 147, in _run_job
    ret = job.run()
  File "C:\Python\lib\site-packages\schedule\__init__.py", line 466, in run
    ret = self.job_func()
  File "F:\Software\python definitief\copieer bron naar copie alleen nieuw data deze werkt op de minuut precies kopie.py", line 23, in task
    with open("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.csv", "a+") as f1:
PermissionError: [Errno 13] Permission denied: 'C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie.csv'
>>> 

And the csv file isnt displayed the comma seperated data in Excel in columns.
How can i achieve that?

Thanks

[ibreeden]

You don't show what the line numbers are in run, run_job and run_pending that are involved with the error "PermissionError: [Errno 13] Permission denied". But my best bet is that you had the file open with Excel.
The csv file does not show the columns in Excel? I think Excel expects a semicolon as list separator instead of a comma. This is normal behaviour if you chose "Dutch" in your language settings. Search in your Windows settings for "lijstscheidingsteken". An other way is to execute "data to columns" in Excel.

[Makada]

Hi,

Thanks for your reply, ill have a look.
Can i use the output csv file from the script as a live source for Excel?

Thanks.