Copy and append data from .dat file

[Makada]

Hi,

Ill try to explain what i want to do

I have a datalogger which output a .dat file which has a maximum of 19 days worth of data.
When new data is written to the .dat file (and it has already 19 days worth of data) the first dataline (at the beginning of the .dat file) is removed to make place for the new dataline (on the end/bottom of the .dat file).

What i want is another .dat file with unlimited days of data.
So the new .dat file has to store and append all data from the source .dat file without removing the oldest data like in the source .dat file.
And it has to watch for/write missing data when the python script with the above function hasnt been running for some time, so it will look for missing data in the source .dat file and write that data in the new .dat file, appended.
The source file is updating every minute.
Maybe an example of python script code can be shown?

The source is : "C:\\Campbellsci\\Loggernet\\CR1000_Table1.dat"
The destination is :"C\\Users\\Mark\\Desktop\\CR1000_Table1.dat"

Im using Windows 10 python 3.8


Thanks.

[michael1789]

Maybe I don't understand, but you can duplicate the line that does the appending and change the destination of one to "C\\Users\\Mark\\Desktop\\CR1000_Table1.dat". I guess you'll have to open it too, but again, just copy the existing line and change the file name. No?

[Makada]

I managed to copy the .dat file with the code below (but only when i restart the script it will copy the file):

import os
import shutil

shutil.copy2 ("C:\\Campbellsci\\LoggerNet\\CR1000_Table1.dat", "C:\\Campbellsci\\LoggerNet\\CR1000_Table1 - kopie.dat")

Now i need to have the contents of the source into the new file.
Every minute and appended.
But the source is a moving 19 days data file, when the max of 19 days is reached, it removes the oldest data to make space for the latest data.
And when this python script hasnt been running for some time, the script needs to search for missing data in the new file compared to the source and append that data to the new file.

Thanks.

[Makada]

I found this piece of code on the internet with the title:

"Read a file content from input and write into output but same content shouldnt append to output"

When i run the code i get an error: "invalid syntax" and the f from file being red.

Heres the code:

Input file: "C:\\Campbellsci\\LoggerNet\\CR1000_Table1.dat"


output file: "C:\\Campbellsci\\LoggerNet\\CR1000_Table1 - kopie.dat"


process file: process.py

""" Not to append duplicate data to output file"""

existingLines = set(line.strip() for line in open("C:\\Campbellsci\\LoggerNet\\CR1000_Table1 - kopie.dat"))

outfile = open("C:\\Campbellsci\\LoggerNet\\CR1000_Table1 - kopie.dat", "a+")
for content in open("C:\\Campbellsci\\LoggerNet\\CR1000_Table1.dat", "r"):
    if content not in existingLines: # to void duplicate lines
        outfile.write(content+'\n')
        existingLines.add(content)
outfile.close()

Thanks.

[Makada]

I changed the "Input file" to "Input_file" and "output file" to "output_file", now is ok.
But now i get the following error:

NameError: name 'process' is not defined.

How can solve this?

Thanks.

[ibreeden]

Remove line 7

[Makada]

Hi,

Ive tried that, that solves the error.
When i run the code, it Works ok.
But when i re-run the code some minute later, i get the data twice, same list of data below the previous copied data, with the missed data in those few minutes added to the copied data.

Thanks.

---

Heres an example of the actual output.
As you can see, it copies the data including the missing 3 minutes, 11:11 11:12 and 11:13:

"TOA5","CR1000","CR1000","20810","CR1000.Std.32.05","CPU:basic - kopie.CR1","13463","Table1"
"TIMESTAMP","RECORD","WS_kph_S_WVT","WindDir_D1_WVT","WindDir_SD1_WVT","WS_kph_Max","AirTC_Avg","AirTC_Max","AirTC_Min","RH_Avg","RH_Max","RH_Min","BP_mbar_Avg","BP_mbar_Max","BP_mbar_Min","Rain_mm_Tot","Rain_mm_Intensity_1_min","Rain_mm_Max_Intensity_1_min","Rain_mm_Min_Intensity_1_min","Tot_Rain_mm_12_sec_Max","Tot_Rain_mm_12_sec_Avg","Lufft_R2S_Mode","Mode(1)","Mode(2)","Mode(3)","Mode(4)","Mode(5)","Mode(6)","SlrW_Avg","SlrW_Max","TdC_Avg","TdC_Max","TdC_Min","TwC_Avg","TwC_Max","TwC_Min","HI_C_Avg","HI_C_Max","SVPWPa_Avg","SVPWPa_Max","SVPWPa_Min","SunHrs_Tot","PrecipitationHrs_Tot","PotSlrW_Avg","PotSlrW_Max","PotSlrW_Min","WC_C_Avg","WC_C_Min"
"TS","RN","kilometers/hour","Deg","Deg","kilometers/hour","Deg C","Deg C","Deg C","%","%","%","mbar","mbar","mbar","mm","","","","","","","","","","","","","W/m^2","W/m^2","Deg C","Deg C","Deg C","Deg C","Deg C","Deg C","Deg C","Deg C","Pa","Pa","Pa","hours","","W/m^2","W/m^2","W/m^2","Deg C","Deg C"
"","","WVc","WVc","WVc","Max","Avg","Max","Min","Avg","Max","Min","Avg","Max","Min","Tot","Smp","Smp","Smp","Max","Avg","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Avg","Max","Avg","Max","Min","Avg","Max","Min","Avg","Max","Avg","Max","Min","Tot","Tot","Avg","Max","Min","Avg","Min"
"2020-01-17 11:07:00",1466,9.97,194.9,34.22,14.52,9.03,9.11,8.92,95.77014,95.8,95.7,1013.228,1013.254,1013.192,0,0,0,0,0,0,0,0,0,0,0,0,0,57.17,61.14,8.4,8.48,8.29,8.69,8.77,8.57,9.03,9.11,1149,1156,1141,0,0,330.7,331.5,329.9,7.519,6.7
"2020-01-17 11:08:00",1467,8.34,186,44.19,13.41,9.04,9.1,8.94,95.76873,95.8,95.7,1013.225,1013.285,1013.192,0,0,0,0,0,0,0,0,0,0,0,0,0,50.71,53.87,8.41,8.48,8.32,8.7,8.76,8.6,9.04,9.1,1151,1155,1143,0,0,332.3,333,331.5,7.864,6.933
"2020-01-17 11:09:00",1468,8.62,173,28.57,10.54,9.05,9.11,8.94,95.76466,95.8,95.7,1013.221,1013.285,1013.192,0,0,0,0,0,0,0,0,0,0,0,0,0,49.06,52.41,8.41,8.49,8.31,8.71,8.77,8.6,9.05,9.11,1151,1156,1143,0,0,333.8,334.6,333.1,7.782,7.38
"2020-01-17 11:10:00",1469,9.86,177.6,18.33,13.82,9.04,9.11,8.97,95.76958,95.8,95.7,1013.238,1013.254,1013.192,0,0,0,0,0,0,0,0,0,0,0,0,0,58.21,62.6,8.41,8.48,8.34,8.7,8.77,8.62,9.04,9.11,1151,1156,1145,0,0,335.4,336.2,334.6,7.571,6.892

"TOA5","CR1000","CR1000","20810","CR1000.Std.32.05","CPU:basic - kopie.CR1","13463","Table1"
"TIMESTAMP","RECORD","WS_kph_S_WVT","WindDir_D1_WVT","WindDir_SD1_WVT","WS_kph_Max","AirTC_Avg","AirTC_Max","AirTC_Min","RH_Avg","RH_Max","RH_Min","BP_mbar_Avg","BP_mbar_Max","BP_mbar_Min","Rain_mm_Tot","Rain_mm_Intensity_1_min","Rain_mm_Max_Intensity_1_min","Rain_mm_Min_Intensity_1_min","Tot_Rain_mm_12_sec_Max","Tot_Rain_mm_12_sec_Avg","Lufft_R2S_Mode","Mode(1)","Mode(2)","Mode(3)","Mode(4)","Mode(5)","Mode(6)","SlrW_Avg","SlrW_Max","TdC_Avg","TdC_Max","TdC_Min","TwC_Avg","TwC_Max","TwC_Min","HI_C_Avg","HI_C_Max","SVPWPa_Avg","SVPWPa_Max","SVPWPa_Min","SunHrs_Tot","PrecipitationHrs_Tot","PotSlrW_Avg","PotSlrW_Max","PotSlrW_Min","WC_C_Avg","WC_C_Min"
"TS","RN","kilometers/hour","Deg","Deg","kilometers/hour","Deg C","Deg C","Deg C","%","%","%","mbar","mbar","mbar","mm","","","","","","","","","","","","","W/m^2","W/m^2","Deg C","Deg C","Deg C","Deg C","Deg C","Deg C","Deg C","Deg C","Pa","Pa","Pa","hours","","W/m^2","W/m^2","W/m^2","Deg C","Deg C"
"","","WVc","WVc","WVc","Max","Avg","Max","Min","Avg","Max","Min","Avg","Max","Min","Tot","Smp","Smp","Smp","Max","Avg","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Avg","Max","Avg","Max","Min","Avg","Max","Min","Avg","Max","Avg","Max","Min","Tot","Tot","Avg","Max","Min","Avg","Min"
"2020-01-17 11:07:00",1466,9.97,194.9,34.22,14.52,9.03,9.11,8.92,95.77014,95.8,95.7,1013.228,1013.254,1013.192,0,0,0,0,0,0,0,0,0,0,0,0,0,57.17,61.14,8.4,8.48,8.29,8.69,8.77,8.57,9.03,9.11,1149,1156,1141,0,0,330.7,331.5,329.9,7.519,6.7
"2020-01-17 11:08:00",1467,8.34,186,44.19,13.41,9.04,9.1,8.94,95.76873,95.8,95.7,1013.225,1013.285,1013.192,0,0,0,0,0,0,0,0,0,0,0,0,0,50.71,53.87,8.41,8.48,8.32,8.7,8.76,8.6,9.04,9.1,1151,1155,1143,0,0,332.3,333,331.5,7.864,6.933
"2020-01-17 11:09:00",1468,8.62,173,28.57,10.54,9.05,9.11,8.94,95.76466,95.8,95.7,1013.221,1013.285,1013.192,0,0,0,0,0,0,0,0,0,0,0,0,0,49.06,52.41,8.41,8.49,8.31,8.71,8.77,8.6,9.05,9.11,1151,1156,1143,0,0,333.8,334.6,333.1,7.782,7.38
"2020-01-17 11:10:00",1469,9.86,177.6,18.33,13.82,9.04,9.11,8.97,95.76958,95.8,95.7,1013.238,1013.254,1013.192,0,0,0,0,0,0,0,0,0,0,0,0,0,58.21,62.6,8.41,8.48,8.34,8.7,8.77,8.62,9.04,9.11,1151,1156,1145,0,0,335.4,336.2,334.6,7.571,6.892
"2020-01-17 11:11:00",1470,7.879,176.4,34.06,11.22,9.05,9.1,8.97,95.78661,95.8,95.7,1013.219,1013.254,1013.161,0,0,0,0,0,0,0,0,0,0,0,0,0,64.3,65.51,8.42,8.47,8.34,8.71,8.76,8.63,9.05,9.1,1151,1155,1145,0,0,336.9,337.7,336.2,7.982,7.326
"2020-01-17 11:12:00",1471,9.48,170.7,29.66,14.01,9.06,9.15,8.99,95.79492,95.8,95.8,1013.218,1013.254,1013.192,0,0,0,0,0,0,0,0,0,0,0,0,0,62.72,64.06,8.44,8.52,8.36,8.72,8.8,8.64,9.06,9.15,1152,1159,1146,0,0,338.5,339.2,337.7,7.661,6.923
"2020-01-17 11:13:00",1472,10.48,184.6,28.24,16.13,9.07,9.16,8.98,95.79506,95.8,95.8,1013.231,1013.285,1013.13,0,0,0,0,0,0,0,0,0,0,0,0,0,54.98,59.69,8.44,8.54,8.35,8.73,8.82,8.64,9.07,9.16,1153,1160,1146,0,0,340,340.7,339.3,7.498,6.579


Thanks.

[ibreeden]

There is a difference in the way you handle the two files. On the "*kopie.dat" file you use the str.strip() function. On the "Table1.dat" you don't. This may corrupt the comparison. Try to change line 15 to:

    if content.strip() not in existingLines: # to avoid duplicate lines

[Makada]

Yes, that solved it, realy nice

Thinking about it, i probably made a mistake in thinking about new data to be added to the output without keeping an eye on the beginning of the sourcefile.

When the source file has reached its maximum lines of data covering 19 days, there will be data erased at the begin of the sourcefile to make space for new data.

Example: sourcefile begin with time/date 2020-01-17 11:08:00 and the sourcefile is full, at next update of the sourcefile it will remove the 11:08:00 to make space for the newest data, and will show 11:09:00. Its like a "moving" 19 days datasource.
But i need to have/keep the 11:08:00 data in the output file.

Am i right with the present code it will remove the 11:08:00 data line in the output datafile when sourcefile is full?

If so, how can i accomplish that the output keeps that 11:08:00 dataline?

Thanks.

[ibreeden]

No, your program contains no code to remove anything from the output file. The output file will grow forever.