Feb-07-2019, 11:44 PM
The main idea was to determine (roughly) the number of rows to be read from the end of the file.
Since we know interval of measurements = 15 min, and the number of days = 28, we can expect that
last 1440*28/15 rows of the file include measurements for last 28 days. So, we don't need to load entire
file. It is sufficient to load last 1440*28/15 rows from it. This is preliminary filtering. From
these rows we can build a data frame. Further, we need to do precise filtering, using pandas, as you do in your code.
You told about 28 days, but in the following line of code, you take data for 14 days. Is it an error?
Since we know interval of measurements = 15 min, and the number of days = 28, we can expect that
last 1440*28/15 rows of the file include measurements for last 28 days. So, we don't need to load entire
file. It is sufficient to load last 1440*28/15 rows from it. This is preliminary filtering. From
these rows we can build a data frame. Further, we need to do precise filtering, using pandas, as you do in your code.
You told about 28 days, but in the following line of code, you take data for 14 days. Is it an error?
df = df.loc[df['rdate'] > datetime.datetime.now() - pd.to_timedelta("14day")]