Python Forum
is a pandas dataframe timeseries time index in a specified range (but ignoring date)?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
is a pandas dataframe timeseries time index in a specified range (but ignoring date)?
#1
I have a dataset of indexed timeseries data in csv file format that I'm reading to a pandas dataframe, and specifying the index as the column of time entries:

import pandas as pd
df = pd.read_csv(filename,header = 1,index_col = 1)
df.index = pd.to_datetime(df.index)
This will be part of a batch processing algorithm that opens the file, checks if any part of the timeseries is within a specified time period, and then either continues with other files in the directory (if no data in the period), or carries out further processing (if in specified range).

The timeseries index is in format yyyy:mm:dd hh:mm:ss.ms, with freq='100ms' (ie in DSP terms, the sampling frequency is 10Hz and the period, or sampling interval is 100ms). 

The specified time period ranges for checking against must be NON DATE-SPECIFIC ranges between two time bounds, in this case the period

23:00:00-07:00:00 (ie an 8h period spanning over two dates)

It is important the range checked against is time and not date-specific, as the files could have any date. I don't want to remove the date information from the index (and not sure that's even possible) as it will be useful later in the process.

WHAT HAVE I ALREADY CONSIDERED?

I have tried to create a Boolean mask for the data using timestamps, eg:

periodstart = pd.Timestamp('23:00:00.000')
periodend = pd.Timestamp('06:59:59.900')
mask = (df.index.time >= periodstart) & (df.index.time <= periodend)
This doesn't work, because the timestamps insert the current date on the clock. I need the algorithm to be non-date specific, as it will be operated in a batch application on data covering many days.

I considered identifying the dates in the datetime index for each datafile:

datesinseries = pd.Series(df.index).map(lambda t: t.date()).unique()
and using these to generate the timestamps, but this seems very cumbersome, indicating there is probably a much simpler way. It could also create a problem if the number of days covered in the datafiles varies beyond 2 (not likely with these data but I'd prefer not to create problems further down the road).

I have also tried to form a comparison series set using

df.index.time
period = pd.DatetimeIndex(start='23:00:00',end='07:00:00',freq='100ms')
mask = df.index.time in period
which returns a single Boolean 'False' no matter if the times are in the period specified. I think this syntax is fundamentally wrong as it treats a datetime index as if it is a list object, when is a type of array.

Finally, I've had a look at pandas.Period, pandas.period_range, pandas.Timedelta and a load of other stuff in the pandas documentation! There is a lot there, and I'm only just starting out with python, let alone pandas, so could do with an experienced helping hand!

Any suggestions for forming this check?

Thanks
Reply
#2
Can you not get the date from the obtained csv data and then set that date on the timestamps used to check between.
Reply
#3
Thanks for your reply. 

It's possible using 

datesinseries = pd.Series(df.index).map(lambda t: t.date()).unique()
but a) it seems a roundabout way to do it, and b) I'm not sure how to combine a date extracted this way with a manually-defined time in a single timestamp as bounds for the range. Any advice on this would be helpful.

Do I just combine strings?

The .unique approach returns a datetime object that isn't obvious how to combine into a timestamp object
Reply
#4
How about adding an hour to the timestamp and then checking the time only is between 00:00:00 and 07:59:59
import datetime
import pandas

timestamps = ('2000-01-01 02:30:45',
              '2005-01-01 23:30:45',
              '2010-01-01 7:30:45',
              '2016-01-01 16:30:45')

for time_stamp in timestamps:
    df = pandas.Timestamp(time_stamp)
    df += datetime.timedelta(hours=1)
    between = datetime.time(0, 0, 0) < df.time() < datetime.time(7, 59, 59)
    print(between)
Output:
True True False False
Reply
#5
Thanks, I like this idea as it's quite practical.

One issue I'm finding is that my datetimeindex is an immutable array of datetime objects. there doesn't seem to be any simple way to convert this directly to a mutable array to do the addition.

So far I've tried copying the whole datetimeindex to a dummy array but not managed to get this to work as python seems very resistant to datetime gymnastics. I've also tried stepping through each value in the datetimeindex to convert to timestamp, which works, but I can't seem to compile the Boolean values into a single mask array to then apply back on to the index. In your example above, 'between' returns individual values at each loop iteration.

I found a class someone created on github for these types of problems, which may explain why I'm finding issues. But I'm trying to minimise dependancies!

https://github.com/tgs/nptime/blob/master/nptime.py
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  update values in one dataframe based on another dataframe - Pandas iliasb 2 561 Aug-14-2021, 12:38 PM
Last Post: jefsummers
  empty row in pandas dataframe rwahdan 3 561 Jun-22-2021, 07:57 PM
Last Post: snippsat
  Does a pandas have a date without a time? AlekseyPython 6 1,616 Feb-10-2021, 09:24 AM
Last Post: Naheed
Question Pandas - Creating additional column in dataframe from another column Azureaus 2 893 Jan-11-2021, 09:53 PM
Last Post: Azureaus
  [split] Getting Index Error - list index out of range krishna 2 855 Jan-09-2021, 08:29 AM
Last Post: buran
  Comparing results within a list and appending to pandas dataframe Aryagm 1 715 Dec-17-2020, 01:08 PM
Last Post: palladium
  Pandas - compute means per category and time rama27 7 1,148 Nov-13-2020, 08:55 AM
Last Post: PsyPy
  How to search for specific string in Pandas dataframe Coding_Jam 1 957 Nov-02-2020, 09:35 AM
Last Post: PsyPy
  Interpolating DataFrame method=‘index’ help tlewick1 1 621 Oct-22-2020, 12:48 AM
Last Post: scidam
  Iterate through dataframe to extract delta of a particular time period lynnette1983 1 594 Oct-22-2020, 12:19 AM
Last Post: scidam

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020