Python Forum
iterate over index and define each range as a day
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
iterate over index and define each range as a day
#1
Hi,
my last question didn't find any help/answer and I found another approach and I wanted to know if it's possible to iterate over a column set as index(DateTime with pandas format:
2019-05-02 00:03:00
2019-05-02 00:08:00
2019-05-02 00:13:00
2019-05-02 00:18:00
2019-05-02 00:23:00
2019-05-02 00:28:00
2019-05-02 00:33:00
...
), so that during the iteration I can specify that the range from 00:03:00 to 23:59:00 is a day
(do something) and so on. I have issues dealing with date objects on pandas.
Thks for your help.
Reply
#2
Do you want to add new column based on time (with values like 'day', 'night' or whatever)? Or do you want iterate and if 'day time' is found do something with row?
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Reply
#3
(Nov-15-2019, 09:23 AM)perfringo Wrote: Do you want to add a new column based on time (with values like 'day', 'night' or whatever)? Or do you want to iterate and if 'day time' is found do something with row?

Thks for replying ... No I don't want to add a new column on time but what I'm trying to do, is to iterate on the index column of the df and do some operations in another column of that Dataframe (my Dataframe has 3 columns and I want to manipulate the df['x'] corresponding to that day range found and so on for others days that would be found!) hope it makes sense.
Reply
#4
I don't know whether it addresses your problem:

import pandas as pd

data = ['2019-05-02 00:03:00', '2019-05-02 00:08:00', '2019-05-02 00:13:00', '2019-05-02 00:18:00',
        '2019-05-02 00:23:00', '2019-05-02 00:28:00', '2019-05-02 00:33:00']

df = pd.DataFrame({'num': range(len(data))}, index=pd.to_datetime(data))
It will create following DataFrame with DatetimeIndex:

Output:
num 2019-05-02 00:03:00 0 2019-05-02 00:08:00 1 2019-05-02 00:13:00 2 2019-05-02 00:18:00 3 2019-05-02 00:23:00 4 2019-05-02 00:28:00 5 2019-05-02 00:33:00 6
Now we can set num value based on time:

df.iloc[df.index.indexer_between_time('00:13:00', '23:59:00')] = 20
Which will give:

Output:
num 2019-05-02 00:03:00 0 2019-05-02 00:08:00 1 2019-05-02 00:13:00 20 2019-05-02 00:18:00 20 2019-05-02 00:23:00 20 2019-05-02 00:28:00 20 2019-05-02 00:33:00 20
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Reply
#5
Previously I wrote a different answer and saw too late, that it was already answered.
If you need something, which generates datetime intervals e.g. for business days:

Here an example with dateutil.
You should read the documentation about dateutil.rrule.

It can generate intervals.
Here an example how to generate intervals with dateutil package.
Maybe it's useful together with pandas, but also without pandas.


import datetime as dt

# python3 -m pip install dateultil --user
# or install it in a venv
# often this package is installed, because other packages depends on this module

from dateutil.rrule import rrule, HOURLY


def hourly(start, interval, count):
    """
    Finite generator for hourly intervals
    count defines how many elements
    """
    for dt_interval in rrule(freq=HOURLY, interval=interval, dtstart=start, count=count):
        yield dt_interval
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply
#6
(Nov-15-2019, 11:48 AM)perfringo Wrote: I don't know whether it addresses your problem:

import pandas as pd

data = ['2019-05-02 00:03:00', '2019-05-02 00:08:00', '2019-05-02 00:13:00', '2019-05-02 00:18:00',
        '2019-05-02 00:23:00', '2019-05-02 00:28:00', '2019-05-02 00:33:00']

df = pd.DataFrame({'num': range(len(data))}, index=pd.to_datetime(data))
It will create following DataFrame with DatetimeIndex:

Output:
num 2019-05-02 00:03:00 0 2019-05-02 00:08:00 1 2019-05-02 00:13:00 2 2019-05-02 00:18:00 3 2019-05-02 00:23:00 4 2019-05-02 00:28:00 5 2019-05-02 00:33:00 6
Now we can set num value based on time:

df.iloc[df.index.indexer_between_time('00:13:00', '23:59:00')] = 20
Which will give:

Output:
num 2019-05-02 00:03:00 0 2019-05-02 00:08:00 1 2019-05-02 00:13:00 20 2019-05-02 00:18:00 20 2019-05-02 00:23:00 20 2019-05-02 00:28:00 20 2019-05-02 00:33:00 20

Thks for your time but its not really what I want to do
Reply
#7
My understanding has let me down. However, how should I interpret the problem:

Quote:I wanted to know if it's possible to iterate over a column set as index(DateTime /.../ so that during the iteration I can specify that the range from 00:03:00 to 23:59:00 is a day
(do something) and so on

I don't want to add a new column on time but what I'm trying to do, is to iterate on the index column of the df and do some operations in another column of that Dataframe (my Dataframe has 3 columns and I want to manipulate the df['x'] corresponding to that day range found and so on for others days that would be found!)

As I see it based on problem statement above:

Dataframe with DateTime as index. Check. Access to rows by time range. Check. Possibility to manipulate values. Check.
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Reply
#8
(Nov-18-2019, 08:06 AM)perfringo Wrote: My understanding has let me down. However, how should I interpret the problem:

Quote:I wanted to know if it's possible to iterate over a column set as index(DateTime /.../ so that during the iteration I can specify that the range from 00:03:00 to 23:59:00 is a day
(do something) and so on

I don't want to add a new column on time but what I'm trying to do, is to iterate on the index column of the df and do some operations in another column of that Dataframe (my Dataframe has 3 columns and I want to manipulate the df['x'] corresponding to that day range found and so on for others days that would be found!)

As I see it based on problem statement above:

Dataframe with DateTime as index. Check. Access to rows by time range. Check. Possibility to manipulate values. Check.


Hi,

I was trying to do something like this ... my bad if I express myself wrong!

import numpy as np
import pandas as pd
from datetime import datetime, timedelta

df = pd.DataFrame(columns=['SomeDatetime', 'x', 'y'], index=[0,1,2,3,4,5])
now = datetime.now()
df.loc[0, 'SomeDatetime'] = now + timedelta(minutes = 10)
df.loc[1, 'SomeDatetime'] = now - timedelta(days = 1)
df.loc[2, 'SomeDatetime'] = now + timedelta(minutes = 15)
df.loc[3, 'SomeDatetime'] = now + timedelta(minutes = 20)
df.loc[4, 'SomeDatetime'] = now + timedelta(minutes = 50)
df.loc[5, 'SomeDatetime'] = now - timedelta(days = 30*4)  - timedelta(days = 3)

df['x'] = pd.Series(np.random.randn(6))
df['y'] = pd.Series(np.random.randn(6))

df.set_index('SomeDatetime', inplace=True)

print("Dataframe\n")
print(df)
print("\nDay Loop\n")

for date in df.index.to_series().dt.date.unique():
    print(date)
    day_value = df[df.index.to_series().dt.date == date]
    print(day_value)
    print('\n')
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Index out of range error standenman 0 1,040 May-22-2023, 10:35 PM
Last Post: standenman
  [split] Getting Index Error - list index out of range krishna 2 2,568 Jan-09-2021, 08:29 AM
Last Post: buran
  Cycle through Numpy range within another range(?) Zero01 0 1,995 Jul-31-2020, 02:37 PM
Last Post: Zero01
  Getting Index Error - list index out of range RahulSingh 2 6,102 Feb-03-2020, 07:17 AM
Last Post: RahulSingh
  pandas.read_sas with chunksize: IndexError list index out of range axelle 0 2,549 Jan-28-2020, 09:30 AM
Last Post: axelle
  IndexEroor : index out of range adithyakrish 7 6,770 May-19-2017, 07:34 PM
Last Post: buran
  is a pandas dataframe timeseries time index in a specified range (but ignoring date)? m_lotinga 4 19,090 Dec-12-2016, 10:51 PM
Last Post: m_lotinga
  Unable to understand reason for error IndexError: tuple index out of range rajat2504 4 54,054 Dec-09-2016, 11:04 AM
Last Post: Kebap

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020