Using shift to compute the percent change in a time series

Using shift to compute the percent change in a time series - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Data Science (https://python-forum.io/forum-44.html)
+--- Thread: Using shift to compute the percent change in a time series (/thread-24602.html)

Using shift to compute the percent change in a time series - new_to_python - Feb-21-2020

Hi, somebody mentioned that one could use ts/ts.shift(1)-1 to calculate the percent changes in a time series. I tried to understand this by hand calculating the percent changes on 2000-04-30. I got -3.1619163378849615 instead of -1.462553 using the formula. Have I done something wrong? I though percent changes is (new value - old value)/old value *100%.

In [91]: ts                                                                                                         
Out[91]: 
2000-01-31   -0.047169
2000-02-29    0.636350
2000-03-31    1.207707
2000-04-30   -0.558628
Freq: M, dtype: float64

In [92]: ts.shift(1)                                                                                                
Out[92]: 
2000-01-31         NaN
2000-02-29   -0.047169
2000-03-31    0.636350
2000-04-30    1.207707
Freq: M, dtype: float64

In [93]: ts/ts.shift(1)-1                                                                                           
Out[93]: 
2000-01-31          NaN
2000-02-29   -14.490780
2000-03-31     0.897866
2000-04-30    -1.462553
Freq: M, dtype: float64

In [94]:                                                                                                            

In [94]: (1.207707+0.558628)/(-0.558628)                                                                            
Out[94]: -3.1619163378849615

RE: Using shift to compute the percent change in a time series - scidam - Feb-22-2020

Hello,
1.207707 is an old value, and -0.558628 is a new value. So, you've computed (old - new)/new in line 27(94).

RE: Using shift to compute the percent change in a time series - new_to_python - Feb-22-2020

Thanks. By using 1.207707 as the old value and -0.558628 as the new value, I got -1.462553 as calculated by ts/ts.shift(1)-1. How come 1.207707 is the old value? The original (old) value in ts on 2000-04-30 is -0.558628. That is why I treated it as an old (original) value. It got updated to the new value of 1.207707.

Am I correct that for the discussion here, old value refers to the value in the previous timestamp in the original ts (i.e. value on 2000-3-31) rather than the original value in ts (i.e. value on 2000-04-30) that is being updated?

RE: Using shift to compute the percent change in a time series - scidam - Feb-22-2020

Ok, maybe I misunderstood something... We have a series:

Output:
x[0], x[1], x[2], .... , x[N]

and you are trying to compute

Output:
(x[1]-x[0])/x[0], (x[2]-x[1])/x[1], ...., (x[N]-x[N-1])/x[N-1]

x[0] is older than x[1], x[1] is older than x[2], etc.

In terms of pandas/shift this series could be rewriting as follows

(x - x.shift(1))/x.shift(1) = x/x.shift(1) - 1

where

[inline]x.shift(1) = [NaN, x[0], x[1], ...., x[N-1]]
x = [x[0], x[1], ..., x[N]][/inline]

Your series is [-0.047169, 0.636350, 1.207707, -0.558628]. So, 1.207707 is older than
-0.558628, isn't it?!

RE: Using shift to compute the percent change in a time series - new_to_python - Feb-22-2020

Everything you wrote in Post#4 makes sense when we treat x as a time series data. For example, x[0] is the starting position of a dog. x[1] is the position 1s later, x[2] is its position 2s later, etc. So, x[1] is the old positional data compared with x[2].
However, when I think of x.shift(1), my own way of reasoning (in Post#3) led to wrong answer. Usually, when do people use the function shift()?

RE: Using shift to compute the percent change in a time series - scidam - Feb-23-2020

(Feb-22-2020, 11:30 PM)new_to_python Wrote: Usually, when do people use the function shift()?

In many cases, when you need to compute differences of neighboring values in the array,
you can just use indexing, e.g.

import numpy as np
x = np.array([1,2,3,4,5,6])
res = (x[1:] - x[:-1])/x[:-1]

The .shift method might be helpful e.g. when working with dates:

import pandas as pd 
index = pd.date_range('01 / 01 / 2020', periods = 5, freq ='10H') 
df = pd.DataFrame({"A":[1, 2, 3, 4, 5],  
                   "B":[10, 20, 30, 40, 50]},
                    index=index)

df

Output:	        A	B
2020-01-01 00:00:00	1	10
2020-01-01 10:00:00	2	20
2020-01-01 20:00:00	3	30
2020-01-02 06:00:00	4	40
2020-01-02 16:00:00	5	50

df.shift(freq="5H")

Output:	        A	B
2020-01-01 05:00:00	1	10
2020-01-01 15:00:00	2	20
2020-01-02 01:00:00	3	30
2020-01-02 11:00:00	4	40
2020-01-02 21:00:00	5	50

If freq parameter is specified, and index is of date/datetime type, this method shifts index values only.

Finally, sometimes it is easier to type df.shift(10) and get shifted data instead of using index-based version, e.g. something like this df.values[10:, ...].

RE: Using shift to compute the percent change in a time series - new_to_python - Mar-03-2020

Thanks