Python Forum
Using shift to compute the percent change in a time series
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Using shift to compute the percent change in a time series
#1
Hi, somebody mentioned that one could use ts/ts.shift(1)-1 to calculate the percent changes in a time series. I tried to understand this by hand calculating the percent changes on 2000-04-30. I got -3.1619163378849615 instead of -1.462553 using the formula. Have I done something wrong? I though percent changes is (new value - old value)/old value *100%.

In [91]: ts                                                                                                         
Out[91]: 
2000-01-31   -0.047169
2000-02-29    0.636350
2000-03-31    1.207707
2000-04-30   -0.558628
Freq: M, dtype: float64

In [92]: ts.shift(1)                                                                                                
Out[92]: 
2000-01-31         NaN
2000-02-29   -0.047169
2000-03-31    0.636350
2000-04-30    1.207707
Freq: M, dtype: float64

In [93]: ts/ts.shift(1)-1                                                                                           
Out[93]: 
2000-01-31          NaN
2000-02-29   -14.490780
2000-03-31     0.897866
2000-04-30    -1.462553
Freq: M, dtype: float64

In [94]:                                                                                                            

In [94]: (1.207707+0.558628)/(-0.558628)                                                                            
Out[94]: -3.1619163378849615
Reply
#2
Hello,
1.207707 is an old value, and -0.558628 is a new value. So, you've computed (old - new)/new in line 27(94).
Reply
#3
Thanks. By using 1.207707 as the old value and -0.558628 as the new value, I got -1.462553 as calculated by ts/ts.shift(1)-1. How come 1.207707 is the old value? The original (old) value in ts on 2000-04-30 is -0.558628. That is why I treated it as an old (original) value. It got updated to the new value of 1.207707.

Am I correct that for the discussion here, old value refers to the value in the previous timestamp in the original ts (i.e. value on 2000-3-31) rather than the original value in ts (i.e. value on 2000-04-30) that is being updated?
Reply
#4
Ok, maybe I misunderstood something... We have a series:

Output:
x[0], x[1], x[2], .... , x[N]
and you are trying to compute

Output:
(x[1]-x[0])/x[0], (x[2]-x[1])/x[1], ...., (x[N]-x[N-1])/x[N-1]
x[0] is older than x[1], x[1] is older than x[2], etc.

In terms of pandas/shift this series could be rewriting as follows

(x - x.shift(1))/x.shift(1) = x/x.shift(1) - 1

where

[inline]x.shift(1) = [NaN, x[0], x[1], ...., x[N-1]]
x = [x[0], x[1], ..., x[N]][/inline]

Your series is [-0.047169, 0.636350, 1.207707, -0.558628]. So, 1.207707 is older than
-0.558628, isn't it?!
Reply
#5
Everything you wrote in Post#4 makes sense when we treat x as a time series data. For example, x[0] is the starting position of a dog. x[1] is the position 1s later, x[2] is its position 2s later, etc. So, x[1] is the old positional data compared with x[2].
However, when I think of x.shift(1), my own way of reasoning (in Post#3) led to wrong answer. Usually, when do people use the function shift()?
Reply
#6
(Feb-22-2020, 11:30 PM)new_to_python Wrote: Usually, when do people use the function shift()?
In many cases, when you need to compute differences of neighboring values in the array,
you can just use indexing, e.g.
import numpy as np
x = np.array([1,2,3,4,5,6])
res = (x[1:] - x[:-1])/x[:-1] 
The .shift method might be helpful e.g. when working with dates:
import pandas as pd 
index = pd.date_range('01 / 01 / 2020', periods = 5, freq ='10H') 
df = pd.DataFrame({"A":[1, 2, 3, 4, 5],  
                   "B":[10, 20, 30, 40, 50]},
                    index=index)
df
Output:
A B 2020-01-01 00:00:00 1 10 2020-01-01 10:00:00 2 20 2020-01-01 20:00:00 3 30 2020-01-02 06:00:00 4 40 2020-01-02 16:00:00 5 50
df.shift(freq="5H")
Output:
A B 2020-01-01 05:00:00 1 10 2020-01-01 15:00:00 2 20 2020-01-02 01:00:00 3 30 2020-01-02 11:00:00 4 40 2020-01-02 21:00:00 5 50
If freq parameter is specified, and index is of date/datetime type, this method shifts index values only.

Finally, sometimes it is easier to type df.shift(10) and get shifted data instead of using index-based version, e.g. something like this df.values[10:, ...].
Reply
#7
Thanks
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  pandas: Compute the % of the unique values in a column JaneTan 0 44 Yesterday, 01:59 PM
Last Post: JaneTan
  Recommendations for ML libraries for time-series forecast AndreasPython 0 629 Jan-06-2021, 01:03 PM
Last Post: AndreasPython
  Combine groupby() and shift() in pandas rama27 0 1,694 Nov-17-2020, 09:49 PM
Last Post: rama27
  Pandas - compute means per category and time rama27 7 1,190 Nov-13-2020, 08:55 AM
Last Post: PsyPy
  Time Series forecating with multiple independent variables Krychol88 1 626 Oct-23-2020, 08:11 AM
Last Post: DPaul
  how to handling time series data file with Python? aupres 4 1,126 Aug-10-2020, 12:40 PM
Last Post: MattKahn13
  How to shift data frame rows of specified column Mekala 0 950 Jul-21-2020, 02:42 PM
Last Post: Mekala
  Changing Time Series from Start to End of Month illmattic 0 813 Jul-16-2020, 10:49 AM
Last Post: illmattic
  HELP- DATA FRAME INTO TIME SERIES- BASIC bntayfur 0 736 Jul-11-2020, 09:04 PM
Last Post: bntayfur
  Differencing Time series and Inverse after Training donnertrud 0 1,784 May-27-2020, 06:11 AM
Last Post: donnertrud

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020