Python Forum
Using shift to compute the percent change in a time series
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Using shift to compute the percent change in a time series
#1
Hi, somebody mentioned that one could use ts/ts.shift(1)-1 to calculate the percent changes in a time series. I tried to understand this by hand calculating the percent changes on 2000-04-30. I got -3.1619163378849615 instead of -1.462553 using the formula. Have I done something wrong? I though percent changes is (new value - old value)/old value *100%.

In [91]: ts                                                                                                         
Out[91]: 
2000-01-31   -0.047169
2000-02-29    0.636350
2000-03-31    1.207707
2000-04-30   -0.558628
Freq: M, dtype: float64

In [92]: ts.shift(1)                                                                                                
Out[92]: 
2000-01-31         NaN
2000-02-29   -0.047169
2000-03-31    0.636350
2000-04-30    1.207707
Freq: M, dtype: float64

In [93]: ts/ts.shift(1)-1                                                                                           
Out[93]: 
2000-01-31          NaN
2000-02-29   -14.490780
2000-03-31     0.897866
2000-04-30    -1.462553
Freq: M, dtype: float64

In [94]:                                                                                                            

In [94]: (1.207707+0.558628)/(-0.558628)                                                                            
Out[94]: -3.1619163378849615
Reply
#2
Hello,
1.207707 is an old value, and -0.558628 is a new value. So, you've computed (old - new)/new in line 27(94).
Reply
#3
Thanks. By using 1.207707 as the old value and -0.558628 as the new value, I got -1.462553 as calculated by ts/ts.shift(1)-1. How come 1.207707 is the old value? The original (old) value in ts on 2000-04-30 is -0.558628. That is why I treated it as an old (original) value. It got updated to the new value of 1.207707.

Am I correct that for the discussion here, old value refers to the value in the previous timestamp in the original ts (i.e. value on 2000-3-31) rather than the original value in ts (i.e. value on 2000-04-30) that is being updated?
Reply
#4
Ok, maybe I misunderstood something... We have a series:

Output:
x[0], x[1], x[2], .... , x[N]
and you are trying to compute

Output:
(x[1]-x[0])/x[0], (x[2]-x[1])/x[1], ...., (x[N]-x[N-1])/x[N-1]
x[0] is older than x[1], x[1] is older than x[2], etc.

In terms of pandas/shift this series could be rewriting as follows

(x - x.shift(1))/x.shift(1) = x/x.shift(1) - 1

where

[inline]x.shift(1) = [NaN, x[0], x[1], ...., x[N-1]]
x = [x[0], x[1], ..., x[N]][/inline]

Your series is [-0.047169, 0.636350, 1.207707, -0.558628]. So, 1.207707 is older than
-0.558628, isn't it?!
Reply
#5
Everything you wrote in Post#4 makes sense when we treat x as a time series data. For example, x[0] is the starting position of a dog. x[1] is the position 1s later, x[2] is its position 2s later, etc. So, x[1] is the old positional data compared with x[2].
However, when I think of x.shift(1), my own way of reasoning (in Post#3) led to wrong answer. Usually, when do people use the function shift()?
Reply
#6
(Feb-22-2020, 11:30 PM)new_to_python Wrote: Usually, when do people use the function shift()?
In many cases, when you need to compute differences of neighboring values in the array,
you can just use indexing, e.g.
import numpy as np
x = np.array([1,2,3,4,5,6])
res = (x[1:] - x[:-1])/x[:-1] 
The .shift method might be helpful e.g. when working with dates:
import pandas as pd 
index = pd.date_range('01 / 01 / 2020', periods = 5, freq ='10H') 
df = pd.DataFrame({"A":[1, 2, 3, 4, 5],  
                   "B":[10, 20, 30, 40, 50]},
                    index=index)
df
Output:
A B 2020-01-01 00:00:00 1 10 2020-01-01 10:00:00 2 20 2020-01-01 20:00:00 3 30 2020-01-02 06:00:00 4 40 2020-01-02 16:00:00 5 50
df.shift(freq="5H")
Output:
A B 2020-01-01 05:00:00 1 10 2020-01-01 15:00:00 2 20 2020-01-02 01:00:00 3 30 2020-01-02 11:00:00 4 40 2020-01-02 21:00:00 5 50
If freq parameter is specified, and index is of date/datetime type, this method shifts index values only.

Finally, sometimes it is easier to type df.shift(10) and get shifted data instead of using index-based version, e.g. something like this df.values[10:, ...].
Reply
#7
Thanks
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Help: Conversion of Electricity Data into Time Series Data SmallGuy 3 1,155 Oct-04-2023, 03:31 PM
Last Post: deanhystad
  Time Series Production Process Problem Mzarour 1 2,096 Feb-28-2023, 12:25 PM
Last Post: get2sid
  validate the existing value and change the series datahub 0 1,191 May-03-2022, 08:00 PM
Last Post: datahub
  reduce time series based on sum condition amdi40 0 1,078 Apr-06-2022, 09:09 AM
Last Post: amdi40
  How to accumulate volume of time series amdi40 3 2,259 Feb-15-2022, 02:23 PM
Last Post: amdi40
  pandas: Compute the % of the unique values in a column JaneTan 1 1,756 Oct-25-2021, 07:55 PM
Last Post: jefsummers
  Recommendations for ML libraries for time-series forecast AndreasPython 0 1,862 Jan-06-2021, 01:03 PM
Last Post: AndreasPython
  Combine groupby() and shift() in pandas rama27 0 4,151 Nov-17-2020, 09:49 PM
Last Post: rama27
  Pandas - compute means per category and time rama27 7 3,416 Nov-13-2020, 08:55 AM
Last Post: PsyPy
  Time Series forecating with multiple independent variables Krychol88 1 1,823 Oct-23-2020, 08:11 AM
Last Post: DPaul

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020