Python Forum
Creating a matrix of rolling variances
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Creating a matrix of rolling variances
#1
I have a pandas data frame of six columns, I would like to iteratively compute the variance along each column. Since I am a newbie, I don't really understand the niceties of the language and common usage patterns. What is the common Python idiom for achieving the following?

vars = []
for i in range(1, 100000):
    v = (data.iloc[range(0, i+1)].var()).values
    if len(vars) == 0:
        vars = v
    else:
        vars = np.vstack((vars, v))
Also, when I run this code, it takes a long time to execute. Can anyone suggest how to improve the running time?
Reply
#2
Notice that vars is a reserved word, it is better not to use it as the name of a variable...

You can obtain what I think is what you are trying to do with:
pd.DataFrame(([data[:k].var() for k in range(1, 10000)]))
In a low level language in this case it might be better to go to the mathematical definition of variance and calculate the mean and the variance in each step, updating your accumulators... in python this might be not so optimum as you will need to access element by element your array.

In general, in python try to not to iterate by index (your range(1, 100000)) as it is specially inefficient. Normally all the python operations are "vectorized" to work with the full column, row or matrix. Take a look at the numpy and pandas documentation.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Rolling window and apply code JunkBoy 6 1,815 Jul-23-2022, 07:00 PM
Last Post: JunkBoy
  Hurst Exponent in Rolling Basis illmattic 1 3,804 Jan-06-2021, 09:49 PM
Last Post: illmattic
  sklearn.neural_network MLPClassifier forecast variances CK1960 1 1,779 Oct-29-2020, 10:13 AM
Last Post: CK1960
  Calculating Beta over Rolling Periods illmattic 2 5,280 Sep-27-2020, 11:27 PM
Last Post: Larz60+
  Creating look up table/matrix from 3d data array chai0404 3 2,825 Apr-09-2020, 04:53 AM
Last Post: buran
  Apply rolling window function over time dimension of 3D data Staph 0 2,160 Jan-01-2020, 08:31 AM
Last Post: Staph
  Grouping data based on rolling conditions kapilan15 0 1,930 Jun-05-2019, 01:07 PM
Last Post: kapilan15
  Pandas .rolling() with some calculations inside irmscher 5 6,130 Apr-04-2019, 11:55 AM
Last Post: scidam
  How to use pandas.rolling mean for 3D input array? Prv_Yadv 1 3,808 Mar-26-2019, 11:49 AM
Last Post: scidam
  Creating matrix counting words in list of strings jazmad 2 3,505 Dec-23-2018, 05:47 PM
Last Post: jazmad

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020