Python Forum
ARIMA.Fit memoryError - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Data Science (https://python-forum.io/forum-44.html)
+--- Thread: ARIMA.Fit memoryError (/thread-16268.html)



ARIMA.Fit memoryError - wissam1974 - Feb-20-2019

hi All python Forum experts
i am using the software pyCharm2018.1.1

i have tried to build ARIMA model in python, my model has been identified by the parameters
(p=0, d=0, q=367), here is the code:

def arima_Model_Static_PlotErrorAC_PAC(series): 
    train, expctd =series , series 
    arima_orders = (0, 0, 367)
    model = ARIMA(series, order=arima_orders)
    results_MA = model.fit(disp=-1, start_params=[.1 for i in range(1 + arima_orders[2])])
    yhatList=results_MA.fittedvalues 
    residuals = [expctd[i] - yhatList[i] for i in range(len(expctd))] 
    mse = mean_squared_error(expctd, yhatList)
    rmse = sqrt(mse)
    print(results_MA.summary())
    print(rmse)

this model is called as follow:

series=DataSetDiff #DataSetDiff is a series with a length of 3652 values
outputResidualError=arima_Model_Static_PlotErrorAC_PAC(series)
an error is loaded with this high q order whici is:
Error:
C:\109_personel\112_pyCharmArima\venv\Scripts\python.exe C:/109_personel/112_pyCharmArima/Presentation_Vers2_ModelOneFunct_3_5.py Traceback (most recent call last): File "C:/109_personel/112_pyCharmArima/Presentation_Vers2_ModelOneFunct_3_5.py", line 243, in arima_Model_Static_PlotErrorAC_PAC results_MA = model.fit(disp=-1, start_params=[.1 for i in range(1 + arima_orders[2])], solver='bfgs') File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\statsmodels\tsa\arima_model.py", line 959, in fit callback=callback, **kwargs) File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\statsmodels\base\model.py", line 466, in fit full_output=full_output) File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\statsmodels\base\optimizer.py", line 191, in _fit hess=hessian) File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\statsmodels\base\optimizer.py", line 327, in _fit_bfgs disp=disp, retall=retall, callback=callback) File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\scipy\optimize\optimize.py", line 916, in fmin_bfgs res = _minimize_bfgs(f, x0, args, fprime, callback=callback, **opts) File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\scipy\optimize\optimize.py", line 970, in _minimize_bfgs gfk = myfprime(x0) File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\scipy\optimize\optimize.py", line 300, in function_wrapper return function(*(wrapper_args + args)) File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\statsmodels\base\model.py", line 451, in score return -self.score(params, *args) / nobs File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\statsmodels\tsa\arima_model.py", line 583, in score return approx_fprime_cs(params, self.loglike, args=(False,)) File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\statsmodels\tools\numdiff.py", line 202, in approx_fprime_cs for i, ih in enumerate(increments)] File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\statsmodels\tools\numdiff.py", line 202, in <listcomp> for i, ih in enumerate(increments)] File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\statsmodels\tsa\arima_model.py", line 780, in loglike return self.loglike_kalman(params, set_sigma2) File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\statsmodels\tsa\arima_model.py", line 790, in loglike_kalman return KalmanFilter.loglike(params, self, set_sigma2) File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\statsmodels\tsa\kalmanf\kalmanfilter.py", line 654, in loglike R_mat, T_mat) File "kalman_loglike.pyx", line 359, in statsmodels.tsa.kalmanf.kalman_loglike.kalman_loglike_complex File "kalman_loglike.pyx", line 228, in statsmodels.tsa.kalmanf.kalman_loglike.kalman_filter_complex File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\numpy\core\numeric.py", line 2200, in identity return eye(n, dtype=dtype) File "C:\109_personel\112_pyCharmArima\venv\lib\site-packages\numpy\lib\twodim_base.py", line 186, in eye m = zeros((N, M), dtype=dtype, order=order) MemoryError Process finished with exit code 1
my model is well running and forecasting until q order MA 150 that mean(0,0,150). the error memoryError raised on the selection of q=367 as order

is any one can help me to solve this error, i have googled this error many time and i did not found a suitable solutions
Thank you for any help


RE: ARIMA.Fit memoryError - scidam - Feb-22-2019

It seems that you operate with big arrays somewhere, or such arrays occur during computations. zeros((N, M)) in Numpy tries to allocate N*M*8 bytes (8 is default, depends on dtype) in RAM, but you don't have enough RAM.


RE: ARIMA.Fit memoryError - wissam1974 - Feb-23-2019

(Feb-22-2019, 02:25 AM)scidam Wrote: It seems that you operate with big arrays somewhere, or such arrays occur during computations. zeros((N, M)) in Numpy tries to allocate N*M*8 bytes (8 is default, depends on dtype) in RAM, but you don't have enough RAM.

hi Mr. I have extended my physical RAM to 20G as a max limit that can hold my computer, and then I re-executed my model until q order equal to 180 as an example "ARIMA (0,0,180)" because my main objective is to go up to 367 as q order. I had kept my computer running for hours (20) no error had been loaded, but the execution of my model had not finished and no output coming. some suggestions found for this kind of problems are recommended to change the program, may be this means I should make some modifications in the predefined ARIMA.fit python function to get best optimization? because it is not an easy choice? if yes, can you help me trying it?