Mar-03-2020, 03:18 PM
(Mar-02-2020, 07:03 PM)micseydel Wrote: You really need to provide more details. If your question is about Python code, we need to see the code that causes the problem. (If that code more than about ten lines, I highly recommend you come up with example code which is minimized to reproduce your problem, hard-code as much as possible.) As for the most recent results you posted, I'm not especially familiar with MS SQL but you'll need to provide your queries if you want people to comment on the results.
hi Micseydel,
here is the code:
from datetime import datetime, timedelta import pandas as pd import numpy as np import ast from warnings import catch_warnings from warnings import filterwarnings from statsmodels.tsa.holtwinters import ExponentialSmoothing import time import sqlite3 import pyodbc as msql from dateutil.relativedelta import relativedelta def lit_eval(strg): if type(strg) == str: return ast.literal_eval(strg) else: return strg def pd_dict2list(row): return [value for key, value in row.items()][0:-1] def frcst_cot(y, model, setup, fcst_hrz): result = None try: # never show warnings when grid searching, too noisy with catch_warnings(): filterwarnings("ignore") t, d, s, p, b, r = setup # fit model model = ExponentialSmoothing(y, seasonal_periods=p, trend=t, seasonal=s, damped=d).fit(use_boxcox=b, remove_bias=r) yhat_v = model.predict(len(y) - fcst_hrz) yhat = model.forecast(fcst_hrz) result = pd.concat([yhat_v, yhat], axis=0) except: error = None result = None # if no forecasting was done applying generic model if result is None: print('result is none, trying generic model...') try: # never show warnings when grid searching, too noisy with catch_warnings(): filterwarnings("ignore") t, d, s, p, b, r = setup # fit model model = ExponentialSmoothing(y).fit() yhat_v = model.predict(len(y) - fcst_hrz) yhat = model.forecast(fcst_hrz) result = pd.concat([yhat_v, yhat], axis=0) except: error = None result = None # if results are still None using Naive model prediction if result is None: print('result is still none, applying Naive model...') try: # never show warnings when grid searching, too noisy with catch_warnings(): filterwarnings("ignore") forecst = y.copy() idx = pd.date_range(start=y.tail(1).index.min(), periods=fcst_hrz, freq='MS') forecst = forecst.reindex(idx, fill_value=0) forecst['Volume'] = y.tail(1).values[0][0] forecst = forecst.squeeze() except: error = None return result # TODO: Add path to pickle file BEST_MODELS FILE best_models = pd.read_pickle(r'C:\Users\XXX\Desktop\..\best_models.pkl') starttime = time.time() for itm in range(len(best_models)): row = best_models[itm:itm + 1] fcst_horizon = 12 cols = (row.T != '(All)') cols.columns = ['chek'] cols = cols[cols.chek == True].T needed_cols = list(set(cols.columns) - {'DATE', 'MODEL', 'SETUP', 'VERTEX'}) qry_str = '\tand '.join([(c + ' == "' + row[c].values[0] + '" ') for c in needed_cols]) # TODO: ADD PICKLE PATH COT_VERTEX CoT_vertex = pd.read_pickle(r'C:\Users\XXX\Desktop\..CoT_vertex.pkl') CoT_vertex = CoT_vertex.copy() CoT_vertex = CoT_vertex.pivot_table(index='Date', values=['Volume'], aggfunc=np.sum).reset_index() if CoT_vertex.Date.min() != CoT_vertex.Date.min(): print('no COT data for:' + qry_str) else: if datetime.strptime(CoT_vertex.Date.max(), '%Y-%m-%d').date() >= ( datetime.now().replace(day=1) + relativedelta(months=-2)).date(): # Added idx = pd.date_range(start=CoT_vertex.Date.min(), end=CoT_vertex.Date.max(), freq='MS') CoT_vertex.set_index('Date', inplace=True) CoT_vertex = CoT_vertex.reindex(idx, fill_value=0) CoT_vertex = CoT_vertex.squeeze() frcst = frcst_cot(CoT_vertex, row['MODEL'].values[0], row['SETUP'].values[0], fcst_horizon) print(frcst) else: print('Last COT older than 3m')I tried to hardcode as much as I can but I am very new at Python so still struggling.
I am now looking to see how to send you the 2 pickle files so that you can run the code...