Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Keras.Predict into Dataframe
#1
Hallo.

i'm working with multiple machine learning models.
I want to have all my predictions into a dataframe. The following is working for my other ml scripts(sklearn), expect for this one where i use Kera library.
It return the error: 'Exception: Data must be 1-dimensional'
My DataFrame for all my features is called 'feature_list' and all my targets is 'target_list'
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import statsmodels.api as sm
import keras 
import tensorflow as tf
from keras.models import Sequential
from keras.layers import Dense
from sklearn.metrics import r2_score

period = df['2011-01-01':'2015-01-01']
X = np.array(period[feature_list])
predictions = pd.DataFrame()
for i in target_list:
    y = np.array(period[i])
    split = sm.add_constant(X)
    train_size = int(0.75*y.shape[0])

    train_features = X[:train_size]
    train_targets = y[:train_size]
    test_features = X[train_size:]
    test_targets = y[train_size:]

    model = Sequential()
    model.add(Dense(5,input_dim=train_features.shape[1],activation='relu'))
    model.add(Dense(5,activation='relu'))
    model.add(Dense(1,activation='linear'))
    model.compile(optimizer='adam', loss='mse')
    model.fit(X, y, epochs=5, verbose=0)
    print(i)
    predictions[i] = model.predict(X[train_size:])
if i remove '[i]' in 'predictions[i] = model.predict(X[train_size:])' in return some values, so i think the mistake is in the compiling in some way.

So how do i compile all my predictions into a dataframe for all my targets?

Best regards Fynpyth from Denmark.
Quote
#2
Code is incomplete so cannot test, but I think line 32 should be
    predictions = model.predict(train_features)

and when you are ready to test change to test_features

and when you are ready to test change to test_features
Quote
#3
(Mar-13-2020, 05:25 PM)jefsummers Wrote: Code is incomplete so cannot test, but I think line 32 should be
    predictions = model.predict(train_features)

and when you are ready to test change to test_features

and when you are ready to test change to test_features

Thanks for answer.
The "train_features" match the "X[train_size:]" and the code is running for both if i remove '[i]' in the predictions[i], but it's only return one columns of predictions, and i need for all my 40 targets for a trading strategy.

I'm sorry i can't upload all my code, but it is for my thesis and my thesis can be flaged as plagiarism if i upload it 1 to 1.

sincerly.
Quote
#4
The colon is on the opposite side in train_features vs X[train_size:], so those are not the same thing.
After correcting that, do
predictions.shape()
and post
Quote
#5
(Mar-14-2020, 02:24 AM)jefsummers Wrote: The colon is on the opposite side in train_features vs X[train_size:], so those are not the same thing.
After correcting that, do
predictions.shape()
and post

You are right about X[train_size:] and train_features is not the same. My bad.

I tried your solution but it did not work, but i kinda found a strange 2-step solution.
maybe you know why it works?
predictions = pd.DataFrame()
for i in target_list:
   y = np.array(period[i])
   split = sm.add_constant(X)
   train_size = int(0.75*y.shape[0])

   train_features = X[:train_size]
   train_targets = y[:train_size]
   test_features = X[train_size:]
   test_targets = y[train_size:]

   model = Sequential()
   model.add(Dense(50,input_dim=train_features.shape[1],activation='relu'))
   model.add(Dense(50,activation='relu'))
   model.add(Dense(1,activation='linear'))
   model.compile(optimizer='adam', loss='mse')
   model.fit(X, y, epochs=1, verbose=1)
   print(i)
   predictions = pd.DataFrame(model.predict(X[train_size:]), columns=[i])
then i remove "predictions = pd.DataFrame()" and add [i] in the last line and get this:

for i in target_list:
   y = np.array(period[i])
   split = sm.add_constant(X)
   train_size = int(0.75*y.shape[0])

   train_features = X[:train_size]
   train_targets = y[:train_size]
   test_features = X[train_size:]
   test_targets = y[train_size:]

   model = Sequential()
   model.add(Dense(50,input_dim=train_features.shape[1],activation='relu'))
   model.add(Dense(50,activation='relu'))
   model.add(Dense(1,activation='linear'))
   model.compile(optimizer='adam', loss='mse')
   model.fit(X, y, epochs=1, verbose=1)
   print(i)
   predictions[i] = pd.DataFrame(model.predict(X[train_size:]), columns=[i])
and now it returns the dataframe, "predictions" with all the predictions for all my targets. If i don't edit the script, it doesn't run Think

best ragards
Quote
#6
The line you remove actually should not do anything - you assign predictions to a blank dataframe and then reassign it in the for loop. Adding the [i] then makes predictions a list of dataframes. I'd be interested if you used
type(predictions)
after your loop to verify that...
Quote
#7
(Mar-14-2020, 11:45 AM)jefsummers Wrote: The line you remove actually should not do anything - you assign predictions to a blank dataframe and then reassign it in the for loop. Adding the [i] then makes predictions a list of dataframes. I'd be interested if you used
type(predictions)
after your loop to verify that...
That was also what I was thinking.

It return nothing for the "type(predictions)"
In the "variable explorer" in spyder, it say it is a DataFrame.
Quote
#8
May be running into 2 variables - predictions and predictions[].

Idea - go back and in line 1 put
predictions[]
Then, in your prediction line use
predictions.append(pd.DataFrame(model.predict(X[train_size:]), columns=[i]))
Then you should have an array of dataframes. You can examine each one, and you can combine them into a single frame later.
Quote
#9
(Mar-14-2020, 04:22 PM)jefsummers Wrote: May be running into 2 variables - predictions and predictions[].

Idea - go back and in line 1 put
predictions[]
Then, in your prediction line use
predictions.append(pd.DataFrame(model.predict(X[train_size:]), columns=[i]))
Then you should have an array of dataframes. You can examine each one, and you can combine them into a single frame later.

When i run
predictions[]
in the first line, it return an "invalid syntax".
Think
Quote
#10
predictions = []. Sorry, forgot the equal sign
It's just to define predictions as a list so you can use append
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Error when import Keras Azadfalah 1 193 Apr-29-2020, 04:45 AM
Last Post: buran
  Error When Using sklearn Predict Function firebird 0 197 Mar-21-2020, 04:34 PM
Last Post: firebird
  Keras + Matplotlib causing crash spearced 3 874 Feb-06-2020, 04:54 PM
Last Post: zljt3216
  Keras Dense layer with wrong input d1r4c 0 203 Jan-02-2020, 02:35 PM
Last Post: d1r4c
  Predict Longitude and Latitude Using Python vibeandvisualize 1 290 Dec-27-2019, 12:10 PM
Last Post: Larz60+
  How to predict with date as input for DecisionTreeRegressor sandeep_ganga 0 222 Dec-12-2019, 03:29 AM
Last Post: sandeep_ganga
  Keras: Time series classification midarq 0 370 Sep-25-2019, 09:03 AM
Last Post: midarq
  Keras. ModuleNotFoundError: No module named 'error' agatte 1 3,025 Sep-18-2019, 12:27 PM
Last Post: shivani
  Keras Target Problem inco 1 926 May-05-2019, 10:47 AM
Last Post: scidam
  keras tensor mainpulation output to next input JohnyBeGood 0 594 Nov-21-2018, 03:30 PM
Last Post: JohnyBeGood

Forum Jump:


Users browsing this thread: 1 Guest(s)