Python Forum
Keras.Predict into Dataframe
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Keras.Predict into Dataframe
#1
Hallo.

i'm working with multiple machine learning models.
I want to have all my predictions into a dataframe. The following is working for my other ml scripts(sklearn), expect for this one where i use Kera library.
It return the error: 'Exception: Data must be 1-dimensional'
My DataFrame for all my features is called 'feature_list' and all my targets is 'target_list'
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import statsmodels.api as sm
import keras 
import tensorflow as tf
from keras.models import Sequential
from keras.layers import Dense
from sklearn.metrics import r2_score

period = df['2011-01-01':'2015-01-01']
X = np.array(period[feature_list])
predictions = pd.DataFrame()
for i in target_list:
    y = np.array(period[i])
    split = sm.add_constant(X)
    train_size = int(0.75*y.shape[0])

    train_features = X[:train_size]
    train_targets = y[:train_size]
    test_features = X[train_size:]
    test_targets = y[train_size:]

    model = Sequential()
    model.add(Dense(5,input_dim=train_features.shape[1],activation='relu'))
    model.add(Dense(5,activation='relu'))
    model.add(Dense(1,activation='linear'))
    model.compile(optimizer='adam', loss='mse')
    model.fit(X, y, epochs=5, verbose=0)
    print(i)
    predictions[i] = model.predict(X[train_size:])
if i remove '[i]' in 'predictions[i] = model.predict(X[train_size:])' in return some values, so i think the mistake is in the compiling in some way.

So how do i compile all my predictions into a dataframe for all my targets?

Best regards Fynpyth from Denmark.
Reply
#2
Code is incomplete so cannot test, but I think line 32 should be
    predictions = model.predict(train_features)

and when you are ready to test change to test_features

and when you are ready to test change to test_features
Reply
#3
(Mar-13-2020, 05:25 PM)jefsummers Wrote: Code is incomplete so cannot test, but I think line 32 should be
    predictions = model.predict(train_features)

and when you are ready to test change to test_features

and when you are ready to test change to test_features

Thanks for answer.
The "train_features" match the "X[train_size:]" and the code is running for both if i remove '[i]' in the predictions[i], but it's only return one columns of predictions, and i need for all my 40 targets for a trading strategy.

I'm sorry i can't upload all my code, but it is for my thesis and my thesis can be flaged as plagiarism if i upload it 1 to 1.

sincerly.
Reply
#4
The colon is on the opposite side in train_features vs X[train_size:], so those are not the same thing.
After correcting that, do
predictions.shape()
and post
Reply
#5
(Mar-14-2020, 02:24 AM)jefsummers Wrote: The colon is on the opposite side in train_features vs X[train_size:], so those are not the same thing.
After correcting that, do
predictions.shape()
and post

You are right about X[train_size:] and train_features is not the same. My bad.

I tried your solution but it did not work, but i kinda found a strange 2-step solution.
maybe you know why it works?
predictions = pd.DataFrame()
for i in target_list:
   y = np.array(period[i])
   split = sm.add_constant(X)
   train_size = int(0.75*y.shape[0])

   train_features = X[:train_size]
   train_targets = y[:train_size]
   test_features = X[train_size:]
   test_targets = y[train_size:]

   model = Sequential()
   model.add(Dense(50,input_dim=train_features.shape[1],activation='relu'))
   model.add(Dense(50,activation='relu'))
   model.add(Dense(1,activation='linear'))
   model.compile(optimizer='adam', loss='mse')
   model.fit(X, y, epochs=1, verbose=1)
   print(i)
   predictions = pd.DataFrame(model.predict(X[train_size:]), columns=[i])
then i remove "predictions = pd.DataFrame()" and add [i] in the last line and get this:

for i in target_list:
   y = np.array(period[i])
   split = sm.add_constant(X)
   train_size = int(0.75*y.shape[0])

   train_features = X[:train_size]
   train_targets = y[:train_size]
   test_features = X[train_size:]
   test_targets = y[train_size:]

   model = Sequential()
   model.add(Dense(50,input_dim=train_features.shape[1],activation='relu'))
   model.add(Dense(50,activation='relu'))
   model.add(Dense(1,activation='linear'))
   model.compile(optimizer='adam', loss='mse')
   model.fit(X, y, epochs=1, verbose=1)
   print(i)
   predictions[i] = pd.DataFrame(model.predict(X[train_size:]), columns=[i])
and now it returns the dataframe, "predictions" with all the predictions for all my targets. If i don't edit the script, it doesn't run Think

best ragards
Reply
#6
The line you remove actually should not do anything - you assign predictions to a blank dataframe and then reassign it in the for loop. Adding the [i] then makes predictions a list of dataframes. I'd be interested if you used
type(predictions)
after your loop to verify that...
Reply
#7
(Mar-14-2020, 11:45 AM)jefsummers Wrote: The line you remove actually should not do anything - you assign predictions to a blank dataframe and then reassign it in the for loop. Adding the [i] then makes predictions a list of dataframes. I'd be interested if you used
type(predictions)
after your loop to verify that...
That was also what I was thinking.

It return nothing for the "type(predictions)"
In the "variable explorer" in spyder, it say it is a DataFrame.
Reply
#8
May be running into 2 variables - predictions and predictions[].

Idea - go back and in line 1 put
predictions[]
Then, in your prediction line use
predictions.append(pd.DataFrame(model.predict(X[train_size:]), columns=[i]))
Then you should have an array of dataframes. You can examine each one, and you can combine them into a single frame later.
Reply
#9
(Mar-14-2020, 04:22 PM)jefsummers Wrote: May be running into 2 variables - predictions and predictions[].

Idea - go back and in line 1 put
predictions[]
Then, in your prediction line use
predictions.append(pd.DataFrame(model.predict(X[train_size:]), columns=[i]))
Then you should have an array of dataframes. You can examine each one, and you can combine them into a single frame later.

When i run
predictions[]
in the first line, it return an "invalid syntax".
Think
Reply
#10
predictions = []. Sorry, forgot the equal sign
It's just to define predictions as a list so you can use append
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Which network architecture should be chosen to predict disease severity? AlekseyPython 2 319 Aug-21-2021, 07:12 PM
Last Post: jefsummers
  How to predict output in a regression problem? Rezaafz 1 684 Apr-03-2021, 04:16 PM
Last Post: jefsummers
  Understanding Keras and TensorFlow and how to use them bytecrunch 1 737 Mar-11-2021, 02:40 PM
Last Post: jefsummers
  Making a Basic Keras Model - Input Shape and Parameters MattKahn13 0 924 Aug-16-2020, 04:36 PM
Last Post: MattKahn13
  Analyze, predict the next step in a sequence. Antonio0608 0 850 Jul-23-2020, 05:53 PM
Last Post: Antonio0608
  Error when import Keras Azadfalah 1 1,085 Apr-29-2020, 04:45 AM
Last Post: buran
  Error When Using sklearn Predict Function firebird 0 889 Mar-21-2020, 04:34 PM
Last Post: firebird
  Keras + Matplotlib causing crash spearced 3 2,061 Feb-06-2020, 04:54 PM
Last Post: zljt3216
  Keras Dense layer with wrong input d1r4c 0 738 Jan-02-2020, 02:35 PM
Last Post: d1r4c
  Predict Longitude and Latitude Using Python vibeandvisualize 1 970 Dec-27-2019, 12:10 PM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020