Python Forum
Keras.Predict into Dataframe
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Keras.Predict into Dataframe
#1
Hallo.

i'm working with multiple machine learning models.
I want to have all my predictions into a dataframe. The following is working for my other ml scripts(sklearn), expect for this one where i use Kera library.
It return the error: 'Exception: Data must be 1-dimensional'
My DataFrame for all my features is called 'feature_list' and all my targets is 'target_list'
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import statsmodels.api as sm
import keras 
import tensorflow as tf
from keras.models import Sequential
from keras.layers import Dense
from sklearn.metrics import r2_score

period = df['2011-01-01':'2015-01-01']
X = np.array(period[feature_list])
predictions = pd.DataFrame()
for i in target_list:
    y = np.array(period[i])
    split = sm.add_constant(X)
    train_size = int(0.75*y.shape[0])

    train_features = X[:train_size]
    train_targets = y[:train_size]
    test_features = X[train_size:]
    test_targets = y[train_size:]

    model = Sequential()
    model.add(Dense(5,input_dim=train_features.shape[1],activation='relu'))
    model.add(Dense(5,activation='relu'))
    model.add(Dense(1,activation='linear'))
    model.compile(optimizer='adam', loss='mse')
    model.fit(X, y, epochs=5, verbose=0)
    print(i)
    predictions[i] = model.predict(X[train_size:])
if i remove '[i]' in 'predictions[i] = model.predict(X[train_size:])' in return some values, so i think the mistake is in the compiling in some way.

So how do i compile all my predictions into a dataframe for all my targets?

Best regards Fynpyth from Denmark.
Reply
#2
Code is incomplete so cannot test, but I think line 32 should be
    predictions = model.predict(train_features)

and when you are ready to test change to test_features

and when you are ready to test change to test_features
Reply
#3
(Mar-13-2020, 05:25 PM)jefsummers Wrote: Code is incomplete so cannot test, but I think line 32 should be
    predictions = model.predict(train_features)

and when you are ready to test change to test_features

and when you are ready to test change to test_features

Thanks for answer.
The "train_features" match the "X[train_size:]" and the code is running for both if i remove '[i]' in the predictions[i], but it's only return one columns of predictions, and i need for all my 40 targets for a trading strategy.

I'm sorry i can't upload all my code, but it is for my thesis and my thesis can be flaged as plagiarism if i upload it 1 to 1.

sincerly.
Reply
#4
The colon is on the opposite side in train_features vs X[train_size:], so those are not the same thing.
After correcting that, do
predictions.shape()
and post
Reply
#5
(Mar-14-2020, 02:24 AM)jefsummers Wrote: The colon is on the opposite side in train_features vs X[train_size:], so those are not the same thing.
After correcting that, do
predictions.shape()
and post

You are right about X[train_size:] and train_features is not the same. My bad.

I tried your solution but it did not work, but i kinda found a strange 2-step solution.
maybe you know why it works?
predictions = pd.DataFrame()
for i in target_list:
   y = np.array(period[i])
   split = sm.add_constant(X)
   train_size = int(0.75*y.shape[0])

   train_features = X[:train_size]
   train_targets = y[:train_size]
   test_features = X[train_size:]
   test_targets = y[train_size:]

   model = Sequential()
   model.add(Dense(50,input_dim=train_features.shape[1],activation='relu'))
   model.add(Dense(50,activation='relu'))
   model.add(Dense(1,activation='linear'))
   model.compile(optimizer='adam', loss='mse')
   model.fit(X, y, epochs=1, verbose=1)
   print(i)
   predictions = pd.DataFrame(model.predict(X[train_size:]), columns=[i])
then i remove "predictions = pd.DataFrame()" and add [i] in the last line and get this:

for i in target_list:
   y = np.array(period[i])
   split = sm.add_constant(X)
   train_size = int(0.75*y.shape[0])

   train_features = X[:train_size]
   train_targets = y[:train_size]
   test_features = X[train_size:]
   test_targets = y[train_size:]

   model = Sequential()
   model.add(Dense(50,input_dim=train_features.shape[1],activation='relu'))
   model.add(Dense(50,activation='relu'))
   model.add(Dense(1,activation='linear'))
   model.compile(optimizer='adam', loss='mse')
   model.fit(X, y, epochs=1, verbose=1)
   print(i)
   predictions[i] = pd.DataFrame(model.predict(X[train_size:]), columns=[i])
and now it returns the dataframe, "predictions" with all the predictions for all my targets. If i don't edit the script, it doesn't run Think

best ragards
Reply
#6
The line you remove actually should not do anything - you assign predictions to a blank dataframe and then reassign it in the for loop. Adding the [i] then makes predictions a list of dataframes. I'd be interested if you used
type(predictions)
after your loop to verify that...
Reply
#7
(Mar-14-2020, 11:45 AM)jefsummers Wrote: The line you remove actually should not do anything - you assign predictions to a blank dataframe and then reassign it in the for loop. Adding the [i] then makes predictions a list of dataframes. I'd be interested if you used
type(predictions)
after your loop to verify that...
That was also what I was thinking.

It return nothing for the "type(predictions)"
In the "variable explorer" in spyder, it say it is a DataFrame.
Reply
#8
May be running into 2 variables - predictions and predictions[].

Idea - go back and in line 1 put
predictions[]
Then, in your prediction line use
predictions.append(pd.DataFrame(model.predict(X[train_size:]), columns=[i]))
Then you should have an array of dataframes. You can examine each one, and you can combine them into a single frame later.
Reply
#9
(Mar-14-2020, 04:22 PM)jefsummers Wrote: May be running into 2 variables - predictions and predictions[].

Idea - go back and in line 1 put
predictions[]
Then, in your prediction line use
predictions.append(pd.DataFrame(model.predict(X[train_size:]), columns=[i]))
Then you should have an array of dataframes. You can examine each one, and you can combine them into a single frame later.

When i run
predictions[]
in the first line, it return an "invalid syntax".
Think
Reply
#10
predictions = []. Sorry, forgot the equal sign
It's just to define predictions as a list so you can use append
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  How to use .predict() method Jack90_ 4 698 Nov-03-2023, 08:21 PM
Last Post: Larz60+
  How to predict Total Hours needed with List as Input? caninan 0 761 Jan-27-2023, 04:20 PM
Last Post: caninan
  1st layer tf.keras output shape set at multiple - need help! Afrodizzyjack 0 1,782 Jun-07-2022, 04:53 PM
Last Post: Afrodizzyjack
  issue displaying summary of whole keras CNN model on TensorFlow with python Afrodizzyjack 0 1,619 Oct-27-2021, 04:07 PM
Last Post: Afrodizzyjack
  Which network architecture should be chosen to predict disease severity? AlekseyPython 2 1,956 Aug-21-2021, 07:12 PM
Last Post: jefsummers
  How to predict output in a regression problem? Rezaafz 1 2,468 Apr-03-2021, 04:16 PM
Last Post: jefsummers
  Understanding Keras and TensorFlow and how to use them bytecrunch 1 2,052 Mar-11-2021, 02:40 PM
Last Post: jefsummers
  Problems feeding live input from my microphone into a keras model (SegFault: 11) zeptozetta 1 2,539 Sep-14-2020, 03:08 AM
Last Post: zeptozetta
  Making a Basic Keras Model - Input Shape and Parameters MattKahn13 0 2,093 Aug-16-2020, 04:36 PM
Last Post: MattKahn13
  Analyze, predict the next step in a sequence. Antonio0608 0 1,970 Jul-23-2020, 05:53 PM
Last Post: Antonio0608

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020