Python Forum
replace nan values by mean group by date.year, date.month - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Data Science (https://python-forum.io/forum-44.html)
+--- Thread: replace nan values by mean group by date.year, date.month (/thread-19599.html)



replace nan values by mean group by date.year, date.month - wissam1974 - Jul-06-2019

Hi for all

i have read a CSV file with tow series columns as follow:
Dateobs TMIN
2006-01-01 NAN
2006-01-02 12.3
2006-01-03 11.3
..
2006-02-01 15.2
2006-02-02 Nan
2006-03-03 11.3
..
2016-04-06 15.8
2016-04-07 11.6
2016-04-08 Nan
..
etc..

Error:
my question is how to replace NaN values for the daily TMIN column by mean group by (year month)
the table shown above was issued when i execute this pseudo code
    data_pd = pd.read_csv('beirut.csv')

    # overwriting data after changing format
    data_pd["Dateobs"] = pd.to_datetime(data_pd["Dateobs"])
    Dateobs_typedatetime64=  data_pd["Dateobs"]
    print(Dateobs_typedatetime64)

    TMIN_typeFloat64=  data_pd["TMIN"]
    print(TMIN_typeFloat64)
any help or suggestions to overpass this complexity will be appreciate.
thank you


RE: replace nan values by mean group by date.year, date.month - scidam - Jul-06-2019

Your code should be something like this:

data_pd.fillna(data_pd.groupby([data_pd.Dateobs.dt.year, df.Dateobs.dt.month]).transform('mean'), inplace=True)



RE: replace nan values by mean group by date.year, date.month - wissam1974 - Jul-06-2019

(Jul-06-2019, 01:10 AM)scidam Wrote: Your code should be something like this:

data_pd.fillna(data_pd.groupby([data_pd.Dateobs.dt.year, df.Dateobs.dt.month]).transform('mean'), inplace=True)

hi Mr first thank you for your reply, i appreciate it
i tried this
result=data_pd.fillna(data_pd.groupby([data_pd.Dateobs.dt.year, data_pd.Dateobs.dt.month]).transform('mean'), inplace=True)
print (result)
and i am getting the following result:
Output:
None Process finished with exit code 0
still the same problem its not replace null value by mean
is there anything else i should try
thank you


RE: replace nan values by mean group by date.year, date.month - scidam - Jul-06-2019

If you are using inplace=True you don't need to assign the result. Remove inplace=True in your code, or print(data_pd). result variable is None, because when inplace=true .fillna method returns None.

This is minimal working example:
import pandas as pd
df = pd.DataFrame({'date': ['2001-09-01', '2001-09-02', '2001-10-01', '2001-10-03', '2001-09-03'],
                   'value': [1, 2, 0, None, None]})
df.date = pd.to_datetime(df.date)
df.fillna(df.groupby([df.date.dt.year, df.date.dt.month]).transform('mean'), inplace=True)



RE: replace nan values by mean group by date.year, date.month - wissam1974 - Jul-06-2019

(Jul-06-2019, 01:49 AM)scidam Wrote: If you are using inplace=True you don't need to assign the result. Remove inplace=True in your code, or print(data_pd). result variable is None, because when inplace=true .fillna method returns None.

This is minimal working example:
import pandas as pd
df = pd.DataFrame({'date': ['2001-09-01', '2001-09-02', '2001-10-01', '2001-10-03', '2001-09-03'],
                   'value': [1, 2, 0, None, None]})
df.date = pd.to_datetime(df.date)
df.fillna(df.groupby([df.date.dt.year, df.date.dt.month]).transform('mean'), inplace=True)

Thank you a lot Mr you have helped me solve my problem Smile


RE: replace nan values by mean group by date.year, date.month - AnkitGupta - Feb-19-2020

I want to replace NAN value of Product_price column using fillna Mean based on product ID how I can implement. Is it possible I cant implement for all columns in my dataset based on Product_id

data['product_id'] = data.groupby('product_id').product_price.transform(lambda x: x.fillna(x.mean()))