Python Forum
replace nan values by mean group by date.year, date.month
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
replace nan values by mean group by date.year, date.month
#1
Hi for all

i have read a CSV file with tow series columns as follow:
Dateobs TMIN
2006-01-01 NAN
2006-01-02 12.3
2006-01-03 11.3
..
2006-02-01 15.2
2006-02-02 Nan
2006-03-03 11.3
..
2016-04-06 15.8
2016-04-07 11.6
2016-04-08 Nan
..
etc..

Error:
my question is how to replace NaN values for the daily TMIN column by mean group by (year month)
the table shown above was issued when i execute this pseudo code
    data_pd = pd.read_csv('beirut.csv')

    # overwriting data after changing format
    data_pd["Dateobs"] = pd.to_datetime(data_pd["Dateobs"])
    Dateobs_typedatetime64=  data_pd["Dateobs"]
    print(Dateobs_typedatetime64)

    TMIN_typeFloat64=  data_pd["TMIN"]
    print(TMIN_typeFloat64)
any help or suggestions to overpass this complexity will be appreciate.
thank you
Reply
#2
Your code should be something like this:

data_pd.fillna(data_pd.groupby([data_pd.Dateobs.dt.year, df.Dateobs.dt.month]).transform('mean'), inplace=True)
Reply
#3
(Jul-06-2019, 01:10 AM)scidam Wrote: Your code should be something like this:

data_pd.fillna(data_pd.groupby([data_pd.Dateobs.dt.year, df.Dateobs.dt.month]).transform('mean'), inplace=True)

hi Mr first thank you for your reply, i appreciate it
i tried this
result=data_pd.fillna(data_pd.groupby([data_pd.Dateobs.dt.year, data_pd.Dateobs.dt.month]).transform('mean'), inplace=True)
print (result)
and i am getting the following result:
Output:
None Process finished with exit code 0
still the same problem its not replace null value by mean
is there anything else i should try
thank you
Reply
#4
If you are using inplace=True you don't need to assign the result. Remove inplace=True in your code, or print(data_pd). result variable is None, because when inplace=true .fillna method returns None.

This is minimal working example:
import pandas as pd
df = pd.DataFrame({'date': ['2001-09-01', '2001-09-02', '2001-10-01', '2001-10-03', '2001-09-03'],
                   'value': [1, 2, 0, None, None]})
df.date = pd.to_datetime(df.date)
df.fillna(df.groupby([df.date.dt.year, df.date.dt.month]).transform('mean'), inplace=True)
Reply
#5
(Jul-06-2019, 01:49 AM)scidam Wrote: If you are using inplace=True you don't need to assign the result. Remove inplace=True in your code, or print(data_pd). result variable is None, because when inplace=true .fillna method returns None.

This is minimal working example:
import pandas as pd
df = pd.DataFrame({'date': ['2001-09-01', '2001-09-02', '2001-10-01', '2001-10-03', '2001-09-03'],
                   'value': [1, 2, 0, None, None]})
df.date = pd.to_datetime(df.date)
df.fillna(df.groupby([df.date.dt.year, df.date.dt.month]).transform('mean'), inplace=True)

Thank you a lot Mr you have helped me solve my problem Smile
Reply
#6
I want to replace NAN value of Product_price column using fillna Mean based on product ID how I can implement. Is it possible I cant implement for all columns in my dataset based on Product_id

data['product_id'] = data.groupby('product_id').product_price.transform(lambda x: x.fillna(x.mean()))
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  [Numpy] Load date/time from .txt to 'datetime64' type. water 4 403 Mar-01-2024, 11:16 PM
Last Post: Gribouillis
  Function to Select a Date standenman 2 893 May-08-2023, 07:12 PM
Last Post: standenman
  Pandas read csv file in 'date/time' chunks MorganSamage 4 1,647 Feb-13-2023, 11:24 AM
Last Post: MorganSamage
  Sort by month in a plot ph4n70m1988 0 992 Dec-11-2022, 06:14 AM
Last Post: ph4n70m1988
  replace sets of values in an array without using loops paul18fr 7 1,629 Jun-20-2022, 08:15 PM
Last Post: paul18fr
  Sorting list from FTP by Date lastyle 2 2,545 Nov-02-2021, 05:46 PM
Last Post: ibreeden
  Does a pandas have a date without a time? AlekseyPython 6 4,865 Feb-10-2021, 09:24 AM
Last Post: Naheed
  Changing Time Series from Start to End of Month illmattic 0 1,827 Jul-16-2020, 10:49 AM
Last Post: illmattic
  How Do I Only Get the Year from Date and Isolate Data for Year? WhatsupSmiley 2 2,133 Apr-14-2020, 11:45 AM
Last Post: snippsat
  why is the ticker and date different junkone 1 1,610 Apr-02-2020, 06:34 PM
Last Post: snippsat

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020