Python Forum
Filling in missing values
Thread Rating:
  • 1 Vote(s) - 4 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Filling in missing values
#1
Hi all,

I'm working with a normalized JSON file and I'm trying to clean the data a bit.
I need to backwards fill the missing values for the name column. I'm trying different things, including the isnull() function with method=bfill and nothing seems to work. It leaves me wondering if it is because it's not a NaN, but instead has nothing listed. I'm new to python so maybe the answer is obvious, but I could use some help!
Since this is my first post unfortunately I can't provide a link to the screenshot or add an attachment.
Reply
#2
Hi melm0,

You should be able to cut'n'paste your code, also please remember to add the python code tags.

Hopefully someone will be able to help you.

Good Luck

Bass

"The good thing about standards is that you have so many to choose from" Andy S. Tanenbaum
Reply
#3
Thanks! here is the code to normalize the json table. From there fillna(method='bfill') does not fill in missing values in the 'name' column

import pandas as pd
import json
from pandas.io.json import json_normalize
import numpy as np
#loading data in file 
file = "world_bank_projects.json"
with open(file) as json_file:
    json_data = json.load(json_file)
#normalizing data with project name column. 
fill = json_normalize(json_data,'mjtheme_namecode', 'project_name')
fill.head(25)
Reply
#4
You need to post the relevant code relating to your question. For instance, you mention an "isnull" function, yet I do not see it in your code and a variable "bfill", again not present in the code you posted. Ideally, you should also post a small sample of the json file that shows an example of what you mean by 'backfill' as well as what you consider a complete line. For information on how to properly ask a question, thereby increasing the chance of a quick and correct answer, refer to our Help/Info document.
If it ain't broke, I just haven't gotten to it yet.
OS: Windows 10, openSuse 42.3, freeBSD 11, Raspian "Stretch"
Python 3.6.5, IDE: PyCharm 2018 Community Edition
Reply
#5
Hello

Not sure about how your json file looks but why not keep things simple and find the Value which is none. I am not sure what exactly you want to achieve but the below may help

import json

data = []
keys = ["field1","field2","field3","field4", "field5", "field6" ]

with open('test_json.json') as json_data:
    for line in json_data:
        dataline = json.loads(line)
        row = []
        for key in keys:
            row.append(dataline.get(key))
        data.append(row)
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Filling NaNs in a financial dataset larzz 11 1,888 Jun-07-2023, 03:40 PM
Last Post: snippsat
  groupby on var with missing values error zenvega 0 1,705 May-07-2021, 07:40 PM
Last Post: zenvega

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020