Replace null values in Json file

Replace null values in Json file - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: Replace null values in Json file (/thread-10860.html)

Pages: 1 2

Replace null values in Json file - burnsssss - Jun-10-2018

Hi all!!
i am new in forum so i have very much to read!

I have one problem with some code in python. I am trying to replace None values in Json file to "vacio" values.

My example of read file is:

data = []
for reng in open("con_error.json", "r"):
    data.append(json.loads(reng))

and the result is:

[{u'contributors': None,
u'coordinates': None,
u'created_at': u'Sun Jun 10 14:21:53 +0000 2018',
u'entities': {u'hashtags': [],
u'symbols': [],
u'urls': [],
u'user_mentions': [{u'id': 254599256,
u'id_str': u'254599256',
u'indices': [3, 11],
u'name': u'Alejandro Fargosi',
u'screen_name': u'fargosi'}]},
u'favorite_count': 0,
u'favorited': False,
u'filter_level': u'low',
u'geo': None,
u'id': 1005817329459097600L,
u'id_str': u'1005817329459097600',
u'in_reply_to_screen_name': None,
u'in_reply_to_status_id': None}]

Now, i use the next code for replace None for "vacio":

json_string=json.dumps(data).replace("null", "vacio")

and the result is:

'[{"quote_count": 0, "contributors": vacio, "truncated": false, "text": "RT @fargosi: Del 20-4 al 10-6-18, nuestros defensores:\\nMauro Maldonado muerto\\nAbigail Alvarez muerta\\nJorge Cusi muerto\\nDaniel Rios, muerto\\u2026", "is_quote_status": false, "in_reply_to_status_id": vacio, "reply_count": 0, "id": 1005817329459097600, "favorite_count": 0, "entities": {"user_mentions": [{"id": 254599256, "indices": [3, 11], "id_str": "254599256", "screen_name": "fargosi", "name": "Alejandro Fargosi"}], "symbols": [], "hashtags": [], "urls": []}, "retweeted": false, "coordinates": vacio, "timestamp_ms": "1528640513182", "source": "<a href=\\"http://twitter.com/download/iphone\\" rel=\\"nofollow\\">Twitter for iPhone</a>", "in_reply_to_screen_name": vacio, "id_str": "1005817329459097600", "retweet_count": 0, "in_reply_to_user_id": vacio, "favorited": false, "retweeted_status":... }]

Now, i want to transform this result to a new object in format of dict for extract data more easy, such as first result. I tried

json.loads(json_string)

but i have error : No JSON object could be decoded

I next would execute next code, for this reason i need access by dict format:

usuarios = []
texto = []
fechas = []
localizaciones = []

for line in open('con_error.json', 'r'):  
    usuarios.append(json.loads(line)['user']['name']) 
    texto.append(json.loads(line)['text']) 
    fechas.append(json.loads(line)['created_at']) 
    
    localizaciones.append(json.loads(line)['place']['name'])

Thank you very much and sorry if there is some mystake in my post. It is my first post!

RE: Replace null values in Json file - Grok_It - Jun-10-2018

retracted

RE: Replace null values in Json file - buran - Jun-10-2018

1. the way you read the json is not right

with open("con_error.json", "r") as f:
    data = json.load(f)

now you have your json as data dict

2. if use str.replace to replace "null"(?!). what you do is to read json, then immediately dump it back as string. So then why read it as json at all? just read the file as text and replace "null" if you are going to do this

3. once you read the json properly as a dict you can loop over it and replace None. However it looks like complex data structure, so I cannot be more specific

RE: Replace null values in Json file - volcano63 - Jun-10-2018

null value is not enclosed in double quotes - but string values are. You have essentially corrupted your JSON file by inserting unquoted string.

Your replace line should have been

json_string=json.dumps(data).replace("null", '"vacio"')

RE: Replace null values in Json file - burnsssss - Jun-11-2018

(Jun-10-2018, 06:50 PM)buran Wrote: 1. the way you read the json is not right
with open("con_error.json", "r") as f:
    data = json.load(f)
now you have your json as data dict

2. if use str.replace to replace "null"(?!). what you do is to read json, then immediately dump it back as string. So then why read it as json at all? just read the file as text and replace "null" if you are going to do this

3. once you read the json properly as a dict you can loop over it and replace None. However it looks like complex data structure, so I cannot be more specific

Thank you very much to all for answer my question. I am trying to realize for loop in a dict object , but some values are replace and other no. The code is:

for i, j in data.iteritems():
    if j is None:
        data[i] = 'vacio'

i dont understand why no replace all values

Thank you very much!

RE: Replace null values in Json file - buran - Jun-11-2018

As I said - you have complex structure, so you need to check if the value is (for example) list/dict, then you need to check values within this data structure too.
In your example you have

Output:u'user_mentions': [{u'id': 254599256,
u'id_str': u'254599256',
u'indices': [3, 11],
u'name': u'Alejandro Fargosi',
u'screen_name': u'fargosi'}]},
u'favorite_count': 0,
u'favorited': False,
u'filter_level': u'low',
u'geo': None,
u'id': 1005817329459097600L,
u'id_str': u'1005817329459097600',
u'in_reply_to_screen_name': None,
u'in_reply_to_status_id': None}]

the value is list of dicts, and there are elements in the dict that has value None.
so it is at least two levels deep that you have to check

what is the best way to do it depends on your data and you know your data best

can you post example of your json file so that we can get better understanding of the data structure that you work with?

RE: Replace null values in Json file - volcano63 - Jun-11-2018

My advice - above- with string replacement would have worked too, but this is a proper way.

You cannot change nested structures without diving in with recursion

def replace_nulls(json_elem):
    if isinstance(json_elem, list):
        return [replace_nulls(elem) for elem in json_elem]
    elif isinstance(json_elem, dict):
        return {key: replace_nulls(value) for key, value in json_elem.items()}
    else:
        return 'vacio' if json_elem is None else json_elem

Output:[{'contributors': 'vacio',
  'coordinates': 'vacio',
  'created_at': 'Sun Jun 10 14:21:53 +0000 2018',
  'entities': {'hashtags': [],
   'symbols': [],
   'urls': [],
   'user_mentions': [{'id': 254599256,
     'id_str': '254599256',
     'indices': [3, 11],
     'name': 'Alejandro Fargosi',
     'screen_name': 'fargosi'}]},
  'favorite_count': 0,
  'favorited': False,
  'filter_level': 'low',
  'geo': 'vacio',
  'id': 1005817329459097600,
  'id_str': '1005817329459097600',
  'in_reply_to_screen_name': 'vacio',
  'in_reply_to_status_id': 'vacio'}]

This simple code produces the same result (but nobody listens Doh

)

json.loads(json.dumps(data).replace('null', '"vacio"'))

RE: Replace null values in Json file - buran - Jun-11-2018

(Jun-11-2018, 08:18 AM)volcano63 Wrote: This simple code produces the same result (but nobody listens Doh )
Python Code: (Double-click to select all)
1

json.loads(json.dumps(data).replace('null', '"vacio"'))

(Jun-10-2018, 06:50 PM)buran Wrote: what you do is to read json, then immediately dump it back as string. So then why read it as json at all? just read the file as text and replace "null" if you are going to do this

the simplest code wouldn't need to import/use json at all

RE: Replace null values in Json file - burnsssss - Jun-11-2018

(Jun-11-2018, 08:18 AM)volcano63 Wrote: My advice - above- with string replacement would have worked too, but this is a proper way. You cannot change nested structures without diving in with recursion

def replace_nulls(json_elem): if isinstance(json_elem, list): return [replace_nulls(elem) for elem in json_elem] elif isinstance(json_elem, dict): return {key: replace_nulls(value) for key, value in json_elem.items()} else: return 'vacio' if json_elem is None else json_elem

Output:
[{'contributors': 'vacio', 'coordinates': 'vacio', 'created_at': 'Sun Jun 10 14:21:53 +0000 2018', 'entities': {'hashtags': [], 'symbols': [], 'urls': [], 'user_mentions': [{'id': 254599256, 'id_str': '254599256', 'indices': [3, 11], 'name': 'Alejandro Fargosi', 'screen_name': 'fargosi'}]}, 'favorite_count': 0, 'favorited': False, 'filter_level': 'low', 'geo': 'vacio', 'id': 1005817329459097600, 'id_str': '1005817329459097600', 'in_reply_to_screen_name': 'vacio', 'in_reply_to_status_id': 'vacio'}]

Thank you very much!! It works very well. It is more fast and better that transform to string. Thank you very much. I learned more python with all of yours! Sorry for my english

RE: Replace null values in Json file - volcano63 - Jun-11-2018

(Jun-11-2018, 08:44 AM)buran Wrote:
(Jun-11-2018, 08:18 AM)volcano63 Wrote: This simple code produces the same result (but nobody listens Doh )
Python Code: (Double-click to select all)
1

json.loads(json.dumps(data).replace('null', '"vacio"'))

(Jun-10-2018, 06:50 PM)buran Wrote: what you do is to read json, then immediately dump it back as string. So then why read it as json at all? just read the file as text and replace "null" if you are going to do this
the simplest code wouldn't need to import/use json at all

My point was that the approach initially taken by OP would have worked with a slight adjustment. Recursion is the proper way - but solution-like it is simpler with json(not everyone is proficient with recursions - it took me a Scala course to grok it better).