Python Forum
Reading json file as pandas data frame?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Reading json file as pandas data frame?
#1
Dear Python Users,

I am using python 3.6 and trying to download json file (350 MB) as pandas dataframe using the code below. However, I get the following error:

Error:
data_json_str = "[" + ",".join(data) + "] "TypeError: sequence item 0: expected str instance, bytes found
How can I fix the error?

import pandas as pd

# read the entire file into a python array
with open('C:/Users/Alberto/nutrients.json', 'rb') as f:
   data = f.readlines()

# remove the trailing "\n" from each line
data = map(lambda x: x.rstrip(), data)

# each element of 'data' is an individual JSON object.
# i want to convert it into an *array* of JSON objects
# which, in and of itself, is one large JSON object
# basically... add square brackets to the beginning
# and end, and have all the individual business JSON objects
# separated by a comma
data_json_str = "[" + ",".join(data) + "]"

# now, load it into pandas
data_df = pd.read_json(data_json_str)
Reply
#2
data_json_str is bytes pd.read_json need string.
>>> data_json_str = b'hello'
>>> type(data_json_str)
<class 'bytes'>

>>> s = data_json_str.decode()
>>> s
'hello'
>>> type(s)
<class 'str'>
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Grouping in pandas/multi-index data frame Aleqsie 3 607 Jan-06-2024, 03:55 PM
Last Post: deanhystad
  Filtering Data Frame, with another value NewBiee 9 1,329 Aug-21-2023, 10:53 AM
Last Post: NewBiee
  Reading large crapy text file in anaconda to profile data syamatunuguntla 0 811 Nov-18-2022, 06:15 PM
Last Post: syamatunuguntla
Smile How to further boost the data read write speed using pandas tjk9501 1 1,228 Nov-14-2022, 01:46 PM
Last Post: jefsummers
Thumbs Up can't access data from URL in pandas/jupyter notebook aaanoushka 1 1,830 Feb-13-2022, 01:19 PM
Last Post: jefsummers
Question Sorting data with pandas TheZaind 4 2,295 Nov-22-2021, 07:33 PM
Last Post: aserian
  Exporting data frame to excel dyerlee91 0 1,604 Oct-05-2021, 11:34 AM
Last Post: dyerlee91
  Pandas Data frame column condition check based on length of the value aditi06 1 2,655 Jul-28-2021, 11:08 AM
Last Post: jefsummers
  How to import an xml file to Pandas sjhazard 0 2,324 Jun-08-2021, 08:19 PM
Last Post: sjhazard
  Adding a new column to a Panda Data Frame rsherry8 2 2,083 Jun-06-2021, 06:49 PM
Last Post: jefsummers

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020