Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Merge JSON files prioritizing the updated values from most recent file
#1
Hi folks!

I have series of JSON files similar to example below. "update" in "header" is the file creation timestamp. "hours" are timestamps for the values "x" and "y".

I would like to combine them in a single CSV file to import into Excel. The problem is most of the files contain updated "x" and "y" values in comparison to preceding file, e.g. first two timestamps are same as the last two timestamps from previous file, but "x" and "y" values were updated. So for the same timestamps there are updated more accurate "x" and "y" values.

With my limited knowledge I have tried to write a script which is ignoring older values from older file, comparing the "update" timestamp. It works only when I ignore "x" and "y" values and it plots series of hours.

Without further investigation why it does not work properly, I would like to ask to guide me to choose the right approach. I am sure there are more convenient ways doing it.

Thanks!


{
"header":{
"update":1555054504000
},
"data":{
"hours":[
1555038000000,
1555048800000,
1555059600000,
1555070400000,
1555081200000,
1555092000000,
1555102800000,
1555113600000
],
"x":[
241.095130810609,
235.6587698951538,
234.52988957999375,
238.14886341887896,
240.9792842156129,
234.37616327308106,
236.4281670519553,
239.34914914407685
],
"y":[
273.9192290114759,
271.7583893617311,
270.7841492576362,
277.412376380971,
279.51083939292204,
280.7639255517393,
277.92215250624633,
272.7410417669065
]
}
}

#! /usr/bin/python3
import os, json

path_to_json = 'data/'
json_files = [pos_json for pos_json in os.listdir(path_to_json) if pos_json.endswith('.json')]

files = list(enumerate(json_files))

def appendJsonContent(fileNumber):
    information = []
    fileName = files[fileNumber][1]
    with open(os.path.join(path_to_json, fileName)) as jsonFile:
        json_text = json.load(jsonFile)
        for e, hours in enumerate(json_text['data']['hours']):
            x = json_text['data']['x'][e]
            y = json_text['data']['x'][e]
            information.append(str(hours) + " | " + str(x) + " | " + str(y))
    return information

# return the number of values which is different from the second list
def uniqueDataNumber(i):
    firstSetData = appendJsonContent(i)
    secondSetData = appendJsonContent(i+1)
    mergedSetData = list(set(firstSetData + secondSetData))
    return len(mergedSetData)-len(firstSetData)

for index in range(0, len(files), 2):
    for e in range(uniqueDataNumber(0)):
        print(appendJsonContent(index)[e])
    for e in range(uniqueDataNumber(1)):
        print(appendJsonContent(index+1)[e])
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Help batch converting .json chosen file to MySQL BrandonKastning 2 149 Mar-14-2020, 09:19 PM
Last Post: BrandonKastning
  Pandas merge csv files karlito 2 277 Dec-16-2019, 10:59 AM
Last Post: karlito
  save my sensor data from the bme680 into a json or csv file Plastefuchs84 1 472 Aug-23-2019, 03:04 AM
Last Post: Plastefuchs84
  More recent package/library for difflib? twinpiques 2 479 Jul-12-2019, 01:34 AM
Last Post: twinpiques
  JSON file Loading issue punna111 2 1,751 Aug-27-2018, 08:35 PM
Last Post: rootVIII
  reading, modifying and writing json file metalray 2 6,980 Jun-06-2018, 03:09 PM
Last Post: metalray
  Selecting special values in txt file Scientifix 1 687 May-01-2018, 04:18 PM
Last Post: killerrex
  Reading json file as pandas data frame? Alberto 1 5,654 Feb-05-2018, 12:43 AM
Last Post: snippsat
  matplotlib help with reading values from CSV file cps13 6 3,171 Aug-22-2017, 10:21 AM
Last Post: andre
  Automatically updated graph in jupyter notebook using qt5 capponero 0 2,024 May-16-2017, 03:08 PM
Last Post: capponero

Forum Jump:


Users browsing this thread: 1 Guest(s)