Python Forum

Hey,
i have this list of tuples, i need to arrange the output in json by order.
in nvm i have 3 list of tuple

>>> nvm = [(10, ['cp5'], ['ERROR'], 201905021800.0, 201905021900.0, [[1.2], [1.5]]), (11, ['cp3'], ['ERROR'], 201905021800.0, 201905021900.0, [[1.2], [1.5]]), (12, ['cp7', 'cp4'], ['ERROR'], 201905021800.0, 201905021900.0, [[1.2], [1.5]])]
>>> type(nvm)
<class 'list'>
>>> type(nvm[0])
<class 'tuple'>
>>>

and i want to order it with loop and append because results always changing and next time
it might be 10 lists.
i tried something with len to help me know how many lists i got but i didn't succeed

        #for lists in len(records)
        #    listnum = range(lists)

        records = {

                "event_id": nvm[0][0] ,
                "cp_ids": nvm[0][1] ,
                "event_type": nvm[0][2] ,
                "start_ts": nvm[0][3] ,
                "end_ts": nvm[0][4] ,
                "record": nvm[0][5]

                  }
        return json.dumps(records)

this does work but only for records[0] list, i need it for all the lists
records[0] records[1] records[2] and if i have more i need some loop\ append to help me with it.
please help

Thank you!

Hi,

first of all: your post would be _much_ more readable if you would write words correctly in upper and lower case and especially would use punctuation.

in Python you can iterate directly over list and tuples, as the are iterable. So no need for range or len or anything. Just do for item in iterable:.

I do not really understand what the final output should be. You a list of three tuples and want to get ... what? The example in your 2nd code block doesn't make to much sense... Do you want a dict for each item in the list, which would be three dicts as per your example. Or do you want to get a dict of dicts?
And why to do need a dict at all? Your input, the list of tuples, is already ordered, isn't it?
And did you check the documentation of your data source if this isn't able already to provide a dict instead of a list?

Regards, noisefloor

(Jun-16-2019, 10:05 AM)noisefloor Wrote: [ -> ]Hi,

first of all: your post would be _much_ more readable if you would write words correctly in upper and lower case and especially would use punctuation.

in Python you can iterate directly over list and tuples, as the are iterable. So no need for range or len or anything. Just do for item in iterable:.

I do not really understand what the final output should be. You a list of three tuples and want to get ... what? The example in your 2nd code block doesn't make to much sense... Do you want a dict for each item in the list, which would be three dicts as per your example. Or do you want to get a dict of dicts?
And why to do need a dict at all? Your input, the list of tuples, is already ordered, isn't it?
And did you check the documentation of your data source if this isn't able already to provide a dict instead of a list?

Regards, noisefloor

Hey, i want it to show as json by list, like this :

[(event_id :10, 
cp_ids : 'cp5',
event_type: 'ERROR',
start_ts : 201905021800.0,
end_ts : 201905021900.0,
record: 1.2, 1.5])
},
event_id :11,
cp_ids: 'cp3',
event_type : 'ERROR',
start_ts: 201905021800.0,
end_ts: 201905021900.0,
record: 1.2, 1.5
},
event_id:12,
cp_ids:'cp7', 'cp4',
event_type:'ERROR',
start_ts: 201905021800.0,
end_ts : 201905021900.0,
record: 1.2, 1.5
}

Even if i do the for i in, it does work but only for one list. the question is how do i make it
go throw all list ( if there is 3 lists like here or even 20 lists)

This is my code :

records = {
 
        "event_id": nvm[0][0] ,
        "cp_ids": nvm[0][1] ,
        "event_type": nvm[0][2] ,
        "start_ts": nvm[0][3] ,
        "end_ts": nvm[0][4] ,
        "record": nvm[0][5]
 
          }

How can i make the nvm[0][0] = nvm[n][0] to know to iterate on all list ?
Im new to python

You're thinking to complicated.

Your structure is following:

EVENT_FRAME = [int, str, str, int, int, List[float]]
The last element is already a nested list, unpacking nested lists is easy with itertools.chain.fromiterable

Example code:

Hide/Show

import sys
import json
import itertools
import pathlib


def records_to_dict_transformation(data):
    """
    Generator to convert the nested list into dicts
    It skips the row, if there are too much or less columns.
    If a conversion fails, he would take the original values.
    This is interesting, if your input data comes from a csv as strings.
    """
    for row_idx, row in enumerate(data):
        try:
            event_id, cp_ids, event_type, start_ts, end_ts, record = row
        except ValueError:
            print(
                f"Inconsistent formatting of row {row_idx}. Too much or less columns",
                file=sys.stderr,
            )
            # printing errors to stderr
            # only reusable data should written to stdout
            continue
        try:
            event_id = int(event_id)
            # unix timestamps should always be integers
            # I guess the data comes from JavaScript
            # otherwise you will hit a some point problems with caching
            # if you use floating-point-timestamps
            start_ts = int(start_ts)
            end_ts = int(end_ts)
        except ValueError:
            print("Could not convert to integers. Skipping conversion", file=sys.stderr)
        # now flatten the record with itertools.chain.fromiterable
        record = list(itertools.chain.from_iterable(record))
        try:
            # trying to convert records to float
            record = list(map(float, record))
        except ValueError:
            print("Illegal value in record detected.", file=sys.stderr)
        yield {
            "event_id": int(event_id),
            "cp_ids": cp_ids,
            "event_type": event_type,
            "start_ts": start_ts,
            "end_ts": end_ts,
            "record": record,
        }

def records_to_dict(data):
    """
    Generator to convert the nested list into dicts
    Only rows are skipped, if the number or columns is too much or less.
    No conversion at all, because the sample data is already a
    Python data structure.
    """
    for row_idx, row in enumerate(data, 1):
        try:
            event_id, cp_ids, event_type, start_ts, end_ts, record = row
        except ValueError:
            print(
                f"Inconsistent formatting of row {row_idx}. Too much or less columns",
                file=sys.stderr,
            )
            continue
        yield {
            "event_id": int(event_id),
            "cp_ids": cp_ids,
            "event_type": event_type,
            "start_ts": start_ts,
            "end_ts": end_ts,
            "record": record,
        }


# now we put the results of dicts in a list
# the function records_to_dict is a generator, when called
# then you have to consume the generator with a for-loop of a built-in function, which takes iterables


if __name__ == "__main__":
    # if no argument is supplied, used sdtin as input for data
    if len(sys.argv) == 1 and not sys.stdin.isatty():
        print("Using stdin", file=sys.stderr)
        input_data = eval(sys.stdin.read()) # eval is EVIL
    # if one argument is supplied, use this as a path to a filen
    elif len(sys.argv) == 2 and sys.stdin.isatty():
        print("Using file", sys.argv[1], file=sys.stderr)
        # pathlib makes it a bit easier to read some text from a file
        # internally a context manager is used, so the file descriptor
        # is closed automatic
        input_data = pathlib.Path(sys.argv[1]).read_text()
        # eval allows to evaluate python literals
        # normally this should not used, I use it just for demonstration
        input_data = eval(input_data)
    else:
        print("Using example data", file=sys.stderr)
        input_data = nvm = [
            (10, ["cp5"], ["ERROR"], 201905021800.0, 201905021900.0, [[1.2], [1.5]]),
            (11, ["cp3"], ["ERROR"], 201905021800.0, 201905021900.0, [[1.2], [1.5]]),
            (
                12,
                ["cp7", "cp4"],
                ["ERROR"],
                201905021800.0,
                201905021900.0,
                [[1.2], [1.5]],
            ),
        ]
    result_as_list = list(records_to_dict(input_data))
    result_as_json = json.dumps(result_as_list)
    print(result_as_json)

If you're working on linux:

# save data in a file, to test the reading
echo "[(10, ['cp5'], ['ERROR'], 201905021800.0, 201905021900.0, [[1.2], [1.5]]), (11, ['cp3'], ['ERROR'], 201905021800.0, 201905021900.0, [[1.2], [1.5]]), (12, ['cp7', 'cp4'], ['ERROR'], 201905021800.0, 201905021900.0, [[1.2], [1.5]])]" > nvm.txt

# now use the script to read and evaluate the data in the file

python nvm.py nvm.txt

Output:

Hide/Show

Output:Using file nvm.txt
[{"event_id": 10, "cp_ids": ["cp5"], "event_type": ["ERROR"], "start_ts": 201905021800, "end_ts": 201905021900, "record": [1.2, 1.5]}, {"event_id": 11, "cp_ids": ["cp3"], "event_type": ["ERROR"], "start_ts": 201905021800, "end_ts": 201905021900, "record": [1.2, 1.5]}, {"event_id": 12, "cp_ids": ["cp7", "cp4"], "event_type": ["ERROR"], "start_ts": 201905021800, "end_ts": 201905021900, "record": [1.2, 1.5]}]

To test, if the format is ok, you can use json.tool

python nvm.py nvm.txt | python -m json.tool

Hide/Show

Output:[
    {
        "event_id": 10,
        "cp_ids": [
            "cp5"
        ],
        "event_type": [
            "ERROR"
        ],
        "start_ts": 201905021800,
        "end_ts": 201905021900,
        "record": [
            1.2,
            1.5
        ]
    },
    {
        "event_id": 11,
        "cp_ids": [
            "cp3"
        ],
        "event_type": [
            "ERROR"
        ],
        "start_ts": 201905021800,
        "end_ts": 201905021900,
        "record": [
            1.2,
            1.5
        ]
    },
    {
        "event_id": 12,
        "cp_ids": [
            "cp7",
            "cp4"
        ],
        "event_type": [
            "ERROR"
        ],
        "start_ts": 201905021800,
        "end_ts": 201905021900,
        "record": [
            1.2,
            1.5
        ]
    }
]

You can see a big difference in indentation. This is because the tool uses by default indent=4
You can use this also with json.dumps(your_object, indent=4)

What you need to investigate is:

for-loops, iteration over nested datatypes:

data = [(1,2),(3,4),(5,6)]
for row in data:
    print(row)
    # to see what happens

Argument unpacking:

data = [(1,2),(3,4),(5,6)]
one, two, three = data
# is very strict, wrong number of elements -> ValueError

Argument unpacking with *:

data = [(1,2),(3,4),(5,6)]
first_element1, *rest = data
*elements, last_element1 = data
first_element2, *between, last_element2 = data

Everything with a star in front, is a list and consumes as much as it can do without an Exception.

Sometimes you have the bad situation of nested results, but if you need each of them in one variable, you can use nested argument unpacking:
```
silly_sequence = ['ok', (200, 'GET'), 'foo']
# now we want to unpack all of them
ok, (status, get), foo = silly_sequence
```

itertools.chain.fromiterable is used to flatten an iteable:

data = [[1,2,3], [4,5,6], [7,8,9],[10,11,12]]
flatten = list(itertools.chain.fromiterable(data)

(nested) (star) argument unpacking works also in a for-loop:

positions = [(1,4,2), (3,2,1), (4,5,3)]
for x, y, z in positions:
    print(x, y, z)

# if you have nested iterables and know you have minimum one element or more, you can use
# the star unpacking
minium_length_1 = [(1,2,3,4,5,6), (16,1), [1,1.3], [1]]
for first, *rest in minium_length_1:
    print(first, rest)
# this will fail, if there is a non-iterable element or an empty sequence

Generators are good, if you have a process chain. For example parsing data line by line,
next generator parses the line and so on. With generators you can also process memory efficient infinite sequences.
Introduction to Python Generators

Hi,

did you work through the official Python tutorial at docs.python.org yet? That's always a good idea, as it is perfect to learn all the basics. And iterating is one of Python's basics, as iteration is used a lot in Python - and iteration is something Python is really powerful at.

Anyway, blueprint for you:

records = []
for item in nvm:
   record = {}
   record['event_id'] = item[0]
   #and so on
   records.append(record)

Regards, noisefloor

Thank you all for the fast comments, i ended up with this code block :

        res = []
        for n in nvm:
            res.append({"event_id": n[0], "cp_ids": n[1], "event_type": n[2], "start_ts": n[3], "end_ts": n[4], "record": n[5]})
        return json.dumps(res)

and it does show all the lists
Thanks!

You could use zip to create the dictionary

import json

nvm = [(10, ['cp5'], ['ERROR'], 201905021800.0,
        201905021900.0, [[1.2], [1.5]]),
       (11, ['cp3'], ['ERROR'], 201905021800.0,
        201905021900.0, [[1.2], [1.5]]),
       (12, ['cp7', 'cp4'], ['ERROR'], 201905021800.0,
        201905021900.0, [[1.2], [1.5]])
       ]
keys = ("event_id", "cp_ids", "event_type", "start_ts", "end_ts", "record")

print(json.dumps([{key: value for key, value in zip(keys, value)}
                  for value in nvm]))

Output:[{"event_id": 10, "cp_ids": ["cp5"], "event_type": ["ERROR"], "start_ts": 201905021800.0, "end_ts": 201905021900.0, "record": [[1.2], [1.5]]}, {"event_id": 11, "cp_ids": ["cp3"],
"event_type": ["ERROR"], "start_ts": 201905021800.0, "end_ts": 201905021900.0, "record": [[1.2], [1.5]]}, {"event_id": 12, "cp_ids": ["cp7", "cp4"], "event_type": ["ERROR"], "start_ts": 201905021800.0, "end_ts": 201905021900.0, "record": [[1.2], [1.5]]}]

if you needs know how many lists

import json
nvm = [(10, ['cp5'], ['ERROR'], 201905021800.0, 201905021900.0, [[1.2], [1.5]]), (11, ['cp3'], ['ERROR'], 201905021800.0, 201905021900.0, [[1.2], [1.5]]), (12, ['cp7', 'cp4'], ['ERROR'], 201905021800.0, 201905021900.0, [[1.2], [1.5]])]
res = []

for k,v in enumerate(nvm):
    # res.append({"event_id": n[0], "cp_ids": n[1], "event_type": n[2], "start_ts": n[3], "end_ts": n[4], "record": n[5]})
    print(k,'->',v)
    # print(json.dumps(res))

batchenr

noisefloor

batchenr

DeaD_EyE

noisefloor

batchenr

Yoriz

Abdullah