Arrange list of tuple using loop

[DeaD_EyE]

You're thinking to complicated.

Your structure is following:

EVENT_FRAME = [int, str, str, int, int, List[float]]
The last element is already a nested list, unpacking nested lists is easy with itertools.chain.fromiterable


Example code:

Hide/Show

import sys
import json
import itertools
import pathlib


def records_to_dict_transformation(data):
    """
    Generator to convert the nested list into dicts
    It skips the row, if there are too much or less columns.
    If a conversion fails, he would take the original values.
    This is interesting, if your input data comes from a csv as strings.
    """
    for row_idx, row in enumerate(data):
        try:
            event_id, cp_ids, event_type, start_ts, end_ts, record = row
        except ValueError:
            print(
                f"Inconsistent formatting of row {row_idx}. Too much or less columns",
                file=sys.stderr,
            )
            # printing errors to stderr
            # only reusable data should written to stdout
            continue
        try:
            event_id = int(event_id)
            # unix timestamps should always be integers
            # I guess the data comes from JavaScript
            # otherwise you will hit a some point problems with caching
            # if you use floating-point-timestamps
            start_ts = int(start_ts)
            end_ts = int(end_ts)
        except ValueError:
            print("Could not convert to integers. Skipping conversion", file=sys.stderr)
        # now flatten the record with itertools.chain.fromiterable
        record = list(itertools.chain.from_iterable(record))
        try:
            # trying to convert records to float
            record = list(map(float, record))
        except ValueError:
            print("Illegal value in record detected.", file=sys.stderr)
        yield {
            "event_id": int(event_id),
            "cp_ids": cp_ids,
            "event_type": event_type,
            "start_ts": start_ts,
            "end_ts": end_ts,
            "record": record,
        }

def records_to_dict(data):
    """
    Generator to convert the nested list into dicts
    Only rows are skipped, if the number or columns is too much or less.
    No conversion at all, because the sample data is already a
    Python data structure.
    """
    for row_idx, row in enumerate(data, 1):
        try:
            event_id, cp_ids, event_type, start_ts, end_ts, record = row
        except ValueError:
            print(
                f"Inconsistent formatting of row {row_idx}. Too much or less columns",
                file=sys.stderr,
            )
            continue
        yield {
            "event_id": int(event_id),
            "cp_ids": cp_ids,
            "event_type": event_type,
            "start_ts": start_ts,
            "end_ts": end_ts,
            "record": record,
        }


# now we put the results of dicts in a list
# the function records_to_dict is a generator, when called
# then you have to consume the generator with a for-loop of a built-in function, which takes iterables


if __name__ == "__main__":
    # if no argument is supplied, used sdtin as input for data
    if len(sys.argv) == 1 and not sys.stdin.isatty():
        print("Using stdin", file=sys.stderr)
        input_data = eval(sys.stdin.read()) # eval is EVIL
    # if one argument is supplied, use this as a path to a filen
    elif len(sys.argv) == 2 and sys.stdin.isatty():
        print("Using file", sys.argv[1], file=sys.stderr)
        # pathlib makes it a bit easier to read some text from a file
        # internally a context manager is used, so the file descriptor
        # is closed automatic
        input_data = pathlib.Path(sys.argv[1]).read_text()
        # eval allows to evaluate python literals
        # normally this should not used, I use it just for demonstration
        input_data = eval(input_data)
    else:
        print("Using example data", file=sys.stderr)
        input_data = nvm = [
            (10, ["cp5"], ["ERROR"], 201905021800.0, 201905021900.0, [[1.2], [1.5]]),
            (11, ["cp3"], ["ERROR"], 201905021800.0, 201905021900.0, [[1.2], [1.5]]),
            (
                12,
                ["cp7", "cp4"],
                ["ERROR"],
                201905021800.0,
                201905021900.0,
                [[1.2], [1.5]],
            ),
        ]
    result_as_list = list(records_to_dict(input_data))
    result_as_json = json.dumps(result_as_list)
    print(result_as_json)

If you're working on linux:

# save data in a file, to test the reading
echo "[(10, ['cp5'], ['ERROR'], 201905021800.0, 201905021900.0, [[1.2], [1.5]]), (11, ['cp3'], ['ERROR'], 201905021800.0, 201905021900.0, [[1.2], [1.5]]), (12, ['cp7', 'cp4'], ['ERROR'], 201905021800.0, 201905021900.0, [[1.2], [1.5]])]" > nvm.txt

# now use the script to read and evaluate the data in the file

python nvm.py nvm.txt

Output:

Hide/Show

Output:
Using file nvm.txt
[{"event_id": 10, "cp_ids": ["cp5"], "event_type": ["ERROR"], "start_ts": 201905021800, "end_ts": 201905021900, "record": [1.2, 1.5]}, {"event_id": 11, "cp_ids": ["cp3"], "event_type": ["ERROR"], "start_ts": 201905021800, "end_ts": 201905021900, "record": [1.2, 1.5]}, {"event_id": 12, "cp_ids": ["cp7", "cp4"], "event_type": ["ERROR"], "start_ts": 201905021800, "end_ts": 201905021900, "record": [1.2, 1.5]}]

To test, if the format is ok, you can use json.tool

python nvm.py nvm.txt | python -m json.tool

Hide/Show

Output:
[
    {
        "event_id": 10,
        "cp_ids": [
            "cp5"
        ],
        "event_type": [
            "ERROR"
        ],
        "start_ts": 201905021800,
        "end_ts": 201905021900,
        "record": [
            1.2,
            1.5
        ]
    },
    {
        "event_id": 11,
        "cp_ids": [
            "cp3"
        ],
        "event_type": [
            "ERROR"
        ],
        "start_ts": 201905021800,
        "end_ts": 201905021900,
        "record": [
            1.2,
            1.5
        ]
    },
    {
        "event_id": 12,
        "cp_ids": [
            "cp7",
            "cp4"
        ],
        "event_type": [
            "ERROR"
        ],
        "start_ts": 201905021800,
        "end_ts": 201905021900,
        "record": [
            1.2,
            1.5
        ]
    }
]

You can see a big difference in indentation. This is because the tool uses by default indent=4
You can use this also with json.dumps(your_object, indent=4)

What you need to investigate is:

• for-loops, iteration over nested datatypes:
  

  data = [(1,2),(3,4),(5,6)]
  for row in data:
      print(row)
      # to see what happens

• Argument unpacking:
  

  data = [(1,2),(3,4),(5,6)]
  one, two, three = data
  # is very strict, wrong number of elements -> ValueError

• Argument unpacking with *:
  

  data = [(1,2),(3,4),(5,6)]
  first_element1, *rest = data
  *elements, last_element1 = data
  first_element2, *between, last_element2 = data

  Everything with a star in front, is a list and consumes as much as it can do without an Exception.
  
• Sometimes you have the bad situation of nested results, but if you need each of them in one variable, you can use nested argument unpacking:
  

  silly_sequence = ['ok', (200, 'GET'), 'foo']
  # now we want to unpack all of them
  ok, (status, get), foo = silly_sequence

• itertools.chain.fromiterable is used to flatten an iteable:
  

  data = [[1,2,3], [4,5,6], [7,8,9],[10,11,12]]
  flatten = list(itertools.chain.fromiterable(data)

• (nested) (star) argument unpacking works also in a for-loop:
  

  positions = [(1,4,2), (3,2,1), (4,5,3)]
  for x, y, z in positions:
      print(x, y, z)
  
  # if you have nested iterables and know you have minimum one element or more, you can use
  # the star unpacking
  minium_length_1 = [(1,2,3,4,5,6), (16,1), [1,1.3], [1]]
  for first, *rest in minium_length_1:
      print(first, rest)
  # this will fail, if there is a non-iterable element or an empty sequence

• Generators are good, if you have a process chain. For example parsing data line by line,
  next generator parses the line and so on. With generators you can also process memory efficient infinite sequences.
  Introduction to Python Generators