Compile list of dictianories out of another list of dictianories by certain keys

Compile list of dictianories out of another list of dictianories by certain keys - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: Compile list of dictianories out of another list of dictianories by certain keys (/thread-33874.html)

Pages: 1 2

Compile list of dictianories out of another list of dictianories by certain keys - CatorCanulis - Jun-05-2021

Hi everyone,

I need to compile a new list of new dictianories out of another list of dictianories. The new list should contain dictianories from the old one, reduced to pre determined keys if the match regarding another predetermined list of keys and values.

It can be assumed that all dictianories have exactly the same keys. The list cointains of thousands of individual dictianories.

What I have in mind is something like:

lod = [{"id": 1, "a": 1, "b": 2, "result": 9.82}, {"id": 2, "a": 1, "b": 2, "result": -5}, {"id": 3, "a": 1, "b": 5, "result": 7.98}]

newList = selectionFunction(lod, ["a", "b"], [1, 2], ["id", "result"])
# Return should be: [{"id": 1, "result": 9.82}, {"id": 2, "result": -5}]

How can I code this efficiently? I have a function but it takes forever.

Thanks in advance
Markus

RE: Compile list of dictianories out of another list of dictianories by certain keys - buran - Jun-05-2021

show the code for selectionFunction

RE: Compile list of dictianories out of another list of dictianories by certain keys - CatorCanulis - Jun-05-2021

def selectFromDictianories(
    dictianories
  , compareKeys
  , compareValues
  , returnKeys
  , booleanPop = False
  , booleanVerbose = True
):
  dictianoriesToReturn = []
  for intDic, currentDictianory in enumerate(dictianories):
    if booleanVerbose:
      print(
          "Durchsuche:", dictianories, "\n"
        , "Vergleiche Schlüssel:", ", ".join(compareKeys)
        , "von:", currentDictianory, "Soll:", compareValues
      )
    try:
      listComparison = [
        currentDictianory[key] for key in compareKeys
      ]
    except KeyError as errorKeyMissing:
      if booleanVerbose:
        print(
            "Warnung! Diese Funktion nimmt an, dass alle Schlüssel"
          , compareKeys
          , "im zu durchsuchenden Wörterbuch"
          , currentDictianory
          , "tatsächlich vorkommt, was nicht der Fall ist.\n"
          , "Es fehlt:", errorKeyMissing
        )
      listComparison = []
    if booleanVerbose:
      print("Vergleiche:", listComparison, "mit:", compareValues)
    # TODO: Kann es passieren, dass
    # ENTEWEDER listComparison und compareValues zufällig gleich sind
    # ODER sie prinzipiell gleich sind, aber die Reihenfolge nicht stimmt.
    if listComparison != [] and listComparison == compareValues:
      print("Treffer für:", currentDictianory)
      dictianoryToReturn = {}
      for k in returnKeys:
        dictianoryToReturn[k] = currentDictianory[k]
      dictianoriesToReturn.append(dictianoryToReturn)
      if booleanPop:
        dictianories.pop(intDic)
  return dictianoriesToReturn

RE: Compile list of dictianories out of another list of dictianories by certain keys - perfringo - Jun-05-2021

Maybe I am missing something but it seems that simple filtering functions would do. Something like:

lod = [{"id": 1, "a": 1, "b": 2, "result": 9.82},
       {"id": 2, "a": 1, "b": 2, "result": -5},
       {"id": 3, "a": 1, "b": 5, "result": 7.98}]

def filter_dicts(data, *args, **kwargs):
    for row in data:
        for k, v in kwargs.items():
            if row[k] != v:
                break
        else:
            yield {key: row[key] for key in args}

print(*filter_dicts(lod, 'id', 'result', a=1, b=2))

# will print -> {'id': 1, 'result': 9.82} {'id': 2, 'result': -5}

RE: Compile list of dictianories out of another list of dictianories by certain keys - CatorCanulis - Jun-06-2021

This seems to do the trick, thank you!

Can you elaborate in what the * does in

*filter_dicts

please?

(Jun-05-2021, 06:28 PM)perfringo Wrote: Maybe I am missing something but it seems that simple filtering functions would do. Something like:

lod = [{"id": 1, "a": 1, "b": 2, "result": 9.82},
       {"id": 2, "a": 1, "b": 2, "result": -5},
       {"id": 3, "a": 1, "b": 5, "result": 7.98}]

def filter_dicts(data, *args, **kwargs):
    for row in data:
        for k, v in kwargs.items():
            if row[k] != v:
                break
        else:
            yield {key: row[key] for key in args}

print(*filter_dicts(lod, 'id', 'result', a=1, b=2))

# will print -> {'id': 1, 'result': 9.82} {'id': 2, 'result': -5}

RE: Compile list of dictianories out of another list of dictianories by certain keys - CatorCanulis - Jun-06-2021

I have another question: How can I "use" the result, that filter_dicts yields? At the end I need a new list of dictianories like the ones printed.

(Jun-05-2021, 06:28 PM)perfringo Wrote: Maybe I am missing something but it seems that simple filtering functions would do. Something like:

lod = [{"id": 1, "a": 1, "b": 2, "result": 9.82},
       {"id": 2, "a": 1, "b": 2, "result": -5},
       {"id": 3, "a": 1, "b": 5, "result": 7.98}]

def filter_dicts(data, *args, **kwargs):
    for row in data:
        for k, v in kwargs.items():
            if row[k] != v:
                break
        else:
            yield {key: row[key] for key in args}

print(*filter_dicts(lod, 'id', 'result', a=1, b=2))

# will print -> {'id': 1, 'result': 9.82} {'id': 2, 'result': -5}

RE: Compile list of dictianories out of another list of dictianories by certain keys - Yoriz - Jun-06-2021

print(*filter_dicts(lod, 'id', 'result', a=1, b=2))

https://docs.python.org/3/tutorial/controlflow.html#unpacking-argument-lists Wrote:Unpacking Argument Lists
The reverse situation occurs when the arguments are already in a list or tuple but need to be unpacked for a function call requiring separate positional arguments. For instance, the built-in range() function expects separate start and stop arguments. If they are not available separately, write the function call with the *-operator to unpack the arguments out of a list or tuple:
>>> list(range(3, 6))            # normal call with separate arguments
[3, 4, 5]
>>> args = [3, 6]
>>> list(range(*args))            # call with arguments unpacked from a list
[3, 4, 5]
In the same fashion, dictionaries can deliver keyword arguments with the **-operator:
>>> def parrot(voltage, state='a stiff', action='voom'):
...     print("-- This parrot wouldn't", action, end=' ')
...     print("if you put", voltage, "volts through it.", end=' ')
...     print("E's", state, "!")
...
>>> d = {"voltage": "four million", "state": "bleedin' demised", "action": "VOOM"}
>>> parrot(**d)
-- This parrot wouldn't VOOM if you put four million volts through it. E's bleedin' demised !

If you want to turn a generator function result into a list, use list

result = filter_dicts(lod, 'id', 'result', a=1, b=2)
result_list = list(result)
print(result_list)

Output:
[{'id': 1, 'result': 9.82}, {'id': 2, 'result': -5}]

RE: Compile list of dictianories out of another list of dictianories by certain keys - Caprone - Jun-06-2021

(Jun-05-2021, 06:28 PM)perfringo Wrote: Maybe I am missing something but it seems that simple filtering functions would do. Something like:

lod = [{"id": 1, "a": 1, "b": 2, "result": 9.82},
       {"id": 2, "a": 1, "b": 2, "result": -5},
       {"id": 3, "a": 1, "b": 5, "result": 7.98}]

def filter_dicts(data, *args, **kwargs):
    for row in data:
        for k, v in kwargs.items():
            if row[k] != v:
                break
        else:
            yield {key: row[key] for key in args}

print(*filter_dicts(lod, 'id', 'result', a=1, b=2))

# will print -> {'id': 1, 'result': 9.82} {'id': 2, 'result': -5}

this is a good solution but not a pythonic solution;
You should always try to avoid nested loop where possible and take advantage of list/dict comprehension;
and more --> try to simplify the problem -->
in this case You only need to know if kwargs' dict is a subset of the 'row' dict (as constraint made by OT) , no checking loop needed:

def filter_dicts(data, *args, **kwargs):
   return  ({key: row[key] for key in args}  for row in data  if row.items() >= kwargs.items())

RE: Compile list of dictianories out of another list of dictianories by certain keys - CatorCanulis - Jun-06-2021

Unbelievable… My original function cut down to two lines. I think perfringos solution is a little easier to read since the behavious of .items() is not as well understood by beginners like me. But this solution is of course very clean.

One little caveat though: I think

if row[k] != v:

would raise KeyError if k doesn't exist. With the shorter way I just get an empty result without warning.

RE: Compile list of dictianories out of another list of dictianories by certain keys - Pedroski55 - Jun-07-2021

Quote:It can be assumed that all dictianories have exactly the same keys. The list cointains of thousands of individual dictianories.

I am not sure what your condition for selection is. I assumed 'a' = 1 and 'b' = 2

I tried this just for fun.

Not sure how to dynamically assign the condition. Then you could quickly change the output myFun1(lexi, condition)

lexis = [{"id": 1, "a": 1, "b": 2, "result": 9.82},
            {"id": 2, "a": 1, "b": 2, "result": -5},
            {"id": 3, "a": 1, "b": 5, "result": 7.98},
            {"id": 4, "a": 1, "b": 2, "result": 11.1},
            {"id": 5, "a": 1, "b": 2, "result": -4},
            {"id": 6, "a": 1, "b": 5, "result": 6.33},
            {"id": 7, "a": 1, "b": 2, "result": 8.88},
            {"id": 8, "a": 1, "b": 2, "result": -2},
            {"id": 9, "a": 1, "b": 5, "result": 5.44}]

results_list = []

def myFun1(lexi):
    # have a look at the data
    for key, value in lexi.items(): 
        print(f'key is: {key}, value is: {value}')
    ergebnis = {}       
    # a condition
    if lexi['a'] == 1 and lexi['b'] == 2:
        ergebnis['id'] = lexi['id']
        ergebnis['result'] = lexi['result']
        results_list.append(ergebnis)
    
for lexi in lexis:
    myFun1(lexi)