Python Forum
Compile list of dictianories out of another list of dictianories by certain keys
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Compile list of dictianories out of another list of dictianories by certain keys
#1
Hi everyone,

I need to compile a new list of new dictianories out of another list of dictianories. The new list should contain dictianories from the old one, reduced to pre determined keys if the match regarding another predetermined list of keys and values.

It can be assumed that all dictianories have exactly the same keys. The list cointains of thousands of individual dictianories.

What I have in mind is something like:
lod = [{"id": 1, "a": 1, "b": 2, "result": 9.82}, {"id": 2, "a": 1, "b": 2, "result": -5}, {"id": 3, "a": 1, "b": 5, "result": 7.98}]

newList = selectionFunction(lod, ["a", "b"], [1, 2], ["id", "result"])
# Return should be: [{"id": 1, "result": 9.82}, {"id": 2, "result": -5}]
How can I code this efficiently? I have a function but it takes forever.

Thanks in advance
Markus
Reply
#2
show the code for selectionFunction
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#3
def selectFromDictianories(
    dictianories
  , compareKeys
  , compareValues
  , returnKeys
  , booleanPop = False
  , booleanVerbose = True
):
  dictianoriesToReturn = []
  for intDic, currentDictianory in enumerate(dictianories):
    if booleanVerbose:
      print(
          "Durchsuche:", dictianories, "\n"
        , "Vergleiche Schlüssel:", ", ".join(compareKeys)
        , "von:", currentDictianory, "Soll:", compareValues
      )
    try:
      listComparison = [
        currentDictianory[key] for key in compareKeys
      ]
    except KeyError as errorKeyMissing:
      if booleanVerbose:
        print(
            "Warnung! Diese Funktion nimmt an, dass alle Schlüssel"
          , compareKeys
          , "im zu durchsuchenden Wörterbuch"
          , currentDictianory
          , "tatsächlich vorkommt, was nicht der Fall ist.\n"
          , "Es fehlt:", errorKeyMissing
        )
      listComparison = []
    if booleanVerbose:
      print("Vergleiche:", listComparison, "mit:", compareValues)
    # TODO: Kann es passieren, dass
    # ENTEWEDER listComparison und compareValues zufällig gleich sind
    # ODER sie prinzipiell gleich sind, aber die Reihenfolge nicht stimmt.
    if listComparison != [] and listComparison == compareValues:
      print("Treffer für:", currentDictianory)
      dictianoryToReturn = {}
      for k in returnKeys:
        dictianoryToReturn[k] = currentDictianory[k]
      dictianoriesToReturn.append(dictianoryToReturn)
      if booleanPop:
        dictianories.pop(intDic)
  return dictianoriesToReturn
Reply
#4
Maybe I am missing something but it seems that simple filtering functions would do. Something like:

lod = [{"id": 1, "a": 1, "b": 2, "result": 9.82},
       {"id": 2, "a": 1, "b": 2, "result": -5},
       {"id": 3, "a": 1, "b": 5, "result": 7.98}]

def filter_dicts(data, *args, **kwargs):
    for row in data:
        for k, v in kwargs.items():
            if row[k] != v:
                break
        else:
            yield {key: row[key] for key in args}

print(*filter_dicts(lod, 'id', 'result', a=1, b=2))

# will print -> {'id': 1, 'result': 9.82} {'id': 2, 'result': -5}
CatorCanulis likes this post
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Reply
#5
This seems to do the trick, thank you!

Can you elaborate in what the * does in
*filter_dicts
please?

(Jun-05-2021, 06:28 PM)perfringo Wrote: Maybe I am missing something but it seems that simple filtering functions would do. Something like:

lod = [{"id": 1, "a": 1, "b": 2, "result": 9.82},
       {"id": 2, "a": 1, "b": 2, "result": -5},
       {"id": 3, "a": 1, "b": 5, "result": 7.98}]

def filter_dicts(data, *args, **kwargs):
    for row in data:
        for k, v in kwargs.items():
            if row[k] != v:
                break
        else:
            yield {key: row[key] for key in args}

print(*filter_dicts(lod, 'id', 'result', a=1, b=2))

# will print -> {'id': 1, 'result': 9.82} {'id': 2, 'result': -5}
Reply
#6
I have another question: How can I "use" the result, that filter_dicts yields? At the end I need a new list of dictianories like the ones printed.

(Jun-05-2021, 06:28 PM)perfringo Wrote: Maybe I am missing something but it seems that simple filtering functions would do. Something like:

lod = [{"id": 1, "a": 1, "b": 2, "result": 9.82},
       {"id": 2, "a": 1, "b": 2, "result": -5},
       {"id": 3, "a": 1, "b": 5, "result": 7.98}]

def filter_dicts(data, *args, **kwargs):
    for row in data:
        for k, v in kwargs.items():
            if row[k] != v:
                break
        else:
            yield {key: row[key] for key in args}

print(*filter_dicts(lod, 'id', 'result', a=1, b=2))

# will print -> {'id': 1, 'result': 9.82} {'id': 2, 'result': -5}
Reply
#7
print(*filter_dicts(lod, 'id', 'result', a=1, b=2))
is
https://docs.python.org/3/tutorial/contr...ment-lists Wrote:Unpacking Argument Lists
The reverse situation occurs when the arguments are already in a list or tuple but need to be unpacked for a function call requiring separate positional arguments. For instance, the built-in range() function expects separate start and stop arguments. If they are not available separately, write the function call with the *-operator to unpack the arguments out of a list or tuple:
>>> list(range(3, 6))            # normal call with separate arguments
[3, 4, 5]
>>> args = [3, 6]
>>> list(range(*args))            # call with arguments unpacked from a list
[3, 4, 5]
In the same fashion, dictionaries can deliver keyword arguments with the **-operator:
>>> def parrot(voltage, state='a stiff', action='voom'):
...     print("-- This parrot wouldn't", action, end=' ')
...     print("if you put", voltage, "volts through it.", end=' ')
...     print("E's", state, "!")
...
>>> d = {"voltage": "four million", "state": "bleedin' demised", "action": "VOOM"}
>>> parrot(**d)
-- This parrot wouldn't VOOM if you put four million volts through it. E's bleedin' demised !

If you want to turn a generator function result into a list, use list
result = filter_dicts(lod, 'id', 'result', a=1, b=2)
result_list = list(result)
print(result_list)
Output:
[{'id': 1, 'result': 9.82}, {'id': 2, 'result': -5}]
Reply
#8
(Jun-05-2021, 06:28 PM)perfringo Wrote: Maybe I am missing something but it seems that simple filtering functions would do. Something like:

lod = [{"id": 1, "a": 1, "b": 2, "result": 9.82},
       {"id": 2, "a": 1, "b": 2, "result": -5},
       {"id": 3, "a": 1, "b": 5, "result": 7.98}]

def filter_dicts(data, *args, **kwargs):
    for row in data:
        for k, v in kwargs.items():
            if row[k] != v:
                break
        else:
            yield {key: row[key] for key in args}

print(*filter_dicts(lod, 'id', 'result', a=1, b=2))

# will print -> {'id': 1, 'result': 9.82} {'id': 2, 'result': -5}

this is a good solution but not a pythonic solution;
You should always try to avoid nested loop where possible and take advantage of list/dict comprehension;
and more --> try to simplify the problem -->
in this case You only need to know if kwargs' dict is a subset of the 'row' dict (as constraint made by OT) , no checking loop needed:

def filter_dicts(data, *args, **kwargs):
   return  ({key: row[key] for key in args}  for row in data  if row.items() >= kwargs.items())
Gribouillis and CatorCanulis like this post
Reply
#9
Unbelievable… My original function cut down to two lines. I think perfringos solution is a little easier to read since the behavious of .items() is not as well understood by beginners like me. But this solution is of course very clean.

One little caveat though: I think
if row[k] != v:
would raise KeyError if k doesn't exist. With the shorter way I just get an empty result without warning.
Reply
#10
Quote:It can be assumed that all dictianories have exactly the same keys. The list cointains of thousands of individual dictianories.

I am not sure what your condition for selection is. I assumed 'a' = 1 and 'b' = 2

I tried this just for fun.

Not sure how to dynamically assign the condition. Then you could quickly change the output myFun1(lexi, condition)

lexis = [{"id": 1, "a": 1, "b": 2, "result": 9.82},
            {"id": 2, "a": 1, "b": 2, "result": -5},
            {"id": 3, "a": 1, "b": 5, "result": 7.98},
            {"id": 4, "a": 1, "b": 2, "result": 11.1},
            {"id": 5, "a": 1, "b": 2, "result": -4},
            {"id": 6, "a": 1, "b": 5, "result": 6.33},
            {"id": 7, "a": 1, "b": 2, "result": 8.88},
            {"id": 8, "a": 1, "b": 2, "result": -2},
            {"id": 9, "a": 1, "b": 5, "result": 5.44}]

results_list = []

def myFun1(lexi):
    # have a look at the data
    for key, value in lexi.items(): 
        print(f'key is: {key}, value is: {value}')
    ergebnis = {}       
    # a condition
    if lexi['a'] == 1 and lexi['b'] == 2:
        ergebnis['id'] = lexi['id']
        ergebnis['result'] = lexi['result']
        results_list.append(ergebnis)
    
for lexi in lexis:
    myFun1(lexi)
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Order a list with successive permutations based on another list yvrob 3 614 Mar-19-2021, 08:20 AM
Last Post: supuflounder
  Saving list in a list quest_ 3 741 Mar-10-2021, 09:58 AM
Last Post: quest_
Star Convert Bytearray into List using list() Shlok 2 508 Feb-18-2021, 10:44 AM
Last Post: deanhystad
  Adding List Element if Second part of the List Elements are the Same quest_ 3 624 Nov-25-2020, 04:33 PM
Last Post: bowlofred
  Count number of occurrences of list items in list of tuples t4keheart 1 576 Nov-03-2020, 05:37 AM
Last Post: deanhystad
Question Save list with nested list into CSV SpongeB0B 1 1,137 Oct-12-2020, 07:26 AM
Last Post: bowlofred
  Appending to list of list in For loop nico_mnbl 2 580 Sep-25-2020, 04:09 PM
Last Post: nico_mnbl
  How to make a list of values from a dictionary list? faryad13 2 652 Sep-03-2020, 03:45 PM
Last Post: faryad13
  Append list into list within a for loop rama27 2 787 Jul-21-2020, 04:49 AM
Last Post: deanhystad
  Compile to Windows exe. SpongeB0B 6 1,319 Jul-06-2020, 08:44 AM
Last Post: snippsat

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020