Getting list inside of a list; how to clean it and retrieve value

mrapple2020 · Apr-09-2019, 08:26 PM

When I print output from a dictionary, I get the following output as shown under "NOW LISTING".
to be honest I cannot understand at what point the data got these multiple [[[. It seems a list inside of a list but I do not know why.

Question: based on this, how would you retrieve value thru this data structure? Or should I fix the origin of these behavior? I am pasting code below.
******************* NOW LISTING ***********************
defaultdict(<class 'list'>, {'PSE2': ['0/7/CPU0', '1/0/CPU0', '1/2/CPU0'], 'IF_PSE1': [['TenGigE0/5/0/21.201', 'PSE2=NO', 21, 'TenGigE0/5', 'REPLICATION=NO'], ['TenGigE0/7/0/3.1', 'PSE2=NO', 3, 'TenGigE0/7', 'REPLICATION=NO']], 'IF_PSE2': [['TenGigE0/7/0/3.1', 'PSE2=YES', 3, 'TenGigE0/7', 'REPLICATION=NO']], 'NODE': [[['1013'], ['1974'], ['0/5']], [['1013'], ['993'], ['0/9']], [['9999'], ['198'], ['0/5']], [['9999'], ['992'], ['0/9']]]})

        
          
          
              
              import paramiko
import copy
import os
import sys
import re
import getpass
import time
import json
from collections import defaultdict
 
#User and Password is Cisco CEC
USERNAME = 'admin'
TARGET_SERVER = '10.1.1.1'
TOLERANCE = 400
MGROUP = "224.0.0.0/24"
 
def olist(raw_show_data):
    #Separate the output for the respective show commands executed
    olist_data  = raw_show_data.split('#show')
    #Separate data from 3 show commands given (show mrib route, show mfib route rate, show platform)
    #print(olist_data[1])
    #print(olist_data[2])
    #print(olist_data[3])
    #==>delete this:using bogus data for now
    olist_data[1] = """TenGigE0/5/0/1.999 Flags: Outgoing Interface List  TenGigE0/5/0/21.201 Flags: F NS, Up: 01:06:23 TenGigE0/7/0/3.1  >>> CRS-X interface ( 1st PSE)"""
     
    #1) Analyze 'show platform' and determine if it is 1 PSE or 2 PSE architecture
     
    var_dict = {}
    show_platform_str  = olist_data[3].strip()
    show_platform_list = show_platform_str.split('\n')
    #Create a dictionary to organize all Nodes which are PSE-2 architecture. Dictionary if_dic will contain list of outgoing interfaces(o-list) which are on PSE2 or PSE1
    var_dict = defaultdict(list)
    for element in show_platform_list:
        match =re.findall(r'(FP-X|MSC-X|SPC-X)',element)
 
        if match:
            #The first element on each line is the Node#:
            break_element = element.split(' ')
            #Build a list of all Node# which are PSE2           
            var_dict['PSE2'].append(break_element[0])
 
 
    #2)Analyze the outgoing interfaces on show mrib route
    #Only consider outgoing interface lists (o-list):
    #A dict to organize interface status
    olist_dic = {}
    raw_int_data = olist_data[1].split('Outgoing Interface List')
    raw_int_out  = raw_int_data[1]
     
    #Get list of interfaces
    match_int  = re.findall(r'(TenGig[A-Za-z0-9\/\.]+)', raw_int_out)
    if match_int:
 
       for entry in match_int:
            raw_int_ids = entry.split('/')
            #Get the interface port ID
            raw_portids = raw_int_ids[3].partition(".")
            port_id = raw_portids[0]
             
            #Get the interface  CPU/Node position
            port_cpu_pos = (raw_int_ids[0]+"/"+raw_int_ids[1])
            match = re.findall(r'(\d\/\d)',port_cpu_pos)
             
            #Copy dictinary so that  I can interate and modify the original one.
            temp_var_dict = var_dict.copy()
             
            for k, v in temp_var_dict.items():
                           match = str(match)
                           match = match.strip("[]")
                           match = match.strip("''")
                           result = [index for index, value in enumerate(v) if match in value]
                           result = str(result)
                           result = result.strip("[]")
                           result = result.strip("'")
                           #====> IF I print(var_dict), at this point I have no error *********  
                           if result == "0":
                               #It means a match between interface on  RP PSE2 model was found; Interface position is on PSE2 architecture
                               print (f'PSE-2 Line cards:{v}     Interfaces on PSE2:{entry}   Interface PortID:{port_id}')
                               port_id = int(port_id)
                               print(port_id)
 
                               if port_id >= 19:
                                   #print(f'interface:{entry}  portID={port_id}   CPU_POS={port_cpu_pos}    REPLICATION=YES')
                                   if_info = [entry,'PSE2=YES',port_id,port_cpu_pos,'REPLICATION=YES'] 
                                   var_dict['IF_PSE2'].append(if_info)
                                   #===> *** This is the point that if i attempt to print var_dict, I get the Error during olist(): dictionary changed size during iteration
                               else:
                                   #print(f'interface:{entry},portID={port_id}   CPU_POS={port_cpu_pos}     REPLICATION=NO')
                                   if_info = [entry,'PSE2=YES',port_id,port_cpu_pos,'REPLICATION=NO']
                                   var_dict['IF_PSE2'].append(if_info)
                           else: 
                                #it means the interface is on single PSE. No replication is applicable. Just check threshold between incoming and outgoing rate.
                                if_info = [entry,'PSE2=NO',int(port_id),port_cpu_pos,'REPLICATION=NO']
 
                                var_dict['IF_PSE1'].append(if_info)
    #3) Retrieve Incoming and Outgoing rate
    #==>To be deleted: This is a temporary hardcoded value for testing.
    olist_data[2] = """RP/0/RP0/CPU0:R7-P1#show mfib route rate 230.207.200.1
    IP Multicast Forwarding Rates
    (Source Address, Group Address)
    Incoming rate: pps/bps (Incoming node)
    Outgoing rate:
    Node: (Outgoing node) : pps/bps
    (50.50.50.2,230.207.200.1)
    Incoming rate : 1013 / 8127181 (0/5/CPU0)
     Outgoing rate : 
     Node : 0/5/CPU0 : 1974 / 15828597. >> it shows double as O-list is on 2 PSE’s.
     Node : 0/9/CPU0 : 993 / 7966755      
    Incoming rate : 9999 / 8127181 (0/5/CPU0)
     Outgoing rate : 
     Node : 0/5/CPU0 : 198 / 15828597. >> it shows double as O-list is on 2 PSE’s.
     Node : 0/9/CPU0 : 992 / 7966755      """
 
    #Format olist_data[2]
    under_Outgoing = False
    list_node = olist_data[2].split('\n')
    for entry in list_node:
             
            in_rate = re.findall(r'Incoming\srate\s:\s(\d+)\s\/\s\d+', entry)
            if in_rate:
                   
                  #Incoming Rate information found. Save it to Node dictionary:
                  save_rate = in_rate
            outgoing = re.findall(r'(Outgoing rate:)', entry)
            if outgoing:
                 #That means it is under the "Outgoing rate:" section. 
                 under_Outgoing = True
            else:
                 under_Outgoing = False
 
                
            if entry.strip().startswith('Node'):
 
                 cpu_node_pos = re.findall(r'(\d\/\d)',entry)
                 if cpu_node_pos:
                      #cpu_node_pos = cpu_node_pos.strip()
                      pass
 
                 node = re.findall(r'Node\s:\s.*?\s:\s(\d+)\s/\s\d+',entry)
                 if node:
 
                      #Now save data: incoming rate, outgoing rate, CPU_pos_info, mcast_group
                      node_data =  [save_rate, node, cpu_node_pos]
                      var_dict['NODE'].append(node_data)
                      #print(f'Incoming Rate:{save_rate}    Outgoing Rate:{node}   CPU_INFO:{cpu_node_pos}')
 
     
     
    return var_dict
     
 
def ssh_nodes(myuser, nodepass,  host,mcastgroup):
 
    print(f'* Connecting to node for username:{USERNAME}')
 
    remote_conn_pre = paramiko.SSHClient()
    remote_conn_pre
    remote_conn_pre.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    host = host.strip()
    remote_conn_pre.connect(host, username=myuser,password=nodepass,look_for_keys=False,allow_agent=False)
 
    print(f'* SSH connection established to {host}')
    remote_conn = remote_conn_pre.invoke_shell()
    remote_conn.send('terminal length 0\n')
    remote_conn.send('show mrib route %s \n' %(mcastgroup))
    time.sleep(15)
    remote_conn.send('show mfib route rate %s \n' %(mcastgroup))
    time.sleep(15)
    remote_conn.send('show platform \n')
    time.sleep(15)
    output=remote_conn.recv(65535)
    time.sleep(10)
    remote_conn.close()
    output_str = output.decode('utf-8')
    return output_str
 
def get_credentials():
    #Assuming USERNAME is already entered
    inputpass = getpass.getpass()
    inputpass = inputpass.strip('\n')   
    return inputpass
 
def verify_threshold(node_PS_if_list):
    print(f'******************* NOW LISTING ***********************')
    print(node_PS_if_list)
 
def main():
 
    try:
        #Get secure password from user. Username is a global variable USERNAME
        node_pass = get_credentials()
    except Exception as err:
        print(f'Error during get_credentials() call: {err}')
     
    try:
        get_platform_data = ssh_nodes(USERNAME, node_pass, TARGET_SERVER, MGROUP)
    except Exception as err:
        print (f'Error during get_platform_data:{err}')
     
    try:
        get_analysis_int = olist(get_platform_data)
    
    except Exception as err:
       print(f'Error during olist(): {err}')
 
    try:
        verify_threshold(get_analysis_int)
    except Exception as err:
        print(f'Error during verify_threshold(): {err}')
    
    sys.exit(0)
 
if __name__ == '__main__':
    main()
 
  
#===end====================

            

        
      

**Larz60+** · (This post was last modified: Apr-09-2019, 09:08 PM by Larz60+.)

This is a defaultdict, and a defaultdict is a dict like object, see: https://docs.python.org/3/library/collec...efaultdict
so the output doesn't look like what you might expect, yet it's perfectly good data:

        
          
          
              
              >>> import collections
>>> zz =  {'PSE2': ['0/7/CPU0', '1/0/CPU0', '1/2/CPU0'], 'IF_PSE1': [['TenGigE0/5/0/21.201', 'PSE2=NO', 21, 'TenGigE0/5', 'REPLICATION=NO'], ['TenGigE0/7/0/3.1', 'PSE2=NO', 3, 'TenGigE0/7', 'REPLICATION=NO']], 'IF_PSE2': [['TenGigE0/7/0/3.1', 'PSE2=YES', 3, 'TenGigE0/7', 'REPLICATION=NO']], 'NODE': [[['1013'], ['1974'], ['0/5']], [['1013'], ['993'], ['0/9']], [['9999'], ['198'], ['0/5']], [['9999'], ['992'], ['0/9']]]}
>>> 
>>> sorted(zz.items())
[('IF_PSE1', [['TenGigE0/5/0/21.201', 'PSE2=NO', 21, 'TenGigE0/5', 'REPLICATION=NO'], ['TenGigE0/7/0/3.1', 'PSE2=NO', 3, 'TenGigE0/7', 'REPLICATION=NO']]), ('IF_PSE2', [['TenGigE0/7/0/3.1', 'PSE2=YES', 3, 'TenGigE0/7', 'REPLICATION=NO']]), ('NODE', [[['1013'], ['1974'], ['0/5']], [['1013'], ['993'], ['0/9']], [['9999'], ['198'], ['0/5']], [['9999'], ['992'], ['0/9']]]), ('PSE2', ['0/7/CPU0', '1/0/CPU0', '1/2/CPU0'])]
>>> 
>>> zz['NODE'][0][1]
['1974']
>>> zz['PSE2']
['0/7/CPU0', '1/0/CPU0', '1/2/CPU0']
>>>
>>> zz.keys()
dict_keys(['PSE2', 'IF_PSE1', 'IF_PSE2', 'NODE'])

            

        
      

mrapple2020 · Apr-09-2019, 09:53 PM

So if I want to list values 1, 2, 3, 4, 5, I did:
'NODE': [[['1013'], ['1'], ['5/6']], [['1013'], ['2'], ['0/6']], [['1013'], ['3'], ['0/9']], [['1013'], ['4'], ['0/1']], [['1013'], ['5'], ['5/6']], [['1013'], ['6'], ['0/6']], [['1013'], ['7'], ['0/9']], [['1013'], ['8'], ['0/1']]]})
expect 1 at position = ['1']
expect 2 at position = ['2']
node position 3 = ['3']
node position = ['4']
node position = ['5']

Question: placing a counter "i" in the for loop like in the below would do it.
It is not clear to me where would be the 'range' command placed to limit the list until it reaches the end?

Thank you!

i@ubuntu:~/Downloads/scripts$

        
              def verify_threshold(node_PS_if_list):
    print(f'******************* NOW LISTING ***********************')
    print(node_PS_if_list)
    for k, v in node_PS_if_list.items():
        if k ==  "NODE":
 
            print (f'expect 1 at position 1 =  {v[i][1]}')
            print (f'expect 2 at  position 1 = {v[i][1]}')
            print (f'node position 3 =         {v[i][1]}')
            print (f'node position           = {v[i][1]}')
            print (f'node position           = {v[i][1]}')

**Larz60+** · Apr-09-2019, 11:33 PM

well, if you look at the code that I below, there are 4 keys

        
          
          
              
              >>> import collections
>>> zz = {'PSE2': ['0/7/CPU0', '1/0/CPU0', '1/2/CPU0'], 'IF_PSE1': [['TenGigE0/5/0/21.201', 'PSE2=NO', 21, 'TenGigE0/5', 'REPLICATION=NO'], ['TenGigE0/7/0/3.1', 'PSE2=NO', 3, 'TenGigE0/7', 'REPLICATION=NO']], 'IF_PSE2': [['TenGigE0/7/0/3.1', 'PSE2=YES', 3, 'TenGigE0/7', 'REPLICATION=NO']], 'NODE': [[['1013'], ['1974'], ['0/5']], [['1013'], ['993'], ['0/9']], [['9999'], ['198'], ['0/5']], [['9999'], ['992'], ['0/9']]]}
>>>
>>> zz.keys()
dict_keys(['PSE2', 'IF_PSE1', 'IF_PSE2', 'NODE'])
>>>
>>> zz['PSE2']
['0/7/CPU0', '1/0/CPU0', '1/2/CPU0']
>>>
zz['IF_PSE1']
[['TenGigE0/5/0/21.201', 'PSE2=NO', 21, 'TenGigE0/5', 'REPLICATION=NO'], ['TenGigE0/7/0/3.1', 'PSE2=NO', 3, 'TenGigE0/7', 'REPLICATION=NO']]
>>>
>>> zz['IF_PSE2']
[['TenGigE0/7/0/3.1', 'PSE2=YES', 3, 'TenGigE0/7', 'REPLICATION=NO']]
>>>
zz['NODE']
[[['1013'], ['1974'], ['0/5']], [['1013'], ['993'], ['0/9']], [['9999'], ['198'], ['0/5']], [['9999'], ['992'], ['0/9']]]

            

        
      

There are only 4 item as seen with zz.keys()
then for example to breakdown the 'NODE' key:

        
          
          
              
              >>> for item in zz['NODE']:
...     print(item)
... 
[['1013'], ['1974'], ['0/5']]
[['1013'], ['993'], ['0/9']]
[['9999'], ['198'], ['0/5']]
[['9999'], ['992'], ['0/9']]
>>> 
>>> for item in zz['NODE']:
...     for item1 in item:
...         print(item1)
... 
['1013']
['1974']
['0/5']
['1013']
['993']
['0/9']
['9999']
['198']
['0/5']
['9999']
['992']
['0/9']
>>>

            

        
      

So from this you should be able to drill down to what you want.

mrapple2020 · Apr-09-2019, 11:51 PM

Sorry if I still didn't get it.
Let's say if I need to list the items I highlighted below (they are always the zz[variable][1].
If possible, can you kindly tell me your way to get a for loop and iterate thru the 'variable' to increment it and make me always get value from position zz[variable][1]? Thanks in advance.

Let's say I need to list

(Apr-09-2019, 11:33 PM)Larz60+ Wrote: well, if you look at the code that I below, there are 4 keys

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

>>> import collections
>>> zz = {'PSE2': ['0/7/CPU0', '1/0/CPU0', '1/2/CPU0'], 'IF_PSE1': [['TenGigE0/5/0/21.201', 'PSE2=NO', 21, 'TenGigE0/5', 'REPLICATION=NO'], ['TenGigE0/7/0/3.1', 'PSE2=NO', 3, 'TenGigE0/7', 'REPLICATION=NO']], 'IF_PSE2': [['TenGigE0/7/0/3.1', 'PSE2=YES', 3, 'TenGigE0/7', 'REPLICATION=NO']], 'NODE': [[['1013'], ['1974'], ['0/5']], [['1013'], ['993'], ['0/9']], [['9999'], ['198'], ['0/5']], [['9999'], ['992'], ['0/9']]]}
>>>
>>> zz.keys()
dict_keys(['PSE2', 'IF_PSE1', 'IF_PSE2', 'NODE'])
>>>
>>> zz['PSE2']
['0/7/CPU0', '1/0/CPU0', '1/2/CPU0']
>>>
zz['IF_PSE1']
[['TenGigE0/5/0/21.201', 'PSE2=NO', 21, 'TenGigE0/5', 'REPLICATION=NO'], ['TenGigE0/7/0/3.1', 'PSE2=NO', 3, 'TenGigE0/7', 'REPLICATION=NO']]
>>>
>>> zz['IF_PSE2']
[['TenGigE0/7/0/3.1', 'PSE2=YES', 3, 'TenGigE0/7', 'REPLICATION=NO']]
>>>
zz['NODE']
[[['1013'], ['1974'], ['0/5']], [['1013'], ['993'], ['0/9']], [['9999'], ['198'], ['0/5']], [['9999'], ['992'], ['0/9']]]

There are only 4 item as seen with zz.keys()
then for example to breakdown the 'NODE' key:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

>>> for item in zz['NODE']:
...     print(item)
...
[['1013'], ['1974'], ['0/5']]
[['1013'], ['993'], ['0/9']]
[['9999'], ['198'], ['0/5']]
[['9999'], ['992'], ['0/9']]
>>>
>>> for item in zz['NODE']:
...     for item1 in item:
...         print(item1)
...
['1013']
[b]['1974'][/b]
['0/5']
['1013']
[b]['993'][/b]
['0/9']
['9999']
[b]['198'][/b]
['0/5']
['9999']
[b]['992'][/b]
['0/9']
>>>

So from this you should be able to drill down to what you want.

**Larz60+** · Apr-10-2019, 09:47 PM

Ok:

        
              zz['IF_PSE1']
[['TenGigE0/5/0/21.201', 'PSE2=NO', 21, 'TenGigE0/5', 'REPLICATION=NO'], ['TenGigE0/7/0/3.1', 'PSE2=NO', 3, 'TenGigE0/7', 'REPLICATION=NO']]

It looks like you want the third element of each list, so:

        
          
          
              
              >>> for key, outerlist in zz.items():
...     for innerlist in outerlist:
...         print(key, innerlist[2])
... 
PSE2 7
PSE2 0
PSE2 2
IF_PSE1 21
IF_PSE1 3
IF_PSE2 3
NODE ['0/5']
NODE ['0/9']
NODE ['0/5']
NODE [['9999'], ['992'], [...]]
>>>

            

        
      

Is this correct?

Possibly Related Threads…
Thread		Author	Replies	Views	Last Post
	Strange behavior list of list	mmhmjanssen	3	1,610	May-09-2024, 11:32 AM Last Post: mmhmjanssen
	Can i clean this code ?	BSDevo	8	2,292	Oct-28-2023, 05:50 PM Last Post: BSDevo
	No matter what I do I get back "List indices must be integers or slices, not list"	Radical	4	2,601	Sep-24-2023, 05:03 AM Last Post: deanhystad
	How do I call sys.argv list inside a function, from the CLI?	billykid999	3	1,892	May-02-2023, 08:40 AM Last Post: Gribouillis
	Delete strings from a list to create a new only number list	Dvdscot	8	3,346	May-01-2023, 09:06 PM Last Post: deanhystad
	List all possibilities of a nested-list by flattened lists	sparkt	1	1,769	Feb-23-2023, 02:21 PM Last Post: sparkt
	Сheck if an element from a list is in another list that contains a namedtuple	elnk	8	3,402	Oct-26-2022, 04:03 PM Last Post: deanhystad
	Keyword to build list from list of objects?	pfdjhfuys	3	2,615	Aug-06-2022, 11:39 PM Last Post: Pedroski55
	Clean Up Script	rotw121	2	1,816	May-25-2022, 03:24 PM Last Post: rotw121
	Split a number to list and list sum must be number	sunny9495	5	3,827	Apr-28-2022, 09:32 AM Last Post: Dexty

Getting list inside of a list; how to clean it and retrieve value

User Panel Messages

Announcements