Hi all,
In these days I wrote a script. It works very well but a piece of his code is very slow because of a custom function. you can see it below:
this text file is composed by five policies (you can identify them keeping in mind three parameters: source zone, destination zone and policy name). my custom function can split them with a duble new line and save them in a new variable. it's the result:
with many lines it's not scalable. how can I improve my custom function to become it scalable and more efficient than before?
In these days I wrote a script. It works very well but a piece of his code is very slow because of a custom function. you can see it below:
def split_rules(rules, *index): rule = "" if len(index) == 1: rules_splitted = rules.split("\n") first_rule_line = rules_splitted[0] index_0 = index[0] rule_name = first_rule_line.split()[index_0] for line in rules_splitted: list = line.split() if rule_name == list[index_0]: rule = rule + line + "\n" else: rule_name = line.split()[index_0] rule = rule + "\n" + line + "\n" rules = rule.rstrip("\n") elif len(index) == 2: rules_splitted = rules.split("\n") first_rule_line = rules_splitted[0] first_rule_line_splitted = first_rule_line.split() index_0 = index[0] index_1 = index[1] rule_name_1 = first_rule_line_splitted[index_0] rule_name_2 = first_rule_line_splitted[index_1] for line in rules_splitted: list = line.split() if rule_name_1 == list[index_0] and rule_name_2 == list[index_1]: rule = rule + line + "\n" else: rule_name_1 = list[index_0] rule_name_2 = list[index_1] rule = rule + "\n" + line + "\n" rules = rule.rstrip("\n") elif len(index) == 3: rules_splitted = rules.split("\n") first_rule_line = rules_splitted[0] first_rule_line_splitted = first_rule_line.split() index_0 = index[0] index_1 = index[1] index_2 = index[2] rule_name_1 = first_rule_line_splitted[index_0] rule_name_2 = first_rule_line_splitted[index_1] rule_name_3 = first_rule_line_splitted[index_2] for line in rules_splitted: list = line.split() if rule_name_1 == list[index_0] and rule_name_2 == list[index_1] and rule_name_3 == list[index_2]: rule = rule + line + "\n" else: rule_name_1 = list[index_0] rule_name_2 = list[index_1] rule_name_3 = list[index_2] rule = rule + "\n" + line + "\n" rules = rule.rstrip("\n") return(rules)the "split_rules" function splits a text file containing many policies. what do I mean with policies? I mean anything like that:
Quote:from-zone A to-zone B policy CICCIO ...
from-zone A to-zone B policy CICCIO ...
from-zone A to-zone B policy PIPPO ...
from-zone X to-zone Y policy PIPPO ...
from-zone X to-zone Y policy PIPPO ...
from-zone Z to-zone Z policy PLUTO ...
from-zone Z to-zone Z policy PLUTO ...
from-zone Z to-zone Z policy PLUTO ...
from-zone Z to-zone Z policy PAPERINO ...
from-zone Z to-zone Z policy PAPERINO ...
from-zone Z to-zone Z policy CICCIO ...
from-zone Z to-zone Z policy CICCIO ...
this text file is composed by five policies (you can identify them keeping in mind three parameters: source zone, destination zone and policy name). my custom function can split them with a duble new line and save them in a new variable. it's the result:
text = open("test.txt", "r") text = text.read() print(split_rules(text, 1, 3, 5)) # print output: # # from-zone A to-zone B policy CICCIO ... # from-zone A to-zone B policy CICCIO ... # # from-zone A to-zone B policy PIPPO ... # # from-zone X to-zone Y policy PIPPO ... # from-zone X to-zone Y policy PIPPO ... # # from-zone Z to-zone Z policy PLUTO ... # from-zone Z to-zone Z policy PLUTO ... # from-zone Z to-zone Z policy PLUTO ... # # from-zone Z to-zone Z policy PAPERINO ... # from-zone Z to-zone Z policy PAPERINO ... # # from-zone Z to-zone Z policy CICCIO ... # from-zone Z to-zone Z policy CICCIO ...this function works, it reaches his goal very well but it seems that it is not enough efficent when it have to process more than 40000 lines. with 40000 lines it takes around 2 minutes to process the policies, but with 150000 lines it takes around 15 minutes!
with many lines it's not scalable. how can I improve my custom function to become it scalable and more efficient than before?