Python Forum
Run function in parallel but inherite dynamic data from the previous function
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Run function in parallel but inherite dynamic data from the previous function
#1
I'm trying to run multiple functions simultaneously:

-or so called functions because they belong to a class:
    from sh import tail
    data = {}
    class my_Class():
        def __init__(self):
            """Nothing to declare for initializing"""
        def get_data(self, filepath):
            """I'm trying to import the data from several files"""
            for line in tail("-f", "-n 1", filepath, _iter=True):
                data[filepath] = line
                print(data)
    my_Class().get_data("path/to/file") #call 1
    my_Class().get_data("path/to/another/file") #call 2
    # ... 14 similar calls
I want each call to append it's data to the dictionary. And so, when I call:
    my_Class().get_data("path/to/file") #call 1
    my_Class().get_data("path/to/another/file") #call 2
    # ... 14 similar calls
The result should print:
    #1 {'filepath1' : line}
    #2 {'filepath1' : line, 'filepath2' : line}
    #3 {'filepath1' : line, 'filepath2' : line, 'filepath3' : line}
    # ... 13 more
At the same time I want the content of dictionary data{...} to keep changing dynamically; because of the data in the files is flexible. For example:
    #1 {'filepath1' : line}
    #2 {'filepath1' : last_line_in_the_file}, 'filepath2' : line}
    #3 {'filepath1' : last_line_in_the_file, 'filepath2' : last_line_in_the_file, 'filepath3' : line}
    # ... 13 more
I've already checked these posts: but it doesn't do what I ask; https://stackoverflow.com/questions/7207...n-parallel, https://stackoverflow.com/questions/2054...-in-python
Thank you! Please tell me if something sounds obscure
Reply
#2
You have not asked a specific question so there are no answers. What is wrong with the code you posted. Note that what is in the dictionary depends on the for statement so insert a print statement to make sure that it finds something. Note also that "data[filepath] = line" will only give you the final line found. Generally, the code would be more like
class my_Class():
    def __init__(self, data, filepath):
        for line in tail("-f", "-n 1", filepath, _iter=True):
            if filepath not in data:
                data[filepath]=[]
            data[filepath].append(line)
            print(data)

data={}
for filepath in many_paths:
    instance=my_Class(data, filepath)  
Reply
#3
(Jul-10-2018, 11:14 PM)woooee Wrote: You have not asked a question so there are no answers. Note that what is in the dictionary depends on the for statement so insert a print statement to make sure that it finds something. Note also that "data[filepath] = line" will only give you the final line found.
Hello I'm asking if what I'm trying to do is possible by someway ? Thank you :)
Reply
#4
Quote:I'm asking if what I'm trying to do is possible by someway ?
I'm not sure I know what that means, but it looks like your code already does that. Have you tried inserting a print statement as I suggested in the previous post/
Reply
#5
There are so many things wrong with this code (including PEP-8 non-compliance and non-specific names).

If you want to use a class - put your data in a class. Using methods (the proper name of class functions) to update external data is usually a bad idea - update object variable. Just create a class and a method that scans folder

class FileScanner:
    def __init__(self):
        self.data = {}

    def collect_tail(self, file_name):
        .....
            if filepath not in data:
                data[filepath]=[]
            self.data[filepath].append(line)

scanner = FileScanner()
scanner.collect_tail(...)
scanner.collect_tail(...)
This is the proper way

BTW, you are running "functions" neither in parallel nor simultaneously - your are running them sequentially, i.e. one after another.

(Jul-10-2018, 11:21 PM)woooee Wrote:
class my_Class():
    def __init__(self, data, filepath):
        for line in tail("-f", "-n 1", filepath, _iter=True):
            if filepath not in data:
                data[filepath]=[]
            data[filepath].append(line)
            print(data)

data={}

You are essentially updating external variable data - this is called side-effect and it is unhealthy coding practice.
Besides, __init__ method is usually used for initialization - not for actual actions

(Jul-10-2018, 11:21 PM)woooee Wrote:
for filepath in many_paths:
    instance=my_Class(data, filepath)  
And instance will be overwritten on each iteration...

This is not a good advice...
Test everything in a Python shell (iPython, Azure Notebook, etc.)
  • Someone gave you an advice you liked? Test it - maybe the advice was actually bad.
  • Someone gave you an advice you think is bad? Test it before arguing - maybe it was good.
  • You posted a claim that something you did not test works? Be prepared to eat your hat.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  The function of double underscore back and front in a class function name? Pedroski55 9 563 Feb-19-2024, 03:51 PM
Last Post: deanhystad
  Create a function for writing to SQL data to csv mg24 4 1,111 Oct-01-2022, 04:30 AM
Last Post: mg24
  Showing an empty chart, then input data via function kgall89 0 946 Jun-02-2022, 01:53 AM
Last Post: kgall89
  How to insert different types of data into a function DrData82 0 1,225 Feb-10-2022, 10:41 PM
Last Post: DrData82
Information Estimating transfer function from frd data ymohammadi 0 1,403 Feb-10-2022, 10:00 AM
Last Post: ymohammadi
  Flask dynamic data ? korenron 10 6,958 Nov-17-2021, 07:37 AM
Last Post: korenron
  Exit function from nested function based on user input Turtle 5 2,859 Oct-10-2021, 12:55 AM
Last Post: Turtle
  Retrieving a column from a data set using a function Bayle 6 2,290 Oct-06-2021, 08:52 PM
Last Post: Bayle
  Help with WebSocket reading data from anoter function korenron 0 1,301 Sep-19-2021, 11:08 AM
Last Post: korenron
Question Stopping a parent function from a nested function? wallgraffiti 1 3,621 May-02-2021, 12:21 PM
Last Post: Gribouillis

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020