Python Forum
Run function in parallel but inherite dynamic data from the previous function
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Run function in parallel but inherite dynamic data from the previous function
#1
I'm trying to run multiple functions simultaneously:

-or so called functions because they belong to a class:
    from sh import tail
    data = {}
    class my_Class():
        def __init__(self):
            """Nothing to declare for initializing"""
        def get_data(self, filepath):
            """I'm trying to import the data from several files"""
            for line in tail("-f", "-n 1", filepath, _iter=True):
                data[filepath] = line
                print(data)
    my_Class().get_data("path/to/file") #call 1
    my_Class().get_data("path/to/another/file") #call 2
    # ... 14 similar calls
I want each call to append it's data to the dictionary. And so, when I call:
    my_Class().get_data("path/to/file") #call 1
    my_Class().get_data("path/to/another/file") #call 2
    # ... 14 similar calls
The result should print:
    #1 {'filepath1' : line}
    #2 {'filepath1' : line, 'filepath2' : line}
    #3 {'filepath1' : line, 'filepath2' : line, 'filepath3' : line}
    # ... 13 more
At the same time I want the content of dictionary data{...} to keep changing dynamically; because of the data in the files is flexible. For example:
    #1 {'filepath1' : line}
    #2 {'filepath1' : last_line_in_the_file}, 'filepath2' : line}
    #3 {'filepath1' : last_line_in_the_file, 'filepath2' : last_line_in_the_file, 'filepath3' : line}
    # ... 13 more
I've already checked these posts: but it doesn't do what I ask; https://stackoverflow.com/questions/7207...n-parallel, https://stackoverflow.com/questions/2054...-in-python
Thank you! Please tell me if something sounds obscure
Reply
#2
You have not asked a specific question so there are no answers. What is wrong with the code you posted. Note that what is in the dictionary depends on the for statement so insert a print statement to make sure that it finds something. Note also that "data[filepath] = line" will only give you the final line found. Generally, the code would be more like
class my_Class():
    def __init__(self, data, filepath):
        for line in tail("-f", "-n 1", filepath, _iter=True):
            if filepath not in data:
                data[filepath]=[]
            data[filepath].append(line)
            print(data)

data={}
for filepath in many_paths:
    instance=my_Class(data, filepath)  
Reply
#3
(Jul-10-2018, 11:14 PM)woooee Wrote: You have not asked a question so there are no answers. Note that what is in the dictionary depends on the for statement so insert a print statement to make sure that it finds something. Note also that "data[filepath] = line" will only give you the final line found.
Hello I'm asking if what I'm trying to do is possible by someway ? Thank you :)
Reply
#4
Quote:I'm asking if what I'm trying to do is possible by someway ?
I'm not sure I know what that means, but it looks like your code already does that. Have you tried inserting a print statement as I suggested in the previous post/
Reply
#5
There are so many things wrong with this code (including PEP-8 non-compliance and non-specific names).

If you want to use a class - put your data in a class. Using methods (the proper name of class functions) to update external data is usually a bad idea - update object variable. Just create a class and a method that scans folder

class FileScanner:
    def __init__(self):
        self.data = {}

    def collect_tail(self, file_name):
        .....
            if filepath not in data:
                data[filepath]=[]
            self.data[filepath].append(line)

scanner = FileScanner()
scanner.collect_tail(...)
scanner.collect_tail(...)
This is the proper way

BTW, you are running "functions" neither in parallel nor simultaneously - your are running them sequentially, i.e. one after another.

(Jul-10-2018, 11:21 PM)woooee Wrote:
class my_Class():
    def __init__(self, data, filepath):
        for line in tail("-f", "-n 1", filepath, _iter=True):
            if filepath not in data:
                data[filepath]=[]
            data[filepath].append(line)
            print(data)

data={}

You are essentially updating external variable data - this is called side-effect and it is unhealthy coding practice.
Besides, __init__ method is usually used for initialization - not for actual actions

(Jul-10-2018, 11:21 PM)woooee Wrote:
for filepath in many_paths:
    instance=my_Class(data, filepath)  
And instance will be overwritten on each iteration...

This is not a good advice...
Test everything in a Python shell (iPython, Azure Notebook, etc.)
  • Someone gave you an advice you liked? Test it - maybe the advice was actually bad.
  • Someone gave you an advice you think is bad? Test it before arguing - maybe it was good.
  • You posted a claim that something you did not test works? Be prepared to eat your hat.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
Question Stopping a parent function from a nested function? wallgraffiti 1 333 May-02-2021, 12:21 PM
Last Post: Gribouillis
  Need help on extract dynamic table data Dr_Strange 0 542 Apr-30-2021, 07:03 AM
Last Post: Dr_Strange
Question exiting the outer function from the inner function banidjamali 3 613 Feb-27-2021, 09:47 AM
Last Post: banidjamali
  Function to delete data SmukasPlays 1 421 Jan-11-2021, 02:26 PM
Last Post: earlwindler
  Overwrite previous live data. Makada 2 647 Nov-07-2020, 07:40 PM
Last Post: Makada
  Passing argument from top-level function to embedded function JaneTan 2 571 Oct-15-2020, 03:50 PM
Last Post: deanhystad
  use NULL as function parameter which is a pointer function? oyster 0 658 Jul-11-2020, 09:02 AM
Last Post: oyster
  How to use function result in another function Ayckinn 4 861 Jun-16-2020, 04:50 PM
Last Post: Ayckinn
  Use dynamic variable from parallel running python script Sicksym 0 550 May-15-2020, 02:52 PM
Last Post: Sicksym
  Need help on Parallel Programming Kishore01 0 385 Mar-23-2020, 03:14 PM
Last Post: Kishore01

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020