Python Forum
Receive data from Redis RQ worker process
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Receive data from Redis RQ worker process
#1
Hi,

I am currently trying to get my Flask Web Application working with Redis RQ.

My application gets an input file and analyzes it for approx. 20 seconds. During the analysis it keeps an filling a dictionary, consisting of dictionaries, which I want to access from my flask application to display the content of this dictionary in a nice way, like plots and graphs.

Now my idea is to allow the user to upload the file and hit "Analyze", which starts my program. Instead of waiting 20 seconds until the file has been processes and redirecting the user then to the page with all the plots I want to let my program run in the background. So if you click "Analyze" the browser directs the user directly to the page with the plots while in the background my analysis program is running.

For the task queue I am using redis RQ. The problem is, that I have no idea how I can access the dictionary, that is being filled by the worker process, through my flask application. Because the content of the dictionary keeps on growing I want to be able to present the user the live output as the analysis process keeps on running.

I tried using the job.meta function but I don't know if this function is really suited for a dictionary that can get quite big in size.

My question is: Can I have my worker process keep on writing the contents of the dictionary to a database, which then can be accessed by my flask application? Sort of like a middle man, that can be polled for results as the database keeps on growing.



Thank you
Reply


Messages In This Thread
Receive data from Redis RQ worker process - by freak14 - Jul-15-2019, 12:39 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  How to receive two passed cmdline parameters and access them inside a Python script? pstein 2 277 Feb-17-2024, 12:29 PM
Last Post: deanhystad
  Receive Input on Same Line? johnywhy 8 606 Jan-16-2024, 03:45 AM
Last Post: johnywhy
  How should I process this data? SuchUmami 2 346 Jan-13-2024, 05:30 PM
Last Post: RockBlok
  How would I process this data? SuchUmami 2 511 Nov-12-2023, 10:05 AM
Last Post: SuchUmami
  Reading data from excel file –> process it >>then write to another excel output file Jennifer_Jone 0 1,046 Mar-14-2023, 07:59 PM
Last Post: Jennifer_Jone
  How to continuously receive messages until specified to stop using Bleak jacobbreen25 3 2,016 Dec-28-2022, 04:25 PM
Last Post: jacobbreen25
  How can I add an end line character after every value that I receive? GiggsB 11 4,997 Mar-06-2022, 08:51 PM
Last Post: GiggsB
  How to receive webcam capture on spout? buzzdarkyear 2 2,589 Jan-12-2022, 02:26 PM
Last Post: buzzdarkyear
  Suspending worker threads olderguy 1 1,988 Apr-21-2021, 10:28 AM
Last Post: Larz60+
  Process Data from one csv file and write to another CSV file specific column ajin9581 1 1,942 Jun-17-2020, 06:00 PM
Last Post: buran

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020