Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Receive data from Redis RQ worker process

I am currently trying to get my Flask Web Application working with Redis RQ.

My application gets an input file and analyzes it for approx. 20 seconds. During the analysis it keeps an filling a dictionary, consisting of dictionaries, which I want to access from my flask application to display the content of this dictionary in a nice way, like plots and graphs.

Now my idea is to allow the user to upload the file and hit "Analyze", which starts my program. Instead of waiting 20 seconds until the file has been processes and redirecting the user then to the page with all the plots I want to let my program run in the background. So if you click "Analyze" the browser directs the user directly to the page with the plots while in the background my analysis program is running.

For the task queue I am using redis RQ. The problem is, that I have no idea how I can access the dictionary, that is being filled by the worker process, through my flask application. Because the content of the dictionary keeps on growing I want to be able to present the user the live output as the analysis process keeps on running.

I tried using the job.meta function but I don't know if this function is really suited for a dictionary that can get quite big in size.

My question is: Can I have my worker process keep on writing the contents of the dictionary to a database, which then can be accessed by my flask application? Sort of like a middle man, that can be polled for results as the database keeps on growing.

Thank you

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Process Data from one csv file and write to another CSV file specific column ajin9581 1 83 Jun-17-2020, 06:00 PM
Last Post: buran
  Jython macro which uses PythonInterpreter and Redis script gives an error rkanumola 2 435 Oct-30-2019, 07:37 AM
Last Post: rkanumola
  How to sharing object between multiple process from main process using Pipe Subrata 1 839 Sep-03-2019, 09:49 PM
Last Post: woooee
  How can I only receive data from selected serial numbers MasterCATZ 7 721 Apr-20-2019, 08:35 AM
Last Post: MasterCATZ
  Receive Serial Data and store in different Variables in Python jenkins43 5 1,343 Dec-28-2018, 01:33 PM
Last Post: snippsat
  Json dictionnary on Redis katsu707 1 636 Dec-04-2018, 11:59 AM
Last Post: DeaD_EyE
  Sending a custom signal from a worker mhc 4 1,446 Jul-03-2018, 04:05 PM
Last Post: mhc
  Send and receive Files(data/images) betwenn a python app and another appl Akhou 2 1,064 May-02-2018, 08:20 AM
Last Post: Akhou
  Avoid output buffering when redirecting large data (40KB) to another process Ramphic 3 1,088 Mar-10-2018, 04:49 AM
Last Post: Larz60+
  how coding microphone receive non-English? TedHanaka 1 832 Feb-12-2018, 02:13 PM
Last Post: sparkz_alot

Forum Jump:

Users browsing this thread: 1 Guest(s)