+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html)
+--- Thread: Web App That Request Data from Another Web Site every 12-hours (/thread-12978.html)
Pages:
1
2
Web App That Request Data from Another Web Site every 12-hours - jomonetta - Sep-21-2018
Hi, I am working on a web app with flask framework. I want to get a data from another web site every 12 hour, save it and display it through my web app, but what should I do? I can get the data from another web site with requests package but what I want is done this every 12 hour and display it through my web app users while the app is online.
RE: Web App That Request Data from Another Web Site every 12-hours - mcmxl22 - Sep-21-2018
You could import the time module and use time.sleep.
RE: Web App That Request Data from Another Web Site every 12-hours - nilamo - Sep-21-2018
I vote against time.sleep for 12(!!) hours. For one thing, if you logout or the machine restarts, then your script never runs again.
I would set it up as two completely independent things. There's a web app, that takes some data and displays it. Then there's a second script that updates that data. The web app doesn't need to care about how often the data is updated, it just pushes it to the browser. And the second script also shouldn't care about how often it should run, it should just fetch the page, and update the data, every time it's run.
And then run the update script once every 12 hours using cron. https://en.wikipedia.org/wiki/Cron
RE: Web App That Request Data from Another Web Site every 12-hours - jomonetta - Sep-22-2018
(Sep-21-2018, 07:58 PM)nilamo Wrote: I vote against time.sleep for 12(!!) hours. For one thing, if you logout or the machine restarts, then your script never runs again. I would set it up as two completely independent things. There's a web app, that takes some data and displays it. Then there's a second script that updates that data. The web app doesn't need to care about how often the data is updated, it just pushes it to the browser. And the second script also shouldn't care about how often it should run, it should just fetch the page, and update the data, every time it's run. And then run the update script once every 12 hours using cron. https://en.wikipedia.org/wiki/CronThank you for your answer. So, if I deploy my web app, the server will run my main app script right? And if I want to send a request to another web site and monitor it through my app, does the server able to execute the other independent script which uses cron?
RE: Web App That Request Data from Another Web Site every 12-hours - snippsat - Sep-23-2018
(Sep-22-2018, 12:31 PM)jomonetta Wrote: does the server able to execute the other independent script which uses cron?Can get some solution with cron to work,but as server is running there are several solution made for this APScheduler,Celery,schedule, ect...
Here example with APScheduler,it's nice as can add job for several functions.
It run with
BackgroundScheduler so it will not interfef with main loop that server run.from flask import Flask, render_template, jsonify
from apscheduler.schedulers.background import BackgroundScheduler
import random
import requests
app = Flask(__name__)
def parse_func():
response = requests.get('https://nghttp2.org/httpbin/get')
r = response.json()
lst = [r['url'], r['origin']]
rand_value = random.choice(lst)
return(rand_value)
@app.route("/")
def template_test():
return render_template('sh2.html', location=parse_func())
if __name__ == '__main__':
scheduler = BackgroundScheduler()
scheduler.add_job(parse_func, 'interval', seconds=15)
scheduler.start()
app.run(debug=True)So now calling parse_func() every 15-sec,longer eg hour=12.Just to see values on client side use jinja,with a reload script.
sh2.html:
<!doctype html>
<html>
<head>
<title>Some title</title>
<link rel=stylesheet type=text/css href="{{ url_for('static', filename='css/style.css') }}" />
</head>
<body>
<p> {{ location }}</p>
</body>
<script>
function timedRefresh(timeoutPeriod) {
setTimeout("location.reload(true);",timeoutPeriod);
}
window.onload = timedRefresh(15000);
</script>
</html>Can also look at this post.
RE: Web App That Request Data from Another Web Site every 12-hours - jomonetta - Sep-24-2018
(Sep-23-2018, 05:44 PM)snippsat Wrote:Thank you for your answer. In this example, parse_func is called inside the render_template, and it sends request every time when the user goes to '/' route. I want to send a request every 12 hours, independent of the user choice of route. How can I avoid to send request every time?(Sep-22-2018, 12:31 PM)jomonetta Wrote: does the server able to execute the other independent script which uses cron?Can get some solution with cron to work,but as server is running there are several solution made for this APScheduler,Celery,schedule, ect... Here example with APScheduler,it's nice as can add job for several functions. It run withBackgroundSchedulerso it will not interfef with main loop that server run.from flask import Flask, render_template, jsonify from apscheduler.schedulers.background import BackgroundScheduler import random import requests app = Flask(__name__) def parse_func(): response = requests.get('https://nghttp2.org/httpbin/get') r = response.json() lst = [r['url'], r['origin']] rand_value = random.choice(lst) return(rand_value) @app.route("/") def template_test(): return render_template('sh2.html', location=parse_func()) if __name__ == '__main__': scheduler = BackgroundScheduler() scheduler.add_job(parse_func, 'interval', seconds=15) scheduler.start() app.run(debug=True)So now callingparse_func()every 15-sec,longer eghour=12. Just to see values on client side use jinja,with a reload script. sh2.html:<!doctype html> <html> <head> <title>Some title</title> <link rel=stylesheet type=text/css href="{{ url_for('static', filename='css/style.css') }}" /> </head> <body> <p> {{ location }}</p> </body> <script> function timedRefresh(timeoutPeriod) { setTimeout("location.reload(true);",timeoutPeriod); } window.onload = timedRefresh(15000); </script> </html>Can also look at this post.
RE: Web App That Request Data from Another Web Site every 12-hours - snippsat - Sep-24-2018
It don't have to be called in route,it work independent of that.
It's was just a demo to also send values from schedule function to client with jinja.
from flask import Flask, render_template, jsonify
from apscheduler.schedulers.background import BackgroundScheduler
import random
import requests
app = Flask(__name__)
def parse_func():
response = requests.get('https://nghttp2.org/httpbin/get')
r = response.json()
lst = [r['url'], r['origin']]
rand_value = random.choice(lst)
print(rand_value)
#return(rand_value)
@app.route("/")
def template():
return render_template('sh2.html')
if __name__ == '__main__':
scheduler = BackgroundScheduler()
scheduler.add_job(parse_func, 'interval', seconds=15)
scheduler.start()
app.run(debug=True)See that it run schedule interval,and routes can to there thing not knowing about this at all.
RE: Web App That Request Data from Another Web Site every 12-hours - jomonetta - Sep-24-2018
(Sep-24-2018, 03:12 AM)snippsat Wrote: It don't have to be called in route,it work independent of that. It's was just a demo to also send values from schedule function to client with jinja. from flask import Flask, render_template, jsonify from apscheduler.schedulers.background import BackgroundScheduler import random import requests app = Flask(__name__) def parse_func(): response = requests.get('https://nghttp2.org/httpbin/get') r = response.json() lst = [r['url'], r['origin']] rand_value = random.choice(lst) print(rand_value) #return(rand_value) @app.route("/") def template(): return render_template('sh2.html') if __name__ == '__main__': scheduler = BackgroundScheduler() scheduler.add_job(parse_func, 'interval', seconds=15) scheduler.start() app.run(debug=True) See that it run schedule interval,and routes can to there thing not knowing about this at all.Thank you. Now it is more clear, but I want to display this printed value through a route without sending request every time the route called. I am really appreciate for you effort. I know, I am asking lots of questions, but I am beginner and in fact I cannot find a solution for this problem. 
RE: Web App That Request Data from Another Web Site every 12-hours - nilamo - Sep-24-2018
The code you quoted does not make the request when you open the page. It makes it once every 15 seconds, regardless of how much or little the page is being requested.
RE: Web App That Request Data from Another Web Site every 12-hours - jomonetta - Sep-24-2018
(Sep-24-2018, 04:01 PM)nilamo Wrote: The code you quoted does not make the request when you open the page. It makes it once every 15 seconds, regardless of how much or little the page is being requested.how can I display the "rand_value" from '/' route or any other routes rather than displaying in the prompt screen?