Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Scripted update of web page
#1
I am not a developer per se. I try to find a way to update our CMDB database using Python (scripted bulk updates) and I stuck in a way.

Here is what I have.
CMDB database running on MS SQL server has IIS web interface.
The web page, when I check it's source, is combination of HTML and JavaScript code.
The IIS web server has an API for reading the database and creating new CIs. Only two methods are implemented GET and POST. The POST written in a way that it only capable of creating of new CIs and attempt to feed it with data for existing CI returns error with the message that record is not unique.
All this is somewhat understandable.
The best (in my opinion) way to up update CIs is to code PATCH method in API, but the person who wrote the API does not work here anymore.

What I tried to do.
I wrote Python code that can read the database and create new CIs. All works through the API and works well.
But I cannot find a way to update the record.
On the other hand I have a fact that from the web browser I can update any record. Which means that task can be done.

Is my assumption right:
That submitted data, before being posted from the web browser, pre-processed by the JavaScript imbedded into the web page, and only then get sent to the web server for processing. This way it does not use API and update database directly.
How I can use this to script my bulk update procedures?
Should I somehow intercept the page before it reaches the database (or the web server) and update it to my needs?
I do not quite understand how to form a Post request and how to send it to the web server bypassing the API.
I also noticed that each request uses some kind of unique key in communication with the server. I guess, it makes communication between browser and web server uncompromisable. Which also means that I, with my python interference, will not be able to enter the data I need. Unless I try to use some Python libraries that are capable of generating such unique key (or request it from the server) and use it in communication with the server.
I need some examples of such code, but I do not know how to ask questions about it.
Reply
#2
Hello! What library do you use?
Usually, after changing a record you have to call the commit method in order to update/save the changes.

Also, the script must have permissions to be able to change the database records or to create new ones.
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Reply
#3
I use requests library. Here is the code that works fine with API (creation of CIs):

url_cmdb_test = "http://idcsrv190/cmdb/api2/ci"
user,password = get_name_and_password( credential_file_name )

report_json_to_file = os.path.normpath(os.path.join(os.getcwd(),"test-ci_2.json"))
file_json_test = open(report_json_to_file, "r", encoding="utf-8").read()
headers = {"Content-Type" : "application/json;charset=utf-8"}
ret = requests.post(url=url_cmdb_test, data=file_json_test, headers=headers, auth=requests.auth.HTTPBasicAuth( user, password ), verify=False)
print(str(ret.status_code))
print(str(ret.text))
Here is one of the latest outputs:
201
{"id":6220,"self":"http://idcsrv190/cmdb/api2/ci/6220"}

The code "201" is the successfully created record. I can select it from the browser. There is no explicit commitment.
I add html page of the screen with the CI details on the next post. I do not see the way to attach file here.
The HTML file in line 162 has a form (<form action="/cmdb/ci/save") to save CI details. I think this is what I need to update and upload from my python script, but do not know how. The link /cmdb/ci/save doe not use API and sends information directly to the web server.

CodePen
Reply
#4
Added without line numbers

Link to code
Reply
#5
Perhaps you could use Selenium. It will handle the session, the cookies, JS and you can interact with any part of the webpage.

Before that try to read the JSON file and deserialize it.

import json

url_cmdb_test = "http://idcsrv190/cmdb/api2/ci"
user,password = get_name_and_password( credential_file_name )
 
report_json_to_file = os.path.normpath(os.path.join(os.getcwd(),"test-ci_2.json"))
file_json_test = open(report_json_to_file, "r", encoding="utf-8").read()
data = json.loads(file_json_test) # here you turn the json into a python dictionary
headers = {"Content-Type" : "application/json;charset=utf-8"}
ret = requests.post(url=url_cmdb_test, data=data, headers=headers, auth=requests.auth.HTTPBasicAuth( user, password ), verify=False) # data=data
print(str(ret.status_code))
print(str(ret.text))
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Reply
#6
Thank you. I'll try.
Reply
#7
(Jan-06-2018, 01:27 AM)treatitwell Wrote: The HTML file in line 162 has a form (<form action="/cmdb/ci/save") to save CI details. I think this is what I need to update and upload from my python script, but do not know how. The link /cmdb/ci/save doe not use API and sends information directly to the web server.
Here is just the form,it's big form with a lot values.
So writing a code for in Requests can be a struggle,you have to look at what's send in eg Chrome/FireFox developer tool.

To show a example with Flask:
Server: app.py
from flask import Flask, render_template, jsonify, request, redirect

app = Flask(__name__)

@app.route('/')
def hello():
    return render_template("index.html")

@app.route('/cmdb/ci/save', methods=['POST'])
def signup():
    email = request.form['email']
    print(f"The email address is {email}")
    # Now can server store form values in eg database
    return redirect('/')

if __name__ == '__main__':
    app.run(debug=True)
index.html:
<!doctype html>
<html>
<head>
  <title>Some title</title>
  <link rel=stylesheet type=text/css href="{{ url_for('static', filename='css/style.css') }}" />
</head>
<body>
  <div id="wrapper">
    <form action="/cmdb/ci/save" method="POST">
      <input type="text" name="email"></input>
      <input type="submit" value="Signup"></input>
    </form>
  </div>
</body>
</html>
So now sending data direct to server with Requests,same as filling out a html form and push submit button.
What server accept like headers info and checking of input data you have to figure out.
import requests

headers = {'Content-type': 'application/x-www-form-urlencoded'}
data = {"email": "[email protected]"}
response = requests.post('http://127.0.0.1:5000/cmdb/ci/save', headers=headers, data=data)
Output:
127.0.0.1 - - [06/Jan/2018 14:00:26] "POST /cmdb/ci/save HTTP/1.1" 302 - 127.0.0.1 - - [06/Jan/2018 14:00:26] "GET / HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2018 14:00:26] "GET /static/css/style.css HTTP/1.1" 200 - The email address is [email protected]
Edit:
Selenium as posted is also an option to do automation of the form data and push button.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  use Xpath in Python :: libxml2 for a page-to-page skip-setting apollo 2 3,578 Mar-19-2020, 06:13 PM
Last Post: apollo

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020