Extracting Specific Lines from text file based on content.

[jokerfmj]

Hey, all. My apologies as I'm new to python in general, so I'm doing a lot of reading that goes over my head. :)

I'm working on a multi-step process where I run one python script that makes an API call to a system. I then have to extract some of those values from that output so that I can submit them, along with a new password, via a second API call that I would make after.

I have two main questions, first, is it possible to pipe the output of the initial API call directly to a text file rather than just to the console?

Second, is it even possible for python to look through a text file and extract certain lines based on the values in that line?

Some sample output from the first API call:

Output:
{
  "LINE_1": true,
  "LINE_2": "Post \"https://a:443/sdk\": dial tcp: lookup a on 1.1.127.252:53: no such host",
  "EXAMPLE_3": "",
  "created_at": 1648040001,
  "deleted": false,
  "deleted_at": 0,
  "delta_interval": 60,
  "description": "API's Testing.",
  "disable_backup": false,
  "hosts_list": [
    {
      "host_name": "TestingHost",
      "port_number": 443
    }
  ],
  "id": "623b1841755f021ebe269xyz",
  "name": "Tester",
  "password": "",
  "scope_id": "5f7e2a15497d4f48a9605687",
  "type": "vcenter",
}

There are four specific lines I am trying to extract:
"name":
"type":
"hosts_list":
"username":

I've come across some things describing how to convert lines of a text file into individual objects, but that only allows me to print lines based on the number line they are in a text file, it doesn't print lines based on the content in their lines.

Also, if this is possible, the "hosts_list" line may cause an issue because it actually has a subsection that can contain multiple entries. In the above example the subsection is:

Output:
    {
      "host_name": "TestingHost",
      "port_number": 443
    }

Thank you for any help you can provide.

[deanhystad]

You don't even have to pipe it to a text file. You can read it as input. Can you show me how you are making the API calls?

[jokerfmj]

You don't even have to pipe it to a text file. You can read it as input. Can you show me how you are making the API calls?

I'm not sure what you mean "read it as input," any chance you can explain?

My API call is:

API_ENDPOINT="https://MyURL.com"

restclient = RestClient(API_ENDPOINT,
                credentials_file='MyCredentialsFile.json',
                verify=False)

resp = restclient.get('API_Call_URI_here')

[deanhystad]

You did not specify you were writing a RESTful client. API is a really common term and I don't do a lot of web stuff. The json makes more sense now.

resp = restclient.get('API_Call_URI_here') returns a response, it does not write to the console. If something is being written to the console it is not because of the API call. How are you running this code? If you are running Python from the interactive window, the REPL automatically prints a reply to each command. For your command it would print resp.json. This is an artifact of how you are running the code and not real output.

resp will hold the response to your "get". resp.status_code is the status code for the request. resp.json is a json string with all the interesting stuff. You can convert the json string to a Python dictionary to make it easier to work with.

import requests
import json

API_ENDPOINT="https://MyURL.com"
 
restclient = requests.RestClient(API_ENDPOINT,
                credentials_file='MyCredentialsFile.json',
                verify=False)
 
resp = restclient.get('API_Call_URI_here')

data = json.loads(resp.json)
print(data["hosts_list"])

This should print:

Output:
[
    {
      "host_name": "TestingHost",
      "port_number": 443
    }
 ]

Using the dictionary you can ask for any of the json fields using their key name. These are the keys in the response you posted.

"LINE_1", "LINE_2", "EXAMPLE_3", "created_at", "deleted", "deleted_at", "delta_interval",
"description", "disable_backup"' "hosts_list", "id", "name", "password"' "scope_id", "type"

Are you familiar with Python dictionaries?

[snippsat]

resp.json is a json string with all the interesting stuff. You can convert the json string to a Python dictionary to make it easier to work with.

deanhystad as info so do Requests have build in json encoder/decoder added a long time ago(2014).
So the extra step with json string trough json library is not recommend.
can also mess up encoding,resp.json() directly from Request dos a much better job of figuring out encoding used.

>>> import requests
>>> 
>>> resp = requests.get('https://github.com/timeline.json')
>>> resp.json()
{'documentation_url': 'https://docs.github.com/v3/activity/events/#list-public-events',
 'message': 'Hello there, wayfaring stranger. If you’re reading this then you '
            'probably didn’t see our blog post a couple of years back '
            'announcing that this API would go away: http://git.io/17AROg Fear '
            'not, you should be able to get what you need from the shiny new '
            'Events API instead.'}

>>> resp.json()['documentation_url']
'https://docs.github.com/v3/activity/events/#list-public-events'

>>> resp.encoding
'utf-8'

[deanhystad]

That makes sense since it is something you would have to do all the time. Can you provide a link to a nice tutorial that demonstrates actually doing something with the json reply? I don't do this kind of thing, but it looks interesting.

[snippsat]

Can you provide a link to a nice tutorial that demonstrates actually doing something with the json reply?

In this post get Tor Mag to work that dos not have puplic API.
First send some post Payload then parse r.json() to get the shortened URL.

In this Thread see use of httpbin,can to testing of HTTP Request.

If search so are many tutorials about this eg this one is ok.
How to Use the Python Requests Module With REST APIs

[jokerfmj]

Thanks for this. When I run the script with

resp.json()

I get no output, but when I run it with

print (json.dumps(resp.json(), indent=2))

I get the full output. When I try to add in specific libraries (such as "hosts_file") I get the error

Error:
resp.json()["hosts_file"]

.

Is there something else I need to add?

======= EDIT =======

I needed to add the indexing piece (not sure correct terminology), but it is the [0] part in the below code.

print (resp.json()[0]['id'])

resp.json is a json string with all the interesting stuff. You can convert the json string to a Python dictionary to make it easier to work with.

deanhystad as info so do Requests have build in json encoder/decoder added a long time ago(2014).
So the extra step with json string trough json library is not recommend.
can also mess up encoding,resp.json() directly from Request dos a much better job of figuring out encoding used.

>>> import requests
>>> 
>>> resp = requests.get('https://github.com/timeline.json')
>>> resp.json()
{'documentation_url': 'https://docs.github.com/v3/activity/events/#list-public-events',
 'message': 'Hello there, wayfaring stranger. If you’re reading this then you '
            'probably didn’t see our blog post a couple of years back '
            'announcing that this API would go away: http://git.io/17AROg Fear '
            'not, you should be able to get what you need from the shiny new '
            'Events API instead.'}

>>> resp.json()['documentation_url']
'https://docs.github.com/v3/activity/events/#list-public-events'

>>> resp.encoding
'utf-8'

[snippsat]

What library as using,there is no RestClient in Requests?
Is the old python-restclient,which link to cheeseshop 🧀,
and use easy_install this is so old that new people to Python don't know what this is.

What showed in this Thread is Requests,and that's what you should use.