Using a String var as URL in Requests.get( )

Using a String var as URL in Requests.get( ) - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: Using a String var as URL in Requests.get( ) (/thread-39838.html)

Using a String var as URL in Requests.get( ) - DBB - Apr-20-2023

New to python so forgive me...

I have an external .csv file containing several lines, each one of which is a fully formed URL, having user names and states as parameters, eg:
'https://MyTargetAPI.com/api/?firstname=bozo&lastname=clown&state=FL'

I'm trying to iterate through the list of URLs and pass each one through the requests.get() function, with the retrieved results saved to a file.

My module is working for one record when the actual URL is used in the requests.get() function, thusly:

r = requests.get('https://MyTargetAPI.com/api/?firstname=bozo&lastname=clown&state=FL')
results_text = r.text
print(results_text)   #displays the results returned from the API

The rest of my code correctly handles the information returned from the API and saves it to file, as intended.

But I have had no luck attempting to pull each URL from the file, assigning it to a variable, then using the variable in the requests.get() function.

I can iterate through each line of the external file and the print(line.strip()) section correctly displays each retrieved URL as I loop through. But if I attempt to assign the retrieved URL string to a variable, then use the variable in the requests.get() function rather than the actual URL, things go south:

#By itself, this seems to be working, and displays each URL retrieved as I loop through the external file:
print(line.strip())
     
#But when I follow on with this, it breaks:
api_url = line.strip()
r = requests.get(api_url)
results_text = r.text
print(results_text)

I'm not sure of the syntax, and so this is probably something obvious to you all, but I'm not sure how to de-reference the variable within the function to get this working. I would really appreciate a nudge in the right direction.

-DBB

RE: Using a String var as URL in Requests.get( ) - snippsat - Apr-21-2023

(Apr-20-2023, 11:52 PM)DBB Wrote: But I have had no luck attempting to pull each URL from the file, assigning it to a variable, then using the variable in the requests.get() function.

You don't assigning to variable then loop,can do in one go when iterate over file object.
Example like this.

url_lst.csv:

Output:https://www.google.com/,
https://www.python.org/,

import csv
import requests

with open('url_lst.csv') as fp:
    for url in csv.reader(fp):
        print(url[0])
        response = requests.get(url[0])
        print(response.status_code)

Output:https://www.google.com/
200
https://www.python.org/
200

Also when use an Api is most normal to work with Json return and not text.

results_text = r.text #Not normal 
result_json = r.json() #The normal way

when have Json can also use Serialization(get data back to original format eg a dictionary in this case).

with open("data.json", "w") as fp:
    json.dump(result_json, fp)
  
with open("data.json") as fp_read:
    result_json = json.load(fp_read)

RE: Using a String var as URL in Requests.get( ) - DBB - Apr-21-2023

Thanks for replying!

(Apr-21-2023, 02:51 AM)snippsat Wrote: You don't assigning to variable then loop,can do in one go when iterate over file object.
Example like this.

url_lst.csv:
Output:https://www.google.com/,
https://www.python.org/,
import csv
import requests

with open('url_lst.csv') as fp:
    for url in csv.reader(fp):
        print(url[0])
        response = requests.get(url[0])
        print(response.status_code)
Output:https://www.google.com/
200
https://www.python.org/
200

I'm not sure I follow the above. To be clear, my input file already is a collection of completely formed URLs that I'm trying to pass to the API, one by one as I step through that file.

Quote:Also when use an Api is most normal to work with Json return and not text.
results_text = r.text #Not normal 
result_json = r.json() #The normal way
when have Json can also use Serialization(get data back to original format eg a dictionary in this case).
with open("data.json", "w") as fp:
    json.dump(result_json, fp)
 
 with open("data.json") as fp_read:
    result_json = json.load(fp_read)

Yeah, that's largely what I'm doing in the code I have in the other part of the module. I was simply trying to pass each line of my input file as a variable out to the API using the r= results.get(My_URL_var) function.

I have scoured the internet looking for a method of doing that, so maybe it's not possible. I ended up reworking my input .csv file to only include the items I needed to include as parameters in the URL string, then created a payload variable to include in the requests.get() function like this:

r=requests.get('https://MyTargetAPI.com/api/', params=payload)

I'm still up to my neck in it, but that seems to work for that portion...

Thanks, again!
-DBB

RE: Using a String var as URL in Requests.get( ) - snippsat - Apr-22-2023

(Apr-21-2023, 11:06 PM)DBB Wrote: I'm not sure I follow the above. To be clear, my input file already is a collection of completely formed URLs that I'm trying to pass to the API, one by one as I step through that file.

That what's my code do,if i take your url it would use that in last in requests.get().

Output:https://www.google.com/,
https://www.python.org/,
'https://MyTargetAPI.com/api/?firstname=bozo&lastname=clown&state=FL

import csv
import requests
 
with open('url_lst.csv') as fp:
    for url in csv.reader(fp):
        print(url[0])
        response = requests.get(url[0])
        print(response.status_code)

Output:https://www.google.com/
200
https://www.python.org/
200
'https://MyTargetAPI.com/api/?firstname=bozo&lastname=clown&state=FL
Traceback (most recent call last):
.....  
requests.exceptions.InvalidSchema: No connection adapters were found for "'https://MyTargetAPI.com/api/?firstname=bozo&lastname=clown&state=FL"

So this would work if it had been a working url(know that is just example) it would give 200 back,and then could use respone.json or respone.text to get return from Api.

Quote:I have scoured the internet looking for a method of doing that, so maybe it's not possible. I ended up reworking my input .csv file to only include the items I needed to include as parameters in the URL string, then created a payload variable to include in the requests.get() function like this

Could split it up,but you mention that is was completely formed URL'S than a a loop as showed should work.