Python Forum

Hello,

I'm learning how to use the "requests" module to download and parse web pages with Python3.

For some reason, I get a 404 when looping through URLs read from a file, while it works if I set the URL manually:

import requests

#Open list of URLs
file1 = open('urls.txt', 'r') 
Lines = file1.readlines() 
for line in Lines: 
	#URL looks OK
	print(line)

	#OK line = "https://www.google.com"
	response = requests.get(line)

	#print(response.text)
	#Says 404
	print(response.status_code)
file1.close()

Any idea what it could be?

Thank you.

readlines() separates the data by newline, but it doesn't remove the newline. All your lines (except possibly the final one) will have a trailing newline that is not part of the url. The newline is in the print you're doing, but isn't obvious.

Thanks much for the info.

Working (and simpler) code:

Lines = open("urls.txt").read().splitlines()
for line in Lines: 
	response = requests.get(line)
	print(response.text)