(Jan-28-2018, 08:43 AM)league55 Wrote: Does that list vary depending on the data retrieved from the webpage accessed by requests.get()Now the list is always the same,it's build as a package wheel.
The list do not change based on OS or external source.
A typical example.
Requests is always getting getting the correct encoding that web site use.
So sending
content
not text
to BS is okay(no need to encode two times),as BS will detected that's utf-8 trough use of Unicode, Dammitfrom bs4 import BeautifulSoup import requests url = 'https://www.python.org/' url_get = requests.get(url) print(url_get.encoding) soup = BeautifulSoup(url_get.content, 'lxml') print(soup.select('head > title')[0].text)
Output:utf-8
Welcome to Python.org