Downloading Page Source From URL List

[Pedroski55]

I don't think you will use \ in hyperlinks.

If your web address looks like:

URL = r"http://gibberish-rubbish-trash.com/more_gibberish?page="

and your regex is like this:

e = re.compile(r"(\w+\.\w+)\\(.*)")

You will not find the whole web address because, notwithstanding all hyphenated words, - is not \w:

Output:
e.search(URL)
<re.Match object; span=(25, 55), match='trash.com\\more_gibberish?page='>

I had trouble with web addresses containing -

This finds the base web address:

URL = r"http://gibberish-rubbish-trash.com/more_gibberish?page="
e = re.compile(r"//(\S+\.\w+)")
res = e.search(URL)
res.group(1)

Output:
'gibberish-rubbish-trash.com'