Downloading Page Source From URL List

[deanhystad]

My sites.txt is just a list of urls like:
https://abc.com
https://def.com
https://ghi.com
Thanks.

If that is what your url's look like, not only is https: a problem. but there are problems with this:

        myfold = myurl[8:10]
        myfn = myurl[8:12]

"https://abc.com"[8:10] == "ab", not "abc". 10-8 = 2 characters, not 3. You would need [8:11], or [7:10] after you fix the https issue. Better yet you should use pattern matching to extract the file and folder names.

import re

URL = r"http://gibberish.com\more_gibberish?page="

if match := re.search(r"(\w+\.\w+)\\(.*)", URL):
    website, document = match.groups()
    name, ext = website.split(".")
    print(name, ext, document)

Output:
gibberish com more_gibberish?page=

[Pedroski55]

I don't think you will use \ in hyperlinks.

If your web address looks like:

URL = r"http://gibberish-rubbish-trash.com/more_gibberish?page="

and your regex is like this:

e = re.compile(r"(\w+\.\w+)\\(.*)")

You will not find the whole web address because, notwithstanding all hyphenated words, - is not \w:

Output:
e.search(URL)
<re.Match object; span=(25, 55), match='trash.com\\more_gibberish?page='>

I had trouble with web addresses containing -

This finds the base web address:

URL = r"http://gibberish-rubbish-trash.com/more_gibberish?page="
e = re.compile(r"//(\S+\.\w+)")
res = e.search(URL)
res.group(1)

Output:
'gibberish-rubbish-trash.com'