+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html)
+--- Thread: web scraping HTML - :( (/thread-33182.html)
RE: web scraping HTML - :( - snippsat - Apr-05-2021
And also move with open into the loop.
import requests
domains = ['https://webdesigner.dk', 'https://www.dk4.dk/item/4128-persondatapolitik']
for domain in domains:
print(domain)
response = requests.get(domain)
print(f"Response data length: {len(response.text)}")
# Remove # from line below to see source
# print(response.text)
with open('source.html', 'a', encoding='utf-8') as f:
f.write(response.text.strip())
RE: web scraping HTML - :( - Kingoman - Apr-05-2021
Thank you for keep trying to help me out here, but it is not working.
I have posted a picture of my screen.
https://imgur.com/hg0P9fq
The 10696 lines is still only from the last mentioned website.
RE: web scraping HTML - :( - ndc85430 - Apr-05-2021
Your writing isn't in the loop, is it?
RE: web scraping HTML - :( - Kingoman - Apr-05-2021
(Apr-05-2021, 03:37 AM)ndc85430 Wrote: Your writing isn't in the loop, is it?
I'm sorry but I don't know what you mean by this.
Please see the link with the image I posted in my previous post
RE: web scraping HTML - :( - supuflounder - Apr-05-2021
Here's a link to some scraping software I did a couple weeks ago:
https://python-forum.io/newreply.php?tid=32984&replyto=139401
RE: web scraping HTML - :( - ndc85430 - Apr-05-2021
The call to
write is done once, after the loop. Is that really what you intended?
RE: web scraping HTML - :( - Kingoman - Apr-05-2021
Thus is the First thing I try to setup as code, so I dont know
The intention is:
Put in my list of addresses, have it combine all the public HTML source code into one long text That I can view.
- thats it, it is just instead of me going through each website, right click and view source
RE: web scraping HTML - :( - Kingoman - Apr-05-2021
I don’t have any idea How to get it to loop, it was an idea for me to try this out, but yeah, doesn’t seem to work out
RE: web scraping HTML - :( - ndc85430 - Apr-05-2021
Do you understand that indentation tells you what's part of a block, like a
for loop?
RE: web scraping HTML - :( - Kingoman - Apr-05-2021
No.
I just wanted to try this out instead of manually doing 100 websites, but It doesn’t
Seem so simple to do, after all :)
T