Importing created module for web scraping with bs4

Truman · Sep-05-2018, 11:49 PM

module that I made:

1

2

3

4

5

6

7

8

9

10

11

12

<!DOCTYPE html>
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
  
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
  
<p class="story">...</p>

web scraping script:

        
              import ht_doc
from bs4 import BeautifulSoup
soup = BeautifulSoup(ht_doc.html, 'html.parser')
print(soup.prettify())

error:

Error:C:\Python36\kodovi>busoup.py
Traceback (most recent call last):
  File "C:\Python36\kodovi\busoup.py", line 1, in <module>
    import ht_doc
ModuleNotFoundError: No module named 'ht_doc'

Decided to practise Beautiful Soup module and hit a snag at the beginning. Wall

Why I can't import ht_doc module?

***metulburr*** · (This post was last modified: Sep-06-2018, 01:00 AM by metulburr.)

you dont import the html doc. IF you are writing the html in the same file as the code, you can just save the content in a variable and put it in like

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

html = '''
<!DOCTYPE html>
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
   
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
   
<p class="story">...</p>
'''
 
...
soup = BeautifulSoup(html, 'html.parser')
...

However if you are obtaining the content from the internet, then you should use the requests module to get the html, and use

1

2

3

...
r = requets.get('https://www.google.com')
soup = BeautifulSoup(r.text, 'html.parser')

If you have the html in a file, then you would have to load that html file like any other standard file and pass BeautifulSoup the whole file via fileObject.read()

more info here at our tutorials
https://python-forum.io/Thread-Web-Scraping-part-1

Truman · Sep-06-2018, 10:09 PM

Thank you, had no idea that I can't use html file as module. Will create a variable.

Truman · Sep-07-2018, 09:03 PM

here is an another way to import html file:

        
              with open("htmll.html") as fp:
    soup = BeautifulSoup(fp)
    print(soup.prettify())

***snippsat*** · (This post was last modified: Sep-07-2018, 10:25 PM by snippsat.)

(Sep-07-2018, 09:03 PM)Truman Wrote: here is an another way to import html file:

Have to be careful with encoding doing this.
Both on how get html from web(save) and open it.
It's easy to mess up Unicode,so try use UTF-8 always in and out.

        
              with open("htmll.html", encoding=utf-8) as fp:
    soup = BeautifulSoup(fp, 'lxml')
    print(soup.prettify())

Requests and BS do this well together,i explain better here Reading a html file.

**buran** · Sep-08-2018, 06:32 AM

(Sep-07-2018, 09:03 PM)Truman Wrote: here is an another way to import html file

try to be technically precise - you don't import the html file, you simply open it and pass the file handler to the BeautifulSoup to process it

Possibly Related Threads…
Thread		Author	Replies	Views	Last Post
	having a problem in importing module	pratheep	1	3,371	Jan-20-2018, 07:54 AM Last Post: buran

Importing created module for web scraping with bs4

User Panel Messages

Announcements