Python Forum

Full Version: Beautiful soup and tags
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Pages: 1 2
Hi everyone,

I'm trying to do a loop inside a div tag with others div tags but how to find all the children of that div!

<div id ="storelist" class>
   <ul> 
        <li id ="00021455" class>
         .
         .
         .
              <div class ="wr-store-details">
                  <h3> class "Store abc"</h3>
                  <span> class ="20005 dc">
                  <span> class ="LA">
    </ul>
</div>

#code
from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup
import csv 
import requests 
  
URL = "http:www.abc.com"
r = requests.get(URL) 
  
soup = BeautifulSoup(r.content, 'html5lib') 
  
store_details_tab = []  # a list to store store_details-tab 
  
table = soup.find('div', attrs = {'id':'storelist'}) 
  
for row in table.findAll('div', attrs = {'class':'store_details'}): 
    store_details = {} 
    store_details['store_name'] = row.div.text 
    store_details['store_address'] = row.span.text 
    store_details['store_city'] = row.span.text 
    store_details_tab.append(store_details) 
  
filename = 'store_details_tab.csv'
with open(filename, 'wb') as f: 
    w = csv.DictWriter(f,['store_name','store_address','store_city']) 
    w.writeheader() 
    for store_details in store_details_tab: 
        w.writerow(store_details)
error: Traceback (most recent call last):
File "haagen_store.py", line 4, in <module>
import requests
ModuleNotFoundError: No module named 'requests'
(Jul-05-2019, 07:10 AM)starter_student Wrote: [ -> ]error: Traceback (most recent call last):
File "haagen_store.py", line 4, in <module>
import requests
ModuleNotFoundError: No module named 'requests'

This error message states that Python can't find requests module. Have you installed it (it's not built-in module)?
(Jul-05-2019, 07:18 AM)perfringo Wrote: [ -> ]
(Jul-05-2019, 07:10 AM)starter_student Wrote: [ -> ]error: Traceback (most recent call last):
File "haagen_store.py", line 4, in <module>
import requests
ModuleNotFoundError: No module named 'requests'

This error message states that Python can't find requests module. Have you installed it (it's not built-in module)?

Hey,
I installed both via anaconda
soup and request[Image: rxbQbgj] [Image: hB2Fx87]
(Jul-05-2019, 08:46 AM)starter_student Wrote: [ -> ]I installed both via anaconda
soup and request[Image: rxbQbgj] [Image: hB2Fx87]

You can make some easy checks from terminal:

pip list - list all packages and their versions

pip show requests detailed information, something like this:
Output:
Name: requests Version: 2.21.0 Summary: Python HTTP for Humans. Home-page: http://python-requests.org Author: Kenneth Reitz Author-email: [email protected] License: Apache 2.0 Location: /anaconda/lib/python3.7/site-packages Requires: certifi, idna, chardet, urllib3 Required-by: twython, Sphinx, smart-open, requests-oauthlib, query-phenomizer, Nikola, jupyterhub, ipythonblocks, conda, anaconda-project, anaconda-client
If there is nothing to show it will return to prompt.
managing multiple versions is a snap with pyenv and virtual environments.
With pyenv, you have access to (in code tags for scrolling capability):
Available versions:
  2.1.3
  2.2.3
  2.3.7
  2.4.0
  2.4.1
  2.4.2
  2.4.3
  2.4.4
  2.4.5
  2.4.6
  2.5.0
  2.5.1
  2.5.2
  2.5.3
  2.5.4
  2.5.5
  2.5.6
  2.6.6
  2.6.7
  2.6.8
  2.6.9
  2.7.0
  2.7-dev
  2.7.1
  2.7.2
  2.7.3
  2.7.4
  2.7.5
  2.7.6
  2.7.7
  2.7.8
  2.7.9
  2.7.10
  2.7.11
  2.7.12
  2.7.13
  2.7.14
  2.7.15
  2.7.16
  3.0.1
  3.1.0
  3.1.1
  3.1.2
  3.1.3
  3.1.4
  3.1.5
  3.2.0
  3.2.1
  3.2.2
  3.2.3
  3.2.4
  3.2.5
  3.2.6
  3.3.0
  3.3.1
  3.3.2
  3.3.3
  3.3.4
  3.3.5
  3.3.6
  3.3.7
  3.4.0
  3.4-dev
  3.4.1
  3.4.2
  3.4.3
  3.4.4
  3.4.5
  3.4.6
  3.4.7
  3.4.8
  3.4.9
  3.4.10
  3.5.0
  3.5-dev
  3.5.1
  3.5.2
  3.5.3
  3.5.4
  3.5.5
  3.5.6
  3.5.7
  3.6.0
  3.6-dev
  3.6.1
  3.6.2
  3.6.3
  3.6.4
  3.6.5
  3.6.6
  3.6.7
  3.6.8
  3.7.0
  3.7-dev
  3.7.1
  3.7.2
  3.7.3
  3.8-dev
  activepython-2.7.14
  activepython-3.5.4
  activepython-3.6.0
  anaconda-1.4.0
  anaconda-1.5.0
  anaconda-1.5.1
  anaconda-1.6.0
  anaconda-1.6.1
  anaconda-1.7.0
  anaconda-1.8.0
  anaconda-1.9.0
  anaconda-1.9.1
  anaconda-1.9.2
  anaconda-2.0.0
  anaconda-2.0.1
  anaconda-2.1.0
  anaconda-2.2.0
  anaconda-2.3.0
  anaconda-2.4.0
  anaconda-4.0.0
  anaconda2-2.4.0
  anaconda2-2.4.1
  anaconda2-2.5.0
  anaconda2-4.0.0
  anaconda2-4.1.0
  anaconda2-4.1.1
  anaconda2-4.2.0
  anaconda2-4.3.0
  anaconda2-4.3.1
  anaconda2-4.4.0
  anaconda2-5.0.0
  anaconda2-5.0.1
  anaconda2-5.1.0
  anaconda2-5.2.0
  anaconda2-5.3.0
  anaconda2-5.3.1
  anaconda2-2018.12
  anaconda2-2019.03
  anaconda3-2.0.0
  anaconda3-2.0.1
  anaconda3-2.1.0
  anaconda3-2.2.0
  anaconda3-2.3.0
  anaconda3-2.4.0
  anaconda3-2.4.1
  anaconda3-2.5.0
  anaconda3-4.0.0
  anaconda3-4.1.0
  anaconda3-4.1.1
  anaconda3-4.2.0
  anaconda3-4.3.0
  anaconda3-4.3.1
  anaconda3-4.4.0
  anaconda3-5.0.0
  anaconda3-5.0.1
  anaconda3-5.1.0
  anaconda3-5.2.0
  anaconda3-5.3.0
  anaconda3-5.3.1
  anaconda3-2018.12
  anaconda3-2019.03
  ironpython-dev
  ironpython-2.7.4
  ironpython-2.7.5
  ironpython-2.7.6.3
  ironpython-2.7.7
  jython-dev
  jython-2.5.0
  jython-2.5-dev
  jython-2.5.1
  jython-2.5.2
  jython-2.5.3
  jython-2.5.4-rc1
  jython-2.7.0
  jython-2.7.1
  micropython-dev
  micropython-1.9.3
  micropython-1.9.4
  micropython-1.10
  miniconda-latest
  miniconda-2.2.2
  miniconda-3.0.0
  miniconda-3.0.4
  miniconda-3.0.5
  miniconda-3.3.0
  miniconda-3.4.2
  miniconda-3.7.0
  miniconda-3.8.3
  miniconda-3.9.1
  miniconda-3.10.1
  miniconda-3.16.0
  miniconda-3.18.3
  miniconda2-latest
  miniconda2-3.18.3
  miniconda2-3.19.0
  miniconda2-4.0.5
  miniconda2-4.1.11
  miniconda2-4.3.14
  miniconda2-4.3.21
  miniconda2-4.3.27
  miniconda2-4.3.30
  miniconda3-latest
  miniconda3-2.2.2
  miniconda3-3.0.0
  miniconda3-3.0.4
  miniconda3-3.0.5
  miniconda3-3.3.0
  miniconda3-3.4.2
  miniconda3-3.7.0
  miniconda3-3.8.3
  miniconda3-3.9.1
  miniconda3-3.10.1
  miniconda3-3.16.0
  miniconda3-3.18.3
  miniconda3-3.19.0
  miniconda3-4.0.5
  miniconda3-4.1.11
  miniconda3-4.2.12
  miniconda3-4.3.11
  miniconda3-4.3.14
  miniconda3-4.3.21
  miniconda3-4.3.27
  miniconda3-4.3.30
  pypy-c-jit-latest
  pypy-c-nojit-latest
  pypy-dev
  pypy-stm-2.3
  pypy-stm-2.5.1
  pypy-1.5-src
  pypy-1.5
  pypy-1.6
  pypy-1.7
  pypy-1.8
  pypy-1.9
  pypy-2.0-src
  pypy-2.0
  pypy-2.0.1-src
  pypy-2.0.1
  pypy-2.0.2-src
  pypy-2.0.2
  pypy-2.1-src
  pypy-2.1
  pypy-2.2-src
  pypy-2.2
  pypy-2.2.1-src
  pypy-2.2.1
  pypy-2.3-src
  pypy-2.3
  pypy-2.3.1-src
  pypy-2.3.1
  pypy-2.4.0-src
  pypy-2.4.0
  pypy-2.5.0-src
  pypy-2.5.0
  pypy-2.5.1-src
  pypy-2.5.1
  pypy-2.6.0-src
  pypy-2.6.0
  pypy-2.6.1-src
  pypy-2.6.1
  pypy-4.0.0-src
  pypy-4.0.0
  pypy-4.0.1-src
  pypy-4.0.1
  pypy-5.0.0-src
  pypy-5.0.0
  pypy-5.0.1-src
  pypy-5.0.1
  pypy-5.1-src
  pypy-5.1
  pypy-5.1.1-src
  pypy-5.1.1
  pypy-5.3-src
  pypy-5.3
  pypy-5.3.1-src
  pypy-5.3.1
  pypy-5.4-src
  pypy-5.4
  pypy-5.4.1-src
  pypy-5.4.1
  pypy-5.6.0-src
  pypy-5.6.0
  pypy-5.7.0-src
  pypy-5.7.0
  pypy-5.7.1-src
  pypy-5.7.1
  pypy2-5.3-src
  pypy2-5.3
  pypy2-5.3.1-src
  pypy2-5.3.1
  pypy2-5.4-src
  pypy2-5.4
  pypy2-5.4.1-src
  pypy2-5.4.1
  pypy2-5.6.0-src
  pypy2-5.6.0
  pypy2-5.7.0-src
  pypy2-5.7.0
  pypy2-5.7.1-src
  pypy2-5.7.1
  pypy2.7-5.8.0-src
  pypy2.7-5.8.0
  pypy2.7-5.9.0-src
  pypy2.7-5.9.0
  pypy2.7-5.10.0-src
  pypy2.7-5.10.0
  pypy2.7-6.0.0-src
  pypy2.7-6.0.0
  pypy2.7-7.0.0-src
  pypy2.7-7.0.0
  pypy2.7-7.1.0-src
  pypy2.7-7.1.0
  pypy2.7-7.1.1-src
  pypy2.7-7.1.1
  pypy3-dev
  pypy3-2.3.1-src
  pypy3-2.3.1
  pypy3-2.4.0-src
  pypy3-2.4.0
  pypy3.3-5.2-alpha1-src
  pypy3.3-5.2-alpha1
  pypy3.3-5.5-alpha-src
  pypy3.3-5.5-alpha
  pypy3.5-c-jit-latest
  pypy3.5-5.7-beta-src
  pypy3.5-5.7-beta
  pypy3.5-5.7.1-beta-src
  pypy3.5-5.7.1-beta
  pypy3.5-5.8.0-src
  pypy3.5-5.8.0
  pypy3.5-5.9.0-src
  pypy3.5-5.9.0
  pypy3.5-5.10.0-src
  pypy3.5-5.10.0
  pypy3.5-5.10.1-src
  pypy3.5-5.10.1
  pypy3.5-6.0.0-src
  pypy3.5-6.0.0
  pypy3.5-7.0.0-src
  pypy3.5-7.0.0
  pypy3.6-7.0.0-src
  pypy3.6-7.0.0
  pypy3.6-7.1.0-src
  pypy3.6-7.1.0
  pypy3.6-7.1.1-src
  pypy3.6-7.1.1
  pyston-0.5.1
  pyston-0.6.0
  pyston-0.6.1
  stackless-dev
  stackless-2.7-dev
  stackless-2.7.2
  stackless-2.7.3
  stackless-2.7.4
  stackless-2.7.5
  stackless-2.7.6
  stackless-2.7.7
  stackless-2.7.8
  stackless-2.7.9
  stackless-2.7.10
  stackless-2.7.11
  stackless-2.7.12
  stackless-2.7.14
  stackless-3.2.2
  stackless-3.2.5
  stackless-3.3.5
  stackless-3.3.7
  stackless-3.4-dev
  stackless-3.4.1
  stackless-3.4.2
  stackless-3.4.7
  stackless-3.5.4
If interested, see: https://python-forum.io/Thread-pyenv-Sim...ight=pyenv
(Jul-05-2019, 09:19 AM)perfringo Wrote: [ -> ]
(Jul-05-2019, 08:46 AM)starter_student Wrote: [ -> ]I installed both via anaconda
soup and request[Image: rxbQbgj] [Image: hB2Fx87]

You can make some easy checks from terminal:

pip list - list all packages and their versions

pip show requests detailed information, something like this:
Output:
Name: requests Version: 2.21.0 Summary: Python HTTP for Humans. Home-page: http://python-requests.org Author: Kenneth Reitz Author-email: [email protected] License: Apache 2.0 Location: /anaconda/lib/python3.7/site-packages Requires: certifi, idna, chardet, urllib3 Required-by: twython, Sphinx, smart-open, requests-oauthlib, query-phenomizer, Nikola, jupyterhub, ipythonblocks, conda, anaconda-project, anaconda-client
If there is nothing to show it will return to prompt.

Thks for your response but I checked it and its already installed also the details were shown
(Jul-08-2019, 06:18 AM)starter_student Wrote: [ -> ]Thks for your response but I checked it and its already installed also the details were shown

Is the location of requests on your path so that Python can find it? You can check it within Python:

>>> import sys
>>> sys.path
['/anaconda/bin',
 '/anaconda/lib/python3.7',
 '/anaconda/lib/python3.7/lib-dynload',
 '/anaconda/lib/python3.7/site-packages'
  /.../]
Now check whether requests location is on path. If not, the you should add it to path.

This issue can also be related to Python versions (did you install it to correct one) or virtual environment (do you use them?).

One can always install request again with command: -m pip install requests which will install it into same Python version which is running in shell.
(Jul-08-2019, 06:39 AM)perfringo Wrote: [ -> ]
(Jul-08-2019, 06:18 AM)starter_student Wrote: [ -> ]Thks for your response but I checked it and its already installed also the details were shown

Is the location of requests on your path so that Python can find it? You can check it within Python:

>>> import sys
>>> sys.path
['/anaconda/bin',
 '/anaconda/lib/python3.7',
 '/anaconda/lib/python3.7/lib-dynload',
 '/anaconda/lib/python3.7/site-packages'
  /.../]
Now check whether requests location is on path. If not, the you should add it to path.

This issue can also be related to Python versions (did you install it to correct one) or virtual environment (do you use them?).

One can always install request again with command: -m pip install requests which will install it into same Python version which is running in shell.

I think i was not importing the module in the right way! Thks
but now I have another error TypeError: a bytes-like object is required, not 'str'... can you help me with the code?
(Jul-08-2019, 07:32 AM)starter_student Wrote: [ -> ]but now I have another error TypeError: a bytes-like object is required, not 'str'... can you help me with the code?
Next time post whole Traceback ,it will also tell line number that problem occur in this case 39.
Change filename, 'wb' to filename, 'w'.
(Jul-08-2019, 10:15 AM)snippsat Wrote: [ -> ]
(Jul-08-2019, 07:32 AM)starter_student Wrote: [ -> ]but now I have another error TypeError: a bytes-like object is required, not 'str'... can you help me with the code?
Next time post whole Traceback ,it will also tell line number that problem occur in this case 39.
Change filename, 'wb' to filename, 'w'.

Thks for the advice ... already done ! :) I removed the 'b'
and now there is no error but the output file is empty just with headers
Pages: 1 2