Posts: 14
Threads: 4
Joined: Jun 2019
Jul-05-2019, 07:10 AM
(This post was last modified: Jul-05-2019, 07:10 AM by starter_student.)
Hi everyone,
I'm trying to do a loop inside a div tag with others div tags but how to find all the children of that div!
<div id ="storelist" class>
<ul>
<li id ="00021455" class>
.
.
.
<div class ="wr-store-details">
<h3> class "Store abc"</h3>
<span> class ="20005 dc">
<span> class ="LA">
</ul>
</div>
#code
from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup
import csv
import requests
URL = "http:www.abc.com"
r = requests.get(URL)
soup = BeautifulSoup(r.content, 'html5lib')
store_details_tab = [] # a list to store store_details-tab
table = soup.find('div', attrs = {'id':'storelist'})
for row in table.findAll('div', attrs = {'class':'store_details'}):
store_details = {}
store_details['store_name'] = row.div.text
store_details['store_address'] = row.span.text
store_details['store_city'] = row.span.text
store_details_tab.append(store_details)
filename = 'store_details_tab.csv'
with open(filename, 'wb') as f:
w = csv.DictWriter(f,['store_name','store_address','store_city'])
w.writeheader()
for store_details in store_details_tab:
w.writerow(store_details)error: Traceback (most recent call last):
File "haagen_store.py", line 4, in <module>
import requests
ModuleNotFoundError: No module named 'requests'
Posts: 1,950
Threads: 8
Joined: Jun 2018
(Jul-05-2019, 07:10 AM)starter_student Wrote: error: Traceback (most recent call last):
File "haagen_store.py", line 4, in <module>
import requests
ModuleNotFoundError: No module named 'requests'
This error message states that Python can't find requests module. Have you installed it (it's not built-in module)?
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy
Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Posts: 14
Threads: 4
Joined: Jun 2019
(Jul-05-2019, 07:18 AM)perfringo Wrote: (Jul-05-2019, 07:10 AM)starter_student Wrote: error: Traceback (most recent call last):
File "haagen_store.py", line 4, in <module>
import requests
ModuleNotFoundError: No module named 'requests'
This error message states that Python can't find requests module. Have you installed it (it's not built-in module)?
Hey,
I installed both via anaconda
soup and request ![[Image: rxbQbgj]](https://ibb.co/rxbQbgj) ![[Image: hB2Fx87]](https://ibb.co/hB2Fx87)
Posts: 1,950
Threads: 8
Joined: Jun 2018
Jul-05-2019, 09:19 AM
(This post was last modified: Jul-05-2019, 09:19 AM by perfringo.)
(Jul-05-2019, 08:46 AM)starter_student Wrote: I installed both via anaconda
soup and request
You can make some easy checks from terminal:
pip list - list all packages and their versions
pip show requests detailed information, something like this:
Output: Name: requests
Version: 2.21.0
Summary: Python HTTP for Humans.
Home-page: http://python-requests.org
Author: Kenneth Reitz
Author-email: [email protected]
License: Apache 2.0
Location: /anaconda/lib/python3.7/site-packages
Requires: certifi, idna, chardet, urllib3
Required-by: twython, Sphinx, smart-open, requests-oauthlib, query-phenomizer, Nikola, jupyterhub, ipythonblocks, conda, anaconda-project, anaconda-client
If there is nothing to show it will return to prompt.
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy
Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Posts: 12,023
Threads: 484
Joined: Sep 2016
managing multiple versions is a snap with pyenv and virtual environments.
With pyenv, you have access to (in code tags for scrolling capability):
Available versions:
2.1.3
2.2.3
2.3.7
2.4.0
2.4.1
2.4.2
2.4.3
2.4.4
2.4.5
2.4.6
2.5.0
2.5.1
2.5.2
2.5.3
2.5.4
2.5.5
2.5.6
2.6.6
2.6.7
2.6.8
2.6.9
2.7.0
2.7-dev
2.7.1
2.7.2
2.7.3
2.7.4
2.7.5
2.7.6
2.7.7
2.7.8
2.7.9
2.7.10
2.7.11
2.7.12
2.7.13
2.7.14
2.7.15
2.7.16
3.0.1
3.1.0
3.1.1
3.1.2
3.1.3
3.1.4
3.1.5
3.2.0
3.2.1
3.2.2
3.2.3
3.2.4
3.2.5
3.2.6
3.3.0
3.3.1
3.3.2
3.3.3
3.3.4
3.3.5
3.3.6
3.3.7
3.4.0
3.4-dev
3.4.1
3.4.2
3.4.3
3.4.4
3.4.5
3.4.6
3.4.7
3.4.8
3.4.9
3.4.10
3.5.0
3.5-dev
3.5.1
3.5.2
3.5.3
3.5.4
3.5.5
3.5.6
3.5.7
3.6.0
3.6-dev
3.6.1
3.6.2
3.6.3
3.6.4
3.6.5
3.6.6
3.6.7
3.6.8
3.7.0
3.7-dev
3.7.1
3.7.2
3.7.3
3.8-dev
activepython-2.7.14
activepython-3.5.4
activepython-3.6.0
anaconda-1.4.0
anaconda-1.5.0
anaconda-1.5.1
anaconda-1.6.0
anaconda-1.6.1
anaconda-1.7.0
anaconda-1.8.0
anaconda-1.9.0
anaconda-1.9.1
anaconda-1.9.2
anaconda-2.0.0
anaconda-2.0.1
anaconda-2.1.0
anaconda-2.2.0
anaconda-2.3.0
anaconda-2.4.0
anaconda-4.0.0
anaconda2-2.4.0
anaconda2-2.4.1
anaconda2-2.5.0
anaconda2-4.0.0
anaconda2-4.1.0
anaconda2-4.1.1
anaconda2-4.2.0
anaconda2-4.3.0
anaconda2-4.3.1
anaconda2-4.4.0
anaconda2-5.0.0
anaconda2-5.0.1
anaconda2-5.1.0
anaconda2-5.2.0
anaconda2-5.3.0
anaconda2-5.3.1
anaconda2-2018.12
anaconda2-2019.03
anaconda3-2.0.0
anaconda3-2.0.1
anaconda3-2.1.0
anaconda3-2.2.0
anaconda3-2.3.0
anaconda3-2.4.0
anaconda3-2.4.1
anaconda3-2.5.0
anaconda3-4.0.0
anaconda3-4.1.0
anaconda3-4.1.1
anaconda3-4.2.0
anaconda3-4.3.0
anaconda3-4.3.1
anaconda3-4.4.0
anaconda3-5.0.0
anaconda3-5.0.1
anaconda3-5.1.0
anaconda3-5.2.0
anaconda3-5.3.0
anaconda3-5.3.1
anaconda3-2018.12
anaconda3-2019.03
ironpython-dev
ironpython-2.7.4
ironpython-2.7.5
ironpython-2.7.6.3
ironpython-2.7.7
jython-dev
jython-2.5.0
jython-2.5-dev
jython-2.5.1
jython-2.5.2
jython-2.5.3
jython-2.5.4-rc1
jython-2.7.0
jython-2.7.1
micropython-dev
micropython-1.9.3
micropython-1.9.4
micropython-1.10
miniconda-latest
miniconda-2.2.2
miniconda-3.0.0
miniconda-3.0.4
miniconda-3.0.5
miniconda-3.3.0
miniconda-3.4.2
miniconda-3.7.0
miniconda-3.8.3
miniconda-3.9.1
miniconda-3.10.1
miniconda-3.16.0
miniconda-3.18.3
miniconda2-latest
miniconda2-3.18.3
miniconda2-3.19.0
miniconda2-4.0.5
miniconda2-4.1.11
miniconda2-4.3.14
miniconda2-4.3.21
miniconda2-4.3.27
miniconda2-4.3.30
miniconda3-latest
miniconda3-2.2.2
miniconda3-3.0.0
miniconda3-3.0.4
miniconda3-3.0.5
miniconda3-3.3.0
miniconda3-3.4.2
miniconda3-3.7.0
miniconda3-3.8.3
miniconda3-3.9.1
miniconda3-3.10.1
miniconda3-3.16.0
miniconda3-3.18.3
miniconda3-3.19.0
miniconda3-4.0.5
miniconda3-4.1.11
miniconda3-4.2.12
miniconda3-4.3.11
miniconda3-4.3.14
miniconda3-4.3.21
miniconda3-4.3.27
miniconda3-4.3.30
pypy-c-jit-latest
pypy-c-nojit-latest
pypy-dev
pypy-stm-2.3
pypy-stm-2.5.1
pypy-1.5-src
pypy-1.5
pypy-1.6
pypy-1.7
pypy-1.8
pypy-1.9
pypy-2.0-src
pypy-2.0
pypy-2.0.1-src
pypy-2.0.1
pypy-2.0.2-src
pypy-2.0.2
pypy-2.1-src
pypy-2.1
pypy-2.2-src
pypy-2.2
pypy-2.2.1-src
pypy-2.2.1
pypy-2.3-src
pypy-2.3
pypy-2.3.1-src
pypy-2.3.1
pypy-2.4.0-src
pypy-2.4.0
pypy-2.5.0-src
pypy-2.5.0
pypy-2.5.1-src
pypy-2.5.1
pypy-2.6.0-src
pypy-2.6.0
pypy-2.6.1-src
pypy-2.6.1
pypy-4.0.0-src
pypy-4.0.0
pypy-4.0.1-src
pypy-4.0.1
pypy-5.0.0-src
pypy-5.0.0
pypy-5.0.1-src
pypy-5.0.1
pypy-5.1-src
pypy-5.1
pypy-5.1.1-src
pypy-5.1.1
pypy-5.3-src
pypy-5.3
pypy-5.3.1-src
pypy-5.3.1
pypy-5.4-src
pypy-5.4
pypy-5.4.1-src
pypy-5.4.1
pypy-5.6.0-src
pypy-5.6.0
pypy-5.7.0-src
pypy-5.7.0
pypy-5.7.1-src
pypy-5.7.1
pypy2-5.3-src
pypy2-5.3
pypy2-5.3.1-src
pypy2-5.3.1
pypy2-5.4-src
pypy2-5.4
pypy2-5.4.1-src
pypy2-5.4.1
pypy2-5.6.0-src
pypy2-5.6.0
pypy2-5.7.0-src
pypy2-5.7.0
pypy2-5.7.1-src
pypy2-5.7.1
pypy2.7-5.8.0-src
pypy2.7-5.8.0
pypy2.7-5.9.0-src
pypy2.7-5.9.0
pypy2.7-5.10.0-src
pypy2.7-5.10.0
pypy2.7-6.0.0-src
pypy2.7-6.0.0
pypy2.7-7.0.0-src
pypy2.7-7.0.0
pypy2.7-7.1.0-src
pypy2.7-7.1.0
pypy2.7-7.1.1-src
pypy2.7-7.1.1
pypy3-dev
pypy3-2.3.1-src
pypy3-2.3.1
pypy3-2.4.0-src
pypy3-2.4.0
pypy3.3-5.2-alpha1-src
pypy3.3-5.2-alpha1
pypy3.3-5.5-alpha-src
pypy3.3-5.5-alpha
pypy3.5-c-jit-latest
pypy3.5-5.7-beta-src
pypy3.5-5.7-beta
pypy3.5-5.7.1-beta-src
pypy3.5-5.7.1-beta
pypy3.5-5.8.0-src
pypy3.5-5.8.0
pypy3.5-5.9.0-src
pypy3.5-5.9.0
pypy3.5-5.10.0-src
pypy3.5-5.10.0
pypy3.5-5.10.1-src
pypy3.5-5.10.1
pypy3.5-6.0.0-src
pypy3.5-6.0.0
pypy3.5-7.0.0-src
pypy3.5-7.0.0
pypy3.6-7.0.0-src
pypy3.6-7.0.0
pypy3.6-7.1.0-src
pypy3.6-7.1.0
pypy3.6-7.1.1-src
pypy3.6-7.1.1
pyston-0.5.1
pyston-0.6.0
pyston-0.6.1
stackless-dev
stackless-2.7-dev
stackless-2.7.2
stackless-2.7.3
stackless-2.7.4
stackless-2.7.5
stackless-2.7.6
stackless-2.7.7
stackless-2.7.8
stackless-2.7.9
stackless-2.7.10
stackless-2.7.11
stackless-2.7.12
stackless-2.7.14
stackless-3.2.2
stackless-3.2.5
stackless-3.3.5
stackless-3.3.7
stackless-3.4-dev
stackless-3.4.1
stackless-3.4.2
stackless-3.4.7
stackless-3.5.4 If interested, see: https://python-forum.io/Thread-pyenv-Sim...ight=pyenv
Posts: 14
Threads: 4
Joined: Jun 2019
(Jul-05-2019, 09:19 AM)perfringo Wrote: (Jul-05-2019, 08:46 AM)starter_student Wrote: I installed both via anaconda
soup and request
You can make some easy checks from terminal:
pip list - list all packages and their versions
pip show requests detailed information, something like this:
Output: Name: requests
Version: 2.21.0
Summary: Python HTTP for Humans.
Home-page: http://python-requests.org
Author: Kenneth Reitz
Author-email: [email protected]
License: Apache 2.0
Location: /anaconda/lib/python3.7/site-packages
Requires: certifi, idna, chardet, urllib3
Required-by: twython, Sphinx, smart-open, requests-oauthlib, query-phenomizer, Nikola, jupyterhub, ipythonblocks, conda, anaconda-project, anaconda-client
If there is nothing to show it will return to prompt.
Thks for your response but I checked it and its already installed also the details were shown
Posts: 1,950
Threads: 8
Joined: Jun 2018
Jul-08-2019, 06:39 AM
(This post was last modified: Jul-08-2019, 06:39 AM by perfringo.)
(Jul-08-2019, 06:18 AM)starter_student Wrote: Thks for your response but I checked it and its already installed also the details were shown
Is the location of requests on your path so that Python can find it? You can check it within Python:
>>> import sys
>>> sys.path
['/anaconda/bin',
'/anaconda/lib/python3.7',
'/anaconda/lib/python3.7/lib-dynload',
'/anaconda/lib/python3.7/site-packages'
/.../] Now check whether requests location is on path. If not, the you should add it to path.
This issue can also be related to Python versions (did you install it to correct one) or virtual environment (do you use them?).
One can always install request again with command: -m pip install requests which will install it into same Python version which is running in shell.
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy
Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Posts: 14
Threads: 4
Joined: Jun 2019
(Jul-08-2019, 06:39 AM)perfringo Wrote: (Jul-08-2019, 06:18 AM)starter_student Wrote: Thks for your response but I checked it and its already installed also the details were shown
Is the location of requests on your path so that Python can find it? You can check it within Python:
>>> import sys
>>> sys.path
['/anaconda/bin',
'/anaconda/lib/python3.7',
'/anaconda/lib/python3.7/lib-dynload',
'/anaconda/lib/python3.7/site-packages'
/.../] Now check whether requests location is on path. If not, the you should add it to path.
This issue can also be related to Python versions (did you install it to correct one) or virtual environment (do you use them?).
One can always install request again with command: -m pip install requests which will install it into same Python version which is running in shell.
I think i was not importing the module in the right way! Thks
but now I have another error TypeError: a bytes-like object is required, not 'str'... can you help me with the code?
Posts: 7,312
Threads: 123
Joined: Sep 2016
(Jul-08-2019, 07:32 AM)starter_student Wrote: but now I have another error TypeError: a bytes-like object is required, not 'str'... can you help me with the code? Next time post whole Traceback ,it will also tell line number that problem occur in this case 39.
Change filename, 'wb' to filename, 'w'.
Posts: 14
Threads: 4
Joined: Jun 2019
(Jul-08-2019, 10:15 AM)snippsat Wrote: (Jul-08-2019, 07:32 AM)starter_student Wrote: but now I have another error TypeError: a bytes-like object is required, not 'str'... can you help me with the code? Next time post whole Traceback ,it will also tell line number that problem occur in this case 39.
Change filename, 'wb' to filename, 'w'.
Thks for the advice ... already done ! :) I removed the 'b'
and now there is no error but the output file is empty just with headers
|
