A problem with installing scrapy

[Truman]

Error:
rial\_dist
    creating build\lib.win32-3.6\twisted\trial\_dist\test
    copying src\twisted\trial\_dist\test\test_distreporter.py -> build\lib.win32
-3.6\twisted\trial\_dist\test
    copying src\twisted\trial\_dist\test\test_disttrial.py -> build\lib.win32-3.
6\twisted\trial\_dist\test
    copying src\twisted\trial\_dist\test\test_options.py -> build\lib.win32-3.6\
twisted\trial\_dist\test
    copying src\twisted\trial\_dist\test\test_worker.py -> build\lib.win32-3.6\t
wisted\trial\_dist\test
    copying src\twisted\trial\_dist\test\test_workerreporter.py -> build\lib.win
32-3.6\twisted\trial\_dist\test
    copying src\twisted\trial\_dist\test\test_workertrial.py -> build\lib.win32-
3.6\twisted\trial\_dist\test
    copying src\twisted\trial\_dist\test\__init__.py -> build\lib.win32-3.6\twis
ted\trial\_dist\test
    creating build\lib.win32-3.6\twisted\web\test
    copying src\twisted\web\test\requesthelper.py -> build\lib.win32-3.6\twisted
\web\test
    copying src\twisted\web\test\test_agent.py -> build\lib.win32-3.6\twisted\we
b\test
    copying src\twisted\web\test\test_cgi.py -> build\lib.win32-3.6\twisted\web\
test
    copying src\twisted\web\test\test_client.py -> build\lib.win32-3.6\twisted\w
eb\test
    copying src\twisted\web\test\test_distrib.py -> build\lib.win32-3.6\twisted\
web\test
    copying src\twisted\web\test\test_domhelpers.py -> build\lib.win32-3.6\twist
ed\web\test
    copying src\twisted\web\test\test_error.py -> build\lib.win32-3.6\twisted\we
b\test
    copying src\twisted\web\test\test_flatten.py -> build\lib.win32-3.6\twisted\
web\test
    copying src\twisted\web\test\test_html.py -> build\lib.win32-3.6\twisted\web
\test
    copying src\twisted\web\test\test_http.py -> build\lib.win32-3.6\twisted\web
\test
    copying src\twisted\web\test\test_http2.py -> build\lib.win32-3.6\twisted\we
b\test
    copying src\twisted\web\test\test_httpauth.py -> build\lib.win32-3.6\twisted
\web\test
    copying src\twisted\web\test\test_http_headers.py -> build\lib.win32-3.6\twi
sted\web\test
    copying src\twisted\web\test\test_newclient.py -> build\lib.win32-3.6\twiste
d\web\test
    copying src\twisted\web\test\test_proxy.py -> build\lib.win32-3.6\twisted\we
b\test
    copying src\twisted\web\test\test_resource.py -> build\lib.win32-3.6\twisted
\web\test
    copying src\twisted\web\test\test_script.py -> build\lib.win32-3.6\twisted\w
eb\test
    copying src\twisted\web\test\test_stan.py -> build\lib.win32-3.6\twisted\web
\test
    copying src\twisted\web\test\test_static.py -> build\lib.win32-3.6\twisted\w
eb\test
    copying src\twisted\web\test\test_tap.py -> build\lib.win32-3.6\twisted\web\
test
    copying src\twisted\web\test\test_template.py -> build\lib.win32-3.6\twisted
\web\test
    copying src\twisted\web\test\test_util.py -> build\lib.win32-3.6\twisted\web
\test
    copying src\twisted\web\test\test_vhost.py -> build\lib.win32-3.6\twisted\we
b\test
    copying src\twisted\web\test\test_web.py -> build\lib.win32-3.6\twisted\web\
test
    copying src\twisted\web\test\test_webclient.py -> build\lib.win32-3.6\twiste
d\web\test
    copying src\twisted\web\test\test_web__responses.py -> build\lib.win32-3.6\t
wisted\web\test
    copying src\twisted\web\test\test_wsgi.py -> build\lib.win32-3.6\twisted\web
\test
    copying src\twisted\web\test\test_xml.py -> build\lib.win32-3.6\twisted\web\
test
    copying src\twisted\web\test\test_xmlrpc.py -> build\lib.win32-3.6\twisted\w
eb\test
    copying src\twisted\web\test\_util.py -> build\lib.win32-3.6\twisted\web\tes
t
    copying src\twisted\web\test\__init__.py -> build\lib.win32-3.6\twisted\web\
test
    creating build\lib.win32-3.6\twisted\web\_auth
    copying src\twisted\web\_auth\basic.py -> build\lib.win32-3.6\twisted\web\_a
uth
    copying src\twisted\web\_auth\digest.py -> build\lib.win32-3.6\twisted\web\_
auth
    copying src\twisted\web\_auth\wrapper.py -> build\lib.win32-3.6\twisted\web\
_auth
    copying src\twisted\web\_auth\__init__.py -> build\lib.win32-3.6\twisted\web
\_auth
    creating build\lib.win32-3.6\twisted\words\im
    copying src\twisted\words\im\baseaccount.py -> build\lib.win32-3.6\twisted\w
ords\im
    copying src\twisted\words\im\basechat.py -> build\lib.win32-3.6\twisted\word
s\im
    copying src\twisted\words\im\basesupport.py -> build\lib.win32-3.6\twisted\w
ords\im
    copying src\twisted\words\im\interfaces.py -> build\lib.win32-3.6\twisted\wo
rds\im
    copying src\twisted\words\im\ircsupport.py -> build\lib.win32-3.6\twisted\wo
rds\im
    copying src\twisted\words\im\locals.py -> build\lib.win32-3.6\twisted\words\
im
    copying src\twisted\words\im\pbsupport.py -> build\lib.win32-3.6\twisted\wor
ds\im
    copying src\twisted\words\im\__init__.py -> build\lib.win32-3.6\twisted\word
s\im
    creating build\lib.win32-3.6\twisted\words\protocols
    copying src\twisted\words\protocols\irc.py -> build\lib.win32-3.6\twisted\wo
rds\protocols
    copying src\twisted\words\protocols\__init__.py -> build\lib.win32-3.6\twist
ed\words\protocols
    creating build\lib.win32-3.6\twisted\words\test
    copying src\twisted\words\test\test_basechat.py -> build\lib.win32-3.6\twist
ed\words\test
    copying src\twisted\words\test\test_basesupport.py -> build\lib.win32-3.6\tw
isted\words\test
    copying src\twisted\words\test\test_domish.py -> build\lib.win32-3.6\twisted
\words\test
    copying src\twisted\words\test\test_irc.py -> build\lib.win32-3.6\twisted\wo
rds\test
    copying src\twisted\words\test\test_ircsupport.py -> build\lib.win32-3.6\twi
sted\words\test
    copying src\twisted\words\test\test_irc_service.py -> build\lib.win32-3.6\tw
isted\words\test
    copying src\twisted\words\test\test_jabberclient.py -> build\lib.win32-3.6\t
wisted\words\test
    copying src\twisted\words\test\test_jabbercomponent.py -> build\lib.win32-3.
6\twisted\words\test
    copying src\twisted\words\test\test_jabbererror.py -> build\lib.win32-3.6\tw
isted\words\test
    copying src\twisted\words\test\test_jabberjid.py -> build\lib.win32-3.6\twis
ted\words\test
    copying src\twisted\words\test\test_jabberjstrports.py -> build\lib.win32-3.
6\twisted\words\test
    copying src\twisted\words\test\test_jabbersasl.py -> build\lib.win32-3.6\twi
sted\words\test
    copying src\twisted\words\test\test_jabbersaslmechanisms.py -> build\lib.win
32-3.6\twisted\words\test
    copying src\twisted\words\test\test_jabberxmlstream.py -> build\lib.win32-3.
6\twisted\words\test
    copying src\twisted\words\test\test_jabberxmppstringprep.py -> build\lib.win
32-3.6\twisted\words\test
    copying src\twisted\words\test\test_service.py -> build\lib.win32-3.6\twiste
d\words\test
    copying src\twisted\words\test\test_tap.py -> build\lib.win32-3.6\twisted\wo
rds\test
    copying src\twisted\words\test\test_xishutil.py -> build\lib.win32-3.6\twist
ed\words\test
    copying src\twisted\words\test\test_xmlstream.py -> build\lib.win32-3.6\twis
ted\words\test
    copying src\twisted\words\test\test_xmpproutertap.py -> build\lib.win32-3.6\
twisted\words\test
    copying src\twisted\words\test\test_xpath.py -> build\lib.win32-3.6\twisted\
words\test
    copying src\twisted\words\test\__init__.py -> build\lib.win32-3.6\twisted\wo
rds\test
    creating build\lib.win32-3.6\twisted\words\xish
    copying src\twisted\words\xish\domish.py -> build\lib.win32-3.6\twisted\word
s\xish
    copying src\twisted\words\xish\utility.py -> build\lib.win32-3.6\twisted\wor
ds\xish
    copying src\twisted\words\xish\xmlstream.py -> build\lib.win32-3.6\twisted\w
ords\xish
    copying src\twisted\words\xish\xpath.py -> build\lib.win32-3.6\twisted\words
\xish
    copying src\twisted\words\xish\xpathparser.py -> build\lib.win32-3.6\twisted
\words\xish
    copying src\twisted\words\xish\__init__.py -> build\lib.win32-3.6\twisted\wo
rds\xish
    creating build\lib.win32-3.6\twisted\words\protocols\jabber
    copying src\twisted\words\protocols\jabber\client.py -> build\lib.win32-3.6\
twisted\words\protocols\jabber
    copying src\twisted\words\protocols\jabber\component.py -> build\lib.win32-3
.6\twisted\words\protocols\jabber
    copying src\twisted\words\protocols\jabber\error.py -> build\lib.win32-3.6\t
wisted\words\protocols\jabber
    copying src\twisted\words\protocols\jabber\ijabber.py -> build\lib.win32-3.6
\twisted\words\protocols\jabber
    copying src\twisted\words\protocols\jabber\jid.py -> build\lib.win32-3.6\twi
sted\words\protocols\jabber
    copying src\twisted\words\protocols\jabber\jstrports.py -> build\lib.win32-3
.6\twisted\words\protocols\jabber
    copying src\twisted\words\protocols\jabber\sasl.py -> build\lib.win32-3.6\tw
isted\words\protocols\jabber
    copying src\twisted\words\protocols\jabber\sasl_mechanisms.py -> build\lib.w
in32-3.6\twisted\words\protocols\jabber
    copying src\twisted\words\protocols\jabber\xmlstream.py -> build\lib.win32-3
.6\twisted\words\protocols\jabber
    copying src\twisted\words\protocols\jabber\xmpp_stringprep.py -> build\lib.w
in32-3.6\twisted\words\protocols\jabber
    copying src\twisted\words\protocols\jabber\__init__.py -> build\lib.win32-3.
6\twisted\words\protocols\jabber
    creating build\lib.win32-3.6\twisted\_threads\test
    copying src\twisted\_threads\test\test_convenience.py -> build\lib.win32-3.6
\twisted\_threads\test
    copying src\twisted\_threads\test\test_memory.py -> build\lib.win32-3.6\twis
ted\_threads\test
    copying src\twisted\_threads\test\test_team.py -> build\lib.win32-3.6\twiste
d\_threads\test
    copying src\twisted\_threads\test\test_threadworker.py -> build\lib.win32-3.
6\twisted\_threads\test
    copying src\twisted\_threads\test\__init__.py -> build\lib.win32-3.6\twisted
\_threads\test
    running egg_info
    writing src\Twisted.egg-info\PKG-INFO
    writing dependency_links to src\Twisted.egg-info\dependency_links.txt
    writing entry points to src\Twisted.egg-info\entry_points.txt
    writing requirements to src\Twisted.egg-info\requires.txt
    writing top-level names to src\Twisted.egg-info\top_level.txt
    reading manifest file 'src\Twisted.egg-info\SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no previously-included files matching '*.misc' found under director
y 'src\twisted'
    warning: no previously-included files matching '*.bugfix' found under direct
ory 'src\twisted'
    warning: no previously-included files matching '*.doc' found under directory
 'src\twisted'
    warning: no previously-included files matching '*.feature' found under direc
tory 'src\twisted'
    warning: no previously-included files matching '*.removal' found under direc
tory 'src\twisted'
    warning: no previously-included files matching 'NEWS' found under directory
'src\twisted'
    warning: no previously-included files matching 'README' found under director
y 'src\twisted'
    warning: no previously-included files matching 'newsfragments' found under d
irectory 'src\twisted'
    warning: no previously-included files found matching 'src\twisted\topfiles\C
REDITS'
    warning: no previously-included files found matching 'src\twisted\topfiles\C
hangeLog.Old'
    warning: no previously-included files found matching 'pyproject.toml'
    warning: no previously-included files found matching 'codecov.yml'
    warning: no previously-included files found matching 'appveyor.yml'
    warning: no previously-included files found matching '.circleci'
    warning: no previously-included files matching '*' found under directory '.c
ircleci'
    no previously-included directories found matching 'bin'
    no previously-included directories found matching 'admin'
    no previously-included directories found matching '.travis'
    no previously-included directories found matching '.github'
    warning: no previously-included files found matching 'docs\historic\2003'
    warning: no previously-included files matching '*' found under directory 'do
cs\historic\2003'
    writing manifest file 'src\Twisted.egg-info\SOURCES.txt'
    copying src\twisted\python\twisted-completion.zsh -> build\lib.win32-3.6\twi
sted\python
    creating build\lib.win32-3.6\twisted\python\_pydoctortemplates
    copying src\twisted\python\_pydoctortemplates\common.html -> build\lib.win32
-3.6\twisted\python\_pydoctortemplates
    copying src\twisted\python\_pydoctortemplates\index.html -> build\lib.win32-
3.6\twisted\python\_pydoctortemplates
    copying src\twisted\python\_pydoctortemplates\summary.html -> build\lib.win3
2-3.6\twisted\python\_pydoctortemplates
    copying src\twisted\test\cert.pem.no_trailing_newline -> build\lib.win32-3.6
\twisted\test
    copying src\twisted\test\key.pem.no_trailing_newline -> build\lib.win32-3.6\
twisted\test
    copying src\twisted\test\server.pem -> build\lib.win32-3.6\twisted\test
    copying src\twisted\test\test_defer.py.3only -> build\lib.win32-3.6\twisted\
test
    copying src\twisted\internet\iocpreactor\notes.txt -> build\lib.win32-3.6\tw
isted\internet\iocpreactor
    copying src\twisted\internet\test\_awaittests.py.3only -> build\lib.win32-3.
6\twisted\internet\test
    copying src\twisted\internet\test\_yieldfromtests.py.3only -> build\lib.win3
2-3.6\twisted\internet\test
    creating build\lib.win32-3.6\twisted\internet\test\fake_CAs
    copying src\twisted\internet\test\fake_CAs\chain.pem -> build\lib.win32-3.6\
twisted\internet\test\fake_CAs
    copying src\twisted\internet\test\fake_CAs\not-a-certificate -> build\lib.wi
n32-3.6\twisted\internet\test\fake_CAs
    copying src\twisted\internet\test\fake_CAs\thing1.pem -> build\lib.win32-3.6
\twisted\internet\test\fake_CAs
    copying src\twisted\internet\test\fake_CAs\thing2-duplicate.pem -> build\lib
.win32-3.6\twisted\internet\test\fake_CAs
    copying src\twisted\internet\test\fake_CAs\thing2.pem -> build\lib.win32-3.6
\twisted\internet\test\fake_CAs
    copying src\twisted\mail\test\rfc822.message -> build\lib.win32-3.6\twisted\
mail\test
    copying src\twisted\python\test\_deprecatetests.py.3only -> build\lib.win32-
3.6\twisted\python\test
    copying src\twisted\words\im\instancemessenger.glade -> build\lib.win32-3.6\
twisted\words\im
    copying src\twisted\words\xish\xpathparser.g -> build\lib.win32-3.6\twisted\
words\xish
    running build_ext
    building 'twisted.test.raiser' extension
    error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual
C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools
    ----------------------------------------
ERROR: Command "'c:\python36\python.exe' -u -c 'import setuptools, tokenize;__fi
le__='"'"'C:\\Users\\user\\AppData\\Local\\Temp\\pip-install-gcuyud9b\\Twisted\\
setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().
replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'
"'exec'"'"'))' install --record 'C:\Users\user\AppData\Local\Temp\pip-record-6xx
qckqy\install-record.txt' --single-version-externally-managed --compile" failed
with error code 1 in C:\Users\user\AppData\Local\Temp\pip-install-gcuyud9b\Twist
ed\

I found a similar thread but it didn't help me much. Installed C++ 14.0.
Any suggestion?

[snippsat]

Did you follow my advice in that link?

Also you first download the Twisted wheel.
Then after install Twisted wheel,you do pip install -U scrapy
There there no need to mess with compiler C++ 14.0.

Here a run:

# Test python
C:\Users\Tom\Downloads
λ python -V
Python 3.7.3

# Test pip
C:\Users\Tom\Downloads
λ pip -V
pip 19.1.1 from c:\python37\lib\site-packages\pip (python 3.7)

# Install Twisted wheel
C:\Users\Tom\Downloads
λ pip install Twisted-19.2.0-cp37-cp37m-win32.whl
Processing c:\users\tom\downloads\twisted-19.2.0-cp37-cp37m-win32.whl
Dowloading ....
Successfully installed Twisted-19.2.0

# Now install scrapy
C:\Users\Tom\Downloads
λ pip install -U scrapy
Collecting scrapy
  Downloading ....
Successfully installed scrapy-1.6.0

# Test that it work
C:\Users\Tom\Downloads
λ python
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import scrapy

>>> scrapy.__version__
'1.6.0'
>>> exit()

[heiner55]

You can try https://www.lfd.uci.edu/~gohlke/pythonlibs/.
Then search for scrapy.

[Truman]

snippsat,

I tried to install Twisted wheel too but got error message.

[snippsat]

I tried to install Twisted wheel too but got error message.

No one here is a mind reader

Type python cmd:
C:\>python
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

So i have Python 3.7 and it's 32-bit,then this is wheel i need to use

pip install Twisted-19.2.0-cp37-cp37m-win32.whl

You use Python 3.6
Then there are two choices.

Twisted‑19.2.0‑cp36‑cp36m‑win32.whl
Twisted‑19.2.0‑cp36‑cp36m‑win_amd64.whl

If you get [MSC v.1916 32 bit (Intel)]

pip install Twisted‑19.2.0‑cp36‑cp36m‑win32.whl

If you get [MSC v.1915 64 bit (AMD64)]

pip install Twisted‑19.2.0‑cp36‑cp36m‑win_amd64.whl

Basic install tutorial Python 3.6/3.7 and pip installation under Windows

[Truman]

Error:
C:\Python36\kodovi>pip install Twisted-19.2.0-cp36-cp36m-win_amd64.whl
ERROR: Twisted-19.2.0-cp36-cp36m-win_amd64.whl is not a supported wheel on this
platform.

[snippsat]

Do you have 64-bit Python?
Reads my post again you test in cmd bye typing python.
Do you get [MSC v.1916 32 bit (Intel)] then you have 32-bit Python and install is.

pip install Twisted‑19.2.0‑cp36‑cp36m‑win32.whl

[Truman]

Thank you. I have windows on 64 bits which confused me.

I installed scrapy too but get this error

Error:
Traceback (most recent call last):
  File "C:\Python36\kodovi\scrap.py", line 1, in <module>
    import scrapy
  File "C:\Python36\lib\site-packages\scrapy\__init__.py", line 34, in <module>
    from scrapy.spiders import Spider
  File "C:\Python36\lib\site-packages\scrapy\spiders\__init__.py", line 10, in <
module>
    from scrapy.http import Request
  File "C:\Python36\lib\site-packages\scrapy\http\__init__.py", line 8, in <modu
le>
    from scrapy.http.headers import Headers
  File "C:\Python36\lib\site-packages\scrapy\http\headers.py", line 4, in <modul
e>
    from scrapy.utils.python import to_unicode
  File "C:\Python36\lib\site-packages\scrapy\utils\python.py", line 14, in <modu
le>
    from scrapy.utils.decorators import deprecated
  File "C:\Python36\lib\site-packages\scrapy\utils\decorators.py", line 4, in <m
odule>
    from twisted.internet import defer, threads
  File "C:\Python36\lib\site-packages\twisted\internet\defer.py", line 31, in <m
odule>
    from twisted.python import lockfile, failure
  File "C:\Python36\lib\site-packages\twisted\python\lockfile.py", line 46, in <
module>
    import pywintypes
  File "C:\Python36\lib\site-packages\win32\lib\pywintypes.py", line 136, in <mo
dule>
    __import_pywin32_system_module__("pywintypes", globals())
  File "C:\Python36\lib\site-packages\win32\lib\pywintypes.py", line 39, in __im
port_pywin32_system_module__
    for suffix_item in imp.get_suffixes():
AttributeError: module 'imp' has no attribute 'get_suffixes'

when trying to execute code:

import scrapy
from scrapy.crawler import CrawlerProcess

class PythonEventsSpider(scrapy.Spider):
	name = 'pythoneventsspider'
	
	start_urls = ['https://www.python.org/events/python-events/',]
	found_events = []
	
	def parse(self, response):
		for event in response.xpath('//ul[contains(@class, "list-recentevents")]/li'):
			event_details = dict()
			event_details['name'] = event.xpath('h3[@class="event-title"]/a/text()').extract_first()
			event_details['location'] = event.xpath('p/span[@class="event-location"]/text()').extract_first()
			event_details['time'] = event.xpath('p/time/text()').extract_first()
			self.found_events.append(event_details)
			
if __name__ == "__main__":
	process = CrawlerProcess({ 'LOG_LEVEL': 'ERROR'})
	process.crawl(PythonEventsSpider)
	spider = next(iter(process.crawlers)).spider
	process.start()
	
	for event in spider.found_events:
		print(event)

As per error message it looks that there are some problems with packages.

Also wondering should I use scrapy at all. This stuff can be done with requests and bs4 in less lines.

[snippsat]

I installed scrapy too but get this error

You should follow tutorial,to make sure that work first.

Also wondering should I use scrapy at all. This stuff can be done with requests and bs4 in less lines.

The user case can be very different,Scrapy is the fully fledged Spider library.
Which mean that it eg can follow a start_url and extract data from eg 10 to millions of pages.

start_urls = ['http://alibaba.com/']

alibaba has a lot of product pages,can eg use Srapy to find product and price of product of choice.
It will start and go trough 1000's of pages,and can save data get back in json,csv...ect
Writing the same code with Requests and BS would be a lot of work.

[Truman]

Ok, will give a try with a tutorial and tell you what happened.