Python Forum
Installing and running a python web scraping app from github to a windows 8.1 system
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Installing and running a python web scraping app from github to a windows 8.1 system
#1
Hi to all forumers Smile
My name is Eugenio and I'm a python and R student. At this moment I'm trying to learn how to install and run python programs. I use python programs in trading, statistical analisys and web scraping. Every time I want to use a new program, the main obstacle is to install it in the right way. The best thing to do is to ask some help to the original developer but there are cases in which that it's not possible.
The next task I would like to accomplish is to use a program to scrape airbnb site to get data about rentals in my zone.
The program is here
https://github.com/tomslee/airbnb-data-collection
The developer is Tomslee but he says he's no more answering question from users:
http://tomslee.net/2017/10/airbnb-data-c...ement.html
So I'm trying to install and run the code by myself. Here are the steps till now
1. I have installed python ver 3.7 on a windows 8.1 system
2. I have changed the path variable in windows/system
3. I have downloaded the software from github and copied it in a subdir of c:\python_37 directory
Then I tried to run the airbnb.py file and I started to get error messages. They regarded python libraries not installed. I solved the first few error msgs by installing the required libraries.

Now the FIRST question arises:
In the Tomslee software there is this file:
requirements
When I type thehelp("modules") command I get this stuff:
installed modules
Then do I have to install all the libraries present in requirements-txt and not present in my python installation?

However when I try to run the airbnb.py file I get this error msg:

ERROR Configuration file eugenio.config not found.

Reading the airbnb_config.py I think that error msg is about a config file that should exist in some windows directory. Is that right?

Thanks a lot to all will help me. I know that I will ask the forum a gazillion question, but I think it should be interesting for users to follow a python software installation from beginning to the end. What I've learned right now is that installing python software is even more important than knowing how to code it. And I haven't find on Coursera and others a lot of tutorials about this task.
Reply
#2
Quote:At this moment I'm trying to learn how to install and run python programs.
please give example of 'program' giving difficulty.
You should not install into the python directory, this is done by the installer.
If the package you are installing is listed in pypi.org, you can install with:
pip3 install package
if not, clone or download package to a download staging area.
from that staging area, run python3 setup.py install

Quote:The next task I would like to accomplish is to use a program to scrape airbnb site to get data about rentals in my zone.
in the future, limit threads to a single question

do the following tutorials:
Web-Scraping part-1
Web-scraping part-2
Reply
#3
Quote:clone or download package to a download staging area.
from that staging area, run python3 setup.py install
Hi,
I've downloaded the package to a staging directory ...but there isn't a setup.py file to install.
Here is an image of the content of staging directory:
https://drive.google.com/open?id=16YHE-T...vruDI5NouQ
Reply
#4
what's in the README file? is this package named airbnb? if so, it's in PyPi.org, so all you need to install is:
pip3 install airbnb
in addition, their github repository https://github.com/nderkach/airbnb-python
has a setup.py file, so I'd be weary of the code you're using.
Reply
#5
Hi,
and tnx for Your answer. I've pip3 installed the airbnb file and some module were installed as stated here:
https://drive.google.com/open?id=1-u45dJ...KH2f56x4lY
Then I tried to run airbnb.py file either from command prompt or from idle interface but in both cases he throw at me an 'lxml module not installed error'.
So I need to manually install all the modules that I see in the requirements file (see my first post) and that aren't present in my python installation, or what?
About the github project from Nicolay Derkach, I'll give a look at it but do You think I would be able to get data like this
http://tomslee.net/airbnb-data-collection-get-the-data
for my city?
Reply
#6
pip install lxml
Quote:Then I tried to run airbnb.py file either from command prompt or from idle interface but in both cases he throw at me an 'lxml module not installed error'.
this is a package, and you have to write code around it. go to the github page, the author shows how to use it.
Reply
#7
I'm trying to install all python libraries needed but I got stuck installing ipywidgets that gives me this error:

Command "python setup.py egg_info" failed with error code 1 in C:\Users\eugenio\
AppData\Local\Temp\pip-install-khk27l8s\pywinpty\

Please note that I already upgraded setuptools (ver 40.4.3) and I've tried to install ipywidgets whl file from pipy.org without success.
I'm on a win 8.1 system and have python 3.7

Thanks to everyone can help me
Reply
#8
...problem solved.

I had to first install the pwinpty package from Gohlke site.
Reply
#9
gohike would be my first choice (for ms windows). They take the time to include all needed packages.
Reply
#10
...now I have to install nb-condacloud and nb-conda libraries. In order to do that I found the only way is a conda command (or I can use pip3?).
So I installed miniconda3 and run the conda command. I get a timeout error becouse I'm behind a proxy. I've tried to create a .condarc file with the right proxy settings but seems like it's not being loaded...

Any suggestions?
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Installing Github-programs AudunNilsen 1 142 Mar-22-2024, 04:28 PM
Last Post: deanhystad
  Advice for installing python app from github ? Adi44 6 1,116 Aug-07-2023, 06:34 PM
Last Post: snippsat
  Python script running under windows over nssm.exe JaroslavZ 0 671 May-12-2023, 09:22 AM
Last Post: JaroslavZ
  Installing Qt for Python. (On Windows 10) davediamond 22 4,361 Apr-18-2022, 06:50 AM
Last Post: ndc85430
  batch file for running python scipt in Windows shell MaartenRo 2 1,825 Jan-21-2022, 02:36 PM
Last Post: MaartenRo
  Installing auto-sklearn on Windows 10 Led_Zeppelin 1 2,621 Apr-15-2021, 08:02 PM
Last Post: bowlofred
  Need help installing infoblox-client on Windows 10 dazmac10 1 2,482 Mar-07-2021, 10:57 PM
Last Post: snippsat
  How to link Sublime Text 3 Build system to Python 3.9 Using Windows 10 Fanman001 2 4,531 Mar-04-2021, 03:09 PM
Last Post: martpogs
  Running python scripts from github etc pacmyc 7 3,609 Mar-03-2021, 10:26 PM
Last Post: pacmyc
  Difference between os.system("clear") and os.system("cls") chmsrohit 7 16,492 Jan-11-2021, 06:30 PM
Last Post: ykumar34

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020