Python Forum
California Public Salary Files - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html)
+--- Thread: California Public Salary Files (/thread-2586.html)



California Public Salary Files - Larz60+ - Mar-26-2017

California posts files containing salary and compensation of many (all?) public positions
on one of their websites. These files are all of the same .csv format which makes them very handy
for various applications.

I have been playing with them, and thanks to Snippsat am able to scrape the ASP.net catalog.

Keeping the list up to date was a pain, so I created a GUI to aid in the process.

In the next few days, I will add:
  • Automatic zip file extraction -- Done
  • command list so that specific files can be automatically loaded using command file.
This software is available here: https://github.com/Larz60p/CaliforniaPublicSalaries

Here's a screenshot:

[attachment=144]


RE: California Public Salary Files - Larz60+ - Mar-27-2017

Added automatic unpacking of zip files


RE: California Public Salary Files - snippsat - Mar-27-2017

You should add a way to run it.
Eg:
pip freeze > requirements.txt
This contain:
appdirs==1.4.3
beautifulsoup4==4.5.3
lxml==3.7.3
packaging==16.8
pyparsing==2.2.0
requests==2.13.0
six==1.10.0
Then run instruction would be:
git clone https://github.com/Larz60p/CaliforniaPublicSalaries.git
pip install -r requirements.txt
CaliforniaPublicSalaries\src\>python CaCompGui.py
This is one way of doing it,other more normal way is pip install my_module.
Then setup.py do it all,find and install 3-party requirements.


RE: California Public Salary Files - Larz60+ - Mar-27-2017

As soon as I wake up completely from my nap (I never used to do that),
I am going to follow your advice.
It's high time I learned (and put into practice) how to create packages.
It's been on my list for quite some time.