Python Forum

Hello,

I have created a Json file that has links to the Open Data site for each US State.
It also has a few cities and departments and I will be adding more from time to time.


The program will create the Json file when run.

It is also available on Github
[url=https://github.com/Larz60p/StateOpenData][/url]

import json


class MakeOpenDataJSON:
    def __init__(self):
        self.jfilename = 'StateOpenData.JSON'
        self.OpenData = {
            'Alabama': {
                'state': 'http://open.alabama.gov/'
            },
            'Alaska': {
                'state': 'None',
                'DeptNaturalResources': 'http://www.asgdc.state.ak.us/'
            },
            'Arizona': {
                'state': 'https://ptl.az.gov/app/transparency/index.html'
            },
            'Arkansas': {
                'state': 'http://transparency.arkansas.gov/'
            },
            'California': {
                'state': 'https://data.ca.gov/',
                'San Diego': 'http://data.sandiego.gov/',
                'San Francisco': 'https://data.sfgov.org/'
            },
            'Colorado': {
                'state': 'https://www.colorado.gov/data/'
            },
            'Connecticut': {
                'state': 'https://data.ct.gov/'
            },
            'Delaware': {
                'state': 'https://data.delaware.gov/',
                'topics': 'http://delaware.gov/topics/data.shtml',
            },
            'District Of Columbia': {
                'state': 'http://opendata.dc.gov/'
            },
            'Florida': {
                'state': 'http://geodata.myflorida.com/',
                'Right To Know': 'http://floridahasarighttoknow.myflorida.com/'
            },
            'Georgia': {
                'state': 'http://www.open.georgia.gov/'
            },
            'Hawaii': {
                'state': 'https://data.hawaii.gov/',
                'Honolulu Land Information System': 'http://gis.hicentral.com/'
            },
            'Idaho': {
                'state': 'http://data.gis.idaho.gov/'
            },
            'Illinois': {
                'state': 'https://data.illinois.gov/',
                'South Suburban Mayors and Managers': 'https://data.illinois.gov/ssmma'
            },
            'Indiana': {
                'state': 'http://www.stats.indiana.edu/'
            },
            'Iowa': {
                'state': 'https://data.iowa.gov/'
            },
            'Kansas': {
                'state': 'http://www.kansas.gov/KanView/'
            },
            'Kentucky': {
                'state': 'http://opendoor.ky.gov/Pages/default.aspx'
            },
            'Louisiana': {
                'state': 'https://wwwcfprd.doa.louisiana.gov/latrac/portal.cfm'
            },
            'Maine': {
                'state': 'https://data.maine.gov/'
            },
            'Maryland': {
                'state': 'https://data.maryland.gov/'
            },
            'Massachusetts': {
                'state': 'http://massbigdata.org/data/the-massachusetts-open-data-catalog/',
                'Boston': 'https://data.cityofboston.gov/',
                'Cambridge': 'https://data.cambridgema.gov/',
                'Framingham': 'http://www.civicdashboards.com/city/framingham-ma-16000US2524960/'
            },
            'Michigan': {
                'state': 'https://data.michigan.gov/'
            },
            'Minnesota': {
                'state': 'http://mn.gov/portal/'
            },
            'Mississippi': {
                'state': 'https://www.arcgis.com/home/item.html?id=a266920482b7486baac8f591a63f19a4'
            },
            'Missouri': {
                'state': 'https://data.mo.gov/'
            },
            'Montana': {
                'state': 'https://data.datamontana.us/browse'
            },
            'Nebraska': {
                'state': 'http://www.nebraska.gov/'
            },
            'Nevada': {
                'state': 'http://open.nv.gov/'
            },
            'New Hampshire': {
                'state': 'https://www.nh.gov/doit/open-source/'
            },
            'New Jersey': {
                'state': 'https://data.nj.gov/'
            },
            'New Mexico': {
                'state': 'http://www.sunshineportalnm.com/'
            },
            'New York': {
                'state': 'https://nycopendata.socrata.com/',
                'NY Dept Of Health': 'https://health.data.ny.gov/'
            },
            'North Carolina': {
                'state': 'https://open-nc.org/',
                'Asheville': 'http://data.ashevillenc.gov/',
                'Cary': 'https://data.townofcary.org/page/home/',
                'Chapel Hill': 'https://www.chapelhillopendata.org/explore/',
                'Charlotte': 'http://clt-charlotte.opendata.arcgis.com/',
                'Durham': 'https://opendurham.nc.gov/page/home/',
                'Fayetteville': 'http://data.fayettevillenc.gov/',
                'Greensboro': 'https://data.greensboro-nc.gov/',
                'Raleigh': 'https://localwiki.org/raleigh/Open_Data_Sources',
                'WakeCounty': 'http://data-wake.opendata.arcgis.com/',
                'GISresources': 'http://www.cgia.state.nc.us/dataresources.aspx',
                'Mecklenburg Open Mapping': 'http://maps.co.mecklenburg.nc.us/openmapping/',
                'OpenDataInstitute Of NC': 'https://www.slideshare.net/TheODINC',
                'State Data Center NC OSBM': 'https://www.osbm.nc.gov/facts-figures/state-data-center'
            },
            'North Dakota': {
                'state': 'None',
                'GIS North Dakota ITD': 'https://www.nd.gov/itd/statewide-alliances/gis'
            },
            'Ohio': {
                'state': 'http://www.ohio.gov/government/transparency/'
            },
            'Oklahoma': {
                'state': 'https://www.ok.gov/about/data.html'
            },
            'Oregon': {
                'state': 'https://data.oregon.gov/'
            },
            'Pennsylvania': {
                'state': 'https://data.pa.gov/',
                'Phidelphia': 'https://www.opendataphilly.org/'
            },
            'Rhode Island': {
                'state': 'http://www.ri.gov/data/'
            },
            'South Carolina': {
                'state': 'http://www.gis.sc.gov/'
            },
            'South Dakota': {
                'state': 'http://open.sd.gov/'
            },
            'Tennessee': {
                'state': 'http://tn.gov/transparenttn'
            },
            'Texas': {
                'state': 'https://data.texas.gov/'
            },
            'Utah': {
                'state': 'https://www.utah.gov/data/'
            },
            'Vermont': {
                'state': 'https://data.vermont.gov/'
            },
            'Virginia': {
                'state': 'https://data.virginia.gov/'
            },
            'Washington': {
                'state': 'https://data.wa.gov/'
            },
            'West Virginia': {
                'state': 'None',
                'Civic Dashboard': 'http://catalog.opendata.city/dataset?license_id=odc-by&tags=West+Virginia',
                'Map West Virginia': 'http://www.mapwv.gov/help-gisdata.php'
            },
            'Wisconsin': {
                'state': 'None',
                'Madison': 'https://data.cityofmadison.com/'
            },
            'Wyoming': {
                'state': 'None',
                'National And Community Service': 'https://data.nationalservice.gov/Volunteering-and-Civic-Engagement/Wyoming/bpid-mwn8/data'
            }
        }
        self.make_file()

    def make_file(self):
        with open(self.jfilename, "w") as f:
            json.dump(self.OpenData, f)


if __name__ == '__main__':
    MakeOpenDataJSON()

With this, for example, you can scrape for lists of restaurants, etc. Boston has a list with latitude  and longitude of about 3000.


Have fun

if the data exists in code, why bother making a JSON file?  why not just make it be a module?  then to load the data is as simple as loading a module and calling the appropriate method or accessing the appropriate attribute.

The json file loads with a very small amount of code:

        with open(self.json_filename, encoding="utf8") as f:

            self.urltree = json.load(f)

Reasons for a json file:

• File load is very fast
  
• The file contents can only be modified if there is intention to do so, whereas if part of the program, can be unintentionally modified.
  
• The source is in one place, so if the file is kept in one place, then all programs using it will get a fresh copy whenever it is updated.
  
• Once loaded, you automatically have a copy of the dictionary in memory.
  
• I'm sure if I thought long enough I could come up with many more reasons.
  

(Mar-05-2017, 07:20 AM)Larz60+ Wrote: [ -> ]Reasons for a json file:

• File load is very fast
  
• The file contents can only be modified if there is intention to do so, whereas if part of the program, can be unintentionally modified.
  
• The source is in one place, so if the file is kept in one place, then all programs using it will get a fresh copy whenever it is updated.
  
• Once loaded, you automatically have a copy of the dictionary in memory.
  
• I'm sure if I thought long enough I could come up with many more reasons.
  

i think you are assuming i mean to insert data code into the program that reads it.  my idea is to make data of appropriate size (it is ok to hold it all in memory for all expected users) into a module, when the data source is suitable to do this.  in the case of your data, it is small enough (your JSON code loads it all).  next concern is speed.  is loading a module faster or slower than an equal amount of data in JSON format.

one advantage of a module is that it is an appropriate place to include methods for working with the data.

here is a very small one i did:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""Amazon Web Services (AWS) instance types"""
version_info = (2017,2,14,0,0,0)
instance_type_list = [
    'i2.xlarge','i2.2xlarge','i2.4xlarge','i2.8xlarge',
    'hi1.4xlarge',
    'hs1.8xlarge',
    'c1.medium','c1.xlarge',
    'c3.large','c3.xlarge','c3.2xlarge','c3.4xlarge','c3.8xlarge',
    'c4.large','c4.xlarge','c4.2xlarge','c4.4xlarge','c4.8xlarge',
    'cc1.4xlarge','cc2.8xlarge',
    'cr1.8xlarge',
    'cg1.4xlarge',
    'g2.2xlarge',
    'r3.large','r3.xlarge','r3.2xlarge','r3.4xlarge','r3.8xlarge',
    'd2.xlarge','d2.2xlarge','d2.4xlarge','d2.8xlarge',
    't1.micro',
    'm1.small','m1.medium','m1.large','m1.xlarge',
    'm3.medium','m3.large','m3.xlarge','m3.2xlarge',
    'm4.large','m4.xlarge','m4.2xlarge','m4.4xlarge','m4.10xlarge',
    't2.nano','t2.micro','t2.small','t2.medium','t2.large',
    'm2.xlarge','m2.2xlarge','m2.4xlarge',
]
instance_type_set = set(instance_type_list)

I assume nothing about what you do or intend to do. You can do as you wish.
I do what works best for me, and I am sharing what I have.
Nothing more and nothing less.

If you put your data into your Python code, your Python code can access it easily. If you put your data into a standard data format, like json, xml, or csv, then all of your code (Python, R, C++, ...) and several of your applications can access it easily.