Need Direction

[illmattic]

Hello everyone,

I just started with python a month ago and it's the first computer language I've attempted to learn. I've gone through the basics and worked with such modules as pandas, requests, beautifulsoup and matplotlib. My goal is to create a monthly economic database that automates the process of pulling the data from the web. Can someone point me in the direction on how to do this best so that I don't waste time going down hopeless routes. As of now my idea would be to set up a SQLite database by querying data from the web using requests and beautifulsoup. Am I right to think this?

Thank you
Matt

[Larz60+]

You might want to take a look at some of the packages available: https://pypi.org/search/?q=%22yahoo+quotes%22&o=

[illmattic]

You might want to take a look at some of the packages available: https://pypi.org/search/?q=%22yahoo+quotes%22&o=

Thanks I'll use this. But as an overall framework what would be the best approach to creating a database, for example, of US economic data that includes GDP, unemployment, inflation etc that updates monthly.

Thank you

[BitPythoner]

This is an interesting idea, you would need to pull information from certain .gov websites, maybe using BeautifulSoup, than output it with some kind of graph. You could create a database with something like this: https://docs.python.org/3/library/sqlite3.html#module-sqlite3.

---

If your program is to run once a month, you would want to use an external program such as task scheduler in windows.

[Larz60+]

There are many ways to go about creating a database.
Formally, you should probably use a UML (Universal Modeling Langiage).
There are many that are available for free, the most common,in order of popularity are:

• Eclipse Papyrus
  
• PlantUML
  
• Violet
  
• Umple
  
• Umbrello
  
• Modelio
  
• ArgoUML
  
• UMLet
  
• Dia
  
• UML DEsigner
  

Personally, unless creating a project for a customer, I bypass all of that in favor of
writing a Python SQL_Alchemy Model.
This is pure python code, and once you create a model, you can run the code and have it create the database automatically
What's mode, you can then import the module into programs that require use of the database, using a simplified version
of query, called SQLAlchemy ORM. You can also create an unpopulated mirror of your database in another DBMS with a single line of code change.
Updates to the database are a snap using the built-in migrate methods.
If interested in this method, see: https://python-forum.io/Thread-SqlAlchem...-Data-Load
This is a simple tutorial which I wrote, and is on this forum.

And there's the seat of the pants method:
Open an SQLite designer ( https://sqlitebrowser.org/ ) and start creating your database table by table.
Using this method, I would at least first determine what tables were required for my application, and writing down a simple list of what's needed in each table, and what the links (keys) are going to be that link (relate) one table to another.

[illmattic]

There are many ways to go about creating a database.
Formally, you should probably use a UML (Universal Modeling Langiage).
There are many that are available for free, the most common,in order of popularity are:

• Eclipse Papyrus
  
• PlantUML
  
• Violet
  
• Umple
  
• Umbrello
  
• Modelio
  
• ArgoUML
  
• UMLet
  
• Dia
  
• UML DEsigner
  

Personally, unless creating a project for a customer, I bypass all of that in favor of
writing a Python SQL_Alchemy Model.
This is pure python code, and once you create a model, you can run the code and have it create the database automatically
What's mode, you can then import the module into programs that require use of the database, using a simplified version
of query, called SQLAlchemy ORM. You can also create an unpopulated mirror of your database in another DBMS with a single line of code change.
Updates to the database are a snap using the built-in migrate methods.
If interested in this method, see: https://python-forum.io/Thread-SqlAlchem...-Data-Load
This is a simple tutorial which I wrote, and is on this forum.

And there's the seat of the pants method:
Open an SQLite designer ( https://sqlitebrowser.org/ ) and start creating your database table by table.
Using this method, I would at least first determine what tables were required for my application, and writing down a simple list of what's needed in each table, and what the links (keys) are going to be that link (relate) one table to another.

Thanks a lot!
Looking over the coding from your SQLAlchemy post, I think the 'seat of the pants' method would be more suitable for someone at my level. I started building the database in excel but came to realize that python was more efficient at handling and processing data so I decided to dedicate my time to learning python.