When do I need a database?

[Mark17]

I've asked a few people about this and not gotten a good answer. In general, I don't understand how a database can save me time.

I'm looking to build a trade backtester where security prices for every trading day are in a .csv file.

So basically, the program has to import all the .csv files at the very beginning or--I download data into a database once and then it's there for good?

But even in a database, Python still has to query it and in doing so, doesn't it retrieve the data into dataframes? So whether I get data into frames straight from .csv or from database, I don't clearly understand the difference.

One person told me the database takes care of parsing... how so and why?

Someone else told me on top of an already-complicated algo, the database would bog the program down more. Does that make sense or not?

Thanks for any help you can give!

[Larz60+]

I would say that if you have several sources of data (a CSV file for example), and each 'record' (row) containing that data is of the same format (columns), but varying contents (value), and each source is related to one or more of the other sources by at least one value (key). A database will allow exploiting that relationship, and would be a good choice.

[jefsummers]

Example of a program I wrote for a local charity using a database:
When a client arrives they need to sign in. They can update their demographics (one table) and enter why they are here (a second table). That list of options of why they are here is populated by a third table. If their attendance gives them "points" that is registered in yet another table, and if they redeem points that is also posted to that table. And on it goes. There are tables for the volunteers and administrators and times they have volunteered.

All of the client tables are linked by a client ID number. The events of the current visit are linked by another Visit ID number. Pulling this all together is easier with a database, and less likely to get corrupted. There are also privacy concerns that make the database a better option.