Apr-19-2022, 07:39 AM
Hi,
I'm currently involved in transforming large legacy "databases" (excel, access, csv...)
into something organised and searchable by the public. We are talking millions of genealogy-type records.
The first reaction is to put it all in a few sqlite3 databases by type (Marriages, baptisms...etc)
Unfortunately the database creation , even on a fast SSD is very, very slow.
A few 100.000 records/hour, when not even indexed.
The alternative is to leave everything in smaller csv files, eg. by village. Speedwise it is lightning fast, compared.
So he question is basically, does one have to put up with slow database creation,
or is there something else to try ?
Paul
I'm currently involved in transforming large legacy "databases" (excel, access, csv...)
into something organised and searchable by the public. We are talking millions of genealogy-type records.
The first reaction is to put it all in a few sqlite3 databases by type (Marriages, baptisms...etc)
Unfortunately the database creation , even on a fast SSD is very, very slow.
A few 100.000 records/hour, when not even indexed.
The alternative is to leave everything in smaller csv files, eg. by village. Speedwise it is lightning fast, compared.
So he question is basically, does one have to put up with slow database creation,
or is there something else to try ?
Paul
It is more important to do the right thing, than to do the thing right.(P.Drucker)
Better is the enemy of good. (Montesquieu) = French version for 'kiss'.
Better is the enemy of good. (Montesquieu) = French version for 'kiss'.