How to delete portion of file already processed?

[deanhystad]

I think the problem may only be in your mind. A big file likely does not slow things down. It takes longer to process a big file because there is more information to process, but making the file smaller does not reduce the time to process each element/transaction/whatever. Trying to remove the processed information is just going to slow things down because not only are you still doing the work to process the file information, but you are doing additional work to change information about the file, maybe even modifying non-volatile memory which is slow relative to most computer operations.

[bowlofred]

If things are taking too long, you should look into profiling the program. Find out which parts are slow and which parts are not. Only worry about the parts that are slow.

One way is to stub out the real work you're doing. Write a program that reads through the lines like you would and passes them to an empty function in the right order. If you can do that quickly then that part doesn't need to be sped up.

[perfringo]

It always worth of remembering:

Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.

---Donald Knuth

[Pedroski55]

There was another question recently about a file with 2 million rows. Is this related? I was curious how that ended, but I didn't see an echo.

I never had big data to process.

But if your data is split into years, why not just check each year?

An electrician once told me, if you have a broken cable, cut it in half and test each half.

2 possibilities:

1. You cut it exactly where it is broken (unlikely).
2. One half is good.

Don't know what answers you are looking for, but checking slices of the readlines() list may ease the burden.

The original file will always be preserved until and unless you overwrite it: original_file.write(new_data) so just open it 'r'.