Dec-29-2017, 07:34 PM
http://aripollak.com/pythongzipbenchmarks/
Looks like the speed depends pretty heavily on which version of python you're running. You might also gain some improvement by wrapping the gzip object in io.BufferedReader.
I wouldn't mind seeing more of your code, though, as 45minutes for 1gb sounds excessive. Depending on what you're doing (and the power of the computer it's running on), maybe we can create a process queue and take advantage of multiple cores/processors.
Looks like the speed depends pretty heavily on which version of python you're running. You might also gain some improvement by wrapping the gzip object in io.BufferedReader.
I wouldn't mind seeing more of your code, though, as 45minutes for 1gb sounds excessive. Depending on what you're doing (and the power of the computer it's running on), maybe we can create a process queue and take advantage of multiple cores/processors.
