Python Forum
Opinion: how should my scripts cache web download files?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Opinion: how should my scripts cache web download files?
#1
I have a script that downloads files from the web. To be a good web citizen, I want to keep a local cache and only re-fetch from the internet if the cache is too old.

I cannot rely on there being a local caching web server.

This is especially important while testing my script, when I might run it dozens of times in a row. I know that the server I am downloading from may block too many requests.

Speaking of requests, for reasons I can only use the stdlib, so no third part solutions, sorry Sad

So my idea is to look for a cached file in a known location:
  • If the file doesn't exist, download from the web.
  • If the file exists, but is older than some amount of time, say X minutes, download from the web.
  • If the file exists, and is younger than X minutes, then use the cached file.

The cached file will have to persist from one run of the script to the next, but it doesn't have to survive rebooting the computer.

Two questions:

  1. Where should I put the cache? Platform independent answers preferred.
  2. What is a reasonable value for X minutes? I'm thinking 15 minutes.

Thanks in advance.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Speeding up code using cache Peter 1 565 Jul-29-2023, 04:52 AM
Last Post: bowlofred
  How to download a list of files from FTP? schnarkle 0 1,028 Jun-21-2022, 10:35 PM
Last Post: schnarkle
  Download mp4 files from an url q988988 2 6,458 Mar-07-2022, 10:11 AM
Last Post: snippsat
  download with internet download manager coral_raha 0 3,003 Jul-18-2021, 03:11 PM
Last Post: coral_raha
  main libvlc error: stale plugins cache: schascheck 2 7,767 Dec-27-2020, 05:24 PM
Last Post: schascheck
  How can I download Python files from GitHub? bitcoin10mil 2 2,871 Aug-26-2020, 09:03 PM
Last Post: Axel_Erfurt
  Clear Cache Path sportcardinal 0 1,639 Jul-05-2020, 05:11 PM
Last Post: sportcardinal
  How to print cache from Decorators with Memoization OlgaM 2 2,107 Jan-29-2020, 05:06 PM
Last Post: OlgaM
  pip cache millpond 3 8,260 Jul-22-2019, 01:12 AM
Last Post: millpond
  python cache for small integer Uchikago 1 2,524 Jun-27-2019, 05:32 PM
Last Post: ichabod801

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020