Python Forum
Opinion: how should my scripts cache web download files?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Opinion: how should my scripts cache web download files?
#1
I have a script that downloads files from the web. To be a good web citizen, I want to keep a local cache and only re-fetch from the internet if the cache is too old.

I cannot rely on there being a local caching web server.

This is especially important while testing my script, when I might run it dozens of times in a row. I know that the server I am downloading from may block too many requests.

Speaking of requests, for reasons I can only use the stdlib, so no third part solutions, sorry Sad

So my idea is to look for a cached file in a known location:
  • If the file doesn't exist, download from the web.
  • If the file exists, but is older than some amount of time, say X minutes, download from the web.
  • If the file exists, and is younger than X minutes, then use the cached file.

The cached file will have to persist from one run of the script to the next, but it doesn't have to survive rebooting the computer.

Two questions:

  1. Where should I put the cache? Platform independent answers preferred.
  2. What is a reasonable value for X minutes? I'm thinking 15 minutes.

Thanks in advance.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Speeding up code using cache Peter 1 550 Jul-29-2023, 04:52 AM
Last Post: bowlofred
  How to download a list of files from FTP? schnarkle 0 1,016 Jun-21-2022, 10:35 PM
Last Post: schnarkle
  Download mp4 files from an url q988988 2 6,390 Mar-07-2022, 10:11 AM
Last Post: snippsat
  download with internet download manager coral_raha 0 2,967 Jul-18-2021, 03:11 PM
Last Post: coral_raha
  main libvlc error: stale plugins cache: schascheck 2 7,742 Dec-27-2020, 05:24 PM
Last Post: schascheck
  How can I download Python files from GitHub? bitcoin10mil 2 2,851 Aug-26-2020, 09:03 PM
Last Post: Axel_Erfurt
  Clear Cache Path sportcardinal 0 1,629 Jul-05-2020, 05:11 PM
Last Post: sportcardinal
  How to print cache from Decorators with Memoization OlgaM 2 2,079 Jan-29-2020, 05:06 PM
Last Post: OlgaM
  pip cache millpond 3 8,245 Jul-22-2019, 01:12 AM
Last Post: millpond
  python cache for small integer Uchikago 1 2,509 Jun-27-2019, 05:32 PM
Last Post: ichabod801

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020