Skip to content

Use requests-cache for persistent HTTP caching in synchronous Python code

Decided · 29 June 2026

Context

PyLD document loaders fetch remote JSON-LD contexts and other documents over HTTP. JSON-LD Best Practices recommends:

Best Practice 14: Cache JSON-LD Contexts

Services providing a JSON-LD Context SHOULD set HTTP cache-control headers to allow liberal caching of such contexts, and clients SHOULD attempt to use a locally cached version of these documents.

— § 8.1 Cache JSON-LD Contexts

Status Quo

PyLD already caches remote contexts in process memory only. ContextResolver shares a module-level LRUCache of resolved contexts, and ResolvedContext keeps a per-document LRUCache for processed contexts relative to an active context.

FrozenDocumentLoader serves files from disk based on its configuration options set in code.

None of this is HTTP-aware or persistent across processes: RequestsDocumentLoader and AioHttpDocumentLoader issue a fresh HTTP request on every load with no Cache-Control handling.

HTTP Caching Standards

Client-side caching semantics are governed by the following standards:

flowchart LR
  rfc2616("<b>RFC 2616</b><br/>HTTP/1.1")
  rfc7234("<b>RFC 7234</b><br/>HTTP/1.1: Caching")
  rfc9111("<b>RFC 9111</b><br/>HTTP Caching")
  rfc9110("<b>RFC 9110</b><br/>HTTP Semantics")
  rfc5861("<b>RFC 5861</b><br/>Cache-Control Extensions for Stale Content")
  rfc8246("<b>RFC 8246</b><br/>HTTP Immutable Responses")

  rfc2616 -->|is superseded by| rfc7234
  rfc7234 -->|is superseded by| rfc9111
  rfc9110 -->|complements| rfc9111
  rfc5861 -->|complements| rfc9111
  rfc8246 -->|complements| rfc9111

  click rfc2616 "https://www.rfc-editor.org/rfc/rfc2616.html" "RFC 2616"
  click rfc7234 "https://www.rfc-editor.org/rfc/rfc7234.html" "RFC 7234"
  click rfc9111 "https://www.rfc-editor.org/rfc/rfc9111.html" "RFC 9111"
  click rfc9110 "https://www.rfc-editor.org/rfc/rfc9110.html" "RFC 9110"
  click rfc5861 "https://www.rfc-editor.org/rfc/rfc5861.html" "RFC 5861"
  click rfc8246 "https://www.rfc-editor.org/rfc/rfc8246.html" "RFC 8246"

This ADR compares Python HTTP caching libraries that could extend document loaders with standards-aware, persistent caching.

Alternatives Rejected

Library HTTP-aware? Persistent?
tkem/cachetools ❌ ❌
grantjenks/python-diskcache ❌ ✅

Neither library implements HTTP cache semantics — cachetools is a generic in-memory structure and diskcache is persistent but not HTTP-aware — so they do not fit document-loader HTTP caching.

Decision

requests-cache/requests-cache psf/cachecontrol requests-cache/aiohttp-client-cache karpetrosyan/hishel
Backend
psf/requests ✅ ✅ ❌ ✅
encode/httpx ❌ ❌ ❌ ✅
aio-libs/aiohttp ❌ ❌ ✅ ❌
HTTP Headers Support
Cache-Control request Broad directive support. max-age, no-cache, no-store, min-fresh; other directives parsed but not necessarily honored. Simplified: max-age, no-cache, no-store. ✅
Cache-Control response Broad directive support. max-age, no-store, validator-driven caching. Simplified: max-age, no-store. ✅
Expires ✅ ✅ ✅ ✅
ETag / If-None-Match ✅ ✅ ⚠ ✅
Last-Modified / If-Modified-Since ✅ ✅ ⚠ ✅
Vary ✅ ✅ ❌ ✅
Meta (last updated: 28 June 2026)
Last release 1.3.2
2026-05-11		 0.14.4
2025-11-14		 0.14.3
2026-01-07		 1.3.0
2026-06-11
GitHub Stars ⭐ 1,495 ⭐ 499 ⭐ 152 ⭐ 395
Decision ✅ Adopt as optional sync HTTP cache support for RequestsDocumentLoader. ❌ Exclude — shares the requests backend with requests-cache, but defaults to an in-memory store with no first-class persistent backend comparable to requests-cache storage options, offers narrower Cache-Control directive support, and trails on maintenance signals in this comparison. ❓ Defer for async support to limit dependency and maintenance footprint; candidate if AioHttpDocumentLoader gains HTTP caching. ❓ Defer — strong HTTP cache policy design, but adoption would target httpx rather than PyLD's current AioHttpDocumentLoader.

⚠ marks indirect or limited support. aiohttp-client-cache has conditional refresh support for validators, but its own documentation describes cache-header handling as a simplified subset rather than full automatic validator-driven HTTP cache revalidation. For hishel, header rows link to the governing RFC 9111 sections because its default SpecificationPolicy implements the full specification rather than documenting per-header support separately.

Consequences

  • PyLD will add optional, HTTP-aware sync caching via requests-cache without making caching a required dependency.
  • Opt-in sync caching will now be provided by SqliteCacheRequestsDocumentLoader, composing RequestsDocumentLoader with a persistent SQLite CachedSession.
  • A custom in-process HTTP cache implementation is rejected; with a fitting library available, PyLD will integrate requests-cache rather than expand the codebase to implement RFC 9111 caching itself.
  • CacheControl is excluded for the sync requests path in favor of requests-cache.
  • Async HTTP caching remains out of scope for this decision; aiohttp-client-cache and hishel stay as future candidates.