# Cache implementation (developer reference)
This is the low-level, contributor-facing companion to the user-facing
[Caching and data freshness](../explanation/caching.md) page. It
documents *how* the cache is built: the on-disk layout, the freshness
model, the fetch-decision flow, negative caching, and the optional
server (L2) tier. If you are changing anything in `src/cache/store.zig`
or the fetch paths in `src/service.zig`, read this first.
Diagrams use [Mermaid](https://mermaid.js.org/), which Forgejo renders
natively.
## Where the data lives
`DataService` (`src/service.zig`) is the sole data-access layer. Both
the CLI and the TUI go through it; nothing else calls a provider
directly. It reads and writes a per-symbol, per-type SRF file cache via
`Store` (`src/cache/store.zig`):
```
{cache_dir}/ default ~/.cache/zfin, set by ZFIN_CACHE_DIR
AAPL/
candles_daily.srf OHLCV bars (appended; replaced on restatement)
candles_meta.srf last_close, last_date, provider, adj_basis + freshness
dividends.srf
splits.srf
options.srf
earnings.srf
classification.srf
etf_metrics.srf
_edgar/
tickers_companies.srf shared EDGAR ticker -> CIK maps
tickers_funds.srf
0000320193/
entity_facts.srf per-CIK XBRL facts
```
[SRF](https://git.lerch.org/lobo/srf) is a line-oriented key-value
format. Files carry `#!`-prefixed directives (`#!expires=`,
`#!created=`) ahead of their records.
### The candle two-file split
Candles are stored as **two** files, and the split is load-bearing:
- `candles_daily.srf` holds the actual OHLCV records and normally grows
by appending: on a cache miss only bars newer than `last_date` are
fetched and appended, not the full history. The exception is a
**restatement**, which replaces the file wholesale - see
[The adjustment basis](#the-adjustment-basis).
- `candles_meta.srf` holds a single small record (`last_close`,
`last_date`, `provider`, `fail_count`, `tiingo_retry_after_s`,
`adj_basis`) plus the `#!expires=` and `#!created=` directives.
Keeping the metadata separate lets every freshness check and last-price
read touch a ~100-byte file instead of deserializing a multi-megabyte
candle history. The price fast-path in `loadAllPrices` never
deserializes `candles_daily.srf` - at most it peeks at the first bytes
to detect a negative entry.
The two files are a unit: `DataService.invalidate` and the torn-file
self-heal clear both together, and a negative cache entry for a
candle-less symbol is keyed off `candles_daily.srf` (see
[Negative caching](#negative-caching)).
### The adjustment basis
Appending bars cannot restate the bars already on disk, and sometimes
they need it.
Providers compute `adj_close` by scaling raw `close` by the product of
the adjustment factors for every distribution *after* that bar. So a
series' adjustment basis is only as current as the fetch that produced
it. Newly appended bars arrive with `adj_close == close`, because
nothing has gone ex after them yet, while every previously cached bar
keeps the basis it was originally fetched with. When the next
distribution goes ex, the bars behind it should be marked down by its
factor - and an append does not do that. Every total return spanning
that ex-date then reads low by roughly the missed yield.
`CandleMeta.adj_basis` records how current a series' basis is. It is
**the date of the newest bar present at the last full fetch**:
- Not a wall-clock timestamp. A fetch that runs before the day's bar is
published gets a basis one bar behind - which is the honest claim,
since a provider's adjusted series is only ever as complete as its
newest bar.
- Not an ex-date. It is compared against ex-dates but is never one.
- Never advanced by an append. `Store.appendCandles` takes the existing
meta and overrides only `last_close` / `last_date`; only
`Store.cacheCandles`, which replaces the whole file, sets a new
basis. That asymmetry is the mechanism, so keep it.
`getCandles` escalates from append to full refetch when
`freshness.adjustmentBasisStale` says the basis predates a corporate
action that has **already gone ex**:
```zig
newest_ex = freshness.newestCorporateAction(alloc, store, sym, meta.last_date)
stale = freshness.adjustmentBasisStale(meta.adj_basis, newest_ex)
```
The already-ex bound lives in `newestCorporateAction`, not in the
verdict, and that placement is load-bearing in both directions. A
declared-but-not-yet-ex distribution is reflected in no provider's
adjusted series, so treating it as something to catch up to would
refetch on every pass forever. But bounding the *verdict* instead -
"is the newest action of all still in the future? then nothing to do" -
lets one forward announcement hide every older unapplied action behind
it. That shipped: a quarterly payer announcing a quarter ahead was
permanently unable to restate.
The check runs on the stale path only, not on the fresh-cache
early-return. Detection is therefore at most one trading day behind,
which the candle TTL guarantees, and the hot portfolio-pricing path
pays nothing.
Steady state for a quarterly payer is four full-history fetches a year,
each landing within about a trading day of an ex-date. `zfin diagnose
SYMBOL` reports the basis against the newest already-ex action.
## Freshness is the `#!expires=` directive, not mtime
A cache entry is fresh when the wall clock is earlier than the
`#!expires=` epoch-seconds directive embedded in the file. File
modification time is **not** consulted for freshness anywhere - mtime is
a fragile signal (it changes on copy, restore, rsync, and filesystem
quirks), so the expiry is written into the content itself.
- On write, `computeExpires` (`store.zig`) sets `#!expires = now + TTL`
for the data type, optionally offset by a per-key deterministic jitter
to avoid thundering-herd refreshes.
- On read, the SRF iterator parses `#!expires=` and `isFresh` compares
it to `Timestamp.now(io, .real)`.
- An entry with **no** `#!expires=` is treated as stale by zfin's
`.fresh_only` reads (a deliberate override of SRF's "no expiry =
always fresh" default), except for negative entries, which are always
fresh.
### TTLs by data type
Base TTLs live in `Ttl` (`store.zig`); jitter is applied per call site
in `DataType.ttl()`.
| Data type | TTL | Jitter | Notes |
|----------------------|------------------|--------|--------------------------------------------------|
| Daily candles | market-aware | n/a | Boundary set by `market.nextCandleExpiry` (below)|
| Historical candles | never (`-1`) | n/a | Bars older than ~1 day are immutable |
| Dividends | 14 days | 11% | Declared well in advance |
| Splits | 14 days | 11% | Rare corporate events |
| Options | 1 hour | 0 | Move continuously during market hours |
| Earnings | 30 days | 8% | Smart-refresh after an announcement date passes |
| Classification | 90 days | 8% | Sector/industry/country from Wikidata |
| ETF metrics | 90 days | 8% | NPORT-P profile, quarterly cadence |
| Entity facts (XBRL) | 30 days | 8% | Per-CIK, quarterly filing cadence |
| EDGAR ticker maps | 30 days | 8% | ticker -> CIK; very stable upstream |
| Quotes | never cached | n/a | Live by definition (see below) |
### Market-aware candle freshness
Daily bars are only meaningful once the session settles, so candle
expiry is keyed to the market clock rather than a rolling window
(`market.nextCandleExpiry` / `market.staleCandleExpiry`,
`market.shouldRefresh`):
- Equities/ETFs expire at **16:55 ET** on the next trading day.
- Mutual funds (NAV) expire at **03:25 ET** the next morning.
If a refresh fires but the provider has not posted the just-closed bar
yet, the entry retries in ~30 minutes; once a due bar is ~90 minutes
overdue, the code concludes the session was an un-modeled closure (Good
Friday, weather) and falls back to the next normal boundary instead of
thrashing all day. See the user page for the cron-timing rationale.
## The fetch decision
### Tiers (the big picture)
```mermaid
flowchart TD
A["Data request via DataService"] --> B{"Local cache fresh?"}
B -->|yes| C["Deserialize and return, no network"]
B -->|no| D{"ZFIN_SERVER set and not force_refresh?"}
D -->|yes| E["GET server, write bytes verbatim"]
E --> F{"Synced entry fresh?"}
F -->|yes| C
F -->|no| G["Provider fetch"]
D -->|no| G
G --> H{"Result?"}
H -->|ok| I["Write cache with new expiry, return"]
H -->|NotFound| J["Write negative cache, return FetchFailed"]
H -->|transient| K["Return error, retry next run"]
```
The `--refresh-data` policy maps to `FetchOptions`:
- `auto` (default): all tiers, honor TTL.
- `force` -> `force_refresh = true`: skip the local-cache and server
tiers, go straight to the provider, re-stamp the cache. Bypasses
negative entries (so it retries dead lookups).
- `never` -> `skip_network = true`: stop at the local cache; return
stale data if present, never touch the network.
### `getCandles` (single symbol)
This is the most involved path because of the daily/meta split, the
incremental-update logic, the adjustment-basis escalation, and the
TwelveData carve-out.
Note also that `force` re-asks the provider but does **not** rebuild
candle history: it skips the TTL and the server tier, then takes the
same incremental top-up. Replacing the series is the restatement path's
job (or `zfin cache clear`).
Provider routing is keyed off `CandleMeta.tiingo_retry_after_s`, not off
`provider`. `provider` is pure provenance - "where did these bars come
from" - and using it to route made a single non-transient Tiingo failure
permanent: Yahoo got tried first, succeeded, rewrote `provider = .yahoo`,
and Tiingo was never consulted again. The backoff is armed only by a
genuine 404, expires after `Ttl.tiingo_backoff` with per-symbol jitter,
and clears the moment Tiingo serves the symbol again.
`CandleProvider` has four variants - `twelvedata`, `yahoo`, `tiingo`,
`external` - and exactly one of them is special-cased anywhere:
`twelvedata`, whose `adj_close` was unreliable, so a cache carrying it is
treated as unusable. That check exists in **two** places, one per entry
into the meta-exists branch (the `skip_network` path reports the symbol
unavailable; the online path forces a full re-fetch). `external` is
covered under [Externally-managed candle
series](#externally-managed-candle-series); it is inert on every serve
path and vetoes only the negative-cache write.
```mermaid
flowchart TD
S["getCandles(symbol, opts)"] --> NG{"negative candles_daily and not force_refresh?"}
NG -->|yes| FF["return FetchFailed, no network"]
NG -->|no| RM{"candles_meta exists?"}
RM -->|yes| SK{"skip_network?"}
SK -->|yes| TWS{"provider is twelvedata?"}
TWS -->|yes| FF
TWS -->|no| RETS["return cached even if stale
(FetchFailed if unreadable)"]
SK -->|no| TW{"provider is twelvedata?"}
TW -->|yes| FULL
TW -->|no| FR{"meta fresh and not force_refresh?"}
FR -->|yes| RET["return cached candles"]
FR -->|no| SS1["syncCandlesFromServer"]
SS1 --> SF1{"fresh AND adj_basis current?"}
SF1 -->|yes| RET
SF1 -->|no| AB{"adj_basis predates an already-ex action?"}
AB -->|yes| REST["refetchFullHistory: restate whole series"]
REST --> RR{"ok?"}
RR -->|yes| RET2["return fetched"]
RR -->|no| INC
AB -->|no| INC{"shouldRefresh?"}
INC -->|no| BUMP["bump TTL, return cached"]
INC -->|yes| INCF["incremental fetch from last_date+1, appendCandles"]
RM -->|no| SN{"skip_network?"}
SN -->|yes| FF
SN -->|no| SS2["syncCandlesFromServer"]
SS2 --> SF2{"fresh now?"}
SF2 -->|yes| RET
SF2 -->|no| FULL["refetchFullHistory: Tiingo, then Yahoo"]
FULL --> RES{"result?"}
RES -->|ok| RET2
RES -->|"NotFound (EVERY provider disclaims it)"| SNC{"prior meta says provider is external?"}
SNC -->|yes| FF
SNC -->|no| WN["writeNegative candles_daily"]
WN --> FF
RES -->|transient| TR["bump fail_count, TransientError"]
RES -->|other| FF
```
Two things about this shape are easy to get wrong.
**The basis check precedes `shouldRefresh`.** A symbol that needs both a
top-up and a restatement costs one full fetch, not an append followed by
a second pass. And a failed restatement falls through to the ordinary
top-up rather than erroring: the existing series is untouched and still
usable, just understated by the missed adjustment, so it retries on the
next stale pass.
**Only a unanimous `NotFound` earns a negative entry.** `writeNegative`
*overwrites* `candles_daily.srf` with a marker, so a verdict of "no such
symbol" from Tiingo alone must not reach it - Yahoo gets asked first, and
`refetchFullHistory` returns `error.NotFound` only when every provider
disclaims the symbol. Anything else (auth trouble, a malformed body, a
network blip) fails the call but leaves the cache alone. The restatement
path above never writes a negative entry at all, for the same reason: it
is reached while holding a working series.
The no-prior-cache branch does not re-check the basis after a server
sync, only freshness. A stale basis inherited from the server is caught
on the next invocation, which takes the meta-exists branch. One
invocation of understated returns, then it self-corrects.
Key invariant: the negative marker for a candle-less symbol lives in
`candles_daily.srf`, and **every** candle decision honors it there -
`isCandleMetaFresh` (the price fast-path gate), `getCachedCandles` (the
cache-only display path), and the `getCandles` short-circuit above. This
matters because `candles_meta.srf` is never created for a symbol that
has no candles, so anything keying freshness off the meta file alone
would treat such a symbol as perpetually stale and re-fetch it forever.
### `fetchCached` (dividends, splits, options, earnings, ...)
Everything that is not candles flows through the generic `fetchCached`,
which is simpler because each type is a single file:
1. `.fresh_only` read; a fresh entry (including a negative one) returns
immediately.
2. `skip_network`: return any cached entry, even stale; else
`FetchFailed`.
3. Server sync (if configured); a fresh synced entry returns.
4. Provider fetch; on success write with the type's TTL; on
`NotFound` write a negative entry; on transient error return
`FetchFailed` without poisoning the cache.
### `loadAllPrices` (portfolio + watchlist price load)
The portfolio price load batches all symbols through three phases.
Phase 2 is the parallel server sync; Phase 3 is the per-symbol provider
fallback that calls `getCandles`.
```mermaid
flowchart TD
ST["loadAllPrices(portfolio + watch syms)"] --> P1["Phase 1: per symbol"]
P1 --> CF{"cache fresh and not force_refresh?"}
CF -->|yes| HIT["use cached last close, cached_count++"]
CF -->|no| ADD["add to needs_fetch"]
HIT --> CHK
ADD --> CHK{"needs_fetch empty?"}
CHK -->|yes| DONE["return"]
CHK -->|no| OFF{"skip_network?"}
OFF -->|yes| STALE["stale-cache fallback or failed_count++"]
OFF -->|no| HASSRV{"ZFIN_SERVER set?"}
HASSRV -->|yes| P2["Phase 2: parallelServerSync"]
HASSRV -->|no| ALLF["all needs_fetch to server_failures"]
P2 --> REM["unsynced to server_failures"]
ALLF --> P3
REM --> P3["Phase 3: sequentialProviderFetch, getCandles each"]
P3 --> END["return prices + counts"]
```
## Negative caching
When a provider says a symbol genuinely has no data of a type - an
`error.NotFound` - zfin writes a **negative cache entry** so it does not
re-run the dead lookup on every invocation. The entry is the sentinel:
```
#!srfv1
# fetch_failed
```
(`Store.negative_cache_content`). Rules:
- **Only `NotFound` qualifies.** `isPermanentProviderFailure` gates the
write. Rate-limit, 5xx, connection, auth, and parse failures are
transient - they fail the call but leave the cache untouched so the
next run retries. (Auth/parse looking permanent but being transient is
exactly why they must not poison a now-sticky negative cache.)
- **Negative entries are always fresh.** They have no `#!expires=`;
`readSlice`, `read`, and `isCandleMetaFresh` special-case the sentinel
as fresh, so they stick until `--refresh-data=force` or `cache clear`.
- **Candles key the negative off `candles_daily.srf`.** `writeNegative`
writes that file; `isCandleMetaFresh`, `getCachedCandles`, and the
`getCandles` short-circuit all recognize it there. `candles_meta.srf`
is intentionally not created for a no-data symbol.
- **Externally-managed series are exempt.** A prior meta saying
`provider::external` vetoes the write via
`DataService.shouldNegativeCache`, because for a series no provider was
ever going to carry, the marker is unrecoverable rather than merely
sticky. See [Externally-managed candle
series](#externally-managed-candle-series).
## Candle-less symbols (crypto and friends)
Some held symbols have **no daily candles available from any candle
provider** - cryptocurrencies on the Yahoo `DOGE-USD` /
`BTC-USD` shape are the common case, and delisted or invalid tickers
behave identically. For these symbols `getCandles` writes a negative
entry and never produces a price from history. "Any" is literal: Tiingo
and Yahoo must both disclaim the symbol, because the marker overwrites
`candles_daily.srf`.
Such symbols are still priced, through two mechanisms that do **not**
touch the candle cache:
- **Live quotes (Yahoo).** `loadLiveQuotes` / `getQuote` fetch an
intraday price from Yahoo, which *does* serve crypto. The TUI overlays
these live quotes on top of the candle-close price map on refresh and
on every streaming tick, so a candle-less holding shows its real
current price there. Live quotes are never cached.
- **Manual price.** A `price::` field on a lot in `portfolio.srf` pins a
value. When neither a candle close nor a live quote is available,
`buildFallbackPrices` (`analytics/valuation.zig`) falls back to the
position's average cost and flags it as a manual/estimated price
(rendered in a warning color).
Practical consequence by surface: the plain CLI `portfolio` command does
not apply the live-quote overlay, so a candle-less holding shows its
average-cost fallback (break-even, warning color); the TUI shows the
live Yahoo price. This is expected - historical-candle commands (`perf`,
charts) simply have no data for these symbols, while quote-driven views
do.
If you want a candle-less symbol to be re-checked against the provider
(for example a ticker that has since started trading), clear its
negative entry with `cache clear` or `--refresh-data=force`; the live
quote and manual-price paths are unaffected by the negative cache.
## Externally-managed candle series
`CandleMeta.provider == .external` labels a series that was produced and
is maintained **outside zfin**, by something that writes
`candles_daily.srf` and `candles_meta.srf` directly into a cache
directory. The motivating case is a unitized trust with no ticker and no
CUSIP, whose daily unit values come from a plan recordkeeper's feed;
every market-data provider 404s it.
Like the previous section, this is a symbol no provider carries. The
difference is that a candle-less symbol has no history *anywhere*, so
negative-caching it is correct, whereas an externally-managed symbol has
a real series that simply did not come from a provider. The label exists
so `candles_meta.srf` can say so honestly instead of naming a provider
that never served it.
Two population routes:
- **Server**: a job writes the pair straight into the `ZFIN_SERVER`
cache directory. That copy is the master.
- **Client**: the existing server-sync path (phase 2 of `loadAllPrices`,
and the two `syncCandlesFromServer` calls inside `getCandles`) pulls
the bytes down verbatim, exactly as it does for any other symbol. No
client-side special-casing is involved.
The label is **pure provenance and drives no routing**. zfin still walks
the normal provider chain for such a symbol and still 404s on every pass;
restricting it to the server tier and skipping the chain entirely is
deferred work. There is exactly one behavior attached to the label, and
it is a veto rather than a routing decision:
**A negative-cache entry on an external symbol is unrecoverable.**
`writeNegative` overwrites `candles_daily.srf` with a marker, negative
entries never expire, and `getCandles` short-circuits on `isNegative`
*before* it reaches `syncCandlesFromServer` - so the marker makes the
only surviving copy permanently unreachable. On a server it is worse:
the externally-populated cache is the master copy, so nothing survives.
`DataService.shouldNegativeCache` therefore refuses the write when the
prior meta says `.external`.
That guard is deliberately partial. It consults the prior meta, so it
covers the case that destroys data: meta present, `candles_daily.srf`
missing or unreadable (a partially-completed server sync writes the two
files in sequence and needs both to report success), which falls through
to the cold-start path. It cannot cover a **true cold start** with both
files absent - there is no label to read - so an external symbol first
touched while `ZFIN_SERVER` is unreachable is poisoned until
`--refresh-data=force` or `cache clear`. Closing that needs the
`isNegative` short-circuit moved after the server tier, which would
re-hit the network for every legitimately candle-less symbol on every
run. It belongs with the deferred routing work.
### Adding a `CandleProvider` variant is a coordinated deployment
Not a backward-compatible change. `provider` has no default, so it is
never elided, and SRF's enum coercion has no lenient mode and no
`srfParse` hook (custom parsers are consulted for struct/union fields
only). A build that does not know a variant cannot fall back:
- srf `>= ea2c358` (zfin's current pin) returns
`StringValueNotValidEnumMember`. `readCandleMeta` swallows it into a
`null`, which reads as a cache miss and routes the symbol down the
cold-start path above - destructive for precisely the series that
cannot be re-fetched.
- srf `<= 4a3e5f0` unwraps a null optional and **panics**. That is what
zfin-server `7103ced` vendors, and a panic is not interceptable by the
`catch return null` every caller depends on. The blast radius is not
one symbol: `handleDiagnostics` runs `freshness.collect` over every key
in the cache directory, so a single unreadable file takes down
`//diagnostics` and the cron refresh sweep. The
byte-serving routes (`/candles`, `/candles_meta`) are unaffected, since
they never parse.
Skew is dangerous in both directions: an old **client** syncing
`candles_meta` bytes from a new server fails the same way, because
`looksCompleteSrf` and the ETag check validate shape and integrity, not
enum values.
So the ordering is: release zfin, then rebuild and deploy every server
that reads the affected cache directory, and only then may a file
carrying the new value exist.
## Server sync (the optional L2 tier)
`ZFIN_SERVER` points zfin at a
[zfin-server](https://git.lerch.org/lobo/zfin-server) instance - a
shared cache between your local cache and the upstream providers. When
unset, every server-sync path silently no-ops.
Client side (`syncFromServer`, `syncCandlesFromServer`,
`parallelServerSync` in `service.zig`):
- Triggered on a local miss/stale entry, before any provider call
(skipped under `force_refresh`).
- `GET {ZFIN_SERVER}/{SYMBOL}/{type}`; the response body is validated
(sha256 ETag, completeness check) and written to the local cache
**verbatim** via `writeRaw` - the client does not re-stamp
`#!expires=`, so it inherits the server's freshness boundary.
- `parallelServerSync` fans out one task per symbol for the portfolio
price load (each worker uses its own HTTP client; the allocator is
thread-safe).
Server side (`serveSrfFile`, `fetchOnMiss` in `zfin-server`):
- A **present** file is served as-is, even if stale - the server's cron
is the freshness authority; reads never trigger a refetch.
- An **absent** file triggers a one-shot `fetchOnMiss` (which calls the
same `getCandles` / `fetchCached` code through the shared zfin
library), then re-reads; if still absent, it returns 404.
Because the server runs the same library, the negative-cache rules above
apply there too: a candle-less symbol gets a negative `candles_daily.srf`
on first miss and is served from it thereafter, rather than re-hitting
the upstream provider on every request.
## Invalidation and atomicity
- `DataService.invalidate(symbol)` clears a symbol's entries; for
candles it removes the `candles_daily` + `candles_meta` pair together.
- `cache clear` wipes the whole cache directory; everything re-fetches
next run.
- All writes are crash-safe: `atomic.zig` writes to a temp file, fsyncs,
and renames into place, so a reader never sees a torn file. A
defensively detected torn candle file self-heals by wiping the pair.
## Key code references
| Concern | Location |
|---------------------------------|-----------------------------------------------------|
| Data-access entry point | `DataService` - `src/service.zig` |
| Per-type generic fetch | `fetchCached` - `src/service.zig` |
| Candle fetch + incremental | `getCandles` - `src/service.zig` |
| Batch price load | `loadAllPrices` - `src/service.zig` |
| Live quotes (uncached) | `loadLiveQuotes`, `getQuote` - `src/service.zig` |
| Cache store, read/write | `Store` - `src/cache/store.zig` |
| Freshness check | `isFresh` (SRF), `isCandleMetaFresh` - `store.zig` |
| TTLs and expiry computation | `Ttl`, `computeExpires` - `src/cache/store.zig` |
| Negative cache | `writeNegative`, `isNegative` - `src/cache/store.zig` |
| Negative-cache veto | `shouldNegativeCache` - `src/service.zig` |
| Candle provenance labels | `CandleProvider` - `src/cache/store.zig` |
| NotFound classification | `isPermanentProviderFailure` - `src/service.zig` |
| Market-aware candle expiry | `nextCandleExpiry`, `shouldRefresh` - `src/market.zig` |
| Price fallback (manual/avg-cost)| `buildFallbackPrices` - `src/analytics/valuation.zig` |
| Server endpoints | `serveSrfFile`, `fetchOnMiss` - `zfin-server/src/main.zig` |
For the user-facing summary and the `--refresh-data` walkthrough, see
[Caching and data freshness](../explanation/caching.md).