# Cache implementation (developer reference)

This is the low-level, contributor-facing companion to the user-facing
[Caching and data freshness](../explanation/caching.md) page. It
documents *how* the cache is built: the on-disk layout, the freshness
model, the fetch-decision flow, negative caching, and the optional
server (L2) tier. If you are changing anything in `src/cache/store.zig`
or the fetch paths in `src/service.zig`, read this first.

Diagrams use [Mermaid](https://mermaid.js.org/), which Forgejo renders
natively.

## Where the data lives

`DataService` (`src/service.zig`) is the sole data-access layer. Both
the CLI and the TUI go through it; nothing else calls a provider
directly. It reads and writes a per-symbol, per-type SRF file cache via
`Store` (`src/cache/store.zig`):

```
{cache_dir}/                      default ~/.cache/zfin, set by ZFIN_CACHE_DIR
  AAPL/
    candles_daily.srf             OHLCV bars (appended; replaced on restatement)
    candles_meta.srf              last_close, last_date, provider, adj_basis + freshness
    dividends.srf
    splits.srf
    options.srf
    earnings.srf
    classification.srf
    etf_metrics.srf
  _edgar/
    tickers_companies.srf         shared EDGAR ticker -> CIK maps
    tickers_funds.srf
  0000320193/
    entity_facts.srf              per-CIK XBRL facts
```

[SRF](https://git.lerch.org/lobo/srf) is a line-oriented key-value
format. Files carry `#!`-prefixed directives (`#!expires=`,
`#!created=`) ahead of their records.

### The candle two-file split

Candles are stored as **two** files, and the split is load-bearing:

- `candles_daily.srf` holds the actual OHLCV records and normally grows
  by appending: on a cache miss only bars newer than `last_date` are
  fetched and appended, not the full history. The exception is a
  **restatement**, which replaces the file wholesale - see
  [The adjustment basis](#the-adjustment-basis).
- `candles_meta.srf` holds a single small record (`last_close`,
  `last_date`, `provider`, `fail_count`, `tiingo_retry_after_s`,
  `adj_basis`) plus the `#!expires=` and `#!created=` directives.

Keeping the metadata separate lets every freshness check and last-price
read touch a ~100-byte file instead of deserializing a multi-megabyte
candle history. The price fast-path in `loadAllPrices` never
deserializes `candles_daily.srf` - at most it peeks at the first bytes
to detect a negative entry.

The two files are a unit: `DataService.invalidate` and the torn-file
self-heal clear both together, and a negative cache entry for a
candle-less symbol is keyed off `candles_daily.srf` (see
[Negative caching](#negative-caching)).

### The adjustment basis

Appending bars cannot restate the bars already on disk, and sometimes
they need it.

Providers compute `adj_close` by scaling raw `close` by the product of
the adjustment factors for every distribution *after* that bar. So a
series' adjustment basis is only as current as the fetch that produced
it. Newly appended bars arrive with `adj_close == close`, because
nothing has gone ex after them yet, while every previously cached bar
keeps the basis it was originally fetched with. When the next
distribution goes ex, the bars behind it should be marked down by its
factor - and an append does not do that. Every total return spanning
that ex-date then reads low by roughly the missed yield.

`CandleMeta.adj_basis` records how current a series' basis is. It is
**the date of the newest bar present at the last full fetch**:

- Not a wall-clock timestamp. A fetch that runs before the day's bar is
  published gets a basis one bar behind - which is the honest claim,
  since a provider's adjusted series is only ever as complete as its
  newest bar.
- Not an ex-date. It is compared against ex-dates but is never one.
- Never advanced by an append. `Store.appendCandles` takes the existing
  meta and overrides only `last_close` / `last_date`; only
  `Store.cacheCandles`, which replaces the whole file, sets a new
  basis. That asymmetry is the mechanism, so keep it.

`getCandles` escalates from append to full refetch when
`freshness.adjustmentBasisStale` says the basis predates a corporate
action that has **already gone ex**:

```zig
newest_ex = freshness.newestCorporateAction(alloc, store, sym, meta.last_date)
stale     = freshness.adjustmentBasisStale(meta.adj_basis, newest_ex)
```

The already-ex bound lives in `newestCorporateAction`, not in the
verdict, and that placement is load-bearing in both directions. A
declared-but-not-yet-ex distribution is reflected in no provider's
adjusted series, so treating it as something to catch up to would
refetch on every pass forever. But bounding the *verdict* instead -
"is the newest action of all still in the future? then nothing to do" -
lets one forward announcement hide every older unapplied action behind
it. That shipped: a quarterly payer announcing a quarter ahead was
permanently unable to restate.

The check runs on the stale path only, not on the fresh-cache
early-return. Detection is therefore at most one trading day behind,
which the candle TTL guarantees, and the hot portfolio-pricing path
pays nothing.

Steady state for a quarterly payer is four full-history fetches a year,
each landing within about a trading day of an ex-date. `zfin diagnose
SYMBOL` reports the basis against the newest already-ex action.

## Freshness is the `#!expires=` directive, not mtime

A cache entry is fresh when the wall clock is earlier than the
`#!expires=` epoch-seconds directive embedded in the file. File
modification time is **not** consulted for freshness anywhere - mtime is
a fragile signal (it changes on copy, restore, rsync, and filesystem
quirks), so the expiry is written into the content itself.

- On write, `computeExpires` (`store.zig`) sets `#!expires = now + TTL`
  for the data type, optionally offset by a per-key deterministic jitter
  to avoid thundering-herd refreshes.
- On read, the SRF iterator parses `#!expires=` and `isFresh` compares
  it to `Timestamp.now(io, .real)`.
- An entry with **no** `#!expires=` is treated as stale by zfin's
  `.fresh_only` reads (a deliberate override of SRF's "no expiry =
  always fresh" default), except for negative entries, which are always
  fresh.

### TTLs by data type

Base TTLs live in `Ttl` (`store.zig`); jitter is applied per call site
in `DataType.ttl()`.

| Data type            | TTL              | Jitter | Notes                                            |
|----------------------|------------------|--------|--------------------------------------------------|
| Daily candles        | market-aware     | n/a    | Boundary set by `market.nextCandleExpiry` (below)|
| Historical candles   | never (`-1`)     | n/a    | Bars older than ~1 day are immutable             |
| Dividends            | 14 days          | 11%    | Declared well in advance                         |
| Splits               | 14 days          | 11%    | Rare corporate events                            |
| Options              | 1 hour           | 0      | Move continuously during market hours            |
| Earnings             | 30 days          | 8%     | Smart-refresh after an announcement date passes  |
| Classification       | 90 days          | 8%     | Sector/industry/country from Wikidata            |
| ETF metrics          | 90 days          | 8%     | NPORT-P profile, quarterly cadence               |
| Entity facts (XBRL)  | 30 days          | 8%     | Per-CIK, quarterly filing cadence                |
| EDGAR ticker maps    | 30 days          | 8%     | ticker -> CIK; very stable upstream              |
| Quotes               | never cached     | n/a    | Live by definition (see below)                   |

### Market-aware candle freshness

Daily bars are only meaningful once the session settles, so candle
expiry is keyed to the market clock rather than a rolling window
(`market.nextCandleExpiry` / `market.staleCandleExpiry`,
`market.shouldRefresh`):

- Equities/ETFs expire at **16:55 ET** on the next trading day.
- Mutual funds (NAV) expire at **03:25 ET** the next morning.

If a refresh fires but the provider has not posted the just-closed bar
yet, the entry retries in ~30 minutes; once a due bar is ~90 minutes
overdue, the code concludes the session was an un-modeled closure (Good
Friday, weather) and falls back to the next normal boundary instead of
thrashing all day. See the user page for the cron-timing rationale.

## The fetch decision

### Tiers (the big picture)

```mermaid
flowchart TD
    A["Data request via DataService"] --> B{"Local cache fresh?"}
    B -->|yes| C["Deserialize and return, no network"]
    B -->|no| D{"ZFIN_SERVER set and not force_refresh?"}
    D -->|yes| E["GET server, write bytes verbatim"]
    E --> F{"Synced entry fresh?"}
    F -->|yes| C
    F -->|no| G["Provider fetch"]
    D -->|no| G
    G --> H{"Result?"}
    H -->|ok| I["Write cache with new expiry, return"]
    H -->|NotFound| J["Write negative cache, return FetchFailed"]
    H -->|transient| K["Return error, retry next run"]
```

The `--refresh-data` policy maps to `FetchOptions`:

- `auto` (default): all tiers, honor TTL.
- `force` -> `force_refresh = true`: skip the local-cache and server
  tiers, go straight to the provider, re-stamp the cache. Bypasses
  negative entries (so it retries dead lookups).
- `never` -> `skip_network = true`: stop at the local cache; return
  stale data if present, never touch the network.

### `getCandles` (single symbol)

This is the most involved path because of the daily/meta split, the
incremental-update logic, the adjustment-basis escalation, and the
TwelveData carve-out.

Note also that `force` re-asks the provider but does **not** rebuild
candle history: it skips the TTL and the server tier, then takes the
same incremental top-up. Replacing the series is the restatement path's
job (or `zfin cache clear`).

Provider routing is keyed off `CandleMeta.tiingo_retry_after_s`, not off
`provider`. `provider` is pure provenance - "where did these bars come
from" - and using it to route made a single non-transient Tiingo failure
permanent: Yahoo got tried first, succeeded, rewrote `provider = .yahoo`,
and Tiingo was never consulted again. The backoff is armed only by a
genuine 404, expires after `Ttl.tiingo_backoff` with per-symbol jitter,
and clears the moment Tiingo serves the symbol again.

```mermaid
flowchart TD
    S["getCandles(symbol, opts)"] --> NG{"negative candles_daily and not force_refresh?"}
    NG -->|yes| FF["return FetchFailed, no network"]
    NG -->|no| RM{"candles_meta exists?"}

    RM -->|yes| SK{"skip_network?"}
    SK -->|yes| RETS["return cached even if stale<br/>(FetchFailed if unreadable)"]
    SK -->|no| TW{"provider is twelvedata?"}
    TW -->|yes| FULL
    TW -->|no| FR{"meta fresh and not force_refresh?"}
    FR -->|yes| RET["return cached candles"]
    FR -->|no| SS1["syncCandlesFromServer"]
    SS1 --> SF1{"fresh AND adj_basis current?"}
    SF1 -->|yes| RET
    SF1 -->|no| AB{"adj_basis predates an already-ex action?"}
    AB -->|yes| REST["refetchFullHistory: restate whole series"]
    REST --> RR{"ok?"}
    RR -->|yes| RET2["return fetched"]
    RR -->|no| INC
    AB -->|no| INC{"shouldRefresh?"}
    INC -->|no| BUMP["bump TTL, return cached"]
    INC -->|yes| INCF["incremental fetch from last_date+1, appendCandles"]

    RM -->|no| SN{"skip_network?"}
    SN -->|yes| FF
    SN -->|no| SS2["syncCandlesFromServer"]
    SS2 --> SF2{"fresh now?"}
    SF2 -->|yes| RET
    SF2 -->|no| FULL["refetchFullHistory: Tiingo, then Yahoo"]
    FULL --> RES{"result?"}
    RES -->|ok| RET2
    RES -->|"NotFound (EVERY provider disclaims it)"| WN["writeNegative candles_daily"]
    WN --> FF
    RES -->|transient| TR["bump fail_count, TransientError"]
    RES -->|other| FF
```

Two things about this shape are easy to get wrong.

**The basis check precedes `shouldRefresh`.** A symbol that needs both a
top-up and a restatement costs one full fetch, not an append followed by
a second pass. And a failed restatement falls through to the ordinary
top-up rather than erroring: the existing series is untouched and still
usable, just understated by the missed adjustment, so it retries on the
next stale pass.

**Only a unanimous `NotFound` earns a negative entry.** `writeNegative`
*overwrites* `candles_daily.srf` with a marker, so a verdict of "no such
symbol" from Tiingo alone must not reach it - Yahoo gets asked first, and
`refetchFullHistory` returns `error.NotFound` only when every provider
disclaims the symbol. Anything else (auth trouble, a malformed body, a
network blip) fails the call but leaves the cache alone. The restatement
path above never writes a negative entry at all, for the same reason: it
is reached while holding a working series.

The no-prior-cache branch does not re-check the basis after a server
sync, only freshness. A stale basis inherited from the server is caught
on the next invocation, which takes the meta-exists branch. One
invocation of understated returns, then it self-corrects.

Key invariant: the negative marker for a candle-less symbol lives in
`candles_daily.srf`, and **every** candle decision honors it there -
`isCandleMetaFresh` (the price fast-path gate), `getCachedCandles` (the
cache-only display path), and the `getCandles` short-circuit above. This
matters because `candles_meta.srf` is never created for a symbol that
has no candles, so anything keying freshness off the meta file alone
would treat such a symbol as perpetually stale and re-fetch it forever.

### `fetchCached` (dividends, splits, options, earnings, ...)

Everything that is not candles flows through the generic `fetchCached`,
which is simpler because each type is a single file:

1. `.fresh_only` read; a fresh entry (including a negative one) returns
   immediately.
2. `skip_network`: return any cached entry, even stale; else
   `FetchFailed`.
3. Server sync (if configured); a fresh synced entry returns.
4. Provider fetch; on success write with the type's TTL; on
   `NotFound` write a negative entry; on transient error return
   `FetchFailed` without poisoning the cache.

### `loadAllPrices` (portfolio + watchlist price load)

The portfolio price load batches all symbols through three phases.
Phase 2 is the parallel server sync; Phase 3 is the per-symbol provider
fallback that calls `getCandles`.

```mermaid
flowchart TD
    ST["loadAllPrices(portfolio + watch syms)"] --> P1["Phase 1: per symbol"]
    P1 --> CF{"cache fresh and not force_refresh?"}
    CF -->|yes| HIT["use cached last close, cached_count++"]
    CF -->|no| ADD["add to needs_fetch"]
    HIT --> CHK
    ADD --> CHK{"needs_fetch empty?"}
    CHK -->|yes| DONE["return"]
    CHK -->|no| OFF{"skip_network?"}
    OFF -->|yes| STALE["stale-cache fallback or failed_count++"]
    OFF -->|no| HASSRV{"ZFIN_SERVER set?"}
    HASSRV -->|yes| P2["Phase 2: parallelServerSync"]
    HASSRV -->|no| ALLF["all needs_fetch to server_failures"]
    P2 --> REM["unsynced to server_failures"]
    ALLF --> P3
    REM --> P3["Phase 3: sequentialProviderFetch, getCandles each"]
    P3 --> END["return prices + counts"]
```

## Negative caching

When a provider says a symbol genuinely has no data of a type - an
`error.NotFound` - zfin writes a **negative cache entry** so it does not
re-run the dead lookup on every invocation. The entry is the sentinel:

```
#!srfv1
# fetch_failed
```

(`Store.negative_cache_content`). Rules:

- **Only `NotFound` qualifies.** `isPermanentProviderFailure` gates the
  write. Rate-limit, 5xx, connection, auth, and parse failures are
  transient - they fail the call but leave the cache untouched so the
  next run retries. (Auth/parse looking permanent but being transient is
  exactly why they must not poison a now-sticky negative cache.)
- **Negative entries are always fresh.** They have no `#!expires=`;
  `readSlice`, `read`, and `isCandleMetaFresh` special-case the sentinel
  as fresh, so they stick until `--refresh-data=force` or `cache clear`.
- **Candles key the negative off `candles_daily.srf`.** `writeNegative`
  writes that file; `isCandleMetaFresh`, `getCachedCandles`, and the
  `getCandles` short-circuit all recognize it there. `candles_meta.srf`
  is intentionally not created for a no-data symbol.

## Candle-less symbols (crypto and friends)

Some held symbols have **no daily candles available from any candle
provider** - cryptocurrencies on the Yahoo `DOGE-USD` /
`BTC-USD` shape are the common case, and delisted or invalid tickers
behave identically. For these symbols `getCandles` writes a negative
entry and never produces a price from history. "Any" is literal: Tiingo
and Yahoo must both disclaim the symbol, because the marker overwrites
`candles_daily.srf`.

Such symbols are still priced, through two mechanisms that do **not**
touch the candle cache:

- **Live quotes (Yahoo).** `loadLiveQuotes` / `getQuote` fetch an
  intraday price from Yahoo, which *does* serve crypto. The TUI overlays
  these live quotes on top of the candle-close price map on refresh and
  on every streaming tick, so a candle-less holding shows its real
  current price there. Live quotes are never cached.
- **Manual price.** A `price::` field on a lot in `portfolio.srf` pins a
  value. When neither a candle close nor a live quote is available,
  `buildFallbackPrices` (`analytics/valuation.zig`) falls back to the
  position's average cost and flags it as a manual/estimated price
  (rendered in a warning color).

Practical consequence by surface: the plain CLI `portfolio` command does
not apply the live-quote overlay, so a candle-less holding shows its
average-cost fallback (break-even, warning color); the TUI shows the
live Yahoo price. This is expected - historical-candle commands (`perf`,
charts) simply have no data for these symbols, while quote-driven views
do.

If you want a candle-less symbol to be re-checked against the provider
(for example a ticker that has since started trading), clear its
negative entry with `cache clear` or `--refresh-data=force`; the live
quote and manual-price paths are unaffected by the negative cache.

## Server sync (the optional L2 tier)

`ZFIN_SERVER` points zfin at a
[zfin-server](https://git.lerch.org/lobo/zfin-server) instance - a
shared cache between your local cache and the upstream providers. When
unset, every server-sync path silently no-ops.

Client side (`syncFromServer`, `syncCandlesFromServer`,
`parallelServerSync` in `service.zig`):

- Triggered on a local miss/stale entry, before any provider call
  (skipped under `force_refresh`).
- `GET {ZFIN_SERVER}/{SYMBOL}/{type}`; the response body is validated
  (sha256 ETag, completeness check) and written to the local cache
  **verbatim** via `writeRaw` - the client does not re-stamp
  `#!expires=`, so it inherits the server's freshness boundary.
- `parallelServerSync` fans out one task per symbol for the portfolio
  price load (each worker uses its own HTTP client; the allocator is
  thread-safe).

Server side (`serveSrfFile`, `fetchOnMiss` in `zfin-server`):

- A **present** file is served as-is, even if stale - the server's cron
  is the freshness authority; reads never trigger a refetch.
- An **absent** file triggers a one-shot `fetchOnMiss` (which calls the
  same `getCandles` / `fetchCached` code through the shared zfin
  library), then re-reads; if still absent, it returns 404.

Because the server runs the same library, the negative-cache rules above
apply there too: a candle-less symbol gets a negative `candles_daily.srf`
on first miss and is served from it thereafter, rather than re-hitting
the upstream provider on every request.

## Invalidation and atomicity

- `DataService.invalidate(symbol)` clears a symbol's entries; for
  candles it removes the `candles_daily` + `candles_meta` pair together.
- `cache clear` wipes the whole cache directory; everything re-fetches
  next run.
- All writes are crash-safe: `atomic.zig` writes to a temp file, fsyncs,
  and renames into place, so a reader never sees a torn file. A
  defensively detected torn candle file self-heals by wiping the pair.

## Key code references

| Concern                         | Location                                            |
|---------------------------------|-----------------------------------------------------|
| Data-access entry point         | `DataService` - `src/service.zig`                   |
| Per-type generic fetch          | `fetchCached` - `src/service.zig`                   |
| Candle fetch + incremental      | `getCandles` - `src/service.zig`                    |
| Batch price load                | `loadAllPrices` - `src/service.zig`                 |
| Live quotes (uncached)          | `loadLiveQuotes`, `getQuote` - `src/service.zig`    |
| Cache store, read/write         | `Store` - `src/cache/store.zig`                     |
| Freshness check                 | `isFresh` (SRF), `isCandleMetaFresh` - `store.zig`  |
| TTLs and expiry computation     | `Ttl`, `computeExpires` - `src/cache/store.zig`     |
| Negative cache                  | `writeNegative`, `isNegative` - `src/cache/store.zig` |
| NotFound classification         | `isPermanentProviderFailure` - `src/service.zig`    |
| Market-aware candle expiry      | `nextCandleExpiry`, `shouldRefresh` - `src/market.zig` |
| Price fallback (manual/avg-cost)| `buildFallbackPrices` - `src/analytics/valuation.zig` |
| Server endpoints                | `serveSrfFile`, `fetchOnMiss` - `zfin-server/src/main.zig` |

For the user-facing summary and the `--refresh-data` walkthrough, see
[Caching and data freshness](../explanation/caching.md).