# Cache implementation (developer reference)

This is the low-level, contributor-facing companion to the user-facing
[Caching and data freshness](../explanation/caching.md) page. It
documents *how* the cache is built: the on-disk layout, the freshness
model, the fetch-decision flow, negative caching, and the optional
server (L2) tier. If you are changing anything in `src/cache/store.zig`
or the fetch paths in `src/service.zig`, read this first.

Diagrams use [Mermaid](https://mermaid.js.org/), which Forgejo renders
natively.

## Where the data lives

`DataService` (`src/service.zig`) is the sole data-access layer. Both
the CLI and the TUI go through it; nothing else calls a provider
directly. It reads and writes a per-symbol, per-type SRF file cache via
`Store` (`src/cache/store.zig`):

```
{cache_dir}/                      default ~/.cache/zfin, set by ZFIN_CACHE_DIR
  AAPL/
    candles_daily.srf             OHLCV bars (append-only)
    candles_meta.srf              last_close, last_date, provider + freshness
    dividends.srf
    splits.srf
    options.srf
    earnings.srf
    classification.srf
    etf_metrics.srf
  _edgar/
    tickers_companies.srf         shared EDGAR ticker -> CIK maps
    tickers_funds.srf
  0000320193/
    entity_facts.srf              per-CIK XBRL facts
```

[SRF](https://git.lerch.org/lobo/srf) is a line-oriented key-value
format. Files carry `#!`-prefixed directives (`#!expires=`,
`#!created=`) ahead of their records.

### The candle two-file split

Candles are stored as **two** files, and the split is load-bearing:

- `candles_daily.srf` holds the actual OHLCV records and grows
  append-only: on a cache miss only bars newer than `last_date` are
  fetched and appended, never the full history.
- `candles_meta.srf` holds a single small record (`last_close`,
  `last_date`, `provider`, `fail_count`) plus the `#!expires=` and
  `#!created=` directives.

Keeping the metadata separate lets every freshness check and last-price
read touch a ~100-byte file instead of deserializing a multi-megabyte
candle history. The price fast-path in `loadAllPrices` never
deserializes `candles_daily.srf` - at most it peeks at the first bytes
to detect a negative entry.

The two files are a unit: `DataService.invalidate` and the torn-file
self-heal clear both together, and a negative cache entry for a
candle-less symbol is keyed off `candles_daily.srf` (see
[Negative caching](#negative-caching)).

## Freshness is the `#!expires=` directive, not mtime

A cache entry is fresh when the wall clock is earlier than the
`#!expires=` epoch-seconds directive embedded in the file. File
modification time is **not** consulted for freshness anywhere - mtime is
a fragile signal (it changes on copy, restore, rsync, and filesystem
quirks), so the expiry is written into the content itself.

- On write, `computeExpires` (`store.zig`) sets `#!expires = now + TTL`
  for the data type, optionally offset by a per-key deterministic jitter
  to avoid thundering-herd refreshes.
- On read, the SRF iterator parses `#!expires=` and `isFresh` compares
  it to `Timestamp.now(io, .real)`.
- An entry with **no** `#!expires=` is treated as stale by zfin's
  `.fresh_only` reads (a deliberate override of SRF's "no expiry =
  always fresh" default), except for negative entries, which are always
  fresh.

### TTLs by data type

Base TTLs live in `Ttl` (`store.zig`); jitter is applied per call site
in `DataType.ttl()`.

| Data type            | TTL              | Jitter | Notes                                            |
|----------------------|------------------|--------|--------------------------------------------------|
| Daily candles        | market-aware     | n/a    | Boundary set by `market.nextCandleExpiry` (below)|
| Historical candles   | never (`-1`)     | n/a    | Bars older than ~1 day are immutable             |
| Dividends            | 14 days          | 11%    | Declared well in advance                         |
| Splits               | 14 days          | 11%    | Rare corporate events                            |
| Options              | 1 hour           | 0      | Move continuously during market hours            |
| Earnings             | 30 days          | 8%     | Smart-refresh after an announcement date passes  |
| Classification       | 90 days          | 8%     | Sector/industry/country from Wikidata            |
| ETF metrics          | 90 days          | 8%     | NPORT-P profile, quarterly cadence               |
| Entity facts (XBRL)  | 30 days          | 8%     | Per-CIK, quarterly filing cadence                |
| EDGAR ticker maps    | 30 days          | 8%     | ticker -> CIK; very stable upstream              |
| Quotes               | never cached     | n/a    | Live by definition (see below)                   |

### Market-aware candle freshness

Daily bars are only meaningful once the session settles, so candle
expiry is keyed to the market clock rather than a rolling window
(`market.nextCandleExpiry` / `market.staleCandleExpiry`,
`market.shouldRefresh`):

- Equities/ETFs expire at **16:55 ET** on the next trading day.
- Mutual funds (NAV) expire at **03:25 ET** the next morning.

If a refresh fires but the provider has not posted the just-closed bar
yet, the entry retries in ~30 minutes; once a due bar is ~90 minutes
overdue, the code concludes the session was an un-modeled closure (Good
Friday, weather) and falls back to the next normal boundary instead of
thrashing all day. See the user page for the cron-timing rationale.

## The fetch decision

### Tiers (the big picture)

```mermaid
flowchart TD
    A["Data request via DataService"] --> B{"Local cache fresh?"}
    B -->|yes| C["Deserialize and return, no network"]
    B -->|no| D{"ZFIN_SERVER set and not force_refresh?"}
    D -->|yes| E["GET server, write bytes verbatim"]
    E --> F{"Synced entry fresh?"}
    F -->|yes| C
    F -->|no| G["Provider fetch"]
    D -->|no| G
    G --> H{"Result?"}
    H -->|ok| I["Write cache with new expiry, return"]
    H -->|NotFound| J["Write negative cache, return FetchFailed"]
    H -->|transient| K["Return error, retry next run"]
```

The `--refresh-data` policy maps to `FetchOptions`:

- `auto` (default): all tiers, honor TTL.
- `force` -> `force_refresh = true`: skip the local-cache and server
  tiers, go straight to the provider, re-stamp the cache. Bypasses
  negative entries (so it retries dead lookups).
- `never` -> `skip_network = true`: stop at the local cache; return
  stale data if present, never touch the network.

### `getCandles` (single symbol)

This is the most involved path because of the daily/meta split, the
incremental-update logic, and the TwelveData carve-out.

```mermaid
flowchart TD
    S["getCandles(symbol, opts)"] --> NG{"negative candles_daily and not force_refresh?"}
    NG -->|yes| FF["return FetchFailed, no network"]
    NG -->|no| RM{"candles_meta exists?"}

    RM -->|yes| TW{"provider is twelvedata?"}
    TW -->|yes| FULL
    TW -->|no| FR{"meta fresh and not force_refresh?"}
    FR -->|yes| RET["return cached candles"]
    FR -->|no| SS1["syncCandlesFromServer"]
    SS1 --> SF1{"fresh now?"}
    SF1 -->|yes| RET
    SF1 -->|no| INC{"shouldRefresh?"}
    INC -->|no| BUMP["bump TTL, return cached"]
    INC -->|yes| INCF["incremental fetch from last_date+1"]

    RM -->|no| SN{"skip_network?"}
    SN -->|yes| FF
    SN -->|no| SS2["syncCandlesFromServer"]
    SS2 --> SF2{"fresh now?"}
    SF2 -->|yes| RET
    SF2 -->|no| FULL["populateAllFromTiingo, full history"]
    FULL --> RES{"result?"}
    RES -->|ok| RET2["return fetched"]
    RES -->|NotFound| WN["writeNegative candles_daily"]
    WN --> FF
    RES -->|transient| TR["bump fail_count, TransientError"]
    RES -->|other| FF
```

Key invariant: the negative marker for a candle-less symbol lives in
`candles_daily.srf`, and **every** candle decision honors it there -
`isCandleMetaFresh` (the price fast-path gate), `getCachedCandles` (the
cache-only display path), and the `getCandles` short-circuit above. This
matters because `candles_meta.srf` is never created for a symbol that
has no candles, so anything keying freshness off the meta file alone
would treat such a symbol as perpetually stale and re-fetch it forever.

### `fetchCached` (dividends, splits, options, earnings, ...)

Everything that is not candles flows through the generic `fetchCached`,
which is simpler because each type is a single file:

1. `.fresh_only` read; a fresh entry (including a negative one) returns
   immediately.
2. `skip_network`: return any cached entry, even stale; else
   `FetchFailed`.
3. Server sync (if configured); a fresh synced entry returns.
4. Provider fetch; on success write with the type's TTL; on
   `NotFound` write a negative entry; on transient error return
   `FetchFailed` without poisoning the cache.

### `loadAllPrices` (portfolio + watchlist price load)

The portfolio price load batches all symbols through three phases.
Phase 2 is the parallel server sync; Phase 3 is the per-symbol provider
fallback that calls `getCandles`.

```mermaid
flowchart TD
    ST["loadAllPrices(portfolio + watch syms)"] --> P1["Phase 1: per symbol"]
    P1 --> CF{"cache fresh and not force_refresh?"}
    CF -->|yes| HIT["use cached last close, cached_count++"]
    CF -->|no| ADD["add to needs_fetch"]
    HIT --> CHK
    ADD --> CHK{"needs_fetch empty?"}
    CHK -->|yes| DONE["return"]
    CHK -->|no| OFF{"skip_network?"}
    OFF -->|yes| STALE["stale-cache fallback or failed_count++"]
    OFF -->|no| HASSRV{"ZFIN_SERVER set?"}
    HASSRV -->|yes| P2["Phase 2: parallelServerSync"]
    HASSRV -->|no| ALLF["all needs_fetch to server_failures"]
    P2 --> REM["unsynced to server_failures"]
    ALLF --> P3
    REM --> P3["Phase 3: sequentialProviderFetch, getCandles each"]
    P3 --> END["return prices + counts"]
```

## Negative caching

When a provider says a symbol genuinely has no data of a type - an
`error.NotFound` - zfin writes a **negative cache entry** so it does not
re-run the dead lookup on every invocation. The entry is the sentinel:

```
#!srfv1
# fetch_failed
```

(`Store.negative_cache_content`). Rules:

- **Only `NotFound` qualifies.** `isPermanentProviderFailure` gates the
  write. Rate-limit, 5xx, connection, auth, and parse failures are
  transient - they fail the call but leave the cache untouched so the
  next run retries. (Auth/parse looking permanent but being transient is
  exactly why they must not poison a now-sticky negative cache.)
- **Negative entries are always fresh.** They have no `#!expires=`;
  `readSlice`, `read`, and `isCandleMetaFresh` special-case the sentinel
  as fresh, so they stick until `--refresh-data=force` or `cache clear`.
- **Candles key the negative off `candles_daily.srf`.** `writeNegative`
  writes that file; `isCandleMetaFresh`, `getCachedCandles`, and the
  `getCandles` short-circuit all recognize it there. `candles_meta.srf`
  is intentionally not created for a no-data symbol.

## Candle-less symbols (crypto and friends)

Some held symbols have **no daily candles available from the candle
provider (Tiingo)** - cryptocurrencies on the Yahoo `DOGE-USD` /
`BTC-USD` shape are the common case, and delisted or invalid tickers
behave identically. For these symbols `getCandles` writes a negative
entry and never produces a price from history.

Such symbols are still priced, through two mechanisms that do **not**
touch the candle cache:

- **Live quotes (Yahoo).** `loadLiveQuotes` / `getQuote` fetch an
  intraday price from Yahoo, which *does* serve crypto. The TUI overlays
  these live quotes on top of the candle-close price map on refresh and
  on every streaming tick, so a candle-less holding shows its real
  current price there. Live quotes are never cached.
- **Manual price.** A `price::` field on a lot in `portfolio.srf` pins a
  value. When neither a candle close nor a live quote is available,
  `buildFallbackPrices` (`analytics/valuation.zig`) falls back to the
  position's average cost and flags it as a manual/estimated price
  (rendered in a warning color).

Practical consequence by surface: the plain CLI `portfolio` command does
not apply the live-quote overlay, so a candle-less holding shows its
average-cost fallback (break-even, warning color); the TUI shows the
live Yahoo price. This is expected - historical-candle commands (`perf`,
charts) simply have no data for these symbols, while quote-driven views
do.

If you want a candle-less symbol to be re-checked against the provider
(for example a ticker that has since started trading), clear its
negative entry with `cache clear` or `--refresh-data=force`; the live
quote and manual-price paths are unaffected by the negative cache.

## Server sync (the optional L2 tier)

`ZFIN_SERVER` points zfin at a
[zfin-server](https://git.lerch.org/lobo/zfin-server) instance - a
shared cache between your local cache and the upstream providers. When
unset, every server-sync path silently no-ops.

Client side (`syncFromServer`, `syncCandlesFromServer`,
`parallelServerSync` in `service.zig`):

- Triggered on a local miss/stale entry, before any provider call
  (skipped under `force_refresh`).
- `GET {ZFIN_SERVER}/{SYMBOL}/{type}`; the response body is validated
  (sha256 ETag, completeness check) and written to the local cache
  **verbatim** via `writeRaw` - the client does not re-stamp
  `#!expires=`, so it inherits the server's freshness boundary.
- `parallelServerSync` fans out one task per symbol for the portfolio
  price load (each worker uses its own HTTP client; the allocator is
  thread-safe).

Server side (`serveSrfFile`, `fetchOnMiss` in `zfin-server`):

- A **present** file is served as-is, even if stale - the server's cron
  is the freshness authority; reads never trigger a refetch.
- An **absent** file triggers a one-shot `fetchOnMiss` (which calls the
  same `getCandles` / `fetchCached` code through the shared zfin
  library), then re-reads; if still absent, it returns 404.

Because the server runs the same library, the negative-cache rules above
apply there too: a candle-less symbol gets a negative `candles_daily.srf`
on first miss and is served from it thereafter, rather than re-hitting
the upstream provider on every request.

## Invalidation and atomicity

- `DataService.invalidate(symbol)` clears a symbol's entries; for
  candles it removes the `candles_daily` + `candles_meta` pair together.
- `cache clear` wipes the whole cache directory; everything re-fetches
  next run.
- All writes are crash-safe: `atomic.zig` writes to a temp file, fsyncs,
  and renames into place, so a reader never sees a torn file. A
  defensively detected torn candle file self-heals by wiping the pair.

## Key code references

| Concern                         | Location                                            |
|---------------------------------|-----------------------------------------------------|
| Data-access entry point         | `DataService` - `src/service.zig`                   |
| Per-type generic fetch          | `fetchCached` - `src/service.zig`                   |
| Candle fetch + incremental      | `getCandles` - `src/service.zig`                    |
| Batch price load                | `loadAllPrices` - `src/service.zig`                 |
| Live quotes (uncached)          | `loadLiveQuotes`, `getQuote` - `src/service.zig`    |
| Cache store, read/write         | `Store` - `src/cache/store.zig`                     |
| Freshness check                 | `isFresh` (SRF), `isCandleMetaFresh` - `store.zig`  |
| TTLs and expiry computation     | `Ttl`, `computeExpires` - `src/cache/store.zig`     |
| Negative cache                  | `writeNegative`, `isNegative` - `src/cache/store.zig` |
| NotFound classification         | `isPermanentProviderFailure` - `src/service.zig`    |
| Market-aware candle expiry      | `nextCandleExpiry`, `shouldRefresh` - `src/market.zig` |
| Price fallback (manual/avg-cost)| `buildFallbackPrices` - `src/analytics/valuation.zig` |
| Server endpoints                | `serveSrfFile`, `fetchOnMiss` - `zfin-server/src/main.zig` |

For the user-facing summary and the `--refresh-data` walkthrough, see
[Caching and data freshness](../explanation/caching.md).
