352 lines
16 KiB
Markdown
352 lines
16 KiB
Markdown
# Cache implementation (developer reference)
|
|
|
|
This is the low-level, contributor-facing companion to the user-facing
|
|
[Caching and data freshness](../explanation/caching.md) page. It
|
|
documents *how* the cache is built: the on-disk layout, the freshness
|
|
model, the fetch-decision flow, negative caching, and the optional
|
|
server (L2) tier. If you are changing anything in `src/cache/store.zig`
|
|
or the fetch paths in `src/service.zig`, read this first.
|
|
|
|
Diagrams use [Mermaid](https://mermaid.js.org/), which Forgejo renders
|
|
natively.
|
|
|
|
## Where the data lives
|
|
|
|
`DataService` (`src/service.zig`) is the sole data-access layer. Both
|
|
the CLI and the TUI go through it; nothing else calls a provider
|
|
directly. It reads and writes a per-symbol, per-type SRF file cache via
|
|
`Store` (`src/cache/store.zig`):
|
|
|
|
```
|
|
{cache_dir}/ default ~/.cache/zfin, set by ZFIN_CACHE_DIR
|
|
AAPL/
|
|
candles_daily.srf OHLCV bars (append-only)
|
|
candles_meta.srf last_close, last_date, provider + freshness
|
|
dividends.srf
|
|
splits.srf
|
|
options.srf
|
|
earnings.srf
|
|
classification.srf
|
|
etf_metrics.srf
|
|
_edgar/
|
|
tickers_companies.srf shared EDGAR ticker -> CIK maps
|
|
tickers_funds.srf
|
|
0000320193/
|
|
entity_facts.srf per-CIK XBRL facts
|
|
```
|
|
|
|
[SRF](https://git.lerch.org/lobo/srf) is a line-oriented key-value
|
|
format. Files carry `#!`-prefixed directives (`#!expires=`,
|
|
`#!created=`) ahead of their records.
|
|
|
|
### The candle two-file split
|
|
|
|
Candles are stored as **two** files, and the split is load-bearing:
|
|
|
|
- `candles_daily.srf` holds the actual OHLCV records and grows
|
|
append-only: on a cache miss only bars newer than `last_date` are
|
|
fetched and appended, never the full history.
|
|
- `candles_meta.srf` holds a single small record (`last_close`,
|
|
`last_date`, `provider`, `fail_count`) plus the `#!expires=` and
|
|
`#!created=` directives.
|
|
|
|
Keeping the metadata separate lets every freshness check and last-price
|
|
read touch a ~100-byte file instead of deserializing a multi-megabyte
|
|
candle history. The price fast-path in `loadAllPrices` never
|
|
deserializes `candles_daily.srf` - at most it peeks at the first bytes
|
|
to detect a negative entry.
|
|
|
|
The two files are a unit: `DataService.invalidate` and the torn-file
|
|
self-heal clear both together, and a negative cache entry for a
|
|
candle-less symbol is keyed off `candles_daily.srf` (see
|
|
[Negative caching](#negative-caching)).
|
|
|
|
## Freshness is the `#!expires=` directive, not mtime
|
|
|
|
A cache entry is fresh when the wall clock is earlier than the
|
|
`#!expires=` epoch-seconds directive embedded in the file. File
|
|
modification time is **not** consulted for freshness anywhere - mtime is
|
|
a fragile signal (it changes on copy, restore, rsync, and filesystem
|
|
quirks), so the expiry is written into the content itself.
|
|
|
|
- On write, `computeExpires` (`store.zig`) sets `#!expires = now + TTL`
|
|
for the data type, optionally offset by a per-key deterministic jitter
|
|
to avoid thundering-herd refreshes.
|
|
- On read, the SRF iterator parses `#!expires=` and `isFresh` compares
|
|
it to `Timestamp.now(io, .real)`.
|
|
- An entry with **no** `#!expires=` is treated as stale by zfin's
|
|
`.fresh_only` reads (a deliberate override of SRF's "no expiry =
|
|
always fresh" default), except for negative entries, which are always
|
|
fresh.
|
|
|
|
### TTLs by data type
|
|
|
|
Base TTLs live in `Ttl` (`store.zig`); jitter is applied per call site
|
|
in `DataType.ttl()`.
|
|
|
|
| Data type | TTL | Jitter | Notes |
|
|
|----------------------|------------------|--------|--------------------------------------------------|
|
|
| Daily candles | market-aware | n/a | Boundary set by `market.nextCandleExpiry` (below)|
|
|
| Historical candles | never (`-1`) | n/a | Bars older than ~1 day are immutable |
|
|
| Dividends | 14 days | 11% | Declared well in advance |
|
|
| Splits | 14 days | 11% | Rare corporate events |
|
|
| Options | 1 hour | 0 | Move continuously during market hours |
|
|
| Earnings | 30 days | 8% | Smart-refresh after an announcement date passes |
|
|
| Classification | 90 days | 8% | Sector/industry/country from Wikidata |
|
|
| ETF metrics | 90 days | 8% | NPORT-P profile, quarterly cadence |
|
|
| Entity facts (XBRL) | 30 days | 8% | Per-CIK, quarterly filing cadence |
|
|
| EDGAR ticker maps | 30 days | 8% | ticker -> CIK; very stable upstream |
|
|
| Quotes | never cached | n/a | Live by definition (see below) |
|
|
|
|
### Market-aware candle freshness
|
|
|
|
Daily bars are only meaningful once the session settles, so candle
|
|
expiry is keyed to the market clock rather than a rolling window
|
|
(`market.nextCandleExpiry` / `market.staleCandleExpiry`,
|
|
`market.shouldRefresh`):
|
|
|
|
- Equities/ETFs expire at **16:55 ET** on the next trading day.
|
|
- Mutual funds (NAV) expire at **03:25 ET** the next morning.
|
|
|
|
If a refresh fires but the provider has not posted the just-closed bar
|
|
yet, the entry retries in ~30 minutes; once a due bar is ~90 minutes
|
|
overdue, the code concludes the session was an un-modeled closure (Good
|
|
Friday, weather) and falls back to the next normal boundary instead of
|
|
thrashing all day. See the user page for the cron-timing rationale.
|
|
|
|
## The fetch decision
|
|
|
|
### Tiers (the big picture)
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
A["Data request via DataService"] --> B{"Local cache fresh?"}
|
|
B -->|yes| C["Deserialize and return, no network"]
|
|
B -->|no| D{"ZFIN_SERVER set and not force_refresh?"}
|
|
D -->|yes| E["GET server, write bytes verbatim"]
|
|
E --> F{"Synced entry fresh?"}
|
|
F -->|yes| C
|
|
F -->|no| G["Provider fetch"]
|
|
D -->|no| G
|
|
G --> H{"Result?"}
|
|
H -->|ok| I["Write cache with new expiry, return"]
|
|
H -->|NotFound| J["Write negative cache, return FetchFailed"]
|
|
H -->|transient| K["Return error, retry next run"]
|
|
```
|
|
|
|
The `--refresh-data` policy maps to `FetchOptions`:
|
|
|
|
- `auto` (default): all tiers, honor TTL.
|
|
- `force` -> `force_refresh = true`: skip the local-cache and server
|
|
tiers, go straight to the provider, re-stamp the cache. Bypasses
|
|
negative entries (so it retries dead lookups).
|
|
- `never` -> `skip_network = true`: stop at the local cache; return
|
|
stale data if present, never touch the network.
|
|
|
|
### `getCandles` (single symbol)
|
|
|
|
This is the most involved path because of the daily/meta split, the
|
|
incremental-update logic, and the TwelveData carve-out.
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
S["getCandles(symbol, opts)"] --> NG{"negative candles_daily and not force_refresh?"}
|
|
NG -->|yes| FF["return FetchFailed, no network"]
|
|
NG -->|no| RM{"candles_meta exists?"}
|
|
|
|
RM -->|yes| TW{"provider is twelvedata?"}
|
|
TW -->|yes| FULL
|
|
TW -->|no| FR{"meta fresh and not force_refresh?"}
|
|
FR -->|yes| RET["return cached candles"]
|
|
FR -->|no| SS1["syncCandlesFromServer"]
|
|
SS1 --> SF1{"fresh now?"}
|
|
SF1 -->|yes| RET
|
|
SF1 -->|no| INC{"shouldRefresh?"}
|
|
INC -->|no| BUMP["bump TTL, return cached"]
|
|
INC -->|yes| INCF["incremental fetch from last_date+1"]
|
|
|
|
RM -->|no| SN{"skip_network?"}
|
|
SN -->|yes| FF
|
|
SN -->|no| SS2["syncCandlesFromServer"]
|
|
SS2 --> SF2{"fresh now?"}
|
|
SF2 -->|yes| RET
|
|
SF2 -->|no| FULL["populateAllFromTiingo, full history"]
|
|
FULL --> RES{"result?"}
|
|
RES -->|ok| RET2["return fetched"]
|
|
RES -->|NotFound| WN["writeNegative candles_daily"]
|
|
WN --> FF
|
|
RES -->|transient| TR["bump fail_count, TransientError"]
|
|
RES -->|other| FF
|
|
```
|
|
|
|
Key invariant: the negative marker for a candle-less symbol lives in
|
|
`candles_daily.srf`, and **every** candle decision honors it there -
|
|
`isCandleMetaFresh` (the price fast-path gate), `getCachedCandles` (the
|
|
cache-only display path), and the `getCandles` short-circuit above. This
|
|
matters because `candles_meta.srf` is never created for a symbol that
|
|
has no candles, so anything keying freshness off the meta file alone
|
|
would treat such a symbol as perpetually stale and re-fetch it forever.
|
|
|
|
### `fetchCached` (dividends, splits, options, earnings, ...)
|
|
|
|
Everything that is not candles flows through the generic `fetchCached`,
|
|
which is simpler because each type is a single file:
|
|
|
|
1. `.fresh_only` read; a fresh entry (including a negative one) returns
|
|
immediately.
|
|
2. `skip_network`: return any cached entry, even stale; else
|
|
`FetchFailed`.
|
|
3. Server sync (if configured); a fresh synced entry returns.
|
|
4. Provider fetch; on success write with the type's TTL; on
|
|
`NotFound` write a negative entry; on transient error return
|
|
`FetchFailed` without poisoning the cache.
|
|
|
|
### `loadAllPrices` (portfolio + watchlist price load)
|
|
|
|
The portfolio price load batches all symbols through three phases.
|
|
Phase 2 is the parallel server sync; Phase 3 is the per-symbol provider
|
|
fallback that calls `getCandles`.
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
ST["loadAllPrices(portfolio + watch syms)"] --> P1["Phase 1: per symbol"]
|
|
P1 --> CF{"cache fresh and not force_refresh?"}
|
|
CF -->|yes| HIT["use cached last close, cached_count++"]
|
|
CF -->|no| ADD["add to needs_fetch"]
|
|
HIT --> CHK
|
|
ADD --> CHK{"needs_fetch empty?"}
|
|
CHK -->|yes| DONE["return"]
|
|
CHK -->|no| OFF{"skip_network?"}
|
|
OFF -->|yes| STALE["stale-cache fallback or failed_count++"]
|
|
OFF -->|no| HASSRV{"ZFIN_SERVER set?"}
|
|
HASSRV -->|yes| P2["Phase 2: parallelServerSync"]
|
|
HASSRV -->|no| ALLF["all needs_fetch to server_failures"]
|
|
P2 --> REM["unsynced to server_failures"]
|
|
ALLF --> P3
|
|
REM --> P3["Phase 3: sequentialProviderFetch, getCandles each"]
|
|
P3 --> END["return prices + counts"]
|
|
```
|
|
|
|
## Negative caching
|
|
|
|
When a provider says a symbol genuinely has no data of a type - an
|
|
`error.NotFound` - zfin writes a **negative cache entry** so it does not
|
|
re-run the dead lookup on every invocation. The entry is the sentinel:
|
|
|
|
```
|
|
#!srfv1
|
|
# fetch_failed
|
|
```
|
|
|
|
(`Store.negative_cache_content`). Rules:
|
|
|
|
- **Only `NotFound` qualifies.** `isPermanentProviderFailure` gates the
|
|
write. Rate-limit, 5xx, connection, auth, and parse failures are
|
|
transient - they fail the call but leave the cache untouched so the
|
|
next run retries. (Auth/parse looking permanent but being transient is
|
|
exactly why they must not poison a now-sticky negative cache.)
|
|
- **Negative entries are always fresh.** They have no `#!expires=`;
|
|
`readSlice`, `read`, and `isCandleMetaFresh` special-case the sentinel
|
|
as fresh, so they stick until `--refresh-data=force` or `cache clear`.
|
|
- **Candles key the negative off `candles_daily.srf`.** `writeNegative`
|
|
writes that file; `isCandleMetaFresh`, `getCachedCandles`, and the
|
|
`getCandles` short-circuit all recognize it there. `candles_meta.srf`
|
|
is intentionally not created for a no-data symbol.
|
|
|
|
## Candle-less symbols (crypto and friends)
|
|
|
|
Some held symbols have **no daily candles available from the candle
|
|
provider (Tiingo)** - cryptocurrencies on the Yahoo `DOGE-USD` /
|
|
`BTC-USD` shape are the common case, and delisted or invalid tickers
|
|
behave identically. For these symbols `getCandles` writes a negative
|
|
entry and never produces a price from history.
|
|
|
|
Such symbols are still priced, through two mechanisms that do **not**
|
|
touch the candle cache:
|
|
|
|
- **Live quotes (Yahoo).** `loadLiveQuotes` / `getQuote` fetch an
|
|
intraday price from Yahoo, which *does* serve crypto. The TUI overlays
|
|
these live quotes on top of the candle-close price map on refresh and
|
|
on every streaming tick, so a candle-less holding shows its real
|
|
current price there. Live quotes are never cached.
|
|
- **Manual price.** A `price::` field on a lot in `portfolio.srf` pins a
|
|
value. When neither a candle close nor a live quote is available,
|
|
`buildFallbackPrices` (`analytics/valuation.zig`) falls back to the
|
|
position's average cost and flags it as a manual/estimated price
|
|
(rendered in a warning color).
|
|
|
|
Practical consequence by surface: the plain CLI `portfolio` command does
|
|
not apply the live-quote overlay, so a candle-less holding shows its
|
|
average-cost fallback (break-even, warning color); the TUI shows the
|
|
live Yahoo price. This is expected - historical-candle commands (`perf`,
|
|
charts) simply have no data for these symbols, while quote-driven views
|
|
do.
|
|
|
|
If you want a candle-less symbol to be re-checked against the provider
|
|
(for example a ticker that has since started trading), clear its
|
|
negative entry with `cache clear` or `--refresh-data=force`; the live
|
|
quote and manual-price paths are unaffected by the negative cache.
|
|
|
|
## Server sync (the optional L2 tier)
|
|
|
|
`ZFIN_SERVER` points zfin at a
|
|
[zfin-server](https://git.lerch.org/lobo/zfin-server) instance - a
|
|
shared cache between your local cache and the upstream providers. When
|
|
unset, every server-sync path silently no-ops.
|
|
|
|
Client side (`syncFromServer`, `syncCandlesFromServer`,
|
|
`parallelServerSync` in `service.zig`):
|
|
|
|
- Triggered on a local miss/stale entry, before any provider call
|
|
(skipped under `force_refresh`).
|
|
- `GET {ZFIN_SERVER}/{SYMBOL}/{type}`; the response body is validated
|
|
(sha256 ETag, completeness check) and written to the local cache
|
|
**verbatim** via `writeRaw` - the client does not re-stamp
|
|
`#!expires=`, so it inherits the server's freshness boundary.
|
|
- `parallelServerSync` fans out one task per symbol for the portfolio
|
|
price load (each worker uses its own HTTP client; the allocator is
|
|
thread-safe).
|
|
|
|
Server side (`serveSrfFile`, `fetchOnMiss` in `zfin-server`):
|
|
|
|
- A **present** file is served as-is, even if stale - the server's cron
|
|
is the freshness authority; reads never trigger a refetch.
|
|
- An **absent** file triggers a one-shot `fetchOnMiss` (which calls the
|
|
same `getCandles` / `fetchCached` code through the shared zfin
|
|
library), then re-reads; if still absent, it returns 404.
|
|
|
|
Because the server runs the same library, the negative-cache rules above
|
|
apply there too: a candle-less symbol gets a negative `candles_daily.srf`
|
|
on first miss and is served from it thereafter, rather than re-hitting
|
|
the upstream provider on every request.
|
|
|
|
## Invalidation and atomicity
|
|
|
|
- `DataService.invalidate(symbol)` clears a symbol's entries; for
|
|
candles it removes the `candles_daily` + `candles_meta` pair together.
|
|
- `cache clear` wipes the whole cache directory; everything re-fetches
|
|
next run.
|
|
- All writes are crash-safe: `atomic.zig` writes to a temp file, fsyncs,
|
|
and renames into place, so a reader never sees a torn file. A
|
|
defensively detected torn candle file self-heals by wiping the pair.
|
|
|
|
## Key code references
|
|
|
|
| Concern | Location |
|
|
|---------------------------------|-----------------------------------------------------|
|
|
| Data-access entry point | `DataService` - `src/service.zig` |
|
|
| Per-type generic fetch | `fetchCached` - `src/service.zig` |
|
|
| Candle fetch + incremental | `getCandles` - `src/service.zig` |
|
|
| Batch price load | `loadAllPrices` - `src/service.zig` |
|
|
| Live quotes (uncached) | `loadLiveQuotes`, `getQuote` - `src/service.zig` |
|
|
| Cache store, read/write | `Store` - `src/cache/store.zig` |
|
|
| Freshness check | `isFresh` (SRF), `isCandleMetaFresh` - `store.zig` |
|
|
| TTLs and expiry computation | `Ttl`, `computeExpires` - `src/cache/store.zig` |
|
|
| Negative cache | `writeNegative`, `isNegative` - `src/cache/store.zig` |
|
|
| NotFound classification | `isPermanentProviderFailure` - `src/service.zig` |
|
|
| Market-aware candle expiry | `nextCandleExpiry`, `shouldRefresh` - `src/market.zig` |
|
|
| Price fallback (manual/avg-cost)| `buildFallbackPrices` - `src/analytics/valuation.zig` |
|
|
| Server endpoints | `serveSrfFile`, `fetchOnMiss` - `zfin-server/src/main.zig` |
|
|
|
|
For the user-facing summary and the `--refresh-data` walkthrough, see
|
|
[Caching and data freshness](../explanation/caching.md).
|