diff --git a/docs/explanation/caching.md b/docs/explanation/caching.md index 3575216..bfff230 100644 --- a/docs/explanation/caching.md +++ b/docs/explanation/caching.md @@ -128,8 +128,8 @@ payer that's about four full fetches a year, each within a day or so of an ex-date. It's why a refresh run occasionally takes noticeably longer than the usual top-up. -`zfin diagnose SYMBOL` reports the basis for one symbol and says whether -a restatement is pending. +[`zfin diagnose SYMBOL`](../reference/cli/diagnose.md) reports the basis +for one symbol and says whether a restatement is pending. ## Negative caching diff --git a/docs/guides/offline-and-refresh.md b/docs/guides/offline-and-refresh.md index fae808f..3907a85 100644 --- a/docs/guides/offline-and-refresh.md +++ b/docs/guides/offline-and-refresh.md @@ -36,7 +36,8 @@ the bars behind it need their adjusted closes marked down, and zfin re-downloads that symbol's full history to pick up the corrected values on its own -- see [Incremental candle updates](../explanation/caching.md#incremental-candle-updates). -`zfin diagnose SYMBOL` says whether one is pending. +[`zfin diagnose SYMBOL`](../reference/cli/diagnose.md) says whether one is +pending. `zfin cache clear` remains the blunt instrument if you want to discard everything and start over. diff --git a/docs/reference/cli/cache.md b/docs/reference/cli/cache.md index 56b7ce9..37c0473 100644 --- a/docs/reference/cli/cache.md +++ b/docs/reference/cli/cache.md @@ -3,32 +3,56 @@ Inspect or clear the local provider-data cache. ``` -Usage: zfin cache +Usage: zfin cache | zfin cache refresh [SYMBOL...] ``` | Subcommand | Does | |------------|------------------------------------------------------------------------------------------------------------------------------------------------------------| | `stats` | List every cached symbol with per-data-type size, age, and freshness state. Stale entries (past TTL) are flagged. Includes `cusip_tickers.srf` if present. | -| `clear` | Delete every file under the cache directory. No confirmation; the next provider call re-fetches everything. | +| `stale` | Find symbols whose newest candle is behind their peers'. Read-only; reports, never fetches. | +| `refresh` | Force-refresh candle data in the **local** cache, bypassing the TTL and the shared server. No arguments refreshes exactly what `stale` reports. | +| `clear` | Delete every file under the cache directory. No confirmation; the next provider call re-fetches everything. | The cache directory is `$ZFIN_CACHE_DIR` if set, otherwise `~/.cache/zfin`. +## `stale` compares against peers, not the clock + +A symbol is called stale when other cached symbols **of the same kind** +(equity vs mutual fund) hold a newer bar than it does. That is a +different question from "is this bar old?", and the peer framing is what +makes it trustworthy: an un-modeled market closure moves every symbol +together, so it cannot be mistaken for one frozen cache entry. + +## `refresh` deliberately skips the server + +`zfin cache refresh` goes straight to the provider. It does **not** ask +`ZFIN_SERVER`, because its job is to repair *your* copy. To refresh the +server's copy instead, use +[`zfin server refresh`](server.md); to find out which side is behind in +the first place, use [`zfin diagnose SYMBOL`](diagnose.md). + ## Examples ```bash -zfin cache stats -zfin cache clear # wipe; everything re-fetches on next use +zfin cache stats # what's cached, sizes, and ages +zfin cache stale # which symbols are behind their peers +zfin cache refresh # re-fetch exactly what `stale` reported +zfin cache refresh SPY AGG # re-fetch these two +zfin cache clear # wipe; everything re-fetches on next use ``` `clear` is safe -- it only removes cached copies of public market data. Reach for it when you suspect corrupt cached data; routine staleness is -handled automatically by the `auto` refresh policy. +handled automatically by the `auto` refresh policy, and a stale +adjustment basis repairs itself on the next refresh. ## See also - [Caching and data freshness](../../explanation/caching.md) -- TTLs and the fetch model. - [Offline use and refreshing data](../../guides/offline-and-refresh.md) -- the `--refresh-data` flag. +- [`diagnose`](diagnose.md) -- trace one symbol through every tier. +- [`server`](server.md) -- refresh the shared server's cache instead. --- diff --git a/docs/reference/cli/diagnose.md b/docs/reference/cli/diagnose.md new file mode 100644 index 0000000..c07092e --- /dev/null +++ b/docs/reference/cli/diagnose.md @@ -0,0 +1,98 @@ +# `zfin diagnose` + +Trace one symbol's candle data through every tier -- local cache, shared +server, upstream provider -- and say where the chain breaks. + +``` +Usage: zfin diagnose SYMBOL +``` + +`diagnose` answers one question: **is the provider behind for this +symbol, or is it us?** It is the counterpart to +[`doctor`](doctor.md), and deliberately its opposite on both axes -- +`doctor` checks the whole setup and never touches the network, while +`diagnose` checks a single symbol and queries every tier including the +provider. + +It is **read-only**: no cache writes, and the provider query does not +populate the cache. Safe to run while investigating. + +## What it reports + +Seven lines, in the order the data flows: + +| Line | Says | +|-------------|-------------------------------------------------------------------------------------------------------| +| `local` | Newest cached bar, TTL state, which provider sourced it, consecutive failures, any Tiingo backoff | +| `adj basis` | How current the cached adjusted closes are, against the newest dividend or split that has already gone ex | +| `peers` | Newest bar held by other cached symbols of the same kind (equity vs mutual fund) | +| `tracked` | Whether a normal run fetches this symbol, or it is only pulled on demand | +| `server` | What `ZFIN_SERVER` offers, whether it is ahead or behind you, and whether it refreshes this symbol | +| `provider` | What the upstream provider actually has right now | +| `verdict` | Where the chain breaks, and what to do about it | + +## Example + +```bash +zfin diagnose SPY +``` + +``` +local newest 2026-08-19, TTL still in the future, tiingo +adj basis 2026-08-17 - current through the newest corporate action (2026-06-18) +peers 23 other equity cached, newest 2026-08-19 +tracked on demand - a projections benchmark symbol +server offers 2026-08-19 - same as local, written 2026-08-19, tracked: yes +provider tiingo newest 2026-08-19 (8 bars in the last 10 days) + +verdict up to date with everything available +``` + +## Reading the `adj basis` line + +This is the least obvious line, so it is worth spelling out. Cached +price history grows by appending, which cannot retroactively correct the +*adjusted* closes of bars already on disk. When a dividend or split goes +ex, those earlier bars need marking down, and until they are, total +returns spanning that ex-date read low by roughly the missed yield. See +[Incremental candle updates](../../explanation/caching.md#incremental-candle-updates). + +Three forms: + +``` +adj basis 2026-08-17 - current through the newest corporate action (2026-06-18) +adj basis 1970-01-01 - STALE, 2026-06-01 went ex behind it; total returns read low until restated +adj basis 2026-08-19 - no dividends or splits cached, nothing to restate +``` + +`STALE` is not something to act on -- the next refresh re-downloads that +symbol's full history on its own. It explains a total return that looks +slightly low in the meantime. + +Two things the date is **not**: it is not when the restatement ran (it is +the newest bar present at the time, which lags when a session's bar has +not posted yet), and it is not an ex-date. `1970-01-01` means "never +restated", which is what a cache predating the feature reads as. + +The compared-against date is the newest ex-date that has **already +passed**. A declared-but-not-yet-ex dividend is deliberately ignored: +no provider has applied it either, so waiting on it would refetch +forever. + +## Rate limit caveat + +The provider query uses its own rate limiter, separate from the budget +the rest of zfin accounts for -- so it is invisible to that budget. On a +free tier close to its cap, `diagnose` can be the request that trips it. +One request per invocation. + +## See also + +- [`doctor`](doctor.md) -- whole-setup health check, no network. +- [`cache`](cache.md) -- `stale` finds symbols behind their peers; `refresh` forces a local re-fetch. +- [`server`](server.md) -- force a refresh in the shared server's cache instead. +- [Caching and data freshness](../../explanation/caching.md) -- the tiers and TTLs. + +--- + +[CLI command reference](index.md) diff --git a/docs/reference/cli/index.md b/docs/reference/cli/index.md index aaa2cac..985b628 100644 --- a/docs/reference/cli/index.md +++ b/docs/reference/cli/index.md @@ -55,12 +55,14 @@ Get help at any time with `zfin help` or per command with ## Infrastructure -| Command | Does | -|---------------------------------|------------------------------------------------| -| [`cache`](cache.md) | Inspect or clear the local data cache | -| [`doctor`](doctor.md) | Health-check files and environment (read-only) | -| [`version`](version.md) | Show version and build info | -| [`interactive`](interactive.md) | Launch the interactive TUI (alias `i`) | +| Command | Does | +|---------------------------------|--------------------------------------------------| +| [`cache`](cache.md) | Inspect or clear the local data cache | +| [`diagnose`](diagnose.md) | Trace one symbol through cache, server, provider | +| [`server`](server.md) | Ask the shared server to refresh its own cache | +| [`doctor`](doctor.md) | Health-check files and environment (read-only) | +| [`version`](version.md) | Show version and build info | +| [`interactive`](interactive.md) | Launch the interactive TUI (alias `i`) | ## Global options diff --git a/docs/reference/cli/server.md b/docs/reference/cli/server.md new file mode 100644 index 0000000..b6c2550 --- /dev/null +++ b/docs/reference/cli/server.md @@ -0,0 +1,65 @@ +# `zfin server` + +Ask the shared server (`ZFIN_SERVER`) to refresh its own cache. + +``` +Usage: zfin server refresh SYMBOL [SYMBOL...] +``` + +Force-refreshes candle data in the **server's** cache, bypassing its +TTL. Nothing local changes -- your next normal run picks up the server's +new copy through the usual sync. + +## Which side are you refreshing? + +Two commands look similar and do opposite things: + +| Command | Refreshes | Goes through the server? | +|-------------------------------|----------------------|--------------------------| +| `zfin cache refresh SYMBOL` | your **local** cache | No, deliberately | +| `zfin server refresh SYMBOL` | the **server's** cache | That is the point | + +Run [`zfin diagnose SYMBOL`](diagnose.md) first -- its `local` and +`server` lines say which side is actually behind, so you do not refresh +the wrong one. + +## `moved` vs `unchanged` + +The per-symbol result reports whether the newest bar actually **moved**, +not merely whether the fetch succeeded. A successful refresh that changes +nothing is the signature of a provider with no newer data -- which is +usually the finding you came for, so it is not reported as success. + +```bash +zfin server refresh SPY AGG +``` + +``` +Asking https://zfin.example.org to refresh 2 symbol(s)... + SPY moved 2026-08-19 + AGG unchanged 2026-08-18 + +1 moved, 1 unchanged, 0 failed +Your local cache is untouched - a normal run will sync the new copy. +``` + +## Requirements and limits + +- **`ZFIN_SERVER`** must be set, and **`ZFIN_SERVER_API_KEY`** when the + server enforces one. Without a server configured there is nothing to + ask, and the command says so rather than falling back to a local fetch. +- The server caps a single request at **25 symbols**. +- It **refuses a second concurrent refresh** rather than queueing it, so + a refresh already in flight (a cron run, say) returns a conflict rather + than doubling the provider load. + +## See also + +- [`diagnose`](diagnose.md) -- which tier is behind, before you pick a side. +- [`cache`](cache.md) -- `refresh` for the local cache instead. +- [Caching and data freshness](../../explanation/caching.md) -- the server as an optional second tier. +- [Environment variables](../config/environment.md) -- `ZFIN_SERVER`, `ZFIN_SERVER_API_KEY`. + +--- + +[CLI command reference](index.md)