update docs with clearer explanation of what is actually happening

AGENTS.md

@@ -882,6 +882,16 @@ command.
 
 - **SRF string lifetimes are handled for you.** `Store.read` parses with a `parse_allocator`, so SRF dupes every owned string into the caller's allocator automatically. **Do not add a `postProcess` callback to dupe strings** - that is not a valid reason for one. `postProcess` exists only for non-trivial post-parse logic, such as recomputing a derived field (see `earningsPostProcess` in `service.zig`, which rebuilds `surprise` from `actual` and `estimate`).
 
+- **A fresh cache entry is not necessarily a complete one.** `fetchCached` takes three optional comptime hooks: `postProcess` (recompute a derived field), `needsRefresh: ?*const fn ([]const T, Date) bool` ("fresh, but is it complete?"), and `ttlFor: ?*const fn ([]const T) cache.TtlSpec` ("how long should this stay fresh, given these records?"). `getDividends` uses `needsRefresh` + `ttlFor`; `getEarnings` uses `postProcess` + `needsRefresh`; `getSplits` and `getOptions` pass `null` for all three. Four rules if you add a `needsRefresh`:
+  - It is consulted at **both** fresh-read sites (local, and post-server-sync) via `serveFreshOrDiscard`. Only covering the first one means a configured `ZFIN_SERVER` defeats the mechanism, because the sync leaves the still-incomplete entry on disk and the second read serves it. This was observed live, not theorised.
+  - `serveFreshOrDiscard` owns the free. Don't discard at a call site; a caller that decides and then forgets is a leak, and the tests rely on `std.testing.allocator` to catch exactly that.
+  - It is suppressed under `skip_network`. Offline mode never refetches, so discarding could only turn a served result into a failure.
+  - It is floored at one ask per symbol per 12 hours (`smart_refresh_recheck_interval` / `askedRecently`). Without that, a hook that stays true because the provider has nothing yet re-fires on **every command**, not once a day.
+
+  **The floor reads `#!expires=`, never `#!created=`.** `serializeWithMeta` re-stamps `created` on every write, and dividend rows arrive as a side effect of Tiingo candle fetches via `writeSupplement` - so `created` means "file last touched" and a candle refresh would fake "we just asked the provider". Only a primary fetch moves `expires` (`.bump`); a supplement restores the old value (`.preserve`). There is a regression test named for this; don't "simplify" it to `created`.
+
+  All three hook branches are comptime-elided for a `null` argument, which is what lets `Split` - a type with no `freeSlice` - keep compiling. See `docs/dev/caching-implementation.md` for the mechanics and `docs/explanation/caching.md` for the worked example.
+
 - **Buffered stdout.** CLI output uses a single `std.Io.Writer` with a 4096-byte stack buffer, flushed once at the end of `main()`. Don't write to stdout through other means.
 
 - **The `color` parameter flows through everything.** CLI commands accept a `color: bool` parameter. Don't use ANSI escapes unconditionally - always gate on the `color` flag.

README.md

@@ -79,17 +79,31 @@ around the runnable example portfolios in [`examples/`](examples/).
 zfin aggregates data from multiple free-tier APIs, using each for what
 it does best, with aggressive caching to stay within free-tier limits.
 
-| Data type             | Provider         | Auth                 | Free-tier limit            | Cache TTL    |
-|-----------------------|------------------|----------------------|----------------------------|--------------|
-| Daily candles (OHLCV) | Tiingo           | `TIINGO_API_KEY`     | 1,000 req/day, 50 req/hour | 23h45m       |
-| Real-time quotes      | Yahoo Finance    | None required        | Unofficial                 | Never cached |
-| Quote fallback        | TwelveData       | `TWELVEDATA_API_KEY` | 8 req/min, 800/day         | Never cached |
-| Dividends             | Polygon          | `POLYGON_API_KEY`    | 5 req/min                  | 14 days      |
-| Splits                | Polygon          | `POLYGON_API_KEY`    | 5 req/min                  | 14 days      |
-| Options chains        | CBOE             | None required        | ~30 req/min (self-imposed) | 1 hour       |
-| Earnings              | FMP              | `FMP_API_KEY`        | 250 req/day                | 30 days*     |
-| ETF profiles          | SEC EDGAR        | `ZFIN_USER_EMAIL`    | 10 req/sec                 | ~90 days     |
-| Classification        | Wikidata + EDGAR | `ZFIN_USER_EMAIL`    | No per-day quota           | Long-lived   |
+| Data type             | Provider         | Auth                 | Free-tier limit            | Cache TTL        |
+|-----------------------|------------------|----------------------|----------------------------|------------------|
+| Daily candles (OHLCV) | Tiingo           | `TIINGO_API_KEY`     | 1,000 req/day, 50 req/hour | 23h45m           |
+| Real-time quotes      | Yahoo Finance    | None required        | Unofficial                 | Never cached     |
+| Quote fallback        | TwelveData       | `TWELVEDATA_API_KEY` | 8 req/min, 800/day         | Never cached     |
+| Dividends             | Polygon          | `POLYGON_API_KEY`    | 5 req/min                  | 6 or 14 days\*\* |
+| Splits                | Polygon          | `POLYGON_API_KEY`    | 5 req/min                  | 14 days          |
+| Options chains        | CBOE             | None required        | ~30 req/min (self-imposed) | 1 hour           |
+| Earnings              | FMP              | `FMP_API_KEY`        | 250 req/day                | 30 days*         |
+| ETF profiles          | SEC EDGAR        | `ZFIN_USER_EMAIL`    | 10 req/sec                 | ~90 days         |
+| Classification        | Wikidata + EDGAR | `ZFIN_USER_EMAIL`    | No per-day quota           | Long-lived       |
+
+\* **Earnings smart refresh** -- cached earnings re-fetch inside the
+30-day window once a report date has passed but the actual result is
+still missing.
+
+\*\* **Dividends get three freshness checks, not one.** An ETF's payment
+amount does not exist until roughly the day it goes ex, and it pays within
+a week, so a fixed expiry time can sail right over the whole event. So
+after the expiry clock says an entry is still fresh, a second check asks
+whether a payment the symbol's own schedule predicts is missing from it -
+and a third stops that from asking the provider more than once every 12
+hours. The expiry clock is 14 days when the schedule is known and 6 days
+when it isn't. See
+[Caching and data freshness](docs/explanation/caching.md#dividends-a-three-level-freshness-check).
 
 Not all keys are required; without a given key, that data type is
 simply unavailable. For per-provider notes, signup links, the full

docs/dev/caching-implementation.md

@@ -146,8 +146,8 @@ in `DataType.ttl()`.
 |----------------------|------------------|--------|--------------------------------------------------|
 | Daily candles        | market-aware     | n/a    | Boundary set by `market.nextCandleExpiry` (below)|
 | Historical candles   | never (`-1`)     | n/a    | Bars older than ~1 day are immutable             |
-| Dividends            | 14 days          | 11%    | Declared well in advance                         |
-| Splits               | 14 days          | 11%    | Rare corporate events                            |
+| Dividends            | 6 days           | 11%    | Backstop only; see dividend smart refresh below  |
+| Splits               | 14 days          | 11%    | Rare corporate events, announced weeks ahead     |
 | Options              | 1 hour           | 0      | Move continuously during market hours            |
 | Earnings             | 30 days          | 8%     | Smart-refresh after an announcement date passes  |
 | Classification       | 90 days          | 8%     | Sector/industry/country from Wikidata            |
@@ -303,19 +303,247 @@ matters because `candles_meta.srf` is never created for a symbol that
 has no candles, so anything keying freshness off the meta file alone
 would treat such a symbol as perpetually stale and re-fetch it forever.
 
-### `fetchCached` (dividends, splits, options, earnings, ...)
+### `fetchCached` (dividends, splits, options, earnings)
 
 Everything that is not candles flows through the generic `fetchCached`,
 which is simpler because each type is a single file:
 
 1. `.fresh_only` read; a fresh entry (including a negative one) returns
-   immediately.
+   immediately -- **unless** the type supplies a `needsRefresh` hook that
+   says the fresh entry is incomplete (see the freshness levels below).
 2. `skip_network`: return any cached entry, even stale; else
    `FetchFailed`.
-3. Server sync (if configured); a fresh synced entry returns.
-4. Provider fetch; on success write with the type's TTL; on
-   `NotFound` write a negative entry; on transient error return
-   `FetchFailed` without poisoning the cache.
+3. Server sync (if configured); a fresh synced entry returns, subject to
+   the same `needsRefresh` check.
+4. Provider fetch; on success write with the TTL from `ttlSpecFor`; on
+   `NotFound` write a negative entry; on `NoApiKey` return `NoApiKey`;
+   on any other transient error return `FetchFailed`. Neither of the
+   last two poisons the cache.
+
+#### The three freshness levels
+
+Dividends and earnings do not get a single freshness verdict; they get
+three, in a fixed order. The user-facing explanation is in
+[Caching and data freshness](../explanation/caching.md#dividends-a-three-level-freshness-check);
+this is where each level lives in code.
+
+| Level | Name | Question | Implementation |
+|-------|--------------------|-----------------------------------------------------------|------------------------------------------------------------|
+| **1** | expiry clock       | "Is this entry old?"                                      | `#!expires=` vs now, enforced by `Store.read(.fresh_only)` |
+| **2** | completeness check | "Is this entry missing something it should already have?" | `needsRefresh` hook, via `serveFreshOrDiscard`             |
+| **3** | recheck floor      | "Did we already ask the provider recently?"               | `askedRecently`, 12h                                       |
+
+**The ordering is the part to get right, and it is the opposite of the
+natural reading.** Level 2 is a **second opinion on a "still fresh" verdict
+from Level 1**, not an extra gate in front of a refresh:
+
+- Level 1 says **stale** -> `Store.read(.fresh_only)` returns `null`, the
+  `if` body in step 1 never executes, and `serveFreshOrDiscard` is never
+  called. **Levels 2 and 3 do not run at all.** There is nothing for them
+  to contribute: a refetch is already happening.
+- Level 1 says **fresh** -> Level 2 runs, and its whole job is to be able
+  to overrule that with "no it isn't, a payment has happened that this
+  entry doesn't contain."
+- Level 2 says **incomplete** -> Level 3 runs, and can overrule Level 2
+  back to "serve the cache anyway" if we asked too recently.
+
+So each level can only be reached by the previous one declining to
+decide. If you ever find yourself reasoning about Level 2 firing on a
+stale entry, the model is inverted.
+
+`skip_network` short-circuits between Levels 1 and 2: offline mode serves
+the fresh entry without consulting either later level, since it can never
+refetch and discarding could only turn a served result into a failure.
+
+`NoApiKey` is propagated rather than collapsed into `FetchFailed`
+because it is the one distinction callers act on differently:
+`commands/earnings.zig` and `tui/earnings_tab.zig` both name the
+environment variable to set, which they cannot do from a generic
+failure. It also must not negative-cache -- nothing is wrong with the
+symbol, and poisoning it would suppress the fetch after the key is
+finally configured.
+
+`getEarnings` is a thin wrapper over this, not a parallel
+implementation: it short-circuits mutual funds (no quarterly earnings
+exist, so there is nothing to cache or fetch) and then delegates,
+supplying `earningsPostProcess` and `earningsNeedsRefreshHook`.
+
+#### The three comptime hooks
+
+`fetchCached` takes three optional comptime function pointers. Passing
+`null` for all three gives the behaviour every type had before they
+existed, which is what `getSplits` and `getOptions` do.
+
+| Hook           | Signature                            | Question it answers                           |
+|----------------|--------------------------------------|-----------------------------------------------|
+| `postProcess`  | `fn (*T, Allocator) anyerror!void`   | Any derived field to recompute after parsing? |
+| `needsRefresh` | `fn ([]const T, Date) bool`          | This entry is fresh, but is it *complete*?    |
+| `ttlFor`       | `fn ([]const T) cache.TtlSpec`       | How long should this data stay fresh?         |
+
+Who passes what:
+
+| Getter        | postProcess            | needsRefresh                | ttlFor         |
+|---------------|------------------------|-----------------------------|----------------|
+| `getDividends`| -                      | `dividendsNeedRefresh`      | `dividendTtl`  |
+| `getEarnings` | `earningsPostProcess`  | `earningsNeedsRefreshHook`  | -              |
+| `getSplits`   | -                      | -                           | -              |
+| `getOptions`  | -                      | -                           | -              |
+
+`postProcess` is **not** for duping strings -- `Store.read` already does
+that. Only `EarningsEvent` uses it, to rebuild `surprise`.
+
+`needsRefresh` is consulted at **both** fresh-read sites (steps 1 and 3
+above) through `serveFreshOrDiscard`, which owns the free so a caller
+cannot decide to discard and then leak. Covering step 3 is not optional:
+with a `ZFIN_SERVER` configured, a sync can leave a still-incomplete entry
+on disk that step 3 would otherwise serve, defeating the mechanism
+entirely. (This is not theoretical - it was observed against a live
+server whose copy was missing the same record the local cache was.) The
+hook is suppressed under `skip_network`, since offline mode never
+refetches and discarding could only turn a served result into a failure.
+
+`ttlFor` exists because the right expiry time for dividends depends on the
+records being written, not just on their type - see `dividendTtl`. It is
+applied through `ttlSpecFor` at both write sites (the normal path and the
+rate-limit retry) so a retry cannot stamp a different clock than the
+first attempt would have.
+
+All three branches are comptime-elided when the argument is `null`, which
+is what lets types without a `freeSlice` (such as `Split`) keep compiling.
+
+#### The recheck floor
+
+A `needsRefresh` hook has a failure mode that is easy to miss. When the
+hook says something is overdue and the provider does not have it yet -
+the *normal* case, not an error - the cache is rewritten with a fresh
+expiry but the same records. The hook therefore still says "overdue", and
+the next command asks again. On a quarter-end run touching fifteen funds
+that is fifteen provider calls per command, repeated every time anything
+runs.
+
+`smart_refresh_recheck_interval` (12 hours) is the floor, enforced by
+`askedRecently`. It applies to any type with a `needsRefresh` hook.
+
+**It reads `#!expires=`, not `#!created=`, and that distinction is the
+whole trick.** `serializeWithMeta` re-stamps `created` on every write, and
+dividend rows also arrive as a side effect of a Tiingo candle fetch via
+`writeSupplement`. So `created` means "file last touched", which a candle
+refresh changes without anyone asking Polygon anything - keying the floor
+off it would suppress a genuinely due refetch for 12 hours. `expires` is
+moved only by a primary-provider fetch (`ExpiryPolicy.bump`); a supplement
+deliberately puts the old value back (`.preserve`). There is a regression
+test for exactly this.
+
+Recovering the write time from `expires` is *exact*, not approximate,
+because `computeExpires` derives its jitter from a hash of the symbol -
+the same symbol always gets the same offset. Calling it with a zero "now"
+returns precisely the TTL that was applied, so `expires - that` is the
+write time to the second.
+
+Two cases fall back to "ask again", which is the safe direction: a file
+with no `#!expires=` at all, and a symbol that crossed from
+unknown-schedule to known-schedule since it was written (stored under the
+short TTL, recomputed with the long one, so the write looks older than it
+was).
+
+#### Dividend schedule prediction
+
+`dividendsNeedRefresh` returns true when at least one full cadence period
+has elapsed since the newest cached ex-date and the payment that should
+have landed in it is still absent.
+
+It exists because a dividend cache being *fresh* and being *complete* are
+unrelated. An operating company declares weeks ahead of the ex-date and
+pays about a month later. An ETF cannot: its distribution is the
+pass-through of whatever the fund earned, so the amount does not exist
+until roughly its own ex-date, and it pays 1-7 days later. The period in
+which the record exists but the cash has not landed is about a week, so
+any longer expiry can be written before the record exists and still be
+fresh after the payment. Quarter-end clustering makes it systematic -- a
+dozen funds go ex within a few days, so one refresh pass puts all of them
+on the same expiry date with the same blind spot.
+
+Cadence comes from `dividendCadenceDays`: the **median** of up to the five
+newest gaps between cached ex-dates. Median rather than mean (a special
+distribution landing days after a regular one drags the mean) and rather
+than minimum (the same outlier destroys the minimum outright, predicting a
+next ex-date days away and firing for the whole chase window after every
+payment). `newestExDates` sorts rather than trusting file order, because
+`writeSupplement`'s merge interleaves two providers' records and a cadence
+computed from negative gaps would forecast into the past forever.
+
+Three properties are load-bearing and each has its own test:
+
+- **A full period must elapse.** Otherwise it refires immediately after a
+  successful fetch, on the record it just stored.
+- **It rolls forward.** The expected date is the latest one at or before
+  today, not the first one after the newest cached record -- so a symbol
+  two periods behind stays recoverable instead of being permanently past
+  its window.
+- **The chase window is bounded** (`dividend_chase_days`, 14). An
+  unbounded chase refetches forever whenever a sponsor skips a period or
+  the cadence estimate is wrong.
+
+`dividendTtl` then picks the expiry from the same cadence calculation:
+`Ttl.dividends_scheduled` (14 days) when a cadence exists, because
+prediction is what finds due payments; `Ttl.dividends` (6 days) when it
+does not, because then the clock is the only mechanism and it has to
+expire before the next weekly review. Six is the largest value that
+guarantees that -- 6 x 1.11 jitter = 6.66 days, under 7, where 7 would
+reach 7.77. `"TTL constants are reasonable"` in store.zig asserts that
+property rather than just the number.
+
+Known blind spots, all left to the expiry time rather than papered over
+with a more eager statistic that would cost requests on every symbol to
+protect against a rare event on one:
+
+- A **cadence change** (quarterly to monthly) keeps the old median for
+  several periods, so the forecast runs late.
+- An **off-cycle special** is on no schedule, so prediction never fires
+  for it.
+- A symbol with **fewer than three cached records** has no cadence at all
+  -- which is exactly the case that gets the 6-day expiry.
+
+#### Why Tiingo can never discover a new dividend
+
+Tiingo returns dividends and splits alongside candles in a single request
+(`populateAllFromTiingo`), written through `writeSupplement` so the merge
+lands new records without moving the primary provider's `#!expires=`
+clock. That makes it look like a free second source for Level 2, and it is
+worth being explicit that it is not - for two independent reasons.
+
+**Structural.** Tiingo carries a dividend as a `divCash` field on a daily
+price bar, so a payment can only appear once a bar exists for it. A
+declared-but-not-yet-ex dividend has no bar and therefore cannot be
+represented at all. This is exactly why Polygon is the primary source -
+see the comment in `fetchFromProvider`, which cites a forward-dated ARCC
+ex-date as the case Tiingo's response cannot express.
+
+**Circular.** `populateAllFromTiingo` is reached from only two places:
+
+1. A **cold candle cache** - the symbol has no bars at all
+   (`getCandles`'s full-history path). Here Tiingo genuinely earns its
+   keep: one request bulk-loads the symbol's entire dividend history.
+2. An **adjustment-basis restatement** - cached `adj_close` values predate
+   a corporate action they should reflect, so the series is refetched in
+   full.
+
+Trigger 2 is decided by `freshness.adjustmentBasisStale` fed by
+`freshness.newestCorporateAction`, and that function reads the **cached
+dividend file** to find the newest action. So zfin only knows to restate
+*after* the dividend record is already on disk - which means Polygon must
+have supplied it first. By the time Tiingo's rows merge in, they add
+nothing about that payment.
+
+Net: Tiingo bulk-loads history and fills field-level gaps in old records.
+Every *new* dividend arrives via Polygon. This is also why the Level 3
+floor reads `#!expires=` and not `#!created=` - a Tiingo supplement
+rewrites the file, and therefore `created`, without having asked Polygon
+anything.
+
+For the user-facing version of all this, including a worked quarterly
+example, see
+[Caching and data freshness](../explanation/caching.md#dividends-a-three-level-freshness-check).
 
 ### `loadAllPrices` (portfolio + watchlist price load)
 
@@ -555,23 +783,28 @@ the upstream provider on every request.
 
 ## Key code references
 
-| Concern                         | Location                                            |
-|---------------------------------|-----------------------------------------------------|
-| Data-access entry point         | `DataService` - `src/service.zig`                   |
-| Per-type generic fetch          | `fetchCached` - `src/service.zig`                   |
-| Candle fetch + incremental      | `getCandles` - `src/service.zig`                    |
-| Batch price load                | `loadAllPrices` - `src/service.zig`                 |
-| Live quotes (uncached)          | `loadLiveQuotes`, `getQuote` - `src/service.zig`    |
-| Cache store, read/write         | `Store` - `src/cache/store.zig`                     |
-| Freshness check                 | `isFresh` (SRF), `isCandleMetaFresh` - `store.zig`  |
-| TTLs and expiry computation     | `Ttl`, `computeExpires` - `src/cache/store.zig`     |
-| Negative cache                  | `writeNegative`, `isNegative` - `src/cache/store.zig` |
-| Negative-cache veto             | `shouldNegativeCache` - `src/service.zig`           |
-| Candle provenance labels        | `CandleProvider` - `src/cache/store.zig`            |
-| NotFound classification         | `isPermanentProviderFailure` - `src/service.zig`    |
-| Market-aware candle expiry      | `nextCandleExpiry`, `shouldRefresh` - `src/market.zig` |
-| Price fallback (manual/avg-cost)| `buildFallbackPrices` - `src/analytics/valuation.zig` |
-| Server endpoints                | `serveSrfFile`, `fetchOnMiss` - `zfin-server/src/main.zig` |
+| Concern                         | Location                                                               |
+|---------------------------------|------------------------------------------------------------------------|
+| Data-access entry point         | `DataService` - `src/service.zig`                                      |
+| Per-type generic fetch          | `fetchCached` - `src/service.zig`                                      |
+| Fresh-but-incomplete decision   | `serveFreshOrDiscard` - `src/service.zig`                              |
+| Recheck floor (12h)             | `askedRecently`, `smart_refresh_recheck_interval` - `src/service.zig`  |
+| Dividend schedule prediction    | `dividendsNeedRefresh`, `dividendCadenceDays` - `src/service.zig`      |
+| Dividend expiry choice          | `dividendTtl`, `ttlSpecFor` - `src/service.zig`                        |
+| Earnings smart refresh          | `earningsNeedsRefresh`, `earningsNeedsRefreshHook` - `src/service.zig` |
+| Candle fetch + incremental      | `getCandles` - `src/service.zig`                                       |
+| Batch price load                | `loadAllPrices` - `src/service.zig`                                    |
+| Live quotes (uncached)          | `loadLiveQuotes`, `getQuote` - `src/service.zig`                       |
+| Cache store, read/write         | `Store` - `src/cache/store.zig`                                        |
+| Freshness check                 | `isFresh` (SRF), `isCandleMetaFresh` - `store.zig`                     |
+| TTLs and expiry computation     | `Ttl`, `computeExpires` - `src/cache/store.zig`                        |
+| Negative cache                  | `writeNegative`, `isNegative` - `src/cache/store.zig`                  |
+| Negative-cache veto             | `shouldNegativeCache` - `src/service.zig`                              |
+| Candle provenance labels        | `CandleProvider` - `src/cache/store.zig`                               |
+| NotFound classification         | `isPermanentProviderFailure` - `src/service.zig`                       |
+| Market-aware candle expiry      | `nextCandleExpiry`, `shouldRefresh` - `src/market.zig`                 |
+| Price fallback (manual/avg-cost)| `buildFallbackPrices` - `src/analytics/valuation.zig`                  |
+| Server endpoints                | `serveSrfFile`, `fetchOnMiss` - `zfin-server/src/main.zig`             |
 
 For the user-facing summary and the `--refresh-data` walkthrough, see
 [Caching and data freshness](../explanation/caching.md).

docs/explanation/caching.md

@@ -39,20 +39,172 @@ The `--refresh-data` policy decides which tiers run:
 
 Different data ages at different rates, so each type has its own TTL:
 
-| Data type     | TTL           | Why                                                                              |
-|---------------|---------------|----------------------------------------------------------------------------------|
-| Daily candles | market-aware  | Keyed to the next time a fresh bar is expected, not a rolling window (see below) |
-| Dividends     | 14 days       | Declared well in advance                                                         |
-| Splits        | 14 days       | Rare corporate events                                                            |
-| Options       | 1 hour        | Prices move continuously when markets are open                                   |
-| Earnings      | 30 days\*     | Quarterly; smart-refreshed around announcements                                  |
-| ETF profiles  | ~30 days      | Holdings and weights change slowly                                               |
-| Quotes        | never cached  | Meant to be a live price check                                                   |
+| Data type     | TTL                | Why                                                                              |
+|---------------|--------------------|----------------------------------------------------------------------------------|
+| Daily candles | market-aware       | Keyed to the next time a fresh bar is expected, not a rolling window (see below) |
+| Dividends     | 6 or 14 days\*     | Depends on whether we know the symbol's payment schedule                         |
+| Splits        | 14 days            | Rare, and announced weeks ahead                                                  |
+| Options       | 1 hour             | Prices move continuously when markets are open                                   |
+| Earnings      | 30 days\*\*        | Quarterly, with an early re-check around announcements                           |
+| ETF profiles  | ~30 days           | Holdings and weights change slowly                                               |
+| Quotes        | never cached       | Meant to be a live price check                                                   |
+
+\* The expiry time is only the first of three checks. See
+[Dividends: a three-level freshness check](#dividends-a-three-level-freshness-check).
+
+\*\* **Earnings** use the same three-level check as dividends, with a
+different Level 2: instead of asking "is a payment overdue?", it asks "has
+a report date passed with the actual result still missing?". So results
+appear promptly after an announcement without daily polling. Level 3
+applies too - one re-check per symbol per 12 hours.
+
+## Dividends: a three-level freshness check
+
+For most data a plain expiry time is enough. For dividends it is not, so
+dividends (and earnings) go through **three checks instead of one**. The
+order matters, and it is the thing most easily got backwards:
+
+| Level | Name                   | Question it answers                                       | Runs when                                         |
+|-------|------------------------|-----------------------------------------------------------|---------------------------------------------------|
+| **1** | **expiry clock**       | "Is this cached entry old?"                               | always                                            |
+| **2** | **completeness check** | "Is this entry missing something it should already have?" | **only if Level 1 said the entry is still fresh** |
+| **3** | **recheck floor**      | "Did we already ask the provider about this recently?"    | only if Level 2 said something is missing         |
+
+**Read that middle row carefully, because the intuition runs the other
+way.** Level 2 is *not* an extra hurdle before a refresh. It is a
+**second opinion on a "no" from Level 1**. If the expiry clock says the
+entry is old, zfin just refetches and Level 2 is never consulted at all -
+there is nothing for it to add, because a refresh is already happening.
+Level 2 only earns its keep in the opposite case: the clock says *"this
+is still fine, serve it"*, and Level 2 gets to answer *"no it isn't -
+a payment has happened that this entry doesn't contain."*
+
+Level 3 then stops Level 2 from being too eager, by refusing to ask the
+provider about the same symbol more than once every 12 hours.
+
+### Why one expiry time isn't enough
+
+A regular company announces a dividend weeks before the ex-date and pays
+about a month after it. There is plenty of time to notice. An ETF is
+different. Its distribution is simply whatever the fund earned, passed
+through to you, so the amount does not exist until roughly the fund's own
+ex-date -- and it pays one to seven days later. The whole period between
+"the payment becomes knowable" and "the cash is in your account" is about
+a week.
+
+A two-week expiry time sails straight over that. Suppose the cache is
+refreshed a few days before an ETF goes ex. It is then considered fresh
+for another eleven days, which covers the ex-date, the payment, *and* the
+weekly review that was supposed to explain the deposit. The dividend is
+invisible exactly when it matters. And it is not bad luck: dozens of
+funds go ex within the same few days at each quarter end, so one refresh
+pass puts all of them on the same expiry date and every one of them has
+the same blind spot.
+
+Level 2 exists to catch precisely that: it works out the symbol's own
+payment schedule from the ex-dates already in the cache, and speaks up
+when a payment is due that the cache does not have.
+
+### Two things that are easy to picture wrongly
+
+**The expiry clock is a duration, not a date.** When a record is cached,
+zfin stamps "expires 14 days from now". It does not store "next check due
+on 30 September". Each refetch re-stamps the duration from scratch.
+
+**The expected payment date is never stored at all.** Level 2 recomputes
+it on every single cache read, from whatever ex-dates are currently
+cached. Nothing on disk holds it.
+
+So capturing a payment does not "set the clock to" the next expected
+payment. The two mechanisms run independently, on different schedules.
+
+### A worked example
+
+Take an ETF that has paid on 1 January, 1 April, 1 July and 1 October for
+the last five years. The newest payment in the cache is 1 July 2026.
+
+Level 2 measures the gaps between the most recent ex-dates -- 91, 90, 92,
+92 and 91 days -- and takes the middle value: **91 days**. Added to 1
+July, that makes **30 September** the date the next payment is expected.
+(The real date is 1 October. Being a day out does not matter, as you will
+see.)
+
+| When             | What the levels say                                                       | Result                                                                                                                   |
+|------------------|---------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------|
+| 2 July - 29 Sept | L1: fresh, then stale every 14 days. L2: nothing due.                     | Nothing is expected, so only the clock causes fetches: **about six over the quarter** (91 days, at 14 days a time).      |
+| 30 Sept - 14 Oct | L1: fresh. L2: **a payment is overdue.** L3: allows one request/12 hours. | The provider is asked, at most twice a day no matter how many commands you run. Stops the moment the new record arrives. |
+| 15 Oct onwards   | L2: gave up -- two weeks past the expected date.                          | Back to the clock alone. Chasing a schedule that is clearly wrong would refetch forever.                                 |
+
+On 30 September the provider probably has nothing yet, because the fund
+has not gone ex. That request is wasted, and it is the price of the estimate
+being a day early. On 1 October the record appears, gets cached, and
+Level 2 immediately goes quiet again -- it now expects the next payment
+around 1 January.
+
+The estimate also corrects itself. Once the 1 October record is cached,
+the newest gaps median to 92 days rather than 91, which puts the next
+expected date exactly on 1 January 2027.
+
+### Why Level 1 has two different lengths
+
+How long the expiry clock runs depends on whether Level 2 can do its job
+for that symbol:
+
+- **14 days when the payment schedule is known.** A symbol with a few
+  years of history is found by Level 2, so the clock only has to cover
+  the one thing a schedule cannot predict: a one-off special
+  distribution. Making it short as well would re-check every symbol
+  every few days to guard against a problem already solved.
+
+- **6 days when the schedule is not known.** With fewer than three
+  cached payments there is no pattern to work from -- typically a holding
+  you have just bought. Level 2 declines, so the clock is the *only*
+  thing that will ever make zfin look again, and it has to expire before
+  a weekly review. Six days guarantees that; seven would not, because
+  the +/-11% spread applied to expiry times can stretch seven days to 7.8.
+
+### Why Level 3 exists
+
+When Level 2 says a payment is due and the provider does not have it yet,
+nothing in the cache changes except the expiry time. Level 2 therefore
+still says "due", so the next command would ask again -- and the one
+after that. On a quarter-end run touching fifteen funds, that is fifteen
+provider calls per command, every time you run anything.
+
+So Level 3 allows a symbol to be asked about at most once every 12 hours.
+Twelve hours sits well inside the shortest gap between a fund going ex
+and paying (one day, in a real portfolio), while still being far more
+frequent than the weekly reconcile that actually uses the answer.
+
+### What about Tiingo?
+
+Tiingo returns dividend data as a side effect of fetching prices, which
+raises the obvious question of whether it can spot a new payment for
+free. It cannot, and it is worth knowing that it never will.
+
+Tiingo reports a dividend as a field on a daily price bar, so the payment
+only shows up once it has already traded -- there is no such thing as a
+forward-looking dividend in that data. Polygon is the primary source
+precisely because it *does* carry declared-but-not-yet-paid dividends.
+And the code path that merges Tiingo's dividend rows in only runs on a
+cold cache or when a known corporate action forces a price-history
+restatement, which means Polygon has already supplied the record by then.
+
+So Tiingo is genuinely useful for bulk-loading a new symbol's entire
+dividend history in one request, and for filling gaps in old records. It
+is simply never the thing that notices a *new* payment.
+
+### What none of this catches
+
+Three cases where Level 2 cannot help, all left to Level 1:
+
+- **A one-off special distribution.** It is on no schedule, so it is
+  found only when the expiry clock runs out.
+- **A fund that changes its schedule** -- quarterly to monthly, say. The
+  old pattern persists for a few periods, so the estimate runs late.
+- **A symbol with fewer than three cached payments.** No pattern to work
+  from, which is exactly why that case gets the 6-day clock.
 
-\* **Earnings smart refresh:** even inside the 30-day window, cached
-earnings re-fetch automatically once an earnings date has passed but
-the cache still lacks the actual result -- so numbers appear promptly
-after an announcement without daily polling.
 
 ## Market-aware candle freshness
 

docs/explanation/data-providers.md

@@ -14,11 +14,19 @@ Each data type has a primary provider chosen for coverage and quality:
 |-------------------------------|----------------------|-----------------------------------------------|
 | Daily candles                 | Tiingo               | Deep history; stocks, ETFs, mutual funds      |
 | Real-time quotes              | Yahoo                | No key required                               |
-| Dividends / splits            | Polygon              | Carries forward-looking declared events       |
+| Dividends / splits            | Polygon              | Carries forward-looking declared events\*      |
 | Options chains                | CBOE                 | No key; 15-minute delayed                     |
 | Earnings                      | FMP                  | Actuals + analyst estimates                   |
 | ETF profiles / classification | SEC EDGAR + Wikidata | Authoritative holdings; needs a contact email |
 
+\* Forward-looking for **splits and operating-company dividends**, which
+are declared weeks to months ahead. It is not a promise about ETF
+distributions: a fund's per-share amount is the pass-through of what it
+earned, so it does not exist until roughly the fund's own ex-date no
+matter which provider you ask. That is why dividends carry a short TTL
+plus a schedule check -- see
+[Caching and data freshness](caching.md).
+
 The [data service](concepts.md#the-data-service-and-caching) hides this
 behind one interface -- commands ask for "candles for VTI," not "call
 Tiingo." That's also what makes the cache and rate limiting uniform

docs/guides/offline-and-refresh.md

@@ -66,15 +66,27 @@ In `auto` mode, each data type has its own time-to-live:
 | Data               | TTL                                            |
 |--------------------|------------------------------------------------|
 | Daily candles      | ~24 hours                                      |
-| Dividends / splits | 14 days                                        |
+| Dividends          | 6 or 14 days (see note below)                  |
+| Splits             | 14 days                                        |
 | Options            | 1 hour                                         |
 | Earnings           | 30 days (refreshed early once a result is due) |
 | ETF profiles       | ~30 days                                       |
 | Quotes             | never cached                                   |
 
 So a second `portfolio` run the same day is instant and network-free
-without any flag. For the full rationale, see
-[Caching and data freshness](../explanation/caching.md).
+without any flag.
+
+Dividends and earnings are the exception to "the clock decides". For
+those, once the clock says an entry is still fresh, a second check asks
+whether it is nonetheless *incomplete* -- for dividends, whether a payment
+the symbol's own schedule predicts is missing from it. An ETF's amount is
+only knowable for a few days before the cash arrives, so a fixed expiry
+can miss the whole event. A third check stops the second one asking the
+provider more than once every 12 hours. The dividend clock itself is 14
+days when the schedule is known and 6 days when it is not.
+
+For the full rationale and a worked example, see
+[Caching and data freshness](../explanation/caching.md#dividends-a-three-level-freshness-check).
 
 ## Inspecting and clearing the cache
 

docs/reference/providers.md

@@ -10,19 +10,26 @@ how keys are configured see
 
 ## Summary
 
-| Data                  | Provider         | Auth                          | Free-tier limit                                | Cache TTL  |
-|-----------------------|------------------|-------------------------------|------------------------------------------------|------------|
-| Daily candles (OHLCV) | Tiingo           | `TIINGO_API_KEY`              | 1,000/day, 50/hour (free); 10,000/hour (Power) | ~24h       |
-| Real-time quotes      | Yahoo            | none                          | unofficial                                     | never      |
-| Real-time IEX quotes  | Tiingo           | `TIINGO_API_KEY`              | any tier (1 req per connect/batch)             | never      |
-| Quote fallback        | TwelveData       | `TWELVEDATA_API_KEY`          | 8/min, 800/day                                 | never      |
-| Dividends             | Polygon          | `POLYGON_API_KEY`             | 5/min                                          | 14 days    |
-| Splits                | Polygon          | `POLYGON_API_KEY`             | 5/min                                          | 14 days    |
-| Options chains        | CBOE             | none                          | ~30/min (self-imposed)                         | 1 hour     |
-| Earnings              | FMP              | `FMP_API_KEY`                 | 250 req/day                                    | 30 days    |
-| ETF profiles          | SEC EDGAR        | `ZFIN_USER_EMAIL`             | 10/sec                                         | ~90 days   |
-| Classification        | Wikidata + EDGAR | `ZFIN_USER_EMAIL`             | no daily quota                                 | long-lived |
-| CUSIP lookup          | OpenFIGI         | `OPENFIGI_API_KEY` (optional) | higher with key                                | indefinite |
+| Data                  | Provider         | Auth                          | Free-tier limit                                | Cache TTL   |
+|-----------------------|------------------|-------------------------------|------------------------------------------------|-------------|
+| Daily candles (OHLCV) | Tiingo           | `TIINGO_API_KEY`              | 1,000/day, 50/hour (free); 10,000/hour (Power) | ~24h        |
+| Real-time quotes      | Yahoo            | none                          | unofficial                                     | never       |
+| Real-time IEX quotes  | Tiingo           | `TIINGO_API_KEY`              | any tier (1 req per connect/batch)             | never       |
+| Quote fallback        | TwelveData       | `TWELVEDATA_API_KEY`          | 8/min, 800/day                                 | never       |
+| Dividends             | Polygon          | `POLYGON_API_KEY`             | 5/min                                          | 6 or 14 d\* |
+| Splits                | Polygon          | `POLYGON_API_KEY`             | 5/min                                          | 14 days     |
+| Options chains        | CBOE             | none                          | ~30/min (self-imposed)                         | 1 hour      |
+| Earnings              | FMP              | `FMP_API_KEY`                 | 250 req/day                                    | 30 days     |
+| ETF profiles          | SEC EDGAR        | `ZFIN_USER_EMAIL`             | 10/sec                                         | ~90 days    |
+| Classification        | Wikidata + EDGAR | `ZFIN_USER_EMAIL`             | no daily quota                                 | long-lived  |
+| CUSIP lookup          | OpenFIGI         | `OPENFIGI_API_KEY` (optional) | higher with key                                | indefinite  |
+
+\* Dividends get three freshness checks rather than one. The expiry time
+shown here is only the first; when it says an entry is still fresh, a
+second check asks whether a payment the symbol's schedule predicts is
+missing from it, and a third limits how often that can hit Polygon. 14
+days when the schedule is known, 6 days when it is not. See
+[Caching and data freshness](../explanation/caching.md#dividends-a-three-level-freshness-check).
 
 ## Where to get a key
 

src/cache/store.zig

@@ -35,9 +35,49 @@ pub const Ttl = struct {
     const s_per_day = std.time.s_per_day;
     /// Historical candles older than 1 day never expire
     pub const candles_historical: i64 = -1; // infinite
-    /// Dividend data refreshes biweekly
-    pub const dividends: i64 = 14 * s_per_day;
-    /// Split data refreshes biweekly
+    /// How long a dividend cache entry stays fresh when we DON'T know the
+    /// symbol's payment schedule. Six days.
+    ///
+    /// This is the cautious case. With no schedule to predict from, the
+    /// clock is the only thing that will ever make us look again, so it
+    /// has to tick faster than the weekly review - otherwise a newly
+    /// bought holding's first distribution can be paid and spent before
+    /// anything notices. Six days is the largest value that still
+    /// guarantees it: the +/-11% jitter tops out at 6.66 days, which is
+    /// under 7. Seven days would reach 7.77 and could stay "fresh"
+    /// straight through a review.
+    ///
+    /// It is also what `DataType.dividends.ttl()` returns, which is what
+    /// `writeSupplement` falls back to when there is no existing file -
+    /// i.e. the first time we ever see a symbol. That is exactly the case
+    /// this value is for.
+    ///
+    /// See `dividends_scheduled` for the case where we DO know the
+    /// schedule, and `dividendsNeedRefresh` in service.zig for how the
+    /// schedule is worked out.
+    pub const dividends: i64 = 6 * s_per_day;
+
+    /// How long a dividend cache entry stays fresh when we DO know the
+    /// symbol's payment schedule. Fourteen days.
+    ///
+    /// Long on purpose. Once a symbol has enough history to establish a
+    /// cadence, `dividendsNeedRefresh` predicts when the next payment is
+    /// due and asks the provider then - so the clock is no longer how we
+    /// find out. Making it short as well would re-check every symbol
+    /// every few days to protect against a problem the prediction has
+    /// already solved.
+    ///
+    /// What the clock still covers here is the one thing a schedule
+    /// cannot predict: an off-cycle special distribution. That can be up
+    /// to 14 days late being noticed, which is the deliberate price of
+    /// not re-checking every symbol constantly.
+    pub const dividends_scheduled: i64 = 14 * s_per_day;
+
+    /// Split data refreshes biweekly. Deliberately NOT shortened
+    /// alongside `dividends`: splits are announced weeks to months
+    /// ahead of their effective date, so a forward-looking record is
+    /// already cached long before it matters and none of the
+    /// short-window exposure above applies.
     pub const splits: i64 = 14 * s_per_day;
     /// Options chains refresh hourly
     pub const options: i64 = std.time.s_per_hour;
@@ -226,9 +266,16 @@ pub const DataType = enum {
     ///
     /// Jitter assignments:
     ///
-    ///   - 11% on dividends/splits (14d base, ~3d total spread).
-    ///     Tuned so a daily cron sees roughly 1/3 of a portfolio's
-    ///     symbols expire each day instead of all in lockstep.
+    ///   - 11% on dividends and splits. Dividends have two base values
+    ///     (6d unscheduled / 14d scheduled - see `Ttl.dividends`) and
+    ///     splits one (14d); all three keep the same percentage, because
+    ///     the goal is the same: a daily cron should see a portfolio's
+    ///     symbols expire across several days rather than all at once.
+    ///
+    ///     Jitter is a load-spreading measure only, never a correctness
+    ///     one. `dividendsNeedRefresh` is what guarantees a due payment
+    ///     is noticed; a run that depended on jitter landing favourably
+    ///     would be relying on luck.
     ///
     ///   - 8% on the longer-TTL types (classification 90d,
     ///     etf_metrics 90d, entity_facts 30d, earnings 30d,
@@ -2567,7 +2614,7 @@ test "writeMerged Dividend: no-change merge still rewrites to refresh expires" {
     s.writeWithSource(Dividend, "TEST", repeat[0..], .{ .seconds = Ttl.dividends }, "polygon");
 
     // Confirm fresh expires landed on disk: read the raw file and
-    // parse out the directive, expecting it to be roughly now+14d.
+    // parse out the directive, expecting it to be roughly now+TTL.
     const path = try std.fs.path.join(allocator, &.{ dir_path, "TEST", "dividends.srf" });
     defer allocator.free(path);
     const data = try std.Io.Dir.cwd().readFileAlloc(io, path, allocator, .limited(1024 * 1024));
@@ -2577,9 +2624,15 @@ test "writeMerged Dividend: no-change merge still rewrites to refresh expires" {
     const it = try srf.iterator(&reader, allocator, .{ .parse_allocator = .none });
     defer it.deinit();
 
+    // Bound derived from the constant and its jitter rather than
+    // hardcoded: this test is about "was the file rewritten with a
+    // fresh clock", not about how long the TTL happens to be, and a
+    // literal here made a TTL change look like a merge regression.
+    const spec = DataType.dividends.ttl();
+    const jitter_floor = @divFloor(spec.seconds * (100 - @as(i64, spec.jitter_pct)), 100);
     const new_expires = it.expires orelse return error.ExpiresMissing;
     try std.testing.expect(new_expires > now_s);
-    try std.testing.expect(new_expires - now_s > 13 * std.time.s_per_day);
+    try std.testing.expect(new_expires - now_s >= jitter_floor);
 }
 
 test "writeSupplement Dividend: preserves an existing future expires" {
@@ -2701,10 +2754,14 @@ test "writeSupplement Dividend: no existing file establishes an initial TTL" {
     const it = try srf.iterator(&reader, allocator, .{ .parse_allocator = .none });
     defer it.deinit();
 
-    // ~14d TTL with ±11% jitter; comfortably bracket the band.
+    // The jittered band, derived from the constant rather than
+    // hardcoded - this test is about "a first supplement stamps a
+    // normal TTL", not about the TTL's length.
+    const spec = DataType.dividends.ttl();
+    const jitter = @divFloor(spec.seconds * @as(i64, spec.jitter_pct), 100);
     const new_expires = it.expires orelse return error.ExpiresMissing;
-    try std.testing.expect(new_expires - now_s > 13 * std.time.s_per_day);
-    try std.testing.expect(new_expires - now_s < 16 * std.time.s_per_day);
+    try std.testing.expect(new_expires - now_s >= spec.seconds - jitter);
+    try std.testing.expect(new_expires - now_s <= spec.seconds + jitter);
 }
 
 test "writeMerged Dividend: field-level upgrade fills nulls (Tiingo-then-Polygon)" {
@@ -3368,10 +3425,25 @@ test "TTL constants are reasonable" {
     // Latest candles use a market-aware expiry computed per-write by
     // market.nextCandleExpiry (no fixed TTL constant here anymore).
 
-    // Dividends and splits refresh biweekly
-    try std.testing.expectEqual(@as(i64, 14 * std.time.s_per_day), Ttl.dividends);
+    // Dividends have two values depending on whether we know the
+    // symbol's payment schedule; splits have one. See `Ttl.dividends`.
+    try std.testing.expectEqual(@as(i64, 6 * std.time.s_per_day), Ttl.dividends);
+    try std.testing.expectEqual(@as(i64, 14 * std.time.s_per_day), Ttl.dividends_scheduled);
     try std.testing.expectEqual(@as(i64, 14 * std.time.s_per_day), Ttl.splits);
 
+    // THE PROPERTY, not just the value. When we don't know a symbol's
+    // schedule the clock is the only thing that will make us look again,
+    // so it has to expire before the next weekly review - including at
+    // the top of the jitter band. 6 days * 1.11 = 6.66, under 7. At 7
+    // days the band reaches 7.77 and an entry could stay fresh straight
+    // through a review, which is the bug this guards against.
+    //
+    // Deliberately NOT asserted for `dividends_scheduled`: there the
+    // prediction finds due payments, so a long clock is correct.
+    const jitter = DataType.dividends.ttl().jitter_pct;
+    const worst_case = Ttl.dividends + @divFloor(Ttl.dividends * @as(i64, jitter), 100);
+    try std.testing.expect(worst_case < 7 * std.time.s_per_day);
+
     // Options refresh hourly
     try std.testing.expectEqual(@as(i64, std.time.s_per_hour), Ttl.options);