update documentation on dividend smart refresh

AGENTS.md

@@ -882,6 +882,13 @@ command.
 
 - **SRF string lifetimes are handled for you.** `Store.read` parses with a `parse_allocator`, so SRF dupes every owned string into the caller's allocator automatically. **Do not add a `postProcess` callback to dupe strings** - that is not a valid reason for one. `postProcess` exists only for non-trivial post-parse logic, such as recomputing a derived field (see `earningsPostProcess` in `service.zig`, which rebuilds `surprise` from `actual` and `estimate`).
 
+- **A fresh cache entry is not necessarily a complete one.** `fetchCached` takes a second comptime hook, `needsRefresh: ?*const fn ([]const T, Date) bool`, for types where TTL cannot express "is this entry missing something it should have by now". `getDividends` passes `dividendsNeedRefresh`; `getSplits` and `getOptions` pass `null`. Three rules if you add one:
+  - It is consulted at **both** fresh-read sites (local, and post-server-sync) via `serveFreshOrDiscard`. Only covering the first one means a configured `ZFIN_SERVER` defeats the mechanism, because the sync leaves the still-incomplete entry on disk and the second read serves it.
+  - `serveFreshOrDiscard` owns the free. Don't discard at a call site; a caller that decides and then forgets is a leak, and the tests rely on `std.testing.allocator` to catch exactly that.
+  - It is suppressed under `skip_network`. Offline mode never refetches, so discarding could only turn a served result into a failure.
+
+  The branch is comptime-elided for a `null` argument, which is what lets `Split` - a type with no `freeSlice` - keep compiling. See `docs/dev/caching-implementation.md` for the dividend rationale and its three documented blind spots.
+
 - **Buffered stdout.** CLI output uses a single `std.Io.Writer` with a 4096-byte stack buffer, flushed once at the end of `main()`. Don't write to stdout through other means.
 
 - **The `color` parameter flows through everything.** CLI commands accept a `color: bool` parameter. Don't use ANSI escapes unconditionally - always gate on the `color` flag.

README.md

@@ -84,13 +84,24 @@ it does best, with aggressive caching to stay within free-tier limits.
 | Daily candles (OHLCV) | Tiingo           | `TIINGO_API_KEY`     | 1,000 req/day, 50 req/hour | 23h45m       |
 | Real-time quotes      | Yahoo Finance    | None required        | Unofficial                 | Never cached |
 | Quote fallback        | TwelveData       | `TWELVEDATA_API_KEY` | 8 req/min, 800/day         | Never cached |
-| Dividends             | Polygon          | `POLYGON_API_KEY`    | 5 req/min                  | 14 days      |
+| Dividends             | Polygon          | `POLYGON_API_KEY`    | 5 req/min                  | 6 days\*\*   |
 | Splits                | Polygon          | `POLYGON_API_KEY`    | 5 req/min                  | 14 days      |
 | Options chains        | CBOE             | None required        | ~30 req/min (self-imposed) | 1 hour       |
 | Earnings              | FMP              | `FMP_API_KEY`        | 250 req/day                | 30 days*     |
 | ETF profiles          | SEC EDGAR        | `ZFIN_USER_EMAIL`    | 10 req/sec                 | ~90 days     |
 | Classification        | Wikidata + EDGAR | `ZFIN_USER_EMAIL`    | No per-day quota           | Long-lived   |
 
+\* **Earnings smart refresh** -- cached earnings re-fetch inside the
+30-day window once a report date has passed but the actual result is
+still missing.
+
+\*\* **Dividend smart refresh** -- the TTL is only a backstop. Cached
+dividends also re-fetch whenever a distribution the symbol's own
+schedule says is due has not arrived. An ETF's amount is not published
+until roughly its own ex-date and it pays 1-7 days later, so a long TTL
+can be written before the record exists and still be fresh after the
+cash has landed.
+
 Not all keys are required; without a given key, that data type is
 simply unavailable. For per-provider notes, signup links, the full
 caching/TTL model, and the complete environment-variable list, see

docs/dev/caching-implementation.md

@@ -146,8 +146,8 @@ in `DataType.ttl()`.
 |----------------------|------------------|--------|--------------------------------------------------|
 | Daily candles        | market-aware     | n/a    | Boundary set by `market.nextCandleExpiry` (below)|
 | Historical candles   | never (`-1`)     | n/a    | Bars older than ~1 day are immutable             |
-| Dividends            | 14 days          | 11%    | Declared well in advance                         |
+| Dividends            | 6 days           | 11%    | Backstop only; see dividend smart refresh below  |
-| Splits               | 14 days          | 11%    | Rare corporate events                            |
+| Splits               | 14 days          | 11%    | Rare corporate events, announced weeks ahead     |
 | Options              | 1 hour           | 0      | Move continuously during market hours            |
 | Earnings             | 30 days          | 8%     | Smart-refresh after an announcement date passes  |
 | Classification       | 90 days          | 8%     | Sector/industry/country from Wikidata            |
@@ -309,14 +309,97 @@ Everything that is not candles flows through the generic `fetchCached`,
 which is simpler because each type is a single file:
 
 1. `.fresh_only` read; a fresh entry (including a negative one) returns
-   immediately.
+   immediately -- **unless** the type supplies a `needsRefresh` hook that
+   says the fresh entry is incomplete (see below).
 2. `skip_network`: return any cached entry, even stale; else
    `FetchFailed`.
-3. Server sync (if configured); a fresh synced entry returns.
+3. Server sync (if configured); a fresh synced entry returns, subject to
+   the same `needsRefresh` check.
 4. Provider fetch; on success write with the type's TTL; on
    `NotFound` write a negative entry; on transient error return
    `FetchFailed` without poisoning the cache.
 
+#### The two comptime hooks
+
+`fetchCached` takes two optional comptime function pointers. Both default
+to "absent" by passing `null`, which is the behaviour every type had
+before they existed.
+
+| Hook            | Signature                                  | Purpose                                             |
+|-----------------|--------------------------------------------|-----------------------------------------------------|
+| `postProcess`   | `fn (*T, Allocator) anyerror!void`         | Recompute a derived field after parse                |
+| `needsRefresh`  | `fn ([]const T, Date) bool`                | "This entry is fresh, but is it *complete*?"         |
+
+`postProcess` is **not** for duping strings -- `Store.read` already does
+that. Today only `EarningsEvent` uses it (to rebuild `surprise`).
+
+`needsRefresh` exists because freshness and completeness are different
+questions. It is consulted at **both** fresh-read sites (step 1 and step
+3) through `serveFreshOrDiscard`, which owns the free so a caller cannot
+decide to discard and then leak. Covering step 3 is not optional: with a
+`ZFIN_SERVER` configured, a sync can leave a still-incomplete entry on
+disk that step 3 would otherwise serve, defeating the mechanism
+entirely. The hook is suppressed under `skip_network`, since offline
+mode never refetches and discarding could only turn a served result into
+a failure.
+
+The hook's branch is comptime-elided when the argument is `null`, which
+is what lets types without a `freeSlice` (such as `Split`) keep
+compiling.
+
+#### Dividend smart refresh
+
+`dividendsNeedRefresh` is the hook `getDividends` passes. It answers
+true when at least one full cadence period has elapsed since the newest
+cached ex-date and the distribution that should have landed in it is
+still absent.
+
+The problem it solves is that dividend TTL and dividend *availability*
+are uncorrelated. An operating company declares weeks ahead of the
+ex-date and pays about a month after it. An ETF cannot: its distribution
+is the pass-through of whatever the fund earned, so the amount is not
+published until roughly its own ex-date, and it pays 1-7 days later. The
+window in which the record exists but the cash has not yet landed is
+about a week, so any longer TTL can be written before the record exists
+and still be fresh after the payment. Quarter-end clustering makes that
+systematic rather than unlucky -- a dozen funds go ex within a few days
+of each other, so one refresh pass puts all of them on the same expiry
+shelf.
+
+Cadence comes from `dividendCadenceDays`: the **median** of up to the
+five newest gaps between cached ex-dates. Median rather than mean (a
+special distribution landing days after a regular one drags the mean) and
+rather than minimum (the same outlier destroys the minimum outright,
+predicting a next ex-date days away and firing for the whole chase window
+after every payment). `newestExDates` sorts rather than trusting file
+order, because `writeSupplement`'s merge interleaves two providers'
+records and a cadence computed from negative gaps would forecast into the
+past forever.
+
+Three properties are load-bearing and each has its own test:
+
+- **A full period must elapse.** Otherwise it refires immediately after
+  a successful fetch, on the record it just stored.
+- **It rolls forward.** The expected date is the latest one at or before
+  today, not the first one after the newest cached record -- so a symbol
+  two periods behind stays recoverable instead of being permanently past
+  its window.
+- **The chase window is bounded** (`dividend_chase_days`, 14). An
+  unbounded chase refetches forever whenever a sponsor skips a period or
+  the cadence estimate is wrong.
+
+Known blind spots, all three deliberately left to the TTL backstop
+rather than papered over with a more eager statistic that would cost
+requests on every symbol to protect against a rare event on one:
+
+- A **cadence change** (quarterly to monthly) keeps the old median for
+  several periods, so the forecast runs late.
+- An **off-cycle special** is on no schedule, so the predicate never
+  fires for it.
+- A symbol with **fewer than three cached records** has no cadence at
+  all -- which is the right direction, since a newly-bought holding has
+  no schedule to be late against.
+
 ### `loadAllPrices` (portfolio + watchlist price load)
 
 The portfolio price load batches all symbols through three phases.
@@ -559,6 +642,9 @@ the upstream provider on every request.
 |---------------------------------|-----------------------------------------------------|
 | Data-access entry point         | `DataService` - `src/service.zig`                   |
 | Per-type generic fetch          | `fetchCached` - `src/service.zig`                   |
+| Fresh-but-incomplete decision   | `serveFreshOrDiscard` - `src/service.zig`           |
+| Dividend schedule check         | `dividendsNeedRefresh`, `dividendCadenceDays` - `src/service.zig` |
+| Earnings smart refresh          | `earningsNeedsRefresh` - `src/service.zig`          |
 | Candle fetch + incremental      | `getCandles` - `src/service.zig`                    |
 | Batch price load                | `loadAllPrices` - `src/service.zig`                 |
 | Live quotes (uncached)          | `loadLiveQuotes`, `getQuote` - `src/service.zig`    |

docs/explanation/caching.md

@@ -42,14 +42,38 @@ Different data ages at different rates, so each type has its own TTL:
 | Data type     | TTL           | Why                                                                              |
 |---------------|---------------|----------------------------------------------------------------------------------|
 | Daily candles | market-aware  | Keyed to the next time a fresh bar is expected, not a rolling window (see below) |
-| Dividends     | 14 days       | Declared well in advance                                                         |
+| Dividends     | 6 days\*      | Short: an ETF distribution is only knowable for a few days before it pays        |
-| Splits        | 14 days       | Rare corporate events                                                            |
+| Splits        | 14 days       | Rare corporate events, announced weeks ahead                                     |
 | Options       | 1 hour        | Prices move continuously when markets are open                                   |
-| Earnings      | 30 days\*     | Quarterly; smart-refreshed around announcements                                  |
+| Earnings      | 30 days\*\*   | Quarterly; smart-refreshed around announcements                                  |
 | ETF profiles  | ~30 days      | Holdings and weights change slowly                                               |
 | Quotes        | never cached  | Meant to be a live price check                                                   |
 
-\* **Earnings smart refresh:** even inside the 30-day window, cached
+\* **Dividend smart refresh:** even inside the 6-day window, cached
+dividends re-fetch automatically once a distribution the symbol's own
+schedule says is due has not arrived. This exists because a dividend
+cache being *fresh* and being *complete* are different things.
+
+An operating company declares a dividend weeks ahead of the ex-date and
+pays about a month after it, so there is plenty of time to notice. An
+**ETF does not work that way.** Its distribution is the pass-through of
+whatever the fund earned, so the amount is not published until roughly
+its own ex-date -- and it pays 1 to 7 days later. The entire window in
+which the record exists but the cash has not yet landed is about a week.
+
+A long TTL straddles that window whole: a cache written a few days
+before the ex-date stays "fresh" until well after the payment, so the
+distribution is invisible to the run that needed it. Quarter-end makes
+it systematic rather than unlucky, because a dozen funds go ex within
+the same few days and one refresh pass puts all of them on the same
+expiry shelf.
+
+So the schedule is the gate, and the TTL is only the backstop. It still
+matters, because the schedule check cannot see three things: a holding
+too new to have an established cadence, an off-cycle special
+distribution, and a symbol with fewer than three cached records.
+
+\*\* **Earnings smart refresh:** even inside the 30-day window, cached
 earnings re-fetch automatically once an earnings date has passed but
 the cache still lacks the actual result -- so numbers appear promptly
 after an announcement without daily polling.

docs/explanation/data-providers.md

@@ -14,11 +14,19 @@ Each data type has a primary provider chosen for coverage and quality:
 |-------------------------------|----------------------|-----------------------------------------------|
 | Daily candles                 | Tiingo               | Deep history; stocks, ETFs, mutual funds      |
 | Real-time quotes              | Yahoo                | No key required                               |
-| Dividends / splits            | Polygon              | Carries forward-looking declared events       |
+| Dividends / splits            | Polygon              | Carries forward-looking declared events\*      |
 | Options chains                | CBOE                 | No key; 15-minute delayed                     |
 | Earnings                      | FMP                  | Actuals + analyst estimates                   |
 | ETF profiles / classification | SEC EDGAR + Wikidata | Authoritative holdings; needs a contact email |
 
+\* Forward-looking for **splits and operating-company dividends**, which
+are declared weeks to months ahead. It is not a promise about ETF
+distributions: a fund's per-share amount is the pass-through of what it
+earned, so it does not exist until roughly the fund's own ex-date no
+matter which provider you ask. That is why dividends carry a short TTL
+plus a schedule check -- see
+[Caching and data freshness](caching.md).
+
 The [data service](concepts.md#the-data-service-and-caching) hides this
 behind one interface -- commands ask for "candles for VTI," not "call
 Tiingo." That's also what makes the cache and rate limiting uniform

docs/guides/offline-and-refresh.md

@@ -66,7 +66,8 @@ In `auto` mode, each data type has its own time-to-live:
 | Data               | TTL                                            |
 |--------------------|------------------------------------------------|
 | Daily candles      | ~24 hours                                      |
-| Dividends / splits | 14 days                                        |
+| Dividends          | 6 days (refreshed early once a payout is due)  |
+| Splits             | 14 days                                        |
 | Options            | 1 hour                                         |
 | Earnings           | 30 days (refreshed early once a result is due) |
 | ETF profiles       | ~30 days                                       |

docs/reference/providers.md

@@ -16,7 +16,7 @@ how keys are configured see
 | Real-time quotes      | Yahoo            | none                          | unofficial                                     | never      |
 | Real-time IEX quotes  | Tiingo           | `TIINGO_API_KEY`              | any tier (1 req per connect/batch)             | never      |
 | Quote fallback        | TwelveData       | `TWELVEDATA_API_KEY`          | 8/min, 800/day                                 | never      |
-| Dividends             | Polygon          | `POLYGON_API_KEY`             | 5/min                                          | 14 days    |
+| Dividends             | Polygon          | `POLYGON_API_KEY`             | 5/min                                          | 6 days\*   |
 | Splits                | Polygon          | `POLYGON_API_KEY`             | 5/min                                          | 14 days    |
 | Options chains        | CBOE             | none                          | ~30/min (self-imposed)                         | 1 hour     |
 | Earnings              | FMP              | `FMP_API_KEY`                 | 250 req/day                                    | 30 days    |
@@ -24,6 +24,12 @@ how keys are configured see
 | Classification        | Wikidata + EDGAR | `ZFIN_USER_EMAIL`             | no daily quota                                 | long-lived |
 | CUSIP lookup          | OpenFIGI         | `OPENFIGI_API_KEY` (optional) | higher with key                                | indefinite |
 
+\* The dividend TTL is only a backstop. Cached dividends also re-fetch
+whenever a distribution the symbol's own schedule says is due has not
+arrived, because an ETF's amount is not published until roughly its
+ex-date and it pays within a week -- see
+[Caching and data freshness](../explanation/caching.md).
+
 ## Where to get a key
 
 | Provider   | Sign up                             | Notes                                                                                |