support external provider
All checks were successful
Generic zig build / build (push) Successful in 5m9s
Generic zig build / publish-macos (push) Successful in 10s
Generic zig build / deploy (push) Successful in 29s

This commit is contained in:
Emil Lerch 2026-08-28 11:02:18 -07:00
parent 18f4f00516
commit df204a3eb2
Signed by: lobo
GPG key ID: A7B62D657EF764F8
5 changed files with 412 additions and 18 deletions

View file

@ -220,6 +220,16 @@ and Tiingo was never consulted again. The backoff is armed only by a
genuine 404, expires after `Ttl.tiingo_backoff` with per-symbol jitter, genuine 404, expires after `Ttl.tiingo_backoff` with per-symbol jitter,
and clears the moment Tiingo serves the symbol again. and clears the moment Tiingo serves the symbol again.
`CandleProvider` has four variants - `twelvedata`, `yahoo`, `tiingo`,
`external` - and exactly one of them is special-cased anywhere:
`twelvedata`, whose `adj_close` was unreliable, so a cache carrying it is
treated as unusable. That check exists in **two** places, one per entry
into the meta-exists branch (the `skip_network` path reports the symbol
unavailable; the online path forces a full re-fetch). `external` is
covered under [Externally-managed candle
series](#externally-managed-candle-series); it is inert on every serve
path and vetoes only the negative-cache write.
```mermaid ```mermaid
flowchart TD flowchart TD
S["getCandles(symbol, opts)"] --> NG{"negative candles_daily and not force_refresh?"} S["getCandles(symbol, opts)"] --> NG{"negative candles_daily and not force_refresh?"}
@ -227,7 +237,9 @@ flowchart TD
NG -->|no| RM{"candles_meta exists?"} NG -->|no| RM{"candles_meta exists?"}
RM -->|yes| SK{"skip_network?"} RM -->|yes| SK{"skip_network?"}
SK -->|yes| RETS["return cached even if stale<br/>(FetchFailed if unreadable)"] SK -->|yes| TWS{"provider is twelvedata?"}
TWS -->|yes| FF
TWS -->|no| RETS["return cached even if stale<br/>(FetchFailed if unreadable)"]
SK -->|no| TW{"provider is twelvedata?"} SK -->|no| TW{"provider is twelvedata?"}
TW -->|yes| FULL TW -->|yes| FULL
TW -->|no| FR{"meta fresh and not force_refresh?"} TW -->|no| FR{"meta fresh and not force_refresh?"}
@ -252,7 +264,9 @@ flowchart TD
SF2 -->|no| FULL["refetchFullHistory: Tiingo, then Yahoo"] SF2 -->|no| FULL["refetchFullHistory: Tiingo, then Yahoo"]
FULL --> RES{"result?"} FULL --> RES{"result?"}
RES -->|ok| RET2 RES -->|ok| RET2
RES -->|"NotFound (EVERY provider disclaims it)"| WN["writeNegative candles_daily"] RES -->|"NotFound (EVERY provider disclaims it)"| SNC{"prior meta says provider is external?"}
SNC -->|yes| FF
SNC -->|no| WN["writeNegative candles_daily"]
WN --> FF WN --> FF
RES -->|transient| TR["bump fail_count, TransientError"] RES -->|transient| TR["bump fail_count, TransientError"]
RES -->|other| FF RES -->|other| FF
@ -354,6 +368,12 @@ re-run the dead lookup on every invocation. The entry is the sentinel:
writes that file; `isCandleMetaFresh`, `getCachedCandles`, and the writes that file; `isCandleMetaFresh`, `getCachedCandles`, and the
`getCandles` short-circuit all recognize it there. `candles_meta.srf` `getCandles` short-circuit all recognize it there. `candles_meta.srf`
is intentionally not created for a no-data symbol. is intentionally not created for a no-data symbol.
- **Externally-managed series are exempt.** A prior meta saying
`provider::external` vetoes the write via
`DataService.shouldNegativeCache`, because for a series no provider was
ever going to carry, the marker is unrecoverable rather than merely
sticky. See [Externally-managed candle
series](#externally-managed-candle-series).
## Candle-less symbols (crypto and friends) ## Candle-less symbols (crypto and friends)
@ -391,6 +411,88 @@ If you want a candle-less symbol to be re-checked against the provider
negative entry with `cache clear` or `--refresh-data=force`; the live negative entry with `cache clear` or `--refresh-data=force`; the live
quote and manual-price paths are unaffected by the negative cache. quote and manual-price paths are unaffected by the negative cache.
## Externally-managed candle series
`CandleMeta.provider == .external` labels a series that was produced and
is maintained **outside zfin**, by something that writes
`candles_daily.srf` and `candles_meta.srf` directly into a cache
directory. The motivating case is a unitized trust with no ticker and no
CUSIP, whose daily unit values come from a plan recordkeeper's feed;
every market-data provider 404s it.
Like the previous section, this is a symbol no provider carries. The
difference is that a candle-less symbol has no history *anywhere*, so
negative-caching it is correct, whereas an externally-managed symbol has
a real series that simply did not come from a provider. The label exists
so `candles_meta.srf` can say so honestly instead of naming a provider
that never served it.
Two population routes:
- **Server**: a job writes the pair straight into the `ZFIN_SERVER`
cache directory. That copy is the master.
- **Client**: the existing server-sync path (phase 2 of `loadAllPrices`,
and the two `syncCandlesFromServer` calls inside `getCandles`) pulls
the bytes down verbatim, exactly as it does for any other symbol. No
client-side special-casing is involved.
The label is **pure provenance and drives no routing**. zfin still walks
the normal provider chain for such a symbol and still 404s on every pass;
restricting it to the server tier and skipping the chain entirely is
deferred work. There is exactly one behavior attached to the label, and
it is a veto rather than a routing decision:
**A negative-cache entry on an external symbol is unrecoverable.**
`writeNegative` overwrites `candles_daily.srf` with a marker, negative
entries never expire, and `getCandles` short-circuits on `isNegative`
*before* it reaches `syncCandlesFromServer` - so the marker makes the
only surviving copy permanently unreachable. On a server it is worse:
the externally-populated cache is the master copy, so nothing survives.
`DataService.shouldNegativeCache` therefore refuses the write when the
prior meta says `.external`.
That guard is deliberately partial. It consults the prior meta, so it
covers the case that destroys data: meta present, `candles_daily.srf`
missing or unreadable (a partially-completed server sync writes the two
files in sequence and needs both to report success), which falls through
to the cold-start path. It cannot cover a **true cold start** with both
files absent - there is no label to read - so an external symbol first
touched while `ZFIN_SERVER` is unreachable is poisoned until
`--refresh-data=force` or `cache clear`. Closing that needs the
`isNegative` short-circuit moved after the server tier, which would
re-hit the network for every legitimately candle-less symbol on every
run. It belongs with the deferred routing work.
### Adding a `CandleProvider` variant is a coordinated deployment
Not a backward-compatible change. `provider` has no default, so it is
never elided, and SRF's enum coercion has no lenient mode and no
`srfParse` hook (custom parsers are consulted for struct/union fields
only). A build that does not know a variant cannot fall back:
- srf `>= ea2c358` (zfin's current pin) returns
`StringValueNotValidEnumMember`. `readCandleMeta` swallows it into a
`null`, which reads as a cache miss and routes the symbol down the
cold-start path above - destructive for precisely the series that
cannot be re-fetched.
- srf `<= 4a3e5f0` unwraps a null optional and **panics**. That is what
zfin-server `7103ced` vendors, and a panic is not interceptable by the
`catch return null` every caller depends on. The blast radius is not
one symbol: `handleDiagnostics` runs `freshness.collect` over every key
in the cache directory, so a single unreadable file takes down
`/<any-symbol>/diagnostics` and the cron refresh sweep. The
byte-serving routes (`/candles`, `/candles_meta`) are unaffected, since
they never parse.
Skew is dangerous in both directions: an old **client** syncing
`candles_meta` bytes from a new server fails the same way, because
`looksCompleteSrf` and the ETag check validate shape and integrity, not
enum values.
So the ordering is: release zfin, then rebuild and deploy every server
that reads the affected cache directory, and only then may a file
carrying the new value exist.
## Server sync (the optional L2 tier) ## Server sync (the optional L2 tier)
`ZFIN_SERVER` points zfin at a `ZFIN_SERVER` points zfin at a
@ -447,6 +549,8 @@ the upstream provider on every request.
| Freshness check | `isFresh` (SRF), `isCandleMetaFresh` - `store.zig` | | Freshness check | `isFresh` (SRF), `isCandleMetaFresh` - `store.zig` |
| TTLs and expiry computation | `Ttl`, `computeExpires` - `src/cache/store.zig` | | TTLs and expiry computation | `Ttl`, `computeExpires` - `src/cache/store.zig` |
| Negative cache | `writeNegative`, `isNegative` - `src/cache/store.zig` | | Negative cache | `writeNegative`, `isNegative` - `src/cache/store.zig` |
| Negative-cache veto | `shouldNegativeCache` - `src/service.zig` |
| Candle provenance labels | `CandleProvider` - `src/cache/store.zig` |
| NotFound classification | `isPermanentProviderFailure` - `src/service.zig` | | NotFound classification | `isPermanentProviderFailure` - `src/service.zig` |
| Market-aware candle expiry | `nextCandleExpiry`, `shouldRefresh` - `src/market.zig` | | Market-aware candle expiry | `nextCandleExpiry`, `shouldRefresh` - `src/market.zig` |
| Price fallback (manual/avg-cost)| `buildFallbackPrices` - `src/analytics/valuation.zig` | | Price fallback (manual/avg-cost)| `buildFallbackPrices` - `src/analytics/valuation.zig` |

View file

@ -138,6 +138,13 @@ ticker, say -- zfin records a negative cache entry so it doesn't retry
the same dead lookup on every run. (Transient failures like rate limits the same dead lookup on every run. (Transient failures like rate limits
are not cached this way; they're retried.) are not cached this way; they're retried.)
One exception: a price history that was loaded into the cache from
outside zfin, rather than fetched from a provider, is never
negative-cached. Such a series has no provider to re-fetch it from, so
recording "nobody has this" would discard the only copy. `zfin diagnose
SYMBOL` names the source of a symbol's bars, and reports `external` for
these.
## Symbols without candle data (crypto) ## Symbols without candle data (crypto)
A few holdings have no daily price history available from zfin's candle A few holdings have no daily price history available from zfin's candle

160
src/cache/store.zig vendored
View file

@ -1415,6 +1415,18 @@ pub const Store = struct {
/// which writes a new meta file with the provider explicit. /// which writes a new meta file with the provider explicit.
/// The wipe happens naturally on first use post-upgrade. /// The wipe happens naturally on first use post-upgrade.
/// ///
/// A field that is *present but carries a value this build
/// does not know* - i.e. a cache written by a NEWER zfin that
/// has added a variant - is a different and nastier failure.
/// SRF's enum coercion has no lenient mode and no `srfParse`
/// hook (it consults custom parsers only for struct/union
/// fields), so the value either raises
/// `StringValueNotValidEnumMember` (srf >= ea2c358, swallowed
/// into the same destructive cache-miss path above) or panics
/// on an unchecked optional unwrap (srf <= 4a3e5f0). Adding a
/// variant is therefore a coordinated deployment, not a
/// backward-compatible change; see `CandleProvider.external`.
///
/// This is also why no *other* field on this struct may be /// This is also why no *other* field on this struct may be
/// default-less: that cold-start path is the destructive one. /// default-less: that cold-start path is the destructive one.
/// It writes a negative-cache marker over `candles_daily.srf` /// It writes a negative-cache marker over `candles_daily.srf`
@ -1527,12 +1539,52 @@ pub const Store = struct {
/// dividends and splits from the same response, and its /// dividends and splits from the same response, and its
/// `adj_close` is what the analytics layer is written against. /// `adj_close` is what the analytics layer is written against.
tiingo, tiingo,
/// Candles were produced and are managed OUTSIDE zfin, by
pub fn fromString(s: []const u8) CandleProvider { /// something that writes `candles_daily.srf` and
if (std.mem.eql(u8, s, "yahoo")) return .yahoo; /// `candles_meta.srf` directly into a cache directory.
if (std.mem.eql(u8, s, "tiingo")) return .tiingo; ///
return .twelvedata; /// Pure provenance like every other variant - it records where
} /// the bars came from and drives no routing. What sets it
/// apart is what it implies about their origin: **no
/// market-data provider carries the symbol at all**. The
/// motivating case is a unitized trust with no ticker and no
/// CUSIP, whose daily unit values come from a plan
/// recordkeeper's feed; Tiingo, Yahoo and TwelveData all 404
/// it. So zfin can never fetch or restate such a series
/// itself. It arrives either by `ZFIN_SERVER` sync (the client
/// case) or by direct population of the cache directory (the
/// server case).
///
/// **A negative-cache entry on such a symbol is unrecoverable**,
/// not merely inconvenient. `writeNegative` overwrites
/// `candles_daily.srf` with a marker, negative entries never
/// expire (`isCandleMetaFresh` reports them fresh forever), and
/// `getCandles` short-circuits on `isNegative` *before* it
/// reaches `syncCandlesFromServer` - so the one copy that still
/// exists becomes permanently unreachable. On a server, where
/// the externally-populated cache IS the master copy, there is
/// no other copy at all. `shouldNegativeCache` in `service.zig`
/// refuses that write for exactly this reason; see its doc
/// comment for the case it cannot cover.
///
/// **Version compatibility is a deployment ordering
/// constraint, not a nicety.** This value is unknown to any
/// zfin built before it existed, and an older reader does not
/// degrade gracefully:
///
/// - srf >= ea2c358 (what zfin pins today) returns
/// `error.StringValueNotValidEnumMember`, which
/// `readCandleMeta` swallows into a `null` - i.e. a cache
/// miss, which routes the symbol down the destructive
/// cold-start path described on the `provider` field.
/// - srf <= 4a3e5f0 unwraps a null optional and **panics**.
/// That is what zfin-server 7103ced vendors, and a panic
/// is not interceptable by `readCandleMeta`'s `catch`.
///
/// So no file carrying `provider::external` may exist until
/// every reader of that cache directory - client and server -
/// runs a zfin that knows the variant.
external,
}; };
// Private I/O // Private I/O
@ -3710,13 +3762,95 @@ test "Store.Freshness enum values" {
try std.testing.expect(Store.Freshness.fresh_only != Store.Freshness.any); try std.testing.expect(Store.Freshness.fresh_only != Store.Freshness.any);
} }
test "CandleProvider.fromString parses provider names" { test "candles_meta with an unknown provider value reads as a cache miss" {
try std.testing.expectEqual(Store.CandleProvider.yahoo, Store.CandleProvider.fromString("yahoo")); // The forward-compatibility contract for `CandleProvider`, pinned.
try std.testing.expectEqual(Store.CandleProvider.tiingo, Store.CandleProvider.fromString("tiingo")); //
try std.testing.expectEqual(Store.CandleProvider.twelvedata, Store.CandleProvider.fromString("twelvedata")); // SRF's enum coercion has no lenient mode and no `srfParse` hook
// Unknown defaults to twelvedata // (custom parsers are consulted for struct/union fields only), so a
try std.testing.expectEqual(Store.CandleProvider.twelvedata, Store.CandleProvider.fromString("unknown")); // `provider::` value this build does not know cannot be mapped to a
try std.testing.expectEqual(Store.CandleProvider.twelvedata, Store.CandleProvider.fromString("")); // fallback. It raises `StringValueNotValidEnumMember`, which
// `deserializeCandleMeta` reports as `InvalidData` and
// `readCandleMeta` swallows into a null - i.e. the symbol looks like
// a cache miss and takes the destructive cold-start path. That is
// why adding a variant is a coordinated deployment; see
// `CandleProvider.external`.
//
// This also acts as a canary on the srf pin. Older srf (<= 4a3e5f0)
// unwrapped a null optional here instead of returning an error, so
// on a downgrade this test PANICS rather than failing - which is
// itself the signal, since a panic is not interceptable by the
// `catch return null` that every caller relies on.
const io = std.testing.io;
const allocator = std.testing.allocator;
const unknown_variant =
\\#!srfv1
\\#!expires=9999999999
\\#!created=1786748010
\\last_close:num:298.97,last_date::2026-05-19,provider::not_a_real_provider
\\
;
try std.testing.expectError(error.InvalidData, Store.deserializeCandleMeta(allocator, unknown_variant));
// And through the real read path: null, not an error, not a partial
// parse with a defaulted provider.
var tmp = std.testing.tmpDir(.{});
defer tmp.cleanup();
const dir_path = try tmp.dir.realPathFileAlloc(io, ".", allocator);
defer allocator.free(dir_path);
var store = Store.init(io, allocator, dir_path);
try store.writeRaw("SMPL", .candles_meta, unknown_variant);
try std.testing.expect(store.readCandleMeta("SMPL") == null);
}
test "CandleMeta round-trips provider::external and never elides it" {
// `provider` has no default precisely so SRF cannot elide it, and
// `external` is the variant where that matters most: a meta file
// that silently omitted it would read back as whatever a future
// default happened to be, and an externally-managed series
// mislabelled as provider-sourced is exactly the lie this variant
// exists to prevent.
const allocator = std.testing.allocator;
const meta = Store.CandleMeta{
.last_close = 24.31,
.last_date = Date.fromYmd(2026, 8, 14),
.provider = .external,
};
const data = try Store.serializeCandleMeta(std.testing.io, allocator, meta, .{ .expires = 1234567890 });
defer allocator.free(data);
try std.testing.expect(std.mem.indexOf(u8, data, "provider::external") != null);
const parsed = try Store.deserializeCandleMeta(allocator, data);
try std.testing.expectEqual(Store.CandleProvider.external, parsed.provider);
try std.testing.expectApproxEqAbs(@as(f64, 24.31), parsed.last_close, 0.001);
try std.testing.expect(parsed.last_date.eql(Date.fromYmd(2026, 8, 14)));
}
test "readCandleMeta round-trips provider::external through the cache" {
// End to end over a real cache directory: `cacheCandles` writes the
// pair, `readCandleMeta` reads the label back. Non-null is the
// assertion that matters - a null here is the destructive cold-start
// path, and for an externally-managed series there is no provider to
// cold-start from.
const io = std.testing.io;
const allocator = std.testing.allocator;
var tmp = std.testing.tmpDir(.{});
defer tmp.cleanup();
const dir_path = try tmp.dir.realPathFileAlloc(io, ".", allocator);
defer allocator.free(dir_path);
var store = Store.init(io, allocator, dir_path);
const bars = [_]Candle{
.{ .date = Date.fromYmd(2026, 8, 13), .open = 24, .high = 24, .low = 24, .close = 24, .adj_close = 24, .volume = 0 },
.{ .date = Date.fromYmd(2026, 8, 14), .open = 25, .high = 25, .low = 25, .close = 25, .adj_close = 25, .volume = 0 },
};
store.cacheCandles("XTRN", bars[0..], .{ .provider = .external }, 9_999_999_999);
const mr = store.readCandleMeta("XTRN") orelse return error.NoCache;
try std.testing.expectEqual(Store.CandleProvider.external, mr.meta.provider);
try std.testing.expect(mr.meta.last_date.eql(Date.fromYmd(2026, 8, 14)));
} }
test "Store init creates valid store" { test "Store init creates valid store" {

View file

@ -1958,7 +1958,7 @@ test "fetchedSymbols: unions all four sources and dedups across them" {
.{ .symbol = "NON40OR52", .ticker = "SPY", .shares = 5, .open_date = Date.fromYmd(2026, 1, 2), .open_price = 90, .security_type = .stock }, .{ .symbol = "NON40OR52", .ticker = "SPY", .shares = 5, .open_date = Date.fromYmd(2026, 1, 2), .open_price = 90, .security_type = .stock },
.{ .symbol = "QTUM", .shares = 0, .open_date = Date.fromYmd(2026, 1, 2), .open_price = 0, .security_type = .watch }, .{ .symbol = "QTUM", .shares = 0, .open_date = Date.fromYmd(2026, 1, 2), .open_price = 0, .security_type = .watch },
// Excluded by stockSymbols: manual price, no ticker alias. // Excluded by stockSymbols: manual price, no ticker alias.
.{ .symbol = "ORCBI", .shares = 3, .open_date = Date.fromYmd(2026, 1, 2), .open_price = 10, .price = 11, .security_type = .stock }, .{ .symbol = "XTRNL", .shares = 3, .open_date = Date.fromYmd(2026, 1, 2), .open_price = 10, .price = 11, .security_type = .stock },
// Excluded: not a stock or watch lot. // Excluded: not a stock or watch lot.
.{ .symbol = "CASHX", .shares = 1, .open_date = Date.fromYmd(2026, 1, 2), .open_price = 1, .security_type = .cash }, .{ .symbol = "CASHX", .shares = 1, .open_date = Date.fromYmd(2026, 1, 2), .open_price = 1, .security_type = .cash },
}; };
@ -1982,7 +1982,7 @@ test "fetchedSymbols: unions all four sources and dedups across them" {
} }
// Manual-price-only and cash lots stay out. // Manual-price-only and cash lots stay out.
for (syms) |s| { for (syms) |s| {
try std.testing.expect(!std.mem.eql(u8, s, "ORCBI")); try std.testing.expect(!std.mem.eql(u8, s, "XTRNL"));
try std.testing.expect(!std.mem.eql(u8, s, "CASHX")); try std.testing.expect(!std.mem.eql(u8, s, "CASHX"));
} }
} }

View file

@ -875,6 +875,43 @@ pub const DataService = struct {
return next; return next;
} }
/// Whether a unanimous `NotFound` from `refetchFullHistory` earns a
/// sticky negative-cache entry, given whatever metadata the symbol
/// carried before the fetch.
///
/// Normally yes: every provider affirmatively disclaimed the
/// symbol, so stop asking. The exception is `.external`, where the
/// bars are managed outside zfin and no provider was ever going to
/// carry them. A negative entry there is not "stop asking a
/// provider that has nothing" - it is unrecoverable data loss.
/// `writeNegative` overwrites `candles_daily.srf` with a marker,
/// negative entries never expire, and `getCandles` short-circuits
/// on `isNegative` *before* it reaches `syncCandlesFromServer`, so
/// the marker makes the only surviving copy of the series
/// permanently unreachable. On a server, where the
/// externally-populated cache IS the master copy, nothing survives
/// at all.
///
/// A `null` prior meta (a true cold start, both cache files absent)
/// still writes the entry: there is no label to consult and nothing
/// local to destroy. That leaves a real hole - a cold start while
/// `ZFIN_SERVER` is unreachable poisons an external symbol until
/// `--refresh-data=force` or `cache clear`. Closing it needs the
/// `isNegative` short-circuit moved after the server tier, which
/// would re-hit the network for every legitimately candle-less
/// symbol (crypto, delisted tickers) on every run. That trade
/// belongs with the deferred external-symbol routing work, not
/// here.
///
/// Deliberately pure so it can be tested directly: reaching the
/// `true` branch through `getCandles` requires two live provider
/// 404s, so an inline conditional at the call site would be
/// permanently uncovered.
fn shouldNegativeCache(prior: ?cache.Store.CandleMeta) bool {
const meta = prior orelse return true;
return meta.provider != .external;
}
/// Fetch candles from providers with error classification. /// Fetch candles from providers with error classification.
/// ///
/// Error handling: /// Error handling:
@ -1336,7 +1373,18 @@ pub const DataService = struct {
// because the negative marker replaces the candle file: a // because the negative marker replaces the candle file: a
// bad negative both suppresses retries until `--refresh` // bad negative both suppresses retries until `--refresh`
// and throws away whatever history was cached. // and throws away whatever history was cached.
if (err == error.NotFound) s.writeNegative(symbol, .candles_daily); //
// And even a unanimous NotFound is not a verdict when the
// series was never a provider's to serve; see
// `shouldNegativeCache`.
if (err == error.NotFound) {
const prior: ?cache.Store.CandleMeta = if (meta_result) |mr| mr.meta else null;
if (shouldNegativeCache(prior)) {
s.writeNegative(symbol, .candles_daily);
} else {
log.warn("{s}: every provider disclaims it, but its cache is externally managed (provider::external) - refusing the negative entry, which would make the series unreachable via ZFIN_SERVER", .{symbol});
}
}
if (err == DataError.AuthError) return DataError.AuthError; if (err == DataError.AuthError) return DataError.AuthError;
return DataError.FetchFailed; return DataError.FetchFailed;
}; };
@ -4327,6 +4375,107 @@ test "getCandles offline never escalates a stale adjustment basis" {
try std.testing.expect(!store.isNegative("SMPL", .candles_daily)); try std.testing.expect(!store.isNegative("SMPL", .candles_daily));
} }
// externally-managed candle series (provider::external)
// The label is pure provenance and must stay behavior-free, with one
// exception: it vetoes the negative-cache write, because for a series
// no provider carries, that marker is unrecoverable rather than merely
// sticky. These tests pin both halves - the label changes nothing about
// how a cache is served, and it does change whether a unanimous 404
// gets remembered.
test "shouldNegativeCache refuses an external-provider symbol" {
// Looped over every variant so a fifth one has to come here and
// declare its intent rather than silently inheriting `true`.
for (std.enums.values(cache.Store.CandleProvider)) |provider| {
const meta = cache.Store.CandleMeta{
.last_close = 10.0,
.last_date = Date.fromYmd(2026, 8, 14),
.provider = provider,
};
const expected = provider != .external;
try std.testing.expectEqual(expected, DataService.shouldNegativeCache(meta));
}
// A true cold start has no label to consult and nothing local to
// destroy, so the entry is still written. This is the documented
// residual hole: an external symbol cold-started while ZFIN_SERVER
// is unreachable stays poisoned until `--refresh-data=force`.
try std.testing.expect(DataService.shouldNegativeCache(null));
}
test "getCandles serves an external-provider cache without touching the network" {
// `provider::external` must be inert on the serve path. In
// particular it must not fall into the `.twelvedata` carve-out,
// which treats a cache as unusable and forces a full re-fetch - for
// a symbol no provider carries, that re-fetch 404s and the
// cold-start path writes a negative marker over the only copy of
// the series. `panic_on_network_attempt` fires if any of that
// regresses.
const allocator = std.testing.allocator;
const io = std.testing.io;
var tmp = std.testing.tmpDir(.{});
defer tmp.cleanup();
const dir_path = try tmp.dir.realPathFileAlloc(io, ".", allocator);
defer allocator.free(dir_path);
const config = Config{ .cache_dir = dir_path };
var svc = DataService.init(io, allocator, config);
defer svc.deinit();
var store = svc.store();
var bars = [_]Candle{
.{ .date = Date.fromYmd(2026, 8, 13), .open = 24, .high = 24, .low = 24, .close = 24, .adj_close = 24, .volume = 0 },
.{ .date = Date.fromYmd(2026, 8, 14), .open = 25, .high = 25, .low = 25, .close = 25, .adj_close = 25, .volume = 0 },
};
store.cacheCandles("XTRN", bars[0..], .{ .provider = .external }, 9_999_999_999);
svc.panic_on_network_attempt = true;
const result = try svc.getCandles("XTRN", .{});
defer result.deinit();
try std.testing.expectEqual(Source.cached, result.source);
try std.testing.expectEqual(@as(usize, 2), result.data.len);
try std.testing.expect(result.data[1].date.eql(Date.fromYmd(2026, 8, 14)));
// Label survived the read, and no marker was written over the file.
const after = (store.readCandleMeta("XTRN") orelse return error.NoCache).meta;
try std.testing.expectEqual(cache.Store.CandleProvider.external, after.provider);
try std.testing.expect(!store.isNegative("XTRN", .candles_daily));
}
test "getCandles offline serves an external-provider cache" {
// The skip_network path has its own `.twelvedata` check, distinct
// from the one on the online path. `.external` must miss that one
// too, otherwise offline mode reports an externally-managed holding
// as unavailable despite a perfectly good local series.
const allocator = std.testing.allocator;
const io = std.testing.io;
var tmp = std.testing.tmpDir(.{});
defer tmp.cleanup();
const dir_path = try tmp.dir.realPathFileAlloc(io, ".", allocator);
defer allocator.free(dir_path);
const config = Config{ .cache_dir = dir_path };
var svc = DataService.init(io, allocator, config);
defer svc.deinit();
var store = svc.store();
var bars = [_]Candle{
.{ .date = Date.fromYmd(2026, 8, 14), .open = 25, .high = 25, .low = 25, .close = 25, .adj_close = 25, .volume = 0 },
};
// Expiry in the past: stale on purpose, since offline mode is meant
// to serve a stale externally-managed series rather than fail.
store.cacheCandles("XTRN", bars[0..], .{ .provider = .external }, 1);
svc.panic_on_network_attempt = true;
const result = try svc.getCandles("XTRN", .{ .skip_network = true });
defer result.deinit();
try std.testing.expectEqual(Source.cached, result.source);
try std.testing.expectEqual(@as(usize, 1), result.data.len);
try std.testing.expect(!store.isNegative("XTRN", .candles_daily));
}
// Tiingo coverage bookkeeping // // Tiingo coverage bookkeeping //
// Regression suite for the one-way drift to Yahoo. The old code // Regression suite for the one-way drift to Yahoo. The old code
// routed on `CandleMeta.provider == .yahoo`, so any non-transient // routed on `CandleMeta.provider == .yahoo`, so any non-transient