support external provider
This commit is contained in:
parent
18f4f00516
commit
df204a3eb2
5 changed files with 412 additions and 18 deletions
|
|
@ -220,6 +220,16 @@ and Tiingo was never consulted again. The backoff is armed only by a
|
|||
genuine 404, expires after `Ttl.tiingo_backoff` with per-symbol jitter,
|
||||
and clears the moment Tiingo serves the symbol again.
|
||||
|
||||
`CandleProvider` has four variants - `twelvedata`, `yahoo`, `tiingo`,
|
||||
`external` - and exactly one of them is special-cased anywhere:
|
||||
`twelvedata`, whose `adj_close` was unreliable, so a cache carrying it is
|
||||
treated as unusable. That check exists in **two** places, one per entry
|
||||
into the meta-exists branch (the `skip_network` path reports the symbol
|
||||
unavailable; the online path forces a full re-fetch). `external` is
|
||||
covered under [Externally-managed candle
|
||||
series](#externally-managed-candle-series); it is inert on every serve
|
||||
path and vetoes only the negative-cache write.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
S["getCandles(symbol, opts)"] --> NG{"negative candles_daily and not force_refresh?"}
|
||||
|
|
@ -227,7 +237,9 @@ flowchart TD
|
|||
NG -->|no| RM{"candles_meta exists?"}
|
||||
|
||||
RM -->|yes| SK{"skip_network?"}
|
||||
SK -->|yes| RETS["return cached even if stale<br/>(FetchFailed if unreadable)"]
|
||||
SK -->|yes| TWS{"provider is twelvedata?"}
|
||||
TWS -->|yes| FF
|
||||
TWS -->|no| RETS["return cached even if stale<br/>(FetchFailed if unreadable)"]
|
||||
SK -->|no| TW{"provider is twelvedata?"}
|
||||
TW -->|yes| FULL
|
||||
TW -->|no| FR{"meta fresh and not force_refresh?"}
|
||||
|
|
@ -252,7 +264,9 @@ flowchart TD
|
|||
SF2 -->|no| FULL["refetchFullHistory: Tiingo, then Yahoo"]
|
||||
FULL --> RES{"result?"}
|
||||
RES -->|ok| RET2
|
||||
RES -->|"NotFound (EVERY provider disclaims it)"| WN["writeNegative candles_daily"]
|
||||
RES -->|"NotFound (EVERY provider disclaims it)"| SNC{"prior meta says provider is external?"}
|
||||
SNC -->|yes| FF
|
||||
SNC -->|no| WN["writeNegative candles_daily"]
|
||||
WN --> FF
|
||||
RES -->|transient| TR["bump fail_count, TransientError"]
|
||||
RES -->|other| FF
|
||||
|
|
@ -354,6 +368,12 @@ re-run the dead lookup on every invocation. The entry is the sentinel:
|
|||
writes that file; `isCandleMetaFresh`, `getCachedCandles`, and the
|
||||
`getCandles` short-circuit all recognize it there. `candles_meta.srf`
|
||||
is intentionally not created for a no-data symbol.
|
||||
- **Externally-managed series are exempt.** A prior meta saying
|
||||
`provider::external` vetoes the write via
|
||||
`DataService.shouldNegativeCache`, because for a series no provider was
|
||||
ever going to carry, the marker is unrecoverable rather than merely
|
||||
sticky. See [Externally-managed candle
|
||||
series](#externally-managed-candle-series).
|
||||
|
||||
## Candle-less symbols (crypto and friends)
|
||||
|
||||
|
|
@ -391,6 +411,88 @@ If you want a candle-less symbol to be re-checked against the provider
|
|||
negative entry with `cache clear` or `--refresh-data=force`; the live
|
||||
quote and manual-price paths are unaffected by the negative cache.
|
||||
|
||||
## Externally-managed candle series
|
||||
|
||||
`CandleMeta.provider == .external` labels a series that was produced and
|
||||
is maintained **outside zfin**, by something that writes
|
||||
`candles_daily.srf` and `candles_meta.srf` directly into a cache
|
||||
directory. The motivating case is a unitized trust with no ticker and no
|
||||
CUSIP, whose daily unit values come from a plan recordkeeper's feed;
|
||||
every market-data provider 404s it.
|
||||
|
||||
Like the previous section, this is a symbol no provider carries. The
|
||||
difference is that a candle-less symbol has no history *anywhere*, so
|
||||
negative-caching it is correct, whereas an externally-managed symbol has
|
||||
a real series that simply did not come from a provider. The label exists
|
||||
so `candles_meta.srf` can say so honestly instead of naming a provider
|
||||
that never served it.
|
||||
|
||||
Two population routes:
|
||||
|
||||
- **Server**: a job writes the pair straight into the `ZFIN_SERVER`
|
||||
cache directory. That copy is the master.
|
||||
- **Client**: the existing server-sync path (phase 2 of `loadAllPrices`,
|
||||
and the two `syncCandlesFromServer` calls inside `getCandles`) pulls
|
||||
the bytes down verbatim, exactly as it does for any other symbol. No
|
||||
client-side special-casing is involved.
|
||||
|
||||
The label is **pure provenance and drives no routing**. zfin still walks
|
||||
the normal provider chain for such a symbol and still 404s on every pass;
|
||||
restricting it to the server tier and skipping the chain entirely is
|
||||
deferred work. There is exactly one behavior attached to the label, and
|
||||
it is a veto rather than a routing decision:
|
||||
|
||||
**A negative-cache entry on an external symbol is unrecoverable.**
|
||||
`writeNegative` overwrites `candles_daily.srf` with a marker, negative
|
||||
entries never expire, and `getCandles` short-circuits on `isNegative`
|
||||
*before* it reaches `syncCandlesFromServer` - so the marker makes the
|
||||
only surviving copy permanently unreachable. On a server it is worse:
|
||||
the externally-populated cache is the master copy, so nothing survives.
|
||||
`DataService.shouldNegativeCache` therefore refuses the write when the
|
||||
prior meta says `.external`.
|
||||
|
||||
That guard is deliberately partial. It consults the prior meta, so it
|
||||
covers the case that destroys data: meta present, `candles_daily.srf`
|
||||
missing or unreadable (a partially-completed server sync writes the two
|
||||
files in sequence and needs both to report success), which falls through
|
||||
to the cold-start path. It cannot cover a **true cold start** with both
|
||||
files absent - there is no label to read - so an external symbol first
|
||||
touched while `ZFIN_SERVER` is unreachable is poisoned until
|
||||
`--refresh-data=force` or `cache clear`. Closing that needs the
|
||||
`isNegative` short-circuit moved after the server tier, which would
|
||||
re-hit the network for every legitimately candle-less symbol on every
|
||||
run. It belongs with the deferred routing work.
|
||||
|
||||
### Adding a `CandleProvider` variant is a coordinated deployment
|
||||
|
||||
Not a backward-compatible change. `provider` has no default, so it is
|
||||
never elided, and SRF's enum coercion has no lenient mode and no
|
||||
`srfParse` hook (custom parsers are consulted for struct/union fields
|
||||
only). A build that does not know a variant cannot fall back:
|
||||
|
||||
- srf `>= ea2c358` (zfin's current pin) returns
|
||||
`StringValueNotValidEnumMember`. `readCandleMeta` swallows it into a
|
||||
`null`, which reads as a cache miss and routes the symbol down the
|
||||
cold-start path above - destructive for precisely the series that
|
||||
cannot be re-fetched.
|
||||
- srf `<= 4a3e5f0` unwraps a null optional and **panics**. That is what
|
||||
zfin-server `7103ced` vendors, and a panic is not interceptable by the
|
||||
`catch return null` every caller depends on. The blast radius is not
|
||||
one symbol: `handleDiagnostics` runs `freshness.collect` over every key
|
||||
in the cache directory, so a single unreadable file takes down
|
||||
`/<any-symbol>/diagnostics` and the cron refresh sweep. The
|
||||
byte-serving routes (`/candles`, `/candles_meta`) are unaffected, since
|
||||
they never parse.
|
||||
|
||||
Skew is dangerous in both directions: an old **client** syncing
|
||||
`candles_meta` bytes from a new server fails the same way, because
|
||||
`looksCompleteSrf` and the ETag check validate shape and integrity, not
|
||||
enum values.
|
||||
|
||||
So the ordering is: release zfin, then rebuild and deploy every server
|
||||
that reads the affected cache directory, and only then may a file
|
||||
carrying the new value exist.
|
||||
|
||||
## Server sync (the optional L2 tier)
|
||||
|
||||
`ZFIN_SERVER` points zfin at a
|
||||
|
|
@ -447,6 +549,8 @@ the upstream provider on every request.
|
|||
| Freshness check | `isFresh` (SRF), `isCandleMetaFresh` - `store.zig` |
|
||||
| TTLs and expiry computation | `Ttl`, `computeExpires` - `src/cache/store.zig` |
|
||||
| Negative cache | `writeNegative`, `isNegative` - `src/cache/store.zig` |
|
||||
| Negative-cache veto | `shouldNegativeCache` - `src/service.zig` |
|
||||
| Candle provenance labels | `CandleProvider` - `src/cache/store.zig` |
|
||||
| NotFound classification | `isPermanentProviderFailure` - `src/service.zig` |
|
||||
| Market-aware candle expiry | `nextCandleExpiry`, `shouldRefresh` - `src/market.zig` |
|
||||
| Price fallback (manual/avg-cost)| `buildFallbackPrices` - `src/analytics/valuation.zig` |
|
||||
|
|
|
|||
|
|
@ -138,6 +138,13 @@ ticker, say -- zfin records a negative cache entry so it doesn't retry
|
|||
the same dead lookup on every run. (Transient failures like rate limits
|
||||
are not cached this way; they're retried.)
|
||||
|
||||
One exception: a price history that was loaded into the cache from
|
||||
outside zfin, rather than fetched from a provider, is never
|
||||
negative-cached. Such a series has no provider to re-fetch it from, so
|
||||
recording "nobody has this" would discard the only copy. `zfin diagnose
|
||||
SYMBOL` names the source of a symbol's bars, and reports `external` for
|
||||
these.
|
||||
|
||||
## Symbols without candle data (crypto)
|
||||
|
||||
A few holdings have no daily price history available from zfin's candle
|
||||
|
|
|
|||
160
src/cache/store.zig
vendored
160
src/cache/store.zig
vendored
|
|
@ -1415,6 +1415,18 @@ pub const Store = struct {
|
|||
/// which writes a new meta file with the provider explicit.
|
||||
/// The wipe happens naturally on first use post-upgrade.
|
||||
///
|
||||
/// A field that is *present but carries a value this build
|
||||
/// does not know* - i.e. a cache written by a NEWER zfin that
|
||||
/// has added a variant - is a different and nastier failure.
|
||||
/// SRF's enum coercion has no lenient mode and no `srfParse`
|
||||
/// hook (it consults custom parsers only for struct/union
|
||||
/// fields), so the value either raises
|
||||
/// `StringValueNotValidEnumMember` (srf >= ea2c358, swallowed
|
||||
/// into the same destructive cache-miss path above) or panics
|
||||
/// on an unchecked optional unwrap (srf <= 4a3e5f0). Adding a
|
||||
/// variant is therefore a coordinated deployment, not a
|
||||
/// backward-compatible change; see `CandleProvider.external`.
|
||||
///
|
||||
/// This is also why no *other* field on this struct may be
|
||||
/// default-less: that cold-start path is the destructive one.
|
||||
/// It writes a negative-cache marker over `candles_daily.srf`
|
||||
|
|
@ -1527,12 +1539,52 @@ pub const Store = struct {
|
|||
/// dividends and splits from the same response, and its
|
||||
/// `adj_close` is what the analytics layer is written against.
|
||||
tiingo,
|
||||
|
||||
pub fn fromString(s: []const u8) CandleProvider {
|
||||
if (std.mem.eql(u8, s, "yahoo")) return .yahoo;
|
||||
if (std.mem.eql(u8, s, "tiingo")) return .tiingo;
|
||||
return .twelvedata;
|
||||
}
|
||||
/// Candles were produced and are managed OUTSIDE zfin, by
|
||||
/// something that writes `candles_daily.srf` and
|
||||
/// `candles_meta.srf` directly into a cache directory.
|
||||
///
|
||||
/// Pure provenance like every other variant - it records where
|
||||
/// the bars came from and drives no routing. What sets it
|
||||
/// apart is what it implies about their origin: **no
|
||||
/// market-data provider carries the symbol at all**. The
|
||||
/// motivating case is a unitized trust with no ticker and no
|
||||
/// CUSIP, whose daily unit values come from a plan
|
||||
/// recordkeeper's feed; Tiingo, Yahoo and TwelveData all 404
|
||||
/// it. So zfin can never fetch or restate such a series
|
||||
/// itself. It arrives either by `ZFIN_SERVER` sync (the client
|
||||
/// case) or by direct population of the cache directory (the
|
||||
/// server case).
|
||||
///
|
||||
/// **A negative-cache entry on such a symbol is unrecoverable**,
|
||||
/// not merely inconvenient. `writeNegative` overwrites
|
||||
/// `candles_daily.srf` with a marker, negative entries never
|
||||
/// expire (`isCandleMetaFresh` reports them fresh forever), and
|
||||
/// `getCandles` short-circuits on `isNegative` *before* it
|
||||
/// reaches `syncCandlesFromServer` - so the one copy that still
|
||||
/// exists becomes permanently unreachable. On a server, where
|
||||
/// the externally-populated cache IS the master copy, there is
|
||||
/// no other copy at all. `shouldNegativeCache` in `service.zig`
|
||||
/// refuses that write for exactly this reason; see its doc
|
||||
/// comment for the case it cannot cover.
|
||||
///
|
||||
/// **Version compatibility is a deployment ordering
|
||||
/// constraint, not a nicety.** This value is unknown to any
|
||||
/// zfin built before it existed, and an older reader does not
|
||||
/// degrade gracefully:
|
||||
///
|
||||
/// - srf >= ea2c358 (what zfin pins today) returns
|
||||
/// `error.StringValueNotValidEnumMember`, which
|
||||
/// `readCandleMeta` swallows into a `null` - i.e. a cache
|
||||
/// miss, which routes the symbol down the destructive
|
||||
/// cold-start path described on the `provider` field.
|
||||
/// - srf <= 4a3e5f0 unwraps a null optional and **panics**.
|
||||
/// That is what zfin-server 7103ced vendors, and a panic
|
||||
/// is not interceptable by `readCandleMeta`'s `catch`.
|
||||
///
|
||||
/// So no file carrying `provider::external` may exist until
|
||||
/// every reader of that cache directory - client and server -
|
||||
/// runs a zfin that knows the variant.
|
||||
external,
|
||||
};
|
||||
|
||||
// ── Private I/O ──────────────────────────────────────────────
|
||||
|
|
@ -3710,13 +3762,95 @@ test "Store.Freshness enum values" {
|
|||
try std.testing.expect(Store.Freshness.fresh_only != Store.Freshness.any);
|
||||
}
|
||||
|
||||
test "CandleProvider.fromString parses provider names" {
|
||||
try std.testing.expectEqual(Store.CandleProvider.yahoo, Store.CandleProvider.fromString("yahoo"));
|
||||
try std.testing.expectEqual(Store.CandleProvider.tiingo, Store.CandleProvider.fromString("tiingo"));
|
||||
try std.testing.expectEqual(Store.CandleProvider.twelvedata, Store.CandleProvider.fromString("twelvedata"));
|
||||
// Unknown defaults to twelvedata
|
||||
try std.testing.expectEqual(Store.CandleProvider.twelvedata, Store.CandleProvider.fromString("unknown"));
|
||||
try std.testing.expectEqual(Store.CandleProvider.twelvedata, Store.CandleProvider.fromString(""));
|
||||
test "candles_meta with an unknown provider value reads as a cache miss" {
|
||||
// The forward-compatibility contract for `CandleProvider`, pinned.
|
||||
//
|
||||
// SRF's enum coercion has no lenient mode and no `srfParse` hook
|
||||
// (custom parsers are consulted for struct/union fields only), so a
|
||||
// `provider::` value this build does not know cannot be mapped to a
|
||||
// fallback. It raises `StringValueNotValidEnumMember`, which
|
||||
// `deserializeCandleMeta` reports as `InvalidData` and
|
||||
// `readCandleMeta` swallows into a null - i.e. the symbol looks like
|
||||
// a cache miss and takes the destructive cold-start path. That is
|
||||
// why adding a variant is a coordinated deployment; see
|
||||
// `CandleProvider.external`.
|
||||
//
|
||||
// This also acts as a canary on the srf pin. Older srf (<= 4a3e5f0)
|
||||
// unwrapped a null optional here instead of returning an error, so
|
||||
// on a downgrade this test PANICS rather than failing - which is
|
||||
// itself the signal, since a panic is not interceptable by the
|
||||
// `catch return null` that every caller relies on.
|
||||
const io = std.testing.io;
|
||||
const allocator = std.testing.allocator;
|
||||
|
||||
const unknown_variant =
|
||||
\\#!srfv1
|
||||
\\#!expires=9999999999
|
||||
\\#!created=1786748010
|
||||
\\last_close:num:298.97,last_date::2026-05-19,provider::not_a_real_provider
|
||||
\\
|
||||
;
|
||||
try std.testing.expectError(error.InvalidData, Store.deserializeCandleMeta(allocator, unknown_variant));
|
||||
|
||||
// And through the real read path: null, not an error, not a partial
|
||||
// parse with a defaulted provider.
|
||||
var tmp = std.testing.tmpDir(.{});
|
||||
defer tmp.cleanup();
|
||||
const dir_path = try tmp.dir.realPathFileAlloc(io, ".", allocator);
|
||||
defer allocator.free(dir_path);
|
||||
|
||||
var store = Store.init(io, allocator, dir_path);
|
||||
try store.writeRaw("SMPL", .candles_meta, unknown_variant);
|
||||
try std.testing.expect(store.readCandleMeta("SMPL") == null);
|
||||
}
|
||||
|
||||
test "CandleMeta round-trips provider::external and never elides it" {
|
||||
// `provider` has no default precisely so SRF cannot elide it, and
|
||||
// `external` is the variant where that matters most: a meta file
|
||||
// that silently omitted it would read back as whatever a future
|
||||
// default happened to be, and an externally-managed series
|
||||
// mislabelled as provider-sourced is exactly the lie this variant
|
||||
// exists to prevent.
|
||||
const allocator = std.testing.allocator;
|
||||
const meta = Store.CandleMeta{
|
||||
.last_close = 24.31,
|
||||
.last_date = Date.fromYmd(2026, 8, 14),
|
||||
.provider = .external,
|
||||
};
|
||||
const data = try Store.serializeCandleMeta(std.testing.io, allocator, meta, .{ .expires = 1234567890 });
|
||||
defer allocator.free(data);
|
||||
|
||||
try std.testing.expect(std.mem.indexOf(u8, data, "provider::external") != null);
|
||||
|
||||
const parsed = try Store.deserializeCandleMeta(allocator, data);
|
||||
try std.testing.expectEqual(Store.CandleProvider.external, parsed.provider);
|
||||
try std.testing.expectApproxEqAbs(@as(f64, 24.31), parsed.last_close, 0.001);
|
||||
try std.testing.expect(parsed.last_date.eql(Date.fromYmd(2026, 8, 14)));
|
||||
}
|
||||
|
||||
test "readCandleMeta round-trips provider::external through the cache" {
|
||||
// End to end over a real cache directory: `cacheCandles` writes the
|
||||
// pair, `readCandleMeta` reads the label back. Non-null is the
|
||||
// assertion that matters - a null here is the destructive cold-start
|
||||
// path, and for an externally-managed series there is no provider to
|
||||
// cold-start from.
|
||||
const io = std.testing.io;
|
||||
const allocator = std.testing.allocator;
|
||||
var tmp = std.testing.tmpDir(.{});
|
||||
defer tmp.cleanup();
|
||||
const dir_path = try tmp.dir.realPathFileAlloc(io, ".", allocator);
|
||||
defer allocator.free(dir_path);
|
||||
|
||||
var store = Store.init(io, allocator, dir_path);
|
||||
const bars = [_]Candle{
|
||||
.{ .date = Date.fromYmd(2026, 8, 13), .open = 24, .high = 24, .low = 24, .close = 24, .adj_close = 24, .volume = 0 },
|
||||
.{ .date = Date.fromYmd(2026, 8, 14), .open = 25, .high = 25, .low = 25, .close = 25, .adj_close = 25, .volume = 0 },
|
||||
};
|
||||
store.cacheCandles("XTRN", bars[0..], .{ .provider = .external }, 9_999_999_999);
|
||||
|
||||
const mr = store.readCandleMeta("XTRN") orelse return error.NoCache;
|
||||
try std.testing.expectEqual(Store.CandleProvider.external, mr.meta.provider);
|
||||
try std.testing.expect(mr.meta.last_date.eql(Date.fromYmd(2026, 8, 14)));
|
||||
}
|
||||
|
||||
test "Store init creates valid store" {
|
||||
|
|
|
|||
|
|
@ -1958,7 +1958,7 @@ test "fetchedSymbols: unions all four sources and dedups across them" {
|
|||
.{ .symbol = "NON40OR52", .ticker = "SPY", .shares = 5, .open_date = Date.fromYmd(2026, 1, 2), .open_price = 90, .security_type = .stock },
|
||||
.{ .symbol = "QTUM", .shares = 0, .open_date = Date.fromYmd(2026, 1, 2), .open_price = 0, .security_type = .watch },
|
||||
// Excluded by stockSymbols: manual price, no ticker alias.
|
||||
.{ .symbol = "ORCBI", .shares = 3, .open_date = Date.fromYmd(2026, 1, 2), .open_price = 10, .price = 11, .security_type = .stock },
|
||||
.{ .symbol = "XTRNL", .shares = 3, .open_date = Date.fromYmd(2026, 1, 2), .open_price = 10, .price = 11, .security_type = .stock },
|
||||
// Excluded: not a stock or watch lot.
|
||||
.{ .symbol = "CASHX", .shares = 1, .open_date = Date.fromYmd(2026, 1, 2), .open_price = 1, .security_type = .cash },
|
||||
};
|
||||
|
|
@ -1982,7 +1982,7 @@ test "fetchedSymbols: unions all four sources and dedups across them" {
|
|||
}
|
||||
// Manual-price-only and cash lots stay out.
|
||||
for (syms) |s| {
|
||||
try std.testing.expect(!std.mem.eql(u8, s, "ORCBI"));
|
||||
try std.testing.expect(!std.mem.eql(u8, s, "XTRNL"));
|
||||
try std.testing.expect(!std.mem.eql(u8, s, "CASHX"));
|
||||
}
|
||||
}
|
||||
|
|
|
|||
151
src/service.zig
151
src/service.zig
|
|
@ -875,6 +875,43 @@ pub const DataService = struct {
|
|||
return next;
|
||||
}
|
||||
|
||||
/// Whether a unanimous `NotFound` from `refetchFullHistory` earns a
|
||||
/// sticky negative-cache entry, given whatever metadata the symbol
|
||||
/// carried before the fetch.
|
||||
///
|
||||
/// Normally yes: every provider affirmatively disclaimed the
|
||||
/// symbol, so stop asking. The exception is `.external`, where the
|
||||
/// bars are managed outside zfin and no provider was ever going to
|
||||
/// carry them. A negative entry there is not "stop asking a
|
||||
/// provider that has nothing" - it is unrecoverable data loss.
|
||||
/// `writeNegative` overwrites `candles_daily.srf` with a marker,
|
||||
/// negative entries never expire, and `getCandles` short-circuits
|
||||
/// on `isNegative` *before* it reaches `syncCandlesFromServer`, so
|
||||
/// the marker makes the only surviving copy of the series
|
||||
/// permanently unreachable. On a server, where the
|
||||
/// externally-populated cache IS the master copy, nothing survives
|
||||
/// at all.
|
||||
///
|
||||
/// A `null` prior meta (a true cold start, both cache files absent)
|
||||
/// still writes the entry: there is no label to consult and nothing
|
||||
/// local to destroy. That leaves a real hole - a cold start while
|
||||
/// `ZFIN_SERVER` is unreachable poisons an external symbol until
|
||||
/// `--refresh-data=force` or `cache clear`. Closing it needs the
|
||||
/// `isNegative` short-circuit moved after the server tier, which
|
||||
/// would re-hit the network for every legitimately candle-less
|
||||
/// symbol (crypto, delisted tickers) on every run. That trade
|
||||
/// belongs with the deferred external-symbol routing work, not
|
||||
/// here.
|
||||
///
|
||||
/// Deliberately pure so it can be tested directly: reaching the
|
||||
/// `true` branch through `getCandles` requires two live provider
|
||||
/// 404s, so an inline conditional at the call site would be
|
||||
/// permanently uncovered.
|
||||
fn shouldNegativeCache(prior: ?cache.Store.CandleMeta) bool {
|
||||
const meta = prior orelse return true;
|
||||
return meta.provider != .external;
|
||||
}
|
||||
|
||||
/// Fetch candles from providers with error classification.
|
||||
///
|
||||
/// Error handling:
|
||||
|
|
@ -1336,7 +1373,18 @@ pub const DataService = struct {
|
|||
// because the negative marker replaces the candle file: a
|
||||
// bad negative both suppresses retries until `--refresh`
|
||||
// and throws away whatever history was cached.
|
||||
if (err == error.NotFound) s.writeNegative(symbol, .candles_daily);
|
||||
//
|
||||
// And even a unanimous NotFound is not a verdict when the
|
||||
// series was never a provider's to serve; see
|
||||
// `shouldNegativeCache`.
|
||||
if (err == error.NotFound) {
|
||||
const prior: ?cache.Store.CandleMeta = if (meta_result) |mr| mr.meta else null;
|
||||
if (shouldNegativeCache(prior)) {
|
||||
s.writeNegative(symbol, .candles_daily);
|
||||
} else {
|
||||
log.warn("{s}: every provider disclaims it, but its cache is externally managed (provider::external) - refusing the negative entry, which would make the series unreachable via ZFIN_SERVER", .{symbol});
|
||||
}
|
||||
}
|
||||
if (err == DataError.AuthError) return DataError.AuthError;
|
||||
return DataError.FetchFailed;
|
||||
};
|
||||
|
|
@ -4327,6 +4375,107 @@ test "getCandles offline never escalates a stale adjustment basis" {
|
|||
try std.testing.expect(!store.isNegative("SMPL", .candles_daily));
|
||||
}
|
||||
|
||||
// ── externally-managed candle series (provider::external) ────
|
||||
// The label is pure provenance and must stay behavior-free, with one
|
||||
// exception: it vetoes the negative-cache write, because for a series
|
||||
// no provider carries, that marker is unrecoverable rather than merely
|
||||
// sticky. These tests pin both halves - the label changes nothing about
|
||||
// how a cache is served, and it does change whether a unanimous 404
|
||||
// gets remembered.
|
||||
|
||||
test "shouldNegativeCache refuses an external-provider symbol" {
|
||||
// Looped over every variant so a fifth one has to come here and
|
||||
// declare its intent rather than silently inheriting `true`.
|
||||
for (std.enums.values(cache.Store.CandleProvider)) |provider| {
|
||||
const meta = cache.Store.CandleMeta{
|
||||
.last_close = 10.0,
|
||||
.last_date = Date.fromYmd(2026, 8, 14),
|
||||
.provider = provider,
|
||||
};
|
||||
const expected = provider != .external;
|
||||
try std.testing.expectEqual(expected, DataService.shouldNegativeCache(meta));
|
||||
}
|
||||
|
||||
// A true cold start has no label to consult and nothing local to
|
||||
// destroy, so the entry is still written. This is the documented
|
||||
// residual hole: an external symbol cold-started while ZFIN_SERVER
|
||||
// is unreachable stays poisoned until `--refresh-data=force`.
|
||||
try std.testing.expect(DataService.shouldNegativeCache(null));
|
||||
}
|
||||
|
||||
test "getCandles serves an external-provider cache without touching the network" {
|
||||
// `provider::external` must be inert on the serve path. In
|
||||
// particular it must not fall into the `.twelvedata` carve-out,
|
||||
// which treats a cache as unusable and forces a full re-fetch - for
|
||||
// a symbol no provider carries, that re-fetch 404s and the
|
||||
// cold-start path writes a negative marker over the only copy of
|
||||
// the series. `panic_on_network_attempt` fires if any of that
|
||||
// regresses.
|
||||
const allocator = std.testing.allocator;
|
||||
const io = std.testing.io;
|
||||
var tmp = std.testing.tmpDir(.{});
|
||||
defer tmp.cleanup();
|
||||
const dir_path = try tmp.dir.realPathFileAlloc(io, ".", allocator);
|
||||
defer allocator.free(dir_path);
|
||||
|
||||
const config = Config{ .cache_dir = dir_path };
|
||||
var svc = DataService.init(io, allocator, config);
|
||||
defer svc.deinit();
|
||||
|
||||
var store = svc.store();
|
||||
var bars = [_]Candle{
|
||||
.{ .date = Date.fromYmd(2026, 8, 13), .open = 24, .high = 24, .low = 24, .close = 24, .adj_close = 24, .volume = 0 },
|
||||
.{ .date = Date.fromYmd(2026, 8, 14), .open = 25, .high = 25, .low = 25, .close = 25, .adj_close = 25, .volume = 0 },
|
||||
};
|
||||
store.cacheCandles("XTRN", bars[0..], .{ .provider = .external }, 9_999_999_999);
|
||||
|
||||
svc.panic_on_network_attempt = true;
|
||||
const result = try svc.getCandles("XTRN", .{});
|
||||
defer result.deinit();
|
||||
|
||||
try std.testing.expectEqual(Source.cached, result.source);
|
||||
try std.testing.expectEqual(@as(usize, 2), result.data.len);
|
||||
try std.testing.expect(result.data[1].date.eql(Date.fromYmd(2026, 8, 14)));
|
||||
|
||||
// Label survived the read, and no marker was written over the file.
|
||||
const after = (store.readCandleMeta("XTRN") orelse return error.NoCache).meta;
|
||||
try std.testing.expectEqual(cache.Store.CandleProvider.external, after.provider);
|
||||
try std.testing.expect(!store.isNegative("XTRN", .candles_daily));
|
||||
}
|
||||
|
||||
test "getCandles offline serves an external-provider cache" {
|
||||
// The skip_network path has its own `.twelvedata` check, distinct
|
||||
// from the one on the online path. `.external` must miss that one
|
||||
// too, otherwise offline mode reports an externally-managed holding
|
||||
// as unavailable despite a perfectly good local series.
|
||||
const allocator = std.testing.allocator;
|
||||
const io = std.testing.io;
|
||||
var tmp = std.testing.tmpDir(.{});
|
||||
defer tmp.cleanup();
|
||||
const dir_path = try tmp.dir.realPathFileAlloc(io, ".", allocator);
|
||||
defer allocator.free(dir_path);
|
||||
|
||||
const config = Config{ .cache_dir = dir_path };
|
||||
var svc = DataService.init(io, allocator, config);
|
||||
defer svc.deinit();
|
||||
|
||||
var store = svc.store();
|
||||
var bars = [_]Candle{
|
||||
.{ .date = Date.fromYmd(2026, 8, 14), .open = 25, .high = 25, .low = 25, .close = 25, .adj_close = 25, .volume = 0 },
|
||||
};
|
||||
// Expiry in the past: stale on purpose, since offline mode is meant
|
||||
// to serve a stale externally-managed series rather than fail.
|
||||
store.cacheCandles("XTRN", bars[0..], .{ .provider = .external }, 1);
|
||||
|
||||
svc.panic_on_network_attempt = true;
|
||||
const result = try svc.getCandles("XTRN", .{ .skip_network = true });
|
||||
defer result.deinit();
|
||||
|
||||
try std.testing.expectEqual(Source.cached, result.source);
|
||||
try std.testing.expectEqual(@as(usize, 1), result.data.len);
|
||||
try std.testing.expect(!store.isNegative("XTRN", .candles_daily));
|
||||
}
|
||||
|
||||
// ── Tiingo coverage bookkeeping ──────────────────────────────//
|
||||
// Regression suite for the one-way drift to Yahoo. The old code
|
||||
// routed on `CandleMeta.provider == .yahoo`, so any non-transient
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue