Compare commits

...

4 commits

Author SHA1 Message Date
c15ea92b0d
fix forward-announced ex-dates masking real restatement work
All checks were successful
Generic zig build / build (push) Successful in 5m7s
Generic zig build / publish-macos (push) Successful in 10s
Generic zig build / deploy (push) Successful in 16s
2026-08-19 13:53:50 -07:00
12eace86b2
clean separation of user vs machine-edited srf files 2026-08-19 13:52:48 -07:00
5706eed3c1
fix regression in contribution 2026-08-18 01:38:57 -07:00
c986df83c1
contributions: support all portfolio files and in account sale/rebuy in same window 2026-08-18 01:29:24 -07:00
24 changed files with 1873 additions and 307 deletions

View file

@ -73,6 +73,15 @@ naive diff sees the receiving account gain lots and counts it as new
money. `transaction_log.srf` is how you tell zfin "this was a transfer,
not new money."
This is only needed for movement **between** accounts. Reallocating
*within* one account -- selling a holding to buy another -- is detected
on its own, with the sale's proceeds offsetting the repurchase, so no
record is required. See
[Internal movement](../reference/cli/contributions.md#internal-movement).
Recording `close_price` on the sold lot is worth the keystrokes: it is
what the sale realized, and zfin values the offset from it rather than
guessing from the current price.
Like the other sibling files, it's **optional and additive**: you only
need it if you move money between your own accounts and want clean
attribution. Without it nothing breaks -- those transfers just show up

View file

@ -7,9 +7,11 @@ points in your portfolio's **git history**.
Usage: zfin contributions [opts]
```
`contributions` diffs two git revisions of your `portfolio.srf` and
attributes the share/lot changes to new money vs. market movement. Your
portfolio must be under git with commits over time.
`contributions` diffs two git revisions of your `portfolio*.srf` files
and attributes the share/lot changes to new money vs. market movement.
Every file matching the glob is read at both revisions and merged, so a
sold lot archived into a sibling `portfolio_closed.srf` is still seen.
Your portfolio must be under git with commits over time.
## Modes
@ -41,9 +43,38 @@ most one of `--until`/`--commit-after`.
zfin contributions --since 1Y
```
Internal transfers between your own accounts are excluded from the
attribution total when declared in
[`transaction_log.srf`](../config/transaction-log-srf.md).
## Internal movement
Money that was already inside an account is not a contribution -- it
just changed form. Two shapes are detected automatically, with no
bookkeeping on your part, and both report under **Internal purchases**
rather than counting toward the total:
- **Buying with cash already in the account.** The buy appears
alongside the account's cash going down.
- **Reallocating -- selling one holding to buy another in the same
account.** The sale's proceeds offset the repurchase.
A sale is valued at `close_price` when you record one (see
[`portfolio.srf`](../config/portfolio-srf.md)), which is what the sale
actually realized. If you delete the lot outright instead, there is no
`close_price` to read and the current market price stands in -- accurate
for a recent sale, less so for one made long before the end of the
window. The report labels which was used: `at close` or `at mark`.
Closing a position that has accumulated a lot per dividend
reinvestment retires many lots at once, so sales collapse to one line
per account and symbol, carrying the lot count and the total.
Proceeds still sitting in cash at the end of the window cannot have
funded anything, and are treated accordingly. On an account marked
`cash_is_contribution::true` they also cancel that account's cash
credit, since the sale is not new money even though cash arrived.
Movement **between** accounts is a different matter -- zfin cannot tell
it from a contribution, so declare it in
[`transaction_log.srf`](../config/transaction-log-srf.md). An explicit
record always wins over the automatic netting above.
## See also

View file

@ -37,8 +37,8 @@ symbol::VTI,shares:num:100,open_date::2024-01-15,open_price:num:220.50,account::
| `shares` | number | Yes | Share count (or face value for cash/CDs). Negative for short option positions. |
| `open_date` | string | Yes\*\* | Purchase date `YYYY-MM-DD`. \*\*Not required for `cash`/`watch`. |
| `open_price` | number | Yes\*\* | Purchase price per share. \*\*Not required for `cash`/`watch`. |
| `close_date` | string | No | Sale date. Omit for an open lot. |
| `close_price` | number | No | Sale price per share. |
| `close_date` | string | No | Sale date. Omit for an open lot. See [Closed lots](#closed-lots). |
| `close_price` | number | No | Sale price per share. See [Closed lots](#closed-lots). |
| `security_type` | string | No | `stock` (default), `option`, `cd`, `cash`, `illiquid`, `watch`. |
| `account` | string | No | Account name. Should match an `account::` entry in [`accounts.srf`](accounts-srf.md). |
| `note` | string | No | Free-text note (shown in cash/CD/illiquid tables). |
@ -96,6 +96,26 @@ security_type::cd,symbol::912797KR0,shares:num:10000,open_date::2024-06-01,open_
security_type::illiquid,symbol::HOME,shares:num:450000,open_date::2020-06-01,open_price:num:350000,note::Primary residence
```
## Closed lots
A lot with `close_date` set is sold. Both fields are optional -- an open
lot has neither -- but a sold lot should carry both, because they drive
two things beyond bookkeeping:
- **Realized gain/loss** comes from `close_price` against `open_price`.
- **[`zfin contributions`](../cli/contributions.md#internal-movement)**
treats the close as a sale and values it at `close_price`. That
matters when you sell one holding to buy another: the proceeds offset
the repurchase instead of it reading as new money. Without a
`close_price` the current market price stands in, which is close
enough for a recent sale and wrong for an old one.
You can either edit the lot where it sits or move it to a sibling file
such as `portfolio_closed.srf` -- the `portfolio*.srf` glob picks it up
either way, so realized gains and back-dated (`--as-of`) views stay
correct. See
[splitting your portfolio](../../guides/set-up-your-portfolio.md).
## Price resolution
For stock lots, the displayed price is resolved in this order:

View file

@ -94,7 +94,11 @@ destination is either fully a transfer or not one at all.
## Scope and limits
- Only `transfer::` records. Buys, sells, and dividends stay inferred
from the portfolio diff.
from the portfolio diff. You do **not** need a record for selling one
holding to buy another inside a single account -- that is detected
automatically, and the sale's proceeds offset the repurchase. See
[`zfin contributions`](../cli/contributions.md#internal-movement) for
how sales are valued and when a record is still required.
- Forward-looking only -- there is no historical reconstruction.
- Account names are matched byte-exactly, so a
[renamed account](../../guides/set-up-accounts.md#renaming-an-account)

View file

@ -5,6 +5,7 @@
const std = @import("std");
const builtin = @import("builtin");
const srf = @import("srf");
const srf_opts = @import("../srf_opts.zig");
const Allocation = @import("valuation.zig").Allocation;
const ClassificationMap = @import("../models/classification.zig").ClassificationMap;
const ClassificationEntry = @import("../models/classification.zig").ClassificationEntry;
@ -546,7 +547,7 @@ pub fn parseAccountsFile(allocator: std.mem.Allocator, data: []const u8) !Accoun
defer it.deinit();
while (try it.next()) |fields| {
const entry = fields.to(AccountTaxEntry, .{}) catch continue;
const entry = fields.to(AccountTaxEntry, srf_opts.user_edited) catch continue;
// A zero/negative large-lot threshold is nonsensical (zero
// flags every new lot; negative is meaningless). Reject it and
@ -1271,6 +1272,28 @@ test "parseAccountsFile: cash_is_contribution default false, opt-in true" {
try std.testing.expect(!am.cashIsContribution("Nonexistent"));
}
test "parseAccountsFile: a hand-typed string separator on a numeric field still parses" {
// `accounts.srf` is hand-edited, so `harvested::5000` instead of
// `harvested:num:5000` is a slip rather than different intent. Under
// SRF's strict default that string reaches an unchecked
// `val.?.number` - a panic in Debug and undefined behaviour in
// ReleaseFast - which is why this parser opts into
// `srf_opts.user_edited`. Pinned because the option is easy to drop
// and the failure is silent in the build zfin actually ships.
const data =
\\#!srfv1
\\account::Sample Brokerage,tax_type::taxable,harvested::5000,harvested_date::2026-06-01
\\account::Sample IRA,tax_type::traditional,audit_large_lot_threshold::25000
;
const allocator = std.testing.allocator;
var am = try parseAccountsFile(allocator, data);
defer am.deinit();
try std.testing.expectEqual(@as(usize, 2), am.entries.len);
try std.testing.expectApproxEqAbs(@as(f64, 5000), am.entries[0].harvested.?, 0.001);
try std.testing.expectApproxEqAbs(@as(f64, 25000), am.entries[1].audit_large_lot_threshold.?, 0.001);
}
test "parseAccountsFile: direct_indexing default false, opt-in true" {
const data =
\\#!srfv1

View file

@ -14,6 +14,7 @@ const builtin = @import("builtin");
const log = std.log.scoped(.projections);
const shiller = @import("../data/shiller.zig");
const srf = @import("srf");
const srf_opts = @import("../srf_opts.zig");
const Date = @import("../Date.zig");
/// `log.warn` wrapper that no-ops under `zig build test`. Used for
@ -762,7 +763,7 @@ pub fn parseProjectionsConfig(data: ?[]const u8) UserConfig {
var annotation_count: u8 = 0;
while (it.next() catch null) |field_it| {
const rec = field_it.to(SrfProjection, .{}) catch continue;
const rec = field_it.to(SrfProjection, srf_opts.user_edited) catch continue;
switch (rec) {
.config => |c| {
config.target_stock_pct = c.target_stock_pct orelse config.target_stock_pct;

View file

@ -47,6 +47,12 @@
//! means the series must be refetched is domain policy, and
//! `store.zig`'s own `updateCandleMeta` doc records the same boundary
//! for market-clock knowledge ("owned by the caller").
//!
//! Note which half owns the "has it actually gone ex yet?" filter:
//! `newestCorporateAction` does, because that is a question about which
//! actions are *relevant*, and relevance is selection. Putting it on the
//! verdict instead is what let one forward-announced dividend mask every
//! older unapplied one.
const std = @import("std");
const Date = @import("../Date.zig");
@ -638,16 +644,38 @@ test "collect-style exclusion: an excluded symbol produces no finding at all" {
// two functions turn that into a verdict, split the same way as
// `collect` / `scan`: one gathers from disk, one decides.
/// Newest corporate-action ex-date cached for a symbol, or null when no
/// dividends or splits are on disk.
/// Newest corporate-action ex-date cached for a symbol that has already
/// gone ex as of `through`, or null when there is no such action.
///
/// Splits count as much as dividends here, and arguably more: raw
/// `close` is not split-adjusted (hence `split.cumulativeSplitRatio`),
/// so an unapplied 2:1 split leaves older `adj_close` values wrong by
/// 50% rather than by a quarter's yield.
///
/// `through` is normally `CandleMeta.last_date`, the newest bar held.
/// Actions dated after it are skipped, and that bound is load-bearing
/// twice over:
///
/// - **Convergence.** A declared-but-not-yet-ex distribution is
/// reflected in no provider's adjustment series, so treating it as
/// something to catch up to would refetch on every pass forever.
/// - **Not masking real work.** The bound belongs here, on the
/// selection, rather than on the verdict. Bounding the verdict
/// instead - "is the newest action of all still in the future? then
/// nothing to do" - lets a single forward announcement hide every
/// older unapplied action behind it. That shipped, and it left NKE
/// permanently unable to restate: it announces roughly a quarter
/// ahead, so its newest cached ex-date is essentially always in the
/// future, which masked a distribution that had genuinely gone ex
/// two months earlier.
///
/// Allocates only transiently - the returned `Date` borrows nothing.
pub fn newestCorporateAction(allocator: std.mem.Allocator, store: *cache.Store, symbol: []const u8) ?Date {
pub fn newestCorporateAction(
allocator: std.mem.Allocator,
store: *cache.Store,
symbol: []const u8,
through: Date,
) ?Date {
var newest: ?Date = null;
// `CacheResult` has no deinit - the caller owns `data`. Dividends
@ -656,12 +684,14 @@ pub fn newestCorporateAction(allocator: std.mem.Allocator, store: *cache.Store,
if (store.read(allocator, Dividend, symbol, null, .any)) |r| {
defer Dividend.freeSlice(allocator, r.data);
for (r.data) |d| {
if (through.lessThan(d.ex_date)) continue;
if (newest == null or newest.?.lessThan(d.ex_date)) newest = d.ex_date;
}
}
if (store.read(allocator, Split, symbol, null, .any)) |r| {
defer allocator.free(r.data);
for (r.data) |sp| {
if (through.lessThan(sp.date)) continue;
if (newest == null or newest.?.lessThan(sp.date)) newest = sp.date;
}
}
@ -674,58 +704,40 @@ pub fn newestCorporateAction(allocator: std.mem.Allocator, store: *cache.Store,
///
/// - `adj_basis`: `CandleMeta.adj_basis` - the bar date through which
/// the cached `adj_close` values reflect corporate actions.
/// - `last_bar`: `CandleMeta.last_date` - the newest bar held.
/// - `newest_action`: from `newestCorporateAction`, or null for a
/// symbol with no cached dividends or splits.
/// - `newest_ex`: from `newestCorporateAction`, which has already
/// discarded anything not yet ex. Null for a symbol with no
/// applicable cached dividends or splits.
///
/// The `last_bar` bound is what keeps this convergent. A distribution
/// announced with a future ex-date is not yet reflected in *any*
/// provider's adjustment series, so treating it as stale would refetch
/// on every pass and never settle.
/// Takes no `last_bar`: the not-yet-ex bound lives in
/// `newestCorporateAction` so that it cannot be applied to the wrong
/// end of the comparison. See that function for what happened when it
/// was applied here instead.
///
/// Pure: three dates in, bool out.
pub fn adjustmentBasisStale(adj_basis: Date, last_bar: Date, newest_action: ?Date) bool {
const newest = newest_action orelse return false;
if (last_bar.lessThan(newest)) return false;
/// Pure: two dates in, bool out.
pub fn adjustmentBasisStale(adj_basis: Date, newest_ex: ?Date) bool {
const newest = newest_ex orelse return false;
return adj_basis.lessThan(newest);
}
test "adjustmentBasisStale: no corporate action means nothing to restate" {
test "adjustmentBasisStale: no applicable action means nothing to restate" {
// A non-payer has no ex-date to exceed the basis, so it must never
// escalate - not even with the epoch sentinel a legacy cache parses to.
try testing.expect(!adjustmentBasisStale(Date.fromYmd(2026, 8, 14), Date.fromYmd(2026, 8, 14), null));
try testing.expect(!adjustmentBasisStale(Date.epoch, Date.fromYmd(2026, 8, 14), null));
try testing.expect(!adjustmentBasisStale(Date.fromYmd(2026, 8, 14), null));
try testing.expect(!adjustmentBasisStale(Date.epoch, null));
}
test "adjustmentBasisStale: action behind the basis is stale" {
const last_bar = Date.fromYmd(2026, 8, 14);
const ex = Date.fromYmd(2026, 6, 18);
// Basis at the newest bar already covers the ex-date.
try testing.expect(!adjustmentBasisStale(last_bar, last_bar, ex));
// Basis past the ex-date already covers it.
try testing.expect(!adjustmentBasisStale(Date.fromYmd(2026, 8, 14), ex));
// Basis exactly at the ex-date covers it too (not `lessThan`).
try testing.expect(!adjustmentBasisStale(ex, last_bar, ex));
try testing.expect(!adjustmentBasisStale(ex, ex));
// Basis behind the ex-date: the bars in between were never marked down.
try testing.expect(adjustmentBasisStale(Date.fromYmd(2026, 5, 1), last_bar, ex));
try testing.expect(adjustmentBasisStale(Date.fromYmd(2026, 5, 1), ex));
// The legacy-cache case: sentinel basis on a dividend payer. This is
// the shape every pre-adj_basis cache lands in.
try testing.expect(adjustmentBasisStale(Date.epoch, last_bar, ex));
}
test "adjustmentBasisStale: a not-yet-ex action is not stale" {
// Ex-date beyond the newest bar we hold. No provider has applied it
// either, so escalating would never converge.
try testing.expect(!adjustmentBasisStale(
Date.fromYmd(2026, 5, 1),
Date.fromYmd(2026, 8, 14),
Date.fromYmd(2026, 9, 17),
));
// Boundary: an ex-date exactly on the newest bar IS covered.
try testing.expect(adjustmentBasisStale(
Date.fromYmd(2026, 5, 1),
Date.fromYmd(2026, 8, 14),
Date.fromYmd(2026, 8, 14),
));
try testing.expect(adjustmentBasisStale(Date.epoch, ex));
}
test "newestCorporateAction: takes the max across dividends and splits" {
@ -737,23 +749,114 @@ test "newestCorporateAction: takes the max across dividends and splits" {
defer a.free(dir_path);
var store = cache.Store.init(io, a, dir_path);
const through = Date.fromYmd(2026, 8, 14);
// Nothing cached at all.
try testing.expect(newestCorporateAction(a, &store, "SMPL") == null);
try testing.expect(newestCorporateAction(a, &store, "SMPL", through) == null);
var divs = [_]Dividend{
.{ .ex_date = Date.fromYmd(2026, 3, 20), .amount = 1.79 },
.{ .ex_date = Date.fromYmd(2026, 6, 18), .amount = 1.90 },
};
store.write(Dividend, "SMPL", divs[0..], .{ .seconds = cache.Ttl.dividends });
try testing.expect(newestCorporateAction(a, &store, "SMPL").?.eql(Date.fromYmd(2026, 6, 18)));
try testing.expect(newestCorporateAction(a, &store, "SMPL", through).?.eql(Date.fromYmd(2026, 6, 18)));
// A later split must win over the later dividend.
var splits = [_]Split{.{ .date = Date.fromYmd(2026, 7, 1), .numerator = 2, .denominator = 1 }};
store.write(Split, "SMPL", splits[0..], .{ .seconds = cache.Ttl.splits });
try testing.expect(newestCorporateAction(a, &store, "SMPL").?.eql(Date.fromYmd(2026, 7, 1)));
try testing.expect(newestCorporateAction(a, &store, "SMPL", through).?.eql(Date.fromYmd(2026, 7, 1)));
// Splits alone (different symbol) still resolve.
store.write(Split, "SMPLB", splits[0..], .{ .seconds = cache.Ttl.splits });
try testing.expect(newestCorporateAction(a, &store, "SMPLB").?.eql(Date.fromYmd(2026, 7, 1)));
try testing.expect(newestCorporateAction(a, &store, "SMPLB", through).?.eql(Date.fromYmd(2026, 7, 1)));
}
test "newestCorporateAction: a forward-announced action does not mask an older unapplied one" {
// The regression. NKE announces roughly a quarter ahead, so its
// newest cached ex-date is essentially always in the future. Bounding
// the *verdict* on "is the newest action still in the future?" made
// that one announcement hide a distribution that had gone ex two
// months earlier, and NKE could never restate. The bound belongs on
// the selection, so the announcement is skipped and the older
// already-ex action is still found.
const io = testing.io;
const a = testing.allocator;
var tmp = testing.tmpDir(.{});
defer tmp.cleanup();
const dir_path = try tmp.dir.realPathFileAlloc(io, ".", a);
defer a.free(dir_path);
var store = cache.Store.init(io, a, dir_path);
const last_bar = Date.fromYmd(2026, 8, 18);
// NKE's exact shape: applied through 2026-03-02, 2026-06-01 went ex
// and was never applied, 2026-09-01 declared but not yet ex.
var divs = [_]Dividend{
.{ .ex_date = Date.fromYmd(2026, 9, 1), .amount = 0.41 },
.{ .ex_date = Date.fromYmd(2026, 6, 1), .amount = 0.41 },
.{ .ex_date = Date.fromYmd(2026, 3, 2), .amount = 0.41 },
};
store.write(Dividend, "SMPL", divs[0..], .{ .seconds = cache.Ttl.dividends });
// The future announcement is skipped; the newest already-ex wins.
const newest = newestCorporateAction(a, &store, "SMPL", last_bar);
try testing.expect(newest != null);
try testing.expect(newest.?.eql(Date.fromYmd(2026, 6, 1)));
// And the verdict is therefore "stale" for a legacy sentinel basis.
try testing.expect(adjustmentBasisStale(Date.epoch, newest));
// ...and for a basis covering 2026-03-02 but not 2026-06-01.
try testing.expect(adjustmentBasisStale(Date.fromYmd(2026, 4, 1), newest));
// ...but not once the basis has caught up.
try testing.expect(!adjustmentBasisStale(Date.fromYmd(2026, 8, 17), newest));
}
test "newestCorporateAction: a declared-but-not-effective split does not mask either" {
const io = testing.io;
const a = testing.allocator;
var tmp = testing.tmpDir(.{});
defer tmp.cleanup();
const dir_path = try tmp.dir.realPathFileAlloc(io, ".", a);
defer a.free(dir_path);
var store = cache.Store.init(io, a, dir_path);
const last_bar = Date.fromYmd(2026, 8, 18);
var splits = [_]Split{
.{ .date = Date.fromYmd(2026, 10, 1), .numerator = 2, .denominator = 1 },
.{ .date = Date.fromYmd(2026, 5, 1), .numerator = 3, .denominator = 1 },
};
store.write(Split, "SMPL", splits[0..], .{ .seconds = cache.Ttl.splits });
const newest = newestCorporateAction(a, &store, "SMPL", last_bar);
try testing.expect(newest.?.eql(Date.fromYmd(2026, 5, 1)));
try testing.expect(adjustmentBasisStale(Date.epoch, newest));
}
test "newestCorporateAction: convergence - an only-future action is not actionable" {
// The property the bad bound was reaching for, preserved. A
// declared-but-not-yet-ex action is in no provider's adjustment
// series, so treating it as something to catch up to would refetch
// the full history on every pass, forever.
const io = testing.io;
const a = testing.allocator;
var tmp = testing.tmpDir(.{});
defer tmp.cleanup();
const dir_path = try tmp.dir.realPathFileAlloc(io, ".", a);
defer a.free(dir_path);
var store = cache.Store.init(io, a, dir_path);
var divs = [_]Dividend{.{ .ex_date = Date.fromYmd(2026, 9, 17), .amount = 1.9 }};
store.write(Dividend, "SMPL", divs[0..], .{ .seconds = cache.Ttl.dividends });
const last_bar = Date.fromYmd(2026, 8, 18);
try testing.expect(newestCorporateAction(a, &store, "SMPL", last_bar) == null);
try testing.expect(!adjustmentBasisStale(Date.epoch, newestCorporateAction(a, &store, "SMPL", last_bar)));
// Boundary: once it goes ex - `through` reaching the ex-date - it
// becomes actionable on that very bar.
const on_ex = Date.fromYmd(2026, 9, 17);
try testing.expect(newestCorporateAction(a, &store, "SMPL", on_ex).?.eql(on_ex));
try testing.expect(adjustmentBasisStale(Date.epoch, newestCorporateAction(a, &store, "SMPL", on_ex)));
}

27
src/cache/store.zig vendored
View file

@ -1,6 +1,7 @@
const std = @import("std");
const log = std.log.scoped(.cache);
const srf = @import("srf");
const srf_opts = @import("../srf_opts.zig");
const atomic = @import("../atomic.zig");
const version = @import("../version.zig");
const Date = @import("../Date.zig");
@ -1343,7 +1344,7 @@ pub const Store = struct {
const created = it.created orelse std.Io.Timestamp.now(self.io, .real).toSeconds();
const fields = (it.next() catch return null) orelse return null;
const meta = fields.to(CandleMeta, .{}) catch return null;
const meta = fields.to(CandleMeta, srf_opts.machine_written) catch return null;
return .{ .meta = meta, .created = created };
}
@ -1889,7 +1890,7 @@ pub const Store = struct {
}
// Per-record coercion. Most types use SRF's generalized
// `fields.to(T, .{})` - correct for any struct shape but
// `fields.to(T, ...)` - correct for any struct shape but
// pays a per-field abstraction cost (coerce() boundary,
// found-bitmap bookkeeping, inline-for dispatch chain).
//
@ -1904,7 +1905,7 @@ pub const Store = struct {
var item: T = if (comptime T == Candle)
coerceCandleSpecialized(fields) catch continue
else
fields.to(T, .{}) catch continue;
fields.to(T, srf_opts.machine_written) catch continue;
if (comptime postProcess) |pp| {
pp(&item, allocator) catch {
if (comptime @hasDecl(T, "deinit")) item.deinit(allocator);
@ -1968,7 +1969,7 @@ pub const Store = struct {
defer it.deinit();
const fields = (try it.next()) orelse return error.InvalidData;
return fields.to(CandleMeta, .{}) catch error.InvalidData;
return fields.to(CandleMeta, srf_opts.machine_written) catch error.InvalidData;
}
// Private serialization: options (bespoke)
@ -2038,7 +2039,7 @@ pub const Store = struct {
}
while (try it.next()) |fields| {
const opt_rec = fields.to(OptionsRecord, .{}) catch continue;
const opt_rec = fields.to(OptionsRecord, srf_opts.machine_written) catch continue;
switch (opt_rec) {
.chain => |ch| {
const idx = chains.items.len;
@ -2111,18 +2112,10 @@ pub fn deserializePortfolio(allocator: std.mem.Allocator, data: []const u8) !Por
var skipped: usize = 0;
while (try it.next()) |fields| {
const line = it.state.line;
// `strings_to_numbers` because these are HUMAN-EDITED files, which is
// exactly the case srf's default strict coercion is not for - its own
// doc says "if you want to use this for human-edited files, turn this
// on". Strict mode assumes the writer was a machine, so a numeric
// field spelled with a string separator (`close_price::200.00` instead
// of `close_price:num:200.00`) reaches an unchecked `val.?.number` and
// takes the whole process down. One such typo was enough to panic every
// `zfin portfolio` run.
//
// The `catch` below still handles genuinely unparseable values; this
// only stops a hand-typed separator from being fatal.
var lot = fields.to(Lot, .{ .strings_to_numbers = true }) catch {
// `user_edited` coercion: see `srf_opts.zig` for why hand-edited
// files get different options from cache files. The `catch`
// below still handles genuinely unparseable values.
var lot = fields.to(Lot, srf_opts.user_edited) catch {
std.log.warn("portfolio: could not parse record at line {d}", .{line});
skipped += 1;
continue;

View file

@ -118,13 +118,17 @@ pub fn run(ctx: *framework.RunCtx, parsed: ParsedArgs) !void {
const verbose = parsed.verbose;
const stale_days = parsed.stale_days;
// Flagless mode: run portfolio hygiene check (single-file
// semantics - git blame, commit SHAs, etc.). Resolve paths
// just to find the anchor; we don't need the merged view.
// Flagless mode: run portfolio hygiene check. Mostly single-file
// semantics (git blame, commit SHAs, etc.), so resolve the anchor -
// but the large-lot check inside diffs portfolio content and needs
// the merged view, so hand it the whole glob too.
if (fidelity_csv == null and schwab_csv == null and !schwab_summary) {
const pf = ctx.resolvePortfolioPath();
defer pf.deinit(allocator);
return hygiene.runHygieneCheck(io, allocator, ctx.environ_map, svc, pf.path, stale_days, verbose, as_of, now_s, color, ctx.globals.refresh_policy, out);
var all = ctx.resolvePortfolioPaths() catch null;
defer if (all) |*p| p.deinit();
const paths: []const []const u8 = if (all) |p| p.paths else &.{pf.path};
return hygiene.runHygieneCheck(io, allocator, ctx.environ_map, svc, pf.path, paths, stale_days, verbose, as_of, now_s, color, ctx.globals.refresh_policy, out);
}
// Reconciliation modes (--fidelity / --schwab / --schwab-summary):

View file

@ -729,6 +729,11 @@ pub fn runHygieneCheck(
env: *const std.process.Environ.Map,
svc: *zfin.DataService,
portfolio_path: []const u8,
/// Every file in the `portfolio*.srf` glob. Only the large-lot
/// check needs this: it diffs portfolio CONTENT and so must see
/// the merged view. The rest of the hygiene report is deliberately
/// single-file (git blame, commit SHAs) and uses `portfolio_path`.
portfolio_paths: []const []const u8,
stale_days: u32,
verbose: bool,
as_of: Date,
@ -1226,7 +1231,7 @@ pub fn runHygieneCheck(
// (not in a git repo). Threshold is per-account: an account's
// `audit_large_lot_threshold` in accounts.srf wins, otherwise the
// filter's built-in default applies.
if (contributions.findUnmatchedLargeLots(io, allocator, env, svc, portfolio_path, &account_map, as_of, color, refresh)) |found| {
if (contributions.findUnmatchedLargeLots(io, allocator, env, svc, portfolio_paths, &account_map, as_of, color, refresh)) |found| {
var found_mut = found;
defer found_mut.deinit();
@ -2182,7 +2187,7 @@ test "runHygieneCheck: Section 7 flags an un-opted-in symbol's split, not an opt
var aw: std.Io.Writer.Allocating = .init(allocator);
defer aw.deinit();
try runHygieneCheck(io, allocator, &env, &svc, pf_path, 3, false, zfin.Date.fromYmd(2026, 1, 1), 1_767_225_600, false, .never, &aw.writer);
try runHygieneCheck(io, allocator, &env, &svc, pf_path, &.{pf_path}, 3, false, zfin.Date.fromYmd(2026, 1, 1), 1_767_225_600, false, .never, &aw.writer);
const output = aw.written();
try std.testing.expect(std.mem.indexOf(u8, output, "Portfolio hygiene") != null);

View file

@ -2,6 +2,7 @@ const std = @import("std");
const builtin = @import("builtin");
const zfin = @import("../root.zig");
const srf = @import("srf");
const srf_opts = @import("../srf_opts.zig");
const history = @import("../history.zig");
const git = @import("../git.zig");
const framework = @import("framework.zig");
@ -1007,7 +1008,7 @@ pub fn loadWatchlist(io: std.Io, allocator: std.mem.Allocator, path: []const u8)
var syms: std.ArrayList([]const u8) = .empty;
while (it.next() catch null) |fields| {
const entry = fields.to(WatchEntry, .{}) catch continue;
const entry = fields.to(WatchEntry, srf_opts.user_edited) catch continue;
const duped = allocator.dupe(u8, entry.symbol) catch continue;
syms.append(allocator, duped) catch {
allocator.free(duped);

View file

@ -290,6 +290,16 @@ pub fn run(ctx: *framework.RunCtx, parsed: ParsedArgs) !void {
defer pf.deinit(allocator);
const portfolio_path = pf.path;
// Attribution diffs portfolio CONTENT, so it needs the whole
// `portfolio*.srf` glob rather than the anchor alone - otherwise a
// sold lot archived into a sibling file reads as a bare
// disappearance and its proceeds can't offset the repurchase.
// Resolution failure is non-fatal: the attribution line is
// optional, so fall back to the anchor on its own.
var attr_pf = ctx.resolvePortfolioPaths() catch null;
defer if (attr_pf) |*p| p.deinit();
const attr_paths: []const []const u8 = if (attr_pf) |p| p.paths else &.{portfolio_path};
const with_projections = parsed.with_projections;
const events_enabled = parsed.events_enabled;
const snapshot_after_live = parsed.after_is_live and parsed.snapshot_after == null;
@ -471,7 +481,7 @@ pub fn run(ctx: *framework.RunCtx, parsed: ParsedArgs) !void {
// Attribution uses the resolved CommitSpecs so --commit-*
// overrides + date fallbacks share one classifier. The caller
// adapts dates to `CommitSpec.date_at_or_before` upstream.
const attribution = contributions.computeAttributionSpec(io, allocator, ctx.environ_map, svc, portfolio_path, attr_before, attr_after_opt, as_of, color, ctx.globals.refresh_policy);
const attribution = contributions.computeAttributionSpec(io, allocator, ctx.environ_map, svc, attr_paths, attr_before, attr_after_opt, as_of, color, ctx.globals.refresh_policy);
try renderFromParts(out, color, allocator, .{
.then_date = then_date,
@ -488,7 +498,7 @@ pub fn run(ctx: *framework.RunCtx, parsed: ParsedArgs) !void {
var now_side = try compare_core.loadSnapshotSide(io, allocator, hist_dir, now_date);
defer now_side.deinit(allocator);
const attribution = contributions.computeAttributionSpec(io, allocator, ctx.environ_map, svc, portfolio_path, attr_before, attr_after_opt, as_of, color, ctx.globals.refresh_policy);
const attribution = contributions.computeAttributionSpec(io, allocator, ctx.environ_map, svc, attr_paths, attr_before, attr_after_opt, as_of, color, ctx.globals.refresh_policy);
try renderFromParts(out, color, allocator, .{
.then_date = then_date,

File diff suppressed because it is too large Load diff

View file

@ -254,8 +254,8 @@ pub fn run(ctx: *framework.RunCtx, parsed: ParsedArgs) !void {
// bars behind it - total returns then read low by roughly the
// missed yield. Report it here because the symptom (a slightly
// low 1Y total return) is otherwise invisible.
if (freshness.newestCorporateAction(arena, &store, symbol)) |newest_action| {
if (freshness.adjustmentBasisStale(m.meta.adj_basis, m.meta.last_date, newest_action)) {
if (freshness.newestCorporateAction(arena, &store, symbol, m.meta.last_date)) |newest_action| {
if (freshness.adjustmentBasisStale(m.meta.adj_basis, newest_action)) {
try out.print(
"adj basis {f} - STALE, {f} went ex behind it; total returns read low until restated\n",
.{ m.meta.adj_basis, newest_action },

View file

@ -66,6 +66,7 @@
const std = @import("std");
const srf = @import("srf");
const srf_opts = @import("../srf_opts.zig");
const Date = @import("../Date.zig");
const atomic = @import("../atomic.zig");
@ -214,7 +215,7 @@ pub fn parse(allocator: std.mem.Allocator, data: []const u8) !Journal {
defer it.deinit();
while (try it.next()) |fields| {
const rec = try fields.to(JournalRecord, .{});
const rec = try fields.to(JournalRecord, srf_opts.user_edited);
switch (rec) {
.acknowledgment => |a| {
try entries.append(allocator, .{

View file

@ -35,6 +35,7 @@
const std = @import("std");
const srf = @import("srf");
const srf_opts = @import("../srf_opts.zig");
const Date = @import("../Date.zig");
// Types
@ -176,7 +177,7 @@ pub fn parseImportedValues(
errdefer points.deinit(allocator);
while (it.next() catch return error.InvalidSrf) |fields| {
const point = fields.to(HistoryPoint, .{}) catch return error.InvalidSrf;
const point = fields.to(HistoryPoint, srf_opts.user_edited) catch return error.InvalidSrf;
try points.append(allocator, point);
}

View file

@ -207,17 +207,29 @@ pub fn findRepo(io: std.Io, allocator: std.mem.Allocator, env: *const std.proces
const root = try allocator.dupe(u8, root_raw);
errdefer allocator.free(root);
// Relative path from root to the file. If `abs_path` starts with the
// repo root (the common case), trim the prefix; otherwise fall back to
// just the basename (extremely unusual - repo root disagrees with
// path).
const rel = try relPathInRoot(allocator, root, abs_path);
return .{ .root = root, .rel_path = rel };
}
/// Relative path from `root` to `abs_path`, as git pathspecs want it.
///
/// If `abs_path` starts with the repo root (the common case), trim the
/// prefix; otherwise fall back to just the basename (extremely unusual
/// - repo root disagrees with path). Caller owns the result.
pub fn relPathInRoot(allocator: std.mem.Allocator, root: []const u8, abs_path: []const u8) ![]const u8 {
const rel_raw = if (std.mem.startsWith(u8, abs_path, root) and abs_path.len > root.len)
std.mem.trimStart(u8, abs_path[root.len..], "/")
else
std.fs.path.basename(abs_path);
const rel = try allocator.dupe(u8, rel_raw);
return allocator.dupe(u8, rel_raw);
}
return .{ .root = root, .rel_path = rel };
/// `relPathInRoot` against an already-discovered repo. Convenience for
/// callers holding a `RepoInfo` that need rel-paths for the anchor's
/// sibling files (e.g. every file in a `portfolio*.srf` glob).
pub fn relPathIn(allocator: std.mem.Allocator, repo: RepoInfo, abs_path: []const u8) ![]const u8 {
return relPathInRoot(allocator, repo.root, abs_path);
}
/// Report the tracked/untracked/modified status of `rel_path` relative to
@ -396,22 +408,26 @@ pub fn lastCommitTimestampForPath(
return std.fmt.parseInt(i64, trimmed, 10) catch return null;
}
/// Return the SHA of the most recent commit that touched `rel_path` at
/// or before `date_iso` (YYYY-MM-DD, inclusive end-of-day semantics via
/// `git log --until`).
/// Return the SHA of the most recent commit that touched any of
/// `rel_paths` at or before `date_iso` (YYYY-MM-DD, inclusive
/// end-of-day semantics via `git log --until`).
///
/// Returns null if no commit before `date_iso` touched `rel_path`.
/// Returns null if no commit before `date_iso` touched any of them.
/// Caller owns the returned string.
///
/// Used by `zfin contributions --since <DATE>` / `--until <DATE>` to
/// resolve a date to the last commit that stamped a given snapshot of
/// the portfolio file.
/// the portfolio. Takes a slice rather than one path because the
/// portfolio is a `portfolio*.srf` glob: resolving against only the
/// first file would pick an older commit whenever the commit in range
/// touched just a sibling (a sold lot moved into
/// `portfolio_closed.srf`), silently widening the window.
pub fn commitAtOrBeforeDate(
io: std.Io,
allocator: std.mem.Allocator,
env: *const std.process.Environ.Map,
root: []const u8,
rel_path: []const u8,
rel_paths: []const []const u8,
date_iso: []const u8,
) Error!?[]const u8 {
// `git log --until=DATE` with a bare YYYY-MM-DD uses the *current
@ -426,11 +442,22 @@ pub fn commitAtOrBeforeDate(
const until_arg = try std.fmt.allocPrint(allocator, "--until={s} 23:59:59", .{date_iso});
defer allocator.free(until_arg);
const result = runGit(io, allocator, env, &.{
// `git log -1 -- p1 p2 ...` already returns the newest commit
// touching ANY of the pathspecs, so a multi-file portfolio glob
// resolves a date correctly even when the only commit in range
// touched just one of the files (e.g. a sold lot moved into
// `portfolio_closed.srf`). Argv is built dynamically because the
// path count is not known at comptime.
var argv: std.ArrayList([]const u8) = .empty;
defer argv.deinit(allocator);
try argv.appendSlice(allocator, &.{
"git", "-C", root,
"log", "-1", "--format=%H",
until_arg, "--", rel_path,
}, .limited(64 * 1024)) catch return error.GitUnavailable;
until_arg, "--",
});
try argv.appendSlice(allocator, rel_paths);
const result = runGit(io, allocator, env, argv.items, .limited(64 * 1024)) catch return error.GitUnavailable;
defer allocator.free(result.stdout);
defer allocator.free(result.stderr);
@ -579,6 +606,7 @@ pub fn resolveCommitRangeSpec(
arena: std.mem.Allocator,
env: *const std.process.Environ.Map,
repo: RepoInfo,
rel_paths: []const []const u8,
before: ?CommitSpec,
after: ?CommitSpec,
dirty: bool,
@ -590,7 +618,7 @@ pub fn resolveCommitRangeSpec(
// Resolve each endpoint independently.
const before_rev: []const u8 = if (before) |b|
try resolveSpec(io, arena, env, repo, b)
try resolveSpec(io, arena, env, repo, rel_paths, b)
else if (dirty)
"HEAD"
else
@ -599,7 +627,7 @@ pub fn resolveCommitRangeSpec(
const after_rev: ?[]const u8 = if (after) |a|
(switch (a) {
.working_copy => null,
else => try resolveSpec(io, arena, env, repo, a),
else => try resolveSpec(io, arena, env, repo, rel_paths, a),
})
else if (dirty)
null
@ -612,14 +640,14 @@ pub fn resolveCommitRangeSpec(
/// Resolve one non-working `CommitSpec` to a string git can consume.
/// Caller handles the `.working_copy` case separately (it's not a
/// git ref).
fn resolveSpec(io: std.Io, arena: std.mem.Allocator, env: *const std.process.Environ.Map, repo: RepoInfo, spec: CommitSpec) Error![]const u8 {
fn resolveSpec(io: std.Io, arena: std.mem.Allocator, env: *const std.process.Environ.Map, repo: RepoInfo, rel_paths: []const []const u8, spec: CommitSpec) Error![]const u8 {
return switch (spec) {
.git_ref => |r| r,
.date_at_or_before => |d| blk: {
var buf: [10]u8 = undefined;
// SAFETY: 10-byte buffer is exactly the size of "YYYY-MM-DD".
const date_str = std.fmt.bufPrint(&buf, "{f}", .{d}) catch buf[0..];
const sha = (try commitAtOrBeforeDate(io, arena, env, repo.root, repo.rel_path, date_str)) orelse
const sha = (try commitAtOrBeforeDate(io, arena, env, repo.root, rel_paths, date_str)) orelse
return error.NoCommitAtOrBefore;
break :blk sha;
},
@ -639,6 +667,7 @@ pub fn resolveCommitRange(
arena: std.mem.Allocator,
env: *const std.process.Environ.Map,
repo: RepoInfo,
rel_paths: []const []const u8,
since: ?Date,
until: ?Date,
dirty: bool,
@ -646,7 +675,7 @@ pub fn resolveCommitRange(
std.debug.assert(!(since == null and until != null));
const before: ?CommitSpec = if (since) |d| .{ .date_at_or_before = d } else null;
const after: ?CommitSpec = if (until) |d| .{ .date_at_or_before = d } else null;
return resolveCommitRangeSpec(io, arena, env, repo, before, after, dirty);
return resolveCommitRangeSpec(io, arena, env, repo, rel_paths, before, after, dirty);
}
// Tests
@ -725,7 +754,7 @@ test "commitAtOrBeforeDate returns a SHA for a past date" {
// Any date well after the repo's creation - commitAtOrBeforeDate
// should find the most recent commit touching build.zig.
const sha_opt = commitAtOrBeforeDate(std.testing.io, allocator, &env, info.root, info.rel_path, "2099-01-01") catch return;
const sha_opt = commitAtOrBeforeDate(std.testing.io, allocator, &env, info.root, &.{info.rel_path}, "2099-01-01") catch return;
try std.testing.expect(sha_opt != null);
const sha = sha_opt.?;
defer allocator.free(sha);
@ -744,7 +773,7 @@ test "commitAtOrBeforeDate returns null for date before repo existed" {
defer allocator.free(info.rel_path);
// Pre-git - before any sensible project history.
const sha_opt = commitAtOrBeforeDate(std.testing.io, allocator, &env, info.root, info.rel_path, "1970-01-02") catch return;
const sha_opt = commitAtOrBeforeDate(std.testing.io, allocator, &env, info.root, &.{info.rel_path}, "1970-01-02") catch return;
try std.testing.expect(sha_opt == null);
}
@ -773,7 +802,7 @@ test "commitAtOrBeforeDate: --until=DATE covers end of day, not current time-of-
// Future-dated cutoff - should always return the tip of history
// regardless of current wall-clock time.
const sha_opt = commitAtOrBeforeDate(std.testing.io, allocator, &env, info.root, info.rel_path, "2099-01-01") catch return;
const sha_opt = commitAtOrBeforeDate(std.testing.io, allocator, &env, info.root, &.{info.rel_path}, "2099-01-01") catch return;
try std.testing.expect(sha_opt != null);
if (sha_opt) |s| allocator.free(s);
}
@ -817,7 +846,7 @@ test "resolveCommitRange: legacy clean -> HEAD~1..HEAD" {
defer env.deinit();
const repo: RepoInfo = .{ .root = "/tmp", .rel_path = "portfolio.srf" };
const range = try resolveCommitRange(std.testing.io, arena_state.allocator(), &env, repo, null, null, false);
const range = try resolveCommitRange(std.testing.io, arena_state.allocator(), &env, repo, &.{repo.rel_path}, null, null, false);
try std.testing.expectEqualStrings("HEAD~1", range.before_rev);
try std.testing.expectEqualStrings("HEAD", range.after_rev.?);
}
@ -829,7 +858,7 @@ test "resolveCommitRange: legacy dirty -> HEAD..working-copy" {
defer env.deinit();
const repo: RepoInfo = .{ .root = "/tmp", .rel_path = "portfolio.srf" };
const range = try resolveCommitRange(std.testing.io, arena_state.allocator(), &env, repo, null, null, true);
const range = try resolveCommitRange(std.testing.io, arena_state.allocator(), &env, repo, &.{repo.rel_path}, null, null, true);
try std.testing.expectEqualStrings("HEAD", range.before_rev);
try std.testing.expect(range.after_rev == null);
}
@ -851,6 +880,7 @@ test "resolveCommitRange: --since resolves to SHA..HEAD for clean tree" {
arena_state.allocator(),
&env,
info,
&.{info.rel_path},
Date.fromYmd(2099, 1, 1),
null,
false,
@ -876,6 +906,7 @@ test "resolveCommitRange: --since with no earlier commit -> NoCommitAtOrBefore"
arena_state.allocator(),
&env,
info,
&.{info.rel_path},
Date.fromYmd(1970, 1, 2),
null,
false,

View file

@ -32,6 +32,7 @@
const std = @import("std");
const builtin = @import("builtin");
const srf = @import("srf");
const srf_opts = @import("srf_opts.zig");
const snapshot = @import("models/snapshot.zig");
const Date = @import("Date.zig");
const timeline = @import("analytics/timeline.zig");
@ -99,7 +100,7 @@ pub fn parseSnapshotBytes(
// record kind we don't know about). Every other srf error
// indicates malformed data in a record we SHOULD understand, so
// we propagate it up rather than silently losing rows.
const rec = field_it.to(SnapshotRecord, .{}) catch |err| switch (err) {
const rec = field_it.to(SnapshotRecord, srf_opts.machine_written) catch |err| switch (err) {
error.ActiveTagDoesNotExist => continue,
else => return error.InvalidSrf,
};

View file

@ -11,6 +11,7 @@
/// symbol::02315N600,asset_class::Bonds,pct:num:15
const std = @import("std");
const srf = @import("srf");
const srf_opts = @import("../srf_opts.zig");
const Date = @import("../Date.zig");
/// A single classification entry for a symbol.
@ -89,7 +90,7 @@ pub fn parseClassificationFile(allocator: std.mem.Allocator, data: []const u8) !
defer it.deinit();
while (try it.next()) |fields| {
const entry = fields.to(ClassificationEntry, .{}) catch continue;
const entry = fields.to(ClassificationEntry, srf_opts.user_edited) catch continue;
// Pre-fill `bucket` if the user didn't curate one. This
// shifts the cost of `deriveBucket` to parse time and
// makes downstream code free to read `entry.bucket`
@ -269,6 +270,24 @@ test "parse classification file: missing name field stays null (backwards compat
try std.testing.expectEqualStrings("Technology", cm.entries[0].sector.?);
}
test "parse classification file: a hand-typed string separator on pct still parses" {
// `metadata.srf` is hand-edited, so `pct::60` instead of `pct:num:60`
// must not be fatal. See `srf_opts.user_edited` - under strict
// coercion this string reaches an unchecked `val.?.number`, which is
// undefined behaviour in ReleaseFast.
const data =
\\#!srfv1
\\symbol::SYM,sector::Technology,pct::60
\\symbol::SYM,sector::Healthcare,pct:num:40
;
var map = try parseClassificationFile(std.testing.allocator, data);
defer map.deinit();
try std.testing.expectEqual(@as(usize, 2), map.entries.len);
try std.testing.expectApproxEqAbs(@as(f64, 60), map.entries[0].pct, 0.001);
try std.testing.expectApproxEqAbs(@as(f64, 40), map.entries[1].pct, 0.001);
}
test "parse classification file: bucket round-trips" {
const data =
\\#!srfv1

View file

@ -60,6 +60,7 @@
const std = @import("std");
const builtin = @import("builtin");
const srf = @import("srf");
const srf_opts = @import("../srf_opts.zig");
const Date = @import("../Date.zig");
const logger = std.log.scoped(.transaction_log);
@ -248,7 +249,7 @@ pub fn parseTransactionLogFile(
defer it.deinit();
while (try it.next()) |fields| {
const parsed = fields.to(TransferRecord, .{}) catch |err| {
const parsed = fields.to(TransferRecord, srf_opts.user_edited) catch |err| {
// Tests intentionally feed malformed records to exercise the
// skip path; real parse failures stay visible outside tests.
if (!builtin.is_test) {
@ -560,6 +561,21 @@ test "parseTransactionLogFile: single cash transfer" {
try testing.expect(r.note == null);
}
test "parseTransactionLogFile: a hand-typed string separator on amount still parses" {
// Hand-edited file, so `amount::50000` instead of `amount:num:50000`
// must not be fatal. See `srf_opts.user_edited` - under strict
// coercion this string reaches an unchecked `val.?.number`, which is
// undefined behaviour in ReleaseFast.
var log = try parseTransactionLogFile(testing.allocator,
\\#!srfv1
\\transfer::2026-05-02,type::cash,amount::50000,from::Acct A,to::Acct B,dest_lot::cash
\\
);
defer log.deinit();
try testing.expectEqual(@as(usize, 1), log.transfers.len);
try testing.expectEqual(@as(f64, 50000), log.transfers[0].amount);
}
test "parseTransactionLogFile: single lot-destination transfer" {
var log = try parseTransactionLogFile(testing.allocator,
\\#!srfv1

View file

@ -270,10 +270,8 @@ pub fn loadPortfolioFromPathsAtRev(
};
defer allocator.free(real);
const rel = if (std.mem.startsWith(u8, real, info.root) and real.len > info.root.len)
std.mem.trimStart(u8, real[info.root.len..], "/")
else
std.fs.path.basename(real);
const rel = git.relPathInRoot(allocator, info.root, real) catch return null;
defer allocator.free(rel);
const data = git.show(io, allocator, env, info.root, rev, rel) catch |err| switch (err) {
error.PathMissingInRev => empty_blk: {

View file

@ -25,6 +25,7 @@ const Config = @import("Config.zig");
const cache = @import("cache/store.zig");
const freshness = @import("cache/freshness.zig");
const srf = @import("srf");
const srf_opts = @import("srf_opts.zig");
const analysis = @import("analytics/analysis.zig");
const transaction_log = @import("models/transaction_log.zig");
const TwelveData = @import("providers/twelvedata.zig").TwelveData;
@ -842,10 +843,21 @@ pub const DataService = struct {
next.provider = provider;
next.fail_count = 0;
// A provider change and a cleared backoff are independent facts,
// and conflating them hid the first one. This used to log only
// when clearing an armed backoff - but the case that mattered
// was a legacy cache carrying `provider = .yahoo` with no
// backoff at all, which is every symbol that had drifted off
// Tiingo before `tiingo_retry_after_s` existed. Twenty-one
// symbols converted back to Tiingo without a single line.
if (meta.provider != provider) {
log.info("{s}: candle provider {t} -> {t}", .{ symbol, meta.provider, provider });
}
switch (coverage) {
.covered => {
if (meta.tiingo_retry_after_s != 0) {
log.info("{s}: provider converted {t} -> tiingo, clearing Tiingo backoff", .{ symbol, meta.provider });
log.info("{s}: Tiingo serving again, clearing backoff", .{symbol});
}
next.tiingo_retry_after_s = 0;
},
@ -1146,9 +1158,9 @@ pub const DataService = struct {
// see a same-dated file whose historical adj_close is
// stale, so the basis has to be re-checked here.
const synced = if (s.readCandleMeta(symbol)) |sm| sm.meta else m;
const synced_action = freshness.newestCorporateAction(self.allocator, &s, symbol);
const synced_action = freshness.newestCorporateAction(self.allocator, &s, symbol, synced.last_date);
if (s.isCandleMetaFresh(symbol) and
!freshness.adjustmentBasisStale(synced.adj_basis, synced.last_date, synced_action))
!freshness.adjustmentBasisStale(synced.adj_basis, synced_action))
{
log.debug("{s}: candles synced from server and fresh", .{symbol});
if (s.read(self.allocator, Candle, symbol, null, .any)) |r|
@ -1179,8 +1191,7 @@ pub const DataService = struct {
// tax the hot portfolio-pricing path for nothing.
if (freshness.adjustmentBasisStale(
m.adj_basis,
m.last_date,
freshness.newestCorporateAction(self.allocator, &s, symbol),
freshness.newestCorporateAction(self.allocator, &s, symbol, m.last_date),
)) {
log.info("{s}: restating full history (adj_basis {f} predates a corporate action)", .{ symbol, m.adj_basis });
if (self.refetchFullHistory(symbol, today, now_s, now_s < m.tiingo_retry_after_s)) |candles| {
@ -2987,7 +2998,7 @@ pub const DataService = struct {
defer it.deinit();
while (it.next() catch return result) |fields| {
const entry = fields.to(CusipEntry, .{}) catch continue;
const entry = fields.to(CusipEntry, srf_opts.machine_written) catch continue;
if (entry.cusip.len == 0 or entry.ticker.len == 0) continue;
// First occurrence wins; getOrPut stores the borrowed
// slices directly - they live in `backing`, no dupe.
@ -3153,7 +3164,7 @@ pub const DataService = struct {
var it = srf.iterator(&reader, arena, .{ .parse_allocator = .none }) catch return;
defer it.deinit();
while (it.next() catch return) |fields| {
const e = fields.to(CusipEntry, .{}) catch continue;
const e = fields.to(CusipEntry, srf_opts.machine_written) catch continue;
if (e.cusip.len == 0 or e.ticker.len == 0) continue;
if (have.contains(e.cusip) or out.contains(e.cusip)) continue;
const kc = arena.dupe(u8, e.cusip) catch continue;
@ -4300,8 +4311,7 @@ test "getCandles offline never escalates a stale adjustment basis" {
store.updateCandleMeta("SMPL", meta, 1); // expiry in the past => stale
try std.testing.expect(freshness.adjustmentBasisStale(
meta.adj_basis,
meta.last_date,
freshness.newestCorporateAction(allocator, &store, "SMPL"),
freshness.newestCorporateAction(allocator, &store, "SMPL", meta.last_date),
));
svc.panic_on_network_attempt = true;

56
src/srf_opts.zig Normal file
View file

@ -0,0 +1,56 @@
//! SRF coercion policy, in one place.
//!
//! SRF encodes a field's type in its separator: `key::v` is a string,
//! `key:num:v` a number, `key:bool:v` a boolean. Coercion into a typed
//! struct therefore depends on the writer having picked the right one.
//!
//! That assumption holds for files zfin writes and breaks for files
//! people write, so the two get different options - and which one a
//! parser wants is a property of where the file came from, not of the
//! struct being parsed. Naming the two policies keeps that decision
//! visible at the call site instead of buried in whichever comment
//! happened to explain it first.
const srf = @import("srf");
/// For files a HUMAN edits: `portfolio.srf`, `accounts.srf`,
/// `metadata.srf`, `watchlist.srf`, `transaction_log.srf`,
/// `projections.srf`, `imported_values.srf`, `acknowledgments.srf`, and
/// the keybind config.
///
/// Accepts a string where a number was declared, because a hand-typed
/// `close_price::200.00` instead of `close_price:num:200.00` is a
/// slip, not a different intent. SRF's own doc says as much: strict
/// coercion is "intended for performant access for cache use cases...
/// if you want to use this for human-edited files, turn this on".
///
/// It is also a safety measure, which is the part worth not
/// forgetting. Under strict coercion a string reaching a numeric field
/// falls through to an unchecked `val.?.number` inside SRF - a panic
/// in Debug/ReleaseSafe and undefined behaviour in ReleaseFast, which
/// is how zfin is built. One `close_price::200.00` once took down
/// every `zfin portfolio` run; the fix was to turn this on for
/// `portfolio.srf` alone, which left every other hand-edited file
/// exposed to the same typo.
///
/// This is mitigation, not a cure. Two holes remain, both needing an
/// upstream fix in SRF's `coerce`:
///
/// - a non-string, non-number value in a numeric field (say
/// `harvested:bool:true`) still reaches the unchecked access;
/// - enum fields ignore this option entirely, so a typo like
/// `security_type::stok` still hits `stringToEnum(...).?`.
pub const user_edited: srf.CoercionOptions = .{ .strings_to_numbers = true };
/// For files ZFIN writes: the candle, quote, and options caches,
/// `cusip_tickers.srf` under `cache_dir`, server responses, and the
/// `history/<date>-portfolio.srf` snapshots produced by
/// `zfin snapshot`.
///
/// Keeps SRF's strict default. The writer is a machine that always
/// emits `:num:` for numbers, so accepting a string instead would only
/// mask a serializer bug rather than tolerate a human slip - these
/// files are not edited after the fact. Strictness is also free
/// performance on the hot paths, where a cached candle file is
/// millions of records.
pub const machine_written: srf.CoercionOptions = .{};

View file

@ -1,6 +1,7 @@
const std = @import("std");
const vaxis = @import("vaxis");
const srf = @import("srf");
const srf_opts = @import("../srf_opts.zig");
pub const Action = enum {
quit,
@ -486,7 +487,7 @@ pub fn loadFromDataChecked(allocator: std.mem.Allocator, data: []const u8) LoadO
var idx: usize = 0;
while (ri.next() catch return .fallback) |fields| : (idx += 1) {
const raw = fields.to(RawRecord, .{}) catch |err| {
const raw = fields.to(RawRecord, srf_opts.user_edited) catch |err| {
// Per-record parse failure (missing field, bad key
// string, unknown action). Don't drop the whole file -
// skip the record and warn the user. Record index is