Compare commits
4 commits
e3ce8f32a1
...
c15ea92b0d
| Author | SHA1 | Date | |
|---|---|---|---|
| c15ea92b0d | |||
| 12eace86b2 | |||
| 5706eed3c1 | |||
| c986df83c1 |
24 changed files with 1873 additions and 307 deletions
|
|
@ -73,6 +73,15 @@ naive diff sees the receiving account gain lots and counts it as new
|
|||
money. `transaction_log.srf` is how you tell zfin "this was a transfer,
|
||||
not new money."
|
||||
|
||||
This is only needed for movement **between** accounts. Reallocating
|
||||
*within* one account -- selling a holding to buy another -- is detected
|
||||
on its own, with the sale's proceeds offsetting the repurchase, so no
|
||||
record is required. See
|
||||
[Internal movement](../reference/cli/contributions.md#internal-movement).
|
||||
Recording `close_price` on the sold lot is worth the keystrokes: it is
|
||||
what the sale realized, and zfin values the offset from it rather than
|
||||
guessing from the current price.
|
||||
|
||||
Like the other sibling files, it's **optional and additive**: you only
|
||||
need it if you move money between your own accounts and want clean
|
||||
attribution. Without it nothing breaks -- those transfers just show up
|
||||
|
|
|
|||
|
|
@ -7,9 +7,11 @@ points in your portfolio's **git history**.
|
|||
Usage: zfin contributions [opts]
|
||||
```
|
||||
|
||||
`contributions` diffs two git revisions of your `portfolio.srf` and
|
||||
attributes the share/lot changes to new money vs. market movement. Your
|
||||
portfolio must be under git with commits over time.
|
||||
`contributions` diffs two git revisions of your `portfolio*.srf` files
|
||||
and attributes the share/lot changes to new money vs. market movement.
|
||||
Every file matching the glob is read at both revisions and merged, so a
|
||||
sold lot archived into a sibling `portfolio_closed.srf` is still seen.
|
||||
Your portfolio must be under git with commits over time.
|
||||
|
||||
## Modes
|
||||
|
||||
|
|
@ -41,9 +43,38 @@ most one of `--until`/`--commit-after`.
|
|||
zfin contributions --since 1Y
|
||||
```
|
||||
|
||||
Internal transfers between your own accounts are excluded from the
|
||||
attribution total when declared in
|
||||
[`transaction_log.srf`](../config/transaction-log-srf.md).
|
||||
## Internal movement
|
||||
|
||||
Money that was already inside an account is not a contribution -- it
|
||||
just changed form. Two shapes are detected automatically, with no
|
||||
bookkeeping on your part, and both report under **Internal purchases**
|
||||
rather than counting toward the total:
|
||||
|
||||
- **Buying with cash already in the account.** The buy appears
|
||||
alongside the account's cash going down.
|
||||
- **Reallocating -- selling one holding to buy another in the same
|
||||
account.** The sale's proceeds offset the repurchase.
|
||||
|
||||
A sale is valued at `close_price` when you record one (see
|
||||
[`portfolio.srf`](../config/portfolio-srf.md)), which is what the sale
|
||||
actually realized. If you delete the lot outright instead, there is no
|
||||
`close_price` to read and the current market price stands in -- accurate
|
||||
for a recent sale, less so for one made long before the end of the
|
||||
window. The report labels which was used: `at close` or `at mark`.
|
||||
|
||||
Closing a position that has accumulated a lot per dividend
|
||||
reinvestment retires many lots at once, so sales collapse to one line
|
||||
per account and symbol, carrying the lot count and the total.
|
||||
|
||||
Proceeds still sitting in cash at the end of the window cannot have
|
||||
funded anything, and are treated accordingly. On an account marked
|
||||
`cash_is_contribution::true` they also cancel that account's cash
|
||||
credit, since the sale is not new money even though cash arrived.
|
||||
|
||||
Movement **between** accounts is a different matter -- zfin cannot tell
|
||||
it from a contribution, so declare it in
|
||||
[`transaction_log.srf`](../config/transaction-log-srf.md). An explicit
|
||||
record always wins over the automatic netting above.
|
||||
|
||||
## See also
|
||||
|
||||
|
|
|
|||
|
|
@ -37,8 +37,8 @@ symbol::VTI,shares:num:100,open_date::2024-01-15,open_price:num:220.50,account::
|
|||
| `shares` | number | Yes | Share count (or face value for cash/CDs). Negative for short option positions. |
|
||||
| `open_date` | string | Yes\*\* | Purchase date `YYYY-MM-DD`. \*\*Not required for `cash`/`watch`. |
|
||||
| `open_price` | number | Yes\*\* | Purchase price per share. \*\*Not required for `cash`/`watch`. |
|
||||
| `close_date` | string | No | Sale date. Omit for an open lot. |
|
||||
| `close_price` | number | No | Sale price per share. |
|
||||
| `close_date` | string | No | Sale date. Omit for an open lot. See [Closed lots](#closed-lots). |
|
||||
| `close_price` | number | No | Sale price per share. See [Closed lots](#closed-lots). |
|
||||
| `security_type` | string | No | `stock` (default), `option`, `cd`, `cash`, `illiquid`, `watch`. |
|
||||
| `account` | string | No | Account name. Should match an `account::` entry in [`accounts.srf`](accounts-srf.md). |
|
||||
| `note` | string | No | Free-text note (shown in cash/CD/illiquid tables). |
|
||||
|
|
@ -96,6 +96,26 @@ security_type::cd,symbol::912797KR0,shares:num:10000,open_date::2024-06-01,open_
|
|||
security_type::illiquid,symbol::HOME,shares:num:450000,open_date::2020-06-01,open_price:num:350000,note::Primary residence
|
||||
```
|
||||
|
||||
## Closed lots
|
||||
|
||||
A lot with `close_date` set is sold. Both fields are optional -- an open
|
||||
lot has neither -- but a sold lot should carry both, because they drive
|
||||
two things beyond bookkeeping:
|
||||
|
||||
- **Realized gain/loss** comes from `close_price` against `open_price`.
|
||||
- **[`zfin contributions`](../cli/contributions.md#internal-movement)**
|
||||
treats the close as a sale and values it at `close_price`. That
|
||||
matters when you sell one holding to buy another: the proceeds offset
|
||||
the repurchase instead of it reading as new money. Without a
|
||||
`close_price` the current market price stands in, which is close
|
||||
enough for a recent sale and wrong for an old one.
|
||||
|
||||
You can either edit the lot where it sits or move it to a sibling file
|
||||
such as `portfolio_closed.srf` -- the `portfolio*.srf` glob picks it up
|
||||
either way, so realized gains and back-dated (`--as-of`) views stay
|
||||
correct. See
|
||||
[splitting your portfolio](../../guides/set-up-your-portfolio.md).
|
||||
|
||||
## Price resolution
|
||||
|
||||
For stock lots, the displayed price is resolved in this order:
|
||||
|
|
|
|||
|
|
@ -94,7 +94,11 @@ destination is either fully a transfer or not one at all.
|
|||
## Scope and limits
|
||||
|
||||
- Only `transfer::` records. Buys, sells, and dividends stay inferred
|
||||
from the portfolio diff.
|
||||
from the portfolio diff. You do **not** need a record for selling one
|
||||
holding to buy another inside a single account -- that is detected
|
||||
automatically, and the sale's proceeds offset the repurchase. See
|
||||
[`zfin contributions`](../cli/contributions.md#internal-movement) for
|
||||
how sales are valued and when a record is still required.
|
||||
- Forward-looking only -- there is no historical reconstruction.
|
||||
- Account names are matched byte-exactly, so a
|
||||
[renamed account](../../guides/set-up-accounts.md#renaming-an-account)
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@
|
|||
const std = @import("std");
|
||||
const builtin = @import("builtin");
|
||||
const srf = @import("srf");
|
||||
const srf_opts = @import("../srf_opts.zig");
|
||||
const Allocation = @import("valuation.zig").Allocation;
|
||||
const ClassificationMap = @import("../models/classification.zig").ClassificationMap;
|
||||
const ClassificationEntry = @import("../models/classification.zig").ClassificationEntry;
|
||||
|
|
@ -546,7 +547,7 @@ pub fn parseAccountsFile(allocator: std.mem.Allocator, data: []const u8) !Accoun
|
|||
defer it.deinit();
|
||||
|
||||
while (try it.next()) |fields| {
|
||||
const entry = fields.to(AccountTaxEntry, .{}) catch continue;
|
||||
const entry = fields.to(AccountTaxEntry, srf_opts.user_edited) catch continue;
|
||||
|
||||
// A zero/negative large-lot threshold is nonsensical (zero
|
||||
// flags every new lot; negative is meaningless). Reject it and
|
||||
|
|
@ -1271,6 +1272,28 @@ test "parseAccountsFile: cash_is_contribution default false, opt-in true" {
|
|||
try std.testing.expect(!am.cashIsContribution("Nonexistent"));
|
||||
}
|
||||
|
||||
test "parseAccountsFile: a hand-typed string separator on a numeric field still parses" {
|
||||
// `accounts.srf` is hand-edited, so `harvested::5000` instead of
|
||||
// `harvested:num:5000` is a slip rather than different intent. Under
|
||||
// SRF's strict default that string reaches an unchecked
|
||||
// `val.?.number` - a panic in Debug and undefined behaviour in
|
||||
// ReleaseFast - which is why this parser opts into
|
||||
// `srf_opts.user_edited`. Pinned because the option is easy to drop
|
||||
// and the failure is silent in the build zfin actually ships.
|
||||
const data =
|
||||
\\#!srfv1
|
||||
\\account::Sample Brokerage,tax_type::taxable,harvested::5000,harvested_date::2026-06-01
|
||||
\\account::Sample IRA,tax_type::traditional,audit_large_lot_threshold::25000
|
||||
;
|
||||
const allocator = std.testing.allocator;
|
||||
var am = try parseAccountsFile(allocator, data);
|
||||
defer am.deinit();
|
||||
|
||||
try std.testing.expectEqual(@as(usize, 2), am.entries.len);
|
||||
try std.testing.expectApproxEqAbs(@as(f64, 5000), am.entries[0].harvested.?, 0.001);
|
||||
try std.testing.expectApproxEqAbs(@as(f64, 25000), am.entries[1].audit_large_lot_threshold.?, 0.001);
|
||||
}
|
||||
|
||||
test "parseAccountsFile: direct_indexing default false, opt-in true" {
|
||||
const data =
|
||||
\\#!srfv1
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@ const builtin = @import("builtin");
|
|||
const log = std.log.scoped(.projections);
|
||||
const shiller = @import("../data/shiller.zig");
|
||||
const srf = @import("srf");
|
||||
const srf_opts = @import("../srf_opts.zig");
|
||||
const Date = @import("../Date.zig");
|
||||
|
||||
/// `log.warn` wrapper that no-ops under `zig build test`. Used for
|
||||
|
|
@ -762,7 +763,7 @@ pub fn parseProjectionsConfig(data: ?[]const u8) UserConfig {
|
|||
var annotation_count: u8 = 0;
|
||||
|
||||
while (it.next() catch null) |field_it| {
|
||||
const rec = field_it.to(SrfProjection, .{}) catch continue;
|
||||
const rec = field_it.to(SrfProjection, srf_opts.user_edited) catch continue;
|
||||
switch (rec) {
|
||||
.config => |c| {
|
||||
config.target_stock_pct = c.target_stock_pct orelse config.target_stock_pct;
|
||||
|
|
|
|||
189
src/cache/freshness.zig
vendored
189
src/cache/freshness.zig
vendored
|
|
@ -47,6 +47,12 @@
|
|||
//! means the series must be refetched is domain policy, and
|
||||
//! `store.zig`'s own `updateCandleMeta` doc records the same boundary
|
||||
//! for market-clock knowledge ("owned by the caller").
|
||||
//!
|
||||
//! Note which half owns the "has it actually gone ex yet?" filter:
|
||||
//! `newestCorporateAction` does, because that is a question about which
|
||||
//! actions are *relevant*, and relevance is selection. Putting it on the
|
||||
//! verdict instead is what let one forward-announced dividend mask every
|
||||
//! older unapplied one.
|
||||
|
||||
const std = @import("std");
|
||||
const Date = @import("../Date.zig");
|
||||
|
|
@ -638,16 +644,38 @@ test "collect-style exclusion: an excluded symbol produces no finding at all" {
|
|||
// two functions turn that into a verdict, split the same way as
|
||||
// `collect` / `scan`: one gathers from disk, one decides.
|
||||
|
||||
/// Newest corporate-action ex-date cached for a symbol, or null when no
|
||||
/// dividends or splits are on disk.
|
||||
/// Newest corporate-action ex-date cached for a symbol that has already
|
||||
/// gone ex as of `through`, or null when there is no such action.
|
||||
///
|
||||
/// Splits count as much as dividends here, and arguably more: raw
|
||||
/// `close` is not split-adjusted (hence `split.cumulativeSplitRatio`),
|
||||
/// so an unapplied 2:1 split leaves older `adj_close` values wrong by
|
||||
/// 50% rather than by a quarter's yield.
|
||||
///
|
||||
/// `through` is normally `CandleMeta.last_date`, the newest bar held.
|
||||
/// Actions dated after it are skipped, and that bound is load-bearing
|
||||
/// twice over:
|
||||
///
|
||||
/// - **Convergence.** A declared-but-not-yet-ex distribution is
|
||||
/// reflected in no provider's adjustment series, so treating it as
|
||||
/// something to catch up to would refetch on every pass forever.
|
||||
/// - **Not masking real work.** The bound belongs here, on the
|
||||
/// selection, rather than on the verdict. Bounding the verdict
|
||||
/// instead - "is the newest action of all still in the future? then
|
||||
/// nothing to do" - lets a single forward announcement hide every
|
||||
/// older unapplied action behind it. That shipped, and it left NKE
|
||||
/// permanently unable to restate: it announces roughly a quarter
|
||||
/// ahead, so its newest cached ex-date is essentially always in the
|
||||
/// future, which masked a distribution that had genuinely gone ex
|
||||
/// two months earlier.
|
||||
///
|
||||
/// Allocates only transiently - the returned `Date` borrows nothing.
|
||||
pub fn newestCorporateAction(allocator: std.mem.Allocator, store: *cache.Store, symbol: []const u8) ?Date {
|
||||
pub fn newestCorporateAction(
|
||||
allocator: std.mem.Allocator,
|
||||
store: *cache.Store,
|
||||
symbol: []const u8,
|
||||
through: Date,
|
||||
) ?Date {
|
||||
var newest: ?Date = null;
|
||||
|
||||
// `CacheResult` has no deinit - the caller owns `data`. Dividends
|
||||
|
|
@ -656,12 +684,14 @@ pub fn newestCorporateAction(allocator: std.mem.Allocator, store: *cache.Store,
|
|||
if (store.read(allocator, Dividend, symbol, null, .any)) |r| {
|
||||
defer Dividend.freeSlice(allocator, r.data);
|
||||
for (r.data) |d| {
|
||||
if (through.lessThan(d.ex_date)) continue;
|
||||
if (newest == null or newest.?.lessThan(d.ex_date)) newest = d.ex_date;
|
||||
}
|
||||
}
|
||||
if (store.read(allocator, Split, symbol, null, .any)) |r| {
|
||||
defer allocator.free(r.data);
|
||||
for (r.data) |sp| {
|
||||
if (through.lessThan(sp.date)) continue;
|
||||
if (newest == null or newest.?.lessThan(sp.date)) newest = sp.date;
|
||||
}
|
||||
}
|
||||
|
|
@ -674,58 +704,40 @@ pub fn newestCorporateAction(allocator: std.mem.Allocator, store: *cache.Store,
|
|||
///
|
||||
/// - `adj_basis`: `CandleMeta.adj_basis` - the bar date through which
|
||||
/// the cached `adj_close` values reflect corporate actions.
|
||||
/// - `last_bar`: `CandleMeta.last_date` - the newest bar held.
|
||||
/// - `newest_action`: from `newestCorporateAction`, or null for a
|
||||
/// symbol with no cached dividends or splits.
|
||||
/// - `newest_ex`: from `newestCorporateAction`, which has already
|
||||
/// discarded anything not yet ex. Null for a symbol with no
|
||||
/// applicable cached dividends or splits.
|
||||
///
|
||||
/// The `last_bar` bound is what keeps this convergent. A distribution
|
||||
/// announced with a future ex-date is not yet reflected in *any*
|
||||
/// provider's adjustment series, so treating it as stale would refetch
|
||||
/// on every pass and never settle.
|
||||
/// Takes no `last_bar`: the not-yet-ex bound lives in
|
||||
/// `newestCorporateAction` so that it cannot be applied to the wrong
|
||||
/// end of the comparison. See that function for what happened when it
|
||||
/// was applied here instead.
|
||||
///
|
||||
/// Pure: three dates in, bool out.
|
||||
pub fn adjustmentBasisStale(adj_basis: Date, last_bar: Date, newest_action: ?Date) bool {
|
||||
const newest = newest_action orelse return false;
|
||||
if (last_bar.lessThan(newest)) return false;
|
||||
/// Pure: two dates in, bool out.
|
||||
pub fn adjustmentBasisStale(adj_basis: Date, newest_ex: ?Date) bool {
|
||||
const newest = newest_ex orelse return false;
|
||||
return adj_basis.lessThan(newest);
|
||||
}
|
||||
|
||||
test "adjustmentBasisStale: no corporate action means nothing to restate" {
|
||||
test "adjustmentBasisStale: no applicable action means nothing to restate" {
|
||||
// A non-payer has no ex-date to exceed the basis, so it must never
|
||||
// escalate - not even with the epoch sentinel a legacy cache parses to.
|
||||
try testing.expect(!adjustmentBasisStale(Date.fromYmd(2026, 8, 14), Date.fromYmd(2026, 8, 14), null));
|
||||
try testing.expect(!adjustmentBasisStale(Date.epoch, Date.fromYmd(2026, 8, 14), null));
|
||||
try testing.expect(!adjustmentBasisStale(Date.fromYmd(2026, 8, 14), null));
|
||||
try testing.expect(!adjustmentBasisStale(Date.epoch, null));
|
||||
}
|
||||
|
||||
test "adjustmentBasisStale: action behind the basis is stale" {
|
||||
const last_bar = Date.fromYmd(2026, 8, 14);
|
||||
const ex = Date.fromYmd(2026, 6, 18);
|
||||
|
||||
// Basis at the newest bar already covers the ex-date.
|
||||
try testing.expect(!adjustmentBasisStale(last_bar, last_bar, ex));
|
||||
// Basis past the ex-date already covers it.
|
||||
try testing.expect(!adjustmentBasisStale(Date.fromYmd(2026, 8, 14), ex));
|
||||
// Basis exactly at the ex-date covers it too (not `lessThan`).
|
||||
try testing.expect(!adjustmentBasisStale(ex, last_bar, ex));
|
||||
try testing.expect(!adjustmentBasisStale(ex, ex));
|
||||
// Basis behind the ex-date: the bars in between were never marked down.
|
||||
try testing.expect(adjustmentBasisStale(Date.fromYmd(2026, 5, 1), last_bar, ex));
|
||||
try testing.expect(adjustmentBasisStale(Date.fromYmd(2026, 5, 1), ex));
|
||||
// The legacy-cache case: sentinel basis on a dividend payer. This is
|
||||
// the shape every pre-adj_basis cache lands in.
|
||||
try testing.expect(adjustmentBasisStale(Date.epoch, last_bar, ex));
|
||||
}
|
||||
|
||||
test "adjustmentBasisStale: a not-yet-ex action is not stale" {
|
||||
// Ex-date beyond the newest bar we hold. No provider has applied it
|
||||
// either, so escalating would never converge.
|
||||
try testing.expect(!adjustmentBasisStale(
|
||||
Date.fromYmd(2026, 5, 1),
|
||||
Date.fromYmd(2026, 8, 14),
|
||||
Date.fromYmd(2026, 9, 17),
|
||||
));
|
||||
// Boundary: an ex-date exactly on the newest bar IS covered.
|
||||
try testing.expect(adjustmentBasisStale(
|
||||
Date.fromYmd(2026, 5, 1),
|
||||
Date.fromYmd(2026, 8, 14),
|
||||
Date.fromYmd(2026, 8, 14),
|
||||
));
|
||||
try testing.expect(adjustmentBasisStale(Date.epoch, ex));
|
||||
}
|
||||
|
||||
test "newestCorporateAction: takes the max across dividends and splits" {
|
||||
|
|
@ -737,23 +749,114 @@ test "newestCorporateAction: takes the max across dividends and splits" {
|
|||
defer a.free(dir_path);
|
||||
|
||||
var store = cache.Store.init(io, a, dir_path);
|
||||
const through = Date.fromYmd(2026, 8, 14);
|
||||
|
||||
// Nothing cached at all.
|
||||
try testing.expect(newestCorporateAction(a, &store, "SMPL") == null);
|
||||
try testing.expect(newestCorporateAction(a, &store, "SMPL", through) == null);
|
||||
|
||||
var divs = [_]Dividend{
|
||||
.{ .ex_date = Date.fromYmd(2026, 3, 20), .amount = 1.79 },
|
||||
.{ .ex_date = Date.fromYmd(2026, 6, 18), .amount = 1.90 },
|
||||
};
|
||||
store.write(Dividend, "SMPL", divs[0..], .{ .seconds = cache.Ttl.dividends });
|
||||
try testing.expect(newestCorporateAction(a, &store, "SMPL").?.eql(Date.fromYmd(2026, 6, 18)));
|
||||
try testing.expect(newestCorporateAction(a, &store, "SMPL", through).?.eql(Date.fromYmd(2026, 6, 18)));
|
||||
|
||||
// A later split must win over the later dividend.
|
||||
var splits = [_]Split{.{ .date = Date.fromYmd(2026, 7, 1), .numerator = 2, .denominator = 1 }};
|
||||
store.write(Split, "SMPL", splits[0..], .{ .seconds = cache.Ttl.splits });
|
||||
try testing.expect(newestCorporateAction(a, &store, "SMPL").?.eql(Date.fromYmd(2026, 7, 1)));
|
||||
try testing.expect(newestCorporateAction(a, &store, "SMPL", through).?.eql(Date.fromYmd(2026, 7, 1)));
|
||||
|
||||
// Splits alone (different symbol) still resolve.
|
||||
store.write(Split, "SMPLB", splits[0..], .{ .seconds = cache.Ttl.splits });
|
||||
try testing.expect(newestCorporateAction(a, &store, "SMPLB").?.eql(Date.fromYmd(2026, 7, 1)));
|
||||
try testing.expect(newestCorporateAction(a, &store, "SMPLB", through).?.eql(Date.fromYmd(2026, 7, 1)));
|
||||
}
|
||||
|
||||
test "newestCorporateAction: a forward-announced action does not mask an older unapplied one" {
|
||||
// The regression. NKE announces roughly a quarter ahead, so its
|
||||
// newest cached ex-date is essentially always in the future. Bounding
|
||||
// the *verdict* on "is the newest action still in the future?" made
|
||||
// that one announcement hide a distribution that had gone ex two
|
||||
// months earlier, and NKE could never restate. The bound belongs on
|
||||
// the selection, so the announcement is skipped and the older
|
||||
// already-ex action is still found.
|
||||
const io = testing.io;
|
||||
const a = testing.allocator;
|
||||
var tmp = testing.tmpDir(.{});
|
||||
defer tmp.cleanup();
|
||||
const dir_path = try tmp.dir.realPathFileAlloc(io, ".", a);
|
||||
defer a.free(dir_path);
|
||||
|
||||
var store = cache.Store.init(io, a, dir_path);
|
||||
const last_bar = Date.fromYmd(2026, 8, 18);
|
||||
|
||||
// NKE's exact shape: applied through 2026-03-02, 2026-06-01 went ex
|
||||
// and was never applied, 2026-09-01 declared but not yet ex.
|
||||
var divs = [_]Dividend{
|
||||
.{ .ex_date = Date.fromYmd(2026, 9, 1), .amount = 0.41 },
|
||||
.{ .ex_date = Date.fromYmd(2026, 6, 1), .amount = 0.41 },
|
||||
.{ .ex_date = Date.fromYmd(2026, 3, 2), .amount = 0.41 },
|
||||
};
|
||||
store.write(Dividend, "SMPL", divs[0..], .{ .seconds = cache.Ttl.dividends });
|
||||
|
||||
// The future announcement is skipped; the newest already-ex wins.
|
||||
const newest = newestCorporateAction(a, &store, "SMPL", last_bar);
|
||||
try testing.expect(newest != null);
|
||||
try testing.expect(newest.?.eql(Date.fromYmd(2026, 6, 1)));
|
||||
|
||||
// And the verdict is therefore "stale" for a legacy sentinel basis.
|
||||
try testing.expect(adjustmentBasisStale(Date.epoch, newest));
|
||||
// ...and for a basis covering 2026-03-02 but not 2026-06-01.
|
||||
try testing.expect(adjustmentBasisStale(Date.fromYmd(2026, 4, 1), newest));
|
||||
// ...but not once the basis has caught up.
|
||||
try testing.expect(!adjustmentBasisStale(Date.fromYmd(2026, 8, 17), newest));
|
||||
}
|
||||
|
||||
test "newestCorporateAction: a declared-but-not-effective split does not mask either" {
|
||||
const io = testing.io;
|
||||
const a = testing.allocator;
|
||||
var tmp = testing.tmpDir(.{});
|
||||
defer tmp.cleanup();
|
||||
const dir_path = try tmp.dir.realPathFileAlloc(io, ".", a);
|
||||
defer a.free(dir_path);
|
||||
|
||||
var store = cache.Store.init(io, a, dir_path);
|
||||
const last_bar = Date.fromYmd(2026, 8, 18);
|
||||
|
||||
var splits = [_]Split{
|
||||
.{ .date = Date.fromYmd(2026, 10, 1), .numerator = 2, .denominator = 1 },
|
||||
.{ .date = Date.fromYmd(2026, 5, 1), .numerator = 3, .denominator = 1 },
|
||||
};
|
||||
store.write(Split, "SMPL", splits[0..], .{ .seconds = cache.Ttl.splits });
|
||||
|
||||
const newest = newestCorporateAction(a, &store, "SMPL", last_bar);
|
||||
try testing.expect(newest.?.eql(Date.fromYmd(2026, 5, 1)));
|
||||
try testing.expect(adjustmentBasisStale(Date.epoch, newest));
|
||||
}
|
||||
|
||||
test "newestCorporateAction: convergence - an only-future action is not actionable" {
|
||||
// The property the bad bound was reaching for, preserved. A
|
||||
// declared-but-not-yet-ex action is in no provider's adjustment
|
||||
// series, so treating it as something to catch up to would refetch
|
||||
// the full history on every pass, forever.
|
||||
const io = testing.io;
|
||||
const a = testing.allocator;
|
||||
var tmp = testing.tmpDir(.{});
|
||||
defer tmp.cleanup();
|
||||
const dir_path = try tmp.dir.realPathFileAlloc(io, ".", a);
|
||||
defer a.free(dir_path);
|
||||
|
||||
var store = cache.Store.init(io, a, dir_path);
|
||||
|
||||
var divs = [_]Dividend{.{ .ex_date = Date.fromYmd(2026, 9, 17), .amount = 1.9 }};
|
||||
store.write(Dividend, "SMPL", divs[0..], .{ .seconds = cache.Ttl.dividends });
|
||||
|
||||
const last_bar = Date.fromYmd(2026, 8, 18);
|
||||
try testing.expect(newestCorporateAction(a, &store, "SMPL", last_bar) == null);
|
||||
try testing.expect(!adjustmentBasisStale(Date.epoch, newestCorporateAction(a, &store, "SMPL", last_bar)));
|
||||
|
||||
// Boundary: once it goes ex - `through` reaching the ex-date - it
|
||||
// becomes actionable on that very bar.
|
||||
const on_ex = Date.fromYmd(2026, 9, 17);
|
||||
try testing.expect(newestCorporateAction(a, &store, "SMPL", on_ex).?.eql(on_ex));
|
||||
try testing.expect(adjustmentBasisStale(Date.epoch, newestCorporateAction(a, &store, "SMPL", on_ex)));
|
||||
}
|
||||
|
|
|
|||
27
src/cache/store.zig
vendored
27
src/cache/store.zig
vendored
|
|
@ -1,6 +1,7 @@
|
|||
const std = @import("std");
|
||||
const log = std.log.scoped(.cache);
|
||||
const srf = @import("srf");
|
||||
const srf_opts = @import("../srf_opts.zig");
|
||||
const atomic = @import("../atomic.zig");
|
||||
const version = @import("../version.zig");
|
||||
const Date = @import("../Date.zig");
|
||||
|
|
@ -1343,7 +1344,7 @@ pub const Store = struct {
|
|||
|
||||
const created = it.created orelse std.Io.Timestamp.now(self.io, .real).toSeconds();
|
||||
const fields = (it.next() catch return null) orelse return null;
|
||||
const meta = fields.to(CandleMeta, .{}) catch return null;
|
||||
const meta = fields.to(CandleMeta, srf_opts.machine_written) catch return null;
|
||||
return .{ .meta = meta, .created = created };
|
||||
}
|
||||
|
||||
|
|
@ -1889,7 +1890,7 @@ pub const Store = struct {
|
|||
}
|
||||
|
||||
// Per-record coercion. Most types use SRF's generalized
|
||||
// `fields.to(T, .{})` - correct for any struct shape but
|
||||
// `fields.to(T, ...)` - correct for any struct shape but
|
||||
// pays a per-field abstraction cost (coerce() boundary,
|
||||
// found-bitmap bookkeeping, inline-for dispatch chain).
|
||||
//
|
||||
|
|
@ -1904,7 +1905,7 @@ pub const Store = struct {
|
|||
var item: T = if (comptime T == Candle)
|
||||
coerceCandleSpecialized(fields) catch continue
|
||||
else
|
||||
fields.to(T, .{}) catch continue;
|
||||
fields.to(T, srf_opts.machine_written) catch continue;
|
||||
if (comptime postProcess) |pp| {
|
||||
pp(&item, allocator) catch {
|
||||
if (comptime @hasDecl(T, "deinit")) item.deinit(allocator);
|
||||
|
|
@ -1968,7 +1969,7 @@ pub const Store = struct {
|
|||
defer it.deinit();
|
||||
|
||||
const fields = (try it.next()) orelse return error.InvalidData;
|
||||
return fields.to(CandleMeta, .{}) catch error.InvalidData;
|
||||
return fields.to(CandleMeta, srf_opts.machine_written) catch error.InvalidData;
|
||||
}
|
||||
|
||||
// ── Private serialization: options (bespoke) ─────────────────
|
||||
|
|
@ -2038,7 +2039,7 @@ pub const Store = struct {
|
|||
}
|
||||
|
||||
while (try it.next()) |fields| {
|
||||
const opt_rec = fields.to(OptionsRecord, .{}) catch continue;
|
||||
const opt_rec = fields.to(OptionsRecord, srf_opts.machine_written) catch continue;
|
||||
switch (opt_rec) {
|
||||
.chain => |ch| {
|
||||
const idx = chains.items.len;
|
||||
|
|
@ -2111,18 +2112,10 @@ pub fn deserializePortfolio(allocator: std.mem.Allocator, data: []const u8) !Por
|
|||
var skipped: usize = 0;
|
||||
while (try it.next()) |fields| {
|
||||
const line = it.state.line;
|
||||
// `strings_to_numbers` because these are HUMAN-EDITED files, which is
|
||||
// exactly the case srf's default strict coercion is not for - its own
|
||||
// doc says "if you want to use this for human-edited files, turn this
|
||||
// on". Strict mode assumes the writer was a machine, so a numeric
|
||||
// field spelled with a string separator (`close_price::200.00` instead
|
||||
// of `close_price:num:200.00`) reaches an unchecked `val.?.number` and
|
||||
// takes the whole process down. One such typo was enough to panic every
|
||||
// `zfin portfolio` run.
|
||||
//
|
||||
// The `catch` below still handles genuinely unparseable values; this
|
||||
// only stops a hand-typed separator from being fatal.
|
||||
var lot = fields.to(Lot, .{ .strings_to_numbers = true }) catch {
|
||||
// `user_edited` coercion: see `srf_opts.zig` for why hand-edited
|
||||
// files get different options from cache files. The `catch`
|
||||
// below still handles genuinely unparseable values.
|
||||
var lot = fields.to(Lot, srf_opts.user_edited) catch {
|
||||
std.log.warn("portfolio: could not parse record at line {d}", .{line});
|
||||
skipped += 1;
|
||||
continue;
|
||||
|
|
|
|||
|
|
@ -118,13 +118,17 @@ pub fn run(ctx: *framework.RunCtx, parsed: ParsedArgs) !void {
|
|||
const verbose = parsed.verbose;
|
||||
const stale_days = parsed.stale_days;
|
||||
|
||||
// Flagless mode: run portfolio hygiene check (single-file
|
||||
// semantics - git blame, commit SHAs, etc.). Resolve paths
|
||||
// just to find the anchor; we don't need the merged view.
|
||||
// Flagless mode: run portfolio hygiene check. Mostly single-file
|
||||
// semantics (git blame, commit SHAs, etc.), so resolve the anchor -
|
||||
// but the large-lot check inside diffs portfolio content and needs
|
||||
// the merged view, so hand it the whole glob too.
|
||||
if (fidelity_csv == null and schwab_csv == null and !schwab_summary) {
|
||||
const pf = ctx.resolvePortfolioPath();
|
||||
defer pf.deinit(allocator);
|
||||
return hygiene.runHygieneCheck(io, allocator, ctx.environ_map, svc, pf.path, stale_days, verbose, as_of, now_s, color, ctx.globals.refresh_policy, out);
|
||||
var all = ctx.resolvePortfolioPaths() catch null;
|
||||
defer if (all) |*p| p.deinit();
|
||||
const paths: []const []const u8 = if (all) |p| p.paths else &.{pf.path};
|
||||
return hygiene.runHygieneCheck(io, allocator, ctx.environ_map, svc, pf.path, paths, stale_days, verbose, as_of, now_s, color, ctx.globals.refresh_policy, out);
|
||||
}
|
||||
|
||||
// Reconciliation modes (--fidelity / --schwab / --schwab-summary):
|
||||
|
|
|
|||
|
|
@ -729,6 +729,11 @@ pub fn runHygieneCheck(
|
|||
env: *const std.process.Environ.Map,
|
||||
svc: *zfin.DataService,
|
||||
portfolio_path: []const u8,
|
||||
/// Every file in the `portfolio*.srf` glob. Only the large-lot
|
||||
/// check needs this: it diffs portfolio CONTENT and so must see
|
||||
/// the merged view. The rest of the hygiene report is deliberately
|
||||
/// single-file (git blame, commit SHAs) and uses `portfolio_path`.
|
||||
portfolio_paths: []const []const u8,
|
||||
stale_days: u32,
|
||||
verbose: bool,
|
||||
as_of: Date,
|
||||
|
|
@ -1226,7 +1231,7 @@ pub fn runHygieneCheck(
|
|||
// (not in a git repo). Threshold is per-account: an account's
|
||||
// `audit_large_lot_threshold` in accounts.srf wins, otherwise the
|
||||
// filter's built-in default applies.
|
||||
if (contributions.findUnmatchedLargeLots(io, allocator, env, svc, portfolio_path, &account_map, as_of, color, refresh)) |found| {
|
||||
if (contributions.findUnmatchedLargeLots(io, allocator, env, svc, portfolio_paths, &account_map, as_of, color, refresh)) |found| {
|
||||
var found_mut = found;
|
||||
defer found_mut.deinit();
|
||||
|
||||
|
|
@ -2182,7 +2187,7 @@ test "runHygieneCheck: Section 7 flags an un-opted-in symbol's split, not an opt
|
|||
var aw: std.Io.Writer.Allocating = .init(allocator);
|
||||
defer aw.deinit();
|
||||
|
||||
try runHygieneCheck(io, allocator, &env, &svc, pf_path, 3, false, zfin.Date.fromYmd(2026, 1, 1), 1_767_225_600, false, .never, &aw.writer);
|
||||
try runHygieneCheck(io, allocator, &env, &svc, pf_path, &.{pf_path}, 3, false, zfin.Date.fromYmd(2026, 1, 1), 1_767_225_600, false, .never, &aw.writer);
|
||||
|
||||
const output = aw.written();
|
||||
try std.testing.expect(std.mem.indexOf(u8, output, "Portfolio hygiene") != null);
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ const std = @import("std");
|
|||
const builtin = @import("builtin");
|
||||
const zfin = @import("../root.zig");
|
||||
const srf = @import("srf");
|
||||
const srf_opts = @import("../srf_opts.zig");
|
||||
const history = @import("../history.zig");
|
||||
const git = @import("../git.zig");
|
||||
const framework = @import("framework.zig");
|
||||
|
|
@ -1007,7 +1008,7 @@ pub fn loadWatchlist(io: std.Io, allocator: std.mem.Allocator, path: []const u8)
|
|||
|
||||
var syms: std.ArrayList([]const u8) = .empty;
|
||||
while (it.next() catch null) |fields| {
|
||||
const entry = fields.to(WatchEntry, .{}) catch continue;
|
||||
const entry = fields.to(WatchEntry, srf_opts.user_edited) catch continue;
|
||||
const duped = allocator.dupe(u8, entry.symbol) catch continue;
|
||||
syms.append(allocator, duped) catch {
|
||||
allocator.free(duped);
|
||||
|
|
|
|||
|
|
@ -290,6 +290,16 @@ pub fn run(ctx: *framework.RunCtx, parsed: ParsedArgs) !void {
|
|||
defer pf.deinit(allocator);
|
||||
const portfolio_path = pf.path;
|
||||
|
||||
// Attribution diffs portfolio CONTENT, so it needs the whole
|
||||
// `portfolio*.srf` glob rather than the anchor alone - otherwise a
|
||||
// sold lot archived into a sibling file reads as a bare
|
||||
// disappearance and its proceeds can't offset the repurchase.
|
||||
// Resolution failure is non-fatal: the attribution line is
|
||||
// optional, so fall back to the anchor on its own.
|
||||
var attr_pf = ctx.resolvePortfolioPaths() catch null;
|
||||
defer if (attr_pf) |*p| p.deinit();
|
||||
const attr_paths: []const []const u8 = if (attr_pf) |p| p.paths else &.{portfolio_path};
|
||||
|
||||
const with_projections = parsed.with_projections;
|
||||
const events_enabled = parsed.events_enabled;
|
||||
const snapshot_after_live = parsed.after_is_live and parsed.snapshot_after == null;
|
||||
|
|
@ -471,7 +481,7 @@ pub fn run(ctx: *framework.RunCtx, parsed: ParsedArgs) !void {
|
|||
// Attribution uses the resolved CommitSpecs so --commit-*
|
||||
// overrides + date fallbacks share one classifier. The caller
|
||||
// adapts dates to `CommitSpec.date_at_or_before` upstream.
|
||||
const attribution = contributions.computeAttributionSpec(io, allocator, ctx.environ_map, svc, portfolio_path, attr_before, attr_after_opt, as_of, color, ctx.globals.refresh_policy);
|
||||
const attribution = contributions.computeAttributionSpec(io, allocator, ctx.environ_map, svc, attr_paths, attr_before, attr_after_opt, as_of, color, ctx.globals.refresh_policy);
|
||||
|
||||
try renderFromParts(out, color, allocator, .{
|
||||
.then_date = then_date,
|
||||
|
|
@ -488,7 +498,7 @@ pub fn run(ctx: *framework.RunCtx, parsed: ParsedArgs) !void {
|
|||
var now_side = try compare_core.loadSnapshotSide(io, allocator, hist_dir, now_date);
|
||||
defer now_side.deinit(allocator);
|
||||
|
||||
const attribution = contributions.computeAttributionSpec(io, allocator, ctx.environ_map, svc, portfolio_path, attr_before, attr_after_opt, as_of, color, ctx.globals.refresh_policy);
|
||||
const attribution = contributions.computeAttributionSpec(io, allocator, ctx.environ_map, svc, attr_paths, attr_before, attr_after_opt, as_of, color, ctx.globals.refresh_policy);
|
||||
|
||||
try renderFromParts(out, color, allocator, .{
|
||||
.then_date = then_date,
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -254,8 +254,8 @@ pub fn run(ctx: *framework.RunCtx, parsed: ParsedArgs) !void {
|
|||
// bars behind it - total returns then read low by roughly the
|
||||
// missed yield. Report it here because the symptom (a slightly
|
||||
// low 1Y total return) is otherwise invisible.
|
||||
if (freshness.newestCorporateAction(arena, &store, symbol)) |newest_action| {
|
||||
if (freshness.adjustmentBasisStale(m.meta.adj_basis, m.meta.last_date, newest_action)) {
|
||||
if (freshness.newestCorporateAction(arena, &store, symbol, m.meta.last_date)) |newest_action| {
|
||||
if (freshness.adjustmentBasisStale(m.meta.adj_basis, newest_action)) {
|
||||
try out.print(
|
||||
"adj basis {f} - STALE, {f} went ex behind it; total returns read low until restated\n",
|
||||
.{ m.meta.adj_basis, newest_action },
|
||||
|
|
|
|||
|
|
@ -66,6 +66,7 @@
|
|||
|
||||
const std = @import("std");
|
||||
const srf = @import("srf");
|
||||
const srf_opts = @import("../srf_opts.zig");
|
||||
const Date = @import("../Date.zig");
|
||||
const atomic = @import("../atomic.zig");
|
||||
|
||||
|
|
@ -214,7 +215,7 @@ pub fn parse(allocator: std.mem.Allocator, data: []const u8) !Journal {
|
|||
defer it.deinit();
|
||||
|
||||
while (try it.next()) |fields| {
|
||||
const rec = try fields.to(JournalRecord, .{});
|
||||
const rec = try fields.to(JournalRecord, srf_opts.user_edited);
|
||||
switch (rec) {
|
||||
.acknowledgment => |a| {
|
||||
try entries.append(allocator, .{
|
||||
|
|
|
|||
|
|
@ -35,6 +35,7 @@
|
|||
|
||||
const std = @import("std");
|
||||
const srf = @import("srf");
|
||||
const srf_opts = @import("../srf_opts.zig");
|
||||
const Date = @import("../Date.zig");
|
||||
|
||||
// ── Types ────────────────────────────────────────────────────
|
||||
|
|
@ -176,7 +177,7 @@ pub fn parseImportedValues(
|
|||
errdefer points.deinit(allocator);
|
||||
|
||||
while (it.next() catch return error.InvalidSrf) |fields| {
|
||||
const point = fields.to(HistoryPoint, .{}) catch return error.InvalidSrf;
|
||||
const point = fields.to(HistoryPoint, srf_opts.user_edited) catch return error.InvalidSrf;
|
||||
try points.append(allocator, point);
|
||||
}
|
||||
|
||||
|
|
|
|||
81
src/git.zig
81
src/git.zig
|
|
@ -207,17 +207,29 @@ pub fn findRepo(io: std.Io, allocator: std.mem.Allocator, env: *const std.proces
|
|||
const root = try allocator.dupe(u8, root_raw);
|
||||
errdefer allocator.free(root);
|
||||
|
||||
// Relative path from root to the file. If `abs_path` starts with the
|
||||
// repo root (the common case), trim the prefix; otherwise fall back to
|
||||
// just the basename (extremely unusual - repo root disagrees with
|
||||
// path).
|
||||
const rel = try relPathInRoot(allocator, root, abs_path);
|
||||
|
||||
return .{ .root = root, .rel_path = rel };
|
||||
}
|
||||
|
||||
/// Relative path from `root` to `abs_path`, as git pathspecs want it.
|
||||
///
|
||||
/// If `abs_path` starts with the repo root (the common case), trim the
|
||||
/// prefix; otherwise fall back to just the basename (extremely unusual
|
||||
/// - repo root disagrees with path). Caller owns the result.
|
||||
pub fn relPathInRoot(allocator: std.mem.Allocator, root: []const u8, abs_path: []const u8) ![]const u8 {
|
||||
const rel_raw = if (std.mem.startsWith(u8, abs_path, root) and abs_path.len > root.len)
|
||||
std.mem.trimStart(u8, abs_path[root.len..], "/")
|
||||
else
|
||||
std.fs.path.basename(abs_path);
|
||||
const rel = try allocator.dupe(u8, rel_raw);
|
||||
return allocator.dupe(u8, rel_raw);
|
||||
}
|
||||
|
||||
return .{ .root = root, .rel_path = rel };
|
||||
/// `relPathInRoot` against an already-discovered repo. Convenience for
|
||||
/// callers holding a `RepoInfo` that need rel-paths for the anchor's
|
||||
/// sibling files (e.g. every file in a `portfolio*.srf` glob).
|
||||
pub fn relPathIn(allocator: std.mem.Allocator, repo: RepoInfo, abs_path: []const u8) ![]const u8 {
|
||||
return relPathInRoot(allocator, repo.root, abs_path);
|
||||
}
|
||||
|
||||
/// Report the tracked/untracked/modified status of `rel_path` relative to
|
||||
|
|
@ -396,22 +408,26 @@ pub fn lastCommitTimestampForPath(
|
|||
return std.fmt.parseInt(i64, trimmed, 10) catch return null;
|
||||
}
|
||||
|
||||
/// Return the SHA of the most recent commit that touched `rel_path` at
|
||||
/// or before `date_iso` (YYYY-MM-DD, inclusive end-of-day semantics via
|
||||
/// `git log --until`).
|
||||
/// Return the SHA of the most recent commit that touched any of
|
||||
/// `rel_paths` at or before `date_iso` (YYYY-MM-DD, inclusive
|
||||
/// end-of-day semantics via `git log --until`).
|
||||
///
|
||||
/// Returns null if no commit before `date_iso` touched `rel_path`.
|
||||
/// Returns null if no commit before `date_iso` touched any of them.
|
||||
/// Caller owns the returned string.
|
||||
///
|
||||
/// Used by `zfin contributions --since <DATE>` / `--until <DATE>` to
|
||||
/// resolve a date to the last commit that stamped a given snapshot of
|
||||
/// the portfolio file.
|
||||
/// the portfolio. Takes a slice rather than one path because the
|
||||
/// portfolio is a `portfolio*.srf` glob: resolving against only the
|
||||
/// first file would pick an older commit whenever the commit in range
|
||||
/// touched just a sibling (a sold lot moved into
|
||||
/// `portfolio_closed.srf`), silently widening the window.
|
||||
pub fn commitAtOrBeforeDate(
|
||||
io: std.Io,
|
||||
allocator: std.mem.Allocator,
|
||||
env: *const std.process.Environ.Map,
|
||||
root: []const u8,
|
||||
rel_path: []const u8,
|
||||
rel_paths: []const []const u8,
|
||||
date_iso: []const u8,
|
||||
) Error!?[]const u8 {
|
||||
// `git log --until=DATE` with a bare YYYY-MM-DD uses the *current
|
||||
|
|
@ -426,11 +442,22 @@ pub fn commitAtOrBeforeDate(
|
|||
const until_arg = try std.fmt.allocPrint(allocator, "--until={s} 23:59:59", .{date_iso});
|
||||
defer allocator.free(until_arg);
|
||||
|
||||
const result = runGit(io, allocator, env, &.{
|
||||
// `git log -1 -- p1 p2 ...` already returns the newest commit
|
||||
// touching ANY of the pathspecs, so a multi-file portfolio glob
|
||||
// resolves a date correctly even when the only commit in range
|
||||
// touched just one of the files (e.g. a sold lot moved into
|
||||
// `portfolio_closed.srf`). Argv is built dynamically because the
|
||||
// path count is not known at comptime.
|
||||
var argv: std.ArrayList([]const u8) = .empty;
|
||||
defer argv.deinit(allocator);
|
||||
try argv.appendSlice(allocator, &.{
|
||||
"git", "-C", root,
|
||||
"log", "-1", "--format=%H",
|
||||
until_arg, "--", rel_path,
|
||||
}, .limited(64 * 1024)) catch return error.GitUnavailable;
|
||||
until_arg, "--",
|
||||
});
|
||||
try argv.appendSlice(allocator, rel_paths);
|
||||
|
||||
const result = runGit(io, allocator, env, argv.items, .limited(64 * 1024)) catch return error.GitUnavailable;
|
||||
defer allocator.free(result.stdout);
|
||||
defer allocator.free(result.stderr);
|
||||
|
||||
|
|
@ -579,6 +606,7 @@ pub fn resolveCommitRangeSpec(
|
|||
arena: std.mem.Allocator,
|
||||
env: *const std.process.Environ.Map,
|
||||
repo: RepoInfo,
|
||||
rel_paths: []const []const u8,
|
||||
before: ?CommitSpec,
|
||||
after: ?CommitSpec,
|
||||
dirty: bool,
|
||||
|
|
@ -590,7 +618,7 @@ pub fn resolveCommitRangeSpec(
|
|||
|
||||
// Resolve each endpoint independently.
|
||||
const before_rev: []const u8 = if (before) |b|
|
||||
try resolveSpec(io, arena, env, repo, b)
|
||||
try resolveSpec(io, arena, env, repo, rel_paths, b)
|
||||
else if (dirty)
|
||||
"HEAD"
|
||||
else
|
||||
|
|
@ -599,7 +627,7 @@ pub fn resolveCommitRangeSpec(
|
|||
const after_rev: ?[]const u8 = if (after) |a|
|
||||
(switch (a) {
|
||||
.working_copy => null,
|
||||
else => try resolveSpec(io, arena, env, repo, a),
|
||||
else => try resolveSpec(io, arena, env, repo, rel_paths, a),
|
||||
})
|
||||
else if (dirty)
|
||||
null
|
||||
|
|
@ -612,14 +640,14 @@ pub fn resolveCommitRangeSpec(
|
|||
/// Resolve one non-working `CommitSpec` to a string git can consume.
|
||||
/// Caller handles the `.working_copy` case separately (it's not a
|
||||
/// git ref).
|
||||
fn resolveSpec(io: std.Io, arena: std.mem.Allocator, env: *const std.process.Environ.Map, repo: RepoInfo, spec: CommitSpec) Error![]const u8 {
|
||||
fn resolveSpec(io: std.Io, arena: std.mem.Allocator, env: *const std.process.Environ.Map, repo: RepoInfo, rel_paths: []const []const u8, spec: CommitSpec) Error![]const u8 {
|
||||
return switch (spec) {
|
||||
.git_ref => |r| r,
|
||||
.date_at_or_before => |d| blk: {
|
||||
var buf: [10]u8 = undefined;
|
||||
// SAFETY: 10-byte buffer is exactly the size of "YYYY-MM-DD".
|
||||
const date_str = std.fmt.bufPrint(&buf, "{f}", .{d}) catch buf[0..];
|
||||
const sha = (try commitAtOrBeforeDate(io, arena, env, repo.root, repo.rel_path, date_str)) orelse
|
||||
const sha = (try commitAtOrBeforeDate(io, arena, env, repo.root, rel_paths, date_str)) orelse
|
||||
return error.NoCommitAtOrBefore;
|
||||
break :blk sha;
|
||||
},
|
||||
|
|
@ -639,6 +667,7 @@ pub fn resolveCommitRange(
|
|||
arena: std.mem.Allocator,
|
||||
env: *const std.process.Environ.Map,
|
||||
repo: RepoInfo,
|
||||
rel_paths: []const []const u8,
|
||||
since: ?Date,
|
||||
until: ?Date,
|
||||
dirty: bool,
|
||||
|
|
@ -646,7 +675,7 @@ pub fn resolveCommitRange(
|
|||
std.debug.assert(!(since == null and until != null));
|
||||
const before: ?CommitSpec = if (since) |d| .{ .date_at_or_before = d } else null;
|
||||
const after: ?CommitSpec = if (until) |d| .{ .date_at_or_before = d } else null;
|
||||
return resolveCommitRangeSpec(io, arena, env, repo, before, after, dirty);
|
||||
return resolveCommitRangeSpec(io, arena, env, repo, rel_paths, before, after, dirty);
|
||||
}
|
||||
|
||||
// ── Tests ────────────────────────────────────────────────────
|
||||
|
|
@ -725,7 +754,7 @@ test "commitAtOrBeforeDate returns a SHA for a past date" {
|
|||
|
||||
// Any date well after the repo's creation - commitAtOrBeforeDate
|
||||
// should find the most recent commit touching build.zig.
|
||||
const sha_opt = commitAtOrBeforeDate(std.testing.io, allocator, &env, info.root, info.rel_path, "2099-01-01") catch return;
|
||||
const sha_opt = commitAtOrBeforeDate(std.testing.io, allocator, &env, info.root, &.{info.rel_path}, "2099-01-01") catch return;
|
||||
try std.testing.expect(sha_opt != null);
|
||||
const sha = sha_opt.?;
|
||||
defer allocator.free(sha);
|
||||
|
|
@ -744,7 +773,7 @@ test "commitAtOrBeforeDate returns null for date before repo existed" {
|
|||
defer allocator.free(info.rel_path);
|
||||
|
||||
// Pre-git - before any sensible project history.
|
||||
const sha_opt = commitAtOrBeforeDate(std.testing.io, allocator, &env, info.root, info.rel_path, "1970-01-02") catch return;
|
||||
const sha_opt = commitAtOrBeforeDate(std.testing.io, allocator, &env, info.root, &.{info.rel_path}, "1970-01-02") catch return;
|
||||
try std.testing.expect(sha_opt == null);
|
||||
}
|
||||
|
||||
|
|
@ -773,7 +802,7 @@ test "commitAtOrBeforeDate: --until=DATE covers end of day, not current time-of-
|
|||
|
||||
// Future-dated cutoff - should always return the tip of history
|
||||
// regardless of current wall-clock time.
|
||||
const sha_opt = commitAtOrBeforeDate(std.testing.io, allocator, &env, info.root, info.rel_path, "2099-01-01") catch return;
|
||||
const sha_opt = commitAtOrBeforeDate(std.testing.io, allocator, &env, info.root, &.{info.rel_path}, "2099-01-01") catch return;
|
||||
try std.testing.expect(sha_opt != null);
|
||||
if (sha_opt) |s| allocator.free(s);
|
||||
}
|
||||
|
|
@ -817,7 +846,7 @@ test "resolveCommitRange: legacy clean -> HEAD~1..HEAD" {
|
|||
defer env.deinit();
|
||||
const repo: RepoInfo = .{ .root = "/tmp", .rel_path = "portfolio.srf" };
|
||||
|
||||
const range = try resolveCommitRange(std.testing.io, arena_state.allocator(), &env, repo, null, null, false);
|
||||
const range = try resolveCommitRange(std.testing.io, arena_state.allocator(), &env, repo, &.{repo.rel_path}, null, null, false);
|
||||
try std.testing.expectEqualStrings("HEAD~1", range.before_rev);
|
||||
try std.testing.expectEqualStrings("HEAD", range.after_rev.?);
|
||||
}
|
||||
|
|
@ -829,7 +858,7 @@ test "resolveCommitRange: legacy dirty -> HEAD..working-copy" {
|
|||
defer env.deinit();
|
||||
const repo: RepoInfo = .{ .root = "/tmp", .rel_path = "portfolio.srf" };
|
||||
|
||||
const range = try resolveCommitRange(std.testing.io, arena_state.allocator(), &env, repo, null, null, true);
|
||||
const range = try resolveCommitRange(std.testing.io, arena_state.allocator(), &env, repo, &.{repo.rel_path}, null, null, true);
|
||||
try std.testing.expectEqualStrings("HEAD", range.before_rev);
|
||||
try std.testing.expect(range.after_rev == null);
|
||||
}
|
||||
|
|
@ -851,6 +880,7 @@ test "resolveCommitRange: --since resolves to SHA..HEAD for clean tree" {
|
|||
arena_state.allocator(),
|
||||
&env,
|
||||
info,
|
||||
&.{info.rel_path},
|
||||
Date.fromYmd(2099, 1, 1),
|
||||
null,
|
||||
false,
|
||||
|
|
@ -876,6 +906,7 @@ test "resolveCommitRange: --since with no earlier commit -> NoCommitAtOrBefore"
|
|||
arena_state.allocator(),
|
||||
&env,
|
||||
info,
|
||||
&.{info.rel_path},
|
||||
Date.fromYmd(1970, 1, 2),
|
||||
null,
|
||||
false,
|
||||
|
|
|
|||
|
|
@ -32,6 +32,7 @@
|
|||
const std = @import("std");
|
||||
const builtin = @import("builtin");
|
||||
const srf = @import("srf");
|
||||
const srf_opts = @import("srf_opts.zig");
|
||||
const snapshot = @import("models/snapshot.zig");
|
||||
const Date = @import("Date.zig");
|
||||
const timeline = @import("analytics/timeline.zig");
|
||||
|
|
@ -99,7 +100,7 @@ pub fn parseSnapshotBytes(
|
|||
// record kind we don't know about). Every other srf error
|
||||
// indicates malformed data in a record we SHOULD understand, so
|
||||
// we propagate it up rather than silently losing rows.
|
||||
const rec = field_it.to(SnapshotRecord, .{}) catch |err| switch (err) {
|
||||
const rec = field_it.to(SnapshotRecord, srf_opts.machine_written) catch |err| switch (err) {
|
||||
error.ActiveTagDoesNotExist => continue,
|
||||
else => return error.InvalidSrf,
|
||||
};
|
||||
|
|
|
|||
|
|
@ -11,6 +11,7 @@
|
|||
/// symbol::02315N600,asset_class::Bonds,pct:num:15
|
||||
const std = @import("std");
|
||||
const srf = @import("srf");
|
||||
const srf_opts = @import("../srf_opts.zig");
|
||||
const Date = @import("../Date.zig");
|
||||
|
||||
/// A single classification entry for a symbol.
|
||||
|
|
@ -89,7 +90,7 @@ pub fn parseClassificationFile(allocator: std.mem.Allocator, data: []const u8) !
|
|||
defer it.deinit();
|
||||
|
||||
while (try it.next()) |fields| {
|
||||
const entry = fields.to(ClassificationEntry, .{}) catch continue;
|
||||
const entry = fields.to(ClassificationEntry, srf_opts.user_edited) catch continue;
|
||||
// Pre-fill `bucket` if the user didn't curate one. This
|
||||
// shifts the cost of `deriveBucket` to parse time and
|
||||
// makes downstream code free to read `entry.bucket`
|
||||
|
|
@ -269,6 +270,24 @@ test "parse classification file: missing name field stays null (backwards compat
|
|||
try std.testing.expectEqualStrings("Technology", cm.entries[0].sector.?);
|
||||
}
|
||||
|
||||
test "parse classification file: a hand-typed string separator on pct still parses" {
|
||||
// `metadata.srf` is hand-edited, so `pct::60` instead of `pct:num:60`
|
||||
// must not be fatal. See `srf_opts.user_edited` - under strict
|
||||
// coercion this string reaches an unchecked `val.?.number`, which is
|
||||
// undefined behaviour in ReleaseFast.
|
||||
const data =
|
||||
\\#!srfv1
|
||||
\\symbol::SYM,sector::Technology,pct::60
|
||||
\\symbol::SYM,sector::Healthcare,pct:num:40
|
||||
;
|
||||
var map = try parseClassificationFile(std.testing.allocator, data);
|
||||
defer map.deinit();
|
||||
|
||||
try std.testing.expectEqual(@as(usize, 2), map.entries.len);
|
||||
try std.testing.expectApproxEqAbs(@as(f64, 60), map.entries[0].pct, 0.001);
|
||||
try std.testing.expectApproxEqAbs(@as(f64, 40), map.entries[1].pct, 0.001);
|
||||
}
|
||||
|
||||
test "parse classification file: bucket round-trips" {
|
||||
const data =
|
||||
\\#!srfv1
|
||||
|
|
|
|||
|
|
@ -60,6 +60,7 @@
|
|||
const std = @import("std");
|
||||
const builtin = @import("builtin");
|
||||
const srf = @import("srf");
|
||||
const srf_opts = @import("../srf_opts.zig");
|
||||
const Date = @import("../Date.zig");
|
||||
|
||||
const logger = std.log.scoped(.transaction_log);
|
||||
|
|
@ -248,7 +249,7 @@ pub fn parseTransactionLogFile(
|
|||
defer it.deinit();
|
||||
|
||||
while (try it.next()) |fields| {
|
||||
const parsed = fields.to(TransferRecord, .{}) catch |err| {
|
||||
const parsed = fields.to(TransferRecord, srf_opts.user_edited) catch |err| {
|
||||
// Tests intentionally feed malformed records to exercise the
|
||||
// skip path; real parse failures stay visible outside tests.
|
||||
if (!builtin.is_test) {
|
||||
|
|
@ -560,6 +561,21 @@ test "parseTransactionLogFile: single cash transfer" {
|
|||
try testing.expect(r.note == null);
|
||||
}
|
||||
|
||||
test "parseTransactionLogFile: a hand-typed string separator on amount still parses" {
|
||||
// Hand-edited file, so `amount::50000` instead of `amount:num:50000`
|
||||
// must not be fatal. See `srf_opts.user_edited` - under strict
|
||||
// coercion this string reaches an unchecked `val.?.number`, which is
|
||||
// undefined behaviour in ReleaseFast.
|
||||
var log = try parseTransactionLogFile(testing.allocator,
|
||||
\\#!srfv1
|
||||
\\transfer::2026-05-02,type::cash,amount::50000,from::Acct A,to::Acct B,dest_lot::cash
|
||||
\\
|
||||
);
|
||||
defer log.deinit();
|
||||
try testing.expectEqual(@as(usize, 1), log.transfers.len);
|
||||
try testing.expectEqual(@as(f64, 50000), log.transfers[0].amount);
|
||||
}
|
||||
|
||||
test "parseTransactionLogFile: single lot-destination transfer" {
|
||||
var log = try parseTransactionLogFile(testing.allocator,
|
||||
\\#!srfv1
|
||||
|
|
|
|||
|
|
@ -270,10 +270,8 @@ pub fn loadPortfolioFromPathsAtRev(
|
|||
};
|
||||
defer allocator.free(real);
|
||||
|
||||
const rel = if (std.mem.startsWith(u8, real, info.root) and real.len > info.root.len)
|
||||
std.mem.trimStart(u8, real[info.root.len..], "/")
|
||||
else
|
||||
std.fs.path.basename(real);
|
||||
const rel = git.relPathInRoot(allocator, info.root, real) catch return null;
|
||||
defer allocator.free(rel);
|
||||
|
||||
const data = git.show(io, allocator, env, info.root, rev, rel) catch |err| switch (err) {
|
||||
error.PathMissingInRev => empty_blk: {
|
||||
|
|
|
|||
|
|
@ -25,6 +25,7 @@ const Config = @import("Config.zig");
|
|||
const cache = @import("cache/store.zig");
|
||||
const freshness = @import("cache/freshness.zig");
|
||||
const srf = @import("srf");
|
||||
const srf_opts = @import("srf_opts.zig");
|
||||
const analysis = @import("analytics/analysis.zig");
|
||||
const transaction_log = @import("models/transaction_log.zig");
|
||||
const TwelveData = @import("providers/twelvedata.zig").TwelveData;
|
||||
|
|
@ -842,10 +843,21 @@ pub const DataService = struct {
|
|||
next.provider = provider;
|
||||
next.fail_count = 0;
|
||||
|
||||
// A provider change and a cleared backoff are independent facts,
|
||||
// and conflating them hid the first one. This used to log only
|
||||
// when clearing an armed backoff - but the case that mattered
|
||||
// was a legacy cache carrying `provider = .yahoo` with no
|
||||
// backoff at all, which is every symbol that had drifted off
|
||||
// Tiingo before `tiingo_retry_after_s` existed. Twenty-one
|
||||
// symbols converted back to Tiingo without a single line.
|
||||
if (meta.provider != provider) {
|
||||
log.info("{s}: candle provider {t} -> {t}", .{ symbol, meta.provider, provider });
|
||||
}
|
||||
|
||||
switch (coverage) {
|
||||
.covered => {
|
||||
if (meta.tiingo_retry_after_s != 0) {
|
||||
log.info("{s}: provider converted {t} -> tiingo, clearing Tiingo backoff", .{ symbol, meta.provider });
|
||||
log.info("{s}: Tiingo serving again, clearing backoff", .{symbol});
|
||||
}
|
||||
next.tiingo_retry_after_s = 0;
|
||||
},
|
||||
|
|
@ -1146,9 +1158,9 @@ pub const DataService = struct {
|
|||
// see a same-dated file whose historical adj_close is
|
||||
// stale, so the basis has to be re-checked here.
|
||||
const synced = if (s.readCandleMeta(symbol)) |sm| sm.meta else m;
|
||||
const synced_action = freshness.newestCorporateAction(self.allocator, &s, symbol);
|
||||
const synced_action = freshness.newestCorporateAction(self.allocator, &s, symbol, synced.last_date);
|
||||
if (s.isCandleMetaFresh(symbol) and
|
||||
!freshness.adjustmentBasisStale(synced.adj_basis, synced.last_date, synced_action))
|
||||
!freshness.adjustmentBasisStale(synced.adj_basis, synced_action))
|
||||
{
|
||||
log.debug("{s}: candles synced from server and fresh", .{symbol});
|
||||
if (s.read(self.allocator, Candle, symbol, null, .any)) |r|
|
||||
|
|
@ -1179,8 +1191,7 @@ pub const DataService = struct {
|
|||
// tax the hot portfolio-pricing path for nothing.
|
||||
if (freshness.adjustmentBasisStale(
|
||||
m.adj_basis,
|
||||
m.last_date,
|
||||
freshness.newestCorporateAction(self.allocator, &s, symbol),
|
||||
freshness.newestCorporateAction(self.allocator, &s, symbol, m.last_date),
|
||||
)) {
|
||||
log.info("{s}: restating full history (adj_basis {f} predates a corporate action)", .{ symbol, m.adj_basis });
|
||||
if (self.refetchFullHistory(symbol, today, now_s, now_s < m.tiingo_retry_after_s)) |candles| {
|
||||
|
|
@ -2987,7 +2998,7 @@ pub const DataService = struct {
|
|||
defer it.deinit();
|
||||
|
||||
while (it.next() catch return result) |fields| {
|
||||
const entry = fields.to(CusipEntry, .{}) catch continue;
|
||||
const entry = fields.to(CusipEntry, srf_opts.machine_written) catch continue;
|
||||
if (entry.cusip.len == 0 or entry.ticker.len == 0) continue;
|
||||
// First occurrence wins; getOrPut stores the borrowed
|
||||
// slices directly - they live in `backing`, no dupe.
|
||||
|
|
@ -3153,7 +3164,7 @@ pub const DataService = struct {
|
|||
var it = srf.iterator(&reader, arena, .{ .parse_allocator = .none }) catch return;
|
||||
defer it.deinit();
|
||||
while (it.next() catch return) |fields| {
|
||||
const e = fields.to(CusipEntry, .{}) catch continue;
|
||||
const e = fields.to(CusipEntry, srf_opts.machine_written) catch continue;
|
||||
if (e.cusip.len == 0 or e.ticker.len == 0) continue;
|
||||
if (have.contains(e.cusip) or out.contains(e.cusip)) continue;
|
||||
const kc = arena.dupe(u8, e.cusip) catch continue;
|
||||
|
|
@ -4300,8 +4311,7 @@ test "getCandles offline never escalates a stale adjustment basis" {
|
|||
store.updateCandleMeta("SMPL", meta, 1); // expiry in the past => stale
|
||||
try std.testing.expect(freshness.adjustmentBasisStale(
|
||||
meta.adj_basis,
|
||||
meta.last_date,
|
||||
freshness.newestCorporateAction(allocator, &store, "SMPL"),
|
||||
freshness.newestCorporateAction(allocator, &store, "SMPL", meta.last_date),
|
||||
));
|
||||
|
||||
svc.panic_on_network_attempt = true;
|
||||
|
|
|
|||
56
src/srf_opts.zig
Normal file
56
src/srf_opts.zig
Normal file
|
|
@ -0,0 +1,56 @@
|
|||
//! SRF coercion policy, in one place.
|
||||
//!
|
||||
//! SRF encodes a field's type in its separator: `key::v` is a string,
|
||||
//! `key:num:v` a number, `key:bool:v` a boolean. Coercion into a typed
|
||||
//! struct therefore depends on the writer having picked the right one.
|
||||
//!
|
||||
//! That assumption holds for files zfin writes and breaks for files
|
||||
//! people write, so the two get different options - and which one a
|
||||
//! parser wants is a property of where the file came from, not of the
|
||||
//! struct being parsed. Naming the two policies keeps that decision
|
||||
//! visible at the call site instead of buried in whichever comment
|
||||
//! happened to explain it first.
|
||||
|
||||
const srf = @import("srf");
|
||||
|
||||
/// For files a HUMAN edits: `portfolio.srf`, `accounts.srf`,
|
||||
/// `metadata.srf`, `watchlist.srf`, `transaction_log.srf`,
|
||||
/// `projections.srf`, `imported_values.srf`, `acknowledgments.srf`, and
|
||||
/// the keybind config.
|
||||
///
|
||||
/// Accepts a string where a number was declared, because a hand-typed
|
||||
/// `close_price::200.00` instead of `close_price:num:200.00` is a
|
||||
/// slip, not a different intent. SRF's own doc says as much: strict
|
||||
/// coercion is "intended for performant access for cache use cases...
|
||||
/// if you want to use this for human-edited files, turn this on".
|
||||
///
|
||||
/// It is also a safety measure, which is the part worth not
|
||||
/// forgetting. Under strict coercion a string reaching a numeric field
|
||||
/// falls through to an unchecked `val.?.number` inside SRF - a panic
|
||||
/// in Debug/ReleaseSafe and undefined behaviour in ReleaseFast, which
|
||||
/// is how zfin is built. One `close_price::200.00` once took down
|
||||
/// every `zfin portfolio` run; the fix was to turn this on for
|
||||
/// `portfolio.srf` alone, which left every other hand-edited file
|
||||
/// exposed to the same typo.
|
||||
///
|
||||
/// This is mitigation, not a cure. Two holes remain, both needing an
|
||||
/// upstream fix in SRF's `coerce`:
|
||||
///
|
||||
/// - a non-string, non-number value in a numeric field (say
|
||||
/// `harvested:bool:true`) still reaches the unchecked access;
|
||||
/// - enum fields ignore this option entirely, so a typo like
|
||||
/// `security_type::stok` still hits `stringToEnum(...).?`.
|
||||
pub const user_edited: srf.CoercionOptions = .{ .strings_to_numbers = true };
|
||||
|
||||
/// For files ZFIN writes: the candle, quote, and options caches,
|
||||
/// `cusip_tickers.srf` under `cache_dir`, server responses, and the
|
||||
/// `history/<date>-portfolio.srf` snapshots produced by
|
||||
/// `zfin snapshot`.
|
||||
///
|
||||
/// Keeps SRF's strict default. The writer is a machine that always
|
||||
/// emits `:num:` for numbers, so accepting a string instead would only
|
||||
/// mask a serializer bug rather than tolerate a human slip - these
|
||||
/// files are not edited after the fact. Strictness is also free
|
||||
/// performance on the hot paths, where a cached candle file is
|
||||
/// millions of records.
|
||||
pub const machine_written: srf.CoercionOptions = .{};
|
||||
|
|
@ -1,6 +1,7 @@
|
|||
const std = @import("std");
|
||||
const vaxis = @import("vaxis");
|
||||
const srf = @import("srf");
|
||||
const srf_opts = @import("../srf_opts.zig");
|
||||
|
||||
pub const Action = enum {
|
||||
quit,
|
||||
|
|
@ -486,7 +487,7 @@ pub fn loadFromDataChecked(allocator: std.mem.Allocator, data: []const u8) LoadO
|
|||
|
||||
var idx: usize = 0;
|
||||
while (ri.next() catch return .fallback) |fields| : (idx += 1) {
|
||||
const raw = fields.to(RawRecord, .{}) catch |err| {
|
||||
const raw = fields.to(RawRecord, srf_opts.user_edited) catch |err| {
|
||||
// Per-record parse failure (missing field, bad key
|
||||
// string, unknown action). Don't drop the whole file -
|
||||
// skip the record and warn the user. Record index is
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue