# debtDetails datasets and construction

Status: Active
Schema version: `1.1` (a job queued before 1.1 completes dense and says `1.0`).
Integration: [banker guide](banker-integration-guide.md).

`get_debt_details_dataset` returns the same outstanding-row builder and
calculation-bundle builder used by the debtDetails page. It does not scrape
HTML or reconstruct cashflows with a different model. Input security selection,
dates, fiscal year end, overlays and visible working copy must match before
comparing outputs. JSON values, not browser formatting or AG component options,
are the exported contract. The initial catalog covers outstanding debt and the
17 calculation-bundle datasets below; additional page workflows are listed
explicitly under Coverage boundaries.

## Envelope and source graph

```text
authorized Borrower / visible working copy
  -> OutstandingSecurity + Issue facts -> outstanding.rows
  -> selected security cashflow engine -> dated principal/interest matrices
     -> tree / groupedRollup
        -> fiscal-period charts and grouped trees
        -> debt-profile pivots and series P&I
        -> payments and calculated journal entries
  -> optional visible project/fund allocation, net-DS and variable overlays
```

The file has `schema_version`, `issuer_id`, `requested_datasets`, `parameters`,
`generated_at`, `units`, `selection`, optional `outstanding`, and optional
`calculations`, plus a browser `citation`. A file requested with
`encoding: "columnar"` that carries `calculations` also has `row_encoding`
(see Row encoding below). With `outstanding`, `series_records`
maps its actual `issue_id` values to `series_id`, `entity_key` and `designation`.
Missing/invalid Issue associations have no entry; never join by display name.
See the [citation guide](citation-guide.md) for workspace identity, login and
row-level citation limits. `outstanding` contains `rows` and `provenance`; `calculations`
contains `meta` and only requested sections. Preserve every warning and coverage
field. Engine/native-path indicators describe how that result was produced;
they are not freshness or financial verification claims.

The page's source routes are session-authenticated XHR endpoints:
`GET /muni/api/getOutstandingDebtData` and
`POST /muni/api/getDebtDetailsCalculationBundle`. Older specialized cashflow,
profile and P&I handlers feed the same families. MCP invokes shared Python
builders internally, not those URLs with an OAuth token.

## Dataset catalog

Names are case-sensitive. Omission selects `outstanding` and `groupedRollup`.

| Dataset | File location | Grain and use |
|---|---|---|
| `outstanding` | `outstanding.rows` | Hierarchical display rows for credit, series and individual securities; principal and bond facts |
| `groupedRollup` | `calculations.groupedRollup` | Dated parallel arrays at credit/series/subcredit grain; preferred whole-issuer source |
| `tree` | `calculations.tree` | Security-level dated cashflows, wide or columnar encoding |
| `annualChart` | `calculations.annualChart` | Debt service and P&I bucketed by fiscal year |
| `semiannualChart` | `calculations.semiannualChart` | Fiscal half-year chart buckets |
| `monthlyChart` | `calculations.monthlyChart` | Monthly debt-service chart buckets |
| `continuousChart` | `calculations.continuousChart` | Exact payment-date chart buckets |
| `annualTree` | `calculations.annualTree` | Fiscal-year grouped tree rows |
| `semiannualTree` | `calculations.semiannualTree` | Half-year grouped tree rows |
| `continuousTree` | `calculations.continuousTree` | Payment-date grouped tree rows |
| `annualProfile` | `calculations.debtProfile.annual_debt_profile_pivoted` | Annual debt-profile pivot |
| `semiannualProfile` | `calculations.debtProfile.semiannual_debt_profile_pivoted` | Semiannual debt-profile pivot |
| `continuousProfile` | `calculations.debtProfile.continuous_debt_profile_pivoted` | Payment-date debt-profile pivot |
| `annualPI` | `calculations.seriesPI.annual_data` | Annual principal and interest by series |
| `semiannualPI` | `calculations.seriesPI.semiannual_series_pi_data` | Semiannual series P&I |
| `continuousPI` | `calculations.seriesPI.continuous_series_pi_data` | Payment-date series P&I |
| `upcomingPayments` | `calculations.upcomingPayments` | `series`, `cusip` payment rows and applied `window` |
| `journalEntries` | `calculations.journalEntries` | Calculated `journal_entries`; `premium_schedule` is currently empty through this bundle; no journal posting |

## Options and selection

The tool's `inputSchema` lists every option's type and allowed values; this
table gives their meaning.

| Option | Type / default | Semantics |
|---|---|---|
| `asOfDate` | ISO date; request day if absent | Pin valuation/cashflow cutoff. `generated_at` is a separate timestamp. |
| `fiscalYearEnd` | Valid `MM/DD`; `12/31` | Fiscal-year and half-year bucketing |
| `variableRate` | Finite JSON number; `1.0` | Percentage points: `1.0` = 1%. Explicit modeling assumption, not an observed index. |
| `securityIds` | Positive integer array | Stable selected OutstandingSecurity IDs |
| `cusips`, `securityKeys` | Nonempty string arrays | Supported security selectors; preserve opaque IDs, including non-CUSIP instruments |
| `allBorrowerSecurities` | Boolean; defaults to true when no selectors | Select the authorized issuer's complete calculation population; true with any nonempty selector array is refused |
| `encoding` | Enum in `inputSchema`; `columnar` default | Row transport for the tree, debt-profile and series P&I row lists (see Row encoding). Values are identical either way. `dense` is exactly the 1.0 default request. MuniPro's in-product assistant defaults to `dense`. |
| `treeEncoding` | Enum in `inputSchema`; unset by default | Dated-tree representation. Unset: a columnar/2 tree under `encoding: "columnar"`, a columnar/1 tree under `encoding: "dense"`. Sent: `wide` rows or a columnar/1 tree under either encoding |
| `sparseRollup` | Boolean | Allow sparse period-index maps in rollup value arrays |
| `includeAsOfCashflow` | Boolean | Include the exact as-of payment date when enabled |
| `includeMaturedCashflows` | Boolean | Include matured cashflows in the calculation selection |
| `includePayingAgent` | Boolean | Carry paying-agent data where available |
| `treeGroup` | Enum in `inputSchema` | Series grouping, or fund regrouping over visible fund allocations |
| `lens` | Enum in `inputSchema` | Security view, or project regrouping over committed visible allocations plus Unallocated |
| `debtServiceMode` | Enum in `inputSchema` | Gross, or net with authorized net debt-service adjustments |
| `netDsAsOfDate`, `netDsScenario` | ISO date / scenario value accepted by page | Net-debt-service valuation context; preserve returned metadata |
| `includeVariableDebt` | Boolean | Attach variable instrument overlays |
| `amort` | Enum in `inputSchema` | Variable debt expected versus worst-case put-acceleration assumptions |
| `paymentsStartDate`, `paymentsEndDate` | ISO date or explicit null | Payment window; omitted values use the page's bounded default. Null opens that end. Read returned `window`. |
| `frequency` | Enum in `inputSchema` | Journal-entry frequency |
| `includeAllocations`, `includePools`, `includeMatured` | Boolean | Outstanding-row display options; company/pool visibility still applies |
| `viewMode`, `leafCredit`, `leafSeries` | Page display/leaf selection values | Outstanding-row view selection, not a substitute for calculation security selection |

Unknown options are refused. No borrower/company/user identity is accepted in
`options`. Booleans must be JSON booleans, not strings. Selector arrays have a
combined 250-item ceiling. Use one selector family per request: `cusips`, or
`securityIds`/`securityKeys`.
Mixing nonempty CUSIPs with either ID/key array is refused. Combining
`allBorrowerSecurities: true` with any nonempty `cusips`, `securityIds` or
`securityKeys` array is also refused before a job is queued. Omit the flag or
set it to false for explicit selections, so no selector is silently ignored.
Dense detail over a larger issuer must be partitioned
into explicit selections of <=250 securities. Whole-issuer gross/security-lens
rollups, charts and grouped trees can use the grouped aggregation path.
Large dense requests refuse rather than truncate. Aggregate partition results
only after confirming disjoint leaf identities and identical parameters.

Outstanding rows are issuer-wide unless their own leaf/display filters are
provided; `securityIds` select calculations, not the outstanding table. This
distinction is included in the file's `selection` metadata. Project+fund and
project+net combinations are invalid and refuse instead of returning a
mislabelled gross dataset.

## Outstanding-row dictionary

The outstanding payload preserves page field names and display types. Important
fields include `orgHierarchy`, `node_type`, `credit`, `series`, `sub_credit`,
`security_id`, `issue_id`, `_rowId`, `cusip9`, maturity/call-date fields,
`outstanding_amt`, `coupon`, and
the returned issued/retired/callable facts and provenance. Fields vary by node
and instrument; absent facts are not zero. Use returned stable IDs for identity;
display labels can repeat across issues and subcredits.

### Bonds without CUSIPs

CUSIP is optional, never the required identity or a row-uniqueness key. Use the
returned `(issuer_id, security_id)` for an OutstandingSecurity and `issue_id`
for its series. Select CUSIP and non-CUSIP bonds together using `securityIds`;
whole-issuer selection includes both. Keep distinct IDs even when names,
maturity, coupon and blank CUSIP values are identical. Never invent a CUSIP,
drop a blank identifier, or deduplicate on CUSIP. Legacy composite `securityKeys`
can match multiple similar records; prefer IDs for exact selection.

Missing CUSIPs do not prevent outstanding debt, cashflows, issuer citations or
uploaded-file retrieval. CUSIP-based EMMA discovery is a separate coverage
boundary: the document manifest reports `securities_without_cusip` and partial
coverage. If no security has a usable CUSIP, no feed is queried and `sources`
is empty; feed availability is unknown. An empty manifest for those bonds does
not mean no legal documents exist. Inspect current uploaded issuer files and obtain additional authorized
sources when needed. Synthetic variable-instrument rows without a security ID
are a different grain; do not coerce them into OutstandingSecurity selectors.

`outstanding_amt` is a formatted currency string on the page (for example
`$1,000,000.00`); `coupon` is a formatted percentage string (`5.000%`). These are
not numeric cashflow matrix columns. Underlying stored coupon ratios such as
`0.05` become displayed percentages; do not multiply a displayed coupon again.
Calculation money values are numeric USD unless explicitly labelled otherwise.
Strip formatting only from documented display-value fields, retain missing
values, and round for presentation after reconciliation.

Parent hierarchy rows already aggregate children. To sum outstanding debt,
choose security leaves with their stable identity, or one returned aggregate
level. Never add parents and leaves together. Treat swaps and other non-principal
instruments according to their instrument metadata; notional is not outstanding
bond principal. Carry provenance `as_of`, `basis`, and `coverage` with any table.

## Dated tree and columnar decoding

Wide dated fields use `cashflow_<date>`, `par_<date>`, `interest_<date>` with
slashes/hyphens replaced by underscores. `cashflow` is debt service; `par` is
principal. The result supplies dates; never manufacture a monthly schedule from
maturity dates. The shared model handles coupons, amortization, calls and supported
instruments, with reconciliation/dropped-security notices in `meta`.

With `treeEncoding=columnar`, the tree declares `encoding: columnar/1`, dates,
date-field metadata and rows carrying `meta`, `at`, and `columns`. Each row's
`at` array indexes the shared date axis. Value arrays within `columns` align
with that row's `at`, not necessarily every date. Retain `meta`; reconstruct
wide dated fields from the supplied prefix/date information. A null/absent value
is not automatically a zero or an observation. Use `wide` when the client cannot
decode the advertised encoding. Do not sum group and security rows together.

## Row encoding

With `encoding: "columnar"` (the default from 1.1) these row lists travel as
columnar/2 blocks: `calculations.tree.tree_data` when no `treeEncoding` is
sent, and the `rows` of the three `calculations.debtProfile.*_pivoted`
sections and the three `calculations.seriesPI` sections. Every other field
keeps its dense form. The encoding changes no value: nothing is dropped,
rounded or reordered. A block is an object with `"encoding": "columnar/2"`,
`keys`, `shapes`, `constants` and `rows`:

- `keys`: every row key, written once, in first-seen order.
- `shapes`: one key order per shape, as flat `start, count` runs of
  consecutive indexes into `keys`.
- `constants`: `[keyIndex, value]` pairs; that key has that value in every row
  whose shape lists it, and rows do not store it.
- `rows`: `[shapeIndex, value, ...]`, one value per non-constant key of the
  shape, in shape order.

To decode, expand each shape's runs into key indexes; for each row, walk its
shape's keys in order, taking a constant key's constant value and every other
key's next row value. A row must use all of its values. The result is the
dense row list with the same keys, key order and values. A list the encoder
cannot represent exactly (including one holding a value nested more than 64
deep) stays an array, so check for `"encoding": "columnar/2"` before
decoding. This block:

```json
{"encoding": "columnar/2", "keys": ["series", "cusip", "fy_2027", "fy_2028"],
 "shapes": [[0, 4], [0, 2]], "constants": [[0, "2019A"]],
 "rows": [[0, "123456AB1", 1000.0, 1000.0], [1, "123456AB2"]]}
```

decodes to `[{"series": "2019A", "cusip": "123456AB1", "fy_2027": 1000.0,
"fy_2028": 1000.0}, {"series": "2019A", "cusip": "123456AB2"}]`.

The file's top-level `row_encoding` object repeats this for a client that has
not read the guide: `format` (`columnar/2`), `locations` (every path a block
can appear at), `decode` (these rules in one paragraph), `docs` (the docs
section; MuniPro sign-in required) and `dense`. It has no `encoding` key and
is never a block. To receive plain row objects instead, send
`encoding: "dense"` for the profile and P&I lists, plus
`treeEncoding: "wide"` for a plain-row tree.

## Rollups, tables and charts

`groupedRollup` has `mode`, `frequency`, `fiscalYearEnd`, `periods`, and
`groups.credit`, `groups.series`, `groups.subcredit`. A group identifies `credit`,
`series`, `sub_credit`, `orgHierarchy`, numeric `outstanding`, and four value
series: `fy_ds`, `pi_principal`, `pi_interest`, `pi_debt_service`. Arrays align to
`periods`; sparse objects map period index to value. The normal zero bucket is
zero. Do not treat absent source facts as observed zeros elsewhere in the file.

For each selected leaf/payment date, principal and interest feed the corresponding
P&I buckets. Debt service uses the calculated cashflow (or principal+interest
fallback). Interest-only fallback can apply when a cashflow lacks split fields;
preserve this distinction when reconciling `fy_ds` against `pi_debt_service`.
Each leaf contributes separately to credit, series and subcredit aggregates.
Those three arrays are alternate views, not additive populations.

For annual charts, group payment dates by the requested fiscal year end and sum
the dated values. Semiannual uses the fiscal half-year boundary, monthly uses
calendar months, and continuous retains exact payment dates. Returned period
labels are authoritative. Chart sections have `mode`, `frequency`, `periods`
and `groups`. Rows expose period columns named `fy_ds_<period>`,
`pi_principal_<period>`, `pi_interest_<period>`, `pi_debt_service_<period>`.
AG Charts uses periods on x, one credit/series/subcredit as each y-series, and
the chosen value prefix. Stack or group per presentation intent; avoid summing
the same population at multiple hierarchy levels.

Compact Tree DS sections declare `mode: annualCompact`, their actual
`frequency`, `groupedOnly: true`, `periods`, and `tree_data`. The mode label is
historical and also appears for semiannual/continuous trees; use `frequency`.
AG Grid reads `orgHierarchy` for tree grouping and the returned period fields
for columns. These grouped trees are not a security-level download; use `tree`.

Debt Profile sections pivot the same dated cashflows and security metadata by
the selected frequency. Series P&I sections pivot principal, interest and debt
service by series and period. Existing page series field keys normalize spaces
and hyphens to underscores, `&` to `and`, and remove parentheses/periods. Such
display keys are not globally unique issue IDs. Keep returned columns/series
metadata. Preserve total rows and do not add `TOTAL` to its own components.

Profile `*_entryN_principal_amt` is a one-time principal display, not the
principal paid on that row's date. Non-CUSIP securities retain independent
displays even when their terms match a CUSIP bond. Annual and semiannual views
place the display in the maturity bucket; continuous uses the first qualifying
principal-payment row. An amortizing security must not repeat its full principal
display at every sinking payment. Other detail fields (such as outstanding and
amortization) can repeat as descriptive security facts; do not sum them across
payment rows. Use dated `par_*` values or the P&I datasets for actual principal
payments, and cashflow/interest values for debt-service charts.

## Callable charts and accounting views

Callable charts are client-side transformations of outstanding leaves. They
filter callable eligibility, apply the page's default VRDB exclusion (including
its already-callable zero-coupon heuristic), optionally shift call dates by
90 days, then aggregate outstanding principal by credit/subcredit/series or
coupon. Annual buckets clamp past call years to the current chart year;
continuous buckets use dates. Coupon keys such as `C_5_000` mean 5.000%; missing
coupon remains unknown. Missing eligible bucket combinations can be zero-filled
for chart rendering. These are callable-par views, not refunding valuations.

Upcoming Payments has both series/date and security/date rows plus the applied
date window. Do not add those two grains together. Legacy payment display rows
can share blank CUSIP and other labels;
retain separate rows and use the security cashflow tree for exact identity.
Journal entries are calculated proposals; exporting them does not post a ledger.
The shared bundle passes no premium security-input dataframe, so its
`premium_schedule` is currently empty even for bonds with premium. That empty
array is a coverage limitation, not evidence that premium amortization is zero.
The page's separate premium calculation is not exported by this banker adapter.
The page's Year-End Adjustments requires security leaves and that separate
premium schedule: group by issue/fund, sum principal and interest by fiscal year, then
construct accrual/reversal/retirement and amortization display entries. Its
client display is not a certified period close or a separate authoritative ledger.

Variable overlays attach `variableInterest`, `variablePrincipal`,
`putAcceleration`, `variableDebtInstruments`, and `variableDebtMeta` where
applicable. They can extend the chart's period axis for commercial paper.
Use the returned axis and overlay metadata. Never add a component already
included in the base series twice. Expected amortization and worst-case puts
are different assumptions. Chart overlays alone do not supply a complete
facility liquidity/expiry timeline.

## Coverage boundaries

The banker catalog exports the core debtDetails outstanding/cashflow/grid/chart
datasets, current uploaded files, and indexed debt documents. It does not grant
all page controllers. These additional debtDetails/related page surfaces need
separate reviewed read adapters before they can be claimed as complete exports:

| Surface | Existing browser source / behavior | Initial banker treatment |
|---|---|---|
| Allocation editor/project/pool workspace | `getAllocationEditorSummary`, `getAllocationEditorSeriesDetail`, `getProjectAllocationRollup`, `getPoolTreeData`, `getPoolCoverageData` | Visible allocation effects available through selected lenses; editor state and pool-coverage datasets not separately exported |
| Premium and certified accounting | Premium calculation, `components/*`, period-close, reconciliation and GL configuration | Core calculated journal entries only; premium schedule empty, complete Year-End Adjustments and certified-close/component ledgers excluded |
| Net debt-service workspace | `netDebtService/*`, subsidy assumptions and snapshots | Selected net calculation overlay supported; configuration/snapshot history excluded |
| Variable debt setup/liquidity | `getVariableDebtSetupGrid`, variable instrument inventory/rate history | Calculation overlays supported; full facility wall requires additional sources |
| DTCC/DCN, holders and secondary activity | Dedicated data/monitor routes and entitlements | Excluded from initial profile; no inferred holding/trading data |
| Refunding history/lineage and Report-tab valuations | `getDebtDetailsRefundingHistory`, `getRefundingLineage`, valuation models | Excluded from initial profile; callable par is not valuation output |
| Edit debt, saved settings, imports and model writes | Session/CSRF write routes | Excluded; only borrower-verification requests have a banker write path |

The [presentation recipes](banker-presentation-guide.md) distinguish supplied
datasets from capabilities needed for later slides. A recipe, source mapping
or schema does not establish deployment, live data coverage or Rogo acceptance.