From 2dee910d422eb24232f29ca956fc88b2802c6a09 Mon Sep 17 00:00:00 2001 From: Denis Angell Date: Wed, 13 May 2026 19:08:27 +0200 Subject: [PATCH] move skills --- .github/scripts/doc-agent/.env.example | 13 + .github/scripts/doc-agent/README.md | 45 ++- .github/scripts/doc-agent/package.json | 11 +- .../doc-agent/prompts/document-file.md | 26 +- .../scripts/doc-agent/prompts/regen-skill.md | 47 +++ .github/scripts/doc-agent/src/document.ts | 24 +- .github/scripts/doc-agent/src/index.ts | 11 + .../scripts/doc-agent/src/prompt-loader.ts | 2 +- .github/scripts/doc-agent/src/regen-skills.ts | 146 ++++++++++ .gitignore | 4 + docs/skills/{soul => }/consensus.md | 0 docs/skills/{soul => }/cryptography.md | 0 docs/skills/{soul => }/ledger.md | 0 docs/skills/{soul => }/nodestore.md | 0 docs/skills/{soul => }/peering.md | 0 docs/skills/protocol.md | 271 ++++++++++++++++++ docs/skills/{soul => }/rpc.md | 0 docs/skills/{soul => }/shamap.md | 0 docs/skills/soul/protocol.md | 64 ----- docs/skills/{soul => }/sql.md | 0 docs/skills/{soul => }/test.md | 0 docs/skills/{soul => }/transactors.md | 0 docs/skills/{soul => }/websockets.md | 0 23 files changed, 571 insertions(+), 93 deletions(-) create mode 100644 .github/scripts/doc-agent/.env.example create mode 100644 .github/scripts/doc-agent/prompts/regen-skill.md create mode 100644 .github/scripts/doc-agent/src/regen-skills.ts rename docs/skills/{soul => }/consensus.md (100%) rename docs/skills/{soul => }/cryptography.md (100%) rename docs/skills/{soul => }/ledger.md (100%) rename docs/skills/{soul => }/nodestore.md (100%) rename docs/skills/{soul => }/peering.md (100%) create mode 100644 docs/skills/protocol.md rename docs/skills/{soul => }/rpc.md (100%) rename docs/skills/{soul => }/shamap.md (100%) delete mode 100644 docs/skills/soul/protocol.md rename docs/skills/{soul => }/sql.md (100%) rename docs/skills/{soul => }/test.md (100%) rename docs/skills/{soul => }/transactors.md (100%) rename docs/skills/{soul => }/websockets.md (100%) diff --git a/.github/scripts/doc-agent/.env.example b/.github/scripts/doc-agent/.env.example new file mode 100644 index 0000000000..1d69051ab1 --- /dev/null +++ b/.github/scripts/doc-agent/.env.example @@ -0,0 +1,13 @@ +# Copy this file to .env and fill in your values. +# .env is gitignored and will never be committed. + +# Required: Anthropic API key for the Claude Agent SDK. +ANTHROPIC_API_KEY=sk-ant-... + +# Optional: Override the path to the xrpld repo root. +# Defaults to three levels up from this directory (the repo this lives in). +# XRPLD_ROOT=/path/to/xrpld + +# Optional: Override the model used by the agent. +# Defaults to claude-opus-4-7. +# DOC_AGENT_MODEL=claude-opus-4-7 diff --git a/.github/scripts/doc-agent/README.md b/.github/scripts/doc-agent/README.md index a0d93d17ac..93d8a70f5a 100644 --- a/.github/scripts/doc-agent/README.md +++ b/.github/scripts/doc-agent/README.md @@ -5,19 +5,23 @@ Claude Agent SDK. ## What it does -Two modes: +Three modes: - **document** — Add Doxygen `/** */` documentation to a C++ file or - directory. The agent reads the file, related tests, and module skill - context, then writes documentation comments per the project standards in - `docs/DOCUMENTATION_STANDARDS.md`. + directory. For each target file, the agent reads the sibling + `.ai.md` (high-signal prose generated by the athenah-ai pipeline), + the module skill, and the file itself, then writes Doxygen comments per + the standards in `docs/DOCUMENTATION_STANDARDS.md`. - **review** — Given a git diff range, detect documentation drift. Used by the `doc-review` GitHub Action and locally for testing. +- **regen-skills** — Rebuild a module's skill file at + `docs/skills/soul/.md` from the `.ai.md` files in that module + and the existing skill content. ## Requirements -- Node.js >= 20 -- `ANTHROPIC_API_KEY` environment variable +- Node.js >= 20.12 (for native `--env-file` support) +- `ANTHROPIC_API_KEY` (in `.env` or exported in shell) - Tools the agent uses: `git`, `gh` (for `--pr`) ## Install @@ -25,8 +29,13 @@ Two modes: ```sh cd .github/scripts/doc-agent npm install +cp .env.example .env +# edit .env and set ANTHROPIC_API_KEY ``` +The npm scripts auto-load `.env` via Node's `--env-file-if-exists` flag. +You can also export the variables in your shell — both work. + ## Build and lint ```sh @@ -41,9 +50,7 @@ npm run check:fix # lint + format + fix ## Usage ```sh -export ANTHROPIC_API_KEY=sk-ant-... - -# Document a single file +# Document a single file (reads sibling .ai.md if present) npm run document include/xrpl/basics/base_uint.h # Document an entire module @@ -54,10 +61,22 @@ npm run review develop..HEAD # Review a PR npm run review -- --pr 1234 + +# Regenerate a skill file from this module's .ai.md inputs +npm run regen-skills protocol +npm run regen-skills ledger ``` -When invoked outside the xrpld repo, set `XRPLD_ROOT` to the path of the -checkout you want to operate on. +When invoked outside the xrpld repo, set `XRPLD_ROOT` in `.env` to the path +of the checkout you want to operate on. + +## ai.md context files + +The doc-agent reads a sibling `.ai.md` next to each source file when +documenting it. These are produced by the upstream `athenah-ai` pipeline +and treated as the authoritative source of intent. They are gitignored +(`*.ai.md` in `.gitignore`) and should be removed once the initial +documentation pass is complete. ## Outputs @@ -76,13 +95,15 @@ doc-agent/ ├── biome.json ├── prompts/ │ ├── document-file.md # System prompt for documentation mode -│ └── review-diff.md # System prompt for review mode +│ ├── review-diff.md # System prompt for review mode +│ └── regen-skill.md # System prompt for regen-skills mode └── src/ ├── index.ts # CLI entry point ├── config.ts # Paths, model, module-skill map ├── prompt-loader.ts # Loads prompts + module skill context ├── document.ts # Document mode ├── review.ts # Review mode + ├── regen-skills.ts # Regen-skills mode └── types.ts # Shared types ``` diff --git a/.github/scripts/doc-agent/package.json b/.github/scripts/doc-agent/package.json index a4a4f0d5b2..4eb459b9e3 100644 --- a/.github/scripts/doc-agent/package.json +++ b/.github/scripts/doc-agent/package.json @@ -9,10 +9,11 @@ }, "scripts": { "build": "tsc", - "start": "node dist/index.js", - "dev": "tsx src/index.ts", - "document": "tsx src/index.ts document", - "review": "tsx src/index.ts review", + "start": "node --env-file-if-exists=.env dist/index.js", + "dev": "tsx --env-file-if-exists=.env src/index.ts", + "document": "tsx --env-file-if-exists=.env src/index.ts document", + "review": "tsx --env-file-if-exists=.env src/index.ts review", + "regen-skills": "tsx --env-file-if-exists=.env src/index.ts regen-skills", "typecheck": "tsc --noEmit", "lint": "biome lint src", "format": "biome format --write src", @@ -29,6 +30,6 @@ "typescript": "^5.7.0" }, "engines": { - "node": ">=20" + "node": ">=20.12" } } diff --git a/.github/scripts/doc-agent/prompts/document-file.md b/.github/scripts/doc-agent/prompts/document-file.md index 701e7cbfc1..d04ed75c73 100644 --- a/.github/scripts/doc-agent/prompts/document-file.md +++ b/.github/scripts/doc-agent/prompts/document-file.md @@ -34,7 +34,7 @@ Read `docs/DOCUMENTATION_STANDARDS.md` for the full specification. Key rules: ## Module Context -Before you start, read the relevant skill file in `docs/skills/soul/` for +Before you start, read the relevant skill file in `docs/skills/` for the module you're working on. These capture per-module conventions, key classes, and gotchas: @@ -42,20 +42,26 @@ classes, and gotchas: - `protocol` — STObject, SField, Serializer, TER codes, Features, Keylets - `ledger` — ReadView/ApplyView, state tables, payment sandbox - `tx` / `transactors` — transaction pipeline -- `consensus`, `peering`, `nodestore`, `shamap`, `rpc` — see `docs/skills/soul/` +- `consensus`, `peering`, `nodestore`, `shamap`, `rpc` — see `docs/skills/` ## Process -1. Read the target file completely -2. Read the corresponding skill file in `docs/skills/soul/` if one applies -3. Identify entities that need documentation (public classes, structs, +1. If "Authoritative AI Context" is provided in the user prompt, treat it as + the source of truth for the file's intent and behavior. Your task is to + translate that prose into structured Doxygen comments on the declarations. +2. Read the target file completely +3. Read the corresponding skill file in `docs/skills/` if one applies +4. Identify entities that need documentation (public classes, structs, public methods, free functions in headers, enums) -4. For each entity: read the implementation (and tests if helpful), then - write a Doxygen comment that captures behavior and intent -5. Use the Edit tool to add the comments to the file -6. Do NOT modify code logic — only add documentation -7. Do NOT add documentation to entities that don't need it (private members +5. For each entity: cross-reference the ai.md context, read the implementation + (and tests if helpful), then write a Doxygen comment that captures behavior + and intent +6. Use the Edit tool to add the comments to the file +7. Do NOT modify code logic — only add documentation +8. Do NOT add documentation to entities that don't need it (private members with obvious purpose, simple getters where the name is self-explanatory) +9. Do NOT read the `.ai.md` file yourself — it is already injected into your + prompt when one exists for the target file When you finish, summarize: - How many entities you documented diff --git a/.github/scripts/doc-agent/prompts/regen-skill.md b/.github/scripts/doc-agent/prompts/regen-skill.md new file mode 100644 index 0000000000..7194f181c8 --- /dev/null +++ b/.github/scripts/doc-agent/prompts/regen-skill.md @@ -0,0 +1,47 @@ +You are updating a per-module skill file for the xrpld codebase. + +A "skill" is a single markdown file at `docs/skills/.md` that +captures the institutional knowledge for one module: what it does, key +classes, conventions, gotchas, and how to work in it. The skill file is +loaded as context whenever an agent works on code in that module. + +## Inputs + +You will be given: +- The current skill file for the module (the baseline to update) +- A list of `.ai.md` files describing the source files in this module + (one per source file, with high-signal prose about purpose and design) + +## Your task + +Produce a new, improved skill file that integrates the knowledge from the +ai.md files into the existing skill. Specifically: + +1. Update the description of the module's responsibility if the ai.md files + reveal more accurate or detailed framing +2. Add any classes, patterns, or invariants the skill is missing +3. Update lists of key files / entry points / conventions +4. Add gotchas and non-obvious behavior surfaced by the ai.md files +5. Keep the structure of the existing skill (don't reorganize for the sake + of it — only restructure if the existing structure is genuinely failing) +6. Be terse. A skill file is a reference card, not a textbook. 200-500 lines + is typical; over 1000 means you're padding. + +## Quality rules + +- **Do not duplicate the ai.md content.** Aggregate, synthesize, distill. + The skill is the module-level view; individual file details belong in + ai.md (and eventually in inline Doxygen comments). +- **Preserve accurate existing content.** Don't rewrite working sections. +- **Cite file paths** for specific claims (e.g., "see `STAmount.h:roundToScale`"). +- **Flag contradictions.** If two ai.md files describe the same concept + differently, surface the conflict rather than silently picking one. +- **Keep prose grounded.** No marketing language. No "robust, scalable, + enterprise-grade" filler. Engineers reading this need facts. + +## Output + +Emit the complete new skill file content as your final assistant message. +Start with the markdown heading. Do not include meta-commentary like "Here +is the updated skill file" — the output is captured verbatim and written +to the skill file path. diff --git a/.github/scripts/doc-agent/src/document.ts b/.github/scripts/doc-agent/src/document.ts index 60d8c54c92..760e35527b 100644 --- a/.github/scripts/doc-agent/src/document.ts +++ b/.github/scripts/doc-agent/src/document.ts @@ -3,6 +3,7 @@ */ import { existsSync, readdirSync, statSync } from 'node:fs'; +import { readFile } from 'node:fs/promises'; import { join, relative, resolve } from 'node:path'; import { query } from '@anthropic-ai/claude-agent-sdk'; import { MODEL, XRPLD_ROOT } from './config.js'; @@ -47,6 +48,21 @@ function findCppFiles(target: string): string[] { return results; } +/** + * Read the sibling .ai.md file for a source file, if one exists. + * + * The athenah-ai pipeline produces a `.ai.md` companion for every + * documented source file (e.g., `Slice.h` -> `Slice.h.ai.md`). When present, + * it is high-signal prose describing the file's purpose, design, and + * non-obvious behavior — the agent should use it as the authoritative + * source of intent. + */ +async function readAiContext(absPath: string): Promise { + const aiPath = `${absPath}.ai.md`; + if (!existsSync(aiPath)) return null; + return await readFile(aiPath, 'utf8'); +} + /** * Document a single file by running the documentation agent against it. */ @@ -55,13 +71,19 @@ async function documentFile(absPath: string): Promise { console.log(`\n=== Documenting: ${relPath} ===`); const systemPrompt = await loadSystemPrompt('document-file', relPath); + const aiContext = await readAiContext(absPath); + const aiContextBlock = + aiContext === null + ? '' + : `\n\n## Authoritative AI Context (${relPath}.ai.md)\n\nThe following is high-signal prose describing this file's purpose, design,\nand non-obvious behavior. Treat it as the source of truth for intent and\nbehavior. Your job is to translate this into structured Doxygen \`/** */\`\ncomments on the actual declarations.\n\n---\n\n${aiContext}\n---`; + const userPrompt = `Add Doxygen documentation to: ${relPath} The file is rooted at ${XRPLD_ROOT}. Use the Read tool to read it, the Edit tool to add documentation, and Glob/Grep to find related tests or callers when needed. -Do not modify any code logic — only add documentation comments.`; +Do not modify any code logic — only add documentation comments.${aiContextBlock}`; const result = query({ prompt: userPrompt, diff --git a/.github/scripts/doc-agent/src/index.ts b/.github/scripts/doc-agent/src/index.ts index 491dbd84c0..962aba599a 100644 --- a/.github/scripts/doc-agent/src/index.ts +++ b/.github/scripts/doc-agent/src/index.ts @@ -7,9 +7,11 @@ * doc-agent document include/xrpl/basics/ * doc-agent review develop..HEAD * doc-agent review --pr 1234 + * doc-agent regen-skills protocol */ import { documentTarget } from './document.js'; +import { regenSkills } from './regen-skills.js'; import { reviewDiff } from './review.js'; const USAGE = ` @@ -19,6 +21,8 @@ Usage: doc-agent document Add Doxygen documentation doc-agent review .. Detect doc drift in range doc-agent review --pr Detect doc drift for a PR + doc-agent regen-skills Regenerate docs/skills/soul/.md + from sibling .ai.md files Environment: ANTHROPIC_API_KEY (required) Anthropic API key @@ -58,6 +62,13 @@ async function main(): Promise { return; } + if (mode === 'regen-skills') { + const moduleName = args[0]; + if (moduleName === undefined) printUsageAndExit(1); + await regenSkills(moduleName); + return; + } + console.error(`Unknown mode: ${mode}`); printUsageAndExit(1); } diff --git a/.github/scripts/doc-agent/src/prompt-loader.ts b/.github/scripts/doc-agent/src/prompt-loader.ts index 05eb63b9f4..55ab194d4e 100644 --- a/.github/scripts/doc-agent/src/prompt-loader.ts +++ b/.github/scripts/doc-agent/src/prompt-loader.ts @@ -24,7 +24,7 @@ export async function loadSystemPrompt(promptName: string, sourcePath: string): return basePrompt; } - const skillPath = resolve(SKILLS_DIR, 'soul', skillFile); + const skillPath = resolve(SKILLS_DIR, skillFile); if (!existsSync(skillPath)) { return basePrompt; } diff --git a/.github/scripts/doc-agent/src/regen-skills.ts b/.github/scripts/doc-agent/src/regen-skills.ts new file mode 100644 index 0000000000..7ccc5e1580 --- /dev/null +++ b/.github/scripts/doc-agent/src/regen-skills.ts @@ -0,0 +1,146 @@ +/** + * Regen-skills mode: rebuild a module's skill file from ai.md inputs. + * + * For a given module (e.g. `protocol`, `ledger`, `consensus`), collect all + * `.ai.md` files under the matching source paths and ask the agent to + * produce an updated `docs/skills/.md`. + */ + +import { existsSync, readdirSync, statSync } from 'node:fs'; +import { readFile, writeFile } from 'node:fs/promises'; +import { join, relative, resolve } from 'node:path'; +import { query } from '@anthropic-ai/claude-agent-sdk'; +import { MODEL, MODULE_SKILL_MAP, PROMPTS_DIR, SKILLS_DIR, XRPLD_ROOT } from './config.js'; + +interface AiFile { + readonly sourcePath: string; + readonly content: string; +} + +/** Resolve which source-tree prefixes feed a given skill file. */ +function prefixesForSkill(skillFile: string): string[] { + return Object.entries(MODULE_SKILL_MAP) + .filter(([, mapped]) => mapped === skillFile) + .map(([prefix]) => prefix); +} + +/** Walk a directory and collect all sibling .ai.md files. */ +function collectAiFiles(prefix: string): string[] { + const absDir = resolve(XRPLD_ROOT, prefix); + if (!existsSync(absDir) || !statSync(absDir).isDirectory()) return []; + + const results: string[] = []; + const walk = (dir: string): void => { + for (const entry of readdirSync(dir, { withFileTypes: true })) { + const full = join(dir, entry.name); + if (entry.isDirectory()) { + walk(full); + } else if (entry.isFile() && entry.name.endsWith('.ai.md')) { + results.push(full); + } + } + }; + walk(absDir); + return results; +} + +async function loadAiFiles(absPaths: readonly string[]): Promise { + const files: AiFile[] = []; + for (const absPath of absPaths) { + const content = await readFile(absPath, 'utf8'); + files.push({ + sourcePath: relative(XRPLD_ROOT, absPath).replace(/\.ai\.md$/, ''), + content, + }); + } + return files; +} + +/** + * Regenerate the skill file for a given module name. + * + * @param moduleName - The skill file name without extension (e.g. "protocol", + * "ledger"). Must match a key in the MODULE_SKILL_MAP value set. + */ +export async function regenSkills(moduleName: string): Promise { + const skillFile = `${moduleName}.md`; + const prefixes = prefixesForSkill(skillFile); + + if (prefixes.length === 0) { + throw new Error( + `Unknown module: ${moduleName}. Valid modules: ${Array.from(new Set(Object.values(MODULE_SKILL_MAP).filter((v): v is string => v !== null))).join(', ')}`, + ); + } + + console.log(`Regenerating skill: ${skillFile}`); + console.log(` Source prefixes: ${prefixes.join(', ')}`); + + const aiPaths = prefixes.flatMap((prefix) => collectAiFiles(prefix)); + if (aiPaths.length === 0) { + console.warn(' No .ai.md files found for this module. Skipping.'); + return; + } + console.log(` Found ${aiPaths.length} .ai.md file(s)`); + + const aiFiles = await loadAiFiles(aiPaths); + const skillPath = resolve(SKILLS_DIR, skillFile); + const existingSkill = existsSync(skillPath) + ? await readFile(skillPath, 'utf8') + : '(no existing skill file — create a new one)'; + + const systemPrompt = await readFile(resolve(PROMPTS_DIR, 'regen-skill.md'), 'utf8'); + + const aiBlocks = aiFiles + .map((f) => `\n### \`${f.sourcePath}\`\n\n${f.content}`) + .join('\n\n---\n'); + + const userPrompt = `Regenerate the skill file: \`docs/skills/${skillFile}\` + +## Existing skill content + +${existingSkill} + +## AI context files for this module + +${aiBlocks} + +Produce the new complete skill file content as your final message.`; + + let response = ''; + const result = query({ + prompt: userPrompt, + options: { + model: MODEL, + systemPrompt, + cwd: XRPLD_ROOT, + allowedTools: ['Read', 'Glob', 'Grep'], + permissionMode: 'acceptEdits', + }, + }); + + for await (const message of result) { + if (message.type === 'assistant') { + const content = message.message?.content; + if (Array.isArray(content)) { + for (const block of content) { + if (block.type === 'text') { + response += block.text; + } + } + } + } + if (message.type === 'result') { + const cost = message.total_cost_usd?.toFixed(4) ?? '?'; + console.log(` [Cost: $${cost}]`); + } + } + + const trimmed = response.trim(); + if (trimmed.length === 0) { + console.error(' Agent returned empty response — skill file not updated.'); + return; + } + + await writeFile(skillPath, `${trimmed}\n`); + console.log(` Wrote: ${relative(XRPLD_ROOT, skillPath)}`); +} diff --git a/.gitignore b/.gitignore index 6bd34ece04..f01378b61a 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,10 @@ # .gitignore # cspell: disable +# AI-generated documentation source (temporary, used by doc-agent +# during the initial documentation pass; removed once docs are merged). +*.ai.md + # Macintosh Desktop Services Store files. .DS_Store diff --git a/docs/skills/soul/consensus.md b/docs/skills/consensus.md similarity index 100% rename from docs/skills/soul/consensus.md rename to docs/skills/consensus.md diff --git a/docs/skills/soul/cryptography.md b/docs/skills/cryptography.md similarity index 100% rename from docs/skills/soul/cryptography.md rename to docs/skills/cryptography.md diff --git a/docs/skills/soul/ledger.md b/docs/skills/ledger.md similarity index 100% rename from docs/skills/soul/ledger.md rename to docs/skills/ledger.md diff --git a/docs/skills/soul/nodestore.md b/docs/skills/nodestore.md similarity index 100% rename from docs/skills/soul/nodestore.md rename to docs/skills/nodestore.md diff --git a/docs/skills/soul/peering.md b/docs/skills/peering.md similarity index 100% rename from docs/skills/soul/peering.md rename to docs/skills/peering.md diff --git a/docs/skills/protocol.md b/docs/skills/protocol.md new file mode 100644 index 0000000000..76ccd24233 --- /dev/null +++ b/docs/skills/protocol.md @@ -0,0 +1,271 @@ +# Protocol and Serialization + +The protocol layer defines XRPL's wire format, type system, and validation rules. It owns the canonical binary encoding required for signatures and consensus, the macro-driven registries for features/transactions/ledger entries, and the typed object model (`STBase` hierarchy) that every transaction and ledger object inhabits. + +## Layered Type System + +``` +Asset = std::variant ← unified asset identity (XRP/IOU/MPT) + Issue = (Currency, AccountID) ← XRP iff currency==zero + MPTIssue = wraps MPTID (192-bit: seq32 || account160) + +Amount types (lean, runtime polymorphic via Asset): + XRPAmount = int64 drops ← integral, no asset + IOUAmount = (mantissa, exponent) ← 15-digit decimal floating point + MPTAmount = int64 ← integral, no asset + +STAmount = unified wire/serialized form holding Asset + value + - holds(), holds(), native(), integral() + - canonicalize() normalizes (mantissa, exponent) per asset rules +``` + +Conversion utilities in `AmountConversions.h`: `toSTAmount`, `toAmount`, `getAsset`. Lean→STAmount is implicit-friendly; STAmount→lean is explicit (`get<>` throws on type mismatch). + +## Key Invariants + +- **Canonical field ordering:** sort by `(SerializedTypeID << 16) | fieldValue`, NOT by raw Field ID bytes — wrong sort breaks signatures +- **Field ID encoding:** 1–3 bytes; both type and field codes <16 → single byte `(type<<4)|name` +- **Hash domain separation:** every signable payload prepends a 4-byte `HashPrefix` (`STX\0`, `SMT\0`, `VAL\0`, `STX\0`, `BCH\0`, `CLM\0`, `LWR\0`, etc.) — never share hashes across domains +- **STObject access semantics:** `obj[sfFoo]` throws `FieldErr` if absent; `obj[~sfFoo]` returns `std::optional` +- **Amendment IDs are deterministic:** `featureFoo == sha512Half(Slice("Foo"))` — never change a feature name +- **Singletons everywhere:** `SField`, `LedgerFormats`, `TxFormats`, `InnerObjectFormats`, `Permission`, `Feature` registry all use Meyer's singletons; registration completes before `main()` via static init +- **Multi-sign signers MUST be sorted ascending by AccountID** (no duplicates, count in [1,32], cannot include tx account) +- **`vfFullyCanonicalSig` always set** by signer; verifiers normalize ECDSA S to low form +- **Amendment-gated arithmetic:** `getSTNumberSwitchover()` is a `LocalValue` (per-coroutine) selecting legacy vs `Number`-based normalization in `IOUAmount`/`STAmount` + +## Macro-Driven Registries (X-Macros) + +Single source of truth for each registry; `.macro` files included multiple times with redefined macros to generate enum, declarations, and definitions. + +| Macro file | Used for | Add requires | +|---|---|---| +| `features.macro` | `XRPL_FEATURE`, `XRPL_FIX`, `XRPL_RETIRE_*` | Bump `numFeatures` in `Feature.h` | +| `transactions.macro` | `TRANSACTION(tag, value, name, delegable, amendment, privileges, fields)` | nothing — count derived | +| `ledger_entries.macro` | `LEDGER_ENTRY(tag, value, name, rpcName, fields)` + `LEDGER_ENTRY_DUPLICATE` for name collisions | nothing | +| `sfields.macro` | `TYPED_SFIELD(name, TYPE, code)`, `UNTYPED_SFIELD` | nothing | +| `permissions.macro` | `PERMISSION(name, txType, value)` (granular permissions ≥65537) | nothing | + +Pattern uses `#pragma push_macro/pop_macro` to protect macro names. `UNWRAP(...)` strips outer parens around field-list initializers so commas don't confuse the preprocessor. `LEDGER_ENTRY_DUPLICATE` exists because `DepositPreauth` is both a transaction type and ledger entry type — `JSS()` can't emit the same string twice. + +## Field Identity (`SField`) + +- **Field code** = `(SerializedTypeID << 16) | fieldValue` — packs type family and per-type index; canonical sort key +- `SField` instances are immutable singletons created at static init via `private_access_tag_t` (only definable inside `SField.cpp`) +- `TypedField` adds compile-time payload type; `OptionaledField` via `operator~(sfField)` +- Metadata flags (`fieldMeta`): `sMD_ChangeOrig`, `sMD_ChangeNew`, `sMD_DeleteFinal`, `sMD_Create`, `sMD_Always`, `sMD_BaseTen` (decimal display), `sMD_PseudoAccount`, `sMD_NeedsAsset` (drives `STTakesAsset` association) +- `IsSigning::no` excludes fields from signing hash (`sfTxnSignature`, `sfSigners`, `sfSignature`, etc.) +- `isBinary()` ⇔ `fieldValue<256` (wire-representable); `isDiscardable()` ⇔ `fieldValue>256` (JSON-only, e.g., `sfHash`, `sfIndex`) + +## Wire Format Reference + +| Item | Encoding | +|---|---| +| XRP STAmount | 8 bytes; bit63=0, bit62=sign(1=pos), 62-bit value | +| MPT STAmount | 8 bytes header (bit63=0, bit61=1) + 192-bit MPTID | +| IOU STAmount | bit63=1, bit62=sign, 8-bit (offset+97), 54-bit mantissa, +20B currency, +20B issuer | +| AccountID | 20 bytes, VL-prefixed when standalone (`STAccount` mimics `STBlob` wire format) | +| STArray | elements between markers; ends with `STI_ARRAY,1` (`0xf1`) | +| STObject | fields in canonical order; ends with `STI_OBJECT,1` (`0xe1`) | +| VL prefix | 1 byte (0–192), 2 bytes (193–12480), 3 bytes (12481–918744); else `std::overflow_error` | +| STIssue | 160-bit currency; if zero → XRP; if next 160 = `noAccount()` → MPT (then 32-bit seq); else IOU issuer | +| LP token currency | byte0 = `0x03`; bytes 1-19 = `sha512Half(min(asset1,asset2), max(asset1,asset2))` low bits | +| Order book quality | `(exponent+100) << 56 \| mantissa`; embedded in last 8 bytes of directory key (big-endian) so SHAMap order = price order | +| NFTokenID (256-bit) | flags(2) + transferFee(2) + issuer(20) + cipheredTaxon(4) + serial(4); low 96 bits = page sort key | + +## Canonical Hashes + +``` +TXN → transactionID SND → txNode (with metadata) +MLN → leafNode MIN → innerNode +LWR → ledgerMaster STX → txSign (single-sig) +SMT → txMultiSign VAL → validation +PRP → proposal MAN → manifest +CLM → paymentChannelClaim BCH → batch +``` + +All hashes use `sha512Half` (first 256 bits of SHA-512). `HashPrefix` constants are protocol-immutable. + +## STObject and STVar + +- `STObject` stores `std::vector`; iterators expose `STBase const&` via transform iterator +- `STVar` is type-erased with 72-byte inline buffer (small-object optimization); larger types heap-allocate +- `copy()`/`move()` virtuals on every ST type delegate to `STBase::emplace()` for placement-new into `STVar`'s buffer +- **Two modes:** + - **Free** (`mType==nullptr`): linear field scan, accepts any field + - **Templated** (`mType` set): O(1) field lookup via `SOTemplate::indices_`, template enforced +- `applyTemplate()` validates after deserialization; `set(SOTemplate)` initializes empty object with template +- Deserialization depth capped at 10 to prevent stack exhaustion + +### Proxy Access Pattern + +```cpp +auto amt = tx[sfAmount]; // ValueProxy: throws FieldErr if absent +auto dst = tx[~sfDestination]; // OptionalProxy: std::optional +tx[sfFlags] = 0; // proxy.assign() — soeDEFAULT zero is silently removed +tx[~sfDestTag] = std::nullopt; // remove field (only valid for soeOPTIONAL) +``` + +Proxies forbid removing `soeREQUIRED` or `soeDEFAULT` fields. + +## SOEStyle (Field Presence) + +| Style | Meaning | +|---|---| +| `soeREQUIRED` | must be present | +| `soeOPTIONAL` | may be absent; if present, may carry default value | +| `soeDEFAULT` | may be absent; if present, must NOT equal default — auto-removed when assigned default | + +`SOETxMPTIssue` flag on amount/issue fields: `soeMPTSupported`, `soeMPTNotSupported`, `soeMPTNone`. Omitting `soeMPTSupported` silently rejects MPT amounts in that field. + +## Common Bug Patterns + +- Adding to `transactions.macro` without adding to `sfields.macro` → silent serialization failures +- Forgetting to bump `numFeatures` after `XRPL_FEATURE` → out-of-bounds access in `FeatureBitset` +- Hand-built binary blobs in non-canonical field order → signature verification failures +- Omitting `soeMPTSupported` on amount field → MPT payments silently rejected +- Mutating `sfTransactionType` inside `STTx` assembler callback → `LogicError` (caught at startup) +- Storing `STBase` subclasses directly in `std::vector` → field names lost on copy-assignment slide; use `STArray`/`STObject` instead +- Storing `Currency` as `"XRP"` ISO code (`badCurrency()`) instead of zero → silently rejected; `to_currency()` legacy returns `badCurrency()` rather than failing +- Forgetting to call `associateAsset(sle, asset)` near end of `doApply()` for vault/loan transactors → unrounded `STNumber` values +- Returning `tec*` from `preflight()` → `NotTEC` type prevents this at compile time (would allow fee theft on unsigned tx) + +## Key Patterns + +### Amendment Registration + +```cpp +// In features.macro: +XRPL_FEATURE(MyFeature, Supported::yes, VoteBehavior::DefaultNo) +XRPL_FIX (MyBugFix, Supported::yes, VoteBehavior::DefaultNo) +XRPL_RETIRE_FEATURE(OldFeature) // code removed; remains registered for ledger compat +``` + +Lifecycle: `Supported::no/DefaultNo` → `Supported::yes/DefaultNo` → (rare) `DefaultYes` for critical fixes. **Never** revert `Supported::yes` to `no` (would amendment-block existing nodes). + +### NotTEC vs TER + +```cpp +NotTEC preflight(...); // can only return tel/tem/tef/ter/tes (no tec) +TER doApply(...); // can return any code including tec* +``` + +`TERSubset` enforces this at compile time via `enable_if`. `TERtoInt(v)` is the authorized free-function conversion (member `explicit operator` would be too permissive in initializer contexts). + +### Signing/Verifying + +```cpp +sign(st, HashPrefix::txSign, KeyType::secp256k1, sk); // writes sfSignature +verify(st, HashPrefix::txSign, pubKey); // returns bool + +// Multi-sign optimization (shared body, per-signer suffix): +auto s = startMultiSigningData(obj); +finishMultiSigningData(signerAccountID, s); // append signer ID to shared payload +``` + +`addWithoutSigningFields()` excludes signature fields from the signed payload — this is what breaks the circularity. + +### STNumber + STTakesAsset + +```cpp +// Vault/Loan/LoanBroker fields use STNumber (no asset embedded). +// In doApply(), after all mutations: +associateAsset(*sle, vaultAsset); // rounds all sMD_NeedsAsset fields, removes zero-defaults +``` + +## Critical Files + +### Foundations +- `include/xrpl/protocol/SField.h`, `src/libxrpl/protocol/SField.cpp` — field registry, X-macro expansion +- `include/xrpl/protocol/Feature.h`, `src/libxrpl/protocol/Feature.cpp` — `numFeatures`, `FeatureBitset`, registration +- `include/xrpl/protocol/Rules.h` — per-coroutine active amendment set; `isFeatureEnabled()` queries thread-local +- `include/xrpl/protocol/HashPrefix.h` — protocol-immutable domain separators + +### Macro Tables (single sources of truth) +- `include/xrpl/protocol/detail/features.macro` +- `include/xrpl/protocol/detail/transactions.macro` +- `include/xrpl/protocol/detail/ledger_entries.macro` +- `include/xrpl/protocol/detail/sfields.macro` +- `include/xrpl/protocol/detail/permissions.macro` + +### Type System Roots +- `STBase.h/cpp` — polymorphic root, `emplace()` SOO helper, `JsonOptions` +- `STObject.h/cpp` — heterogeneous container, proxy system, template enforcement +- `STVar` (`detail/STVar.h`) — 72-byte inline variant, depth guard at 10 +- `SOTemplate.h/cpp` — schema with O(1) field index; move-only + +### Format Registries +- `TxFormats.h/cpp`, `LedgerFormats.h/cpp`, `InnerObjectFormats.h/cpp` — all inherit `KnownFormats` with `forward_list` (pointer-stable) + dual flat_maps + +### Amount/Asset Stack +- `Asset.h/cpp` — variant of Issue/MPTIssue; `visit()`, `equalTokens()`, `BadAsset` sentinel +- `Issue.h/cpp`, `MPTIssue.h/cpp` — XRP/IOU and MPT identity +- `STAmount.h/cpp` — unified serialized amount; `canMul`/`canAdd`/`canSubtract` safety checks; `mulRound`/`mulRoundStrict` (legacy vs precise rounding) +- `IOUAmount.h/cpp`, `XRPAmount.h`, `MPTAmount.h/cpp` — lean representations +- `Number` (in `xrpl/basics/`) — high-precision arithmetic; `MantissaRange::large` enabled by SingleAssetVault/LendingProtocol amendments + +### Cryptography +- `PublicKey.h/cpp` — 33-byte unified format (0xED prefix for Ed25519); `ECDSACanonicality` enum (canonical vs fullyCanonical), libsecp256k1 normalization +- `SecretKey.h/cpp` — `secure_erase` in dtor; deleted `==`/`<<`; XRPL-specific secp256k1 derivation via `Generator` +- `Seed.h/cpp` — 128-bit; `parseGenericSeed()` cascades hex→base58→RFC1751→passphrase, rejecting other key types first +- `digest.h` — `sha512Half`, `sha512_half_hasher_s` (secure erase variant) +- `tokens.h/cpp` — Base58Check; fast path uses base 58^10 intermediate (10–15× speedup, gated on non-MSVC for `__int128`) + +### Wire I/O +- `Serializer.h/cpp` — accumulator; `addVL`, `addFieldID`, big-endian integers, `getSHA512Half()` +- `SerialIter` — non-owning forward cursor over a byte buffer; throws on underrun +- `Sign.h/cpp` — `sign`/`verify` with HashPrefix prepended to `addWithoutSigningFields()` output + +### Higher-Level Objects +- `STTx.h/cpp` — caches `tid_` and `tx_type_`; `passesLocalChecks` (memos, pseudo-tx, MPT support, batch nesting); `sterilize()` round-trip +- `STLedgerEntry.h/cpp` (alias `SLE`) — typed ledger object; `thread()` updates `sfPreviousTxnID`; `isThreadedType()` gated by `fixPreviousTxnID` +- `STValidation.h/cpp` — lazy `valid_` cache; `mTrusted` separate from validity; `lookupNodeID` callback decouples manifest system + +### Indexes and Keys +- `Indexes.h/cpp` — `keylet::*` factories with `LedgerNameSpace` tagged hashing; `keylet::quality()` embeds 64-bit quality in last 8 bytes (big-endian) +- `Keylet.h/cpp` — type-tagged `(uint256, LedgerEntryType)`; `ltANY` wildcard, `ltCHILD` rejects directories +- NFT pages: composite keys (high 160 = owner AccountID, low 96 = token range); `nft::pageMask` is the boundary + +### Validation Helpers (return NotTEC, preflight-time) +- `AMMCore.h/cpp` — `invalidAMMAsset`, `invalidAMMAssetPair`, `invalidAMMAmount`; `ammLPTCurrency()` uses canonical `std::minmax` +- `Permissions.h/cpp` — singleton; `isDelegable()` checks granular vs transaction-level (``, `STInteger`, `STBlob`, `STArray`, `STVector256`, `STCurrency`, `STPathSet`, `STXChainBridge`, `STNumber` (asset-contextual) +- `Quality.h/cpp` — inverted encoding (lower uint64 = higher quality); `ceil_in`/`ceil_out` proportional scaling; `_strict` variants honor Number rounding mode +- `QualityFunction.h/cpp` — linear `q(out)=m*out+b`; AMMTag (slope from pool) vs CLOBLikeTag (m=0); `combine()` for multi-step strands +- `XChainAttestations.h/cpp` — Attestations:: namespace (full, with signature) vs xrpl:: (stored); `match()` returns three-state `AttestationMatch` + +### Pseudo-Account Fields (sMD_PseudoAccount) +- `sfAMMID`, `sfVaultID`, `sfLoanBrokerID` — 256-bit hash representing a synthesized account address + +## Numeric Encoding Reference + +``` +IOU canonical: mantissa ∈ [10^15, 10^16), exponent ∈ [-96, +80] + zero = (mantissa=0, exponent=-100) — sorts below smallest positive +XRP max: cMaxNativeN = 10^17 drops (100 billion XRP) +MPT max: maxMPTokenAmount = INT64_MAX = 0x7FFFFFFFFFFFFFFF +Transfer rate: Rate{value} where value/1_000_000_000 = 1.0 (parityRate = 1:1) +NFT transfer fee: uint16 basis points (0–50000), convert via nft::transferFeeAsRate (×10000) +``` + +## Protocol-Stable Constants (NEVER CHANGE) + +- `LedgerEntryType` numeric values (in ledger objects) +- `TxType` numeric values (in signed transactions) +- `SerializedTypeID` and `SField` codes (in serialized fields) +- `LedgerNameSpace` discriminator characters (in keylet derivation) +- `HashPrefix` enum values (in signature/hash domain separation) +- `error_code_i` numeric values (clients depend on them; append-only) +- `FLAG_LEDGER_INTERVAL = 256` (drives consensus timing) +- `INITIAL_XRP = 100B × 10^6 drops` (validated by `static_assert` against `Number::maxRep`) +- NFT taxon LCG constants (`384160001 * seq + 2459`) +- All flag bit values (`tf*`, `lsf*`, `asf*`) + +Changing any requires an amendment with explicit detection logic for old/new behavior. diff --git a/docs/skills/soul/rpc.md b/docs/skills/rpc.md similarity index 100% rename from docs/skills/soul/rpc.md rename to docs/skills/rpc.md diff --git a/docs/skills/soul/shamap.md b/docs/skills/shamap.md similarity index 100% rename from docs/skills/soul/shamap.md rename to docs/skills/shamap.md diff --git a/docs/skills/soul/protocol.md b/docs/skills/soul/protocol.md deleted file mode 100644 index ff41015ffc..0000000000 --- a/docs/skills/soul/protocol.md +++ /dev/null @@ -1,64 +0,0 @@ -# Protocol and Serialization - -Macro-driven system for defining features, transactions, ledger entries, and serialized fields. Canonical binary format is required for signatures and consensus. - -## Key Invariants - -- Fields are sorted by (type code, field code) for canonical serialization; sorting by Field ID bytes produces WRONG results -- Field ID encoding: 1-3 bytes depending on type/field code values (both < 16 = 1 byte) -- Signing hash prefix: `0x53545800` for single-signing, `0x534D5400` for multi-signing -- `STObject[sfFoo]` returns value or default; `STObject[~sfFoo]` returns optional (nothing if absent) -- All ST types inherit from `STBase`; `STVar` provides type-erased storage with stack/heap allocation - -## Macro System - -- `XRPL_FEATURE(name, supported, vote)` / `XRPL_FIX` / `XRPL_RETIRE` in `features.macro` -- `TRANSACTION(tag, value, class, delegation, fields)` in `transactions.macro` -- `LEDGER_ENTRY(type, code, class, name, fields)` in `ledger_entries.macro` -- `TYPED_SFIELD(name, TYPE, code)` in `sfields.macro` -- Adding any new definition requires updating the count in the corresponding header - -## Serialization Format - -- XRP Amount: 8 bytes, MSB=0, next bit=1 for positive, remaining 62 bits = value -- Token Amount: 8 bytes mantissa/exponent + 20 bytes currency + 20 bytes issuer -- AccountID: 20 bytes, length-prefixed when top-level -- STArray: elements between start (`0xf0`) and end (`0xf1`) markers -- STObject: fields in canonical order between start (`0xe0`) and end (`0xe1`) markers -- Length prefixing: 1 byte (0-192), 2 bytes (193-12480), 3 bytes (12481-918744) - -## Common Bug Patterns - -- Adding a field to `transactions.macro` without adding it to `sfields.macro` causes silent serialization failures -- Forgetting to increment `numFeatures` after adding to `features.macro` causes out-of-bounds access -- Non-canonical field ordering in hand-built binary blobs causes signature verification failures -- `soeMPTSupported` flag on amount fields enables MPT token support; omitting it silently rejects MPT payments - -## Key Patterns - -### Amendment Registration -```cpp -// In features.macro — REQUIRED format: -XRPL_FEATURE(MyNewFeature, Supported::yes, VoteBehavior::DefaultNo) -XRPL_FIX (MyBugFix, Supported::yes, VoteBehavior::DefaultNo) -// MUST also increment numFeatures in Feature.h — omitting causes OOB access -``` - -### STObject Field Access -```cpp -// Safe: optional access — returns std::optional, never throws -if (auto const val = tx[~sfAmount]) - use(*val); - -// Throws if field is absent — only safe when preflight guarantees presence -auto const amount = tx[sfAmount]; -``` - -## Key Files - -- `include/xrpl/protocol/detail/features.macro` - amendment definitions -- `include/xrpl/protocol/detail/transactions.macro` - transaction types -- `include/xrpl/protocol/detail/ledger_entries.macro` - ledger objects -- `include/xrpl/protocol/detail/sfields.macro` - field definitions -- `include/xrpl/protocol/Feature.h` - `numFeatures` constant -- `src/libxrpl/protocol/STObject.cpp` - core serialized object diff --git a/docs/skills/soul/sql.md b/docs/skills/sql.md similarity index 100% rename from docs/skills/soul/sql.md rename to docs/skills/sql.md diff --git a/docs/skills/soul/test.md b/docs/skills/test.md similarity index 100% rename from docs/skills/soul/test.md rename to docs/skills/test.md diff --git a/docs/skills/soul/transactors.md b/docs/skills/transactors.md similarity index 100% rename from docs/skills/soul/transactors.md rename to docs/skills/transactors.md diff --git a/docs/skills/soul/websockets.md b/docs/skills/websockets.md similarity index 100% rename from docs/skills/soul/websockets.md rename to docs/skills/websockets.md