retroSoC/scripts/README.md

# Build and Quality Scripts

This directory contains the Python implementation of setup, dependency locking,
build manifests, flow execution, simulation checks, regression orchestration,
quality checks, metrics, packaging, and cleanup.

`apu_mcasm.py`, `apu_isa.py`, `apu_interpreter.py`, and `apu_primitives.py` are
the frozen APU microcode V1 assembler/ABI model, P3/P4/P5 target interpreters, and
bit-accurate P4 primitive model. They accept only the APU instruction language
and deliberately have no C, ELF, RV32, dynamic linking, or runtime
code-generation path.

`setup_apu_reference.py` installs the checksum-pinned libFLAC and official
FLAC corpus as host-only P5 verification inputs and builds libFLAC with Ogg
disabled below the selected build variant. These inputs never enter firmware,
RTL, or shipped APU microcode.
`qualify_apu_p5_corpus.py` enumerates that exact corpus, records source and
decoded-PCM hashes, and classifies each file against the frozen P5 profile.
`run_apu_p5_corpus_rtl.py` then runs every classified file through one shared,
verification-only `apb4_apu` fixture compiled for Icarus and Verilator. It
records exact production result/accounting and PCM hashes in the same manifest;
per-case results are resumable and neither the fixture nor corpus enters a
product filelist.

Scripts are part of the build contract. Prefer existing helpers over ad-hoc
shell behavior, preserve structured JSON results, and keep setup/download
behavior controlled by `dependencies/dependencies.lock.json`.

`generate_apu_kws_rtl_constants.py` also emits the frozen APU-P9 APUC image
and 15-bank layout manifest. `apu_kws_coeff.py` owns the independent APUC 1.0
schema and release validator; generated assets remain below the selected
variant's `apu/coefficients/` directory.
`apu_p9_evidence.py` initializes or assembles the six frozen P9 evidence files;
missing qualifying runs stay explicitly `unrun` rather than becoming passes.
`apu_p9_memory_ab.py` performs the like-for-like inferred-memory, macro-count,
synthesis-time and process-tree peak-RSS comparison from two explicit variant
roots; it never creates or switches Git worktrees itself.

The frozen P9 qualification entry points are explicit and do not turn smoke
runs into release evidence:

```sh
make CONFIG=configs/ci/ihp130.mk setup-apu-kws-reference apu-p9-coefficients apu-p9-evidence
python3 scripts/run_apu_p7_kws_rtl.py \
  --build-dir build/<candidate>/apu/p9/accuracy --profile ihp130-p9
python3 scripts/run_apu_p7_concurrent_rtl.py \
  --build-dir build/<candidate>/apu/p9/concurrent --seconds 60
make CONFIG=configs/ci/ihp130.mk apu-p9-memory-ab \
  APU_P9_BASELINE_ROOT=build/<baseline> APU_P9_CANDIDATE_ROOT=build/<candidate>
```

Both A/B roots must come from clean committed revisions; the baseline is the
frozen pre-P9 revision and both configurations must enable P7 with the same
IHP130 tool, PDK, recipe, and dependency inputs. Assemble the six final reports
with `scripts/apu_p9_evidence.py assemble`; a `passed` input is accepted only
when its report-specific checks and named source artifacts are complete.

`setup_npu_reference.py` prepares the locked KWS/VWW inputs and the existing
TensorFlow/gemmlowp reference sources. `npu_framework_reference.py` builds the
host-only adapter in `tests/cpp/npu_framework_reference.cc`; its integer kernels
consume original tensor constants and quantization, never NPU packed artifacts.
Only raw model parsing is shared with the NPU compiler. Run
`make CONFIG=configs/ci/ihp130.mk npu-p0-qualify` after
`make CONFIG=configs/ci/ihp130.mk setup-npu-reference` to compare both complete
1000-input corpora against the Python graph reference and compiled executor.
The VWW archive is fetched from the ordered HTTPS URLs in the dependency lock;
every mirror must produce the same locked SHA-256 before extraction.
The selected variant's `npu/p0/` directory retains oracle build provenance,
per-input golden tensors, three-way hashes, logs and `qualification-p0.json`.
The runner uses at most 16 workers (`JOBS`); missing sources/tools, numerical
differences and incomplete runs cannot pass. A direct `--limit` invocation is
debug-only and returns a nonzero status even when its selected cases match.

`npu_compiler.py` is the P5 production entry point. It accepts one static
batch-one INT8 TFLite v3 subgraph, rejects unsupported placement with an
operator-specific diagnostic, and emits ABI-1 descriptors, packed constants,
`npu.json`, and a model-prefixed freestanding C plan. The generated plan uses
caller-owned aligned descriptor/arena storage, checked physical relocation,
bounded HAL waits, 64-byte Zicbom maintenance when compiled for HP, and the
model-derived terminal integer Softmax. Generated files remain below the
selected build variant. `make CONFIG=configs/ci/ihp130.mk npu-p5-deployments`
builds the locked KWS and VWW packages reproducibly.

`qualify_npu_p5.py` builds one production NPU fixture per simulator and runs
the frozen ten-input sets from both manifests. Every accepted output byte is
checked in actual tile-write order, so later arena reuse cannot hide an
intermediate mismatch. Invoke it through
`make CONFIG=configs/ci/ihp130.mk npu-p5-rtl`; missing tools, inputs, terminal
markers, or any layer write fail the required run.

`npu_p5_report.py` fails closed while assembling P5 evidence. It verifies P0,
compiler/parity and embedded-C/HAL results; exact dual-simulator case counts;
source, trace, log and deployment hashes; LP/HP build manifests and firmware;
cross-profile KWS package identity; and zero skipped/rejected cases. Invoke it
through `npu-p5-report` with the retained LP/HP variant roots and config
digests. The generated report records its fully expanded command.

NPU-P6 qualification is split into auditable drivers. `npu_p6_corpus.py`
builds deterministic CRC-protected 100-case KWS/VWW shards from retained P0
goldens. `run_npu_p6_verilator.py` builds the existing `hp_boot` composition,
runs all shards on one PRODUCT Verilator model, and aggregates architectural
cycles and NPU counters. `run_npu_p6_netlist.py` runs macro-aware isolated
synthesis/STA and four directed synthesized-block transactions covering dense,
depthwise, rejection and accumulation-overflow behavior;
`run_npu_p6_physical.py` invokes the unchanged full PRODUCT Yosys/OpenSTA and
warning/metric gates. `run_npu_p6_regression.py` retains PR/nightly verdicts,
and `npu_p6_report.py` requires current-revision PASS evidence for all of
NPU-V015 through NPU-V018. None of these scripts treats FPGA execution or host
elapsed time as mandatory performance evidence.

`development_environment.py` is the shared Docker, Nix, and manual bootstrap
entry point. It installs only the checksum-verified open-source tool bundles and
hash-pinned Python dependencies; project-local PDK and source setup remains under
the existing Make targets.

`publish_fatfs_artifact.sh` is the manual release helper for the lock-pinned
FatFs R0.16 archive. It verifies the archive, GitHub authentication, release
absence, and the published asset checksum before reporting the checksum-pinned
URL.

`parse_performance_log.py` converts the `APP=benchmark` UART `PERF` records
into `meta/performance.json`. The parser requires the terminal
`PERF_BENCHMARK_PASS` marker. `parse_coremark_log.py` converts the fixed
SRAM CoreMark quick report into `meta/coremark.json`; it requires one valid
`COREMARK_RESULT` record and `COREMARK_PASS`. Both reports complement, rather
than replace, the common `SIM_TEST_PASS` simulation verdict.

`generate_vexiiriscv.py` verifies the locked VexiiRiscv revision, rejects local
source changes before and after generation, and generates the fixed HP core
below `build/`; generated RTL is never tracked.
`setup_hp_linux.py` installs the locked OpenSBI, Linux, and Buildroot revisions
and can resume a checkout left without `HEAD` by an interrupted fetch.
`build_hp_linux.py` builds the RV64 `lp64d` userspace/OpenSBI image set with the repo-owned
external OpenSBI platform. `package_hp_boot.py` creates the V2 typed LP/HP flash
bundle and its SHA-256/CRC manifest. `check_lp_hp_performance.py` applies the
measured 2.5x CoreMark/MHz gate.

`setup_hp_rtthread.py` installs the locked kernel, RV64 bare-metal compiler and
isolated SCons environment. `build_hp_rtthread.py` stages only the repository BSP
under the variant and records the source/compiler/configuration inputs.
`hp_tools.py` checks source revisions and RV64 ELF class. `run_hp_sim.py` keeps
Linux, smoke and RT-Thread logs/verdicts separate and checks every required
workload marker together with a successful process exit.

`run_debug_session.py` is the local Hazard3 debug acceptance driver. It starts
the Verilator remote-bitbang endpoint, the lock-pinned OpenOCD binary, and
RISC-V GDB, then records their logs and a structured result. Invoke it through
`make CONFIG=configs/ci/ihp130-debug.mk SIMU=VERILATOR debug-sim`; see
[`../docs/hazard3-debug.md`](../docs/hazard3-debug.md) for its scope and
limitations.

`program_xpi_flash.py` drives the SRAM-resident XPI loader through an existing
OpenOCD GDB endpoint. It validates and splits a BIN at 4 KiB sector boundaries,
generates a reviewable GDB script, and performs erase/program only when
`--execute` is explicit. See [`../docs/ip/xpi.md`](../docs/ip/xpi.md).

`check_rtl_style.py` applies the ownership-aware RTL style rules from
[`../rtl/rtl_style_manifest.json`](../rtl/rtl_style_manifest.json). The CI
target checks changed self-owned RTL for positional module connections, legacy
constructs, and the staged naming contract. `rtl-style-check-all` retains the
historical full-tree structural baseline while naming debt is migrated in
module-sized batches. Existing findings must not be expanded by a new change.

`migrate_rtl_connections.py` is the conservative migration helper for legacy
positional instances. It discovers ANSI-style module declarations in the
production RTL and locked technology/IP trees, rewrites only exact positional
parameter and port lists to named connections, and reports ambiguous instances
without modifying them. Run `make rtl-migrate-connections`, then
`make rtl-format rtl-style-check`; review the generated diff before committing.

`migrate_rtl_names.py` shortens only local identifiers beginning with `s_` or
`r_` (plus local automatic variables such as `read_request`). Public module
ports and interface fields are intentionally unchanged. Run
`make rtl-migrate-names`, then format and lint the resulting diff.

`crypto_constants.py` packs the frozen CRYC1 image from
`data/crypto_constants.json` and emits an 848-byte compact firmware header;
`crypto_p0.py` independently checks it against the mathematical oracle in
`tests/crypto_reference.py`. The P0 runner also executes existing V1 tests,
full-width RSA-2048 public/private/recheck vectors, SHA padding boundaries and
a real DMA-to-AES-to-memory test. Missing tools or missing success markers are
errors. It invokes the shared simulation verdict checker; an Icarus `$fatal`
that exits with code zero is still a failed test.

The `crypto-p0-constants`, `crypto-p0-rtl`, `crypto-p0-baseline` and
`crypto-p0-report` Make targets write below `build/<variant>/crypto/p0/`.
Select `CONFIG=configs/ci/ihp130.mk SYNTH=YOSYS` and one BUILD_TIMESTAMP for
all four on the frozen V1 checkout. The runner rejects V2 before writing any
P0 artifact. P0 records V1 lifecycle limitations. Block synthesis calls the
unchanged `synth.tcl` and `abc_balanced.script`, adding JSON checkpoints and
standalone interface elaboration only. Each new synthesis attempt uses a fresh
`synth-*` artifact directory recorded in the report, preventing an old netlist
from masking a new failed attempt. Its 20833 ps target is derived from
the PCLK inventory; it is not a full-chip timing result. A timeout/interruption
retains partial checkpoints and cannot pass the aggregate report. See
[`../docs/ip/crypto.md`](../docs/ip/crypto.md).

`crypto_p1.py` owns the separate `crypto-p1-constants`, `crypto-p1-rtl`,
`crypto-p1-formal`, `crypto-p1-synth` and `crypto-p1-report` targets. P1 artifacts are below
`build/<variant>/crypto/p1/`; P0 artifacts are never reused as candidate
outputs. The RTL target exercises public APB initialization, AES/SHA and full
RSA-2048 on the IHP FUNCTIONAL model, actual DMA channels 4/5, storage masks
and Montgomery carries, then the explicit synchronous fallback. Formal
checks use running-clock/synchronous-response assumptions and verify control
invariants, private-result release, bounded scrub, APB access ownership and
AES output retention across abort/fatal faults. DMA fixtures also inject AXI
read and write errors. The exact proof/test mapping is in
[`../docs/ip/crypto-verification.md`](../docs/ip/crypto-verification.md).
They do not prove the cryptographic arithmetic or physical resistance.
The RTL target also runs `crypto_response_fault_tb.sv`, which models delayed,
duplicate, stale-epoch and missing SRAM responses; the product store defaults
to fixed one-cycle responses.
Candidate synthesis preserves the P0 SAT/ABC recipe and 10800-second limit.
The report requires `CRYPTO_P0_BASELINE_ROOT`, `CRYPTO_P1_LP_ROOT` and a
successful `CRYPTO_P1_CI_ROOT` for its focused gates. Capture full-SoC evidence
with `crypto_p1.py firmware --variant-root <variant> --firmware-kind lp|ci`
after creating the corresponding committed-profile manifest. This runs the
build/simulation and binds the actual configuration, input snapshot, ELF and
log/result hashes. `crypto_p1.py quality --variant-root <variant>` records
focused formatter and policy results. Report assembly rejects stale inputs
or artifacts; it cannot qualify an earlier binary using current source hashes.
`crypto_p2.py` is the fail-closed P2 aggregator. It requires a clean candidate
revision and records whole-SoC netlist, STA, physical, metrics, warning and
P0/P1 evidence; missing P2 artifacts remain `NOT_RUN`.

Update or add tests in [`../tests`](../tests) for script behavior. Run
`ruff check .` and `python3 -m pytest -q`; build-flow changes also require the
relevant dry-run or regression profile from [`AGENTS.md`](../AGENTS.md).