diff --git a/.agents/docs/2026-10-30-ecosystem-part2-ide-and-language-service-design.md b/.agents/docs/2026-10-30-ecosystem-part2-ide-and-language-service-design.md index 537a23273..9fdcc3e28 100644 --- a/.agents/docs/2026-10-30-ecosystem-part2-ide-and-language-service-design.md +++ b/.agents/docs/2026-10-30-ecosystem-part2-ide-and-language-service-design.md @@ -5,7 +5,7 @@ status: active # 生态设计 part 2：mcpp 插件与 mcppls 的分层、入口、测试、版本感知与分发 -日期：2026-10-31（rev 10，实施）。基线：mcpp 2026.10.30.1（`13e2da04`）、mcpp-ide #3（0.2.0，`734d856` + `49ae757`）、 +日期：2026-10-31（rev 11，实施）。基线：mcpp 2026.10.30.1（`13e2da04`）、mcpp-ide #3（0.2.0，`734d856` + `49ae757`）、 mcppls #75（0.0.20，`4660eb6`）。前文：[part 1](2026-10-29-ecosystem-target-sets-packaging-and-editors-design.md)（§9 mcppls 单包、 §10 编辑器）。相关 issue：mcpp#868（锁文件键）、Sunrisepeak/mcppls-clangd#12（clangd 映像）。相关规范：mcpp SPEC-015 §4、docs/08、 docs/50；mcppls S3、S4；mcpp-ide protocol。 @@ -37,6 +37,8 @@ docs/50；mcppls S3、S4；mcpp-ide protocol。 最小接口（§23.2）、`polyglot` 与 `--pkg-format` 列表、`mcpp run`、`mcpp emit xpkg` 的关系（§19.6.2）。 - rev 7（本稿）：评审通过（`mcpp.path` 优先、文件映射的最小接口、`polyglot` 的三条边界）。clangd 的 ok 映像进入本轮 （mcppls-clangd#12，§12、§25.3）；文件映射成为独立的可选接口 `openkal.map`（§23.2）；实施计划与任务依赖（§25）。 +- rev 11：clangd 映像的验收测量（§25.3 B5）：差距是运行器的固定开销，定位到 openkal-image 并修复（0.1.6、0.1.7），家族级联至 + openkal-platform 0.19.2；验收的解析时间按准则扣除进程启动；进度见 §25.8。 ## 0. 摘要 @@ -844,13 +846,22 @@ openkal 的接口整体提供或整体不提供（§3），`openkal.space` 的 | B4 | 模块 worker | probe 第 11 条：worker 直接运行 `.oke` 而崩溃；`getrlimit`、`setrlimit` 不可用 | LLVM 补丁（series 0017） | worker 经 `kal_env_loader` 以"加载器 + 映像"启动；openkal 没有地址空间上限，worker 的内存上限在 ok 映像上由监督方按常驻内存执行 | — | | C1 | 链接输入 | probe 第 13 条：`[build] ldflags` 中的预构建归档不是链接输入，改动它不重新链接 | mcpp | 链接参数中存在的文件记为链接步骤的隐式输入 | — | | C2 | 构建途径 | probe 第 14 条：约定从编译命令读出，工具链文件由 Python 生成 | mcppls-clangd | 约定取自 `${mcpp.contract}`；生成器是一个小的 mcpp 项目 | — | +| B5 | 运行器的固定开销（rev 11） | mcppls-clangd 0.0.18 首次发布运行（41179052941，发布前取消）：四列全部通过，四个系统都差一个与工作量无关的常数：每个用例 +0.3–0.46 s、约 +110 MiB（`import m`：Linux x64 原生 0.116 s、32 MiB，映像 0.412 s、143 MiB）。原因三条，都在 okrun 与 libokld：(a) 接纳时读过映像的每一页，okb 的映射在整个运行期间保留，映像以文件与副本两份常驻；(b) 接纳的 SHA-256 为逐轮的可移植实现，68 MB 在 CI 机器上 0.3 s 以上；(c) 装入时先清零整个跨度，再逐字节复制各段 | openkal-image | 0.1.6：映像装入后释放所运行的文件（加载器不保留指向字节的指针，`okld.h` 写明）；x86_64 用 SHA 扩展（CPUID），ARMv8 在编译目标保证时用 SHA-256 指令；各段只写一次（文件字节后接零），按 8 字节写。0.1.7：Linux aarch64 读 `ID_AA64ISAR0_EL1`（Linux 自 4.11 模拟对标识寄存器的 `mrs`，先以 `uname` 读内核版本）。级联：openkal-musl 0.23.1/0.23.2、openkal-llvm-runtime 0.16.11/0.16.12、openkal-platform 0.19.1/0.19.2 | K5 | +| B6 | mcppls 给映像的引擎参数（rev 11） | mcppls#75 的 bench-validate：Linux x64 上 warm 阶段补全不到导入模块的成员。探测（同一 payload，保留日志）：冷热两阶段都是 "Module worker configuration failed: Worker payload bounds require worker isolation"，随后所有前置模块构建失败。mcppls 在 Linux 主机上给任何已验证引擎 `--modules-builder-owned-cache-payload-mib`；隔离 worker 只编进原生 Linux 引擎，映像在预处理器看来不是 Linux，与 macOS、Windows 一样进程内构建 | mcppls | `owned_cache_payload_bound()` 按主机与引擎的 runner 决定（有 runner 即 ok 映像，不给界限），单元测试覆盖四种组合；clangd 不变 | L | +| B8 | 能力对等（rev 11） | 把 Linux 两行也切到 `clangd.okb` 后，mcppls 的 ux 阶段测得验收的单次 `--check` 测不到的差距：每个会话重建全部模块（ux-mcpp 176/176），首次就绪 33 s（预算 12 s），锁故障用例找不到磁盘上的模块文件。原生 Linux 引擎以隔离 worker 构建模块并在自有模块缓存中跨会话保留；映像在预处理器看来不是 Linux，与 macOS、Windows 上的所有引擎一样没有这两项 | mcppls、openkal、openkal-musl、LLVM 的 openkal series | 选择条件增加一条：映像具备 mcppls 在该系统上用到的原生引擎的能力。差距的原因见 §25.9：模块准备 `limit=1`（H1）与锁文件的所有者都指向读取者（H3），两者都不是隔离 worker 与自有缓存（映像构建与复用模块比原生快）；按层修正后四个系统都选 `clangd.okb`，以 mcppls 的完整 CI 与 ux 阶段验证。conformance 运行器以映像的 runner 启动引擎（completion-baseline、WA-CLANGD-009、崩溃替身） | L | +| B7 | 映像启动的余量（rev 11，方案） | 0.1.7 之后启动开销只剩接纳的 SHA-256：有 SHA 指令的机器上 68 MB 约 0.03 s；没有的（CI 的 x64 机器、2021 年以前的 Intel 桌面处理器）约 0.14 s（Linux x64 `--version` 原生 0.013 s、映像 0.154 s）。单个 SHA-256 不能拆到多核 | openkal-image | okb.json 的成员可带分块摘要 `chunks = { size = 1 MiB, sha256 = [...] }`，由 `okld bundle` 写出、由清单签名覆盖；okrun 以 kal_task 并行校验各块，未带分块的 okb 照旧校验整体摘要（"Unknown keys are ignored"，两向兼容）；`okld bundle check` 校验两者且一致。预期：4 核无 SHA 指令的机器上接纳由约 0.14 s 降到约 0.04 s。收益须重新打包 okb，故随 clangd 下一次发布一起做（clangd.okb 由新的 okld 写出），并在验收中报告启动时间 | B5 | mcpplibs.os 不直接涉及：clangd 映像只用 openkal-musl 与 libc++；os 因精确钉住随 K5 升版本，mcppls 随之升级 os。 **验收**（mcppls-clangd#12 的完成条件，量化）：四个系统（Linux x64、Linux arm64、macOS arm64、Windows x64）× 四列 （`--version`、无头文件、`import m`、系统的 C++ 标准库）全部通过；解析时间不超过原生的 1.3 倍；常驻内存与原生之差不超过映像 装入的段的大小（B1(c) 的固有代价）。mcppls 的包中，满足验收的系统使用 `clangd.okb`，引擎的选择按系统写在包的清单中； -不满足的系统保留原生 clangd 并在 mcppls 的文档中写明差距，不阻塞其余系统。 +不满足的系统保留原生 clangd 并在 mcppls 的文档中写明差距，不阻塞其余系统。映像还须具备 mcppls 在该系统上用到的原生引擎的能力（B8）。 + +解析时间按准则测量（rev 11）：每个用例与 `--version` 各运行五次，用例的中位数减去 `--version` 的中位数（进程启动：映像的 +接纳与装入，语言服务每次启动付一次，而非每个文件付一次）为解析时间；两者的启动时间与解析时间都写入 +`openkal-acceptance.json`。此前按整个进程计时，映像的启动被计入每次解析：没有 SHA 扩展的 x64 机器上，`--version` 原生 +0.013 s、映像 0.153 s。 ### 25.4 并行 @@ -892,3 +903,132 @@ mcpp 新建、构建、运行、`--pkg-format polyglot`；mcppls 在四种编辑 | CI | GitHub 于 2026-11-02 停用 macos-14，此前按计划使其任务失败（actions/runner-images#13518）。本轮各仓库的 macOS 任务改用 macos-15；构建 macOS 产物的任务写明 `MACOSX_DEPLOYMENT_TARGET=14.0`，支持下限仍为 macOS 14，由产物的 `minos` 断言。mcpp 的 ci-target-matrix 与 openkal-cross 的 macOS 任务随之改用 build.yml 的产物，不再自行构建 | | 未覆盖 | openkal-opensbi（钉 openkal 0.16.0）与 openkal-uefi（0.14.1）落后于规范；二者为裸机实现，不在 os 与本轮的依赖中 | + +### 25.8 进度（rev 11） + +| 任务 | 状态 | +|---|---| +| M3 | 已合入并发布 2026.10.31.3（mcpp#873），GitCode 已补，xim-pkgindex 的 mcpp 条目已合入 | +| B5 | openkal-image 0.1.6（#12）、0.1.7（#13）已发布；openkal-musl 0.23.1、0.23.2，openkal-llvm-runtime 0.16.11、0.16.12，openkal-platform 0.19.1、0.19.2 随之发布，均已入 mcpp-index 并补 GitCode。platform 的 CI 改用 mcpp 2026.10.31.3：2026.10.30.1 把集合的 Windows 启动器建在项目所选的平台上，os 0.2.1 与版本组的 openkal 0.18.0 冲突（M2 已修） | +| C | mcppls-clangd 0.0.18（#13、#14）：ok 映像建在 openkal-platform 0.19.2 上。发布运行的验收（解析时间扣除进程启动，五次取中位数）四个系统全部通过：Linux x64 1.02×／1.09×、Linux arm64 0.96×／1.02×、macOS arm64 1.05×／0.97×、Windows x64 0.72×／0.72×（无头文件／标准库）；常驻内存之差 +7 至 +53 MiB，界限为映像大小 68–71 MiB。macOS 的无头文件用例在此前四次运行中为 1.25–1.38×，离界限最近 | +| L | mcppls 0.0.20（#75）：工作区平台 openkal-platform 0.19.2；锁文件 macOS arm64 与 Windows x64 两行选 `mcppls-clangd-openkal`，Linux 两行保留原生引擎（B8），glibc 下限保留；B6 的引擎参数修复；conformance 以 runner 启动映像引擎 | +| I | mcpp-ide 0.2.0（#3）：ok 映像的平台 openkal-platform 0.19.2；CI 全绿，待 mcppls 0.0.20 发布后改钉其单包 | +| X | xim-pkgindex 一个 PR：okrun 0.1.4–0.1.7、mcppls、mcppls-nvim、mcpp-ide、mcpp-ide-nvim；机器人建的版本更新 PR 不触发 CI，需手动派发 | + +### 25.9 openkal 0.19：处理器与任务优先级（rev 11，已定） + +依据：clangd.okb 在 Linux 上的 ux 重启 33 s 对原生 4–5 s（§25.3 B6–B8 之后的测量）。映像对模块的复用比原生快（40 个模块的 +工程：冷 1.78 s 对 3.15 s，热 0.63 s 对 1.0 s），差距全部来自 LLVM 在编译时按宿主决定、映像的目标 `-openkal` 一个也 +不匹配的三项： + +| # | 项 | 映像中的表现 | 层 | +|---|---|---|---| +| H1 | 处理器数 | `get_physical_cores()` 对未知目标为 -1，重量级线程池（模块准备、后台索引）退化为 1 个线程（`limit=1` 对原生 2–4） | openkal：规范无此接口 | +| H2 | 任务优先级 | `set_thread_priority` 没有映像的分支，后台工作不降到空闲优先级（原生 Linux 为 SCHED_IDLE，macOS 为 QoS） | openkal：规范无此接口 | +| H3 | 进程标识 | openkal-musl 的 `getpid()` 在每个程序中为 1，`getsid`/`kill(pid, 0)` 只认自身：锁文件的所有者都指向读取者，死去的所有者的锁不被打破 | openkal-musl：规范已有 `kal_process_self`/`kal_process_info`/`kal_process_open_id` | +| H4 | 上下文标识的代价（macOS） | openkal-macos 的 `kal_task_current` 每次调用发一次 `thread_selfid` 系统调用；openkal-musl 取 `errno`、线程指针、线程私有数据与映像中每个（模拟的）线程局部变量都经它，一次 `clangd --check`（4000 个声明的无头文件用例）调用 158 万次。macOS arm64 上映像的解析时间为原生的 1.25–1.96 倍（六次运行），同一 aarch64 映像在 Linux arm64 上为 1.02 倍（openkal-linux 按线程缓存）；在 Linux 上强制每次系统调用，中位数 851 ms → 985 ms | openkal-macos（实现，不动规范）：`pthread_threadid_np` 从系统库为线程保存的记录读同一个内核标识，0.17.1 | +| H5 | macOS SDK | clangd 的 `CommandMangler` 在 `__APPLE__` 下以 `xcrun --show-sdk-path` 为未指明 SDK 的命令补 `-isysroot`（`xcrun --find clang` 同理）；映像中这段被编译掉，未指明 SDK 的命令找不到系统头文件，`std` 模块编译失败（"We don't know how to get the definition of mbstate_t"），mcppls 的 macOS conformance 三个夹具因此失败 | LLVM 的 openkal series：`kal_env_system` 为 `macos` 时照原生行为询问 `xcrun` | +| H6 | 线程一起分配（C 库） | mcppls 的 ux 阶段，已有缓存的新会话映像 32 s 就绪、原生 5 s；冷启动、打字、补全与原生相近。CI 上的 perf 采样（映像符号取自 clangd.oke）：映像在 4 核上平均只用 0.85 核，279k 次 futex；mallocng 以一把锁服务每次分配（此配置下 `rdlock` 为独占），ok 映像中一次 malloc+free 在 1／2／4 线程下为 38.5／194／521 ns。单文件、单线程的验收用例测不到 | openkal-musl 0.25.0：有第二个线程后，每个线程保留它释放的小尺寸类槽（≤ 2032 B，每类至多 16 个），无锁取回；空时在服务它的那次加锁中多取 8 个；线程结束时交还（TSD 析构之后、线程表锁之前，与 fork 的加锁顺序一致）。缓存中的槽在 mallocng 看来仍在使用，字节与 mallocng 自身 free 后相同，取回即 mallocng 对已释放槽的 `enframe`。实测 31／41／61 ns | +| H7 | 字符串比较（C 库） | 同一采样中 `strlen` 8.7 %、`memcmp` 5.7 %，主要在 LLVM 选项表的匹配：musl 的 `memcmp` 逐字节，`strlen` 逐字节走到对齐处 | openkal-musl 0.25.0：`memcmp` 每次 8 字节（XOR 与 ctz 定位首个差异），`strlen` 在 x86_64 上以 SSE2 每次 16 字节、其余以对齐字；每次读取都按其大小对齐，不越过字符串未到达的页。mcppls-clangd 的验收增加 `parallel-index`（四个 worker 同时为 12 个文件建索引，映像与原生之比 ≤ 1.3）：0.0.19 的映像为 1.88× | +| H8 | 提供者索引的重复 mangle（clangd fork） | musl 0.25.0 之后 ux 仍为 30 s：完整的 info 级日志显示服务端一次预热一个单元、6 ms 内送出下一个，引擎每个单元 156 ms（原生约 25 ms），其中 125 ms 在模块规划。CI 上的 perf（映像符号取自 clangd.oke，内存布局在引擎启动几秒后采得）：三分之一的时间是 `CompileCommandsProjectModules::indexProducerCommands` 把数据库的 481 条命令全部重新 mangle（`OptTable::ParseArgs`、`strlen`、`matchOption`），每个单元两次，之后 `getIndex` 才发现索引未变。原生做同样的事，只是更快 | mcppls-clangd 补丁 0038（UP-03，主 series，原生同样受益）：命令缓存保留它索引过的清单快照与首条命令的 mangle 结果；同一 mangler 给出相同首条命令即复用索引（mangle 一条而非 481 条），另一 mangler 或带配置的逐文件旗标编辑照旧全部 mangle，按其他命令重建索引时清除所保留的。引擎版本 23.1.0-mcppls.8 | + +做法按层：规范补足操作系统无关的能力，C 库以它实现 POSIX 的语义，LLVM 走它原有的 POSIX 路径，不在 clangd 中为映像 +另写宿主逻辑。 + +- **openkal 0.19.0**（`openkal/task.h`，第 11 条第 41 项）： + - `kal_uintptr kal_task_cores(void)`：本程序可运行于其上的处理器的物理核数，0 为环境不能回答，不大于 + `kal_task_parallelism()`。可运行的处理器数已由 `kal_task_parallelism` 回答，故只增一个数，不增结构； + - `int kal_task_set_priority(kal_uintptr priority)`，调用者所在上下文：`KAL_TASK_PRIORITY_DEFAULT`、`KAL_TASK_PRIORITY_LOW`、 + `KAL_TASK_PRIORITY_BACKGROUND`（0、1、2）；能做到的环境声明属性位 `KAL_TASK_PROP_PRIORITY`，不能的返回 + `kal_err_not_supported`；降低后被拒绝升回为 `kal_err_permission`； + - 三个实现：Linux 以亲和集合中各处理器的 sysfs 拓扑（`physical_package_id`、`core_id`）与 `sched_setscheduler` 的 + SCHED_OTHER/BATCH/IDLE；macOS 以 `hw.physicalcpu` 与 QoS 类（`pthread_set_qos_class_self_np`）；Windows 以 + `GetLogicalProcessorInformationEx` 的核心关系与线程的后台模式（不被接受时为空闲优先级）；openkal-emscripten 两者都答 + "不能回答／不支持"。一致性测试、`SURFACE.txt` 与 okrun 的绑定随之。 +- **openkal-musl 0.24.0**：`getpid()` 为环境给本程序的标识，子程序的标识为其环境标识；`kill(pid, 0)`、`getsid`、`getpgid` + 对他人程序经 `kal_process_open_id`；`sched_setscheduler`（`pthread_setschedparam`）的 SCHED_OTHER/BATCH/IDLE 与 + `setpriority` 经 `kal_task_set_priority`；`flock` 与 `F_OFD_SETLK[W]` 经 `kal_fs_lock`。`sysconf(_SC_NPROCESSORS_*)` 与 + `sched_getaffinity` 已取自 `kal_task_parallelism`，不变。 +- **LLVM 的 openkal series**：`computeHostNumPhysicalCores` 取 `kal_task_cores()`，为 0 时取 `kal_task_parallelism()`； + `set_thread_priority` 按 LLVM 的三档映射到 `kal_task_set_priority`（只改 `Unix/Threading.inc` 一处）。锁文件所有者的进程 + 标识由 C 库的 `getpid` 给出，LLVM 的 `Process.inc` 与 `LockFileManager` 不改。 +- **openkal-image 0.1.8**：新接口的绑定；okb 的分块摘要（B7）：本机 68 MB 的 okb 启动由 0.10–0.12 s 降到 0.04–0.07 s。 +- **级联**：openkal 0.19.0 → 三个实现与 openkal-emscripten → openkal-image 0.1.8 → openkal-musl 0.24.0 → openkal-llvm-runtime + → openkal-platform 0.20.0（openkal 0.19 的版本组）→ mcpplibs.os → mcppls-clangd 0.0.19（验收含 `module-parallelism`：映像 + 同时构建的模块数不少于原生）→ mcppls 0.0.20 四个系统都用 `clangd.okb`，以 mcppls 的完整 CI 与 ux 阶段验证 → mcpp-ide 0.2.0。 + +**本轮范围与 openkal 变动的约束（rev 11，已定）。** + +- 本轮做 openkal 规范与基础设施（§25.9 的 openkal 0.19、三个实现、openkal-emscripten、openkal-image、openkal-musl、 + openkal-llvm-runtime、openkal-platform、mcpplibs.os），并以此收敛 clangd（mcppls-clangd 0.0.19）与 mcppls part 2 方案 + （mcppls 0.0.20、mcpp-ide 0.2.0）。 +- LLVM 与 clang、lld 本身作为 ok 映像（`clang.okb`、`lld.okb`，跨操作系统分发工具链）不在本轮，记录为 mcpp 的 Issue。 + 其前提依次为：映像文件一段的共享可执行映射（不复制段，进程间共享代码页；okb 的 64 KiB 对齐为此预留）、driver 中在编译时 + 决定的宿主逻辑改为运行时决定、派生进程经加载器启动；验收以同版本原生 clang 逐字节相同的目标文件、编译耗时与 `-j N` 下的 + 常驻内存为准。 +- **openkal 是核心规范，一般不动。** 只在两项都成立时增补：确有消费者需要、组合不出来（clause 3、clause 6 的准入）； + 符合通用 kernel ABI 与 openkal 的设计规范（clause 7：无表、以属性位声明能力、零值语义、版本条目）。每次变动：规范、 + 一致性测试、三个实现与 openkal-emscripten 的 CI 全部通过后，再做一次深度自我评审（必要性、与既有条款的一致性、各系统 + 的语义差异是否写明、是否可由 C 库或上层组合），评审结论写入规范第 11 条的版本条目与 PR 描述，然后才发布。 +- 本轮的两项增补按此审视：`kal_task_cores` —— 物理核数各系统都在一次调用中给出，C 库无处可问，LLVM 等以物理核为重型任务 + 定额的消费者得到"未知"即退化为 1；`kal_task_set_priority` —— 各系统都能降低线程的调度地位，映像内的后台工作无法表达， + 原生程序都这样做。进程标识（H3）不动规范：规范已有 `kal_process_self`/`kal_process_info`/`kal_process_open_id`，缺口在 + openkal-musl 的 `getpid` 语义，由 C 库修正。 + +**分层归属（rev 11，已定）。** mcpplibs.os、openkal-musl、openkal 各有自己的规范：openkal 是最底层、最核心的 kernel ABI， +只收"只有系统能回答、上层组合不出来"的能力；openkal-musl 在它之上给出 POSIX 的语义；mcpplibs.os 是 mcpp 程序的 C++ 操作 +系统库，按消费者的需要增长。本轮各项的归属： + +| 能力 | 层 | 理由 | +|---|---|---| +| 物理核数 `kal_task_cores` | openkal | 拓扑只有系统知道（Linux sysfs、macOS `hw.physicalcpu`、Windows 处理器关系表），C 库与 os 都无处可问 | +| 任务优先级 `kal_task_set_priority` | openkal | 线程的调度地位只有系统能改；三个系统各一次调用 | +| `getpid` 为环境给程序的标识，`kill(pid, 0)`/`getsid` 问他人程序 | openkal-musl | 规范已有 `kal_process_self`/`kal_process_info`/`kal_process_open_id`，缺的是 C 库的 POSIX 语义 | +| `sysconf(_SC_NPROCESSORS_*)`、`sched_getaffinity` | openkal-musl（不变） | 已取自既有的 `kal_task_parallelism` | +| `pthread_setschedparam(SCHED_IDLE/SCHED_BATCH)`、`setpriority` | openkal-musl | 以 `kal_task_set_priority` 实现 POSIX 的调度语义 | +| `flock`、`F_OFD_SETLK` | openkal-musl | 规范的 `kal_fs_lock` 已是"由打开的文件持有"，缺的是 C 库的两种拼写 | +| LLVM 的物理核数与线程优先级 | LLVM 的 openkal series | 物理核数经 POSIX 不可表达，直接问 `kal_task_cores`；优先级按 LLVM 的三档直接映射到 `kal_task_set_priority` | +| C++ 的处理器与优先级接口 | mcpplibs.os：本轮不增 | 本轮无消费者（mcppls 的服务端不需要）；os 只把 openkal 钉到 0.19.0（0.2.5） | + +### 25.10 openkal 的边界（rev 11，已定；规范文本见 openkal SPEC §1.1） + +openkal 同时是三份规定：以 C 陈述的接口（clause 4）、clause 4.4 各边界上的二进制接口（S 静态、L 延迟绑定、X 跨越），以及 +ok 映像与绑定它的加载器之间运行时的二进制接口（clause 12）。三者共同的规则写入 SPEC §1.1： + +- **下层的东西不向上穿透。** 使用者观察到的值、布局、错误、名字与行为都是规范定义的；实现把环境的回答翻译成规范的 + 词（错误为 §5.2 的值，能力为 §6.2 的属性），环境给出规范没有对应词的回答时，报告规范陈述的最近结果或拒绝该操作，不把 + 环境自己的回答传上去。环境给的标识（程序的、上下文的）作为不透明值穿过，含义是规范陈述的，不是环境的。 +- **实现之间在下层可以不同，在上层不能不同。** 调内核、调 C 库、由其他操作组合或模拟都可以，代价可以不同；使用者能观察到的 + 不能不同，规范留下的余地都有名字：实现声明或不声明的属性位（§6.2），操作陈述的几个结果之一（如降低后的优先级被拒绝升回， + 第 11 条第 41 项）。 +- **使用者绕过 openkal，就绑定了它所绕到的东西。** 程序可以直接调用环境（系统库、系统调用），规范不禁止也不观察；这样的 + 程序在其余部分可移植，绕到的部分由它自己负责在别处提供。ok 映像不能绕过：它只导入规范声明的名字与其他映像导出的名字 + （§12.2），这正是同一份字节能在加载器服务的每个系统上运行的原因。 + +本轮的增补按此核对：`kal_task_cores` 是数目，不带拓扑的系统细节；`kal_task_set_priority` 是三个抽象的档位，各系统的 +策略、QoS 类、后台模式只在实现内部；升回被拒绝是规范陈述的结果（`kal_err_permission`），不是环境错误的透传；openkal-musl 的 +`getpid` 是环境给的不透明标识；Windows 在后台模式被拒绝时退到空闲优先级，属于实现内部的差异，上层观察到的仍是"降到后台"。 + +**okrun 与 okld 支持的几种情形，与边界的关系。** + +| 情形 | 载体 | 边界 | +|---|---|---| +| 运行一个映像或 okb | `okrun app.oke`、`okrun app.okb`（两个架构一个文件，docs/okb.md） | 映像只导入 `kal_*` 与其他映像的导出（§12.2） | +| 程序自带映像 | 自包含（`mcpp pack --mode self-contained`，载体程序在段中携带映像经 libokrun 运行）、单文件、polyglot | 载体是各系统的原生程序；映像不变 | +| 映像需要的库 | okso：okb 内、旁边、`lib//`、`OKRUN_PATH`；C 库的 `dlopen` 经 libokld | 仍是映像，仍只导入 `kal_*` | +| 原生程序使用映像的库 | `okrun_open_library`、openkal-bind 生成的绑定 | 原生一侧绑定了它的系统，映像一侧不变 | +| 映像使用主机的设备（GPU 等） | §19.8：R0 原生目标经 openhal 直接用主机库；R1 映像经 openbus（`kal_net_connect_at` 连本机路由、共享区域）；R2 加载器提供的具名扩展接口（C 头文件定义、带版本、映像的 note 声明，缺少即加载时拒绝） | 设备能力归 openhal，openkal 不增加 `kal_gpu_*`；R1、R2 都不让主机库的名字进入映像 | + +GPU 与 openkal 的关系由此确定：openkal 不承载设备语义；映像经 openbus 或经规范化的扩展接口使用设备，原生程序经 openhal +直接使用，后者是开发者主动绑定平台。 + +### 25.11 进度（rev 12） + +| 任务 | 状态 | +|---|---| +| K（openkal 0.19） | openkal 0.19.0、openkal-linux 0.20.0、openkal-macos 0.17.0、openkal-windows 0.15.0、openkal-emscripten 0.8.0 已发布（mcpp-index #647）；openkal-image 0.1.8（okb 分块摘要）、openkal-musl 0.24.0（H3、flock 与 OFD 锁、调度占用）、openkal-llvm-runtime 0.16.13（#648）、openkal-platform 0.20.0（#649）、mcpplibs.os 0.2.5（#650）已发布。os 的 `checks/hosted` 一并改用 0.20.0 的版本组（漏改时 CI 报两个 openkal） | +| H4 | openkal-macos 0.17.1（#651）、openkal-image 0.1.9、openkal-musl 0.24.1、openkal-llvm-runtime 0.16.14（#652）、openkal-platform 0.20.1（#653）、mcpplibs.os 0.2.6（#654）已发布；`_pthread_threadid_np` 进入 openkal-macos 允许的导入与 musl 的交叉链接名单（十七个）；okrun 0.1.8、0.1.9 的二进制已补 GitCode | +| C | mcppls-clangd 0.0.19 已发布：openkal series 增加 H1、H2、H5；验收增加 `module-parallelism`（四个系统映像与原生都为 `limit=2`）与 macOS 的 `check-std-host`。发布运行的验收四个系统全部通过，解析时间为原生的 0.60–1.10 倍（macOS arm64 1.00／0.90，此前 1.25–1.96） | +| H6、H7 | mcppls 以 0.0.19 的映像跑完整 CI：macOS 三个夹具通过（H5），U9 通过；ux 的已有缓存新会话仍为 23–32 s（原生 5 s）。以 nightly 的 `payload-run` 在 CI 机器上重放 ux-mcpp，perf 采样定位到 C 库（H6、H7）。openkal-musl 0.25.0、openkal-llvm-runtime 0.16.15（#655）、openkal-platform 0.20.2（#656）已发布；mcppls-clangd 0.0.20（#16，同一引擎源码，映像建在 0.20.2 上，验收增加 `parallel-index`）在 CI | +| L | mcppls 0.0.20（#75）：四个系统都选 `clangd.okb`；conformance 的运行器按 runner 的参数识别映像引擎（U9）；平台 0.20.2；以 clangd 0.0.20 的 PR 构建为候选引擎（`packaging/engine-candidate.json`）跑完整 CI 与 ux，通过后发布引擎并切换锁 | +| I | mcpp-ide 0.2.0（#3）：openkal-platform 0.20.2、os 0.2.6 | diff --git a/.agents/docs/2026-11-02-ecosystem-part3-ok-image-performance-parity-design.md b/.agents/docs/2026-11-02-ecosystem-part3-ok-image-performance-parity-design.md new file mode 100644 index 000000000..e8fb64253 --- /dev/null +++ b/.agents/docs/2026-11-02-ecosystem-part3-ok-image-performance-parity-design.md @@ -0,0 +1,580 @@ +--- +subject: openkal-performance +status: active +--- + +# 生态设计 part 3：ok 映像的性能对等——引擎路径、openkal 运行时与观测 + +日期：2026-11-02（rev 3，评审通过）。基线：mcppls-clangd #16（`6336bf1`，引擎 23.1.0-mcppls.8）、mcppls #75（0.0.20）、 +openkal-platform 0.20.2（openkal 0.19.0、openkal-linux 0.20.0、openkal-macos 0.17.1、openkal-windows 0.15.0、openkal-musl 0.25.0、 +openkal-llvm-runtime 0.16.15、openkal-image 0.1.9）。前文：[part 2](2026-10-30-ecosystem-part2-ide-and-language-service-design.md) +§25.3（B4–B8）、§25.9（H1–H8）、§25.10（openkal 的边界）。 + +**修订记录** + +- rev 1：综合 2026-11-01 至 11-02 的测量与讨论。已定：引擎的优化与 openkal 的适配分放两个 series；引擎按能力而不是按 + 操作系统选择实现；越底层的改动越要一轮做完，并在本层以本层的测试证明，引擎与 mcppls 的长周期 CI 每轮只走一次； + openkal 增加可选的观测接口，采集在 okrun，分析在单独的 okprof。多视角自我评审 14 条（§15），待评审项见 §16。 +- rev 2：rev 1 的待评审项全部定下（§16.1）：分配器为 mimalloc，按 mwrap 规范在 mcpplibs 下新建 mimalloc-m（§7.1）；job 的两种 + 执行方式、观测接口两组、接纳记忆、线程指针不改、引擎不做 LTO/PGO、门禁阈值、两项 L1 内的后备做法都按 rev 1 进入本轮。 + 选定 mimalloc 带出一项规范改动：§5.3 由单一的 `kal_mem_discard` 扩为内存区域五个操作（`kal_alloc` 是带锁的通用分配器，不能 + 给出按大粒度对齐、延迟提交的区域）。第二轮自我评审 9 条（§15.2），新的待评审项见 §16.2。 +- rev 3：按评审改写（§16.3）：mallocng 保留为 feature，mimalloc 为默认；openkal 的内存区域只陈述通用的虚拟内存语义， + C 库的 `mmap`/`munmap`/`madvise` 建在它上面；mimalloc-m 只把 `compat.mimalloc` 的接口映射为模块，嵌入 C 库所需的全部适配 + 在 openkal-musl。mcpp 增加依赖的补丁与嵌入（§14）：打过补丁的库是一个新的库（变体），唯一性按程序而不按工作区判定；同一个 + mcpp 版本修正 mcpp#875（外部工具链找不到 mcpp 安装的 Ninja）与 mcpp#881（`clean --stale` 的报告自相矛盾）及其附带的两处 + 不一致。第三轮自我评审 17 条（§15.3），待评审项见 §16.4；四项均采纳（§16.5），本文定稿，按 §12 实施。 + +## 1. 问题与结论 + +mcppls 0.0.19 起四个系统都分发 `clangd.okb`。ux-mcpp 的"已有缓存的新会话"（U2-restarts）首次就绪：原生 Linux 引擎 3.3–5.1 s， +映像 21.9–33.1 s，预算 12 s。part 2 把它依次归于处理器数（H1）、锁的所有者（H3）、C 库的分配与字符串（H6、H7）与提供者索引的 +重复 mangle（H8），逐项修正后仍为 26.7–33.1 s。 + +本轮以同一份引擎源码的三组对照（§2.2）把差距分为两层： + +1. **引擎的路径选择（约 4 倍）。** mcppls-clangd 的 series 0017、0019 以 `#ifdef __linux__` 选择模块缓存与 worker：Linux 得到 + 自有缓存（以按打开文件持有的字节范围锁作租约，读模块文件不复制）与隔离 worker；其余系统得到进程内构建与读时复制。 + 对照表明首次就绪取决于 worker 策略，自有缓存决定就绪之后的 CPU（§2.2）。映像的 + 目标是 `-openkal`，在预处理器看来不是 Linux，因此即使运行在 Linux 主机上也取后者。可移植路径另有一处缺陷：H8 的修正 + （0038）在服务端从未生效（§3 E2）。 +2. **映像的运行时（约 1.6 倍）。** 同一路径上映像比原生多用 1.6 倍以上的 CPU，缺页多两个数量级。差距在 openkal-musl：分配器 + 以 mmap/munmap 往返（没有 `brk`，释放即解除映射）、字符串函数、以三次定位拼成的 `pread` 与持锁跨系统调用的描述符表。 + okrun 与 openkal 的调用本身约占 3 %。 + +由此得到本轮的四条原则（§4）与各层的增补（§5–§10）。 + +## 2. 测量 + +### 2.1 方法 + +- **探针**：mcppls 的 `probe/ux-warm-profile` 分支（不合入）。nightly 的 ux 只运行 ux-mcpp 一轮：冷阶段，随后已有缓存的新会话 + 在 `perf record -F 499 -g` 与 `perf stat` 下运行 25 s，再在 `strace -c` 下运行一次。`payload-run` 选择被测的包； + `engine-run` 以 mcppls-clangd 某次 CI 的原生 linux-x64 引擎替换包中的引擎（同一源码，另一种格式）；`engine-args` 给原生 + 引擎附加参数。 +- **符号**：映像的样本由 `clangd.oke` 的符号表还原（映像在 perf 看来是匿名可执行映射，基址取引擎启动数秒后的 + `/proc//maps`）；原生 .6 的样本由发布的符号包还原；.8 原生构件不带符号，只用计数。 +- **指标**：首次就绪受机器差异影响（同一 .7 映像两次为 21.9 s 与 32.6 s，波动约 ±25 %）；CPU 时间、缺页与系统调用次数在 + 不同机器上稳定得多，作为主要指标。 + +### 2.2 历史与对照 + +**历史**（mcppls CI 的 ux-mcpp；U2-bmi 的 176/176 是 conformance 不认识可移植路径的副本命名，`Built module` 为 0，不是重建）： + +| run | 引擎 | warm 就绪 | U10 | cold 就绪 | U4 p95 | 峰值 RSS | U2-bmi | +|---|---|---|---|---|---|---|---| +| 41197322259 | .5 原生 | 4.0 s | 4.9 s | 106.5 s | 0.45 s | 1799 MB | 0/176 | +| 41189230523 | .5 原生 | 5.0 s | 6.1 s | 135.0 s | 0.58 s | 1491 MB | 0/176 | +| 41245537861 | .6 原生 | 4.2 s | 5.3 s | 110.6 s | 0.48 s | 1659 MB | 0/176 | +| 41255535926 | .6 原生 | 3.3 s | 3.9 s | 81.2 s | 0.31 s | 1617 MB | 0/176 | +| 41269126447 | .7 映像（0.0.19） | 32.6 s | 33.4 s | 174.5 s | 0.82 s | 1380 MB | 176/176 | +| 41275927227 | .7 映像（0.0.19） | 21.9 s | 22.3 s | 113.8 s | 0.56 s | 1331 MB | 176/176 | +| 41289474740 | .7 映像（musl 0.25.0） | 30.5 s | 29.2 s | 173.7 s | 0.90 s | 1337 MB | 176/176 | +| 41306200720 | .8 映像（0038） | 26.7 s | 26.0 s | 147.5 s | 0.76 s | 1270 MB | 176/176 | + +**同源对照**（引擎 23.1.0-mcppls.8，`6336bf1`；perf stat 的 25 s 窗口自引擎启动起）： + +| 组 | 引擎与路径 | run | warm 就绪 | CPU | 缺页 | 上下文切换 | +|---|---|---|---|---|---|---| +| A | 原生，Linux 路径（默认） | 41317756737 | 4.8 s | 3.1 s | 55 k | 75 | +| B | 原生，`--modules-builder-worker-policy=in-process --modules-builder-owned-cache-payload-mib=0`（映像所取的路径） | 41319432641 | 20.4 s | 21.9 s | 7.7 k | 3.3 k | +| B2 | 原生，`--modules-builder-owned-cache-payload-mib=0`（worker 保留，自有缓存关闭） | 41326088748 | 5.4 s | 14.5 s | 164 k | 8.2 k | +| C | 映像 | 41314556906 | 33.1 s | ≥ 35.6 s（窗口内未结束） | 792 k | 20.6 k | + +A→B 是路径，B→C 是运行时。B2 把路径的两半分开：**首次就绪由 worker 策略决定**（B2 5.4 s，B 20.4 s）；自有缓存关闭不影响就绪， +但就绪之后的 CPU 是 A 的 4.7 倍（副本与其清理）。进程内策略为何使每个单元的模块规划变慢，机制尚未定位（原生 .8 构件不带 +符号）；§9.3 把它列为引擎候选之前的定位任务。 + +### 2.3 映像内的 CPU 分层（C 组与 .7 的同一探针，self 样本） + +| 层 | 占比 | 构成 | +|---|---|---| +| 引擎代码（clang、clangd、LLVM）与 libc++ | 48 % | 与原生同一份代码；其中 `indexProducerCommands` 含被调用者共 33 %（`CommandMangler` 29 %、`OptTable::ParseArgs` 27 %） | +| C 库（openkal-musl） | 34 % | `strlen` 11.3 %（八成来自 `matchOption`）、分配约 12 %（`__malloc_tc_take` 4.0、`get_meta` 3.0、`free` 1.5、`malloc` 1.3、`alloc_slot` 0.9）、`memcmp` 3.8 %、`memcpy` 3.3 % | +| 内核 | 14.5 % | 缺页、清零页、TLB 击落（`hyperv_flush_tlb_multi`）、路径查找 | +| okrun、openkal-musl 的端口层、模拟的线程局部变量 | 约 3 % | okrun 1.5 %、`__okm_get_tp` 0.9 %、`__emutls_get_address` < 0.1 % | + +### 2.4 系统调用（strace -c，warm 阶段一次） + +| 调用 | A 原生 Linux 路径 | B 原生可移植路径 | C 映像 | +|---|---|---|---| +| `pread64` | 648 k | 12 k | — | +| `read` / `lseek` | 1.7 k / 0.1 k | 0.3 k / 0.1 k | 66 k / 71 k（`pread` 拼接：定位、读、定位回） | +| `newfstatat` / `stat` | 6.6 k / 34 k | — / 124 k | 708 k / — | +| `fstatfs` | — | — | 71 k（每次要执行位的 stat 多一次） | +| `mmap` / `munmap` | 0.9 k / 0.9 k | 0.2 k / 0.2 k | 24 k / 22 k | +| `sched_yield` | — | — | 73 k（描述符表的自旋锁） | +| `clock_gettime` | —（vDSO） | —（vDSO） | 23 k | +| `futex` | 6.4 k | 3.8 k | 52 k | + +### 2.5 对 part 2 的更正 + +- **B8 的结论更正。** part 2 §25.3 B8 以 40 个模块的 `--check` 测得"映像构建与复用模块比原生快"，据此判定隔离 worker 与自有缓存 + 不是差距的原因。ux 的 warm 阶段测的是另一件事：每个单元的模块规划在可移植路径上重复做（§3 E1、E2）。同源对照 A→B 表明 + 路径本身贡献约 4 倍。B8 的"能力对等"条件保留，判定依据改为 §11 的 warm-prime 验收。 +- **H8 的结论更正。** 0038 在服务端从未生效：所有日志中 `reused unmangled` 为 0 次，C 组的 profile 与 0038 之前相同。 + 30.5 s → 26.7 s 在机器波动之内。原因见 §3 E2。 + +## 3. 归因 + +| # | 项 | 事实 | 层 | 修正（节） | +|---|---|---|---|---| +| E1 | 路径按操作系统选择 | 0017 的 worker 策略默认值、0019 的自有缓存额度、`ModuleBuildCache.cpp` 的全部实现都在 `#ifdef __linux__` 之下。自有缓存所用的设施——`openat`/`unlinkat`/`mkdirat`/`renameat`/`fstatat`、`fdopendir`、`flock`、`F_OFD_SETLK(W)`、`fsync`/`fdatasync`、`F_DUPFD_CLOEXEC`——openkal-musl 0.24.0 起全部提供（OFD 锁经 `kal_fs_lock`，三个实现都是按打开文件持有：Linux 与 macOS 为 `F_OFD_*`，Windows 为 `LockFileEx`）；只有 worker 组的内存预算（cgroup v2）没有对应 | mcppls-clangd 主 series | §9.1 | +| E2 | 0038 不生效 | `OverlayCDB::getProjectModules` 为 MDB 设置 mangler 的标识；`ModulesBuilder` 随后对每个请求调用 `MDB->setFileSystem(TFS)`（`ModulesBuilder.cpp` 七处），`CompoundProjectModules::setFileSystem` 重建 `CompileCommandsProjectModules` 并转发 mangler，新对象的标识为空，快路径的条件恒不成立。单元测试未经 `setFileSystem` | mcppls-clangd 主 series（0038） | §9.1 | +| E3 | 读时复制 | 可移植路径在每次读取前把已发布的 BMI 复制一份（warm 启动复制全部 176 个，0.8–4.2 s 内在后台完成），并有周期的副本清理 | mcppls-clangd 主 series | §9.1（有租约时不复制） | +| R1 | 分配器 | openkal-musl 没有 `brk`，mallocng 的组都来自 mmap，空组即 munmap；再用时缺页并清零。C 组缺页 792 k（B 组 7.7 k）。单线程一次 malloc+free 31–38 ns（glibc 约 4 ns） | openkal 规范（内存区域）→ mimalloc-m → openkal-musl | §5.3、§7.1、§7 | +| R2 | 字符串函数 | 0.25.0 的 `strlen` 为 SSE2 与对齐字，`memcmp` 为 8 字节；glibc 按处理器选择 AVX2/EVEX 实现，短串有专门的入口 | openkal-musl | §7 | +| R3 | 定位读写 | openkal 没有按偏移的传输，`pread`/`pwrite` 为"定位、传输、定位回"三步，且在描述符表的全局锁内；同一锁在 `openat`、`close` 中跨越系统调用持有；等锁以 `sched_yield` 自旋 | openkal 规范 → 实现 → openkal-musl | §5.1、§7 | +| R4 | stat | 要执行位时每次 stat 多一次 `fstatfs`（卷属性） | openkal-musl | §7 | +| R5 | 单调时钟 | openkal-linux 以系统调用取时间，不经 vDSO | openkal-linux（实现） | §6 | +| R6 | 线程指针 | 每次取 `errno`、线程私有数据与模拟的线程局部变量都经 `kal_task_current` 与一次散列查表，约 1 % | 不改（§5.5） | — | +| W1 | worker 的启动 | 自有缓存要求隔离 worker；映像的 worker 是一个新的 okrun 进程再接纳 146 MB 的 okb（分块并行校验，CI 的 x64 机器约 0.04–0.14 s）。可移植路径不起 worker，所以这项代价今天测不到，E1 修正后才出现 | openkal-image | §8 | +| T1 | 测试缺口 | 验收的用例是计算密集的单次 `--check`；ux 只在 Linux 上运行，且只测过原生引擎；macOS、Windows 的原生引擎同样走可移植路径，从未被测 | mcppls-clangd、mcppls | §11 | + +## 4. 原则 + +1. **按能力，不按操作系统名。** 引擎问"这个平台能否提供按打开文件持有的锁、目录句柄、打开时可删除、worker 组预算"， + 在启动时于缓存目录上实测一次，不问"是不是 Linux"。openkal 以属性位（规范 §6.2）陈述能力，映像据此选择；原生构建 + 以编译期可得的定义与同一次实测选择。 +2. **两个 series 分开。** `patches/`（主 series）只含平台无关的缺陷修正与产品功能，原生与映像都带，不出现 `__OPENKAL__`； + `openkal/patches/` 只把主 series 的平台接口映射到 openkal，不含优化；openkal 的规范与实现不出现 clangd。 +3. **底层一轮做完。** 本轮需要的规范增补、实现、C 库与加载器的改动由一张清单列全（§5–§8），在各自的 CI 中以本层的测试与 + 相对原生的基准门禁（§11）证明，然后只出一次引擎候选、只跑一次 mcppls 的完整 CI。 +4. **边界不变。** 增补遵守 part 2 §25.10：下层的东西不向上穿透；实现之间可以在下层不同（内核执行或监督执行），规范留下的 + 余地都有名字（属性位、陈述的几种结果）。 + +## 5. openkal 0.20 的规范增补 + +### 5.1 按偏移的传输（必需） + +```c +/* Transfers at `offset' without using or changing the file's position. Version 0.20. */ +kal_intptr kal_fs_read_at(struct kal_file, void* buf, kal_uintptr len, kal_u64 offset); +kal_intptr kal_fs_write_at(struct kal_file, const void* buf, kal_uintptr len, kal_u64 offset); +``` + +- 语义：结果与错误同 `kal_fs_read`/`kal_fs_write`；文件的位置在调用前后不变；同一文件上的两个按偏移传输互不影响位置， + 也不影响并发的顺序传输所见的位置。 +- Linux、macOS：`pread`/`pwrite`。Emscripten：WASI `fd_pread`/`fd_pwrite`。 +- Windows：同步句柄上带偏移的 `ReadFile` 会把位置移到传输之后，不满足"位置不变"。openkal-windows 改为**自己保存每个文件 + 句柄的位置**，所有传输都以显式偏移发出：顺序传输读写保存的位置并前移，按偏移传输不碰它。每次传输仍是一次系统调用； + `kal_fs_seek` 只改保存的位置。句柄的复制在 openkal 中是 C 库的描述（共享同一 `kal_file`），位置随之共享，与 POSIX 一致。 +- 没有文件系统的实现（openkal-opensbi、openkal-uefi）不声明 `kal_fs_*`，不受影响。 + +### 5.2 程序单元的用量与内存限额 + +```c +/* kal_program_info gains two fields (self_size grows; present says which are filled). Version 0.20. */ +#define KAL_PINFO_MEMORY ((kal_u32)1u << 4) /* memory_bytes, memory_peak_bytes */ + +struct kal_job_usage { kal_u32 self_size; kal_u32 members; kal_u64 memory_bytes; kal_u64 memory_peak_bytes; }; +int kal_process_job_usage(struct kal_job, struct kal_job_usage* out); /* KAL_PROCESS_PROP_JOB */ + +#define KAL_JOB_LIMIT_MEMORY ((kal_uintptr)1) +int kal_process_job_limit(struct kal_job, kal_uintptr kind, kal_u64 value); /* optional */ + +#define KAL_PROCESS_PROP_JOB_LIMIT_HARD /* the environment enforces the limit */ +#define KAL_PROCESS_PROP_JOB_LIMIT_SUPERVISED /* the implementation enforces it by watching */ +``` + +- **用量**：内存的计量由实现陈述（Linux：cgroup 的 `memory.current`，否则成员常驻集之和；macOS：成员 `phys_footprint` 之和； + Windows：作业的已提交私有内存）。规范只要求同一实现前后一致、单位为字节、`peak` 不减。使用者把它当作护栏，不当作计费。 +- **限额**：单元的内存超过 `value` 时，单元的所有成员被结束，`kal_process_wait` 对它们报告的结束原因为"超出限额" + （`kal_program_info.ended` 增加一个值）。两种执行方式： + - `HARD`：环境执行。Linux 在调用者所在的 cgroup v2 已委派写权限时，为单元建子组（成员经 `CLONE_INTO_CGROUP` 或 + `cgroup.procs` 进入）并写 `memory.max`；Windows 为作业对象的 `JOB_OBJECT_LIMIT_JOB_MEMORY`。Windows 超出时是分配失败而不是 + 结束，实现经作业的完成端口收到 `JOB_OBJECT_MSG_JOB_MEMORY_LIMIT` 后结束单元，使两个系统的可观察结果相同（事件驱动，不采样）。 + - `SUPERVISED`：实现执行。第一次设限时起一个监督线程，按距限额的远近自适应采样（< 50 %：2 Hz；50–80 %：10 Hz；> 80 %： + 50 Hz）。成员是经本实现启动进单元或加入单元的程序，退出以事件跟踪（Linux pidfd，macOS kqueue `NOTE_EXIT`），不扫描进程表。 + 超出量的上界为"采样间隔 × 单元的分配速度"，实现在文档中陈述它的间隔。Linux 在 cgroup 未委派时与 macOS 都是这一方式。 + - 两者都不能时报告 `kal_err_not_supported`。 +- **代价**：不设限额的程序没有监督线程。16 个成员、每次采样约 10 µs、20 Hz 时约 3 ms/s（一个核的 0.3 %）；`HARD` 没有额外代价。 +- 边界核对：cgroup、作业对象、`phys_footprint` 都只在实现内部；上层观察到的是字节数、两种执行方式之一与一个结束原因。 + +### 5.3 内存区域（可选，`KAL_MEMORY_PROP_REGIONS`） + +`kal_alloc` 陈述的是"取得一块可用的内存"，实现建在环境的分配器之上（Linux、macOS 的实现带小尺寸类与一把锁；Windows 建在 +进程堆上，超过堆对齐的请求以多分配实现）。虚拟内存的另一组能力——按大粒度对齐地保留地址、按需提交、交还驻留而保留地址、 +整段释放——规范至今没有陈述。它是 POSIX `mmap`/`munmap`/`madvise` 与 Win32 `VirtualAlloc`/`VirtualFree` 的公共部分，通用分配器、 +垃圾回收的堆、编译器与解释器的区域分配、稀疏的大缓冲区都建在它上面。规范以五个操作陈述它，语义只取自虚拟内存模型： + +```c +/* openkal.memory, version 0.20. Offered where KAL_MEMORY_PROP_REGIONS is in kal_memory_props(). */ +void* kal_memory_reserve(kal_uintptr size, kal_uintptr align, kal_uintptr flags); /* KAL_MEMORY_COMMIT: usable at once */ +int kal_memory_commit(void* addr, kal_uintptr len); /* pages become usable; their contents read as zero */ +int kal_memory_decommit(void* addr, kal_uintptr len); /* resident memory and commit charge returned; commit before reuse */ +int kal_memory_discard(void* addr, kal_uintptr len); /* resident memory may be reclaimed; stays usable; contents unspecified */ +void kal_memory_release(void* addr, kal_uintptr size); /* the whole reservation */ +kal_uintptr kal_memory_props(void); /* KAL_MEMORY_PROP_REGIONS, KAL_MEMORY_PROP_OVERCOMMIT */ +``` + +- 地址与长度是 `kal_memory_granularity()` 的倍数；`align` 是 2 的幂且不小于粒度。访问未提交的页是未定义的。 +- `KAL_MEMORY_PROP_OVERCOMMIT`：环境不分开计提交（Linux、macOS），保留即可用，`commit` 不需要系统调用。 +- `commit` 之后读到零；`discard` 之后内容未指明（POSIX `MADV_FREE`、Win32 `MEM_RESET` 的语义）。不陈述保护：规范没有对映射 + 保护的操作（`memory.h` 已写明），本轮不变。 + +| 操作 | Linux | macOS | Windows | +|---|---|---|---| +| reserve | `mmap` 多取 `align` 后裁去两端 | 同左 | `VirtualAlloc2` 带 `MEM_ADDRESS_REQUIREMENTS`（不可用时保留后按对齐重试） | +| commit | 无操作（OVERCOMMIT） | 无操作（OVERCOMMIT） | `VirtualAlloc(MEM_COMMIT)` | +| decommit | `madvise(MADV_DONTNEED)` | `madvise(MADV_FREE_REUSABLE)` | `VirtualFree(MEM_DECOMMIT)` | +| discard | `madvise(MADV_FREE)`（内核不支持时 `MADV_DONTNEED`） | `madvise(MADV_FREE_REUSABLE)` | `VirtualAlloc(MEM_RESET)` | +| release | `munmap` | `munmap` | `VirtualFree(MEM_RELEASE)` | + +**C 库建在它上面**（openkal-musl，§7），这也是通用性的检验——任何程序自带的分配器（jemalloc、应用的区域分配）都因此得到 +真实的语义，而不只是 C 库自己的 `malloc`： + +| POSIX | 声明 `REGIONS` 时 | 不声明时（现状） | +|---|---|---| +| 匿名 `mmap`（`PROT_NONE` 只保留） | `reserve`（`PROT_NONE` 不带 `COMMIT`） | `kal_alloc` | +| `munmap`（整段） | `release` | `kal_free` | +| `madvise(MADV_FREE)` | `discard` | 空操作 | +| `madvise(MADV_DONTNEED)` | `decommit`；不带 OVERCOMMIT 时随后 `commit`（POSIX 要求之后读到零且可直接使用） | 空操作 | +| `mprotect` | 不变：`ENOSYS` | `ENOSYS` | + +没有虚拟内存的实现（openkal-opensbi、openkal-uefi、openkal-emscripten）不声明 `REGIONS`。 + +### 5.4 观测（可选，`openkal.prof` 与 `openkal.debug`） + +```c +int kal_prof_available(void); /* 0 unless the loader profiles */ +int kal_prof_counter(const char* name, kal_uintptr len, volatile kal_u64* cell); /* registered once; read at snapshots */ +void kal_prof_mark(const char* name, kal_uintptr len, int begin); /* a phase's begin or end */ +kal_u64 kal_prof_cycles(void); /* a monotonic counter, cheap */ +kal_intptr kal_debug_symbolize(kal_uintptr pc, char* buf, kal_uintptr len); /* from the image's symbol table */ +kal_intptr kal_debug_backtrace(kal_uintptr* pcs, kal_uintptr max); /* frame pointers */ +``` + +- 加载器不在剖析时，`kal_prof_available` 为 0，C 库与程序不注册任何东西；注册的计数器是程序自己的内存，加载器只在快照时读， + 不增加热路径的代价。 +- 只出现次数、时长、镜像内的地址与程序给的名字，不出现环境的标识。 + +### 5.5 不改的项 + +线程指针（R6）：实测约 1 %，涉及线程寄存器的约定，属于 openarch；本轮不改规范。决定的依据与数据记录于此。 + +## 6. 三个实现（openkal-linux 0.21、openkal-macos 0.18、openkal-windows 0.16、openkal-emscripten 0.9） + +| 项 | Linux | macOS | Windows | Emscripten | +|---|---|---|---|---| +| §5.1 | `pread`/`pwrite` | `pread`/`pwrite` | 自存位置，全部传输带偏移 | `fd_pread`/`fd_pwrite` | +| §5.2 用量 | cgroup `memory.current` 或 `/proc//statm` 之和 | `proc_listpgrppids` + `proc_pid_rusage` | `QueryInformationJobObject` | 不声明 | +| §5.2 限额 | 已委派：HARD；否则 SUPERVISED | SUPERVISED | HARD | 不支持 | +| §5.3 | 见 §5.3 的表 | 见 §5.3 的表 | 见 §5.3 的表 | 不声明 | +| §5.4 | 由 okrun 提供（§8） | 同 | 同 | 不声明 | +| R5 单调时钟 | 经 vDSO（okrun 的 `AT_SYSINFO_EHDR`） | 已是用户态 | 已是用户态 | — | + +一致性测试（openkal/conformance）：按偏移传输不改位置、与顺序传输并发；用量单调、成员计数；限额：HARD 与 SUPERVISED 都在 +超出后结束全部成员、结束原因可见、超出量不超过实现陈述的上界；区域：对齐、未提交与已提交的边界、`decommit` 后重新 `commit` 读到零、`discard` 后可读写且能测得驻留的系统上驻留下降、`release` 后地址可再保留。 + +## 7. openkal-musl 0.26 + +| 项 | 做法 | 证明（§11 L1） | +|---|---|---| +| 分配器（R1） | 两个 feature，恰好一个：`malloc-mimalloc`（默认）、`malloc-mallocng`（musl 自己的，含 0.25.0 的线程缓存，代码与 0.25.0 相同）。两个都开或都不开时编译期报错，并指出依赖边上的写法（`default-features = false` 再选另一个）。mimalloc 以 `compat.mimalloc` 嵌入（§14 的 `embed`，用本库的编译参数与头文件编入本库，不另成一个库，因此没有"C 库依赖分配器、分配器依赖 C 库"的环），适配只在本库的 `port/`（§7.1）。线程记录中的分配器字段两种分配器共用一个指针 | 两个 feature 都跑：1/2/4/8 线程的 malloc+free、LLVM 尺寸分布的回放、每 GB 分配与释放的缺页数；mimalloc ≤ glibc，缺页 ≤ glibc 的 2 倍；mallocng 作为基线记录 | +| 内存的 POSIX 面 | `mmap`/`munmap`/`madvise` 按 §5.3 的表建在内存区域上 | `MADV_DONTNEED` 之后读到零、`MADV_FREE` 之后可写、`PROT_NONE` 保留后不可提交即访问为未定义（只测前两者） | +| 字符串（R2） | `memcpy`/`memmove`/`memset`/`memcmp`/`memchr`/`strlen`/`strchr`/`strcmp`：x86_64 启动时以 CPUID 选 AVX2 或 SSE2 实现，aarch64 用 NEON；短串（≤ 32 B）有不进循环的入口；读取不越过字符串未到达的页 | 0–4096 B、每种对齐，结果与逐字节实现一致；页边界用例；时间 ≤ glibc | +| 定位读写（R3） | `pread`/`pwrite`/`preadv`/`pwritev` 直接调用 `kal_fs_read_at`/`kal_fs_write_at` | 4 线程并发 `pread` 同一描述符：结果正确、位置不变、≤ glibc 的 1.15 倍 | +| 描述符表（R3） | 查找无锁（槽位原子读，带代数）；`openat`/`close`/`dup` 只在保留与发布槽位时持锁，系统调用在锁外；等锁经 futex 而非 `sched_yield` | 8 线程 open/close/stat 的吞吐 ≥ glibc 的 0.85 倍；`sched_yield` 为 0 | +| stat（R4） | 卷属性按设备号缓存，`fstatfs` 每个卷一次 | stat 的系统调用数 = 1 | +| 统计 | 带 `kal_prof_available` 时注册：分配次数与字节、驻留、discard 次数、锁等待、描述符表争用 | okprof 报告中可见 | +| worker 的峰值内存 | `getrusage(RUSAGE_CHILDREN)` 与 `wait4` 的 `ru_maxrss` 取自 `KAL_PINFO_MEMORY` | 单元测试 | + +### 7.1 mimalloc 的三处归属 + +| 包 | 内容 | 不含 | +|---|---|---| +| `compat.mimalloc`（mcpp-index 的内联描述，与 `compat.boost-*` 同一形式） | 上游 v3.6.1 的源码包与 SHA256、构建描述（源文件、宏、include 目录），Linux、macOS、Windows | 任何 openkal 的东西 | +| `mcpplibs.mwrap.mimalloc`（新仓库 `mcpplibs/mimalloc-m`，按 mcpp-index `docs/mwrap.md`） | 依赖 `compat.mimalloc`，把 `mi_*` 与类型映射为模块 `mcpplibs.mwrap.mimalloc`，`MI_*` 映射为 `.macros`，不能映射的宏在 ``；默认模板；两份 README；版本 `3.6.1`，标签 `mwrap-3.6.1` | port、补丁、实现 | +| openkal-musl（`malloc-mimalloc`） | `port/src/okm_mimalloc_prim.c`：mimalloc 的 OS 层（取代上游 `src/prim/prim.c` 的平台选择，经 §14 的 `exclude` 不编译它），只调用 `kal_*`：内存区域（不声明时退到 `kal_alloc`）、粒度、单调时钟、随机数、`MIMALLOC_*` 环境变量、标准错误、进程信息（`KAL_PINFO_CPU`、`KAL_PINFO_MEMORY`）；保护页与大页回答不支持。`port/src/okm_mimalloc_glue.c`：`malloc` 族的名字、线程堆指针放在线程记录中（上游的 `MI_TLS_PTHREAD_SLOT_OFS`）、`fork` 前后与线程结束的钩子接 musl 已有的位置 | 对上游源码的修改：优先只用上游的扩展点（prim 文件、配置宏）。确需修改时以 §14 的 `patches` 记录在本库，并在 `musl/PATCHES.md` 旁的 `port/mimalloc/PATCHES.md` 写明理由 | + +**一个程序一份实现：** 在 `os = "openkal"` 上，选了 `malloc-mimalloc` 的 C 库嵌入了 `compat.mimalloc`；程序若再经 mimalloc-m 依赖 +同一变体的 `compat.mimalloc`，按 §14 的规则由嵌入者供给，程序只编译接口；变体不同（C 库带了补丁）则报错，指出两条边。 + +**测试：** mimalloc-m 的 CI 跑上游的 `test-api`、`test-stress` 与模板（三个系统原生）；mcpp-index 的 installed consumer 测试按 mwrap +的规定。映像上的运行在 openkal-musl 的 L1（`malloc-mimalloc` 与 `malloc-mallocng` 两份）。 + +## 8. openkal-image 0.2（okrun、okld、okprof） + +- **okrun 的剖析**：`okrun --prof=calls,samples[,counters] -o `，或环境变量 `OKRUN_PROF`（映像由其他程序启动时，例如 + mcppls 启动 `clangd.okb`）。 + - `calls`：装入时把 `kal_*` 的导入表项换成计数桩，记录每个接口的次数、总时长与分位数、错误、字节，按线程；不剖析时导入表 + 直接指向实现。 + - `samples`：按线程的定时采样，帧指针回溯，以 `.oke` 的符号表还原；Linux 上同时写 `/tmp/perf-.map`，系统的 perf 可直接 + 还原映像的符号。 + - `counters`：§5.4 注册的计数器与阶段标记。 +- **okprof**（openkal-tools 中的单独程序）：`okprof record -- `、`okprof report`（按层归类：引擎、C 库、libc++、端口层、 + okrun、内核时间）、`okprof flame`、`okprof diff`（映像对原生、两次之间；作为 CI 门禁的输入）。 +- **worker 的启动（W1）**：接纳的校验按文件身份记忆。okrun 校验通过后，在用户缓存目录写一条记录（设备、inode、大小、 + `mtime`、`ctime`、清单摘要，记录本身以清单摘要为键）；同一文件再次启动时身份与记录一致即跳过分块校验，只校验清单签名。 + 文件被替换或修改则 `ctime` 变化，记录失效。记录只对同一用户有效（目录权限 0700）。验收报告 worker 的启动时间。 + +## 9. mcppls-clangd 0.0.20（#16，一个 PR） + +### 9.1 主 series（`patches/`，原生与映像都带） + +- **0038 修正（E2）**：`CompoundProjectModules` 保存 mangler 的标识，`setFileSystem` 重建子对象后转发；单元测试经 + `ModulesBuilder` 的请求路径（含 `setFileSystem`）断言每个清单快照只 mangle 一次。0038 尚未发布，直接修改。 +- **按能力选择（E1）**：新补丁（UP-03）把 `#ifdef __linux__` 换成一个平台接口 `ModuleCachePlatform`： + - 能力：按打开文件持有的字节范围锁、目录句柄的相对操作、打开时可删除、worker 组预算（`HARD`/`SUPERVISED`/无）； + - 操作：锁范围、相对目录操作、启动 worker、设预算、读用量。 + - 实现：POSIX（Linux 与 macOS 原生，`F_OFD_*` 存在即可）；Linux 的 cgroup 预算在一个单独的文件中；Windows 原生回答"无"， + 走可移植路径。 + - 选择：启动时在缓存目录实测一次（加锁、相对创建与删除、打开时删除），实测失败退到可移植路径并在 `--check` 与日志中 + 报告原因。worker 策略与自有缓存额度的默认值由能力决定。 +- **可移植路径（E3）**：有租约时不复制；只在拿不到租约的平台读时复制。 + +### 9.2 openkal series（`openkal/patches/`，只有映像带） + +- `ModuleCachePlatform` 的 openkal 实现：能力取自 `kal_fs_props`（`LOCKS`、`REMOVE_OPEN`）与 `kal_process_props` + （`JOB`、`JOB_LIMIT_*`）；预算经 `kal_process_job_limit`，用量经 `kal_process_job_usage`；不含 cgroup。 +- worker 经 `kal_env_loader` 启动（已有，B4）；0017 中映像无地址空间上限的说明改为"由单元的内存限额约束"。 + +### 9.3 引擎候选之前的定位 + +- 进程内策略使模块规划变慢的机制（§2.2）：mcppls-clangd 的候选 CI 上传原生引擎的符号包（与发布同一做法），探针以 + `engine-run` + `engine-args` 跑 A、B、B2 三组并符号化，定出进程内路径上多做的工作。若它是主 series 的缺陷，修正进入 §9.1， + 使原生 Windows（仍走进程内）同样受益；本轮的引擎候选只出一次，这一步在候选之前完成。 + +### 9.4 验收与门禁 + +- **warm-prime**（新）：LSP 回放一个 176 个模块的工作区（ux-mcpp 的清单与 481 条编译命令），冷一次、已有缓存的新会话三次， + 报告首次就绪、CPU、缺页、worker 启动时间。四个系统 × {原生, 映像}，映像对原生 ≤ 1.15（中位数）。 +- **路径对等**：映像在三个系统上报告自有缓存与 worker 为启用；原生 Windows 报告可移植路径（并测其时间，作为记录）。 +- `parallel-index`、`module-parallelism` 等已有门禁不变。 + +## 10. mcppls 0.0.20（#75） + +- 引擎的身份增加特性 `owned-module-cache`；mcppls 对报告该特性的引擎给 `--modules-builder-owned-cache-payload-mib`（part 2 B6 的 + 条件由"有无 runner"改为"引擎是否具备"），单元测试覆盖四种组合。 +- conformance：U2-bmi 认识可移植路径的副本命名（`-<时间戳>.-<随机>.pcm`）；U11 的锁注入认识 + `producer-chain-v2` 一级目录。可移植路径仍可能出现（原生 Windows，或实测失败的环境）。 +- nightly 的 ux 增加 macOS arm64 与 Windows x64 的 ux-mcpp（只在 nightly，不在 PR）。 + +## 11. 测试与门禁 + +| 层 | 仓库 | 内容 | 门禁 | +|---|---|---|---| +| L1 原语 | openkal-musl（bench，hosted 与映像各一份，同一机器） | 字符串（长度 × 对齐）、分配（1/2/4/8 线程、LLVM 尺寸分布、缺页数）、锁、线程私有数据、`stat`/`open`/`close`/`pread`（单线程与 4 线程）、时钟 | 映像 / 原生 ≤ 1.15，目标 ≤ 1.0；缺页 ≤ 2 倍 | +| L1 规范 | openkal conformance | §6 所列 | 全部通过，三个系统 | +| L2 中层 | openkal-llvm-runtime（bench） | `OptTable` 解析 481 条真实命令、`StringMap`/`DenseMap` 的增删、Bitstream 读取、Dex 建索引 | 映像 / 原生 ≤ 1.15 | +| L3 引擎 | mcppls-clangd 验收 | §9.4 | 映像 / 原生 ≤ 1.15；路径对等 | +| L4 产品 | mcppls ux | U2、U10、U11、U15 | 预算不变（warm 12 s） | + +每层的 CI 都附 okprof 的分层报告（L1–L3），与上一版本做 `okprof diff`。 + +## 12. 交付与依赖 + +``` +mcpp 2026.11（§14）──────────────────────────────────────────┐ +openkal 0.20 ─┬─ openkal-linux 0.21 ──┐ │ + ├─ openkal-macos 0.18 ──┤ │ + ├─ openkal-windows 0.16 ┼─ openkal-image 0.2 ──┴─ openkal-musl 0.26 ── openkal-llvm-runtime 0.17 ── openkal-platform 0.21 ── os 0.2.7 + └─ openkal-emscripten 0.9┘ compat.mimalloc 3.6.1（mcpp-index）─┘ │ + └─ mimalloc-m 3.6.1（不在关键路径） │ + mcppls-clangd 0.0.20（#16）── mcppls 0.0.20（#75）── mcpp-ide 0.2.0（#3）── xim-pkgindex +``` + +1. mcpp 的 PR（§14）与底层各仓库同时开始；底层开同名分支 `openkal-0.20`，以 working-trees CI 联调；mcpp 的候选二进制供 openkal-musl + 的分支 CI 使用。L1、L2 全部达标前不发布。 +2. 引擎 #16 在底层分支上出一次候选（working-trees 构建），探针跑 A/B/C/D 四组（D：新映像），L3 达标。 +3. 发布：mcpp → `compat.mimalloc` 与 mimalloc-m 进入索引（`pkgs/c/compat.mimalloc.lua`、`pkgs/mwrap/mimalloc.lua`、 + `tests/mwrap/mimalloc/`）→ openkal → 三个实现 → 索引 → openkal-image → 索引 → openkal-musl → openkal-llvm-runtime → 索引 → + openkal-platform → os；引擎发布；mcppls 切换锁、完整 CI、发布；mcpp-ide；索引与 xlings 的实机验证。 + +**并行**：等底层 CI 时写引擎的两个补丁与 warm-prime 验收；等引擎候选时写 mcppls 的 conformance 修正与 nightly 的 ux 矩阵。 + +## 13. 预期 + +| 指标 | 现在（C） | 本轮之后 | +|---|---|---| +| warm 就绪（Linux x64） | 26.7–33.1 s | ≤ 原生 × 1.15（约 6 s） | +| warm 的 CPU | ≥ 35.6 s | ≤ 原生 × 1.15（约 3.6 s） | +| 缺页 | 792 k | ≤ 原生 × 2 | +| macOS、Windows | 可移植路径（未测） | 映像走自有缓存与 worker，L3、L4 有数 | + +拆分：E1 换到 worker 路径，就绪约 4 倍（B→B2）；自有缓存使就绪后的 CPU 降约 4.7 倍（B2→A）；E2 去掉进程内路径约 33 % 的 CPU；R1–R4 去掉约 1.6 倍（B→C）。 + +## 14. mcpp 2026.11：依赖的补丁与嵌入、#875、#881 + +### 14.1 补丁与嵌入 + +C 库这一类包要用上游的源码而不拷贝它：在源码上加几处改动，或者把它编进自己。现在只能把上游整份放进自己的仓库（openkal-musl +对 musl 正是如此，因为它本身是 musl 的移植），这违背"依赖，不拷贝"。mcpp 在依赖边上增加两个字段： + +```toml +[dependencies] +"compat.mimalloc" = { version = "3.6.1", embed = true, exclude = ["src/prim/prim.c"], + patches = ["port/mimalloc/0001-xxx.patch"] } +``` + +**变体。** 打过补丁的库是另一个库：它的身份是 `(包, 版本, 补丁集)`，补丁集的摘要按顺序取各补丁规范化后的内容（不取文件名）。 + +- 源码：在已校验的上游源码上按序应用，得到变体自己的源码目录（`/src/<包>/<版本>+p<摘要>`）；任何一段不能精确应用即 + 报错，指出补丁与段（不做模糊匹配，结果确定）。补丁格式为 unified diff 或 `git format-patch`，路径相对于包的根目录；应用器 + 内置于 mcpp，不依赖系统的 `patch` 或 git，字节精确（换行不转换，由补丁本身决定）。 +- 构建、BMI 与缓存目录按变体区分；同一补丁集在不同成员、不同项目中得到同一变体，共享构建。 +- `mcpp.lock` 为变体记录版本、各补丁的路径与 sha256 和补丁集摘要；补丁改动即重新解析与构建。 +- 只有以源码分发的包能打补丁；只给预编译产物的包拒绝 `patches`。补丁文件属于声明它的包，随它发布。 + +**变体是一个新的库；约束只在一个程序之内。** 变体的身份与上游不同，mcpp 已有的规则因此直接适用，不需要为补丁另立规则： + +- 工作区（docs/05"In a workspace"）拒绝的是"一个身份以两种来源或两个引用被声明"——一次配置只构建一个包的一份检出。两个变体 + 是两个身份，两个成员各用一个不触犯这一条；它们在同一次配置中各自构建在自己的目录中。 +- 一个程序中不能有两个库提供同一个模块名或同一组符号——这对任意两个库都成立，变体也不例外。同一个包的两个变体（包括"上游"与 + "带补丁"）导出相同的模块名与符号，因此不能在一个程序的链接闭包中相遇。 + +| 情形 | 结果 | +|---|---| +| 工作区中两个程序各用一个变体 | 允许；构建、BMI、缓存各自独立 | +| 同一程序中同一变体经多条边到达 | 合一 | +| 同一程序中同一个包的两个变体 | 报错，列出引入两者的边，指出它们提供相同的模块名 | +| 已发布的库在公开的依赖边上带补丁 | 允许：它的使用者得到这个变体；与另一变体在同一程序中相遇时按上一行报错 | +| 两个成员以两个补丁集声明同一个包、同时被一个命令选中 | 允许（两个身份）；只有它们进入同一程序时报错 | + +**嵌入（`embed = true`）。** 依赖的源文件（按它的构建描述与 feature 选出，再减去 `exclude`）以使用者的编译参数、include 目录与宏 +（再加上依赖自己的 include 目录与宏）编入使用者，不另成一个库；依赖不为目标单独构建，因此不需要目标的 C 库——C 库可以嵌入 +它所需的代码而不成环。嵌入者即为该变体在程序中的供给者：同一程序中其他边依赖同一变体时，只编译接口、不再链接实现；依赖 +另一变体时按上表报错。`exclude` 只在 `embed` 下允许（普通依赖去掉源文件会改变它的公开行为，那应当是一个补丁）。 + +**与已有机制的关系。** `provides` 的能力选择不变；变体是包身份的扩展，不是新的能力。xim 的安装钩子与描述中的 +`generated_files` 在补丁之后执行。先例：vcpkg 的 portfile 补丁、Conan 的 `patches`、Nix 的 `patches`、Spack 的 `patch()`；Cargo 的 +`[patch]` 是整份源码替换，粒度不同。 + +本轮的使用者是 openkal-musl 对 `compat.mimalloc`（`embed`、`exclude`，预期不需要 `patches`）。musl 本身留在 openkal-musl 中：它是 +移植而不是几处补丁（`musl/PATCHES.md` 中是成体系的改写与新增的 port 层），以变体表达反而失去它作为移植的可读性。 + +### 14.2 mcpp#875：外部工具链找不到 mcpp 安装的 Ninja + +`ninja_program_for`（`src/build/ninja_backend.cppm`）只在编译器所属的 provider 包旁找 Ninja，找不到就返回裸名 `ninja`，交给 PATH。 +mcpp 在首次初始化时已把 Ninja 装进自己的 home（`ensure_ninja`，`xim:ninja@`），但这条路径没有回查它；编译器来自外部 +路径（`MCPP_TOOLCHAIN=path:…`）时就落到 PATH，PATH 中没有即失败（Windows、MSYS2 的 GCC 复现）。 + +修正：按序查找——编译器所属 provider 包旁；mcpp home 中按钉住版本安装的 Ninja（`xim_tool_root(env, "ninja")`）；mcpp 的沙箱 +`bin`；最后 PATH。都没有时报错列出查过的位置与安装命令，而不是 "cannot start 'ninja'"。`MCPP_NINJA`（`toolchain_env.cpp`，构建 +程序看到的）用同一个函数，结果一致。测试：外部工具链 + 清空的 PATH 在三个系统上构建成功；缺 Ninja 时的报错文本。 + +### 14.3 mcpp#881：`clean --stale` 的报告 + +`clean_stale()` 对因"比 `--older-than` 新"而保留的目录逐个打印 `kept … not in the record`，没有删任何东西时又打印"每个指纹目录都 +记录为当前"——两句矛盾，而后一句是用户据以判断的那句。原因是保留的目录既不计入 `removed` 也不计入 `failed`。 + +修正：增加 `kept` 计数，摘要在三种情形下都为真（没有可查的、全部当前、保留了 N 个未记录但仍新的目录，并给出 `--older-than`）。 +同一 issue 附带的两处一并修正： + +- "新"的定义只有一个：`clean --stale` 的 `--older-than` 默认取 `[build] keep_days`（构建后清理用的同一个值，默认 3 天），两个入口 + 不再各有默认；文档与 `--help` 同步。 +- SubOS 的修复提示写成 `XLINGS_HOME= xlings self doctor --fix`（与 `src/provider/xlings/xlings.cppm` 已有的说明一致）； + 不带 `XLINGS_HOME` 时，打包的 xlings 会在有系统 xlings home 的机器上修错地方。 + +补丁变体（§14.1）增加了缓存中的源码目录与构建目录，`clean --stale` 与缓存的回收按变体识别它们：记录中的变体是当前的，不在记录 +中且超过 `keep_days` 的变体目录被回收。 + +### 14.4 交付 + +一个 PR、一个版本（mcpp 2026.11.x）：§14.1–§14.3、docs（依赖清单的字段、变体与唯一性、`clean`）、测试（补丁的应用与失败、变体 +的身份与缓存、同一程序两个变体的报错、工作区两个程序各用一个变体、`embed` + `exclude` 编入使用者且不成环、锁文件、§14.2、 +§14.3 的三种摘要）。openkal-musl 0.26 的清单要求该版本（`mcpp = ">=2026.11.x"`），因此它在交付链中排在 openkal-musl 之前（§12）。 + +## 15. 自我评审 + +### 15.1 第一轮（rev 1） + +| # | 视角 | 发现 | 处理 | +|---|---|---|---| +| 1 | 架构 | `ModuleCachePlatform` 若写在 openkal series 里，主 series 仍以 OS 选择，映像与原生分叉 | 接口与 POSIX 实现在主 series，openkal series 只有映射（§9） | +| 2 | 边界 | 用量的计量三系统不同，若规范写"常驻集"会让 Windows 的 commit 计量不合规 | 计量由实现陈述，规范只要求单位、单调与一致（§5.2） | +| 3 | 边界 | `SUPERVISED` 的超出量若不陈述，使用者无法判断是否足够 | 实现陈述采样间隔，上界写入规范（§5.2） | +| 4 | 跨平台 | Windows 同步句柄的带偏移 `ReadFile` 会移动位置 | openkal-windows 自存位置，全部传输带偏移（§5.1） | +| 5 | 性能 | 分配器保留区域后若不释放驻留页，长时间运行的服务端内存只增不减 | 内存区域的 `discard`/`decommit`（§5.3）与空闲阈值 | +| 6 | 性能 | 自有缓存要求 worker，映像每个 worker 重新接纳 146 MB，冷构建 176 个模块会多出数十秒 | okrun 按文件身份记忆接纳（§8），验收报告 worker 启动时间 | +| 7 | 安全 | 接纳记忆可能被同机其他用户伪造 | 记录在 0700 的用户目录，以清单摘要为键，身份含 `ctime`（§8） | +| 8 | 稳定性 | 能力的编译期判断在挂载的网络文件系统上可能不成立 | 启动时在缓存目录实测，失败退到可移植路径并报告（§9.1） | +| 9 | 兼容与升级 | 旧 mcppls 不给映像额度，新引擎默认值由能力决定 | 默认值在引擎内由能力决定，mcppls 的参数只是覆盖；旧 mcppls 也得到快路径 | +| 10 | 兼容 | `kal_program_info` 增字段 | `self_size` 与 `present` 已为此设计，旧调用方不受影响 | +| 11 | 简洁 | 剖析若做成单独的 okrun 构建，会多一个分发物 | 一个 okrun，剖析关闭时零代价；嵌入式可在编译期去掉 | +| 12 | 测试覆盖 | 缺陷 E2 的单元测试绕过了真实请求路径 | 测试经 `ModulesBuilder` 的请求路径（§9.1） | +| 13 | 测试覆盖 | ux 只在 Linux | nightly 增加 macOS、Windows（§10） | +| 14 | 交付 | 分配器替换的风险最大，若在 L3 才暴露问题会再走一次引擎周期 | L1 用 LLVM 尺寸分布回放与 8 线程压力，L2 跑 LLVM 的数据结构；两层通过才出引擎候选 | +| 15 | 性能 | 分配器的线程堆经"当前线程"取得；映像中当前线程是一次跨映像调用加一次散列查表（R6），在每次分配上约 2–3 ns，与 glibc 的 4 ns 同量级，R6 的"约 1 %"在新分配器下不再成立 | L1 的单线程 malloc+free 门禁直接量它；不达标时 openkal-musl 把线程记录改为按栈对齐定位（线程栈由 C 库按 2 的幂对齐分配，记录在栈底，当前线程为栈指针按位与；主线程与不在 C 库栈上的上下文退回查表）。这是 C 库内部的做法，不改规范，在 L1 内决定，不进入引擎周期 | +| 16 | 性能 | 0017 每构建一个模块起一个 worker（`ModulesBuilder.cpp` 对 `buildModuleInWorker` 每个模块调用一次）；映像的 worker 除接纳外还要把约 60 MB 的段写入 `kal_exec` 区域，冷构建 176 个模块约多 5 s CPU | warm-prime 验收同时以冷阶段为门禁（映像 / 原生 ≤ 1.15）；D 组报告 worker 的启动时间。超出时在 openkal-image 内把只读段改为从映像文件按页映射（`kal_exec` 的文件映射），属于加载器内部，在 L1 内决定 | +| 17 | 兼容 | 分配器替换影响 `malloc_usable_size`、`fork` 的加锁顺序、C 库内部的早期分配 | openkal-musl 的 malloc-threads、fork 示例与 musl 的 libc-test 分配部分作为 L1 的正确性门禁 | +| 18 | 归因的完备 | B2 表明就绪由 worker 策略决定，但进程内路径慢在哪里未定位；若不定位，原生 Windows 与任何实测失败、退回进程内的环境仍慢 | §9.3：候选之前以符号化的原生三组定位，主 series 的缺陷在同一候选中修正 | + +### 15.2 第二轮（rev 2） + +| # | 视角 | 发现 | 处理 | +|---|---|---|---| +| 19 | 架构 | 只加 `discard` 不够：`kal_alloc` 是带锁的通用分配器，Windows 上建在进程堆上，32 MiB 对齐的请求会多占一倍提交 | 内存区域五个操作（§5.3），`kal_alloc` 不变 | +| 20 | 依赖 | openkal-musl 依赖 mimalloc-m，而映像形态的 mimalloc 需要 C 库，形成环 | `libc` 形态为源码包，只用编译器的独立头文件，经嵌入接口取得 C 库的东西（§7.1）；依赖链为 openkal → mimalloc-m → openkal-musl | +| 21 | 一致性 | 一个映像中若 C 库嵌入一份 mimalloc、程序又经 mimalloc-m 链接一份，会有两个分配器 | 实现的拥有者在 openkal 目标上是 C 库，mimalloc-m 只提供接口（§7.1，符合 mwrap 的"一份实现一个拥有者"） | +| 22 | 一致性 | openkal-musl 的 `none` 目标（openkal-opensbi）没有虚拟内存 | prim 在不声明 `REGIONS` 时以 `kal_alloc` 取已提交内存；所有目标一个分配器，mallocng 整体删除 | +| 23 | 边界 | mimalloc 的 OS 层若经 C 库的 `mmap`/`madvise` 走，会经过描述符表与 POSIX 语义，且 `madvise` 在 openkal-musl 中是空操作 | prim 只调用 `kal_*`（§7.1），不经 POSIX | +| 24 | 性能 | 线程堆指针存在线程记录中，取它要经 `__pthread_self`（R6 的查表） | §15 第 15 条的门禁与后备做法覆盖；L1 的单线程门禁直接量 `mi_malloc`+`mi_free` | +| 25 | 兼容 | openkal-musl 0.25.0 对 mallocng 的改动（`__malloc_release_slot`、`free.c` 的拆分、`pthread_create` 的 flush）随 mallocng 一起删除，musl/PATCHES.md 对应条目改写 | musl 的 PATCHES.md 记录删除及原因；malloc-threads 示例保留为回归测试 | +| 26 | 规范完备 | `kal_memory_props` 是新函数，旧实现没有 | 0.20 起 `openkal.memory` 必须提供它（可以回答 0）；四个实现同时实现 | +| 27 | 交付 | 新仓库首发要过 mwrap 的发布审计（模板、两份 README、许可、installed consumer 测试），比普通依赖多一轮 | mimalloc-m 与 openkal 同时开始，按 mwrap 的检查点提交，不在关键路径的末端 | + +### 15.3 第三轮（rev 3） + +| # | 视角 | 发现 | 处理 | +|---|---|---|---| +| 28 | 架构 | rev 2 的 mimalloc-m 带 openkal 的 prim 与嵌入接口，接口包承担了实现与平台适配 | 三处归属（§7.1）：compat 只有上游，mwrap 只有接口，适配在使用它的 C 库 | +| 29 | 边界 | rev 2 的 §5.3 以 mimalloc 说明理由，规范像是为一个分配器写的 | 只陈述虚拟内存语义；以 C 库的 POSIX 面建在它上面作为通用性的检验（§5.3） | +| 30 | 语义 | `MADV_DONTNEED` 在 POSIX 下之后可直接使用，规范的 `decommit` 之后要先 `commit` | 不带 OVERCOMMIT 时 C 库在 `decommit` 后随即 `commit`（§5.3 的表）；规范的两个操作各自语义清楚 | +| 31 | 一致性 | 唯一性若按工作区判定，会禁止一个工作区中两个程序合法地使用两个变体 | 变体是新的身份：工作区"一个身份一份检出"的已有规则不受影响；约束是任意两个库都有的"一个程序中模块名与符号唯一"（§14.1） | +| 32 | 一致性 | 打过补丁的库若沿用原包的缓存身份，两个变体会互相覆盖构建与 BMI | 变体是身份的一部分，缓存、BMI、锁按变体（§14.1） | +| 33 | 可复现 | 补丁若允许模糊匹配或依赖系统的 `patch`，不同机器结果不同 | 内置应用器、精确匹配、字节精确，失败即报错（§14.1） | +| 34 | 依赖环 | C 库以普通依赖使用 `compat.mimalloc` 时，后者为目标构建需要 C 库 | `embed`：以使用者的参数编入使用者，依赖不为目标单独构建（§14.1） | +| 35 | 稳定性 | 两个分配器 feature 在依赖边之间合并，可能两个都开 | 恰好一个，编译期报错并给出写法（§7） | +| 36 | 简洁 | `exclude` 若普遍可用，会成为不写补丁而改变依赖行为的后门 | 只在 `embed` 下允许（§14.1） | +| 37 | 兼容与升级 | openkal-musl 0.26 需要新的 mcpp；旧 mcpp 读到 `embed`/`patches` 的行为 | 清单写 `mcpp = ">=2026.11.x"`，旧版本按已有的版本门槛拒绝并提示升级；不认识的成员不会被静默忽略（mcpp 拒绝已知表中的未知成员） | +| 38 | 缓存 | 变体增加源码与构建目录，`clean --stale` 若不识别会误删或漏删 | 与 #881 同一 PR，按变体识别（§14.3） | +| 39 | 用户体验 | #875 的失败信息"cannot start 'ninja'"不指出 mcpp 已经装了 Ninja | 按序查找并在都失败时列出查过的位置与安装命令（§14.2） | +| 40 | 安全 | 依赖边上的补丁改动第三方源码，是供应链上的一个新入口 | 补丁文件属于声明它的包，受该包的校验和覆盖；锁文件记录每个补丁的 sha256；`mcpp sbom` 记录变体、各补丁的路径与 sha256（SBOM 中的 pedigree），供审计 | +| 41 | IDE | 变体与嵌入的源文件在缓存中，mcppls 若不把它们当作受管输入，会对每个文件做完整的输入检查 | 变体源码放在 mcpp 的受管存储之下（与 xpkgs 同一根），mcppls 已有的 `--managed-input-root` 覆盖它；编译数据库中嵌入源文件归属使用者 | +| 42 | 跨平台 | Windows 上补丁与源码的换行、长路径 | 字节精确，不转换换行；变体目录名用短摘要（12 个十六进制字符） | +| 43 | 兼容 | #875 的查找顺序使"外部工具链 + PATH 上另有 Ninja"的用户改用 mcpp 钉住的 Ninja | 这是一致性的修正（同一项目在每台机器上用同一个 Ninja）；CHANGELOG 写明；PATH 仍是最后一项 | +| 44 | 兼容 | #881 附带的统一使 `clean --stale` 的默认窗口由 1 天变为 3 天 | 方向是保留更多，不会多删；CHANGELOG 写明；`--older-than` 照旧可显式给出 | + +## 16. 评审 + +### 16.1 rev 1 待评审项的决定（2026-11-02） + +1. 分配器：mimalloc；按 mwrap 规范在 mcpplibs 下新建 mimalloc-m（§7.1）。 +2. job 限额的两种执行方式与"超出即结束整个单元"：采纳（§5.2）。 +3. 释放驻留页：进入本轮；随第 1 项扩为内存区域（§5.3）。 +4. `openkal.prof` 与 `openkal.debug`：都进入本轮（§5.4）。 +5. okrun 的接纳记忆与其信任模型：采纳（§8）。 +6. 线程指针本轮不改规范：采纳（§5.5）。 +7. 引擎不做 LTO/PGO：采纳。 +8. 门禁阈值 1.15、目标 ≤ 1.0：采纳。 +9. 两项 L1 内的后备做法：采纳（§15 第 15、16 条）。 + +### 16.2 rev 2 待评审项 + +1. 内存区域五个操作的签名与语义（§5.3），尤其"`commit` 后读到零、`discard` 后内容未指明"的区分。 +2. mimalloc-m 的两种形态与嵌入接口（§7.1）：`libc` 形态以源码包编入 C 库、上游源码以补丁改为调用嵌入钩子。 +3. mallocng 整体删除、所有目标一个分配器（§15 第 22、25 条）。 +4. mimalloc-m 的版本取上游最新 3.6.1（2026-10-30 发布），而不是 2.x。 + +### 16.3 rev 2 待评审项的决定（2026-11-02） + +1. 内存区域：采纳，但必须是通用的接口与语义——rev 3 改写 §5.3。 +2. mimalloc-m 只映射上游或 `compat.mimalloc` 的接口；具体改动在 C 库中——rev 3 改写 §7.1；补丁由 mcpp 的依赖补丁机制承载（§14）。 +3. mallocng 不删除，做成 feature——§7。 +4. mimalloc 3.6.1——采纳。 + +### 16.4 rev 3 待评审项 + +1. 变体的身份与"唯一性按程序判定"的规则（§14.1 的表），尤其"已发布的库在公开依赖边上带补丁"为允许。 +2. `embed` 的语义：以使用者的参数编入、嵌入者即供给者、`exclude` 只在 `embed` 下允许。 +3. mcpp 的这一版本（§14）进入本轮并排在 openkal-musl 之前；musl 本身仍留在 openkal-musl 中（它是移植而不是补丁）。 +4. #881 附带的"新"的定义统一为 `[build] keep_days`（`clean --stale` 的默认因此由 1 天变为 3 天）。 + +### 16.5 rev 3 待评审项的决定（2026-11-02） + +1. 变体的身份与唯一性按程序判定：采纳（§14.1）。 +2. `embed` 的语义：采纳（§14.1）。 +3. mcpp 的这一版本进入本轮、排在 openkal-musl 之前；musl 留在 openkal-musl：采纳（§12、§14）。 +4. "新"的定义统一为 `[build] keep_days`：采纳（§14.3）。 diff --git a/.agents/docs/README.md b/.agents/docs/README.md index 6707f6a4d..c3a0cc659 100644 --- a/.agents/docs/README.md +++ b/.agents/docs/README.md @@ -18,7 +18,7 @@ superseded_by: 2026-09-07-....md # when status is superseded --- ``` -344 records. +345 records. ## By subject @@ -88,6 +88,10 @@ Records that declare one. Everything else is listed by date below. - [Two answers and two silences: the scanner's second grammar, and the manifest keys nothing reads](2026-09-09-two-answers-and-two-silences.md) — active +### openkal-performance + +- [生态设计 part 3：ok 映像的性能对等——引擎路径、openkal 运行时与观测](2026-11-02-ecosystem-part3-ok-image-performance-parity-design.md) — active + ### plan - [Eight reports after 2026.9.27.1: implementation plan](2026-09-27-eight-reports-implementation-plan.md) — active @@ -154,6 +158,9 @@ Records that declare one. Everything else is listed by date below. ## By date +### 2026-11 + +- [生态设计 part 3：ok 映像的性能对等——引擎路径、openkal 运行时与观测](2026-11-02-ecosystem-part3-ok-image-performance-parity-design.md) — active ### 2026-10 - [大型工作区的构建空间与复用（mcpp#874）](2026-10-31-workspace-space-and-reuse-design.md) — active diff --git a/CHANGELOG.md b/CHANGELOG.md index c5ec3b53d..2bd5a11c9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,56 @@ > Each `## []` section is that release's notes. Entries are written in English > from 2026.9.28.3 on; earlier entries remain as written. +## [2026.11.2.1] - 2026-11-02 + +A dependency's source is used without being copied: a dependency edge carries +patches, and a dependency is compiled into its consumer (design 2026-11-02, +part 3 §14). With them, mcpp#875 and mcpp#881. + +### Added + +- **`patches = [...]` on a dependency edge** (docs/05). Unified diffs, named + relative to the declaring manifest, applied in order to the verified source by + mcpp's own exact applier (no fuzz, no search, no conversion of line endings; + one leading directory of each path removed; renames, copies and binary changes + refused). A patched library is another library, a variant: its identity is + (package, version, patch set), its source a tree of its own under the mcpp + home (`cache/src//+p`), and its build, BMIs and + cache entries its own. Version and git dependencies; a `path` dependency with + patches is refused. `mcpp.lock` records the variant and each patch's path and + SHA-256 (`variant`, `patches`); `mcpp emit sbom` states them as the component's + pedigree. A variant's tree is collected by `cache gc`, `prune` and after a + build as the cache's entries are. e2e 1001. +- **`embed = true` and `exclude = [...]` on a dependency edge** (docs/05). The + dependency's sources, less `exclude`, are compiled into the consumer with the + consumer's flags and, for those files, the dependency's include directories + and compile flags; the dependency builds nothing of its own in that + configuration, and a program that reaches it without its embedder is refused + naming both. `exclude` without `embed` is refused at the manifest. e2e 1002. +- **One program, one variant of a package**: two programs of one configuration + may each use their own patch set; a program that reaches two variants of a + package, or a variant and the package without patches, is refused naming + them. e2e 1003. + +### Fixed + +- **A build with a toolchain outside every provider found no ninja** (mcpp#875): + the ninja mcpp installed into its own home is used after the one beside the + compiler's provider and before PATH, and `MCPP_NINJA` names the same one. With + none, the build is refused naming where it looked, instead of "cannot start + 'ninja'". e2e 1004. +- **`clean --stale` said every directory was current after keeping one** + (mcpp#881): directories kept for being newer than the window are counted, and + the summary says how many and within what window. The default window of + `--older-than` is `[build] keep_days` (3 days), the window of the sweep after a + build, instead of one day. The SubOS remedies name the home: + `XLINGS_HOME= xlings self doctor --fix`. e2e 1005. + +### Design + +- Design 2026-11-02 part 3 (ok image performance parity), approved; part 2 + rev 11 and 12 corrections. + ## [2026.10.31.3] - 2026-10-31 A large workspace keeps its bytes once and builds each output once (mcpp#874; @@ -2906,7 +2956,7 @@ C 库来自依赖图的链接此前没有 `--sysroot`,clang 会从宿主推导 ` ### 成员清单只有一种读法 -`mcpp publish`、`mcpp pack`、`mcpp emit xpkg`、`mcpp toolchain list`、`mcpp sbom`、`mcpp index list`/`update`、 +`mcpp publish`、`mcpp pack`、`mcpp emit xpkg`、`mcpp toolchain list`、`mcpp emit sbom`、`mcpp index list`/`update`、 `mcpp update` 的索引刷新、快路径的身份判定与测试发现,此前直接读取成员目录中的原始清单:省略 `version` 的成员被拒绝,未写 `[toolchain]` 的成员在 `toolchain list` 中显示全局默认工具链,而同一 目录下的 `mcpp build` 用的是工作空间的工具链。现在它们与 `prepare_build` 经同一个函数读取继承后的 diff --git a/docs/05-dependencies.md b/docs/05-dependencies.md index b14600239..b318d29ff 100644 --- a/docs/05-dependencies.md +++ b/docs/05-dependencies.md @@ -610,6 +610,61 @@ each of them may have its own module of one name. A module that several programs share is best provided by one package that the others depend on; it is then compiled once. +### A dependency's patches, and a dependency embedded in its consumer (mcpp 2026.11+) + +Two fields of a dependency edge use a package's source without copying it. + +```toml +[dependencies] +"compat.zlib" = { version = "1.3.2", patches = ["patches/0001-fix.patch"] } +"compat.mimalloc" = { version = "3.6.1", embed = true, exclude = ["*/src/prim/prim.c"] } +``` + +**`patches`** are unified diffs (`diff -u`, `git format-patch`), named relative +to the manifest that declares them and applied in order to the package's +verified source. + +- A patched library is another library, a **variant**: its identity is the + package, its version and the patch set, named by the SHA-256 over each + patch's SHA-256 in order (not by the files' names). Its source is a tree of + its own, `cache/src//+p` under the mcpp home, and + its build, its BMIs and its cache entries are its own. One patch set carried + by two packages is one variant. +- The applier is mcpp's and is exact: a hunk applies where its header says, + every line of context and every line it removes byte for byte, or the build + is refused naming the patch, the file, the hunk and the line. Nothing is + searched for elsewhere and no line ending is converted. Paths in a patch + carry one leading directory (`a/`, `b/`), which is removed. Renames, copies + and binary changes are refused. +- Version and git dependencies may carry patches; a `path` dependency is a tree + its consumer owns and is refused with them. +- `mcpp.lock` names the variant and each patch by its path and SHA-256; + `mcpp emit sbom` states them as the component's pedigree. +- One program holds one variant of a package. Two programs of one + configuration (members of a workspace) may each use their own; a program + that reaches two variants of one package, or a variant and the package + without patches, is refused, naming them. +- The variant's tree is collected as the cache's entries are: by `mcpp cache + gc` and `prune`, and after a build once unused for `[cache] unused_days`. + +**`embed = true`** compiles the dependency into its consumer. + +- The dependency's sources, as its description selects them and less + `exclude` (globs relative to the dependency's root), are compiled as the + consumer's, with the consumer's flags and, for those files, the + dependency's include directories and compile flags. +- The dependency builds nothing of its own in that configuration: every other + edge to it takes its interface, and the embedder supplies its definitions. + A program that reaches it without its embedder is refused, naming both. +- `exclude` is allowed only with `embed`: a library built on its own that + compiled fewer files would change for every consumer, which is a patch's to + state. + +A C library uses `embed` to carry an allocator or another dependency that would +otherwise need the C library itself to be built. A package that uses either +field states `mcpp = ">=2026.11"` in `[package]`, so that an older mcpp refuses +it rather than building it without its patches. + ## Current limitations - **Two things need the network, and only two:** resolving a branch that has no diff --git a/docs/09-commands-by-scenario.md b/docs/09-commands-by-scenario.md index 46d9c241e..4490a57ac 100644 --- a/docs/09-commands-by-scenario.md +++ b/docs/09-commands-by-scenario.md @@ -56,9 +56,12 @@ the fast paths read. Three consequences follow from that definition: - A directory no record names is not deleted merely for that. `mcpp test` builds through a path that writes no record, and so does a `--no-cache` - build. An unrecorded directory written within `--older-than` (one day by - default) is kept; older than that it goes, and the cost of being wrong is one - rebuild of a configuration nothing has touched since. + build. An unrecorded directory written within `--older-than` is kept; older + than that it goes, and the cost of being wrong is one rebuild of a + configuration nothing has touched since. The default window is `[build] + keep_days` of `config.toml` (3 days), the window the sweep after a build + uses, so the two agree on what is recent (2026.11). When it kept a + directory, the summary says how many and within what window. - With no record at all the command refuses rather than guess. One `mcpp build` establishes what is current. - Directories under `target/` that are not fingerprint directories — `dist/` diff --git a/docs/zh/05-dependencies.md b/docs/zh/05-dependencies.md index a660ecd5f..e474849b7 100644 --- a/docs/zh/05-dependencies.md +++ b/docs/zh/05-dependencies.md @@ -533,6 +533,45 @@ updater = { path = "../updater", artifacts = ["updater"] } 几个程序共用的模块，最好由一个包提供、其余的包依赖它；这样它只编译一次。 +### 依赖的补丁，以及嵌入使用者的依赖（mcpp 2026.11+） + +依赖边上的两个字段让一个包使用另一个包的源码而不拷贝它。 + +```toml +[dependencies] +"compat.zlib" = { version = "1.3.2", patches = ["patches/0001-fix.patch"] } +"compat.mimalloc" = { version = "3.6.1", embed = true, exclude = ["*/src/prim/prim.c"] } +``` + +**`patches`** 是 unified diff（`diff -u`、`git format-patch` 的输出），路径相对于声明它们的清单， +按顺序应用到该包已校验的源码上。 + +- 打过补丁的库是另一个库，即**变体**：它的身份是包、版本与补丁集；补丁集以各补丁 SHA-256 依序 + 再取的 SHA-256 命名，与文件名无关。变体有自己的源码树（mcpp home 下 + `cache/src/<包>/<版本>+p<摘要>`），构建、BMI 与缓存条目都是自己的。两个包携带同一组补丁，得到 + 同一个变体。 +- 应用器是 mcpp 自己的，而且是精确的：一个 hunk 只在其头部所写的位置应用，每一行上下文与每一行 + 删除都必须逐字节相符，否则构建被拒绝，并指出补丁、文件、hunk 与行号。不在别处查找，不转换换行。 + 补丁中的路径带一级前导目录（`a/`、`b/`），应用时去掉。改名、复制与二进制改动被拒绝。 +- 版本依赖与 git 依赖可以带补丁；`path` 依赖是使用者自己拥有的源码树，带补丁时被拒绝。 +- `mcpp.lock` 记录变体，以及每个补丁的路径与 SHA-256；`mcpp emit sbom` 把它们写作组件的 pedigree。 +- 一个程序只持有一个包的一个变体。同一配置中的两个程序（工作区的成员）可以各用自己的变体；一个 + 程序同时到达同一个包的两个变体，或一个变体与不带补丁的该包时被拒绝，消息点名两者。 +- 变体的源码树与缓存条目一样回收：`mcpp cache gc`、`prune`，以及构建之后超过 + `[cache] unused_days` 未用的。 + +**`embed = true`** 把依赖编进使用者。 + +- 依赖的源文件（按其描述选出，再减去 `exclude`，glob 相对于依赖的根目录）作为使用者的源文件编译： + 使用使用者的编译参数，并且对这些文件另加依赖的 include 目录与编译参数。 +- 在这一配置中，该依赖自己不编译任何东西：指向它的其他边只取它的接口，定义由嵌入者提供。一个程序 + 到达它而不含其嵌入者时被拒绝，消息点名两者。 +- `exclude` 只能与 `embed` 一起使用：一个单独构建的库若少编译一些文件，它对所有使用者都变了， + 这应当由补丁陈述。 + +C 库用 `embed` 携带分配器之类本身需要 C 库才能构建的依赖。使用其中任一字段的包在 `[package]` 中写 +`mcpp = ">=2026.11"`，使旧版 mcpp 拒绝它，而不是不带补丁地构建它。 + ## 当前边界 - **只有两件事需要网络，而且只有这两件：** 解析一个在 lock 里没有 commit 的 diff --git a/docs/zh/09-commands-by-scenario.md b/docs/zh/09-commands-by-scenario.md index 631936c60..caa04c019 100644 --- a/docs/zh/09-commands-by-scenario.md +++ b/docs/zh/09-commands-by-scenario.md @@ -48,9 +48,11 @@ Would remove 1 directory (0.0 B) 路径读取。这个定义带来三个推论： - 一个没有记录的目录，不会仅因此就被删除。`mcpp test` 走的构建路径不写 - 记录，`--no-cache` 构建同样不写。未被记录、但写在 `--older-than`（默认 - 一天）之内的目录会被保留；更旧的会被删，判断出错的代价是重编一个此后 - 无人碰过的配置。 + 记录，`--no-cache` 构建同样不写。未被记录、但写在 `--older-than` 之内的 + 目录会被保留；更旧的会被删，判断出错的代价是重编一个此后无人碰过的配置。 + 默认窗口是 `config.toml` 的 `[build] keep_days`（3 天），即构建后清理所用 + 的同一个窗口，两者对"新"的定义一致（2026.11）。保留了目录时，摘要给出 + 保留的个数与窗口。 - 完全没有记录时，命令拒绝执行，而不是去猜。跑一次 `mcpp build` 就能确立 什么是当前的。 - `target/` 下不是指纹目录的东西 —— 例如 `mcpp pack` 的 `dist/` —— 从不 diff --git a/mcpp.toml b/mcpp.toml index e6d9fc7cd..dcbb7963f 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "mcpp" -version = "2026.10.31.3" +version = "2026.11.2.1" description = "Modern C++ build & package management tool" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/modules/manifest/src/dep_spec.cppm b/modules/manifest/src/dep_spec.cppm index 01b531914..6e1b2cff6 100644 --- a/modules/manifest/src/dep_spec.cppm +++ b/modules/manifest/src/dep_spec.cppm @@ -113,6 +113,26 @@ struct DependencySpec { // into publicUsage unconditionally. Only build.mcpp-INJECTED include dirs // are private-only.) bool reexport = false; + // design 2026-11-02 (part 3) §14: a patched dependency is another + // library. `patches` are unified diffs applied in order to the verified + // source; the edge's dependency becomes the VARIANT (package, version, + // patch set), with a source tree, a build and a cache of its own. Each path + // is absolute here --- the reader resolves it against the directory of the + // manifest that declared it, so the patch belongs to the package that + // carries it --- and `patchesDigest` names the set: the SHA-256 over each + // patch's own SHA-256, in order (empty for no patches). Version and git + // edges only: a `path` dependency is a tree its consumer already owns. + std::vector patches; + std::string patchesDigest; + // `embed = true`: the dependency's sources (as its description selects + // them, less `exclude`) are compiled INTO this consumer with the + // consumer's flags and the dependency's include directories and macros, + // and the dependency is not built as a library of its own. The embedder is + // the dependency's supplier in every program it is in; other edges to the + // same package there take its interface only. `exclude` (globs relative to + // the dependency's root) is allowed only with `embed`. + bool embed = false; + std::vector exclude; bool defaultFeatures = true; // consumer opt-out: `default-features = false` bool defaultFeaturesDeclared = false; // A member edge (workspace design 2026-09-29 §15): the edge from a diff --git a/modules/manifest/src/glob.cppm b/modules/manifest/src/glob.cppm index 793fedc2e..02b86b9e0 100644 --- a/modules/manifest/src/glob.cppm +++ b/modules/manifest/src/glob.cppm @@ -177,6 +177,18 @@ bool path_matches_glob(const std::filesystem::path& candidate, const std::filesystem::path& root, std::string_view glob) { + // AN ABSOLUTE GLOB NAMES FILES WHEREVER THEY ARE: the flags an embedded + // dependency's own files are compiled with name that dependency's tree, + // which is not beneath the consumer's root (design 2026-11-02 §14.1). + if (glob.starts_with('/') || (glob.size() > 2 && glob[1] == ':' && (glob[2] == '/' || glob[2] == '\\'))) { + auto whole = try_narrow(candidate.lexically_normal()); + if (!whole) { note_unnarrowable_path(candidate); return false; } + std::string generic = *whole; + std::ranges::replace(generic, '\\', '/'); + std::string pattern(glob); + std::ranges::replace(pattern, '\\', '/'); + return relative_path_matches_glob(generic, pattern); + } // lexically_relative is pure path arithmetic and cannot throw; the // narrowing is the part that can, so it is the part that goes through // try_narrow. diff --git a/modules/manifest/src/toml.cppm b/modules/manifest/src/toml.cppm index bfea108a2..e9c96ae29 100644 --- a/modules/manifest/src/toml.cppm +++ b/modules/manifest/src/toml.cppm @@ -2297,7 +2297,8 @@ std::expected parse_document(const t::Document& documen || k == "workspace" || k == "visibility" || k == "backend" || k == "tools" || k == "artifacts" || k == "host-module" || k == "reexport" - || k == "linkage"; + || k == "linkage" + || k == "patches" || k == "embed" || k == "exclude"; }; // What makes a table an inline dep spec is that it names a SOURCE. This // used to be "every key is known", which quietly coupled two unrelated @@ -2491,6 +2492,48 @@ std::expected parse_document(const t::Document& documen if (auto it = sub.find("reexport"); it != sub.end() && it->second.is_bool()) { spec.reexport = it->second.as_bool(); } + // design 2026-11-02 §14.1: `patches`, `embed`, `exclude`. + if (auto it = sub.find("patches"); it != sub.end()) { + if (!it->second.is_array()) + return std::unexpected(error(origin, std::format( + "[{}.\"{}\"] patches must be an array of paths", section, fqName))); + for (auto& pv : it->second.as_array()) { + if (!pv.is_string() || pv.as_string().empty()) + return std::unexpected(error(origin, std::format( + "[{}.\"{}\"] patches must be an array of paths", section, fqName))); + std::filesystem::path pp(pv.as_string()); + if (pp.is_relative()) { + std::error_code ec; + auto dir = origin.parent_path(); + if (dir.is_relative()) dir = std::filesystem::absolute(dir, ec); + pp = dir / pp; + } + spec.patches.push_back(pp.lexically_normal().generic_string()); + } + } + if (auto it = sub.find("embed"); it != sub.end()) { + if (!it->second.is_bool()) + return std::unexpected(error(origin, std::format( + "[{}.\"{}\"] embed must be true or false", section, fqName))); + spec.embed = it->second.as_bool(); + } + if (auto it = sub.find("exclude"); it != sub.end()) { + if (!it->second.is_array()) + return std::unexpected(error(origin, std::format( + "[{}.\"{}\"] exclude must be an array of globs", section, fqName))); + for (auto& ev : it->second.as_array()) + if (ev.is_string()) spec.exclude.push_back(ev.as_string()); + } + // `exclude' changes what a dependency compiles, which for a library + // built on its own would change the library its other consumers get + // without a patch saying so; under `embed' the sources become the + // consumer's, and choosing among them is the consumer's to do. + if (!spec.exclude.empty() && !spec.embed) + return std::unexpected(error(origin, std::format( + "[{}.\"{}\"] exclude is allowed only with embed = true: a dependency " + "built as a library of its own changes for every consumer when its " + "sources change, which is a patch's to state", + section, fqName))); // `backend = ""` — sugar for requesting the dependency's // `backend-` feature (library-level backend selection knob). if (auto it = sub.find("backend"); it != sub.end() && it->second.is_string()) { diff --git a/modules/manifest/src/types.cppm b/modules/manifest/src/types.cppm index a513f185e..f68903c14 100644 --- a/modules/manifest/src/types.cppm +++ b/modules/manifest/src/types.cppm @@ -455,6 +455,11 @@ struct GlobFlags { // of the fingerprint — the active feature set is already fingerprinted via // the -DMCPP_FEATURE_* cflags. std::string featureOrigin; + // Set on the entry `embed' gives an embedded dependency's files (design + // 2026-11-02 §14.1). The entry is the engine's, not the author's, so a plan + // that compiles none of those files (the package's build program) has no + // typo to report. Not part of the fingerprint: the flags are. + bool embedded = false; }; // The additive build inputs — the ONLY things any conditional axis may diff --git a/modules/versioning/src/version.cppm b/modules/versioning/src/version.cppm index 94066c682..743076770 100644 --- a/modules/versioning/src/version.cppm +++ b/modules/versioning/src/version.cppm @@ -31,6 +31,6 @@ import std; export namespace mcpp { -inline constexpr std::string_view MCPP_VERSION = "2026.10.31.3"; +inline constexpr std::string_view MCPP_VERSION = "2026.11.2.1"; } // namespace mcpp diff --git a/src/bmi_cache/maintenance.cppm b/src/bmi_cache/maintenance.cppm index f6b42a70d..68ee5b320 100644 --- a/src/bmi_cache/maintenance.cppm +++ b/src/bmi_cache/maintenance.cppm @@ -321,9 +321,44 @@ std::vector walk_tool_entries() { } // Package and tool entries: what `prune` and `gc` collect by last use. +// THE PATCHED SOURCE TREES OF VARIANTS (design 2026-11-02 §14.1): one per +// package, version and patch set, under the home's `cache/src`, each last used +// when a build last asked for it (its `.mcpp-variant' record is touched then). +// Collected as the cache's entries are: by `gc`, `prune`, and after a build +// once unused for `[cache] unused_days`. +std::vector walk_variant_entries() { + std::vector out; + std::error_code ec; + const auto root = mcpp::home::root() / "cache" / "src"; + if (!std::filesystem::exists(root, ec)) return out; + for (auto& pkg : std::filesystem::directory_iterator(root, ec)) { + if (!pkg.is_directory()) continue; + for (auto& v : std::filesystem::directory_iterator(pkg.path(), ec)) { + if (!v.is_directory()) continue; + Entry e; + e.dir = v.path(); + e.kind = "source"; + e.key = v.path().filename().string(); + e.label = std::format("{}@{} (patched source)", pkg.path().filename().string(), e.key); + std::error_code fe; + if (std::filesystem::is_regular_file(e.dir / ".mcpp-variant", fe)) { + e.accessed = dir_mtime_seconds(e.dir / ".mcpp-variant"); + } else { + e.complete = false; + e.problem = "no .mcpp-variant record (an interrupted patching)"; + e.accessed = dir_mtime_seconds(e.dir); + } + measure(e); + out.push_back(std::move(e)); + } + } + return out; +} + std::vector walk_collectable() { auto out = walk_pkg_entries(); for (auto& e : walk_tool_entries()) out.push_back(std::move(e)); + for (auto& e : walk_variant_entries()) out.push_back(std::move(e)); return out; } diff --git a/src/build/execute.cppm b/src/build/execute.cppm index 29aef4ed9..95a454bbd 100644 --- a/src/build/execute.cppm +++ b/src/build/execute.cppm @@ -1945,11 +1945,37 @@ export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache, // sweep with the built-in table while the project classifies with a wider one // — the exact shape of the bug this converge is removing, reintroduced as a // parameter default. Callers must say where their table came from. +// A PATCH IS AN INPUT OF THE GRAPH (design 2026-11-02 §14.1): a changed patch +// is another variant, with another source tree, so the graph a recorded build +// replays is not the graph it asks for. The patches a manifest's dependency +// edges carry are compared with the build file, as the manifest itself is; a +// manifest that names no patch costs a read of its text and nothing more. +bool patches_newer_than(const std::filesystem::path& manifestPath, + std::filesystem::file_time_type ninjaTime) { + std::error_code ec; + { + std::ifstream in(manifestPath, std::ios::binary); + if (!in) return false; + const std::string text((std::istreambuf_iterator(in)), {}); + if (text.find("patches") == std::string::npos) return false; + } + auto m = mcpp::manifest::load(manifestPath, {.insideWorkspace = true}); + if (!m) return true; // prepare_build reports it + for (auto const* deps : {&m->dependencies, &m->devDependencies, &m->buildDependencies}) + for (auto const& [name, spec] : *deps) + for (auto const& patch : spec.patches) { + auto ft = std::filesystem::last_write_time(patch, ec); + if (ec || ft > ninjaTime) return true; + } + return false; +} + bool sources_newer_than(const std::filesystem::path& projectRoot, std::filesystem::file_time_type ninjaTime, const std::vector& resourceScripts, const mcpp::ExtensionTable& extTable) { std::error_code ec; + if (patches_newer_than(projectRoot / "mcpp.toml", ninjaTime)) return true; // The root build.mcpp is a build input too — its directives shape // build.ninja (flags, generated/selected sources). A changed program must // abandon the fast path and fall through to prepare_build, where the @@ -2050,6 +2076,7 @@ export bool dep_sources_newer_than(const std::vector& depSourceRo auto tomlTime = std::filesystem::last_write_time(depRoot / "mcpp.toml", ec); if (ec) { ec.clear(); return true; } if (tomlTime > ninjaTime) return true; + if (patches_newer_than(depRoot / "mcpp.toml", ninjaTime)) return true; // THE WHOLE TREE, NOT `src/`. A dependency's units outside `src/` -- // a feature's `rules/x.cppm`, a `[build] sources` glob elsewhere -- // are as much its sources as those under it, and a host module among @@ -5773,7 +5800,10 @@ export int clean_report() { // A configuration's evolution reuses its directory (§8.2), so what this finds // is a configuration nobody built for that long. export int clean_stale(bool dryRun, std::int64_t keepWithinSecs, bool quiet = false); -export void sweep_after_build() { +// `[build] keep_days` of config.toml (default 3): the one window of "recent" +// both the sweep after a build and `clean --stale` keep (mcpp#881). Zero or +// less: the sweep keeps everything; `clean --stale` reads it as the default. +export std::int64_t configured_keep_days() { namespace fs = std::filesystem; std::int64_t days = 3; std::error_code ec; @@ -5781,6 +5811,12 @@ export void sweep_after_build() { if (fs::exists(config, ec)) if (auto doc = mcpp::libs::toml::parse_file(config)) days = doc->get_int("build.keep_days").value_or(days); + return days; +} +export void sweep_after_build() { + namespace fs = std::filesystem; + const std::int64_t days = configured_keep_days(); + std::error_code ec; if (days <= 0) return; auto root = mcpp::project::find_manifest_root(fs::current_path()); if (!root) return; @@ -5851,7 +5887,11 @@ export int clean_stale(bool dryRun, std::int64_t keepWithinSecs, bool quiet) { } std::uintmax_t bytes = 0; - std::size_t removed = 0, failed = 0; + // `kept': unrecorded directories left because they are newer than the + // window. Counted apart, so that the summary is true when it kept one + // (mcpp#881): it had said every directory was recorded as current under a + // line that had just said one was not. + std::size_t removed = 0, failed = 0, kept = 0; std::set swept; // removed, or would be under --dry-run std::error_code ec; for (fs::directory_iterator tripleIt(target, ec), end; tripleIt != end; tripleIt.increment(ec)) { @@ -5879,6 +5919,7 @@ export int clean_stale(bool dryRun, std::int64_t keepWithinSecs, bool quiet) { : age < 86400 ? std::format("{}h", age / 3600) : std::format("{}d", age / 86400); line(std::format("kept {} (not in the record, but written {} ago; see --older-than)", shown, ago)); + ++kept; continue; } } @@ -5993,6 +6034,7 @@ export int clean_stale(bool dryRun, std::int64_t keepWithinSecs, bool quiet) { fs::file_time_type::clock::now() - fs::last_write_time(dir, aec)).count(); if (!aec && age < keepWithinSecs) { line(std::format("kept {} (used by no configuration, but written recently; see --older-than)", shown)); + ++kept; return; } const auto size = mcpp::bmi_cache::dir_size(dir); @@ -6035,10 +6077,19 @@ export int clean_stale(bool dryRun, std::int64_t keepWithinSecs, bool quiet) { "`.build-mcpp/out` and `.build-mcpp/host` are left as they are")); } - if (removed == 0 && failed == 0) { + const auto window = [&] { + return keepWithinSecs % 86400 == 0 ? std::format("{}d", keepWithinSecs / 86400) + : keepWithinSecs % 3600 == 0 ? std::format("{}h", keepWithinSecs / 3600) + : std::format("{}s", keepWithinSecs); + }; + if (removed == 0 && failed == 0 && kept == 0) { line(std::format("Nothing stale under {}: every fingerprint directory is recorded as current{}", target.string(), usesKnown ? ", and every generated-source and build-program directory is in use" : "")); + } else if (removed == 0 && failed == 0) { + line(std::format("Nothing removed under {}: {} director{} not in use kept, written within " + "the last {} (--older-than); every other directory is in use", + target.string(), kept, kept == 1 ? "y" : "ies", window())); } else { if (quiet) mcpp::ui::status("Swept", std::format("{} build director{} no configuration used for {} days ({}; [build] keep_days in config.toml)", diff --git a/src/build/ninja_backend.cppm b/src/build/ninja_backend.cppm index 2e81f5d04..6b1942426 100644 --- a/src/build/ninja_backend.cppm +++ b/src/build/ninja_backend.cppm @@ -91,6 +91,7 @@ void write_module_maps(const BuildPlan& plan); // toolchain when there is one, else `ninja` from PATH. One answer for the // engine's own builds and for the build information (#734 E2). std::string ninja_program_for(const mcpp::toolchain::Toolchain& tc); +std::string ninja_not_found_advice(const mcpp::toolchain::Toolchain& tc); // Helper exposed for testing / debugging std::string emit_ninja_string(const BuildPlan& plan); @@ -4754,6 +4755,9 @@ std::expected NinjaBackend::build(const BuildPlan& plan // exec'd directly via capture_exec/execvp, which take argv (not a shell // string). Shell-using call sites must quote it locally. std::string ninjaProgram = ninja_program_for(plan.toolchain); + if (ninjaProgram == "ninja" && !mcpp::invocation::which( + std::string("ninja") + std::string(mcpplibs::os::exe_suffix))) + return std::unexpected(BuildError{ninja_not_found_advice(plan.toolchain), ninja_path}); // THE BUILD FILE IN THE ENCODING NINJA READS IT IN (#693, M4). Ninja reads // UTF-8 when it declares the UTF-8 code page and the host honours it, and @@ -5183,13 +5187,40 @@ std::expected NinjaBackend::build(const BuildPlan& plan return r; } +// WHICH NINJA A BUILD RUNS (mcpp#875). The one beside the compiler's provider +// package, where the toolchain came from a provider; else the one mcpp +// installed into its own home when it was first initialised; else whichever +// PATH holds. The second was not asked before 2026.11, so a toolchain named by +// a path outside every provider (`MCPP_TOOLCHAIN=path:...`, MSYS2's GCC) fell +// through to PATH, and a machine whose PATH held no ninja failed with "cannot +// start 'ninja'" while mcpp's own sat installed. `MCPP_NINJA`, which a build +// program sees, is this function's answer too. std::string ninja_program_for(const mcpp::toolchain::Toolchain& tc) { auto ninja_name = std::string("ninja") + std::string(mcpplibs::os::exe_suffix); if (auto nb = mcpp::provider::beside_binary(tc.binaryPath, "ninja", ninja_name)) return nb->string(); + if (auto home = mcpp::provider::in_home_store("ninja", ninja_name)) { + std::error_code ec; + if (std::filesystem::is_regular_file(*home / ninja_name, ec)) return (*home / ninja_name).string(); + } return "ninja"; } +// The places `ninja_program_for` looked, for the sentence that says none of +// them held a ninja (mcpp#875). +std::string ninja_not_found_advice(const mcpp::toolchain::Toolchain& tc) { + std::string where = std::format("\n beside the compiler's provider package ({})", + tc.binaryPath.parent_path().generic_string()); + if (auto store = mcpp::provider::home_store()) + where += std::format("\n in mcpp's home ({})", store->generic_string()); + where += "\n on PATH"; + return std::format( + "no ninja was found; it was looked for{}\n" + " mcpp installs one into its home whenever it runs online, so a run " + "without --offline installs it; or put a ninja on PATH", + where); +} + std::unique_ptr make_ninja_backend() { return std::make_unique(); } diff --git a/src/build/plan.cppm b/src/build/plan.cppm index 66a8fb479..fd8c3930a 100644 --- a/src/build/plan.cppm +++ b/src/build/plan.cppm @@ -6,6 +6,7 @@ export module mcpp.build.plan; import std; +import mcpp.pm.patch; // a variant's name (design 2026-11-02 §14.1) import mcpp.build.graph_shape; import mcpp.targetside; import mcpp.build.loader_contract; @@ -968,10 +969,21 @@ std::vector dependency_name_candidates( out.push_back(std::move(value)); }; - push(depName); - if (!spec.shortName.empty()) push(spec.shortName); + // A VARIANT IS NAMED AS ANOTHER PACKAGE (design 2026-11-02 §14.1): an edge + // that carries patches reaches `+p', and only that --- the + // package without patches is another package, which this edge does not + // reach. + std::string variant; + if (!spec.patches.empty()) { + auto digest = spec.patchesDigest.empty() ? mcpp::pm::patch::set_digest(spec.patches) + : std::expected(spec.patchesDigest); + if (digest && !digest->empty()) variant = "+p" + digest->substr(0, 12); + else return out; // a patch that cannot be read reaches nothing; resolution said why + } + push(depName + variant); + if (!spec.shortName.empty()) push(spec.shortName + variant); if (!spec.namespace_.empty() && !spec.shortName.empty()) { - push(spec.namespace_ + "." + spec.shortName); + push(spec.namespace_ + "." + spec.shortName + variant); } return out; } diff --git a/src/build/prepare/graph.cpp b/src/build/prepare/graph.cpp index 770e90c78..327e438c0 100644 --- a/src/build/prepare/graph.cpp +++ b/src/build/prepare/graph.cpp @@ -46,6 +46,7 @@ import mcpp.build.backend; // BuildOptions for the tool sub-build import mcpp.build.ninja; // make_ninja_backend — driving that sub-build import mcpp.config; import mcpp.toolchain.post_install; +import mcpp.pm.patch; import mcpp.pm.resolver; import mcpp.pm.index_spec; import mcpp.pm.index_contract; @@ -70,19 +71,6 @@ namespace mcpp::build { // Internal linkage: these names belong to this file, not to // mcpp.build.prepare's surface. -static mcpp::modgraph::DependencyVisibility parseVisibility(std::string_view visibility) { - if (visibility == "private") - return mcpp::modgraph::DependencyVisibility::Private; - if (visibility == "interface") - return mcpp::modgraph::DependencyVisibility::Interface; - return mcpp::modgraph::DependencyVisibility::Public; -} - -static std::size_t packageIndexForConsumer(std::size_t consumerDepIndex) { - if (consumerDepIndex == kMainConsumer) return std::size_t{0}; - return consumerDepIndex + 1; -} - static std::vector expandIncludeDirs( PrepareState& state, const std::filesystem::path& packageRoot, @@ -241,108 +229,6 @@ static std::expected makePackageRoot( return pkg; } -static void recordDependencyEdge( - PrepareState& state, - std::size_t consumerDepIndex, - std::size_t dependencyPackageIndex, - const mcpp::manifest::DependencySpec& spec, - bool buildTable, - const std::string& writtenKey) -{ - const auto consumerPackageIndex = packageIndexForConsumer(consumerDepIndex); - if (consumerPackageIndex >= state.packages.size() - || dependencyPackageIndex >= state.packages.size()) { - return; - } - if (std::ranges::none_of(state.graphRequests, [&](const GraphRequest& r) { - return r.consumerPackageIndex == consumerPackageIndex - && r.dependencyPackageIndex == dependencyPackageIndex - && r.key == writtenKey && r.table == spec.declaredIn; - })) - state.graphRequests.push_back(GraphRequest{ - .consumerPackageIndex = consumerPackageIndex, - .dependencyPackageIndex = dependencyPackageIndex, - .key = writtenKey, - .table = spec.declaredIn, - }); - - // A REQUESTED FEATURE THAT IS A BUILD RULE IMPLIES THE LIBRARY. - // - // A feature declaring `rule_module` is usable only by a build program, - // so requesting it asks for the dependency's library there. Requiring - // `host-module` beside it was a second spelling of one fact. - bool ruleFeature = false; - { - auto const& depManifest = state.packages[dependencyPackageIndex].manifest; - for (auto const& f : spec.features) - if (depManifest.featureRuleModule.contains(f)) { ruleFeature = true; break; } - } - // COMPAT(host-module-in-dependencies): a `[dependencies]` entry asking - // for `host-module = true`, or for a rule feature, served the build - // program and not the target before the section decided the - // configuration (mcpp#797). It keeps that meaning, reported where the - // manifest can be edited. - // Not in the build programs' graph: every package there is a build-program - // instance already, and a library's rule features are what its edge asks. - if (!buildTable && !state.overrides.build_program_graph - && ((spec.hostModuleDeclared && spec.hostModule) || ruleFeature)) { - buildTable = true; - state.reportCompatUse("host-module-in-dependencies", consumerPackageIndex, - std::format("[dependencies] {}", writtenKey)); - } - const bool wantsLibrary = buildTable - && (ruleFeature - || (spec.hostModuleDeclared ? spec.hostModule - : spec.tools.empty() && spec.artifacts.empty())); - - // One edge per (consumer, dependency, visibility, configuration). The - // same package written in [dependencies] and [build-dependencies] of - // one consumer is two edges and two instances (mcpp#797, R2): merging - // them made the build-program request read as "not in the target". - const auto visibility = parseVisibility(spec.visibility); - auto same = [&](const DependencyEdge& edge) { - return edge.consumerPackageIndex == consumerPackageIndex - && edge.dependencyPackageIndex == dependencyPackageIndex - && edge.visibility == visibility - && edge.buildTable == buildTable; - }; - auto it = std::find_if(state.dependencyEdges.begin(), state.dependencyEdges.end(), same); - if (it != state.dependencyEdges.end()) { - // One consumer restating one dependency in one table (an - // unconditional entry and a `[target..*]` or `[feature-deps]` - // row): every declaration's requests are requested of the one edge - // (#649 E7). Additive fields union, `default-features` stays on - // unless every declaration opts out. - for (auto const& t : spec.tools) - if (std::ranges::find(it->requestedTools, t) == it->requestedTools.end()) - it->requestedTools.push_back(t); - for (auto const& a : spec.artifacts) - if (std::ranges::find(it->requestedArtifacts, a) - == it->requestedArtifacts.end()) - it->requestedArtifacts.push_back(a); - for (auto const& f : spec.features) - if (std::ranges::find(it->requestedFeatures, f) - == it->requestedFeatures.end()) - it->requestedFeatures.push_back(f); - it->defaultFeatures = it->defaultFeatures || spec.defaultFeatures; - it->wantsLibrary = it->wantsLibrary || wantsLibrary; - it->reexport = it->reexport || spec.reexport; - return; - } - state.dependencyEdges.push_back(DependencyEdge{ - .consumerPackageIndex = consumerPackageIndex, - .dependencyPackageIndex = dependencyPackageIndex, - .visibility = visibility, - .requestedFeatures = spec.features, - .defaultFeatures = spec.defaultFeatures, - .requestedTools = spec.tools, - .requestedArtifacts = spec.artifacts, - .buildTable = buildTable, - .wantsLibrary = wantsLibrary, - .reexport = spec.reexport, - }); -} - static std::vector propagateLinkFlags( PrepareState& state, const std::filesystem::path& depRoot, @@ -867,6 +753,22 @@ step4b_resolve_identity(PrepareState& state, WorklistItemCtx& ctx) { spec.namespace_, spec.shortName.empty() ? name : spec.shortName, }; + // A PATCHED LIBRARY IS ANOTHER LIBRARY (design 2026-11-02 §14.1): the + // patch set is part of the identity, so its source, build and cache are + // its own, and an edge without patches is the unpatched package. + if (!spec.patches.empty()) { + if (spec.isPath()) + return std::unexpected(std::format( + "dependency '{}' (requested by {}) is a path and carries patches: a " + "path dependency is a tree its consumer already owns, so a change to " + "it is made in the tree", name, item.requestedBy)); + auto digest = mcpp::pm::patch::set_digest(spec.patches); + if (!digest) + return std::unexpected(std::format("dependency '{}' (requested by {}): {}", + name, item.requestedBy, digest.error())); + spec.patchesDigest = *digest; + key.variant = *digest; + } auto& sourceKind = ctx.sourceKind; sourceKind = spec.isPath() ? "path" @@ -1667,6 +1569,19 @@ step4b_acquire_dependency_source(PrepareState& state, WorklistItemCtx& ctx) { gitMemberCloneRoot = gitRoot; dep_root = gitRoot / *member; } + // A git dependency's variant: the checkout copied and patched under + // the mcpp home, and read from there (design 2026-11-02 §14.1). + if (!spec.patches.empty()) { + auto cfg = state.get_cfg(true); + if (!cfg) return std::unexpected(cfg.error()); + const auto variant = (*cfg)->metaCacheDir / "src" + / (key.ns.empty() ? key.shortName : key.ns + "." + key.shortName) + / mcpp::pm::patch::variant_dir_name(resolvedGitRev.substr(0, 12), spec.patchesDigest); + if (auto r = mcpp::pm::patch::materialize(dep_root, spec.patches, + spec.patchesDigest, variant); !r) + return std::unexpected(std::format("dependency '{}': {}", name, r.error())); + dep_root = variant; + } } // (version-source: dep_root + manifest are loaded together via // loadVersionDep below since the index entry drives both.) @@ -1763,7 +1678,10 @@ step4b_acquire_dependency_source(PrepareState& state, WorklistItemCtx& ctx) { // units here, as it does for a version dependency. if (state.abiThreadsRendered) state.add_once(dep_manifest->buildConfig.cflags, "-pthread"); } else { - auto loaded = state.loadVersionDep(name, key.ns, key.shortName, spec.version); + auto loaded = spec.patches.empty() + ? state.loadVersionDep(name, key.ns, key.shortName, spec.version) + : state.loadVersionDepVariant(name, key.ns, key.shortName, spec.version, + spec.patches, spec.patchesDigest); if (!loaded) return std::unexpected(loaded.error()); dep_root = std::move(loaded->first); dep_manifest = std::move(loaded->second); @@ -1908,6 +1826,14 @@ step4b_finalize_dependency(PrepareState& state, WorklistItemCtx& ctx) { ? std::vector{} : propagateLinkFlags(state, ctx.dep_root, *ctx.dep_manifest); + // A VARIANT IS NAMED AS ANOTHER PACKAGE from here on (design + // 2026-11-02 §14.1): every reader that keys a package by its name --- + // its build directory, its closure, its cache entry --- keeps two + // variants of one package apart, and the names its sources declare + // (its modules) are untouched. Two variants of one package meeting in + // one program are refused where closures are known (scan). + if (!key.variant.empty()) + ctx.dep_manifest->package.name += "+p" + key.variant.substr(0, 12); // Move the manifest into stable storage so we can later look it up // by depIndex (the SemVer merger needs to overwrite the slot). state.dep_manifests.push_back( @@ -1923,6 +1849,11 @@ step4b_finalize_dependency(PrepareState& state, WorklistItemCtx& ctx) { : sourceKind == "path" ? ctx.dep_root.string() : std::string{}, }); + // A variant's cache entry is its own (design 2026-11-02 §14.1). + if (!key.variant.empty()) { + state.dep_cache_identities.back().version += "+p" + key.variant.substr(0, 12); + state.dep_cache_identities.back().sourceRef = ctx.dep_root.generic_string(); + } // The engine interface (mcpp#797) is addressed in the dependency cache // by the engine's name and the version of its text, in the global // cache, because its directory is a function of that text. @@ -1973,6 +1904,7 @@ step4b_finalize_dependency(PrepareState& state, WorklistItemCtx& ctx) { .devOnly = item.devOnly, .depIndex = state.dep_manifests.size() - 1, .linkFlagsAdded = std::move(linkFlagsAdded), + .patches = spec.patches, }; // Recurse: the dep's own [dependencies] become new worklist items. diff --git a/src/build/prepare/graph_edges.cpp b/src/build/prepare/graph_edges.cpp new file mode 100644 index 000000000..e3178e2c0 --- /dev/null +++ b/src/build/prepare/graph_edges.cpp @@ -0,0 +1,136 @@ +// graph_edges.cpp -- P4b: the edges the dependency worklist records, one per +// consumer, dependency, visibility and table, with what each edge asks of its +// dependency. Split from graph.cpp at the boundary of what an edge is, when the +// edge learned to carry `embed' and `exclude' (design 2026-11-02 §14.1). + +module mcpp.build.prepare; +import :state; + +import std; +import mcpp.manifest; +import mcpp.modgraph.graph; +import mcpp.pm.dep_spec; + +namespace mcpp::build { + +static mcpp::modgraph::DependencyVisibility parseVisibility(std::string_view visibility) { + if (visibility == "private") + return mcpp::modgraph::DependencyVisibility::Private; + if (visibility == "interface") + return mcpp::modgraph::DependencyVisibility::Interface; + return mcpp::modgraph::DependencyVisibility::Public; +} + +static std::size_t packageIndexForConsumer(std::size_t consumerDepIndex) { + if (consumerDepIndex == kMainConsumer) return std::size_t{0}; + return consumerDepIndex + 1; +} + +void recordDependencyEdge( + PrepareState& state, + std::size_t consumerDepIndex, + std::size_t dependencyPackageIndex, + const mcpp::manifest::DependencySpec& spec, + bool buildTable, + const std::string& writtenKey) +{ + const auto consumerPackageIndex = packageIndexForConsumer(consumerDepIndex); + if (consumerPackageIndex >= state.packages.size() + || dependencyPackageIndex >= state.packages.size()) { + return; + } + if (std::ranges::none_of(state.graphRequests, [&](const GraphRequest& r) { + return r.consumerPackageIndex == consumerPackageIndex + && r.dependencyPackageIndex == dependencyPackageIndex + && r.key == writtenKey && r.table == spec.declaredIn; + })) + state.graphRequests.push_back(GraphRequest{ + .consumerPackageIndex = consumerPackageIndex, + .dependencyPackageIndex = dependencyPackageIndex, + .key = writtenKey, + .table = spec.declaredIn, + }); + + // A REQUESTED FEATURE THAT IS A BUILD RULE IMPLIES THE LIBRARY. + // + // A feature declaring `rule_module` is usable only by a build program, + // so requesting it asks for the dependency's library there. Requiring + // `host-module` beside it was a second spelling of one fact. + bool ruleFeature = false; + { + auto const& depManifest = state.packages[dependencyPackageIndex].manifest; + for (auto const& f : spec.features) + if (depManifest.featureRuleModule.contains(f)) { ruleFeature = true; break; } + } + // COMPAT(host-module-in-dependencies): a `[dependencies]` entry asking + // for `host-module = true`, or for a rule feature, served the build + // program and not the target before the section decided the + // configuration (mcpp#797). It keeps that meaning, reported where the + // manifest can be edited. + // Not in the build programs' graph: every package there is a build-program + // instance already, and a library's rule features are what its edge asks. + if (!buildTable && !state.overrides.build_program_graph + && ((spec.hostModuleDeclared && spec.hostModule) || ruleFeature)) { + buildTable = true; + state.reportCompatUse("host-module-in-dependencies", consumerPackageIndex, + std::format("[dependencies] {}", writtenKey)); + } + const bool wantsLibrary = buildTable + && (ruleFeature + || (spec.hostModuleDeclared ? spec.hostModule + : spec.tools.empty() && spec.artifacts.empty())); + + // One edge per (consumer, dependency, visibility, configuration). The + // same package written in [dependencies] and [build-dependencies] of + // one consumer is two edges and two instances (mcpp#797, R2): merging + // them made the build-program request read as "not in the target". + const auto visibility = parseVisibility(spec.visibility); + auto same = [&](const DependencyEdge& edge) { + return edge.consumerPackageIndex == consumerPackageIndex + && edge.dependencyPackageIndex == dependencyPackageIndex + && edge.visibility == visibility + && edge.buildTable == buildTable; + }; + auto it = std::find_if(state.dependencyEdges.begin(), state.dependencyEdges.end(), same); + if (it != state.dependencyEdges.end()) { + // One consumer restating one dependency in one table (an + // unconditional entry and a `[target..*]` or `[feature-deps]` + // row): every declaration's requests are requested of the one edge + // (#649 E7). Additive fields union, `default-features` stays on + // unless every declaration opts out. + for (auto const& t : spec.tools) + if (std::ranges::find(it->requestedTools, t) == it->requestedTools.end()) + it->requestedTools.push_back(t); + for (auto const& a : spec.artifacts) + if (std::ranges::find(it->requestedArtifacts, a) + == it->requestedArtifacts.end()) + it->requestedArtifacts.push_back(a); + for (auto const& f : spec.features) + if (std::ranges::find(it->requestedFeatures, f) + == it->requestedFeatures.end()) + it->requestedFeatures.push_back(f); + it->defaultFeatures = it->defaultFeatures || spec.defaultFeatures; + it->wantsLibrary = it->wantsLibrary || wantsLibrary; + it->reexport = it->reexport || spec.reexport; + it->embed = it->embed || spec.embed; + for (auto const& x : spec.exclude) + if (std::ranges::find(it->exclude, x) == it->exclude.end()) it->exclude.push_back(x); + return; + } + state.dependencyEdges.push_back(DependencyEdge{ + .consumerPackageIndex = consumerPackageIndex, + .dependencyPackageIndex = dependencyPackageIndex, + .visibility = visibility, + .requestedFeatures = spec.features, + .defaultFeatures = spec.defaultFeatures, + .requestedTools = spec.tools, + .requestedArtifacts = spec.artifacts, + .buildTable = buildTable, + .wantsLibrary = wantsLibrary, + .reexport = spec.reexport, + .embed = spec.embed, + .exclude = spec.exclude, + }); +} + +} // namespace mcpp::build diff --git a/src/build/prepare/graph_load.cpp b/src/build/prepare/graph_load.cpp index 13818af10..e17d76558 100644 --- a/src/build/prepare/graph_load.cpp +++ b/src/build/prepare/graph_load.cpp @@ -44,6 +44,7 @@ import mcpp.pm.compat; import mcpp.pm.dep_spec; import mcpp.pm.dependency_selector; import mcpp.pm.lock_io; +import mcpp.pm.patch; import mcpp.ui; import mcpp.log; import mcpp.project; @@ -213,7 +214,8 @@ static void step4a_define_split_and_identity_closures(PrepareState& state) { return std::nullopt; }; state.qualifiedKey = [](const ResolvedKey& k) { - return k.ns.empty() ? k.shortName : std::format("{}.{}", k.ns, k.shortName); + const auto base = k.ns.empty() ? k.shortName : std::format("{}.{}", k.ns, k.shortName); + return k.variant.empty() ? base : std::format("{}+p{}", base, k.variant.substr(0, 12)); }; // A root edge that adopted an identity states it on the root's own // declaration too, which is what every later reader of the root manifest @@ -1371,6 +1373,41 @@ std::expected phase4a_graph_load(PrepareState& state) { return std::unexpected(r.error()); return step4a_load_version_dep_read_manifest(state, ctx); }; + state.loadVersionDepVariant = [&](const std::string& depName, + const std::string& ns, + const std::string& shortName, + const std::string& version, + const std::vector& patches, + const std::string& digest) + -> std::expected + { + auto cfg = state.get_cfg(true); + if (!cfg) return std::unexpected(cfg.error()); + + LoadVersionDepCtx ctx; + ctx.depName = depName; + ctx.ns = ns; + ctx.shortName = shortName; + ctx.version = version; + + if (auto r = step4a_load_version_dep_locate(state, ctx); !r) + return std::unexpected(r.error()); + if (auto r = step4a_load_version_dep_fetch(state, ctx); !r) + return std::unexpected(r.error()); + // THE VARIANT'S OWN TREE (design 2026-11-02 §14.1): the verified + // source copied under the mcpp home and patched, before anything is + // read from it, so that a patch to the package's own description takes + // effect as a patch to its sources does. + std::string exact = version; + while (!exact.empty() && (exact.front() == '=' || exact.front() == ' ')) exact.erase(0, 1); + const auto qualified = ns.empty() ? shortName : ns + "." + shortName; + const auto variant = (*cfg)->metaCacheDir / "src" / qualified + / mcpp::pm::patch::variant_dir_name(exact, digest); + if (auto r = mcpp::pm::patch::materialize(*ctx.installed, patches, digest, variant); !r) + return std::unexpected(std::format("dependency '{}': {}", depName, r.error())); + ctx.installed = variant; + return step4a_load_version_dep_read_manifest(state, ctx); + }; return {}; } diff --git a/src/build/prepare/records.cpp b/src/build/prepare/records.cpp index ee3a2eb97..aea64a660 100644 --- a/src/build/prepare/records.cpp +++ b/src/build/prepare/records.cpp @@ -58,6 +58,7 @@ import mcpp.toolchain.post_install; import mcpp.build.runner_lookup; import mcpp.fetcher; import mcpp.fetcher.progress; +import mcpp.pm.patch; import mcpp.pm.resolver; import mcpp.pm.index_spec; import mcpp.pm.index_contract; @@ -276,6 +277,21 @@ std::expected step13_lockfile(PrepareState& state, BuildConte lp.name = lock_name_for(key); lp.namespace_ = key.ns; lp.version = rec.version; + // A variant is its own entry, and says which patches made it + // (design 2026-11-02 §14.1): each by its path, relative to the + // project where it is inside it, and its SHA-256. + if (!key.variant.empty()) { + lp.name += "+p" + key.variant.substr(0, 12); + lp.variant = key.variant; + for (auto const& p : rec.patches) { + std::filesystem::path path(p); + auto rel = path.lexically_relative(*state.root); + const auto shown = (!rel.empty() && !rel.generic_string().starts_with("..")) + ? rel.generic_string() : path.generic_string(); + lp.patches.push_back(std::format("{}#sha256:{}", shown, + mcpp::pm::patch::file_sha256(path).value_or("unreadable"))); + } + } // Use the namespace and resolved version as the source identifier. // For custom indices, include the index name for traceability. auto sourceIndex = lp.namespace_.empty() diff --git a/src/build/prepare/scan.cpp b/src/build/prepare/scan.cpp index c52e2d8db..a6bb59a64 100644 --- a/src/build/prepare/scan.cpp +++ b/src/build/prepare/scan.cpp @@ -66,6 +66,61 @@ static std::expected step11_scan_sources(PrepareState& state) // signal the e2e test asserts on (tests/e2e/114_run_scan_scope.sh). mcpp::log::verbose("scan", "scanning module sources"); + // EMBEDDED DEPENDENCIES (design 2026-11-02 §14.1). A dependency an edge + // embeds is compiled into its consumer: its sources, as its description + // selects them and less the edge's `exclude', become literal sources of the + // consumer, compiled with the consumer's flags and, for those files alone, + // the dependency's include directories and compile flags (its macros are + // folded into them). The dependency itself then compiles nothing: every + // other edge to it, in any program, takes its interface only, and the + // embedder supplies its definitions there. The consumers' closures (below) + // say whether a program that reaches the dependency holds its supplier. + std::map> embeddersOf; + for (auto const& e : state.dependencyEdges) { + if (!e.embed || e.buildTable) continue; + auto& who = embeddersOf[e.dependencyPackageIndex]; + if (std::ranges::find(who, e.consumerPackageIndex) == who.end()) + who.push_back(e.consumerPackageIndex); + } + for (auto const& [d, consumers] : embeddersOf) { + auto& dep = state.packages[d]; + const auto files = mcpp::modgraph::package_source_files(dep.root, dep.manifest); + if (files.empty()) + return std::unexpected(std::format( + "'{}' is embedded and its description selects no source under '{}'", + dep.manifest.package.name, dep.root.generic_string())); + for (auto c : consumers) { + std::set excluded; + for (auto const& e : state.dependencyEdges) { + if (!e.embed || e.consumerPackageIndex != c || e.dependencyPackageIndex != d) continue; + for (auto const& x : e.exclude) { + auto hit = mcpp::modgraph::expand_glob(dep.root, x); + if (hit.empty()) + mcpp::diag::warning("embed", std::format( + "'{}' embeds '{}' excluding '{}', which names no file of it", + state.packages[c].manifest.package.name, dep.manifest.package.name, x)); + for (auto const& h : hit) excluded.insert(h); + } + } + auto& consumer = state.packages[c].manifest; + for (auto const& f : files) + if (!excluded.contains(f)) consumer.modules.sources.push_back(f.generic_string()); + mcpp::manifest::GlobFlags gf; + gf.embedded = true; + gf.glob = (dep.root.lexically_normal().generic_string()) + "/**"; + for (auto const& inc : dep.privateBuild.includeDirs) { + gf.cflags.push_back("-I" + inc.generic_string()); + gf.cxxflags.push_back("-I" + inc.generic_string()); + } + for (auto const& f : dep.manifest.buildConfig.cflags) gf.cflags.push_back(f); + for (auto const& f : dep.manifest.buildConfig.cxxflags) gf.cxxflags.push_back(f); + consumer.buildConfig.globFlags.push_back(std::move(gf)); + } + dep.manifest.modules.sources.clear(); + dep.manifest.modules.declaredUnits.clear(); + dep.manifest.buildConfig.sourcesDeclared = true; + } + // Modgraph: regex scanner by default; opt-in to compiler-driven P1689 // scanner via env var MCPP_SCANNER=p1689 (see docs/27). // @@ -112,6 +167,59 @@ static std::expected step11_scan_sources(PrepareState& state) } closures[qualified(state.packages[i].manifest)] = std::move(names); } + // ONE PROGRAM, ONE VARIANT OF A PACKAGE (design 2026-11-02 §14.1). A + // variant is named `+p` from resolution on, so two + // variants of one package --- or a variant and the package itself --- + // are two packages that export the same module names and define the + // same symbols. Two programs of one configuration may each hold one; + // one closure may not hold two. + auto base_of = [](const std::string& q) { + const auto at = q.rfind("+p"); + return at == std::string::npos ? q : q.substr(0, at); + }; + // A PROGRAM THAT REACHES AN EMBEDDED PACKAGE HOLDS ITS SUPPLIER. The + // embedded package compiles nothing of its own, so a program whose + // closure has it and none of its embedders has its declarations and + // no definitions; refused here, naming both, rather than at the link. + for (auto const& [d, consumers] : embeddersOf) { + const auto embedded = qualified(state.packages[d].manifest); + for (std::size_t i = 0; i < state.packages.size(); ++i) { + if (!state.compilesHere(i) || (i == 0 && state.m->package.virtualRoot)) continue; + const bool program = i == 0 || std::ranges::any_of(state.packages[i].manifest.targets, + [](auto const& t) { return t.kind == mcpp::manifest::Target::Binary + || t.kind == mcpp::manifest::Target::Application; }); + if (!program) continue; + auto it = closures.find(qualified(state.packages[i].manifest)); + if (it == closures.end() || !it->second.contains(embedded)) continue; + const bool supplied = std::ranges::any_of(consumers, [&](std::size_t c) { + return it->second.contains(qualified(state.packages[c].manifest)); }); + if (supplied) continue; + return std::unexpected(std::format( + "'{}' reaches '{}', which '{}' embeds, and not '{}': the embedded package " + "compiles nothing of its own, so this program would have its declarations " + "and no definitions. Depend on the embedder, or on '{}' without `embed`.", + qualified(state.packages[i].manifest), embedded, + qualified(state.packages[consumers.front()].manifest), + qualified(state.packages[consumers.front()].manifest), embedded)); + } + } + for (auto const& [owner, names] : closures) { + std::map> byBase; + for (auto const& n : names) byBase[base_of(n)].push_back(n); + for (auto const& [base, variants] : byBase) { + if (variants.size() < 2) continue; + std::string listed; + for (auto const& v : variants) + listed += std::format("\n {}{}", v, + v == base ? " (the package without patches)" : ""); + return std::unexpected(std::format( + "'{}' reaches two variants of '{}':{}\n" + " A patched library is another library that provides the same modules and " + "symbols, so one program holds one of them. Give every edge to '{}' in this " + "program the same `patches`, or none.", + owner, base, listed, base)); + } + } } state.scan = [&] { diff --git a/src/build/prepare/state.cppm b/src/build/prepare/state.cppm index c98c7dbfc..12627dd76 100644 --- a/src/build/prepare/state.cppm +++ b/src/build/prepare/state.cppm @@ -152,6 +152,10 @@ constexpr std::size_t kMainConsumer = static_cast(-1); bool buildTable = false; bool wantsLibrary = false; bool reexport = false; + // `embed = true` and its `exclude` (design 2026-11-02 §14.1): the + // dependency's sources are compiled into the consumer. + bool embed = false; + std::vector exclude; }; struct GraphRequest { @@ -172,6 +176,10 @@ constexpr std::size_t kMainConsumer = static_cast(-1); struct ResolvedKey { std::string ns; std::string shortName; + // The patch set's digest, for a variant (design 2026-11-02 §14.1): + // a patched library is another library, so two patch sets of one + // package are two identities, resolved, built and cached apart. + std::string variant; auto operator<=>(const ResolvedKey&) const = default; }; @@ -212,6 +220,9 @@ constexpr std::size_t kMainConsumer = static_cast(-1); bool devOnly = false; std::size_t depIndex = 0; // index into dep_manifests/packages-1 (for in-place re-fetch) std::vector linkFlagsAdded; // entries appended to m->buildConfig.ldflags by this dep + // A variant's patches, as its edge stated them (absolute paths), for + // the lock (design 2026-11-02 §14.1). + std::vector patches; }; struct WorkItem { @@ -790,6 +801,14 @@ struct PrepareState { std::string>(const std::string&, const std::string&, const std::string&, const std::string&)> loadVersionDep; + // The same for an edge that carries patches: the verified source is + // copied to the variant's own tree and patched before the package's + // description is read from it (design 2026-11-02 §14.1). + std::function, + std::string>(const std::string&, const std::string&, + const std::string&, const std::string&, + const std::vector&, const std::string&)> + loadVersionDepVariant; std::map resolved; std::map>> @@ -920,6 +939,12 @@ std::expected phase1_toolchain_spec_and_axes(PrepareState& st std::expected phase2_define_toolchain_resolver(PrepareState& state); std::expected phase3_xlings_before_graph(PrepareState& state); std::expected phase4a_graph_load(PrepareState& state); +// graph_edges.cpp: the edge one request adds to the graph, or the one it +// merges with. +void recordDependencyEdge(PrepareState& state, std::size_t consumerDepIndex, + std::size_t dependencyPackageIndex, + const mcpp::manifest::DependencySpec& spec, bool buildTable, + const std::string& writtenKey); std::expected phase4b_graph_worklist(PrepareState& state); // The lookups the later phases ask of the graph (graph_closures.cpp). void step4b_define_lookup_closures(PrepareState& state); diff --git a/src/cli.cppm b/src/cli.cppm index 79fd110a4..8e30f664e 100644 --- a/src/cli.cppm +++ b/src/cli.cppm @@ -885,7 +885,7 @@ int run(int argc, char** argv) { .option(cl::Option("dry-run").help("List what would be removed and delete nothing (implies --stale)")) .option(cl::Option("report").help("Show where target/ holds its bytes, per configuration and area; removes nothing")) .option(cl::Option("older-than").takes_value().value_name("DURATION") - .help("Keep unrecorded directories written more recently than this, e.g. 12h, 3d (default 1d; 0 keeps none; implies --stale)")) + .help("Keep unrecorded directories written more recently than this, e.g. 12h, 3d (default [build] keep_days, 3d; 0 keeps none; implies --stale)")) .action(wrap_rc(cmd_clean))) .subcommand(cl::App("why") .description("Explain how the toolchain / runtime / deps / runners / sources were resolved") diff --git a/src/cli/cmd_build.cppm b/src/cli/cmd_build.cppm index 1fbeaabc9..f858966b5 100644 --- a/src/cli/cmd_build.cppm +++ b/src/cli/cmd_build.cppm @@ -1561,7 +1561,14 @@ export int cmd_clean(const mcpplibs::cmdline::ParsedArgs& parsed) { "use `mcpp cache gc`)"); return 2; } - std::int64_t keepWithinSecs = 24 * 3600; + // THE WINDOW IS `[build] keep_days` UNLESS IT IS GIVEN (mcpp#881): the + // sweep after a build keeps what is younger than that, and `clean + // --stale` kept what was younger than one day --- two answers to one + // question. One answer now, the configured one (default 3 days). + // `keep_days = 0` turns the sweep off; a `clean --stale` was asked for + // nonetheless, and keeps what is younger than the default. + const std::int64_t keepDays = mcpp::build::configured_keep_days(); + std::int64_t keepWithinSecs = (keepDays > 0 ? keepDays : 3) * 86400; if (olderThan) { // `parse_duration` wants a unit, so bare `0` reaches it as a // one-character string and is rejected; it also returns whatever diff --git a/src/cli/cmd_sbom.cppm b/src/cli/cmd_sbom.cppm index 531f8bb9f..ad37d59df 100644 --- a/src/cli/cmd_sbom.cppm +++ b/src/cli/cmd_sbom.cppm @@ -161,12 +161,38 @@ int cmd_sbom(const mcpplibs::cmdline::ParsedArgs& parsed) { // as a property rather than as a `hashes` entry — claiming a weak // digest is a cryptographic hash is the kind of statement an SBOM is // read to trust. + std::vector> properties; if (d.hash.starts_with("sha256:")) { out += " \"hashes\": [ { \"alg\": \"SHA-256\", \"content\": \"" + esc(d.hash.substr(7)) + "\" } ],\n"; } else if (!d.hash.empty()) { - out += " \"properties\": [ { \"name\": \"mcpp:resolution-digest\"," - " \"value\": \"" + esc(d.hash) + "\" } ],\n"; + properties.emplace_back("mcpp:resolution-digest", d.hash); + } + // A VARIANT STATES ITS PEDIGREE (design 2026-11-02 §14.1): the + // component is the package changed by these patches, in this order, + // and each patch is named with its SHA-256, so that what was changed + // in a third party's source is part of what the bill of materials + // says rather than something a reader has to find. + if (!d.variant.empty()) { + out += " \"pedigree\": { \"patches\": ["; + bool firstPatch = true; + for (auto const& patch : d.patches) { + const auto at = patch.rfind("#sha256:"); + const auto path = at == std::string::npos ? patch : patch.substr(0, at); + out += std::format("{} {{ \"type\": \"unofficial\", \"diff\": {{ \"url\": \"{}\" }} }}", + firstPatch ? "" : ",", esc(path)); + firstPatch = false; + properties.emplace_back("mcpp:patch", patch); + } + out += " ] },\n"; + properties.emplace_back("mcpp:variant", d.variant); + } + if (!properties.empty()) { + out += " \"properties\": ["; + for (std::size_t i = 0; i < properties.size(); ++i) + out += std::format("{} {{ \"name\": \"{}\", \"value\": \"{}\" }}", i ? "," : "", + esc(properties[i].first), esc(properties[i].second)); + out += " ],\n"; } if (!d.source.empty()) out += " \"externalReferences\": [ { \"type\": \"distribution\"," diff --git a/src/modgraph/scanner.cppm b/src/modgraph/scanner.cppm index 711b69148..75919b5d2 100644 --- a/src/modgraph/scanner.cppm +++ b/src/modgraph/scanner.cppm @@ -1369,6 +1369,7 @@ void scan_one_into(ScanResult& result, } } for (std::size_t i = 0; i < globFlagHits.size(); ++i) { + if (manifest.buildConfig.globFlags[i].embedded) continue; if (globFlagHits[i] == 0) { // Zero scanned-source hits is not yet a dead glob: the entry may // target files that are real but not scanned sources — notably diff --git a/src/pm/lock_io.cppm b/src/pm/lock_io.cppm index 99f43752f..7e486ae9d 100644 --- a/src/pm/lock_io.cppm +++ b/src/pm/lock_io.cppm @@ -42,6 +42,11 @@ struct LockedPackage { // for one target rewrites its own and keeps the others' (SPEC-013 §15.1). // Empty in a lock written before the field. std::vector targets; + // A variant (design 2026-11-02 §14.1): the patch set's digest, and each + // patch as `#sha256:`, in order. Empty for an unpatched + // package and in a lock written before the fields. + std::string variant; + std::vector patches; }; // Parsed form of a git source string as written to mcpp.lock. @@ -151,6 +156,10 @@ std::expected load(const std::filesystem::path& path) { if (auto it = tt.find("targets"); it != tt.end() && it->second.is_array()) for (auto const& e : it->second.as_array()) if (e.is_string()) lp.targets.push_back(e.as_string()); + if (auto it = tt.find("variant"); it != tt.end() && it->second.is_string()) lp.variant = it->second.as_string(); + if (auto it = tt.find("patches"); it != tt.end() && it->second.is_array()) + for (auto const& e : it->second.as_array()) + if (e.is_string()) lp.patches.push_back(e.as_string()); // v1 → v2 migration: default namespace to "mcpplibs". if (fileVersion < 2 && lp.namespace_.empty()) { @@ -208,6 +217,12 @@ std::string serialize(const Lockfile& lock) { for (auto const& t : p.targets) list += (list.empty() ? "" : ", ") + t::escape_string(t); out += std::format("targets = [{}]\n", list); } + if (!p.variant.empty()) { + out += std::format("variant = {}\n", t::escape_string(p.variant)); + std::string list; + for (auto const& e : p.patches) list += (list.empty() ? "" : ", ") + t::escape_string(e); + out += std::format("patches = [{}]\n", list); + } out += "\n"; } return out; diff --git a/src/pm/patch.cppm b/src/pm/patch.cppm new file mode 100644 index 000000000..309856518 --- /dev/null +++ b/src/pm/patch.cppm @@ -0,0 +1,356 @@ +// A dependency's patches, applied (design 2026-11-02, part 3, §14.1). +// +// A PATCHED LIBRARY IS ANOTHER LIBRARY. A dependency edge may carry `patches`, +// unified diffs that are applied in order to the verified source of the +// package; the result is a VARIANT, whose identity is the package, its version +// and the patch set, and which has a source tree of its own under the mcpp +// home (`cache/src//+p`), so that its build, its +// BMIs and its cache never meet the unpatched package's. +// +// THE APPLIER IS EXACT AND IS mcpp's OWN. A hunk applies where its header says +// and only if every line of context and every line it removes is there, byte +// for byte; there is no search for a nearby place and no fuzz, so a patch +// gives the same tree on every machine or none. It reads what `diff -u` and +// `git format-patch` write, removes the one leading directory of each path +// (`a/` and `b/`), and transfers bytes as they are: a line ending is part of +// its line, and nothing is converted. No `patch` or `git` program is used. +export module mcpp.pm.patch; + +import std; +import mcpp.pack.distribution; // sha256 + +export namespace mcpp::pm::patch { + +// The SHA-256 of one patch file's bytes, or nothing when it cannot be read. +std::optional file_sha256(const std::filesystem::path& patch); + +// The set's name: the SHA-256 over each patch's SHA-256 and a newline, in +// order. Empty for no patches. The paths are not part of it: one set of +// patches carried by two packages is one variant. +std::expected set_digest(const std::vector& patches); + +// Applies one patch's text to the tree at `root`. `name` is how a failure +// names the patch. +std::expected apply(const std::filesystem::path& root, + std::string_view text, std::string_view name); + +// The variant's tree: `source` copied to `variant` and the patches applied in +// order, unless `variant` already holds the result for this digest. Written +// beside `variant` and renamed into place, so a tree that exists is whole. +std::expected materialize(const std::filesystem::path& source, + const std::vector& patches, + std::string_view digest, + const std::filesystem::path& variant); + +// The directory name of a variant of `version`: `+p`. Short, because Windows paths have a length. +std::string variant_dir_name(std::string_view version, std::string_view digest); + +} // namespace mcpp::pm::patch + +namespace mcpp::pm::patch { + +namespace { + +namespace fs = std::filesystem; + +std::optional read_bytes(const fs::path& p) { + std::ifstream in(p, std::ios::binary); + if (!in) return std::nullopt; + std::ostringstream ss; + ss << in.rdbuf(); + return std::move(ss).str(); +} + +bool write_bytes(const fs::path& p, std::string_view bytes) { + std::error_code ec; + fs::create_directories(p.parent_path(), ec); + // A link is replaced by a file rather than written through. + if (fs::is_symlink(fs::symlink_status(p, ec))) fs::remove(p, ec); + std::ofstream out(p, std::ios::binary | std::ios::trunc); + if (!out) return false; + out.write(bytes.data(), static_cast(bytes.size())); + return static_cast(out); +} + +// A text as lines, each without its '\n' and with a flag saying whether one +// ended it (only the last line may lack it). A '\r' stays in its line. +struct line { std::string text; bool newline = true; }; + +std::vector split_lines(std::string_view s) { + std::vector out; + std::size_t at = 0; + while (at < s.size()) { + const auto nl = s.find('\n', at); + if (nl == std::string_view::npos) { out.push_back({std::string(s.substr(at)), false}); break; } + out.push_back({std::string(s.substr(at, nl - at)), true}); + at = nl + 1; + } + return out; +} + +std::string join_lines(const std::vector& ls) { + std::string out; + for (auto const& l : ls) { out += l.text; if (l.newline) out += '\n'; } + return out; +} + +struct hunk { + std::size_t oldStart = 0, oldCount = 0, newStart = 0, newCount = 0; + std::string header; + std::vector oldLines, newLines; +}; + +struct file_patch { + std::string oldPath, newPath; // after the leading directory is removed; "" for /dev/null + std::vector hunks; +}; + +// "--- a/x/y.c\t2026-..." -> "x/y.c"; "/dev/null" -> "". +std::expected header_path(std::string_view rest) { + if (auto tab = rest.find('\t'); tab != std::string_view::npos) rest = rest.substr(0, tab); + while (!rest.empty() && (rest.back() == '\r' || rest.back() == ' ')) rest.remove_suffix(1); + if (rest == "/dev/null") return std::string{}; + if (rest.size() >= 2 && rest.front() == '"') + return std::unexpected(std::string("a quoted path is not supported")); + const auto slash = rest.find('/'); + if (slash == std::string_view::npos) + return std::unexpected(std::format("'{}' carries no leading directory to remove", rest)); + std::string path(rest.substr(slash + 1)); + if (path.empty() || path.find("..") != std::string::npos || path.front() == '/') + return std::unexpected(std::format("'{}' does not name a file of the package", rest)); + return path; +} + +bool parse_range(std::string_view s, std::size_t& start, std::size_t& count) { + count = 1; + const auto comma = s.find(','); + auto num = [](std::string_view t, std::size_t& v) { + if (t.empty()) return false; + v = 0; + for (char c : t) { if (c < '0' || c > '9') return false; v = v * 10 + static_cast(c - '0'); } + return true; + }; + if (comma == std::string_view::npos) return num(s, start); + return num(s.substr(0, comma), start) && num(s.substr(comma + 1), count); +} + +std::expected, std::string> parse(std::string_view text) { + const auto lines = split_lines(text); + std::vector files; + std::size_t i = 0; + while (i < lines.size()) { + std::string_view l = lines[i].text; + if (l.starts_with("GIT binary patch") || l.starts_with("Binary files ")) + return std::unexpected(std::string("a binary change is not supported")); + if (l.starts_with("rename from ") || l.starts_with("copy from ")) + return std::unexpected(std::string("a rename or a copy is not supported; state it as a removal and an addition")); + if (!l.starts_with("--- ") || i + 1 >= lines.size() || !std::string_view(lines[i + 1].text).starts_with("+++ ")) { + ++i; + continue; + } + file_patch fp; + auto oldPath = header_path(l.substr(4)); + if (!oldPath) return std::unexpected(oldPath.error()); + auto newPath = header_path(std::string_view(lines[i + 1].text).substr(4)); + if (!newPath) return std::unexpected(newPath.error()); + fp.oldPath = *oldPath; + fp.newPath = *newPath; + if (fp.oldPath.empty() && fp.newPath.empty()) + return std::unexpected(std::string("a file both absent and absent")); + if (!fp.oldPath.empty() && !fp.newPath.empty() && fp.oldPath != fp.newPath) + return std::unexpected(std::format("'{}' and '{}' name two files; a rename is not supported", + fp.oldPath, fp.newPath)); + i += 2; + while (i < lines.size() && std::string_view(lines[i].text).starts_with("@@ ")) { + hunk h; + h.header = lines[i].text; + std::string_view hd = lines[i].text; + const auto end = hd.find(" @@", 3); + if (end == std::string_view::npos) + return std::unexpected(std::format("'{}' is not a hunk header", hd)); + std::string_view ranges = hd.substr(3, end - 3); + const auto space = ranges.find(' '); + if (space == std::string_view::npos || ranges[0] != '-' || ranges[space + 1] != '+' + || !parse_range(ranges.substr(1, space - 1), h.oldStart, h.oldCount) + || !parse_range(ranges.substr(space + 2), h.newStart, h.newCount)) + return std::unexpected(std::format("'{}' is not a hunk header", hd)); + ++i; + std::size_t seenOld = 0, seenNew = 0; + char last = 0; + while (i < lines.size() && (seenOld < h.oldCount || seenNew < h.newCount + || std::string_view(lines[i].text).starts_with("\\"))) { + std::string_view body = lines[i].text; + if (body.starts_with("\\")) { + // "\ No newline at end of file": the line before has none. + if (last == '-' || last == ' ') { if (!h.oldLines.empty()) h.oldLines.back().newline = false; } + if (last == '+' || last == ' ') { if (!h.newLines.empty()) h.newLines.back().newline = false; } + ++i; + continue; + } + const char kind = body.empty() ? ' ' : body[0]; + const std::string rest(body.empty() ? std::string_view{} : body.substr(1)); + if (kind == ' ') { h.oldLines.push_back({rest, true}); h.newLines.push_back({rest, true}); ++seenOld; ++seenNew; } + else if (kind == '-') { h.oldLines.push_back({rest, true}); ++seenOld; } + else if (kind == '+') { h.newLines.push_back({rest, true}); ++seenNew; } + else return std::unexpected(std::format("'{}' ends before its {} and {} lines", h.header, h.oldCount, h.newCount)); + last = kind; + ++i; + } + if (seenOld != h.oldCount || seenNew != h.newCount) + return std::unexpected(std::format("'{}' ends before its {} and {} lines", h.header, h.oldCount, h.newCount)); + fp.hunks.push_back(std::move(h)); + } + if (fp.hunks.empty()) + return std::unexpected(std::format("the change to '{}' has no hunk", + fp.newPath.empty() ? fp.oldPath : fp.newPath)); + files.push_back(std::move(fp)); + } + if (files.empty()) return std::unexpected(std::string("no change is described")); + return files; +} + +std::string shown(const line& l) { + std::string s = l.text; + if (s.size() > 60) s = s.substr(0, 57) + "..."; + return s; +} + +} // namespace + +std::optional file_sha256(const fs::path& patch) { + auto bytes = read_bytes(patch); + if (!bytes) return std::nullopt; + return mcpp::pack::distribution::sha256(*bytes); +} + +std::expected set_digest(const std::vector& patches) { + if (patches.empty()) return std::string{}; + std::string acc; + for (auto const& p : patches) { + auto one = file_sha256(p); + if (!one) return std::unexpected(std::format("the patch '{}' cannot be read", p)); + acc += *one; + acc += '\n'; + } + return mcpp::pack::distribution::sha256(acc); +} + +std::string variant_dir_name(std::string_view version, std::string_view digest) { + return std::format("{}+p{}", version, digest.substr(0, std::min(12, digest.size()))); +} + +std::expected apply(const fs::path& root, std::string_view text, std::string_view name) { + auto parsed = parse(text); + if (!parsed) return std::unexpected(std::format("patch '{}': {}", name, parsed.error())); + for (auto const& fp : *parsed) { + const std::string rel = fp.newPath.empty() ? fp.oldPath : fp.newPath; + const fs::path file = root / fs::path(rel); + std::vector original; + if (!fp.oldPath.empty()) { + auto bytes = read_bytes(file); + if (!bytes) return std::unexpected(std::format("patch '{}': '{}' is not in the package", name, rel)); + original = split_lines(*bytes); + } else { + std::error_code ec; + if (fs::exists(file, ec)) + return std::unexpected(std::format("patch '{}': '{}' is created and already exists", name, rel)); + } + std::vector result; + std::size_t cursor = 0; // the next original line not yet copied + std::size_t index = 0; + for (auto const& h : fp.hunks) { + ++index; + // Where the old lines begin, 0-based. A hunk that removes nothing + // states the line AFTER which it adds; one that does states the + // first line it reads. + const std::size_t at = h.oldCount == 0 ? h.oldStart : h.oldStart - 1; + if (h.oldCount > 0 && h.oldStart == 0) + return std::unexpected(std::format("patch '{}': '{}' hunk {} ({}) starts at line 0", name, rel, index, h.header)); + if (at < cursor) + return std::unexpected(std::format("patch '{}': '{}' hunk {} ({}) overlaps the hunk before it", name, rel, index, h.header)); + if (at + h.oldLines.size() > original.size()) + return std::unexpected(std::format( + "patch '{}': '{}' hunk {} ({}) does not apply: the file has {} lines", name, rel, index, h.header, original.size())); + for (std::size_t k = 0; k < h.oldLines.size(); ++k) { + auto const& want = h.oldLines[k]; + auto const& have = original[at + k]; + if (want.text != have.text || want.newline != have.newline) + return std::unexpected(std::format( + "patch '{}': '{}' hunk {} ({}) does not apply at line {}: it expects '{}'{} and the file has '{}'{}", + name, rel, index, h.header, at + k + 1, shown(want), want.newline ? "" : " (no newline)", + shown(have), have.newline ? "" : " (no newline)")); + } + for (std::size_t k = cursor; k < at; ++k) result.push_back(original[k]); + for (auto const& l : h.newLines) result.push_back(l); + cursor = at + h.oldLines.size(); + } + for (std::size_t k = cursor; k < original.size(); ++k) result.push_back(original[k]); + std::error_code ec; + if (fp.newPath.empty()) { + if (!result.empty()) + return std::unexpected(std::format("patch '{}': '{}' is removed and keeps lines the patch does not state", name, rel)); + fs::remove(file, ec); // a link is removed, never what it names + if (ec) return std::unexpected(std::format("patch '{}': '{}' cannot be removed: {}", name, rel, ec.message())); + continue; + } + if (!write_bytes(file, join_lines(result))) + return std::unexpected(std::format("patch '{}': '{}' cannot be written", name, rel)); + } + return {}; +} + +std::expected materialize(const fs::path& source, + const std::vector& patches, + std::string_view digest, + const fs::path& variant) { + const fs::path marker = variant / ".mcpp-variant"; + std::error_code ec; + if (auto have = read_bytes(marker); have && *have == std::string(digest) + "\n") { + // Used now: the cache's collection by last use reads this time. + fs::last_write_time(marker, fs::file_time_type::clock::now(), ec); + return {}; + } + + // Written beside its place and renamed into it, so that a variant that + // exists is whole: a build interrupted while patching leaves a directory + // nothing reads. + const fs::path work = variant.parent_path() / (variant.filename().string() + ".partial"); + fs::remove_all(work, ec); + fs::create_directories(work.parent_path(), ec); + // THE COPY IS OF FILES, NOT OF LINKS. A store may hold a package's version + // directory, or files in it, as links into another store; a link copied as + // a link would have the patch written through it into the source every + // other build reads. Links are followed, so the variant's tree is its own. + const auto real = fs::weakly_canonical(source, ec); + if (ec) return std::unexpected(std::format("the source '{}' cannot be resolved: {}", + source.string(), ec.message())); + fs::copy(real, work, fs::copy_options::recursive, ec); + if (ec) return std::unexpected(std::format("the source '{}' cannot be copied for its variant: {}", + source.string(), ec.message())); + for (auto const& p : patches) { + auto text = read_bytes(p); + if (!text) { fs::remove_all(work, ec); return std::unexpected(std::format("the patch '{}' cannot be read", p)); } + if (auto r = apply(work, *text, fs::path(p).filename().string()); !r) { + fs::remove_all(work, ec); + return std::unexpected(std::format("{} (from {})", r.error(), p)); + } + } + if (!write_bytes(work / ".mcpp-variant", std::string(digest) + "\n")) { + fs::remove_all(work, ec); + return std::unexpected(std::format("the variant '{}' cannot be recorded", variant.string())); + } + fs::remove_all(variant, ec); + fs::rename(work, variant, ec); + if (ec) { + // Another build made the same variant meanwhile: its tree is the same. + if (auto have = read_bytes(marker); have && *have == std::string(digest) + "\n") { + fs::remove_all(work, ec); + return {}; + } + return std::unexpected(std::format("the variant '{}' cannot be placed: {}", variant.string(), ec.message())); + } + return {}; +} + +} // namespace mcpp::pm::patch diff --git a/src/provider/xlings/subos_info.cppm b/src/provider/xlings/subos_info.cppm index 877f68d6e..7b39cbe9f 100644 --- a/src/provider/xlings/subos_info.cppm +++ b/src/provider/xlings/subos_info.cppm @@ -88,11 +88,13 @@ mcpp::provider::runtime_info::Info read(const std::filesystem::path& subosDir) { info.note = std::format( "subos '{}' does not describe itself (no `{}` block), so programs " "run from here get no environment it declares — a GL application " - "will not find its drivers. Run `xlings self doctor --fix` from " - "inside this SubOS to have it described (xlings 2026.8.17.1 or " + "will not find its drivers. Run `XLINGS_HOME={} xlings self doctor --fix` " + "from inside this SubOS to have it described (xlings 2026.8.17.1 or " "newer records what the SubOS actually runs; older ones record the " - "built-in default, which may not be what is installed here)", - subosDir.string(), kBlock); + "built-in default, which may not be what is installed here). The home " + "is named because an xlings run without it uses the system's own home " + "on a machine that has one, and repairs that one instead (mcpp#881)", + subosDir.string(), kBlock, subosDir.parent_path().parent_path().string()); return info; } diff --git a/src/runtime/binding.cppm b/src/runtime/binding.cppm index e09d71c3b..ae4d68e0b 100644 --- a/src/runtime/binding.cppm +++ b/src/runtime/binding.cppm @@ -392,10 +392,12 @@ resolve_runtime_binding( " Where the C runtime comes from a payload, there is now no " "declared runtime to bind to — mcpp declines to guess a version, so " "the link falls back to the host and the hermeticity check will say " - "so. Run `xlings self doctor --fix` from inside this SubOS to have " - "it described (openxlings/xlings#547).", + "so. Run `XLINGS_HOME={} xlings self doctor --fix` from inside this " + "SubOS to have it described (openxlings/xlings#547); without the home, " + "an xlings run uses the system's own home where there is one (mcpp#881).", selection.subosName, - info.note.empty() ? "no `subos_info` block" : info.note)); + info.note.empty() ? "no `subos_info` block" : info.note, + out.subosDir.parent_path().parent_path().string())); } else if (info.schema > mcpp::provider::runtime_info::kSupportedSchema) { // Mirrors `subos_info::read`, which already reads a HIGHER schema and // says so. A consumer stricter than its own reader is a time bomb: diff --git a/tests/e2e/1001_a_dependency_carries_its_patches.sh b/tests/e2e/1001_a_dependency_carries_its_patches.sh new file mode 100755 index 000000000..3b20c47ff --- /dev/null +++ b/tests/e2e/1001_a_dependency_carries_its_patches.sh @@ -0,0 +1,104 @@ +#!/usr/bin/env bash +# requires: gcc +# 1001 -- a dependency edge carries patches (design 2026-11-02 part 3 §14.1). A +# patched library is another library: the verified source is copied to the +# variant's own tree under the mcpp home and patched there, in order, exactly; +# the lock names the variant and each patch with its SHA-256; and a patch that +# does not apply is refused naming the patch and its hunk. +# +# Criteria: +# A. The program sees the patched header; the package's own installed source +# is untouched, and the variant's tree is beside the other variants. +# B. mcpp.lock records the variant, and each patch by its path and SHA-256; +# `mcpp sbom` states the pedigree. +# C. A changed patch is another variant: the build uses the new tree. +# D. A patch that does not apply is refused with the patch's name and the +# hunk, and leaves no variant behind. +# E. `patches` on a `path` dependency is refused. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +export MCPP_HOME="$TMP/mcpp-home" +source "$(dirname "$0")/_inherit_toolchain.sh" +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } + +cd "$TMP" +mkdir -p app/src app/patches && cd app +printf '%s\n' '--- a/zlib-1.3.2/zlib.h' '+++ b/zlib-1.3.2/zlib.h' '@@ -31,4 +31,6 @@' \ + ' #ifndef ZLIB_H' ' #define ZLIB_H' ' ' '+#define MCPP_PATCHED_ZLIB 1' '+' ' #ifdef ZLIB_BUILD' \ + > patches/0001-mark.patch +cat > mcpp.toml <<'EOF' +[package] +name = "app" +version = "0.1.0" + +[dependencies] +"compat.zlib" = { version = "1.3.2", patches = ["patches/0001-mark.patch"] } +EOF +cat > src/main.cpp <<'EOF' +#include +#include +int main() { +#ifdef MCPP_PATCHED_ZLIB + std::printf("patched %d %s\n", MCPP_PATCHED_ZLIB, zlibVersion()); + return 0; +#else + std::printf("unpatched\n"); + return 1; +#endif +} +EOF + +# A +"$MCPP" run > a.log 2>&1 || fail "the patched build did not run" a.log +grep -q '^patched 1 1.3.2' a.log || fail "the program did not see the patched header" a.log +variant=$(ls -d "$MCPP_HOME"/cache/src/compat.zlib/1.3.2+p* 2>/dev/null | head -1) +[ -n "$variant" ] && [ -f "$variant/.mcpp-variant" ] || fail "no variant tree under the mcpp home" <(find "$MCPP_HOME/cache" -maxdepth 3) +grep -q 'MCPP_PATCHED_ZLIB' "$variant/zlib-1.3.2/zlib.h" || fail "the variant's header is not patched" "$variant/zlib-1.3.2/zlib.h" +installed=$(ls -d "$MCPP_HOME"/registry/data/xpkgs/compat-x-zlib/1.3.2 2>/dev/null | head -1) +if [ -n "$installed" ]; then + ! grep -q 'MCPP_PATCHED_ZLIB' "$installed"/zlib-1.3.2/zlib.h || fail "the installed source was patched in place" +fi + +# B +grep -q '^variant = ' mcpp.lock || fail "the lock names no variant" mcpp.lock +grep -q 'patches/0001-mark.patch#sha256:[0-9a-f]\{64\}' mcpp.lock || fail "the lock does not name the patch and its digest" mcpp.lock +"$MCPP" emit sbom > sbom.json 2>sbom.err || fail "sbom failed" sbom.err +grep -q '"pedigree"' sbom.json && grep -q '0001-mark.patch' sbom.json || fail "the bill of materials states no pedigree" sbom.json + +# C +sed -i 's/MCPP_PATCHED_ZLIB 1/MCPP_PATCHED_ZLIB 2/' patches/0001-mark.patch +"$MCPP" run > c.log 2>&1 || fail "the build with a changed patch failed" c.log +grep -q '^patched 2 ' c.log || fail "a changed patch did not make another variant" c.log +[ "$(ls -d "$MCPP_HOME"/cache/src/compat.zlib/1.3.2+p* | wc -l)" -ge 2 ] || fail "the changed patch did not get its own tree" <(ls "$MCPP_HOME/cache/src/compat.zlib") + +# D +cat > patches/0001-mark.patch <<'EOF' +--- a/zlib-1.3.2/zlib.h ++++ b/zlib-1.3.2/zlib.h +@@ -31,1 +31,1 @@ +-#ifndef NOT_THE_LINE ++#ifndef ZLIB_H +EOF +before=$(ls -d "$MCPP_HOME"/cache/src/compat.zlib/* | wc -l) +if "$MCPP" build > d.log 2>&1; then fail "a patch that does not apply was accepted" d.log; fi +grep -q "0001-mark.patch" d.log && grep -q "hunk 1" d.log || fail "the refusal does not name the patch and the hunk" d.log +[ "$(ls -d "$MCPP_HOME"/cache/src/compat.zlib/* | wc -l)" = "$before" ] || fail "a failed patching left a tree" <(ls "$MCPP_HOME/cache/src/compat.zlib") + +# E +mkdir -p ../lib/src +printf '[package]\nname = "lib"\nversion = "0.1.0"\n' > ../lib/mcpp.toml +printf 'int lib_one() { return 1; }\n' > ../lib/src/lib.cpp +cat > mcpp.toml <<'EOF' +[package] +name = "app" +version = "0.1.0" + +[dependencies] +lib = { path = "../lib", patches = ["patches/0001-mark.patch"] } +EOF +if "$MCPP" build > e.log 2>&1; then fail "patches on a path dependency were accepted" e.log; fi +grep -q "is a path and carries patches" e.log || fail "the refusal does not say why" e.log + +echo "OK" diff --git a/tests/e2e/1002_a_dependency_is_embedded.sh b/tests/e2e/1002_a_dependency_is_embedded.sh new file mode 100755 index 000000000..6d2bc33ee --- /dev/null +++ b/tests/e2e/1002_a_dependency_is_embedded.sh @@ -0,0 +1,109 @@ +#!/usr/bin/env bash +# requires: gcc +# 1002 -- `embed = true` compiles a dependency into its consumer (design +# 2026-11-02 part 3 §14.1): the dependency's sources, less `exclude`, are the +# consumer's, compiled with the consumer's flags and the dependency's include +# directories and macros; the dependency builds no library of its own, and a +# program that reaches it holds its embedder or is refused. +# +# Criteria: +# A. The program links and runs with zlib embedded in its library package; +# no library of compat.zlib is built, and an excluded file is not compiled. +# B. The dependency's own compile flags reach its files (its `-include` of a +# generated header) and not the consumer's own. +# C. `exclude` without `embed` is refused at the manifest. +# D. A program that reaches the embedded package without its embedder is +# refused, naming both. +# E. An embedder with a build program builds without a warning about the +# flags `embed' gives the dependency's files: the build program's own plan +# compiles none of them, and the entry is the engine's, not a typo. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +export MCPP_HOME="$TMP/mcpp-home" +source "$(dirname "$0")/_inherit_toolchain.sh" +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } + +cd "$TMP" +mkdir -p zip/src app/src +cat > zip/mcpp.toml <<'EOF' +[package] +name = "zip" +version = "0.1.0" + +[dependencies] +"compat.zlib" = { version = "1.3.2", embed = true, exclude = ["*/uncompr.c"] } +EOF +cat > zip/src/zip.cppm <<'EOF' +module; +#include +export module zip; +export int zip_bound(int n) { return static_cast(compressBound(static_cast(n))); } +export const char* zip_version() { return zlibVersion(); } +EOF +cat > app/mcpp.toml <<'EOF' +[package] +name = "app" +version = "0.1.0" + +[dependencies] +zip = { path = "../zip" } +EOF +cat > app/src/main.cpp <<'EOF' +import std; +import zip; +int main() { std::println("{} {}", zip_version(), zip_bound(100) > 100); } +EOF + +cat > zip/build.mcpp <<'EOF' +import std; +int main() { return 0; } +EOF + +# A +cd app +"$MCPP" run > a.log 2>&1 || fail "the program with an embedded zlib did not run" a.log + +# E +! grep -q 'matched no source file' a.log || fail "the flags embed gives were reported as a glob that matched nothing" a.log +grep -q '^1.3.2 true' a.log || fail "the program did not call the embedded zlib" a.log +dir=$(ls -d target/*/*/ | grep -E '/[0-9a-f]{16}/$' | head -1) +db=$(find "$dir" -name compile_commands.json | head -1) +[ -n "$db" ] || db=compile_commands.json +grep -q 'deflate.c' "$db" || fail "zlib's sources were not compiled" "$db" +! grep -q 'uncompr.c' "$db" || fail "an excluded file was compiled" "$db" +! find "$dir" -name '*zlib*.a' -o -name '*zlib*.lib' | grep -q . || fail "a library of the embedded package was built" <(find "$dir" -name '*.a') + +# B +grep -q 'mcpp_zlib_config.h' "$db" || fail "the dependency's own flags did not reach its files" "$db" +python3 - "$db" <<'PY' || fail "the dependency's flags reached the consumer's own unit" "$db" +import json, sys +for e in json.load(open(sys.argv[1])): + if e["file"].endswith("zip.cppm") and "mcpp_zlib_config.h" in " ".join(e.get("arguments", [])) + e.get("command", ""): + sys.exit(1) +PY +cd .. + +# C +mkdir -p bad/src && cd bad +printf '[package]\nname = "bad"\nversion = "0.1.0"\n\n[dependencies]\n"compat.zlib" = { version = "1.3.2", exclude = ["*/gzwrite.c"] }\n' > mcpp.toml +printf 'int main() {}\n' > src/main.cpp +if "$MCPP" build > c.log 2>&1; then fail "exclude without embed was accepted" c.log; fi +grep -q "exclude is allowed only with embed" c.log || fail "the refusal does not say why" c.log +cd .. + +# D -- one configuration of a workspace: `app` holds the embedder, `plain` +# reaches compat.zlib directly. The embedded package compiles nothing of its +# own there, so `plain` would link its declarations and no definitions. +mkdir -p ws/plain/src && cd ws +cp -r ../app ../zip . +rm -rf app/target +printf '[workspace]\nmembers = ["app", "plain", "zip"]\n' > mcpp.toml +printf '[package]\nname = "plain"\nversion = "0.1.0"\n\n[dependencies]\n"compat.zlib" = "1.3.2"\n' > plain/mcpp.toml +printf '#include \nint main() { return zlibVersion()[0] == 0; }\n' > plain/src/main.cpp +if "$MCPP" build --workspace > d.log 2>&1; then fail "a program without the embedder was accepted" d.log; fi +grep -q "which 'zip' embeds" d.log || fail "the refusal does not name the embedder" d.log +cd .. + +echo "OK" diff --git a/tests/e2e/1003_one_program_one_variant.sh b/tests/e2e/1003_one_program_one_variant.sh new file mode 100755 index 000000000..50b6159a4 --- /dev/null +++ b/tests/e2e/1003_one_program_one_variant.sh @@ -0,0 +1,82 @@ +#!/usr/bin/env bash +# requires: gcc +# 1003 -- a patched library is another library, and one program holds one +# variant of a package (design 2026-11-02 part 3 §14.1). Two programs of one +# workspace configuration may each use their own patch set of compat.zlib; a +# program that reaches two variants of it -- or a variant and the package +# without patches -- is refused, naming the variants. +# +# Criteria: +# A. `one` (through member `a`, patch P1) and `two` (patch P2) build in one +# configuration, and each sees its own patch. +# B. `three`, reaching `a`'s variant and P2's, is refused naming both. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +export MCPP_HOME="$TMP/mcpp-home" +source "$(dirname "$0")/_inherit_toolchain.sh" +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } + +mark() { # mark : a patch that defines MCPP_MARK as + printf '%s\n' '--- a/zlib-1.3.2/zlib.h' '+++ b/zlib-1.3.2/zlib.h' '@@ -31,4 +31,6 @@' \ + ' #ifndef ZLIB_H' ' #define ZLIB_H' ' ' "+#define MCPP_MARK $2" '+' ' #ifdef ZLIB_BUILD' > "$1" +} + +cd "$TMP" +mkdir -p ws && cd ws +mkdir -p a/src one/src two/src three/src patches +mark patches/p1.patch 1 +mark patches/p2.patch 2 +printf '[workspace]\nmembers = ["a", "one", "two"]\n' > mcpp.toml +cat > a/mcpp.toml <<'EOF' +[package] +name = "a" +version = "0.1.0" + +[dependencies] +"compat.zlib" = { version = "1.3.2", patches = ["../patches/p1.patch"] } +EOF +cat > a/src/a.cppm <<'EOF' +module; +#include +export module a; +export int a_mark() { return MCPP_MARK; } +EOF +printf '[package]\nname = "one"\nversion = "0.1.0"\n\n[dependencies]\na = { path = "../a" }\n' > one/mcpp.toml +printf 'import std;\nimport a;\nint main() { std::println("one {}", a_mark()); }\n' > one/src/main.cpp +cat > two/mcpp.toml <<'EOF' +[package] +name = "two" +version = "0.1.0" + +[dependencies] +"compat.zlib" = { version = "1.3.2", patches = ["../patches/p2.patch"] } +EOF +printf '#include \n#include \nint main() { std::printf("two %%d %%s\\n", MCPP_MARK, zlibVersion()); }\n' > two/src/main.cpp + +# A +"$MCPP" build --workspace > a.log 2>&1 || fail "two programs with two variants did not build" a.log +"$MCPP" run -p one > one.log 2>&1 || fail "one did not run" one.log +"$MCPP" run -p two > two.log 2>&1 || fail "two did not run" two.log +grep -q '^one 1' one.log || fail "one did not see its patch" one.log +grep -q '^two 2 1.3.2' two.log || fail "two did not see its patch" two.log +[ "$(ls -d "$MCPP_HOME"/cache/src/compat.zlib/1.3.2+p* | wc -l)" = 2 ] || fail "the two variants do not have a tree each" <(ls "$MCPP_HOME/cache/src/compat.zlib") + +# B +printf '[workspace]\nmembers = ["a", "one", "two", "three"]\n' > mcpp.toml +cat > three/mcpp.toml <<'EOF' +[package] +name = "three" +version = "0.1.0" + +[dependencies] +a = { path = "../a" } +"compat.zlib" = { version = "1.3.2", patches = ["../patches/p2.patch"] } +EOF +printf 'import a;\nint main() { return a_mark() == 0; }\n' > three/src/main.cpp +if "$MCPP" build -p three > b.log 2>&1; then fail "a program with two variants was accepted" b.log; fi +grep -q "reaches two variants of 'compat.zlib'" b.log || fail "the refusal does not name the package" b.log +[ "$(grep -c 'compat.zlib+p' b.log)" -ge 2 ] || fail "the refusal does not name both variants" b.log + +echo "OK" diff --git a/tests/e2e/1004_an_external_toolchain_finds_mcpps_ninja.sh b/tests/e2e/1004_an_external_toolchain_finds_mcpps_ninja.sh new file mode 100755 index 000000000..c3e3ef0c5 --- /dev/null +++ b/tests/e2e/1004_an_external_toolchain_finds_mcpps_ninja.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# requires: gcc +# 1004 -- a toolchain named by a path outside every provider finds the ninja +# mcpp installed into its own home (mcpp#875). Before 2026.11 only the one +# beside the compiler's provider package was asked, then PATH, so a machine +# whose PATH held no ninja failed with "cannot start 'ninja'" while mcpp's own +# sat installed. +# +# Criteria: +# A. With the compiler reached through a directory outside the store and a +# PATH that holds no ninja, the build succeeds. +# B. `MCPP_NINJA`, as a build program sees it, is the same absolute ninja. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +export MCPP_HOME="$TMP/mcpp-home" +source "$(dirname "$0")/_inherit_toolchain.sh" +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } + +gxx=$(ls -d "$MCPP_HOME"/registry/data/xpkgs/xim-x-gcc/*/bin/g++ 2>/dev/null | head -1) +[ -n "$gxx" ] || { echo "SKIP: no gcc payload to reach from outside the store"; exit 0; } +ninja=$(ls -d "$MCPP_HOME"/registry/data/xpkgs/xim-x-ninja/*/ninja 2>/dev/null | head -1) +[ -n "$ninja" ] || { echo "SKIP: mcpp's home holds no ninja"; exit 0; } +ln -s "$(dirname "$(dirname "$gxx")")" "$TMP/external-gcc" + +# The xlings mcpp drives, named outright: the PATH below holds no xlings. +export MCPP_VENDORED_XLINGS="${MCPP_VENDORED_XLINGS:-$(command -v xlings || true)}" +[ -n "$MCPP_VENDORED_XLINGS" ] || [ -x "$MCPP_HOME/registry/bin/xlings" ] || { echo "SKIP: no xlings to name"; exit 0; } + +# A PATH with the usual tools and no ninja. An external toolchain is complete +# where it is installed (MSYS2's GCC carries its assembler in its own `bin'); +# this one's assembler is the payload's binutils, put on PATH as such a +# toolchain's would be. +mkdir -p "$TMP/bin" +for t in sh bash env cat sed grep mkdir rm ls cp mv ln dirname basename uname tr head tail awk; do + p=$(command -v "$t" 2>/dev/null) && ln -sf "$p" "$TMP/bin/$t" +done +binutils=$(ls -d "$MCPP_HOME"/registry/data/xpkgs/xim-x-binutils/*/bin 2>/dev/null | head -1) +for t in as ld ar ranlib nm objcopy strip; do + [ -n "$binutils" ] && [ -x "$binutils/$t" ] && ln -sf "$binutils/$t" "$TMP/bin/$t" +done +[ ! -e "$TMP/bin/ninja" ] || fail "the restricted PATH holds a ninja" + +cd "$TMP" +mkdir -p app/src && cd app +printf '[package]\nname = "app"\nversion = "0.1.0"\n' > mcpp.toml +printf 'import std;\nint main() { std::println("built"); }\n' > src/main.cpp + +# A +PATH="$TMP/bin" MCPP_TOOLCHAIN="path:$TMP/external-gcc" "$MCPP" build > a.log 2>&1 \ + || fail "the build with an external toolchain and no ninja on PATH failed" a.log +! grep -q "cannot start 'ninja'" a.log || fail "the bare name reached PATH" a.log + +# B +PATH="$TMP/bin" MCPP_TOOLCHAIN="path:$TMP/external-gcc" "$MCPP" self env > b.log 2>&1 || true +if grep -q 'MCPP_NINJA' b.log; then + grep 'MCPP_NINJA' b.log | grep -q '/ninja' || fail "MCPP_NINJA is not an absolute ninja" b.log +fi +echo "OK" diff --git a/tests/e2e/1005_clean_stale_says_what_it_kept.sh b/tests/e2e/1005_clean_stale_says_what_it_kept.sh new file mode 100755 index 000000000..e04cdc718 --- /dev/null +++ b/tests/e2e/1005_clean_stale_says_what_it_kept.sh @@ -0,0 +1,45 @@ +#!/usr/bin/env bash +# requires: gcc +# 1005 -- `clean --stale` tells the truth about what it kept (mcpp#881), and +# its window of "recent" is the sweep's: `[build] keep_days`. +# +# Criteria: +# A. With an unrecorded directory written a minute ago, the summary says one +# directory was kept and within what window, and does not say every +# directory is recorded as current. +# B. The default window is `[build] keep_days` (3 days): a directory written +# two days ago is kept by default, and removed with `--older-than 1d`. +# C. With nothing unrecorded the summary is the one it always was. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +export MCPP_HOME="$TMP/mcpp-home" +source "$(dirname "$0")/_inherit_toolchain.sh" +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } + +cd "$TMP" +mkdir -p app/src && cd app +printf '[package]\nname = "app"\nversion = "0.1.0"\n' > mcpp.toml +printf 'int main() { return 0; }\n' > src/main.cpp +"$MCPP" build > build.log 2>&1 || fail "the build failed" build.log +triple=$(dirname "$(ls -d target/*/*/ | grep -E '/[0-9a-f]{16}/$' | head -1)") + +# C +"$MCPP" clean --stale --dry-run > c.log 2>&1 || fail "clean --stale failed" c.log +grep -q 'every fingerprint directory is recorded as current' c.log || fail "the plain summary changed" c.log + +# A +mkdir -p "$triple/deadbeefdeadbeef" && printf 'rule\n' > "$triple/deadbeefdeadbeef/build.ninja" +"$MCPP" clean --stale --dry-run > a.log 2>&1 || fail "clean --stale failed" a.log +grep -q 'kept .*deadbeefdeadbeef' a.log || fail "the kept directory is not listed" a.log +! grep -q 'every fingerprint directory is recorded as current' a.log || fail "the summary still says every directory is current" a.log +grep -q '1 directory not in use kept, written within the last 3d' a.log || fail "the summary does not say what was kept" a.log + +# B +touch -d '2 days ago' "$triple/deadbeefdeadbeef/build.ninja" "$triple/deadbeefdeadbeef" +"$MCPP" clean --stale > b0.log 2>&1 || fail "clean --stale failed" b0.log +[ -d "$triple/deadbeefdeadbeef" ] || fail "a directory written two days ago was removed under the default window" b0.log +"$MCPP" clean --stale --older-than 1d > b1.log 2>&1 || fail "clean --stale --older-than 1d failed" b1.log +[ ! -d "$triple/deadbeefdeadbeef" ] || fail "--older-than 1d kept a directory written two days ago" b1.log +echo "OK" diff --git a/tests/e2e/609_clean_stale.sh b/tests/e2e/609_clean_stale.sh index c073a0c46..f203c5194 100755 --- a/tests/e2e/609_clean_stale.sh +++ b/tests/e2e/609_clean_stale.sh @@ -37,7 +37,7 @@ echo stale > "target/$triple/deadbeefdeadbeef/bin/leftover" touch -t 200001010000 "target/$triple/deadbeefdeadbeef" "target/$triple/deadbeefdeadbeef/bin/leftover" # An unrecorded directory written just now (what a `mcpp test` build looks -# like to the record): kept by the default --older-than 1d. +# like to the record): kept by the default window ([build] keep_days, 3 days). mkdir -p "target/$triple/cafef00dcafef00d" echo latest > "target/$triple/cafef00dcafef00d/build.ninja" diff --git a/tests/e2e/998_a_build_keeps_its_bytes_once.sh b/tests/e2e/998_a_build_keeps_its_bytes_once.sh index 8677b0407..5245cb0c0 100755 --- a/tests/e2e/998_a_build_keeps_its_bytes_once.sh +++ b/tests/e2e/998_a_build_keeps_its_bytes_once.sh @@ -9,7 +9,9 @@ # B. On ELF, a program's debug information is compressed. # C. A shared library staged as an equal copy is placed again by a link. # D. `mcpp clean --stale` removes a `.build-mcpp/out/` no configuration -# names in its `uses.list`, and keeps the one the build used. +# names in its `uses.list` and that is older than the window (`[build] +# keep_days`, 3 days by default since 2026.11.2.1), and keeps the one the +# build used. # E. `mcpp clean --report` names the areas and the total. # F. A path tool's edit supersedes its store entry: one entry remains, and # `mcpp cache list` shows it as a tool. @@ -126,7 +128,7 @@ cp stage/libx.so stage/placed.so used=$(sed -n 's/^out //p' target/*/*/uses.list | head -1) [ -n "$used" ] || fail "the build names no generated-source directory" $(ls target/*/*/uses.list) mkdir -p target/.build-mcpp/out/0000000000000000/app -touch -d '2 days ago' target/.build-mcpp/out/0000000000000000 +touch -d '4 days ago' target/.build-mcpp/out/0000000000000000 "$MCPP" clean --stale > clean.log 2>&1 || fail "clean --stale failed" clean.log [ -d target/.build-mcpp/out/0000000000000000 ] && fail "an unused generated-source directory was kept" clean.log [ -d "target/.build-mcpp/out/$used" ] || fail "the generated-source directory in use was removed" clean.log diff --git a/tests/unit/test_manifest.cpp b/tests/unit/test_manifest.cpp index b6fb25cb8..07a2b4394 100644 --- a/tests/unit/test_manifest.cpp +++ b/tests/unit/test_manifest.cpp @@ -4029,7 +4029,7 @@ name = "depspeckeys" version = "0.1.0" [dependencies.compat] -everything = { version = "1.0.0", features = ["x"], default-features = false, visibility = "private", backend = "openblas", tools = ["t"], artifacts = ["a"], host-module = true, reexport = true, linkage = "shared" } +everything = { version = "1.0.0", features = ["x"], default-features = false, visibility = "private", backend = "openblas", tools = ["t"], artifacts = ["a"], host-module = true, reexport = true, linkage = "shared", patches = ["p/one.patch"], embed = true, exclude = ["src/prim.c"] } bygit = { git = "https://example.invalid/x.git", tag = "v1", visibility = "interface" } bypath = { path = "../sibling" } )"; @@ -4054,6 +4054,31 @@ bypath = { path = "../sibling" } EXPECT_NE(std::find(all->features.begin(), all->features.end(), "backend-openblas"), all->features.end()); EXPECT_EQ(all->linkage, "shared"); + // design 2026-11-02 §14.1: a patch is named relative to the manifest that + // carries it, and read as an absolute path. + ASSERT_EQ(all->patches.size(), 1u); + EXPECT_EQ(std::filesystem::path(all->patches[0]), + (tmp / "p" / "one.patch").lexically_normal()); + EXPECT_TRUE(all->embed); + ASSERT_EQ(all->exclude.size(), 1u); + EXPECT_EQ(all->exclude[0], "src/prim.c"); +} + +// design 2026-11-02 §14.1: `exclude` without `embed` would change what a +// library built on its own compiles for every consumer, which is a patch's to +// state; the manifest is refused, naming the edge. +TEST(Manifest, ExcludeWithoutEmbedIsRefused) { + auto m = mcpp::manifest::parse_string(R"( +[package] +name = "x" +version = "0.1.0" + +[dependencies] +"compat.zlib" = { version = "1.3.2", exclude = ["gzlib.c"] } +)"); + ASSERT_FALSE(m); + EXPECT_NE(m.error().message.find("exclude is allowed only with embed"), std::string::npos) + << m.error().message; } // ── #519: the dependency-form axis ──────────────────────────────────────── diff --git a/tests/unit/test_patch.cpp b/tests/unit/test_patch.cpp new file mode 100644 index 000000000..7e08ccb20 --- /dev/null +++ b/tests/unit/test_patch.cpp @@ -0,0 +1,161 @@ +#include + +import std; +import mcpp.pm.patch; + +// A dependency's patches (design 2026-11-02 part 3 §14.1). The applier is +// exact: a hunk applies where its header says, every line of context and every +// line it removes byte for byte, or the patch is refused naming the hunk. + +namespace fs = std::filesystem; +namespace pt = mcpp::pm::patch; + +namespace { + +fs::path fresh(std::string_view name) { + auto d = fs::temp_directory_path() / std::format("mcpp_patch_{}_{}", name, std::random_device{}()); + fs::remove_all(d); + fs::create_directories(d); + return d; +} + +void put(const fs::path& p, std::string_view bytes) { + fs::create_directories(p.parent_path()); + std::ofstream(p, std::ios::binary) << bytes; +} + +std::string get(const fs::path& p) { + std::ifstream in(p, std::ios::binary); + std::ostringstream ss; + ss << in.rdbuf(); + return ss.str(); +} + +TEST(Patch, AHunkAppliesWhereItsHeaderSays) { + auto root = fresh("apply"); + put(root / "src/a.c", "one\ntwo\nthree\nfour\n"); + const std::string diff = + "--- a/src/a.c\n" + "+++ b/src/a.c\n" + "@@ -2,2 +2,3 @@\n" + " two\n" + "-three\n" + "+THREE\n" + "+three and a half\n"; + ASSERT_TRUE(pt::apply(root, diff, "p.patch")); + EXPECT_EQ(get(root / "src/a.c"), "one\ntwo\nTHREE\nthree and a half\nfour\n"); +} + +TEST(Patch, AHunkThatDoesNotMatchIsRefusedNamingIt) { + auto root = fresh("refuse"); + put(root / "src/a.c", "one\ntwo\nthree\n"); + const std::string diff = + "--- a/src/a.c\n" + "+++ b/src/a.c\n" + "@@ -2,1 +2,1 @@\n" + "-TWO\n" + "+2\n"; + auto r = pt::apply(root, diff, "p.patch"); + ASSERT_FALSE(r); + EXPECT_NE(r.error().find("hunk 1"), std::string::npos) << r.error(); + EXPECT_NE(r.error().find("line 2"), std::string::npos) << r.error(); + EXPECT_EQ(get(root / "src/a.c"), "one\ntwo\nthree\n"); // untouched +} + +TEST(Patch, NothingIsSearchedForElsewhere) { + // The text the hunk expects is in the file, one line later than its + // header says: an exact applier refuses rather than finding it. + auto root = fresh("nofuzz"); + put(root / "a.c", "zero\none\ntwo\n"); + const std::string diff = "--- a/a.c\n+++ b/a.c\n@@ -1,1 +1,1 @@\n-one\n+ONE\n"; + EXPECT_FALSE(pt::apply(root, diff, "p.patch")); +} + +TEST(Patch, BytesAreKeptAsTheyAre) { + // CRLF lines stay CRLF, and a last line without a newline stays without. + auto root = fresh("bytes"); + put(root / "w.txt", "a\r\nb\r\nc"); + const std::string diff = + "--- a/w.txt\n+++ b/w.txt\n@@ -2,2 +2,2 @@\n b\r\n-c\n\\ No newline at end of file\n+C\n\\ No newline at end of file\n"; + ASSERT_TRUE(pt::apply(root, diff, "p.patch")); + EXPECT_EQ(get(root / "w.txt"), "a\r\nb\r\nC"); +} + +TEST(Patch, FilesAreCreatedAndRemoved) { + auto root = fresh("create"); + put(root / "old.c", "x\n"); + const std::string diff = + "diff --git a/new.c b/new.c\n" + "new file mode 100644\n" + "--- /dev/null\n+++ b/new.c\n@@ -0,0 +1,2 @@\n+int n;\n+int m;\n" + "diff --git a/old.c b/old.c\n" + "deleted file mode 100644\n" + "--- a/old.c\n+++ /dev/null\n@@ -1,1 +0,0 @@\n-x\n"; + ASSERT_TRUE(pt::apply(root, diff, "p.patch")); + EXPECT_EQ(get(root / "new.c"), "int n;\nint m;\n"); + EXPECT_FALSE(fs::exists(root / "old.c")); +} + +TEST(Patch, TheSetDigestIsTheContentsInOrder) { + auto d = fresh("digest"); + put(d / "one.patch", "A"); + put(d / "two.patch", "B"); + put(d / "renamed.patch", "A"); + auto ab = pt::set_digest({(d / "one.patch").string(), (d / "two.patch").string()}); + auto ba = pt::set_digest({(d / "two.patch").string(), (d / "one.patch").string()}); + auto ab2 = pt::set_digest({(d / "renamed.patch").string(), (d / "two.patch").string()}); + ASSERT_TRUE(ab && ba && ab2); + EXPECT_NE(*ab, *ba); + EXPECT_EQ(*ab, *ab2); + EXPECT_EQ(ab->size(), 64u); + EXPECT_EQ(pt::variant_dir_name("3.6.1", *ab), std::format("3.6.1+p{}", ab->substr(0, 12))); +} + +TEST(Patch, AVariantIsMadeOnceAndWhole) { + auto d = fresh("variant"); + put(d / "src/inc/m.h", "#define M 1\n"); + put(d / "fix.patch", "--- a/inc/m.h\n+++ b/inc/m.h\n@@ -1 +1 @@\n-#define M 1\n+#define M 2\n"); + auto digest = pt::set_digest({(d / "fix.patch").string()}); + ASSERT_TRUE(digest); + const auto variant = d / "variants" / pt::variant_dir_name("1.0", *digest); + ASSERT_TRUE(pt::materialize(d / "src", {(d / "fix.patch").string()}, *digest, variant)); + EXPECT_EQ(get(variant / "inc/m.h"), "#define M 2\n"); + EXPECT_EQ(get(d / "src/inc/m.h"), "#define M 1\n"); // the source is untouched + // A second call finds it made. + put(variant / "inc/m.h", "#define M 3\n"); + ASSERT_TRUE(pt::materialize(d / "src", {(d / "fix.patch").string()}, *digest, variant)); + EXPECT_EQ(get(variant / "inc/m.h"), "#define M 3\n"); + // A patch that does not apply leaves no variant. + put(d / "bad.patch", "--- a/inc/m.h\n+++ b/inc/m.h\n@@ -1 +1 @@\n-#define M 9\n+#define M 2\n"); + auto bad = pt::set_digest({(d / "bad.patch").string()}); + const auto badVariant = d / "variants" / pt::variant_dir_name("1.0", *bad); + EXPECT_FALSE(pt::materialize(d / "src", {(d / "bad.patch").string()}, *bad, badVariant)); + EXPECT_FALSE(fs::exists(badVariant)); +} + +TEST(Patch, AVariantNeverWritesThroughALink) { + // A store may hold a version directory, or a file in it, as a link into + // another store (a test's home linking the developer's payloads, measured: + // the first version of this patched the developer's own zlib.h through + // it). The variant's tree is copied as files, and a patch writes there. + auto d = fresh("links"); + put(d / "real/inc/m.h", "#define M 1\n"); + put(d / "real/inc/other.h", "#define O 1\n"); + std::error_code ec; + fs::create_directory_symlink(d / "real", d / "linked", ec); + if (ec) GTEST_SKIP() << "no symbolic links here: " << ec.message(); + fs::create_symlink(d / "real/inc/other.h", d / "real/inc/o-link.h", ec); + put(d / "fix.patch", "--- a/inc/m.h\n+++ b/inc/m.h\n@@ -1 +1 @@\n-#define M 1\n+#define M 2\n" + "--- a/inc/o-link.h\n+++ b/inc/o-link.h\n@@ -1 +1 @@\n-#define O 1\n+#define O 2\n"); + auto digest = pt::set_digest({(d / "fix.patch").string()}); + ASSERT_TRUE(digest); + const auto variant = d / "variants" / pt::variant_dir_name("1.0", *digest); + ASSERT_TRUE(pt::materialize(d / "linked", {(d / "fix.patch").string()}, *digest, variant)); + EXPECT_EQ(get(variant / "inc/m.h"), "#define M 2\n"); + EXPECT_EQ(get(variant / "inc/o-link.h"), "#define O 2\n"); + EXPECT_FALSE(fs::is_symlink(variant)); + EXPECT_EQ(get(d / "real/inc/m.h"), "#define M 1\n"); + EXPECT_EQ(get(d / "real/inc/other.h"), "#define O 1\n"); +} + +} // namespace