diff --git a/.agents/docs/2026-10-30-ecosystem-part2-ide-and-language-service-design.md b/.agents/docs/2026-10-30-ecosystem-part2-ide-and-language-service-design.md index 54b37fc0b..537a23273 100644 --- a/.agents/docs/2026-10-30-ecosystem-part2-ide-and-language-service-design.md +++ b/.agents/docs/2026-10-30-ecosystem-part2-ide-and-language-service-design.md @@ -5,7 +5,7 @@ status: active # 生态设计 part 2：mcpp 插件与 mcppls 的分层、入口、测试、版本感知与分发 -日期：2026-10-30（rev 9，实施）。基线：mcpp 2026.10.30.1（`13e2da04`）、mcpp-ide #3（0.2.0，`734d856` + `49ae757`）、 +日期：2026-10-31（rev 10，实施）。基线：mcpp 2026.10.30.1（`13e2da04`）、mcpp-ide #3（0.2.0，`734d856` + `49ae757`）、 mcppls #75（0.0.20，`4660eb6`）。前文：[part 1](2026-10-29-ecosystem-target-sets-packaging-and-editors-design.md)（§9 mcppls 单包、 §10 编辑器）。相关 issue：mcpp#868（锁文件键）、Sunrisepeak/mcppls-clangd#12（clangd 映像）。相关规范：mcpp SPEC-015 §4、docs/08、 docs/50；mcppls S3、S4；mcpp-ide protocol。 @@ -819,12 +819,13 @@ openkal 的接口整体提供或整体不提供（§3），`openkal.space` 的 |---|---|---|---| | M | mcpp（#869，2026.10.31.1） | 检查点 ① `features` 与 `program`；② mcpp#868；③ §17；④ §11 与 docs/50 的 §4.0；⑤ `polyglot`；⑥ C1（§25.3） | — | | M2 | mcpp（2026.10.31.2） | `mcpp test --print-launch`（`test-launch`，§7.4）；构建契约的目标侧（C2）；Windows 启动器的平台（K5 的发现）；本文 rev 8、rev 9 | M | +| M3 | mcpp（2026.10.31.3） | 有封装步骤的平台上，链接边的输出由一个函数给出，排序检查不再把封装边当作链接（C 的发现）；大型工作区的构建空间与复用（mcpp#874，[设计](2026-10-31-workspace-space-and-reuse-design.md)）；本文 rev 10 | M2 | | K1 | openkal（0.18.0） | `openkal.map`：SPEC 条款、头文件与模块、`SURFACE.txt`、一致性测试 | — | | K2 | openkal-linux、openkal-macos、openkal-windows | 实现 `openkal.map`；§25.3 定位到实现的修复 | K1 | | K3 | openkal-image | okrun 以 `openkal.map` 读入映像与 okb（§25.3 B1） | K2 | | K4 | openkal-musl | 文件 `mmap`、`munmap`、`msync` 经 `openkal.map`；§25.3 定位到 musl 的修复 | K1、K3（musl 钉住 openkal-ld） | | K5 | openkal-llvm-runtime、openkal-platform、mcpplibs.os | 钉住新版本；platform 的版本组 | K4 | -| C | mcppls-clangd（0.0.18） | clangd 的 `x86_64-openkal`、`aarch64-openkal` 映像与 `clangd.okb`（§25.3） | K5；补丁与隔离实验不依赖，先行 | +| C | mcppls-clangd（0.0.18） | clangd 的 `x86_64-openkal`、`aarch64-openkal` 映像与 `clangd.okb`（§25.3） | K5、M3；补丁与隔离实验不依赖，先行 | | L | mcppls #75（0.0.20） | §5.2 `status`；§4.6；§10 D3–D8；引擎改为 `clangd.okb`（§25.3 的验收） | C | | I | mcpp-ide #3（0.2.0） | §14 的检查点 ①–⑥；mcppls 0.0.20 单包依赖 | L 的发布（CLion 的 mcppls 依赖）；开发不依赖 | | X | mcpp-index、xim-pkgindex | openkal 家族的条目随级联逐个进入 mcpp-index（下游 CI 从已发布的索引解析，故不能合为一个 PR）；xim-pkgindex 一个 PR：okrun、mcppls、mcppls-nvim、mcpp-ide、mcpp-ide-nvim | 各自的发布 | @@ -879,3 +880,15 @@ mcpp 新建、构建、运行、`--pkg-format polyglot`；mcppls 在四种编辑 | M2 | 版本定为 2026.10.31.2：`test-launch`、构建契约的目标侧（C2 的前提）、Windows 启动器的平台 | | I | 检查点 ①–⑤ 已推送：⑤ 为规则层移入后端（LSP，协议 2）、三个编辑器的接入、F15 的降级模式及其 CI 任务；⑥ CLion 原生界面在进行 | +### 25.7 进度（rev 10） + +| 任务 | 状态 | +|---|---| +| M2 | 已合入并发布 2026.10.31.2，GitCode 已补，xim-pkgindex 的 mcpp 条目已合入 | +| K5 | mcpplibs.os 0.2.4 已发布，其 CI 的 ok 映像任务以 openkal-platform 0.19.0 构建；openkal-emscripten 0.7.0 随 openkal 0.18.0 发布（0.6.0 钉 openkal 0.17.1，os 的 emscripten 任务因此失败）；openkal-image 的 release-binaries 接受 `platform`，okrun 0.1.5 的二进制以 platform 0.19.0 构建 | +| C | B2、B3、B4 的修复与 Windows 的内存前导名（`getInMemoryPreamblePath` 在 `LLVM_ON_UNIX` 下为 POSIX 路径，Windows 上读 PCH 失败；`__OPENKAL__` 下按系统的路径风格命名）合为一个补丁。只为 ok 映像适配 LLVM 的补丁放在 `openkal/patches/`，有自己的 series 与台账，仅在构建 ok 映像时于主 series 与 overlay 之后应用（`apply.sh --openkal`）；原生引擎不带它。`clangd.okb` 的身份覆盖两个 series。映像任务在 mcpp 2026.10.31.2 上被排序检查拒绝：`prepare` 动作构建 clangd 的库，检查要求每条链接边等待它，却把发布 `bin/clangd.oke` 的封装边当作链接；M3 修复 | +| L | 引擎条目可为 ok 映像：`engines.clangd` 的 `path` 指向 `clangd.okb`、`runner` 指向 okrun，身份的平台为 `openkal`（S4-4-12）；锁文件行 `clangd = "mcppls-clangd-openkal"` 选择它；工作区平台升至 openkal-platform 0.19.0、os 0.2.4 | +| X | xim-pkgindex 的 CI 计划以引用的坐标与精确版本计算受影响的包：精确钉住的版本只在该版本条目或配方其余部分改变时成为依赖边；一次 mcpp 版本更新从 71 个包、10 个任务降为 4 个包、6 个任务 | +| CI | GitHub 于 2026-11-02 停用 macos-14，此前按计划使其任务失败（actions/runner-images#13518）。本轮各仓库的 macOS 任务改用 macos-15；构建 macOS 产物的任务写明 `MACOSX_DEPLOYMENT_TARGET=14.0`，支持下限仍为 macOS 14，由产物的 `minos` 断言。mcpp 的 ci-target-matrix 与 openkal-cross 的 macOS 任务随之改用 build.yml 的产物，不再自行构建 | +| 未覆盖 | openkal-opensbi（钉 openkal 0.16.0）与 openkal-uefi（0.14.1）落后于规范；二者为裸机实现，不在 os 与本轮的依赖中 | + diff --git a/.agents/docs/2026-10-31-workspace-space-and-reuse-design.md b/.agents/docs/2026-10-31-workspace-space-and-reuse-design.md new file mode 100644 index 000000000..3b7e571f9 --- /dev/null +++ b/.agents/docs/2026-10-31-workspace-space-and-reuse-design.md @@ -0,0 +1,303 @@ +--- +subject: build +status: active +--- + +# 大型工作区的构建空间与复用（mcpp#874） + +日期：2026-10-31。版本：mcpp 2026.10.31.3（PR #873）。前文：[part 2](2026-10-30-ecosystem-part2-ide-and-language-service-design.md) §25。 +相关：mcpp#874（mui：112 个成员，`target/` 25 GB，`mcpp test --workspace` 407 分钟构建、3.4 分钟测试）、mcpp#702、SPEC-013 §4.6。 + +## 0. 结论 + +1. 一个开发构建的字节主要是**调试信息的副本**，不是代码。mcppls 的测试程序 `test_env` 为 96.0 MiB，其中调试段 91.2 MiB、 + `.text` 0.6 MiB；mui 的 `site_test` 中调试段占 80%。每个程序静态链接其闭包，并复制闭包全部对象的 DWARF。 +2. 重复构建的原因是**键与图不稳定**，不是缺少缓存：host tool 的键取自本次命令走到的图，在 `test` 与 `build`、不同成员之间变化； + action 规则没有 `restat`，生成器原样保留输出也使下游全部失效。 +3. 本轮的做法分三类：去掉副本（调试信息、归档、放置、构建程序）、使键与命令无关（工具键、`restat`）、回收无人使用的内容 + （`uses.list`、全局缓存按最后使用回收）。每一项都不改变构建出的程序的行为。 +4. 在 mcppls（6 个成员、61 个测试程序、clang）上实测：`bin/` 由 4 914 MiB 降至 1 217 MiB（−75%），测试链接时间合计由 75 s + 降至 32 s（−58%），编译 CPU 增加约 5%（clang；gcc 在测量波动内）。 +5. 开发周期中的增长来自配置目录的数量：mcpp 升级、工具链变化使同一配置开出完整的新目录（mcppls 每次 2.9 GB）。 + 复用的最细粒度是 ninja 的边，已经存在；本轮让同一谱系的演进在同一目录中发生（§8.3），并在构建后回收 3 天未用的目录 + （§8.4）。增长的上界由"历史的长度"变为"使用中的谱系数"。 + +## 1. 实测 + +### 1.1 mui（mcpp#874，mcpp 2026.10.27.1，gcc 16.1.0，dev，一个配置） + +| 区域 | 大小 | 成因 | +|---|---:|---| +| `obj/` | 1.5 GB | 2 666 个对象 | +| `bin/` 测试程序（135 个） | 13.7 GB | 每个约 100 MB，DWARF 复制 135 次 | +| `bin/` 成员归档（15 个） | 5.3 GB | 11 433 个归档成员对应 2 666 个对象 | +| `bin/` 共享库放置（299 个） | 0.92 GB | 650 MB 为同一 `.so` 的副本 | +| `.build-mcpp/host/` | 743 MB | host 工作区内又嵌套了一份自身 | +| `.build-mcpp/out/` | 10 个哈希目录 | 7 个已无配置使用，`clean --stale` 不覆盖 | +| `~/.mcpp/build-cache/v1/tool/` | 611 MB | 17 个 `+path.` 键，从不回收 | +| `~/.mcpp/build-cache/v1/std/` | 1.8 GB | 57 份 std 模块 | + +一次 `mcpp test --workspace` 的 `.ninja_log`：6 512 条边对应 1 896 个输出，1 473 个输出被构建不止一次， +99.6 编译分钟中 69 分钟为重复构建；`runtime.o` 以同一命令哈希编译 7 次。 + +### 1.2 mcppls（本轮测量，clang 23.1.3，dev，`mcpp test --workspace --no-run`，61 个测试程序） + +| 变体 | `obj/` | `bin/` | `test_env` | 其中调试段 | 测试链接合计 | +|---|---:|---:|---:|---:|---:| +| V0 2026.10.31.2（`-g`） | 824 MiB | 4 914 MiB | 96.0 MiB | 91.2 MiB | 75.1 s | +| V1 `-gz=zlib`（编译与链接） | 514 MiB | 1 937 MiB | 35.7 MiB | 30.9 MiB | 54.4 s | +| V2 V1 + `-fdebug-types-section` | 581 MiB | 1 799 MiB | 32.8 MiB | 27.9 MiB | — | +| V4 V1 + 本地单元 `-gsplit-dwarf` | 486 MiB（`.dwo` 55） | 1 217 MiB | 20.2 MiB | 15.4 MiB | 31.8 s | +| V5 本地单元 `-gsplit-dwarf`，不压缩 | 934 MiB（`.dwo` 324） | 1 217 MiB | 20.3 MiB | 15.4 MiB | — | +| V6 仅链接时压缩 | 823 MiB | 1 937 MiB | 35.7 MiB | 30.9 MiB | 52.2 s | + +V4 中剩余的 15.4 MiB 调试段来自缓存提供的依赖对象（zstd、openkal-musl、mbedtls、mcpplibs.os、openkal-llvm-runtime、 +libarchive 等，以压缩的形式留在段内，见 §4.1 (c)）。链接时间取自 `.ninja_log`，机器同时承载其他负载，作为方向而非精确值； +编译 CPU 以单个翻译单元重复编译、取用户态加内核态时间测得（§4.1）。 + +缓存提供的依赖对象在 `obj/` 中约 55 MiB，本地编译单元约 770 MiB：缓存到构建目录的拷贝不是主要成本（§4.8）。 + +## 2. 约束 + +- **不产生错误产物。** 缓存键逐字段比较；宁可多构建一次，也不复用输入不同的产物。 +- **默认值不改变程序的行为。** 改变链接或运行方式的手段（共享闭包库、合并测试程序）不作为默认。 +- **平台差异在实现内决定。** 一项手段按目标的对象格式、链接方式与调试器约定启用，不要求 mcpp.toml 区分平台。 +- **可关闭。** 每项默认都有对应的关闭方式。 +- **可验证。** 每项改变对应一个 e2e 判据，在旧版失败、新版通过。 + +## 3. 构成分析 + +| 来源 | 机制 | 占比（mui） | +|---|---|---| +| 调试信息副本 | 每个程序链接闭包的全部对象，链接器把每个对象的 DWARF 拷入输出；C++ 模块与 `import std` 使每个翻译单元重复携带标准库的类型描述 | `bin/` 的主要部分 | +| 成员归档 | 静态库链接单元收入了计划中全部编译单元的对象 | 5.3 GB | +| 共享库放置 | 已放置的等价副本在后续构建中被判为"已就绪"，不再换成链接 | 0.65 GB | +| 构建程序 | host 工作区的子构建在自身 `target/` 下计算 host 根，嵌套一份；构建程序旁的副本是拷贝 | 0.2–0.5 GB | +| 未回收目录 | `.build-mcpp/out/`、`host/` 不在构建记录中 | 0.2–0.3 GB | +| 全局缓存 | `cache gc` 只覆盖 `pkg/`；std 与 tool 的命中不记录使用时间 | 2.4 GB | + +| 重复构建来源 | 机制 | +|---|---| +| host tool 键 | 键的依赖闭包取自本次图：工具包只经构建依赖的 `tools` 到达时其依赖不被遍历（`test --workspace` 下闭包为空），作为成员构建时完整（`build`），被测成员的 dev-dependency 也会进入；键变则工具路径变，运行它的 action 全部重跑 | +| 无 `restat` | 生成器保留原输出时，ninja 仍视输出为新，扫描、BMI 与全部导入者重编 | +| 成员分组 | `test --workspace` 已按配置分组、每组规划与构建一次（成员选择设计 S4、D1）；组内的重复来自上两项 | + +## 4. 方案 + +### 4.1 调试信息只存一份 + +**(a) 压缩调试段。** 有调试信息、目标为 ELF 时，编译与链接都加 `-gz=zlib`；平台以 `ld.lld` 直接链接的目标 +（ok 映像）在链接行加 `--compress-debug-sections=zlib`。实测： + +| 编译器 | `-g` | `-g -gz` | 对象大小 | +|---|---:|---:|---:| +| clang 23.1.3（mcppls 三个最大翻译单元，3 次） | 6.68 s | 6.66 s | 21.6 → 12.8 MiB | +| gcc 16.1.0（`import std` 与容器的 dev 翻译单元，8 次） | 3.19 s | 2.82 s | 程序 11.94 → 5.72 MiB | + +是否启用还取决于工具链的构建：没有 zlib 的 LLVM 对 `-gz` 只给警告并写出未压缩的段，其 `ld.lld` 拒绝 +`--compress-debug-sections`（实测：xim 索引中 Windows 主机的 LLVM 23.1.3，CI 的 openkal-cross windows 任务）； +不支持压缩段的 GCC 拒绝 `-gz`。因此询问一次并缓存结果：LLVM 工具链做一次真实的压缩链接——空单元以 `-g` 编译为 ELF 对象，经其驱动实际运行的 +链接器（`-print-prog-name=ld.lld`）以 `--compress-debug-sections=zlib -shared` 链接，成功与否即答案，答案按该链接器的 +路径、大小与时间保存；GCC 问其驱动（`-gz=zlib -###`）。支持时才启用。windows-2022 上的实测（临时 probe 工作流）：xim 的 +LLVM 23.1.3 不带 `ld.lld`，其驱动经 PATH 运行镜像自带的 LLVM（`C:\Program Files\LLVM\bin\ld.lld.exe`），后者不含 zlib； +该 clang 对 `-gz` 不报警告。压缩不增加可测的编译 CPU（两种编译器的差异都在 ±10% 的测量波动内），链接时间下降约 28%。gcc 的对象中调试重定位（`.rela.debug_info`）不压缩，对象缩小有限；链接后重定位消失，程序缩小 52%。gdb 与 lldb 读取压缩段。ok 映像：x86_64 映像 +8.07 MiB → 3.93 MiB，okld 封装与 okrun 运行不受影响。 + +**(b) 本地单元的调试信息留在对象旁（`split_debuginfo`，新增 profile 键）。** profile 键 `split_debuginfo = "unpacked" | "off"`， +名称与取值取自 Cargo。`"unpacked"`：在构建目录中编译的单元以 `-gsplit-dwarf` 编译，`.dwo` 留在对象旁，程序中只有骨架， +链接器不再拷贝 DWARF。默认：`dev` 与 `mcpp test` 为 `"unpacked"`，其余 profile 为 `"off"`。 + +- 只在 `trim_paths` 关闭时生效：调试器按 `DW_AT_comp_dir` 加 `DW_AT_dwo_name` 查找 `.dwo`，`trim_paths` 把构建目录映射为 + `/mcpp/build`，查找会失败。`trim_paths` 默认只在 `release` 与 `dist` 开启，而这两个 profile 默认 `"off"`。 +- `mcpp pack` 构建的 profile 默认 `"off"`，发布的产物带完整调试信息或被 strip，与现状相同。 +- 实测：clang 编译 CPU +4.5%（与压缩同时使用），mcppls 的 `bin/` 再降 37%（1 937 → 1 217 MiB），测试链接合计再降 42%；gcc 编译 CPU 差异在波动内。收益随本地单元的数量增长：单个翻译单元的程序（调试信息主要来自 std 模块与库）只再降 4%（5.72 → 5.48 MiB）。 + +**(c) 缓存提供的单元保持压缩的段内调试信息。** 骨架记录的 `DW_AT_dwo_name` 是编译时的输出路径；缓存条目在临时目录中编译后 +改名到最终位置，gcc 与 clang 都不对 `DW_AT_dwo_name` 应用 `-fdebug-prefix-map`（本轮实测）。要使骨架指向条目的最终位置， +需要逐边设置工作目录，或使用 clang 独有的 cc1 选项，代价与收益不成比例。缓存单元因此以压缩的段内 DWARF 为终点。 + +**(d) 不采用：类型单元（`-fdebug-types-section`）。** 单独使用时 `bin/` 降 7%，与 (b) 同时使用时无收益（V4 与 V2+V4 相同）。 + +**平台与工具链矩阵：** + +| 目标 | 对象格式与调试信息 | (a) 压缩 | (b) split | +|---|---|---|---| +| Linux（gcc、clang）、交叉 Linux | ELF，DWARF 拷入程序 | 编译与链接 `-gz=zlib` | 默认 `unpacked`（dev/test） | +| ok 映像（`-openkal`） | ELF，`ld.lld` 直接链接 | 编译 `-gz`，链接 `--compress-debug-sections=zlib` | 默认 `unpacked`（实测骨架可封装、可运行） | +| 裸机（freestanding） | ELF，厂商调试器可能不读压缩段与 `.dwo` | 不启用 | `off` | +| macOS（Mach-O） | 调试信息留在对象中（debug map），程序不含 DWARF | 不适用 | 不适用（已是 unpacked） | +| Windows MSVC、clang-cl | PDB，每个程序一份，类型在 PDB 内合并 | 不适用 | 不适用 | +| MinGW（PE + DWARF） | DWARF 拷入 PE；汇编器不支持 PE 的压缩段 | 不启用 | `off`（PE 上的 split 不受支持） | +| WebAssembly（emscripten） | DWARF 在 wasm 自定义段 | 不启用 | `off` | + +### 4.2 静态库只含本包对象（已实现） + +静态库链接单元在计划完成后删去其他包的编译单元对象，与 docs/12"静态归档不携带其依赖的代码"一致。构建内的程序直接链接对象， +从不读归档，链接不受影响；`object` action 的输出不是编译单元，保留。mui：5.3 GB → 不超过各成员自身对象之和。 + +`mcpp pack` 发布的归档此前也携带依赖对象（对 path 依赖相当于隐式 vendor，对索引依赖则与消费者解析的同一依赖形成重复定义）； +现在只含本包对象，依赖由打包清单记录、由消费者解析，与文档一致。在 CHANGELOG 的 Changed 中写明。 + +### 4.3 共享库放置为链接（已实现） + +`stage` 在目标已存在且内容相同时，若源为共享库且两者不是同一文件，以硬链接替换；无法链接时保留原副本。 + +### 4.4 构建程序（已实现） + +- host 根由发起链路的构建传入子构建（`BuildOverrides::host_programs_root`），内层组为同级的 `.`，不再嵌套。 +- 构建程序旁的 `build.mcpp.bin` 是 host 组二进制的硬链接，失败时拷贝。 +- 不同配置（dev、release、wasm）的 host 组本来就由（host 工具链、标准、版本）决定，已共享。 + +### 4.5 项目内回收（已实现） + +- 每个配置目录写 `uses.list`：所用的 `.build-mcpp/out/` 与 `.build-mcpp/host/`。 +- `mcpp clean --stale` 以保留下来的配置目录的 `uses.list` 之并为在用集合，回收其余（`--older-than` 窗口内的保留）； + 没有任何 `uses.list`（旧版写的目录）时不回收这两类。 +- `mcpp clean --report`：按配置给出 `obj`、BMI、`bin` 与 `.build-mcpp` 各区的大小。 +- 构建前，构建目录所在文件系统可用空间小于 2 GiB 时警告，并给出回收命令。 + +### 4.6 全局缓存（已实现） + +- `cache list/prune/gc/clean/verify` 覆盖 `tool/`；`clean --tools`。 +- path 工具写入新条目时删除同一源树的旧 `+path.` 条目。 +- **最后使用时间。** 条目写入后文件不再改变，文件时间只记录写入；`noatime`/`relatime` 挂载不记录读取。因此由 mcpp + 陈述使用：构建使用一个条目时，把条目记录文件（pkg 与 tool 的 `entry.json`、std 的 `std-module.json`）的时间设为 + 当前时间（`mcpp::home::mark_used`），12 小时内至多一次，一次使用的代价是一次 `stat`。pkg 命中此前每次解析并重写 + `entry.json`，现在不再写内容。回收读取记录文件时间与 `accessed` 字段中较晚者。 +- **重放构建也计为使用。** 快速路径不经规划、不访问缓存；每个配置目录的 `uses.list` 记录它用过的条目 + （`cache <记录文件>`），构建后的回收（§8.4，每项目每天一次）把它们记为已使用。 +- **保留期。** 全局缓存由多个项目共用，默认 30 天（config.toml `[cache] unused_days`）；项目内构建目录默认 3 天 + （`[build] keep_days`）。`mcpp self config --cache-unused-days --keep-days ` 写入，`mcpp self config` 显示； + 0 关闭。 +- 构建成功后至多每天一次，回收 `[cache] unused_days` 内未使用的 pkg、tool 与 std 条目。 + +### 4.7 重复构建（已实现） + +- host tool 键的依赖闭包由工具包自身的 manifest 传递计算（与其子构建的解析一致）：path 依赖为版本与源树 stamp，git 依赖为 + 声明的 rev，索引依赖为声明的版本要求；dev-dependency 不在其中。键与命令、成员无关。 +- action 规则加 `restat = 1`：生成器保留原输出时，重建止于生成器。 + +### 4.8 评估后不采用 + +| 手段 | 不采用的依据 | +|---|---| +| 测试链接共享闭包库 | 代码只占测试程序的 0.6 MiB（`test_env`），(b) 之后剩余主要是缓存单元的调试段；收益集中在链接时间（V4 已为平均 0.52 s/个）。代价：`-fvisibility=hidden` 的包无法导出测试所需符号；MSVC 的数据符号需要 `dllimport`；ok 映像与裸机只有静态链接；静态初始化与单例的行为改变。 | +| 构建目录链接到缓存对象 | 缓存提供的对象只占 `obj/` 的约 7%；构建目录中的对象若是缓存的硬链接，此后本地编译写同一路径时会就地改写缓存条目。 | +| 项目级内容寻址对象库 | 同一配置内 ninja 已按命令哈希避免重复；跨配置的可复用部分以 dev/test 为主，而二者本来共用一个配置目录。收益不足以抵消 staging、`restat` 与 Windows 文件锁的改动面。 | +| 合并测试程序 | 改变测试的进程隔离，属于测试框架的选择，不属于构建系统。 | + +## 5. 收益汇总 + +| 项 | mui（推算，依据 §1.1） | mcppls（实测） | +|---|---|---| +| 测试程序 | 13.7 GB → 约 3.4 GB（按 mcppls 的 V4/V0 之比 0.25 推算） | 4 914 → 1 217 MiB（−75%） | +| 成员归档 | 5.3 GB → 不超过各成员自身对象之和（全部对象 1.5 GB，压缩前） | — | +| 共享库放置 | 0.92 → 0.27 GB | — | +| host 工作区 | 嵌套的 153 MB 与重复的构建程序副本 | — | +| 重复编译 | 69/99.6 编译分钟 → 0（e2e 997、998 G） | — | +| 全局缓存 | 按 30 天未使用回收 | — | +| 编译 CPU | gcc 在 ±10% 波动内 | clang +4.5% | +| 测试链接 | — | 75 → 32 s | + +## 6. 兼容与升级 + +- 编译行加入 `-gz=zlib` 与 `-gsplit-dwarf`，指纹变化，升级后每个配置完整重建一次；全局包缓存与 std 缓存各生成一份新条目， + 旧条目由 §4.6 的回收在 30 天后移除。 +- 静态库内容变化（§4.2）：以 `bin/lib.a` 在 mcpp 之外链接、且依赖其中携带依赖代码的用法需要同时链接依赖的归档。 +- `uses.list` 由本版写入；旧目录在下一次构建后获得。 +- `[cache] unused_days`、`split_debuginfo` 有关闭方式；`-gz=none` 写在包的 flags 中可关闭压缩。 + +## 7. 验证 + +| e2e | 判据 | +|---|---| +| 996 | 带 `prepare` action 的 ok 映像构建并运行；链接边等待 action | +| 997 | `test --workspace` 与 `build` 之后工具只有一个键，含其依赖、不含 dev-dependency；第二次测试不重建工具 | +| 998 A–H | 归档只含本包对象；ELF 调试段压缩；共享库放置为链接；`clean --stale` 回收无人使用的 `out/`；`clean --report`；path 工具编辑后旧条目被取代、`cache list` 显示 tool；生成器保留输出时不重编导入者；嵌套构建程序为同级组、副本为链接 | +| 999 | dev 构建的本地单元有 `.dwo`、索引包单元没有，程序调试段压缩，gdb 经骨架读取源码；`--release` 与 `split_debuginfo = "off"` 无 `.dwo`；未知取值被拒绝 | +| 996 C | ok 映像的调试段由其 `ld.lld` 链接压缩，单元有 `.dwo`，映像封装并运行 | + +每条在旧版失败、新版通过；CI 覆盖 Linux（gcc、clang）、macOS、Windows（MSVC、MinGW）、ok 映像与交叉目标。 + +## 8. 开发周期中的累积与划分的粒度 + +§1–§7 处理一次构建的大小。开发周期中 `target/` 的增长另有来源：配置目录的数量随时间增加。Cargo 的 `target/` 有同样的问题 +（依赖的每个版本与 feature 组合各占一份，旧的不回收，需要 cargo-sweep 一类的外部工具）。 + +### 8.1 来源（代码与实测） + +配置目录名 `target///` 的指纹有 11 个部分（`mcpp build --print-fingerprint`）：编译器名、版本、驱动身份、目标、 +标准库与其版本、语言标准、配置级编译选项、mcpp 版本、`mcpp.lock` 哈希、std BMI、运行时契约。其中： + +- `mcpp.lock` 哈希恒为空（`dependencyLockHash = ""`），依赖版本变化不开新目录。 +- 编译选项一项是 `canonical_configuration_flags`：只含配置级的值（profile 键、方言选项、部署目标、c-abi 实现、调度、加速器）。 + 包自身的 flags、feature 与依赖是节点的命令，在同一目录中由 ninja 增量重建。 + +因此随时间开出新目录的输入是：**mcpp 版本**（每次升级）、编译器版本与驱动身份、标准库版本、运行时契约、std BMI，以及配置级 +选项的编辑。每次都是完整的新目录与一次全量重建，旧目录保留到手动 `clean --stale`。实测：mcppls 的同一配置因 mcpp 版本不同 +有两个目录，共 4.0 GB；mui 的 `target/` 为 20 GB、3 个目录。 + +被删除或改名的源文件留下的对象与 BMI 与删除的代码量成比例，不是主要来源。不采用 ninja `cleandead` 式的"图外输出即删除"： +`mcpp build` 与 `mcpp test` 交替写同一目录的 build.ninja，按当前图删除会删去另一种图的输出。 + +### 8.2 划分的粒度 + +| 粒度 | 机制 | 复用的范围 | 结论 | +|---|---|---|---| +| 目录（现状） | 指纹的任一部分变化即新目录 | 同一指纹 | 演进输入使整份构建重复 | +| 边 | ninja 按每条边的命令哈希（`.ninja_log`）与输入时间决定重建 | 同一目录内命令未变的边 | 已是最细；前提是演进在同一目录内发生 | +| 包（索引包） | 全局缓存，键逐字段比较 | 所有项目、所有配置 | 已有 | +| 包（本地包）跨目录共享 | 项目级按包键存储，多个配置目录引用 | 不同配置中命令相同的包 | 不采用：各配置目录的 `.ninja_log` 互不知道对方对共享输出的改写，一个配置会把另一个配置写的输出当作自己的结果（命令哈希相同才安全，而键必须覆盖依赖的全部输出才能保证）；共享一个日志则把同一三元组的构建串行化。收益只在 feature 或成员选择不同、而 profile 与工具链相同的配置之间 | +| 单元变体 | 对象路径带命令哈希，多个变体并存 | 来回切换的单元 | 不采用：每次演进都留下变体，增长只能靠回收约束；对象路径不再稳定，所有按路径读取对象的工具（调试、编译数据库的消费者、测试）随之改变 | + +结论：复用的最细粒度是边，ninja 已经实现；缺的是让演进发生在同一目录中。 + +### 8.3 谱系：一个配置的目录延续（已实现） + +指纹分为两部分： + +- **谱系**：编译器族、目标、标准库族、语言标准、配置级选项。开发者在其间切换、需要并存的值。工作空间中配置不同的组 + （`root_position_key` 不同）谱系不同，各有目录，不会互相覆盖。 +- **演进**：编译器版本与驱动身份、标准库版本、mcpp 版本、std BMI、运行时契约。随时间单向变化的值。 + +目录名仍是完整指纹，构建记录、快速路径、`clean --stale` 的语义不变。每个配置目录记录 `lineage` 与 `toolchain`。规划到一个 +尚无 build.ninja 的目录时，在同一三元组下找谱系相同、最近使用（build.ninja 最新）的目录： + +- 工具链相同（只有 mcpp 版本、std BMI 或运行时契约变化）：其条目逐个移入新目录（包括 `.ninja_log` 与 `.ninja_deps`）， + 原目录删除。ninja 以命令哈希判断重建：编译、扫描与链接命令不含 mcpp 的路径，命令不变的边全部复用；引用 `$mcpp` 的 + `stage`、`dyndep` 边重新执行，`stage` 有 `restat`，内容相同时不写、不向下传播。mcppls 的构建图：222 条编译与扫描边与 19 条 + 链接边复用，111 条 dyndep 边与 2 096 条 stage 边重新执行。 +- 工具链不同：其中的对象与 BMI 不可复用（驱动身份变化时命令可能相同而产物格式不同），删除它，在一个目录中全量重建。 +- 另一个构建持有它的锁：不动它，本次构建另开目录（由 §8.4 回收）。 +- 逐个移入而不是重命名目录：快速路径在发现记录的目录没有 build.ninja 之前已经取得该目录的锁（因此目录已存在）， + 且 Windows 不能重命名有句柄打开的目录。 + +一致性：`build`、`run`、`test`、`pack` 经同一规划入口，规则相同。 + +### 8.4 构建后的回收（已实现） + +`mcpp build` 与 `mcpp test` 成功后，每个项目每天至多一次，执行 `clean --stale` 的同一逻辑：构建记录之外、且 `keep_days` +（config.toml `[build]`，默认 3，0 关闭）内未写过的配置目录，以及无配置在用的 `.build-mcpp/out`、`host/` 目录，被移除，并以 +一行报告。§8.3 之后，这类目录只来自不再使用的谱系（不再构建的目标、profile、工具链族）。 + +### 8.5 多视角 + +| 视角 | 说明 | +|---|---| +| 架构 | 指纹与目录名的语义不变；新增的只是目录之间的延续关系（`lineage`、`toolchain` 两个文件）与回收时机 | +| 稳定性 | 复用的正确性由 ninja 的命令哈希保证；工具链变化时不复用；移入失败时退回到新目录，§8.4 回收 | +| 用户体验 | 无需操作；移除旧工具链目录时报告 `Replaced`，回收时报告 `Swept` | +| 兼容 | 本版之前写的目录没有 `lineage`，不被延续；升级到本版后第一次构建全量一次，旧目录在 3 天后回收（若仍在构建记录中则保留，可 `clean --stale`） | +| 跨平台 | 逐个条目移动，避开 Windows 对打开目录的重命名限制；锁用非阻塞尝试，持有者不受影响 | +| 代价 | 同一谱系不能并存两个演进版本（例如对比两个 mcpp 版本的构建）；需要时用不同的 profile 名 | + +### 8.6 收益 + +- 一个配置在整个开发周期中只占一个目录：增长上界由"历史的长度"变为"使用中的谱系数"。 +- mcpp 升级由全量重建变为只重跑 `dyndep` 与 `stage` 边（编译选项不变时）；mcppls 的单个配置目录为 2.9 GB，每次升级不再新增。 +- 不再使用的谱系在 3 天后自动回收。 + +### 8.7 验证 + +e2e 1000：同一谱系的前驱被延续、不重新编译；工具链不同的前驱被移除并报告；其他谱系的目录保留；构建后回收 `keep_days` +之外的目录，`keep_days = 0` 时保留。 diff --git a/.agents/docs/README.md b/.agents/docs/README.md index 3772791ac..6707f6a4d 100644 --- a/.agents/docs/README.md +++ b/.agents/docs/README.md @@ -18,7 +18,7 @@ superseded_by: 2026-09-07-....md # when status is superseded --- ``` -343 records. +344 records. ## By subject @@ -26,6 +26,7 @@ Records that declare one. Everything else is listed by date below. ### build +- [大型工作区的构建空间与复用（mcpp#874）](2026-10-31-workspace-space-and-reuse-design.md) — active - [依赖 image 的资源归属、模块扫描边界与锁文件稳定写入](2026-10-21-resource-ownership-module-scan-and-lock-write-design.md) — active ### build-program @@ -155,6 +156,7 @@ Records that declare one. Everything else is listed by date below. ### 2026-10 +- [大型工作区的构建空间与复用（mcpp#874）](2026-10-31-workspace-space-and-reuse-design.md) — active - [生态设计 part 2：mcpp 插件与 mcppls 的分层、入口、测试、版本感知与分发](2026-10-30-ecosystem-part2-ide-and-language-service-design.md) — active - [另一架构的 ok 映像的运行、测试与打包，以及 ok 映像与 Web 的关系（2026.10.29.1）](2026-10-29-openkal-run-test-pack-design.md) — active - [生态设计：目标集合与腿、打包模型 rev、运行覆盖、不回落、外部构建接入、mcppls 单包与编辑器](2026-10-29-ecosystem-target-sets-packaging-and-editors-design.md) — active diff --git a/.github/workflows/ci-arm64-published-cn.yml b/.github/workflows/ci-arm64-published-cn.yml index f0fdfa5b3..1a6ba488c 100644 --- a/.github/workflows/ci-arm64-published-cn.yml +++ b/.github/workflows/ci-arm64-published-cn.yml @@ -71,7 +71,7 @@ jobs: matrix: include: - {system: linux, runner: ubuntu-24.04, file: linux} - - {system: macos, runner: macos-14, file: macos} + - {system: macos, runner: macos-15, file: macos} - {system: windows, runner: windows-2022, file: windows.exe} runs-on: ${{ matrix.runner }} timeout-minutes: 10 diff --git a/.github/workflows/ci-fresh-install.yml b/.github/workflows/ci-fresh-install.yml index eaffb49dc..97bccff54 100644 --- a/.github/workflows/ci-fresh-install.yml +++ b/.github/workflows/ci-fresh-install.yml @@ -425,10 +425,10 @@ jobs: if: ${{ github.event_name != 'workflow_run' || github.event.workflow_run.conclusion == 'success' }} # Two images, the two ends of the supported range. # - # macos-14: the support floor (mcpp ≥0.0.50 / xlings ≥0.4.50 ship - # minos=14.0 static-libc++ binaries). A fresh install passing here - # is the continuous proof of the macOS 14 floor — and of host-tool - # independence (this image has no sha256sum; macos-15 does). + # macos-15: the oldest image GitHub serves (macos-14 was retired on + # 2026-11-02). The support floor is macOS 14: the released binaries carry + # minos=14.0 with a static libc++ (release.yml asserts it where they are + # built), and the install below asserts it of what it installed. # # xcode-27: macOS 27, the newest release, served under the preview label # `xcode-27` (actions/runner-images#14404; there is no `macos-27` label). @@ -439,7 +439,7 @@ jobs: fail-fast: false matrix: include: - - image: macos-14 + - image: macos-15 - image: xcode-27 runs-on: ${{ matrix.image }} # While the line was on LLVM 22.1.8 the xcode-27 leg was known red (#669, @@ -490,6 +490,12 @@ jobs: which mcpp cat $HOME/.xlings/.xlings.json + - name: The installed mcpp declares the macOS 14 floor + run: | + bin=$(mcpp self env --format json | python3 -c 'import json,sys; print(json.load(sys.stdin)["data"]["program"])') + otool -l "$bin" | grep -A4 LC_BUILD_VERSION | grep -q "minos 14.0" \ + || { otool -l "$bin" | grep -A4 LC_BUILD_VERSION; echo "::error::$bin does not declare minos 14.0"; exit 1; } + - name: "LLVM: mcpp new → run" run: | cd "$(mktemp -d)" @@ -551,13 +557,13 @@ jobs: env: MCPP_PIN: ${{ needs.resolve-version.outputs.version }} # Same two images as macos-fresh: the formula declares `depends_on macos: - # :sonoma` + arm64, macos-14 is the oldest image satisfying it, and - # xcode-27 is macOS 27, the newest release. + # :sonoma` + arm64, macos-15 is the oldest image GitHub serves that + # satisfies it, and xcode-27 is macOS 27, the newest release. strategy: fail-fast: false matrix: include: - - image: macos-14 + - image: macos-15 - image: xcode-27 runs-on: ${{ matrix.image }} # While the line was on LLVM 22.1.8 the xcode-27 leg was known red (#669, diff --git a/.github/workflows/ci-target-matrix.yml b/.github/workflows/ci-target-matrix.yml index 1a8bbd891..b6bdcb4bc 100644 --- a/.github/workflows/ci-target-matrix.yml +++ b/.github/workflows/ci-target-matrix.yml @@ -34,7 +34,7 @@ jobs: # `linux` key would have them overwrite each other in expected.tsv. - { host: linux-x86_64, runner: ubuntu-24.04 } - { host: linux-aarch64, runner: ubuntu-24.04-arm } - - { host: macos-arm64, runner: macos-14 } + - { host: macos-arm64, runner: macos-15 } - { host: windows-x86_64, runner: windows-2022 } defaults: run: @@ -44,36 +44,16 @@ jobs: - uses: ./.github/actions/bootstrap-mcpp # This commit's mcpp: the one build.yml produced (use-built-mcpp, rule R1 - # of the 2026-10-02 CI record), except on the macOS leg. That leg runs on - # macos-14 and the macOS artifact is built on macos-15; whether it runs on - # macos-14 is not measured (the record's Part VII), so the leg builds its - # own until it is. - - if: matrix.host != 'macos-arm64' - uses: ./.github/actions/use-built-mcpp + # of the 2026-10-02 CI record). Every leg runs on the image its artifact + # was built on; the macOS leg moved to macos-15 when GitHub retired + # macos-14 (2026-11-02), and with it the reason it built its own. + - uses: ./.github/actions/use-built-mcpp with: host: ${{ matrix.host }} - name: Name this commit's mcpp MCPP_UNDER_TEST - if: matrix.host != 'macos-arm64' run: echo "MCPP_UNDER_TEST=$MCPP_FRESH" >> "$GITHUB_ENV" - - name: Build the mcpp in this pull request - if: matrix.host == 'macos-arm64' # ci-lint: allow-r1: the macOS leg builds its own mcpp until the artifact is measured on macos-14 - run: | - set -euo pipefail - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$XLINGS_BIN" config --mirror GLOBAL 2>/dev/null || true - "$MCPP" self config --mirror GLOBAL 2>/dev/null || true - "$MCPP" build --dev - # 两种拼写,且按 mtime 取最新 —— target/ 是缓存恢复的,`head -1` - # 会挑到上一次推送留下的二进制,版本号一样而代码是旧的。 - BUILT=$(find target -type f \( -name 'mcpp' -o -name 'mcpp.exe' \) \ - -newer mcpp.toml | head -1) - [ -n "$BUILT" ] || { echo "::error::mcpp did not build"; exit 1; } - BUILT=$(cd "$(dirname "$BUILT")" && pwd)/$(basename "$BUILT") - echo "MCPP_UNDER_TEST=$BUILT" >> "$GITHUB_ENV" - "$BUILT" --version - - name: The invariants run: | set -euo pipefail @@ -249,7 +229,7 @@ jobs: # `linux` key would have them overwrite each other in expected.tsv. - { host: linux-x86_64, runner: ubuntu-24.04 } - { host: linux-aarch64, runner: ubuntu-24.04-arm } - - { host: macos-arm64, runner: macos-14 } + - { host: macos-arm64, runner: macos-15 } - { host: windows-x86_64, runner: windows-2022 } defaults: run: @@ -259,39 +239,16 @@ jobs: - uses: ./.github/actions/bootstrap-mcpp # This commit's mcpp: the one build.yml produced (use-built-mcpp, rule R1 - # of the 2026-10-02 CI record), except on the macOS leg. That leg runs on - # macos-14 and the macOS artifact is built on macos-15; whether it runs on - # macos-14 is not measured (the record's Part VII), so the leg builds its - # own until it is. - - if: matrix.host != 'macos-arm64' - uses: ./.github/actions/use-built-mcpp + # of the 2026-10-02 CI record). Every leg runs on the image its artifact + # was built on; the macOS leg moved to macos-15 when GitHub retired + # macos-14 (2026-11-02), and with it the reason it built its own. + - uses: ./.github/actions/use-built-mcpp with: host: ${{ matrix.host }} - name: Name this commit's mcpp MCPP_UNDER_TEST - if: matrix.host != 'macos-arm64' run: echo "MCPP_UNDER_TEST=$MCPP_FRESH" >> "$GITHUB_ENV" - - name: Build the mcpp in this pull request - if: matrix.host == 'macos-arm64' # ci-lint: allow-r1: the macOS leg builds its own mcpp until the artifact is measured on macos-14 - run: | - set -euo pipefail - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$MCPP" self config --mirror GLOBAL 2>/dev/null || true - "$MCPP" build --dev - BUILT=$(find target -type f \( -name 'mcpp' -o -name 'mcpp.exe' \) \ - -newer mcpp.toml | head -1) - [ -n "$BUILT" ] || { echo "::error::mcpp did not build"; exit 1; } - echo "MCPP_UNDER_TEST=$(cd "$(dirname "$BUILT")" && pwd)/$(basename "$BUILT")" >> "$GITHUB_ENV" - - # 格数不能是缓存状态的函数。 - # - # 实测:同一台 ubuntu-24.04,一轮装了 gcc+llvm(扫 40 格),下一轮只有 gcc - # (扫 20 格)。`expected.tsv` 声明的是前者,于是后者会把所有 llvm 行报成 - # 「期望表说有而扫描没跑到」—— 而那句报错是对的,问题在于覆盖面**漂移**了。 - # - # 矩阵要声明它扫哪些工具链,并把它们装上。装不上就红在这里,而不是 - # 变成一屏「没跑到」。 - name: Install the toolchains this matrix declares run: | set -uo pipefail diff --git a/.github/workflows/openkal-cross.yml b/.github/workflows/openkal-cross.yml index 827fb43c8..06c015660 100644 --- a/.github/workflows/openkal-cross.yml +++ b/.github/workflows/openkal-cross.yml @@ -177,7 +177,7 @@ jobs: include: - { host: linux, runner: ubuntu-24.04, artifact: linux-x86_64 } - { host: linux-aarch64, runner: ubuntu-24.04-arm, artifact: linux-aarch64 } - - { host: macos, runner: macos-14, artifact: macos-aarch64 } + - { host: macos, runner: macos-15, artifact: macos-arm64 } - { host: windows, runner: windows-2022, artifact: windows-x86_64 } defaults: run: @@ -204,83 +204,19 @@ jobs: # was wrong within a day of being written. - uses: ./.github/actions/bootstrap-mcpp - # THE mcpp UNDER TEST. Everything after this uses the binary this step - # produces; the bootstrapped one above is only what compiles it. + # THE mcpp UNDER TEST. Everything after this uses this commit's binary; + # the bootstrapped one above only restores the sandbox. # This commit's mcpp: the one build.yml produced (use-built-mcpp, rule R1 - # of the 2026-10-02 CI record), except on the macOS leg. That leg runs on - # macos-14 and the macOS artifact is built on macos-15; whether it runs on - # macos-14 is not measured (the record's Part VII), so the leg builds its - # own until it is. - - if: matrix.host != 'macos' - uses: ./.github/actions/use-built-mcpp + # of the 2026-10-02 CI record). Every leg runs on the image its artifact + # was built on; the macOS leg moved to macos-15 when GitHub retired + # macos-14 (2026-11-02), and with it the reason it built its own. + - uses: ./.github/actions/use-built-mcpp with: host: ${{ matrix.artifact }} - name: Name this commit's mcpp MCPP_UNDER_TEST - if: matrix.host != 'macos' run: echo "MCPP_UNDER_TEST=$MCPP_FRESH" >> "$GITHUB_ENV" - - name: Build the mcpp in this pull request - if: matrix.host == 'macos' # ci-lint: allow-r1: the macOS leg builds its own mcpp until the artifact is measured on macos-14 - run: | - set -euo pipefail - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$XLINGS_BIN" config --mirror GLOBAL 2>/dev/null || true - "$MCPP" self config --mirror GLOBAL 2>/dev/null || true - # `--dev` and not `--release`. What is under test is a set of - # decisions about compile and link flags; an optimisation level - # changes none of them and a release self-build is most of the budget - # of a job on a two-core runner. - # REMOVE ANY mcpp ALREADY UNDER `target/` FIRST, so that after the - # build there is exactly ONE and it is the one this step produced. - # - # `target/` is a RESTORED CACHE whose key hashes `src/**`. A source - # change misses the key, `restore-keys` hands back the nearest older - # tree anyway — that is the point of the layered restore — and the - # build then writes its output into a NEW `//` - # directory beside the ones already there. `find … | head -1` returns - # whichever directory the walk reaches first, which is not the newest - # and is not this build's. - # - # MEASURED, LOCALLY AND ON CI. On this developer's own tree the - # command picks `007bc0f2b78f7fa5` while the build just wrote - # `148448dec6c4a7a7`. On CI it picked a stale binary whose ELF - # interpreter names a glibc payload the runner no longer has: - # - # …/bin/mcpp: cannot execute: required file not found (exit 127) - # - # — which reads as a broken build of THIS commit and is nothing of the - # kind. Two runs, same stale fingerprint, after seven green ones: the - # trigger is simply a source change large enough to leave two - # directories in the restored tree. - # - # mtime CANNOT BE THE CRITERION HERE. A cache restore writes every - # file at extraction time, so "newest" is meaningless on exactly the - # tree where this goes wrong — `ls -t | head -1` has picked a stale - # fingerprint in this repository before. Absence can't be faked: - # delete them, and whatever exists afterwards was linked just now. - # - # The objects and BMIs stay cached — this costs one link. - find target -type f \( -name 'mcpp' -o -name 'mcpp.exe' \) -delete 2>/dev/null || true - "$MCPP" build --dev - # EXACTLY ONE, not "at least one". Two would mean the delete above - # missed a tree and the ambiguity this step exists to remove is back. - # - # NO `mapfile`. This job also runs on macos, whose `/bin/bash` is - # 3.2 and has no such builtin — a bashism here would fail on one row - # of the matrix for a reason unrelated to what the row tests. - find target -type f \( -name 'mcpp' -o -name 'mcpp.exe' \) > /tmp/mcpp-built.txt - COUNT=$(wc -l < /tmp/mcpp-built.txt | tr -d ' ') - if [ "$COUNT" -ne 1 ]; then - echo "::error::expected exactly one built mcpp, found $COUNT" - sed 's/^/ /' /tmp/mcpp-built.txt - exit 1 - fi - BUILT=$(cat /tmp/mcpp-built.txt) - BUILT=$(cd "$(dirname "$BUILT")" && pwd)/$(basename "$BUILT") - echo "MCPP_UNDER_TEST=$BUILT" >> "$GITHUB_ENV" - "$BUILT" --version - - name: Select the toolchain the openkal packages ask for run: | set -euo pipefail @@ -530,7 +466,7 @@ jobs: matrix: include: - { system: linux, runner: ubuntu-24.04, file: linux } - - { system: macos, runner: macos-14, file: macos } + - { system: macos, runner: macos-15, file: macos } - { system: windows, runner: windows-2022, file: windows.exe } defaults: run: diff --git a/.github/workflows/pypi-publish.yml b/.github/workflows/pypi-publish.yml index 6d211ed98..5c27cde87 100644 --- a/.github/workflows/pypi-publish.yml +++ b/.github/workflows/pypi-publish.yml @@ -140,7 +140,7 @@ jobs: strategy: fail-fast: false matrix: - os: [ubuntu-24.04, ubuntu-24.04-arm, macos-14, windows-latest] + os: [ubuntu-24.04, ubuntu-24.04-arm, macos-15, windows-latest] runs-on: ${{ matrix.os }} timeout-minutes: 20 steps: diff --git a/CHANGELOG.md b/CHANGELOG.md index 65b623740..c5ec3b53d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,104 @@ > Each `## []` section is that release's notes. Entries are written in English > from 2026.9.28.3 on; earlier entries remain as written. +## [2026.10.31.3] - 2026-10-31 + +A large workspace keeps its bytes once and builds each output once (mcpp#874; +design 2026-10-31, workspace space and reuse). Measured on mcppls (6 members, +61 test programs, clang): `bin/` 4 914 MiB -> 1 217 MiB, the tests' links +75 s -> 32 s. + +### Added + +- **`split_debuginfo` under `[profile.]`** (`"unpacked"` | `"off"`, + docs/04 §2.9). `"unpacked"`, the default of `dev` and `debug`, leaves the + debug information of the units compiled in the build directory beside their + objects (`-gsplit-dwarf`) on an ELF target with a loader, ok images included; + each program holds a skeleton that names it, and a test no longer carries a + copy of its closure's debug information. An index package's units keep it in + the object, since their objects enter the build cache. It applies with + `trim_paths` off. e2e 999. +- **`mcpp clean --report`**: the bytes of each configuration's objects, BMIs and + products and of each `.build-mcpp` area. +- **`mcpp cache clean --tools`** and host tool entries in `cache list`, `prune`, + `gc` and `verify`. +- **Collection by last use after a build**: at most once a day, the build + cache's package, host tool and `std` entries unused for `[cache] unused_days` + (config.toml; default 30, `0` keeps all) are removed. A use sets the time of + the entry's record file, at most once in 12 hours (an entry's files never + change, and `noatime` mounts keep no reads); a package hit no longer rewrites + its `entry.json`. A build that replays its graph without planning marks the + entries its configuration names in `uses.list` through the sweep after it. +- **`mcpp self config --cache-unused-days --keep-days `** writes the two + retentions into config.toml; `mcpp self config` shows them (docs/25). +- **A warning before a build** when the file system it writes to has less than + 2 GiB free. +- **The sweep after a build**: after `mcpp build` or `mcpp test` succeeds, at + most once a day, the build directories of the project that no configuration + used for `[build] keep_days` days (config.toml; default 3, `0` keeps all) are + removed, with the generated-source and build-program directories none uses. + +### Changed + +- **Debug information is compressed on ELF** (`-gz=zlib` on the compile and the + link; `--compress-debug-sections=zlib` on an ok image's `ld.lld` link): a test + program of mcppls 96.0 -> 35.7 MiB, with no measurable cost in compile time. + Only with a toolchain whose link reads them: an LLVM toolchain is asked by + one real link of an empty unit through the `ld.lld` its driver runs + (`-print-prog-name=ld.lld`) with `--compress-debug-sections=zlib`, a GCC by + its driver (`-gz=zlib -###`), once, and the answer kept under the probe cache. + Measured on windows-2022: the xim LLVM 23.1.3 ships no `ld.lld`, its driver + runs the image's own LLVM from PATH, and that one has no zlib. + Mach-O, PDB, MinGW, WebAssembly and freestanding targets are unchanged; + `-gz=none` in a package's flags turns it off. +- **A static library holds its own package's objects** (docs/12): a member's + `lib.a` held the objects of its whole closure (mui: 11 433 archive + members for 2 666 objects, 5.3 GB). Programs of the build link objects and are + unaffected; a consumer outside mcpp links each dependency's archive. +- **A configuration keeps one build directory over time** (design 2026-10-31 + §8). A new fingerprint of a lineage (compiler family, target, standard + library family, standard, configuration flags) takes over the directory the + lineage used last, and ninja rebuilds the edges whose commands changed: an + mcpp upgrade that leaves the flags as they were compiles no unit again, where + it opened a full new directory and rebuilt everything (mcppls: 2.9 GB per + configuration per upgrade). A predecessor of another compiler build is + removed. e2e 1000. +- **`mcpp clean --stale` also removes `.build-mcpp/out/` and + `.build-mcpp/host/`** that no kept configuration names in its + `uses.list`. + +### Fixed + +- **A host tool has one store key whatever the command and the member.** Its + dependency closure is read from the tool package's manifests, as its build + resolves them; it was read from the graph the command had walked -- empty + under `mcpp test --workspace`, complete under `mcpp build`, and holding the + dev-dependencies of the member under test -- so the tool's path changed and + every action that ran it ran again (69 of 99.6 compile-minutes repeated in + mui's run). e2e 997. +- **An action that leaves its output as it was rebuilds nothing below it** + (`restat = 1` on action rules). e2e 998 G. +- **A shared library staged as an equal copy is placed again by a link**, and a + build program beside its record is a link to its group's binary. +- **The build programs' graph builds its own build programs beside its group** + (`.`), not in a copy of the host workspace under its `target/`. +- **`mcpp clean --stale` keeps a rooted workspace's own build.** It removed the + members' build directories from before the workspace was the unit of build, + and a workspace that lists itself (`members = ["."]`) had its own + configuration directory removed as one of them. e2e 1000 F. +- **A path host tool's superseded entries are removed** when its tree changes. +- e2e 998 covers these and the archive, the placement and the collection. + +- **An ok image whose graph declares a `prepare` action builds** (SPEC-013 + §4.6, mcpp#702). Every link edge waits for the phony of each package with a + `prepare` action, and the ordering guard checks that it does. Under a platform + with seal steps the link writes beneath `obj/platform/` and a `platform_seal` + edge publishes the artifact's name; the guard took that edge for the link and + refused every such build as an emitter defect (mcppls-clangd's `clangd.okb`, + whose libraries a `prepare` action builds). The emitter and the guard now name + the link edge's output by one function. e2e 996 builds and runs such an image + and finds the link edge waiting for the action. + ## [2026.10.31.2] - 2026-10-31 ### Added diff --git a/docs/01-getting-started.md b/docs/01-getting-started.md index cdf0c62d0..c70fc9e2f 100644 --- a/docs/01-getting-started.md +++ b/docs/01-getting-started.md @@ -212,6 +212,7 @@ mcpp build # incremental build mcpp clean # clean target/ mcpp clean --stale # drop only target/// dirs no build still uses # (--dry-run lists and deletes nothing; --older-than 3d keeps newer unrecorded ones) +mcpp clean --report # where target/ holds its bytes, per configuration and area mcpp test # compile and run tests/**/*.cpp — one binary per file, # framework-agnostic (bare main, or gtest via [dev-dependencies]) mcpp test # only tests whose name contains diff --git a/docs/04-mcpp-toml.md b/docs/04-mcpp-toml.md index 33208322f..de5c01984 100644 --- a/docs/04-mcpp-toml.md +++ b/docs/04-mcpp-toml.md @@ -1621,6 +1621,7 @@ lto = true # -flto; /GL + /LTCG on cl.exe (note: some packaged gc strip = true # -s at link time ndebug = true # -DNDEBUG for every unit of the graph (default: on for release/dist) trim_paths = true # -ffile-prefix-map per unit (default: on for release/dist) +split_debuginfo = "off" # "unpacked": debug information beside the objects (default: dev/debug) # passthrough escape hatch (fixed keys, open values): cflags = ["-fno-plt"] cxxflags = ["-fno-plt"] @@ -1659,11 +1660,39 @@ ldflags = [] map the paths back for a debugger. cl.exe has no documented equivalent and the build says so once. Write `ndebug = false` or `trim_paths = false` under the profile to keep the previous artifact; the first shipping build after the upgrade says this once. +- **Debug information (2026.10.31.3+).** On an ELF target with debug information + (Linux, a cross Linux target, an ok image), the compile and the link compress + it (`-gz=zlib`; an ok image's `ld.lld` link, `--compress-debug-sections=zlib`) + when the toolchain's link reads them (asked once, by one compressed link + through the `ld.lld` an LLVM driver runs, or of a GCC's driver, and kept: a + Windows host whose `ld.lld` has no zlib stays uncompressed). + `split_debuginfo = "unpacked"`, the default of `dev` and `debug`, also leaves + the debug information of the units compiled in the build directory -- the + project's, its workspace members' and its `path` and `git` dependencies' -- + beside their objects (`-gsplit-dwarf`), and each program holds a skeleton that + names them, so a test does not carry a copy of its closure's debug + information. An index package's units keep theirs in the object: their + objects enter the build cache, which other projects stage. It applies only + with `trim_paths` off, since a debugger finds a unit's `.dwo` through the + build directory the skeleton names. `"off"` keeps all of it in the artifact. + Mach-O (the debug map), PDB, MinGW, WebAssembly and freestanding targets are + unchanged. `-gz=none` in a package's flags turns compression off. - **Each profile owns its own build directory.** The resolved profile knobs participate in the fingerprint, so `target//` holds one hash directory per profile and switching between them is incremental instead of a full rebuild. It also means the disk cost scales with the number of profiles actually use. +- **A configuration keeps one directory over time (2026.10.31.3+).** The part of + the fingerprint a developer switches between -- the compiler family, the + target, the standard library's family, the standard and the configuration's + flags -- is the directory's lineage; the compiler's version, mcpp's version, + the std module and the runtime contract evolve. A new fingerprint of a lineage + takes over the directory that lineage used last, and ninja rebuilds the edges + whose commands changed: after an mcpp upgrade that leaves the flags as they + were, no unit is compiled again. A predecessor of another compiler build is + removed instead (its objects are not reusable). After a build or a test run, + at most once a day, the directories no configuration used for `[build] + keep_days` days (config.toml; default 3, `0` keeps all) are removed. ### 2.10 `[build] cache` — The Global Dependency Cache @@ -1710,9 +1739,9 @@ mcpp cache dir # where the cache lives mcpp cache list [--json] # entries, sizes, last use mcpp cache info @ # one entry, including the key inputs it was built with mcpp cache verify # every entry's file list against the disk -mcpp cache gc --max-size 5GiB # LRU-collect package entries to a budget +mcpp cache gc --max-size 5GiB # LRU-collect package and host tool entries to a budget mcpp cache gc --older-than 30d # ...or by how long since they were last used -mcpp cache clean [--deps|--std|--all|--legacy] +mcpp cache clean [--deps|--std|--tools|--all|--legacy] ``` The on-disk entry layout is versioned. An mcpp release that changes it retires diff --git a/docs/09-commands-by-scenario.md b/docs/09-commands-by-scenario.md index 9e10d95d7..46d9c241e 100644 --- a/docs/09-commands-by-scenario.md +++ b/docs/09-commands-by-scenario.md @@ -30,12 +30,20 @@ them. Confusing them costs a full rebuild. | Store | Scope | Growth trigger | Emptied by | |---|---|---|---| -| `target///` | one project | a configuration fingerprint changes and opens a new directory | `mcpp clean`, `mcpp clean --stale` | -| the build cache (`mcpp cache dir`) | the whole machine | any project compiles a dependency or a `std` module, or builds a host tool, or compiles the `mcpp` module its build program imports | `mcpp cache gc`, `mcpp cache prune`, `mcpp cache clean` | +| `target///`, `target/.build-mcpp/{out,host}/` | one project | a configuration fingerprint changes and opens a new directory; a build program's outputs and the build programs are kept per configuration | `mcpp clean`, `mcpp clean --stale` | +| the build cache (`mcpp cache dir`) | the whole machine | any project compiles a dependency or a `std` module, or builds a host tool, or compiles the `mcpp` module its build program imports | collected after a build when unused for `[cache] unused_days`; `mcpp cache gc`, `mcpp cache prune`, `mcpp cache clean` | `mcpp clean` removes `target/` entirely, and the next build recompiles everything. `mcpp clean --stale` removes only the fingerprint directories that -no recorded build still uses, so the configurations in use survive: +no recorded build still uses, so the configurations in use survive. Each +configuration directory names in `uses.list` the generated-source directories +(`.build-mcpp/out/`) and build-program groups (`.build-mcpp/host/`) +it uses; `--stale` also removes those no kept configuration names +(2026.10.31.3+). A configuration's new fingerprint takes over the directory of +its lineage instead of opening another (docs/04 §2.9), and after a build or a +test run, at most once a day, `--stale` runs by itself with the window `[build] +keep_days` (config.toml; default 3 days). `mcpp clean --report` gives the size of each configuration's +objects, BMIs and products and of each `.build-mcpp` area, and removes nothing: ``` $ mcpp clean --stale --dry-run @@ -74,10 +82,18 @@ key kind size last used package ``` `mcpp cache gc` requires `--max-size`, `--older-than`, or both, and evicts -package entries only. A `std` BMI is shared by every project on the machine, and -the implementation excludes it from size-driven eviction on the grounds that -rebuilding one trades a lot of time for a little disk. `mcpp cache clean --std` -remains the explicit way to remove it. +package and host tool entries. A `std` BMI is shared by every project on the +machine, and the implementation excludes it from size-driven eviction on the +grounds that rebuilding one trades a lot of time for a little disk. `mcpp cache +clean --std` remains the explicit way to remove it. + +After a successful build, at most once a day, every entry -- package, host tool +or `std` -- that no build has used for `[cache] unused_days` days (config.toml; +default 30, `0` turns it off) is removed. A hit marks an entry used. A `path` +host tool's entry is replaced when its tree changes: the entry built from the +same tree under the previous stamp is removed as the new one is written +(2026.10.31.3+). Before a build, mcpp warns when the file system it writes to +has less than 2 GiB free. ## The versions a package publishes diff --git a/docs/25-home-indices-and-store.md b/docs/25-home-indices-and-store.md index e61dfaf4d..efcb51133 100644 --- a/docs/25-home-indices-and-store.md +++ b/docs/25-home-indices-and-store.md @@ -87,6 +87,27 @@ rules are in [SPEC-010](specs/home-initialization.md). without installing tools. `self config --mirror auto|CN|GLOBAL` updates only local configuration, so it remains usable while the network is broken. +### Retention of build outputs (2026.10.31.3+) + +The build cache under `build-cache/v1/` is shared by every project of the home; +a project's `target/` is its own. Each has a retention, in config.toml: + +| key | default | removed | +|---|---|---| +| `[cache] unused_days` | 30 | after a build, at most once a day: cache entries (packages, `std` modules, host tools) no build used for that many days | +| `[build] keep_days` | 3 | after `mcpp build` or `mcpp test`, at most once a day per project: build directories no configuration used for that many days | + +`0` keeps everything. `mcpp self config --cache-unused-days ` and +`--keep-days ` write them; `mcpp self config` shows them. + +A cache entry's files never change after they are written, and a file system +mounted `noatime` or `relatime` does not record reads, so mcpp states a use: it +sets the time of the entry's record file (`entry.json`, `std-module.json`) to +now when a build uses the entry, at most once in 12 hours, so a use costs one +`stat`. A build that replays its graph without planning does not reach the +cache; each configuration directory names the entries it used in `uses.list`, +and the sweep after a build marks them used. + Before a missing bootstrap tool requires network access, initialization displays bounded checks of the provider's configured resources. `self doctor` reports the same checks as warnings; local errors keep their meaning. `--offline` skips the diff --git a/docs/zh/01-getting-started.md b/docs/zh/01-getting-started.md index 5772115c9..c0dc0830d 100644 --- a/docs/zh/01-getting-started.md +++ b/docs/zh/01-getting-started.md @@ -197,6 +197,7 @@ mcpp build # incremental build mcpp clean # clean target/ mcpp clean --stale # drop only target/// dirs no build still uses # (--dry-run lists and deletes nothing; --older-than 3d keeps newer unrecorded ones) +mcpp clean --report # where target/ holds its bytes, per configuration and area mcpp test # compile and run tests/**/*.cpp — one binary per file, # framework-agnostic (bare main, or gtest via [dev-dependencies]) mcpp test # only tests whose name contains diff --git a/docs/zh/04-mcpp-toml.md b/docs/zh/04-mcpp-toml.md index ae9d279d4..856c9e95b 100644 --- a/docs/zh/04-mcpp-toml.md +++ b/docs/zh/04-mcpp-toml.md @@ -1496,6 +1496,7 @@ lto = true # -flto; /GL + /LTCG on cl.exe (note: some packaged gc strip = true # -s at link time ndebug = true # 整个图的每个单元 -DNDEBUG(release/dist 默认开) trim_paths = true # 按单元的 -ffile-prefix-map(release/dist 默认开) +split_debuginfo = "off" # "unpacked":调试信息留在对象旁(dev/debug 默认) # passthrough escape hatch (fixed keys, open values): cflags = ["-fno-plt"] cxxflags = ["-fno-plt"] @@ -1529,10 +1530,25 @@ ldflags = [] 且常借 `__FILE__` 找到它的夹具。带调试信息的构建还在输出旁写出 `source-map.gdbinit` 与 `source-map.lldbinit`, 供调试器把路径映射回来。cl.exe 没有文档化的等价开关,构建会报告一次。在 profile 下写 `ndebug = false` 或 `trim_paths = false` 可保留以前的产物;升级后第一次发布档构建会提示一次。 +- **调试信息(2026.10.31.3+)。** 带调试信息、目标为 ELF 时(Linux、交叉 Linux 目标、ok 映像), + 在工具链构建时支持的前提下,编译与链接都压缩调试信息(`-gz=zlib`;ok 映像的 `ld.lld` 链接为 + `--compress-debug-sections=zlib`)。是否支持询问一次并缓存(LLVM 工具链经其驱动实际运行的 `ld.lld` 做一次压缩链接,GCC 问其驱动):链接器不含 zlib 的 Windows 主机 + 保持不压缩。 + `split_debuginfo = "unpacked"` 是 `dev` 与 `debug` 的默认值:在构建目录中编译的单元——本工程、其工作空间成员 + 以及 `path`、`git` 依赖的单元——的调试信息留在对象旁(`-gsplit-dwarf`),每个程序只携带指向它们的骨架, + 测试程序因此不再复制其闭包的调试信息。索引包的单元保留在对象内:它们的对象进入构建缓存,由其他工程取用。 + 它只在 `trim_paths` 关闭时生效,因为调试器经骨架所记的构建目录查找单元的 `.dwo`。`"off"` 把全部调试信息 + 留在产物中。Mach-O(debug map)、PDB、MinGW、WebAssembly 与裸机目标不变。在包的 flags 中写 `-gz=none` + 可关闭压缩。 - **每个 profile 拥有自己的构建目录。** 解析出的 profile 旋钮参与 指纹，所以 `target//` 下每个 profile 各有一个哈希目录，在 它们之间切换是增量的，不是一次完整重建。这也意味着磁盘开销随实际 用到的 profile 数量而增长。 +- **一个配置在时间上只占一个目录(2026.10.31.3+)。** 指纹中开发者在其间切换的部分——编译器族、目标、 + 标准库族、语言标准与配置级选项——是目录的谱系;编译器版本、mcpp 版本、std 模块与运行时契约随时间演进。 + 同一谱系的新指纹接管该谱系最近使用的目录,由 ninja 重建命令变化的边:编译选项不变的 mcpp 升级之后, + 没有单元被重新编译。编译器构建不同的前驱则被删除(其中的对象不可复用)。构建或测试成功后,每天至多一次, + 删除 `[build] keep_days` 天(config.toml;默认 3,`0` 全部保留)内没有配置使用过的目录。 ### 2.10 `[build] cache` —— 依赖的全局构建缓存 @@ -1576,9 +1592,9 @@ mcpp cache dir # where the cache lives mcpp cache list [--json] # entries, sizes, last use mcpp cache info @ # one entry, including the key inputs it was built with mcpp cache verify # every entry's file list against the disk -mcpp cache gc --max-size 5GiB # LRU-collect package entries to a budget +mcpp cache gc --max-size 5GiB # LRU-collect package and host tool entries to a budget mcpp cache gc --older-than 30d # ...or by how long since they were last used -mcpp cache clean [--deps|--std|--all|--legacy] +mcpp cache clean [--deps|--std|--tools|--all|--legacy] ``` 磁盘上的条目布局是版本化的。一个改变了它的 mcpp 发布版，会一次性 diff --git a/docs/zh/09-commands-by-scenario.md b/docs/zh/09-commands-by-scenario.md index b1322706e..631936c60 100644 --- a/docs/zh/09-commands-by-scenario.md +++ b/docs/zh/09-commands-by-scenario.md @@ -27,11 +27,16 @@ | 存储 | 作用域 | 增长时机 | 清空方式 | |---|---|---|---| -| `target/<三元组>/<指纹>/` | 单个工程 | 一个配置指纹变化，开出一个新目录 | `mcpp clean`、`mcpp clean --stale` | -| 构建缓存（`mcpp cache dir`） | 整台机器 | 任何工程编译一个依赖、一个 `std` 模块，或构建一个 host 工具，或编译其构建程序 import 的 `mcpp` 模块 | `mcpp cache gc`、`mcpp cache prune`、`mcpp cache clean` | +| `target/<三元组>/<指纹>/`、`target/.build-mcpp/{out,host}/` | 单个工程 | 一个配置指纹变化，开出一个新目录；构建程序的输出与构建程序按配置保存 | `mcpp clean`、`mcpp clean --stale` | +| 构建缓存（`mcpp cache dir`） | 整台机器 | 任何工程编译一个依赖、一个 `std` 模块，或构建一个 host 工具，或编译其构建程序 import 的 `mcpp` 模块 | 构建后回收 `[cache] unused_days` 内未使用的条目；`mcpp cache gc`、`mcpp cache prune`、`mcpp cache clean` | `mcpp clean` 整个删掉 `target/`，下次构建重编一切。`mcpp clean --stale` -只删已无构建记录使用的指纹目录，在用的配置保留： +只删已无构建记录使用的指纹目录，在用的配置保留。每个配置目录在 `uses.list` 中 +写明它使用的生成源码目录（`.build-mcpp/out/`）与构建程序组 +（`.build-mcpp/host/`）；`--stale` 同时删除保留的配置都未写明的这两类目录 +（2026.10.31.3+）。一个配置的新指纹接管其谱系的目录，不另开目录（docs/04 §2.9）；构建或测试成功后，每天至多 +一次，`--stale` 以 `[build] keep_days`（config.toml；默认 3 天）为窗口自动执行。`mcpp clean --report` 给出每个配置的对象、BMI 与产物以及 +`.build-mcpp` 各区的大小，不删除任何东西： ``` $ mcpp clean --stale --dry-run @@ -66,11 +71,16 @@ key kind size last used package 9234eed9ef786c13 std 0.0 B 2d ago std (incomplete) ``` -`mcpp cache gc` 要求给出 `--max-size`、`--older-than` 或两者，并且只驱逐 -包条目。一份 `std` BMI 被机器上每个工程共享，实现以"重建它是用大量时间 +`mcpp cache gc` 要求给出 `--max-size`、`--older-than` 或两者，驱逐包条目与 +host 工具条目。一份 `std` BMI 被机器上每个工程共享，实现以"重建它是用大量时间 换少量磁盘"为由，把它排除在按体积驱逐之外。`mcpp cache clean --std` 仍是 显式移除它的做法。 +构建成功后，每天至多一次，移除 `[cache] unused_days` 天（config.toml；默认 30， +`0` 关闭）内没有构建使用过的条目，包、host 工具与 `std` 均在其列；命中即记为使用。 +`path` host 工具的源树变化时，写入新条目的同时移除同一源树在旧 stamp 下的条目 +（2026.10.31.3+）。构建前，若所写入的文件系统可用空间不足 2 GiB，mcpp 给出警告。 + ## 一个包已发布的版本 `mcpp search` 按子串匹配，并在每个命中行后附上该包发布的版本 —— 跨描述符 diff --git a/docs/zh/25-home-indices-and-store.md b/docs/zh/25-home-indices-and-store.md index a106dd5c8..6e1a8556d 100644 --- a/docs/zh/25-home-indices-and-store.md +++ b/docs/zh/25-home-indices-and-store.md @@ -71,6 +71,24 @@ Windows 首次显式 `self init` 时，若 D 是可写的本地固定磁盘， `self env`、`self config`、`self doctor` 和 `toolchain list` 读取本地状态，不安装工具。 `self config --mirror auto|CN|GLOBAL` 仅更新本地配置，因此网络故障时仍可使用。 +### 构建产物的保留期（2026.10.31.3+） + +`build-cache/v1/` 下的构建缓存由 home 中所有工程共享；工程的 `target/` 属于工程自身。两者各有保留期， +写在 config.toml 中： + +| 键 | 默认 | 移除的内容 | +|---|---|---| +| `[cache] unused_days` | 30 | 构建后每天至多一次：该天数内没有构建使用过的缓存条目（包、`std` 模块、host 工具） | +| `[build] keep_days` | 3 | `mcpp build` 或 `mcpp test` 后，每个工程每天至多一次：该天数内没有配置使用过的构建目录 | + +`0` 表示全部保留。`mcpp self config --cache-unused-days ` 与 `--keep-days ` 写入它们， +`mcpp self config` 显示它们。 + +缓存条目的文件写入后不再改变，以 `noatime` 或 `relatime` 挂载的文件系统也不记录读取，因此由 mcpp +记录使用：构建使用一个条目时，把条目记录文件（`entry.json`、`std-module.json`）的时间设为当前时间， +每个条目 12 小时内至多一次，一次使用的代价是一次 `stat`。不经规划、直接重放构建图的构建不访问缓存； +每个配置目录在 `uses.list` 中记录它用过的条目，构建后的回收把它们记为已使用。 + 初始化因缺少基础工具而需要联网前，会显示提供者已配置资源的有界检查结果。 `self doctor` 将同类检查结果作为警告报告，本地错误的语义保持不变。 `--offline` 跳过检查与获取；已有工具和缓存包仍可使用。 diff --git a/mcpp.toml b/mcpp.toml index 9d3b5b1d0..e6d9fc7cd 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "mcpp" -version = "2026.10.31.2" +version = "2026.10.31.3" description = "Modern C++ build & package management tool" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/modules/buildmcpp/src/tool_store.cppm b/modules/buildmcpp/src/tool_store.cppm index c5a0c5b15..e7ac173da 100644 --- a/modules/buildmcpp/src/tool_store.cppm +++ b/modules/buildmcpp/src/tool_store.cppm @@ -148,7 +148,18 @@ std::filesystem::path bin_path(const std::filesystem::path& entryDir, bool entry_valid(const std::filesystem::path& entryDir, const Key& k, std::string_view toolName, std::string_view exeSuffix); -void write_entry(const std::filesystem::path& entryDir, const Key& k); +// Writes the entry's record. `source` is the tree a `path` package was +// built from: written beside the record, and an entry of the same package +// from the same tree under another stamp (`+path.`) is +// superseded and removed -- a path package's every edit was a new key, and +// none was ever collected (mcpp#874). +void write_entry(const std::filesystem::path& entryDir, const Key& k, + const std::filesystem::path& source = {}); + +// Marks the entry used now: `mcpp cache` collects by last use, and the +// record's time is that use (the record's content is the key, compared +// field by field, and does not change). +void touch(const std::filesystem::path& entryDir); // ── Overrides (the escape hatch every comparable system provides) ────────── // @@ -292,11 +303,47 @@ bool entry_valid(const fs::path& entryDir, const Key& k, } } -void write_entry(const fs::path& entryDir, const Key& k) { +void write_entry(const fs::path& entryDir, const Key& k, const fs::path& source) { std::error_code ec; fs::create_directories(entryDir, ec); - std::ofstream os(entryDir / "entry.json", std::ios::trunc); - if (os) os << to_json(k).dump(2) << '\n'; + { + std::ofstream os(entryDir / "entry.json", std::ios::trunc); + if (os) os << to_json(k).dump(2) << '\n'; + } + const auto plus = k.version.find("+path."); + if (source.empty() || plus == std::string::npos) return; + const auto origin = source.lexically_normal().generic_u8string(); + const std::string text(origin.begin(), origin.end()); + std::ofstream(entryDir / "source", std::ios::trunc) << text; + // `/@+path.//`: the siblings of this + // version directory that differ only in the stamp. + const auto versionDir = entryDir.parent_path(); + const auto prefix = std::format("{}@{}", k.packageName, k.version.substr(0, plus + 6)); + auto same_source = [&](const fs::path& keyDir) { + std::ifstream in(keyDir / "source"); + if (!in) return false; + const std::string was{std::istreambuf_iterator(in), {}}; + return was == text; + }; + for (auto const& v : fs::directory_iterator(versionDir.parent_path(), ec)) { + const auto name = v.path().filename().string(); + if (v.path() == versionDir || !name.starts_with(prefix)) continue; + std::error_code kec; + for (auto const& key : fs::directory_iterator(v.path(), kec)) + if (same_source(key.path())) fs::remove_all(key.path(), kec); + fs::remove(v.path(), kec); // only when nothing else is left in it + } +} + +// The rule of mcpp::home::mark_used, which this module cannot import: the +// record's time, at most once in 12 hours. +void touch(const fs::path& entryDir) { + std::error_code ec; + const auto record = entryDir / "entry.json"; + const auto now = fs::file_time_type::clock::now(); + const auto was = fs::last_write_time(record, ec); + if (ec || now - was < std::chrono::hours(12)) return; + fs::last_write_time(record, now, ec); } std::optional diff --git a/modules/manifest/src/toml.cppm b/modules/manifest/src/toml.cppm index 715f4adc0..bfea108a2 100644 --- a/modules/manifest/src/toml.cppm +++ b/modules/manifest/src/toml.cppm @@ -1260,6 +1260,10 @@ std::expected parse_document(const t::Document& documen pr.trimPaths = it->second.as_bool(); pr.trimPathsDeclared = true; } + if (auto it = tt.find("split_debuginfo"); it != tt.end() && it->second.is_string()) { + pr.splitDebuginfo = it->second.as_string(); + pr.splitDebuginfoDeclared = true; + } auto read_list = [&](const char* key, std::vector& out) { if (auto it = tt.find(key); it != tt.end() && it->second.is_array()) for (auto& v : it->second.as_array()) diff --git a/modules/manifest/src/types.cppm b/modules/manifest/src/types.cppm index b522834ca..a513f185e 100644 --- a/modules/manifest/src/types.cppm +++ b/modules/manifest/src/types.cppm @@ -1184,6 +1184,9 @@ struct BuildConfig : BuildInputs { // `-ffile-prefix-map` for every unit (the profile's `trim_paths`): the // paths an artifact carries name packages and roles, not this machine. bool trimPaths = false; + // The units compiled in the build directory keep their debug information + // beside their objects (`split_debuginfo = "unpacked"`, resolved). + bool splitDebuginfo = false; // `[build].default-profile` (alias: `profile`) — the project's DEFAULT // profile when no --profile/--dev/--release is passed. The global convention // default stays "release"; this lets a project opt its plain `mcpp build` @@ -2108,6 +2111,12 @@ struct Profile { bool ndebugDeclared = false; bool trimPaths = false; bool trimPathsDeclared = false; + // `split_debuginfo` (design 2026-10-31 §4.1): "unpacked" leaves the + // debug information of the units compiled in the build directory beside + // their objects, "off" puts it in the artifact. Default by the profile's + // name, as `trim_paths`: unpacked for `dev` and `debug`. + std::string splitDebuginfo; + bool splitDebuginfoDeclared = false; // `dependency_linkage`, per profile (#519). // // DECLARED-OR-NOT IS LOAD-BEARING, and that is why there are two members diff --git a/modules/versioning/src/version.cppm b/modules/versioning/src/version.cppm index 115e2ff9b..94066c682 100644 --- a/modules/versioning/src/version.cppm +++ b/modules/versioning/src/version.cppm @@ -31,6 +31,6 @@ import std; export namespace mcpp { -inline constexpr std::string_view MCPP_VERSION = "2026.10.31.2"; +inline constexpr std::string_view MCPP_VERSION = "2026.10.31.3"; } // namespace mcpp diff --git a/src/bmi_cache.cppm b/src/bmi_cache.cppm index f5c40ea36..1ef04495c 100644 --- a/src/bmi_cache.cppm +++ b/src/bmi_cache.cppm @@ -195,6 +195,7 @@ std::expected resolve_cached(const CacheKey& key); // dependency that hit on every build looked stale. void touch_accessed(const CacheKey& key); + // Copy fresh build outputs from projectTarget/{bmiDirName,obj} into the entry, // then write entry.json last as the sentinel. std::expected @@ -379,13 +380,10 @@ std::expected resolve_cached(const CacheKey& key) { } void touch_accessed(const CacheKey& key) { - auto j = read_entry(key.entryFile()); - if (!j) return; - (*j)["accessed"] = now_iso8601(); - // An entry written before the footprint was recorded gains it on its - // first use, so `cache list` -- which writes nothing -- stops measuring it. - if (!j->contains("size") || !j->contains("files")) record_footprint(*j, key.dir()); - (void)write_entry(key.entryFile(), *j); + // The record's time, not its content (mcpp::home::mark_used): rewriting + // entry.json on every hit cost a parse and a write per cached package per + // build. + mcpp::home::mark_used(key.entryFile()); } std::expected diff --git a/src/bmi_cache/maintenance.cppm b/src/bmi_cache/maintenance.cppm index 8abb62efc..f6b42a70d 100644 --- a/src/bmi_cache/maintenance.cppm +++ b/src/bmi_cache/maintenance.cppm @@ -4,6 +4,8 @@ // Layout walked here (see bmi_cache.cppm): // $MCPP_HOME/build-cache/v1/pkg//@//entry.json // $MCPP_HOME/build-cache/v1/std//std-module.json +// $MCPP_HOME/build-cache/v1/tool//@//entry.json +// (a host tool, mcpp.build.tool_store; last use = the record's time) // // Two things this file did wrong before the layout change and that the new // entry.json fixes: @@ -26,6 +28,7 @@ export module mcpp.bmi_cache.maintenance; import std; import mcpp.home; import mcpp.libs.json; +import mcpp.libs.toml; import mcpp.ui; namespace mcpp::bmi_cache { @@ -97,10 +100,11 @@ namespace { std::filesystem::path pkg_root() { return mcpp::home::cache_root() / "pkg"; } std::filesystem::path std_root() { return mcpp::home::cache_root() / "std"; } +std::filesystem::path tool_root() { return mcpp::home::cache_root() / "tool"; } struct Entry { std::filesystem::path dir; - std::string kind; // "pkg" | "std" + std::string kind; // "pkg" | "std" | "tool" std::string label; // "/@" | "std" std::string key; std::uintmax_t size = 0; @@ -229,6 +233,9 @@ std::vector walk_pkg_entries(bool verify = false) { e.accessed = stamp_of(*j, "accessed"); if (verify) check_pkg_files(e, *j); } + // The later of the stamp an entry was written with and its + // record's time, which a use sets (mcpp::home::mark_used). + e.accessed = std::max(e.accessed, dir_mtime_seconds(e.dir / "entry.json")); if (e.accessed == 0) e.accessed = dir_mtime_seconds(e.dir); // The footprint the entry recorded when it was written (see // record_footprint); measured only for an entry from before @@ -271,13 +278,55 @@ std::vector walk_std_entries() { j->value("cpp_standard", std::string{"?"}), j->value("stdlib", std::string{"?"})); } - e.accessed = dir_mtime_seconds(e.dir); + // A use sets the record's time (mcpp::home::mark_used). + e.accessed = std::max(dir_mtime_seconds(e.dir), dir_mtime_seconds(e.dir / "std-module.json")); measure(e); out.push_back(std::move(e)); } return out; } +// Host tools (mcpp#874). An entry's last use is its record's time, which a +// hit refreshes (tool_store::touch); `.build/` holds the sub-builds in flight. +std::vector walk_tool_entries() { + std::vector out; + std::error_code ec; + auto root = tool_root(); + if (!std::filesystem::exists(root, ec)) return out; + for (auto& idx : std::filesystem::directory_iterator(root, ec)) { + if (!idx.is_directory() || idx.path().filename() == ".build") continue; + for (auto& pkg : std::filesystem::directory_iterator(idx.path(), ec)) { + if (!pkg.is_directory()) continue; + for (auto& key : std::filesystem::directory_iterator(pkg.path(), ec)) { + if (!key.is_directory()) continue; + Entry e; + e.dir = key.path(); + e.kind = "tool"; + e.key = key.path().filename().string(); + e.label = idx.path().filename().string() + "/" + pkg.path().filename().string(); + std::error_code fe; + if (!std::filesystem::is_regular_file(e.dir / "entry.json", fe)) { + e.complete = false; + e.problem = "no entry.json"; + e.accessed = dir_mtime_seconds(e.dir); + } else { + e.accessed = dir_mtime_seconds(e.dir / "entry.json"); + } + measure(e); + out.push_back(std::move(e)); + } + } + } + return out; +} + +// Package and tool entries: what `prune` and `gc` collect by last use. +std::vector walk_collectable() { + auto out = walk_pkg_entries(); + for (auto& e : walk_tool_entries()) out.push_back(std::move(e)); + return out; +} + std::string format_age(std::int64_t stampSeconds) { if (stampSeconds == 0) return "unknown"; auto now = std::chrono::duration_cast( @@ -330,7 +379,7 @@ export int cache_dir() { // repository's e2e asserts those keys, so it keeps that shape for ever -- // but they must always describe the same cache. export nlohmann::json cache_list_json() { - auto entries = walk_pkg_entries(); + auto entries = walk_collectable(); auto stds = walk_std_entries(); nlohmann::json j; j["root"] = mcpp::home::cache_root().string(); @@ -359,7 +408,7 @@ export int cache_list(bool asJson) { std::println("{}", cache_list_json().dump(2)); return 0; } - auto entries = walk_pkg_entries(); + auto entries = walk_collectable(); auto stds = walk_std_entries(); if (entries.empty() && stds.empty()) { @@ -429,7 +478,7 @@ export int cache_prune(const std::string& v) { int removed = 0; std::uintmax_t freed = 0; - for (auto& e : walk_pkg_entries()) { + for (auto& e : walk_collectable()) { if (e.accessed != 0 && e.accessed >= cutoff) continue; auto n = remove_entry(e); if (n || !std::filesystem::exists(e.dir)) { @@ -472,10 +521,11 @@ export int cache_gc(const std::string& maxSizeArg, const std::string& olderThanA } } - // Package entries only. A std BMI is shared by every project on the machine - // and costs ~30 s to rebuild; evicting one to hit a size target trades a lot - // of time for a little disk. `cache clean --std` remains the explicit way. - auto entries = walk_pkg_entries(); + // Package and tool entries. A std BMI is shared by every project on the + // machine and costs ~30 s to rebuild; evicting one to hit a size target + // trades a lot of time for a little disk. `cache clean --std` remains the + // explicit way, and the collection by last use after a build covers it. + auto entries = walk_collectable(); std::ranges::sort(entries, [](const Entry& a, const Entry& b) { return a.accessed < b.accessed; // oldest first }); @@ -506,14 +556,14 @@ export int cache_gc(const std::string& maxSizeArg, const std::string& olderThanA // because std entries are deliberately out of scope here. Reporting it as // the cache size would read as "the cache is now empty" while tens of MB of // std BMIs sit right next to it. - std::println("Collected {} entries, freed {} (package entries now {})", + std::println("Collected {} entries, freed {} (package and tool entries now {})", removed, human_bytes(freed), human_bytes(live)); if (maxSize && live > *maxSize) { // Say it rather than silently under-delivering: a size target that // cannot be met without evicting std BMIs is a real outcome, and a // caller that thinks the budget held would be misled. mcpp::ui::warning(std::format( - "still {} over the {} budget: package entries are exhausted. " + "still {} over the {} budget: package and tool entries are exhausted. " "std BMIs are excluded from gc; use `mcpp cache clean --std`", human_bytes(live - *maxSize), human_bytes(*maxSize))); } @@ -521,9 +571,9 @@ export int cache_gc(const std::string& maxSizeArg, const std::string& olderThanA } // `mcpp cache clean [--deps] [--std] [--all] [--legacy]`. -export int cache_clean(bool deps, bool stds, bool all, bool legacy) { - if (all) { deps = true; stds = true; } - if (!deps && !stds && !legacy) deps = true; // bare `clean` = dep entries +export int cache_clean(bool deps, bool stds, bool tools, bool all, bool legacy) { + if (all) { deps = true; stds = true; tools = true; } + if (!deps && !stds && !tools && !legacy) deps = true; // bare `clean` = dep entries std::error_code ec; if (deps) { @@ -536,6 +586,11 @@ export int cache_clean(bool deps, bool stds, bool all, bool legacy) { std::filesystem::remove_all(std_root(), ec); std::println("Cleaned std module entries ({})", human_bytes(n)); } + if (tools) { + auto n = dir_size(tool_root()); + std::filesystem::remove_all(tool_root(), ec); + std::println("Cleaned host tool entries ({})", human_bytes(n)); + } if (legacy) { auto legacyRoot = mcpp::home::legacy_bmi_root(); if (std::filesystem::exists(legacyRoot, ec)) { @@ -550,9 +605,48 @@ export int cache_clean(bool deps, bool stds, bool all, bool legacy) { return 0; } +// COLLECTION BY LAST USE, AFTER A BUILD (mcpp#874). Nothing collected the +// cache unless a user ran `gc`: superseded package versions, every std +// module a toolchain ever built and every variant of every host tool stayed. +// After a successful build, at most once a day, an entry of any kind not used +// for `[cache] unused_days` (config.toml; default 30, 0 turns it off) is +// removed -- the same entries `cache prune --older-than d` removes, std +// modules included, each of which a hit marks used. +export void collect_unused_entries() { + std::int64_t days = 30; + std::error_code ec; + const auto config = mcpp::home::root() / "config.toml"; + if (std::filesystem::exists(config, ec)) + if (auto doc = mcpp::libs::toml::parse_file(config)) + days = doc->get_int("cache.unused_days").value_or(days); + if (days <= 0) return; + const auto stamp = mcpp::home::cache_root() / ".collected"; + const auto now = std::chrono::duration_cast( + std::chrono::system_clock::now().time_since_epoch()).count(); + if (std::filesystem::exists(stamp, ec) && now - dir_mtime_seconds(stamp) < 86400) return; + std::filesystem::create_directories(stamp.parent_path(), ec); + std::ofstream(stamp, std::ios::trunc) << now << '\n'; + const auto cutoff = now - days * 86400; + auto entries = walk_collectable(); + for (auto& e : walk_std_entries()) entries.push_back(std::move(e)); + std::size_t removed = 0; + std::uintmax_t freed = 0; + for (auto const& e : entries) { + if (e.accessed == 0 || e.accessed >= cutoff) continue; + remove_entry(e); + ++removed; + freed += e.size; + } + if (removed) + mcpp::ui::status("Collected", std::format( + "{} cache entr{} unused for {} days ({}; [cache] unused_days in config.toml)", + removed, removed == 1 ? "y" : "ies", days, human_bytes(freed))); +} + // `mcpp cache verify` — entry.json against the disk, for every entry. export int cache_verify() { auto entries = walk_pkg_entries(/*verify=*/true); + for (auto& e : walk_tool_entries()) entries.push_back(std::move(e)); auto stds = walk_std_entries(); int bad = 0; for (auto* set : {&entries, &stds}) { diff --git a/src/build/execute.cppm b/src/build/execute.cppm index 46d080ae5..29aef4ed9 100644 --- a/src/build/execute.cppm +++ b/src/build/execute.cppm @@ -1756,6 +1756,18 @@ export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache, std::error_code ec; std::filesystem::remove_all(ctx.outputDir, ec); } + // Said before the build rather than as a link that fails for want of + // space (mcpp#874: `final link failed: No space left on device`). + { + std::error_code sec; + const auto free = std::filesystem::space(ctx.outputDir.parent_path(), sec).available; + if (!sec && free < (2ull << 30)) + mcpp::ui::warning(std::format( + "{} free where this build writes ({}). `mcpp clean --stale` removes the build " + "directories no configuration uses, and `mcpp cache gc --older-than 30d` the " + "cache entries none has used", + mcpp::bmi_cache::human_bytes(free), ctx.outputDir.parent_path().string())); + } // The generated `*link:` spec lives in the output directory, and the line // above is allowed to delete that directory. prepare wrote the file before @@ -1907,6 +1919,7 @@ export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache, mcpp::build::progress::finished( ctx.profile, mcpp::build::profile_descriptor(ctx.plan.manifest.buildConfig)); report_freestanding_size(ctx); + mcpp::bmi_cache::collect_unused_entries(); return 0; } @@ -5711,7 +5724,91 @@ export int clean_project(bool wipe_bmi) { // `target/` before that: none of them is read by a workspace build. A // member's `target/.build-mcpp/`, where its build program's outputs are, is // kept. -export int clean_stale(bool dryRun, std::int64_t keepWithinSecs) { +// `mcpp clean --report` (mcpp#874): where target/ holds its bytes. Per +// configuration directory, its objects, its BMIs and its products; then the +// configuration-independent `.build-mcpp` (build programs' groups, generated +// sources, dependencies' build-program copies). Removes nothing; `--stale +// --dry-run` says what of it is unused. +export int clean_report() { + namespace fs = std::filesystem; + auto root = mcpp::project::find_manifest_root(fs::current_path()); + if (!root) { std::println(stderr, "error: not in an mcpp package"); return 2; } + if (auto ws = mcpp::project::find_workspace_root(*root); !ws.empty()) root = ws; + const fs::path target = *root / "target"; + using mcpp::bmi_cache::dir_size, mcpp::bmi_cache::human_bytes; + std::uintmax_t total = 0; + std::println("{:<44} {:>10} {:>10} {:>10} {:>10}", "configuration", "obj", "bmi", "bin", "total"); + std::error_code ec; + for (fs::directory_iterator t(target, ec), end; t != end; t.increment(ec)) { + const auto triple = t->path().filename().string(); + std::error_code fec; + if (triple.starts_with(".") || !t->is_directory(fec)) continue; + for (fs::directory_iterator f(t->path(), fec), fend; f != fend; f.increment(fec)) { + if (!f->is_directory()) continue; + const auto obj = dir_size(f->path() / "obj"); + const auto bmi = dir_size(f->path() / "gcm.cache") + dir_size(f->path() / "pcm.cache"); + const auto bin = dir_size(f->path() / "bin"); + const auto all = dir_size(f->path()); + total += all; + std::println("{:<44} {:>10} {:>10} {:>10} {:>10}", + triple + "/" + f->path().filename().string(), human_bytes(obj), + human_bytes(bmi), human_bytes(bin), human_bytes(all)); + } + } + const auto bm = target / ".build-mcpp"; + std::println(""); + for (auto const* area : {"host", "out", "deps"}) { + const auto n = dir_size(bm / area); + total += n; + std::println("{:<44} {:>10}", std::format(".build-mcpp/{}", area), human_bytes(n)); + } + std::println(""); + std::println("{:<44} {:>10}", "target/", human_bytes(total)); + return 0; +} + +// After a successful `mcpp build` or `mcpp test`, at most once a day per +// project (design 2026-10-31 §8.3): `clean --stale` with the window +// `[build] keep_days` (config.toml; default 3, 0 keeps all), said in one line. +// A configuration's evolution reuses its directory (§8.2), so what this finds +// is a configuration nobody built for that long. +export int clean_stale(bool dryRun, std::int64_t keepWithinSecs, bool quiet = false); +export void sweep_after_build() { + namespace fs = std::filesystem; + std::int64_t days = 3; + std::error_code ec; + const auto config = mcpp::home::root() / "config.toml"; + if (fs::exists(config, ec)) + if (auto doc = mcpp::libs::toml::parse_file(config)) + days = doc->get_int("build.keep_days").value_or(days); + if (days <= 0) return; + auto root = mcpp::project::find_manifest_root(fs::current_path()); + if (!root) return; + if (auto ws = mcpp::project::find_workspace_root(*root); !ws.empty()) root = ws; + const auto stamp = *root / "target" / ".mcpp-swept"; + if (fs::exists(stamp, ec)) { + const auto age = std::chrono::duration_cast( + fs::file_time_type::clock::now() - fs::last_write_time(stamp, ec)).count(); + if (!ec && age < 24) return; + } + if (!fs::is_directory(*root / "target", ec)) return; + std::ofstream(stamp, std::ios::trunc) << "swept\n"; + clean_stale(false, days * 86400, true); + // The build-cache entries the remaining configurations use are used today, + // whether this build planned or replayed its graph (design 2026-10-31 §4.6). + for (fs::recursive_directory_iterator it(*root / "target", ec), end; it != end; it.increment(ec)) { + if (it.depth() >= 2) it.disable_recursion_pending(); + if (it->path().filename() != "uses.list") continue; + std::ifstream in(it->path()); + for (std::string l; std::getline(in, l);) + if (l.starts_with("cache ")) mcpp::home::mark_used(l.substr(6)); + } +} + +export int clean_stale(bool dryRun, std::int64_t keepWithinSecs, bool quiet) { + // Quiet: the sweep after a build (design 2026-10-31 §8.3) says one line + // when it removed something, and nothing about what it kept. + auto line = [quiet](const std::string& text) { if (!quiet) std::println("{}", text); }; namespace fs = std::filesystem; auto root = mcpp::project::find_manifest_root(fs::current_path()); if (!root) { std::println(stderr, "error: not in an mcpp package"); return 2; } @@ -5746,6 +5843,7 @@ export int clean_stale(bool dryRun, std::int64_t keepWithinSecs) { if (!fp.empty()) current[triple].insert(fp); } if (current.empty()) { + if (quiet) return 0; std::println(stderr, "error: {} has no build record, so nothing is known to be current; " "run `mcpp build` once, then retry", (*root / kBuildCacheFile).string()); @@ -5754,6 +5852,7 @@ export int clean_stale(bool dryRun, std::int64_t keepWithinSecs) { std::uintmax_t bytes = 0; std::size_t removed = 0, failed = 0; + std::set swept; // removed, or would be under --dry-run std::error_code ec; for (fs::directory_iterator tripleIt(target, ec), end; tripleIt != end; tripleIt.increment(ec)) { std::error_code tec; @@ -5779,13 +5878,14 @@ export int clean_stale(bool dryRun, std::int64_t keepWithinSecs) { const std::string ago = age < 3600 ? std::format("{}m", std::max(age / 60, 1)) : age < 86400 ? std::format("{}h", age / 3600) : std::format("{}d", age / 86400); - std::println("kept {} (not in the record, but written {} ago; see --older-than)", shown, ago); + line(std::format("kept {} (not in the record, but written {} ago; see --older-than)", shown, ago)); continue; } } const auto size = mcpp::bmi_cache::dir_size(dir); + swept.insert(dir); if (dryRun) { - std::println("would remove {} ({})", shown, mcpp::bmi_cache::human_bytes(size)); + line(std::format("would remove {} ({})", shown, mcpp::bmi_cache::human_bytes(size))); } else { std::error_code rec; fs::remove_all(dir, rec); @@ -5794,7 +5894,7 @@ export int clean_stale(bool dryRun, std::int64_t keepWithinSecs) { ++failed; continue; } - std::println("removed {} ({})", shown, mcpp::bmi_cache::human_bytes(size)); + line(std::format("removed {} ({})", shown, mcpp::bmi_cache::human_bytes(size))); } bytes += size; ++removed; @@ -5816,8 +5916,12 @@ export int clean_stale(bool dryRun, std::int64_t keepWithinSecs) { return (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f'); }); }; + // Not the sweep after a build, which collects only by age; and never the + // workspace's own directory, which a rooted workspace lists as `"."`. for (auto const& memberDir : memberDirs) { + if (quiet) break; std::error_code mec; + if (fs::equivalent(memberDir, *root, mec)) continue; const auto memberTarget = memberDir / "target"; if (!fs::is_directory(memberTarget, mec)) continue; for (fs::directory_iterator tripleIt(memberTarget, mec), end; tripleIt != end; @@ -5834,8 +5938,8 @@ export int clean_stale(bool dryRun, std::int64_t keepWithinSecs) { const std::string shown = dir.lexically_relative(*root).generic_string(); const auto size = mcpp::bmi_cache::dir_size(dir); if (dryRun) { - std::println("would remove {} ({}; the workspace builds in {})", shown, - mcpp::bmi_cache::human_bytes(size), target.string()); + line(std::format("would remove {} ({}; the workspace builds in {})", shown, + mcpp::bmi_cache::human_bytes(size), target.string())); } else { std::error_code rec; fs::remove_all(dir, rec); @@ -5844,8 +5948,8 @@ export int clean_stale(bool dryRun, std::int64_t keepWithinSecs) { ++failed; continue; } - std::println("removed {} ({}; the workspace builds in {})", shown, - mcpp::bmi_cache::human_bytes(size), target.string()); + line(std::format("removed {} ({}; the workspace builds in {})", shown, + mcpp::bmi_cache::human_bytes(size), target.string())); } bytes += size; ++removed; @@ -5853,12 +5957,94 @@ export int clean_stale(bool dryRun, std::int64_t keepWithinSecs) { } } + // WHAT THE CONFIGURATIONS USE BESIDE THEIR DIRECTORIES (mcpp#874): the + // generated sources of `.build-mcpp/out/` and the build programs' + // `.build-mcpp/host/`. Each fingerprint directory names what it + // used (`uses.list`); one that is kept keeps what it names, recorded or + // not, since `mcpp test` builds without a record entry. With no list + // anywhere -- directories an earlier release wrote -- nothing is known to + // be unused and nothing here is removed. + std::set usedOut, usedHost; + bool usesKnown = false; + auto readUses = [&](const fs::path& fpDir) { + if (swept.contains(fpDir)) return; + std::ifstream in(fpDir / "uses.list"); + if (!in) return; + usesKnown = true; + for (std::string l; std::getline(in, l);) { + const auto sp = l.find(' '); + if (sp == std::string::npos) continue; + if (l.starts_with("out ")) usedOut.insert(l.substr(sp + 1)); + else if (l.starts_with("host ")) usedHost.insert(l.substr(sp + 1)); + } + }; + { + std::error_code uec; + for (fs::directory_iterator t(target, uec), end; t != end; t.increment(uec)) { + std::error_code fec; + if (!t->is_directory(fec) || fec) continue; + for (fs::directory_iterator f(t->path(), fec), fend; f != fend; f.increment(fec)) + readUses(f->path()); + } + } + auto collect = [&](const fs::path& dir, const std::string& shown) { + std::error_code aec; + const auto age = std::chrono::duration_cast( + fs::file_time_type::clock::now() - fs::last_write_time(dir, aec)).count(); + if (!aec && age < keepWithinSecs) { + line(std::format("kept {} (used by no configuration, but written recently; see --older-than)", shown)); + return; + } + const auto size = mcpp::bmi_cache::dir_size(dir); + if (!dryRun) { + std::error_code rec; + fs::remove_all(dir, rec); + if (rec) { + std::println(stderr, "error: cannot remove {}: {}", shown, rec.message()); + ++failed; + return; + } + } + line(std::format("{} {} ({}; used by no configuration)", dryRun ? "would remove" : "removed", + shown, mcpp::bmi_cache::human_bytes(size))); + bytes += size; + ++removed; + }; + if (usesKnown) { + std::vector bases{*root}; + bases.insert(bases.end(), memberDirs.begin(), memberDirs.end()); + for (auto const& base : bases) { + std::error_code oec; + for (fs::directory_iterator o(base / "target" / ".build-mcpp" / "out", oec), end; o != end; + o.increment(oec)) + if (!usedOut.contains(o->path().filename().string())) + collect(o->path(), o->path().lexically_relative(*root).generic_string()); + } + // A group the build programs' graph needed is `.`, kept + // with its group; `digests` is the shared source-digest cache. + std::error_code hec; + for (fs::directory_iterator h(target / ".build-mcpp" / "host", hec), end; h != end; + h.increment(hec)) { + const auto name = h->path().filename().string(); + if (name == "digests" || !h->is_directory()) continue; + if (!usedHost.contains(name.substr(0, name.find('.')))) + collect(h->path(), h->path().lexically_relative(*root).generic_string()); + } + } else { + line(std::format("note: no build directory names what it uses (built by an earlier release); " + "`.build-mcpp/out` and `.build-mcpp/host` are left as they are")); + } + if (removed == 0 && failed == 0) { - std::println("Nothing stale under {}: every fingerprint directory is recorded as current", - target.string()); + line(std::format("Nothing stale under {}: every fingerprint directory is recorded as current{}", + target.string(), usesKnown + ? ", and every generated-source and build-program directory is in use" : "")); } else { - std::println("{} {} director{} ({})", dryRun ? "Would remove" : "Removed", removed, - removed == 1 ? "y" : "ies", mcpp::bmi_cache::human_bytes(bytes)); + if (quiet) + mcpp::ui::status("Swept", std::format("{} build director{} no configuration used for {} days ({}; [build] keep_days in config.toml)", + removed, removed == 1 ? "y" : "ies", keepWithinSecs / 86400, mcpp::bmi_cache::human_bytes(bytes))); + line(std::format("{} {} director{} ({})", dryRun ? "Would remove" : "Removed", removed, + removed == 1 ? "y" : "ies", mcpp::bmi_cache::human_bytes(bytes))); } return failed ? 1 : 0; } diff --git a/src/build/flags.cppm b/src/build/flags.cppm index 906ad16bf..73d0c97b2 100644 --- a/src/build/flags.cppm +++ b/src/build/flags.cppm @@ -36,6 +36,7 @@ import mcpp.toolchain.hostflags; import mcpp.toolchain.linkmodel; import mcpp.toolchain.model; import mcpp.toolchain.provider; +import mcpp.toolchain.probe; // compresses_debug_sections import mcpp.toolchain.registry; export namespace mcpp::build { @@ -1173,6 +1174,25 @@ CompileFlags compute_flags(const BuildPlan& plan) { ? " /Od" // MSVC's no-opt spelling (there is no /O0) : std::format(" {}{}", d.optPrefix, optLevel); if (prof.debug) opt_flag += std::format(" {}", d.debugFlags); + // DEBUG INFORMATION IS COMPRESSED where the linker copies it into every + // artifact (mcpp#874): on ELF each program of a member, every test among + // them, carries the DWARF of its whole closure, and zlib halves it (a + // workspace's 315 MB test program to 149 MB). The objects are written + // compressed and the driver link keeps them so (`-gz=zlib` on both). Not on + // Mach-O, whose debug map leaves the DWARF in the objects; not with cl.exe, + // whose PDB holds it once; not on a freestanding target, whose boards' + // debuggers may read neither. An ok image, linked by `ld.lld` itself, has + // its sections compressed by that link (measured: okld seals it and okrun + // runs it). gdb and lldb read compressed sections; `-gz=none` in a + // package's flags comes after this and turns it off. + // And only with a toolchain built to write and read them (`-gz=zlib` is a + // warning without zlib, and `ld.lld` then refuses the link). + const bool compressDebug = prof.debug && !isMsvcDialect + && targetObjectFormat == mcpp::build::dist::Format::Elf + && !isFreestandingTarget + && mcpp::toolchain::compresses_debug_sections(plan.toolchain.binaryPath, + mcpp::toolchain::compiler_env(plan.toolchain)); + if (compressDebug) opt_flag += " -gz=zlib"; // LTO ON EVERY COMPILER (2026.10.5.2). cl.exe spells it `/GL` here and // `/LTCG` on the link and the archive; before, it was not spelled at all, // and a build with `lto = true` reported `+ lto` while doing none. @@ -1890,6 +1910,7 @@ CompileFlags compute_flags(const BuildPlan& plan) { std::string link_extra; if (prof.lto) link_extra += " -flto"; + if (compressDebug) link_extra += " -gz=zlib"; if (prof.strip) link_extra += " -s"; // MinGW PE link — keyed on the TARGET (is_mingw_target), NOT the host: a @@ -2540,6 +2561,7 @@ CompileFlags compute_flags(const BuildPlan& plan) { // one of them and not `ld.lld` is asked for the ELF linker // explicitly. if (f.ldDriver.stem() != "ld.lld") f.ld = " -flavor gnu" + f.ld; + if (compressDebug) f.ld += " --compress-debug-sections=zlib"; f.ldC = f.ld; f.sysroot.clear(); f.bFlag.clear(); diff --git a/src/build/ninja_backend.cppm b/src/build/ninja_backend.cppm index a448cebab..2e81f5d04 100644 --- a/src/build/ninja_backend.cppm +++ b/src/build/ninja_backend.cppm @@ -871,6 +871,21 @@ std::string action_phony_name(std::string_view pkg) { // declaring package's. File-scope for the same reason // `action_precedes_compilation`/`action_phony_name` are: the emitter and // `check_action_ordering` ask the identical question. +// What a unit's LINK edge writes. Under a platform with seal steps (SPEC-013 +// §4.6) the link writes beneath `obj/platform/` and a `platform_seal` edge +// publishes the artifact's own name; that edge reads no `$ldflags` and is not a +// link. File-scope because the emitter and `check_action_ordering` must name the +// same edge: the guard once took the seal edge for the link and refused every ok +// image whose graph declares a `prepare` action. +std::filesystem::path link_edge_output(const BuildPlan& plan, const LinkUnit& lu) { + if (!plan.platform.active || plan.platform.fields.link.seal.empty() + || lu.kind == LinkUnit::StaticLibrary) + return lu.output; + auto unsealed = std::filesystem::path("obj/platform") / lu.output.relative_path(); + unsealed += ".unsealed"; + return unsealed; +} + std::set packages_with_prepare_actions(const BuildPlan& plan) { std::set pkgs; for (auto const& a : plan.actions) @@ -3423,8 +3438,7 @@ std::string emit_ninja_string(const BuildPlan& plan, std::string* placements, if (platformUnit) for (auto const& in : platformInputs) ins += " " + escape_ninja_path(in); const bool sealUnit = platformSeal && lu.kind != LinkUnit::StaticLibrary; - auto unsealed = std::filesystem::path("obj/platform") / lu.output.relative_path(); - unsealed += ".unsealed"; + const auto unsealed = link_edge_output(plan, lu); // mcpp#426. `separateLinker` (link.exe) takes no driver at all, so the // question does not arise there and the msvc rules are untouched. @@ -3581,7 +3595,7 @@ std::string emit_ninja_string(const BuildPlan& plan, std::string* placements, // order-only list of the edge. if (lu.kind != LinkUnit::StaticLibrary) orderOnly += linkOrderOnly; std::string out_line = std::format("build {}{} : {}{}{}{}\n", - escape_ninja_path(sealUnit ? unsealed : lu.output), implicitOut, rule, ins, + escape_ninja_path(link_edge_output(plan, lu)), implicitOut, rule, ins, implicit.empty() ? std::string{} : " |" + implicit, orderOnly.empty() ? std::string{} : " ||" + orderOnly); if (auto flag = shared_soname_flag(lu, plan); !flag.empty()) @@ -3946,6 +3960,13 @@ std::string emit_ninja_string(const BuildPlan& plan, std::string* placements, : a.role == mcpp::manifest::BuildAction::Role::Object ? "OBJECT" : "GENERATE", a.description.empty() ? a.id : a.description)); + // A generator that leaves an output as it was -- it writes only when + // the content changed -- ends the rebuild there: ninja compares the + // outputs' mtimes after the command and prunes what depends on an + // unchanged one. Without it every run of an action dirtied the scan, + // the module and every importer below it (mcpp#874). A stamped action + // writes its stamp each run and is unaffected. + append(" restat = 1\n"); // The command's OWN emitted dependency file, when it declared one. // `a.inputs` is FIXED AT SUBMISSION, before the command has run; a // device-source compiler (glslangValidator, glslc, slangc, nvcc/clang) @@ -4166,7 +4187,7 @@ std::optional check_action_ordering(const std::string& manifest, std::set linkOutputs; for (auto const& lu : plan.linkUnits) { if (lu.kind == LinkUnit::StaticLibrary) continue; - linkOutputs.insert(escape_ninja_path(lu.output)); + linkOutputs.insert(escape_ninja_path(link_edge_output(plan, lu))); } std::map seenLinks; // link output -> edges checked for (auto line : manifest | std::views::split('\n')) { diff --git a/src/build/plan.cppm b/src/build/plan.cppm index 9890abceb..66a8fb479 100644 --- a/src/build/plan.cppm +++ b/src/build/plan.cppm @@ -4042,6 +4042,25 @@ make_plan(const mcpp::manifest::Manifest& manifest, } } + // A STATIC LIBRARY HOLDS ITS OWN PACKAGE'S OBJECTS (docs/12, "a static + // archive does not carry its dependencies' code"). Each dependency's code + // is its own package's, which a consumer's graph resolves; a program of + // this plan links objects and never reads an archive. Collected as every + // unit's objects, a workspace's member archives held its closure: 11 433 + // members for 2 666 objects, 5.3 GB (mcpp#874). An object that is not a + // compile unit's -- an `object` action's -- is the package's own and stays. + { + std::map ownerOfObject; + for (auto const& cu : plan.compileUnits) ownerOfObject.emplace(cu.object, cu.packageName); + for (auto& lu : plan.linkUnits) { + if (lu.kind != LinkUnit::StaticLibrary) continue; + std::erase_if(lu.objects, [&](const std::filesystem::path& o) { + auto it = ownerOfObject.find(o); + return it != ownerOfObject.end() && it->second != lu.package; + }); + } + } + return plan; } diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 26afcb179..4a4d3d7ca 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -865,6 +865,12 @@ export struct BuildOverrides { // not apply to them. std::map resolution_pins; bool build_program_graph = false; + // Where the build programs of this chain live: the host directory of the + // build that started it. Empty = this build's own. A build programs' graph + // whose libraries have build programs of their own builds them beside its + // own group there, and not in a copy of the host workspace under its own + // `target/` (mcpp#874). + std::filesystem::path host_programs_root; // The build targets the machine doing it: a sub-build of build programs // or of a host tool. Neither `[build] target` nor the configured default // target applies, since both name the target of the build that asked. diff --git a/src/build/prepare/build_dir.cpp b/src/build/prepare/build_dir.cpp new file mode 100644 index 000000000..5e2992e3b --- /dev/null +++ b/src/build/prepare/build_dir.cpp @@ -0,0 +1,125 @@ +// build_dir.cpp -- a configuration's build directory over time (design +// 2026-10-31 §8): the predecessor a new fingerprint of its lineage takes over, +// and what the directory records for the next build and for `mcpp clean --stale`. + +module mcpp.build.prepare; +import :state; + +import std; +import mcpp.bmi_cache.maintenance; // dir_size, human_bytes +import mcpp.build.dirlock; +import mcpp.home.files; +import mcpp.log; +import mcpp.toolchain.fingerprint; +import mcpp.ui; + +namespace mcpp::build { + +namespace { + +// A CONFIGURATION'S LINEAGE (design 2026-10-31 §8.2): the fingerprint's +// parts a developer switches between and keeps side by side -- the compiler +// family, the target, the standard library's family, the language standard and +// the configuration's flags. The rest evolve with time: the compiler's version +// and identity, the standard library's version, mcpp's version, the std +// module and the runtime contract. A new fingerprint of a lineage that has a +// directory takes that directory over instead of opening another copy of the +// build: ninja then rebuilds the edges whose commands changed and reuses the +// rest. Each mcpp upgrade used to open a full new directory (mcppls: 2.9 GB per +// configuration) and rebuild everything. +std::string lineage_text(const mcpp::toolchain::Fingerprint& fp) { + const auto stdlibFamily = fp.parts[4].substr(0, fp.parts[4].find(' ')); + return std::format("compiler {}\ntarget {}\nstdlib {}\nstandard {}\nconfiguration {}\n", + fp.parts[0], fp.parts[3], stdlibFamily, fp.parts[5], fp.parts[6]); +} + +// The toolchain's part of the evolution. Objects and BMIs of another compiler +// build are never reused, so a predecessor that differs here is removed rather +// than taken over: the build starts over in one directory instead of two. +std::string toolchain_text(const mcpp::toolchain::Fingerprint& fp) { + return std::format("compiler {}\ndriver {}\nstdlib {}\n", fp.parts[1], fp.parts[2], fp.parts[4]); +} + +std::string read_small(const std::filesystem::path& file) { + std::ifstream in(file, std::ios::binary); + return {std::istreambuf_iterator(in), {}}; +} + +} // namespace + +void adopt_predecessor_dir(const PrepareState& state, const std::filesystem::path& outputDir) { + namespace fs = std::filesystem; + std::error_code ec; + // A directory without a graph is not yet a build: the fast path takes the + // lock of the directory the record names before it finds no graph there. + if (fs::exists(outputDir / "build.ninja", ec)) return; + const auto lineage = lineage_text(state.fp); + std::optional best; + fs::file_time_type bestTime{}; + for (auto const& d : fs::directory_iterator(outputDir.parent_path(), ec)) { + std::error_code dec; + if (d.path() == outputDir || !d.is_directory(dec) + || read_small(d.path() / "lineage") != lineage) + continue; + // The one this configuration used last: build.ninja is written by every plan. + const auto t = fs::last_write_time(d.path() / "build.ninja", dec); + if (!dec && (!best || t > bestTime)) { best = d.path(); bestTime = t; } + } + if (!best) return; + // A predecessor another build holds is left to it; this build opens its own. + auto held = mcpp::home::FileLock::try_acquire(*best, mcpp::build::dirlock::kLockName); + if (!held) return; + const auto shown = best->lexically_relative(state.workRoot).generic_string(); + const bool sameToolchain = read_small(*best / "toolchain") == toolchain_text(state.fp); + if (sameToolchain) { + // Its entries move in, one rename each, beside the lock this build + // may already have created there. + fs::create_directories(outputDir, ec); + const auto lock = std::string(mcpp::build::dirlock::kLockName); + for (auto const& e : fs::directory_iterator(*best, ec)) { + const auto name = e.path().filename().string(); + if (name == lock || name == lock + ".owner") continue; + std::error_code mec; + fs::remove_all(outputDir / name, mec); + fs::rename(e.path(), outputDir / name, mec); + } + mcpp::log::info("build/dir", std::format("{} continues as {}", shown, + outputDir.lexically_relative(state.workRoot).generic_string())); + } + held.reset(); + const auto bytes = sameToolchain ? 0 : mcpp::bmi_cache::dir_size(*best); + fs::remove_all(*best, ec); + if (!sameToolchain && !ec) + mcpp::ui::status("Replaced", std::format( + "{} ({}): the toolchain changed, and nothing compiled there is reused", shown, + mcpp::bmi_cache::human_bytes(bytes))); +} + +// What this configuration uses beside its directory (`mcpp clean --stale`), and +// its lineage and toolchain, which the next fingerprint of the configuration +// reads to take the directory over. +void record_build_dir(const PrepareState& state, const std::filesystem::path& outputDir) { + std::string uses; + for (auto const& h : state.usedProgramOut) uses += "out " + h + "\n"; + for (auto const& g : state.usedHostGroups) uses += "host " + g + "\n"; + // The build-cache entries it used: the sweep after a build marks them used + // for a build that replays this graph without planning. + auto records = state.usedCacheRecords; + if (state.describedStdModule) + records.insert((state.describedStdModule->cacheDir / "std-module.json").string()); + for (auto const& r : records) uses += "cache " + r + "\n"; + std::error_code uec; + std::filesystem::create_directories(outputDir, uec); + std::ifstream in(outputDir / "uses.list", std::ios::binary); + const std::string was{std::istreambuf_iterator(in), {}}; + in.close(); + // Written when it is absent too: an empty list states that nothing is used. + if (was != uses || !std::filesystem::exists(outputDir / "uses.list", uec)) + std::ofstream(outputDir / "uses.list", std::ios::binary) << uses; + for (auto const& [name, text] : {std::pair{"lineage", lineage_text(state.fp)}, + std::pair{"toolchain", toolchain_text(state.fp)}}) + if (read_small(outputDir / name) != text) + std::ofstream(outputDir / name, std::ios::binary) << text; +} + +} // namespace mcpp::build diff --git a/src/build/prepare/features.cpp b/src/build/prepare/features.cpp index f0eb91442..faae728d6 100644 --- a/src/build/prepare/features.cpp +++ b/src/build/prepare/features.cpp @@ -1348,6 +1348,42 @@ step6_check_tool_self_request(PrepareState& state, HostToolCtx& ctx) { return {}; } +// The dependency closure a host tool's own build resolves (mcpp#874): its +// package's `[dependencies]`, transitively, read from the manifests. A path +// package is its version and its tree's stamp, and is walked into; a git or +// index package is what the manifest asks for. Dev-dependencies are never in +// it: the tool's build builds no tests. +static std::vector tool_upstream_keys(const std::filesystem::path& root, + const mcpp::manifest::Manifest& manifest) { + namespace fs = std::filesystem; + std::vector keys; + std::error_code ec; + std::set seen{fs::weakly_canonical(root, ec)}; + std::vector> work{{root, manifest}}; + while (!work.empty()) { + auto [dir, m] = std::move(work.back()); + work.pop_back(); + for (auto const& [name, spec] : m.dependencies) { + if (!spec.git.empty()) { + keys.push_back(std::format("{}@git.{}#{}", name, spec.git, spec.gitRev)); + continue; + } + if (spec.path.empty()) { + keys.push_back(std::format("{}@{}", name, spec.version)); + continue; + } + const auto at = fs::weakly_canonical(dir / spec.path, ec); + auto dm = mcpp::manifest::load(at / "mcpp.toml", {.insideWorkspace = true}); + keys.push_back(std::format("{}@{}+path.{}", name, dm ? dm->package.version : std::string{}, + mcpp::build::tool_store::tree_stamp(at))); + if (dm && seen.insert(at).second) work.emplace_back(at, std::move(*dm)); + } + } + std::ranges::sort(keys); + keys.erase(std::unique(keys.begin(), keys.end()), keys.end()); + return keys; +} + // Returns true when the tool is already resolved (a valid store entry, or a // plan-only deferral), in which case the caller's per-tool work is done. static std::expected @@ -1428,10 +1464,17 @@ step6_resolve_tool_key(PrepareState& state, HostToolCtx& ctx) { // but a path dependency can: bump something two levels down // and the tool's direct list is unchanged, so a stale binary // stays in the store — a silently wrong artifact. - for (auto up : dg::transitive_dependencies(state.dependencyEdges, ctx.depIdx)) - ctx.key.upstreamKeys.push_back(std::format("{}@{}", - state.packages[up].manifest.package.name, - source_keyed_version(up))); + // + // Read from the manifests, as the tool's own build resolves them, and + // not from this graph (mcpp#874). How much of the tool package's closure + // this graph walked depends on the command and the member it serves: a + // package only a build dependency's `tools` reach is not walked at all + // (the closure was empty under `mcpp test --workspace` and complete under + // `mcpp build`), and a root's dev-dependencies are walked while its tests + // are built. Either way the key changed between commands, the tool's path + // with it, and every action that ran the tool ran again. + ctx.key.upstreamKeys = tool_upstream_keys(state.packages[ctx.depIdx].root, + state.packages[ctx.depIdx].manifest); std::ranges::sort(ctx.key.upstreamKeys); ctx.cacheRoot = mcpp::home::cache_root(); @@ -1442,6 +1485,8 @@ step6_resolve_tool_key(PrepareState& state, HostToolCtx& ctx) { if (mcpp::build::tool_store::entry_valid(ctx.entry, ctx.key, ctx.toolName, exeSuffix)) { + mcpp::build::tool_store::touch(ctx.entry); + state.usedCacheRecords.insert((ctx.entry / "entry.json").string()); step6_record_tool_provision(state, ctx, ctx.binOut); return true; } @@ -1660,7 +1705,16 @@ step6_publish_tool(PrepareState& state, HostToolCtx& ctx) { "publishing host tool '{}:{}' failed: {}", ctx.depName, ctx.toolName, cpEc.message())); } - mcpp::build::tool_store::write_entry(ctx.entry, ctx.key); + // A path package's tree, so that its superseded entries are collected. + std::filesystem::path toolSource; + if (ctx.depIdx >= 1 && ctx.depIdx - 1 < state.dep_cache_identities.size() + && state.dep_cache_identities[ctx.depIdx - 1].sourceKind == "path") { + const auto& id = state.dep_cache_identities[ctx.depIdx - 1]; + toolSource = id.sourceRef.empty() ? state.packages[ctx.depIdx].root + : std::filesystem::path(id.sourceRef); + } + mcpp::build::tool_store::write_entry(ctx.entry, ctx.key, toolSource); + state.usedCacheRecords.insert((ctx.entry / "entry.json").string()); // The sub-build tree is large (protoc is several hundred // objects) and the key covers every input, so a hit never // needs it again. Removes only THIS consumer's scratch. diff --git a/src/build/prepare/graph.cpp b/src/build/prepare/graph.cpp index 82e32c3cf..770e90c78 100644 --- a/src/build/prepare/graph.cpp +++ b/src/build/prepare/graph.cpp @@ -2173,8 +2173,8 @@ step4b_define_provisioning_closures(PrepareState& state) { ? pm.package.name : pm.package.namespace_ + "." + pm.package.name; for (auto& c : name) if (!std::isalnum(static_cast(c)) && c != '.' && c != '-' && c != '_') c = '_'; - return base / "target" / ".build-mcpp" / "out" - / mcpp::toolchain::hash_string(id) / (name.empty() ? std::string("root") : name); + return base / "target" / ".build-mcpp" / "out" // named in `uses.list` (mcpp#874) + / *state.usedProgramOut.insert(mcpp::toolchain::hash_string(id)).first / (name.empty() ? std::string("root") : name); }; state.adoptActionOutputs = [](mcpp::manifest::Manifest& mm, diff --git a/src/build/prepare/host_graph.cpp b/src/build/prepare/host_graph.cpp index ff5813f08..23e663d55 100644 --- a/src/build/prepare/host_graph.cpp +++ b/src/build/prepare/host_graph.cpp @@ -241,6 +241,7 @@ std::expected host_toolchain(PrepareState& state) { } fs::path host_root(const PrepareState& state) { + if (!state.overrides.host_programs_root.empty()) return state.overrides.host_programs_root; return state.workRoot / "target" / ".build-mcpp" / "host"; } @@ -537,7 +538,11 @@ static void place_program_copy(const PrepareState& state, std::size_t i, const f if (ec) return; if (std::error_code dc; fs::last_write_time(dest, dc) == src && !dc) return; fs::create_directories(dest.parent_path(), ec); - fs::copy_file(bin, dest, fs::copy_options::overwrite_existing, ec); + // A link where the file system allows one: the program is the host + // group's binary, and a copy beside every record doubled each one (mcpp#874). + fs::remove(dest, ec); + fs::create_hard_link(bin, dest, ec); + if (ec) fs::copy_file(bin, dest, fs::copy_options::overwrite_existing, ec); if (!ec) fs::last_write_time(dest, src, ec); if (ec) mcpp::log::verbose("build-program", std::format( "cannot place {} beside its record: {}", path_text(dest), ec.message())); @@ -747,6 +752,7 @@ build_group(PrepareState& state, const fs::path& dir, const std::vector std::make_shared(state.runtimeBindingSnapshot); sub.resolution_pins = resolution_pins(state); sub.build_program_graph = true; + sub.host_programs_root = host_root(state); sub.project_index_root = state.projectIndexRoot(); for (auto const& p : all) sub.workspace_members.push_back(p.member); @@ -1008,7 +1014,13 @@ ensure_build_programs(PrepareState& state, const std::vector& packa } for (auto& [standard, programs] : groups) { - const auto dir = host_root(state) / programs.front().group; + // A group the build programs' graph needs is its sibling, named by its + // depth: the group that asked holds its own directory's lock while it + // builds. + const auto dir = host_root(state) / (state.overrides.build_program_graph + ? std::format("{}.{}", programs.front().group, state.overrides.tool_depth) + : programs.front().group); + state.usedHostGroups.insert(programs.front().group); std::error_code ec; fs::create_directories(dir, ec); // Two configuration groups of one command may share a host toolchain diff --git a/src/build/prepare/plan.cpp b/src/build/prepare/plan.cpp index 14975b910..ef77dd1be 100644 --- a/src/build/prepare/plan.cpp +++ b/src/build/prepare/plan.cpp @@ -1958,6 +1958,7 @@ static std::expected step13_dependency_cache(PrepareState& st } } mcpp::bmi_cache::touch_accessed(key); + state.usedCacheRecords.insert(key.entryFile().string()); ctx.cachedDeps.push_back({depName, depVer, unitIdx.size()}); continue; // no populate task; it is already cached } @@ -1984,6 +1985,35 @@ static std::expected step13_dependency_cache(PrepareState& st return {}; } +// `split_debuginfo = "unpacked"` (design 2026-10-31 §4.1 (b)): a unit +// compiled in this build directory keeps its debug information beside its +// object (`-gsplit-dwarf`), and the artifacts carry a skeleton that names it, +// so a closure's DWARF is no longer copied into every program and test. An +// index package's units are excluded: their objects enter the global cache and +// are staged by other projects, whose skeletons would name this build +// directory -- compilers apply no prefix map to the name of a `.dwo`. On ELF +// targets with a loader, ok images included; a freestanding image keeps it in +// band for the debuggers of its boards. +static void step13_split_debuginfo(PrepareState& state, BuildContext& ctx) { + if (!state.m->buildConfig.splitDebuginfo || !state.tc) return; + const auto t = mcpp::toolchain::triple::parse(state.tc->targetTriple); + const auto target = t ? *t : mcpp::toolchain::triple::host_triple(); + if (target.object_format() != mcpp::toolchain::triple::ObjectFormat::Elf + || target.is_freestanding()) + return; + std::set fromIndex; + for (std::size_t i = 1; i < state.packages.size(); ++i) + if (i - 1 < state.dep_cache_identities.size() + && state.dep_cache_identities[i - 1].sourceKind == "version") + fromIndex.insert(mcpp::build::qualified_package_name(state.packages[i].manifest)); + const auto split = mcpp::manifest::flag_element("-gsplit-dwarf"); + for (auto& cu : ctx.plan.compileUnits) { + if (cu.servedFromCache || fromIndex.contains(cu.packageName)) continue; + cu.packageCflags.push_back(split); + cu.packageCxxflags.push_back(split); + } +} + static std::expected step13_runtime_provider_overrides(PrepareState& state, BuildContext& ctx) { // Apply [runtime.] provider = "" overrides. Canonical // identity wins; the old short spelling is accepted only when it denotes @@ -2186,6 +2216,7 @@ std::expected phase13_finish(PrepareState& state) { "effect denied: write-project -- the build writes {}, and the caller denied writing " "the project (--deny write-project)", ctx.outputDir.generic_string())); } + if (!state.overrides.plan_only) adopt_predecessor_dir(state, ctx.outputDir); if (!state.overrides.plan_only) ctx.buildLock = mcpp::build::dirlock::acquire(ctx.outputDir); // Each step states its duration under `build/stage` in the log file, as @@ -2226,6 +2257,7 @@ std::expected phase13_finish(PrepareState& state) { return std::unexpected(r.error()); if (auto r = timed("windows resources", [&] { return step13_windows_resources(state, ctx); }); !r) return std::unexpected(r.error()); + timed("split debuginfo", [&] { step13_split_debuginfo(state, ctx); return 0; }); if (auto r = timed("dependency cache", [&] { return step13_dependency_cache(state, ctx); }); !r) return std::unexpected(r.error()); if (auto r = timed("lockfile", [&] { return step13_lockfile(state, ctx); }); !r) @@ -2238,7 +2270,8 @@ std::expected phase13_finish(PrepareState& state) { if (auto r = timed("empty link check", [&] { return step13_empty_link_check(state, ctx); }); !r) return std::unexpected(r.error()); timed("report packages", [&] { step13_report_packages(state, ctx); return 0; }); - + // Last: what the plan used is known only now (the dependency cache's hits). + if (!state.overrides.plan_only) record_build_dir(state, ctx.outputDir); ctx.planNotes = std::move(state.planNotes); return ctx; diff --git a/src/build/prepare/state.cppm b/src/build/prepare/state.cppm index d4ae73c97..c98c7dbfc 100644 --- a/src/build/prepare/state.cppm +++ b/src/build/prepare/state.cppm @@ -694,6 +694,16 @@ struct PrepareState { std::function&)> programOutDir; + // What this configuration uses outside its own directory, written into it + // as `uses.list` so that `mcpp clean --stale` keeps these and collects the + // rest (mcpp#874): the `.build-mcpp/out/` of each program it ran and + // the `.build-mcpp/host/` of the build programs it built. + std::set usedProgramOut; + std::set usedHostGroups; + // The record files of the build-cache entries this configuration used (a + // package's, a host tool's), so that a build that replays the graph without + // planning still counts as their use (design 2026-10-31 §4.6). + std::set usedCacheRecords; std::function fillXpkgDirs; std::function step1_define_early_toolchain_closures(Pr // its paths; what build, run and pack produce is trimmed. state.m->buildConfig.trimPaths = (pr.trimPathsDeclared ? pr.trimPaths : shipping) && !state.includeDevDeps; + // `split_debuginfo` (design 2026-10-31 §4.1 (b)): unpacked for the + // profiles a developer debugs. A debugger finds a unit's `.dwo` through + // the build directory the skeleton names, which `trim_paths` replaces. + if (pr.splitDebuginfoDeclared && pr.splitDebuginfo != "unpacked" && pr.splitDebuginfo != "off") + return std::unexpected(std::format( + "[profile.{}] split_debuginfo = \"{}\": expected \"unpacked\" or \"off\"", + pname, pr.splitDebuginfo)); + const bool unpacked = pr.splitDebuginfoDeclared ? pr.splitDebuginfo == "unpacked" + : (pname == "dev" || pname == "debug"); + state.m->buildConfig.splitDebuginfo = unpacked && pr.debug && !state.m->buildConfig.trimPaths; // SAID ONCE PER HOME, the first time a shipping profile takes the new // defaults without stating them: the artifact changed, and the line // that changes it back is one key. diff --git a/src/build/prepare_inputs.cppm b/src/build/prepare_inputs.cppm index 81fa0396b..a8a5bd6bb 100644 --- a/src/build/prepare_inputs.cppm +++ b/src/build/prepare_inputs.cppm @@ -576,6 +576,7 @@ std::string canonical_configuration_flags(const mcpp::manifest::Manifest& root, // Only when on: a profile that has neither keeps the identity it had. if (root.buildConfig.ndebug) s += " ndebug=1"; if (root.buildConfig.trimPaths) s += " trim_paths=1"; + if (root.buildConfig.splitDebuginfo) s += " split_debuginfo=1"; if (!root.buildConfig.dependencyLinkage.empty()) { s += " deplinkage="; s += root.buildConfig.dependencyLinkage; diff --git a/src/build/stage.cppm b/src/build/stage.cppm index 34350c263..9501940a7 100644 --- a/src/build/stage.cppm +++ b/src/build/stage.cppm @@ -381,6 +381,22 @@ std::expected stage_file(const std::filesystem::path& && (same_content(src, dst) || (opts.verify == Verify::Size && !std::ifstream(dst, std::ios::binary).is_open())); if (equivalent) { + // A shared library staged as a copy -- by a release before + // placements were linked, or where a link once failed -- is placed + // again by a link, so a library every member's directory loads is + // one file and not one per member (mcpp#874: 650 MB of copies + // across 27 product directories). A link that cannot be made + // leaves the equal copy where it is. + std::error_code qec; + if (links_on_placement(src) && !std::filesystem::equivalent(src, dst, qec) && !qec) { + auto tmp = temp_sibling(dst); + std::error_code lec, rec; + std::filesystem::create_hard_link(src, tmp, lec); + if (!lec) { + std::filesystem::rename(tmp, dst, rec); + if (rec) std::filesystem::remove(tmp, rec); + } + } // Timestamps left untouched on purpose — see module comment. return StageOutcome{.copied = false}; } diff --git a/src/cli.cppm b/src/cli.cppm index 4ae83812e..79fd110a4 100644 --- a/src/cli.cppm +++ b/src/cli.cppm @@ -41,6 +41,7 @@ import mcpplibs.os.posix; import mcpp.home.files; import mcpp.build.progress; // __action reports its start import mcpp.build.dirlock; // the build directory lock names this command +import mcpp.build.execute; // the sweep after a build (design 2026-10-31 §8.3) import mcpp.ui; import mcpp.log; import mcpp.home; // --system: the system home @@ -700,7 +701,11 @@ int run(int argc, char** argv) { .help("Play a game in the status row while it builds: --play-game=snake|stack|runner, or one at random")) .option(cl::Option("message-format").takes_value().value_name("FMT") .help("human (default) | json: NDJSON events on stdout, then the mcpp.build envelope")) - .action(wrap_rc(cmd_build))) + .action(wrap_rc([](const cl::ParsedArgs& p) { + const int rc = cmd_build(p); + if (rc == 0) mcpp::build::sweep_after_build(); + return rc; + }))) .subcommand(cl::App("run") .description("Build + run a binary target (after `--`, args are passed to it)") // Named `bin`, NOT `target`, and the rename is load-bearing. @@ -869,13 +874,16 @@ int run(int argc, char** argv) { .option(cl::Option("play-game") .help("Play a game in the status row while it builds: --play-game=snake|stack|runner, or one at random")) .action(wrap_rc([&passthrough](const cl::ParsedArgs& p) { - return cmd_test(p, std::span(passthrough)); + const int rc = cmd_test(p, std::span(passthrough)); + if (rc == 0) mcpp::build::sweep_after_build(); + return rc; }))) .subcommand(cl::App("clean") - .description("Remove target/, or with --stale only the fingerprint directories under it that no recorded build still uses") + .description("Remove target/, or with --stale only what no configuration still uses: fingerprint directories, generated sources, build-program groups") .option(cl::Option("bmi-cache").help("Also wipe the global build cache (see `mcpp cache clean`)")) .option(cl::Option("stale").help("Only remove target/// directories that no recorded build considers current")) .option(cl::Option("dry-run").help("List what would be removed and delete nothing (implies --stale)")) + .option(cl::Option("report").help("Show where target/ holds its bytes, per configuration and area; removes nothing")) .option(cl::Option("older-than").takes_value().value_name("DURATION") .help("Keep unrecorded directories written more recently than this, e.g. 12h, 3d (default 1d; 0 keeps none; implies --stale)")) .action(wrap_rc(cmd_clean))) @@ -1181,7 +1189,7 @@ int run(int argc, char** argv) { .option(cl::Option("older-than").takes_value().value_name("N{s|m|h|d}") .help("Age threshold (e.g. 30d)"))) .subcommand(cl::App("gc") - .description("LRU-collect package entries to a size and/or age budget") + .description("LRU-collect package and host tool entries to a size and/or age budget") .option(cl::Option("max-size").takes_value().value_name("N{MiB|GiB}") .help("Keep the package cache under this size (e.g. 5GiB)")) .option(cl::Option("older-than").takes_value().value_name("N{s|m|h|d}") @@ -1190,7 +1198,8 @@ int run(int argc, char** argv) { .description("Drop cache entries (default: package entries only)") .option(cl::Option("deps").help("Drop package entries (default)")) .option(cl::Option("std").help("Drop std module entries")) - .option(cl::Option("all").help("Drop both")) + .option(cl::Option("tools").help("Drop host tool entries")) + .option(cl::Option("all").help("Drop every kind of entry")) .option(cl::Option("legacy") .help("Remove the unused pre-v1 cache at $MCPP_HOME/bmi"))) .subcommand(cl::App("verify") @@ -1278,9 +1287,13 @@ int run(int argc, char** argv) { .option(cl::Option("format").takes_value().value_name("json") .help("Machine-readable output (enveloped; see docs/50-machine-output.md)"))) .subcommand(cl::App("config") - .description("Show or modify mcpp's private xlings configuration") + .description("Show or modify mcpp's configuration: the xlings mirror and the retention of build outputs") .option(cl::Option("mirror").takes_value().value_name("auto|CN|GLOBAL") - .help("Set xlings mirror for mcpp's private registry"))) + .help("Set xlings mirror for mcpp's private registry")) + .option(cl::Option("cache-unused-days").takes_value().value_name("DAYS") + .help("Collect build-cache entries no build used for DAYS days (default 30; 0 keeps all)")) + .option(cl::Option("keep-days").takes_value().value_name("DAYS") + .help("Sweep a project's build directories no configuration used for DAYS days (default 3; 0 keeps all)"))) .subcommand(cl::App("version") .description("Show mcpp version")) .subcommand(cl::App("explain") diff --git a/src/cli/cmd_build.cppm b/src/cli/cmd_build.cppm index 206166990..1fbeaabc9 100644 --- a/src/cli/cmd_build.cppm +++ b/src/cli/cmd_build.cppm @@ -1553,6 +1553,7 @@ export int cmd_clean(const mcpplibs::cmdline::ParsedArgs& parsed) { // under target/, silently, which is the outcome `--stale` exists to avoid. const bool dryRun = parsed.is_flag_set("dry-run"); const auto olderThan = parsed.value("older-than"); + if (parsed.is_flag_set("report")) return mcpp::build::clean_report(); if (parsed.is_flag_set("stale") || dryRun || olderThan) { if (parsed.is_flag_set("bmi-cache")) { std::println(stderr, "error: --stale/--dry-run/--older-than cannot be combined with " diff --git a/src/cli/cmd_cache.cppm b/src/cli/cmd_cache.cppm index 5a1212002..fb8ac1b78 100644 --- a/src/cli/cmd_cache.cppm +++ b/src/cli/cmd_cache.cppm @@ -63,6 +63,7 @@ export int cmd_cache_gc(const mcpplibs::cmdline::ParsedArgs& parsed) { export int cmd_cache_clean(const mcpplibs::cmdline::ParsedArgs& parsed) { return mcpp::bmi_cache::cache_clean(parsed.is_flag_set("deps"), parsed.is_flag_set("std"), + parsed.is_flag_set("tools"), parsed.is_flag_set("all"), parsed.is_flag_set("legacy")); } diff --git a/src/cli/cmd_self.cppm b/src/cli/cmd_self.cppm index edf348365..671da978f 100644 --- a/src/cli/cmd_self.cppm +++ b/src/cli/cmd_self.cppm @@ -202,7 +202,13 @@ export int cmd_self_init(const mcpplibs::cmdline::ParsedArgs& parsed) { } export int cmd_self_config(const mcpplibs::cmdline::ParsedArgs& parsed) { - return mcpp::doctor::self_config(parsed.option_or_empty("mirror").value()); + auto days = [&](const char* name) -> std::optional { + auto v = parsed.value(name); + if (!v) return std::nullopt; + try { return std::stoll(*v); } catch (...) { return -1; } // refused as not a count + }; + return mcpp::doctor::self_config(parsed.option_or_empty("mirror").value(), + days("cache-unused-days"), days("keep-days")); } // `mcpp self store dedupe [--apply]`. diff --git a/src/config.cppm b/src/config.cppm index 1b4067740..b3c02af7d 100644 --- a/src/config.cppm +++ b/src/config.cppm @@ -113,6 +113,12 @@ struct GlobalConfig { // Whether a build refreshes is decided by mcpp.pm.index_refresh from local // resolvability, not from a clock. std::int64_t searchTtlSeconds = 3600; + // [cache] unused_days: the build cache's entries no build used for this + // many days are collected after a build (design 2026-10-31 §4.6; 0 keeps). + std::int64_t cacheUnusedDays = 30; + // [build] keep_days: a project's build directories no configuration used + // for this many days are swept after a build (§8.4; 0 keeps). + std::int64_t buildKeepDays = 3; // From config.toml [index] auto_refresh // Machine-level opt-out from the automatic on-miss index refresh. Distinct @@ -393,6 +399,12 @@ void canonicalize_legacy_index_names(GlobalConfig& cfg); std::expected write_default_toolchain(const GlobalConfig& cfg, std::string_view spec); +// One ` = ` under `[
]` of config.toml, the other lines +// as they were (`mcpp self config --cache-unused-days`, `--keep-days`). +std::expected +write_config_value(const GlobalConfig& cfg, std::string_view section, std::string_view key, + std::string_view literal); + // Persist [toolchain].default_target ("" = host / clear). std::expected write_default_target(const GlobalConfig& cfg, std::string_view targetTriple); @@ -458,9 +470,15 @@ artifact = { GLOBAL = "https://github.com/xlings-res/mcpp-index", CN = "https:// # Bounds `mcpp search` and the age shown by `mcpp index status`. It is NOT # what decides whether a build refreshes the index — see [index] auto_refresh. search_ttl_seconds = 3600 +# After a build, at most once a day, the build cache's entries (packages, +# std modules, host tools) unused for this many days are removed. 0 keeps all. +unused_days = 30 [build] default_jobs = 0 +# After a build, at most once a day, the build directories of this project that +# no configuration used for this many days are removed. 0 keeps all. +keep_days = 3 # Publish each project's compile database at this path under its root, for a # project whose mcpp.toml does not say ([build] compile_commands). Unset: the # database stays in the build directory (target///). @@ -980,6 +998,8 @@ std::expected load_or_init( cfg.indexRefreshTimeoutSeconds = *t; mcpp::provider::set_index_refresh_timeout(std::chrono::seconds{cfg.indexRefreshTimeoutSeconds}); cfg.searchTtlSeconds = doc->get_int("cache.search_ttl_seconds").value_or(3600); + cfg.cacheUnusedDays = doc->get_int("cache.unused_days").value_or(30); + cfg.buildKeepDays = doc->get_int("build.keep_days").value_or(3); cfg.defaultJobs = doc->get_int("build.default_jobs").value_or(0); cfg.compileCommands = doc->get_string("build.compile_commands").value_or(""); cfg.defaultToolchain = doc->get_string("toolchain.default").value_or(""); @@ -1409,26 +1429,35 @@ bool ensure_project_index_dir( return true; } -// M5.5: persist [toolchain].default into config.toml without disturbing +// M5.5: persist one key of a section into config.toml without disturbing // other fields. Naive: read text, replace/insert one line. std::expected -write_toolchain_key(const GlobalConfig& cfg, std::string_view key, - std::string_view value) { +write_config_value(const GlobalConfig& cfg, std::string_view section, std::string_view key, + std::string_view literal) { std::ifstream is(cfg.configFile); if (!is) return std::unexpected(ConfigError{ std::format("cannot open '{}'", cfg.configFile.string())}); std::stringstream ss; ss << is.rdbuf(); std::string text = ss.str(); - std::string line = std::format("{} = \"{}\"\n", key, value); - - auto sectionPos = text.find("[toolchain]"); + std::string line = std::format("{} = {}\n", key, literal); + const auto header = std::format("[{}]", section); + + // The header at the start of a line, so `[toolchain]` is not found inside + // `[toolchain.x]` or a comment. + auto sectionPos = std::string::npos; + for (auto p = text.find(header); p != std::string::npos; p = text.find(header, p + 1)) + if ((p == 0 || text[p - 1] == '\n') + && (p + header.size() == text.size() || text[p + header.size()] == '\n')) { + sectionPos = p; + break; + } if (sectionPos == std::string::npos) { - // Append a [toolchain] block at end. + // Append the section's block at end. if (!text.empty() && text.back() != '\n') text += '\n'; - text += std::format("\n[toolchain]\n{}", line); + text += std::format("\n{}\n{}", header, line); } else { - // Locate an existing ` = ...` line within [toolchain]. Match at + // Locate an existing ` = ...` line within the section. Match at // line starts with a word boundary after the key, so `default` never // matches the `default_target` line (and vice versa). auto eol = text.find('\n', sectionPos); @@ -1471,6 +1500,11 @@ write_toolchain_key(const GlobalConfig& cfg, std::string_view key, return {}; } +std::expected +write_toolchain_key(const GlobalConfig& cfg, std::string_view key, std::string_view value) { + return write_config_value(cfg, "toolchain", key, std::format("\"{}\"", value)); +} + std::expected write_default_toolchain(const GlobalConfig& cfg, std::string_view spec) { return write_toolchain_key(cfg, "default", spec); diff --git a/src/doctor.cppm b/src/doctor.cppm index fecbaba81..64f8608ff 100644 --- a/src/doctor.cppm +++ b/src/doctor.cppm @@ -1961,7 +1961,35 @@ export int store_dedupe(bool apply) { } // `mcpp self config [--mirror CN|GLOBAL]` (mirror = raw option value). -export int self_config(std::string mirror) { +// `--cache-unused-days` and `--keep-days` (design 2026-10-31 §4.6, §8.4): the +// retention of the shared build cache and of a project's build directories, +// in config.toml. Negative values are refused; 0 keeps everything. +export int self_config(std::string mirror, std::optional cacheUnusedDays = {}, + std::optional keepDays = {}) { + if (cacheUnusedDays || keepDays) { + auto cfg = mcpp::config::load_or_init(/*quiet=*/false, {}, {}, + mcpp::config::Initialization::Local); + if (!cfg) { + mcpp::ui::error(cfg.error().message); + return 4; + } + for (auto const& [value, section, key] : {std::tuple{cacheUnusedDays, "cache", "unused_days"}, + std::tuple{keepDays, "build", "keep_days"}}) { + if (!value) continue; + if (*value < 0) { + mcpp::ui::error(std::format("{}.{} = {}: expected a number of days, 0 or more", + section, key, *value)); + return 2; + } + if (auto w = mcpp::config::write_config_value(*cfg, section, key, std::to_string(*value)); !w) { + mcpp::ui::error(w.error().message); + return 1; + } + mcpp::ui::status("Configured", std::format("{}.{} = {}{}", section, key, *value, + *value == 0 ? " (keep all)" : " days")); + } + if (mirror.empty()) return 0; + } if (!mirror.empty()) { mirror = upper_ascii(std::move(mirror)); if (mirror != "CN" && mirror != "GLOBAL" && mirror != "AUTO") { @@ -1985,6 +2013,10 @@ export int self_config(std::string mirror) { auto env = mcpp::config::make_xlings_env(*cfg); if (mirror.empty()) { report_home(*cfg); + mcpp::ui::status("Retention", std::format( + "build cache: entries unused for {} days ([cache] unused_days); a project's build " + "directories: unused for {} days ([build] keep_days)", cfg->cacheUnusedDays, + cfg->buildKeepDays)); auto rc = mcpp::provider::open(env)->show_config(); return rc == 0 ? 0 : 1; } diff --git a/src/home/files.cpp b/src/home/files.cpp index fea364db7..bd20477e4 100644 --- a/src/home/files.cpp +++ b/src/home/files.cpp @@ -106,4 +106,13 @@ std::string untrusted_because(const std::filesystem::path& dir) { return {}; } +void mark_used(const std::filesystem::path& recordFile) { + namespace fs = std::filesystem; + std::error_code ec; + const auto now = fs::file_time_type::clock::now(); + const auto was = fs::last_write_time(recordFile, ec); + if (ec || now - was < std::chrono::hours(12)) return; + fs::last_write_time(recordFile, now, ec); +} + } // namespace mcpp::home diff --git a/src/home/files.cppm b/src/home/files.cppm index c074f81f9..8a5156b01 100644 --- a/src/home/files.cppm +++ b/src/home/files.cppm @@ -74,6 +74,14 @@ bool replace_file(const std::filesystem::path& source, const std::filesystem::pa // that person can replace at any time. std::string untrusted_because(const std::filesystem::path& dir); +// THE LAST USE OF A STORED ENTRY THAT NEVER CHANGES (design 2026-10-31 §4.6): +// a build cache entry, a host tool, a std module. Its files keep the time they +// were written, and a file system mounted `noatime` or `relatime` does not keep +// reads, so a use is stated: the time of the entry's record file is set to +// now, at most once in 12 hours, so a use costs one `stat`. `mcpp cache` and +// the collection after a build read this time. +void mark_used(const std::filesystem::path& recordFile); + } // namespace mcpp::home // The one primitive per system (src/native//lock.cpp). Not exported. diff --git a/src/project.cppm b/src/project.cppm index b5006b1b3..304ad22fc 100644 --- a/src/project.cppm +++ b/src/project.cppm @@ -368,6 +368,10 @@ void merge_layer_root_position(mcpp::manifest::Manifest& member, to.trimPaths = from.trimPaths; to.trimPathsDeclared = true; } + if (!to.splitDebuginfoDeclared && from.splitDebuginfoDeclared) { + to.splitDebuginfo = from.splitDebuginfo; + to.splitDebuginfoDeclared = true; + } if (!to.dependencyLinkageDeclared && from.dependencyLinkageDeclared) { to.dependencyLinkage = from.dependencyLinkage; to.dependencyLinkageDeclared = true; diff --git a/src/toolchain/probe.cppm b/src/toolchain/probe.cppm index df24801da..f93516b7c 100644 --- a/src/toolchain/probe.cppm +++ b/src/toolchain/probe.cppm @@ -32,6 +32,15 @@ std::expected run_capture(const std::vector> v; + if (v == "yes" || v == "no") return v == "yes"; + } + std::filesystem::create_directories(work, lec); + const auto src = work / "empty.c"; + if (!std::filesystem::exists(src, lec)) std::ofstream(src, std::ios::binary).flush(); + const auto obj = work / "empty.o"; + const auto out = work / "empty.so"; + auto compiled = mcpp::run::execute( + {compiler.string(), "--target=x86_64-unknown-linux-gnu", "-g", "-c", "-x", "c", + src.string(), "-o", obj.string()}, + {.env = env, .host_tool = true, .streams = mcpp::run::Streams::merged, + .report_spawn_failure = false}); + if (compiled.exit_code != 0) return false; // not asked; nothing kept + std::vector link{lld.string()}; + if (name.starts_with("lld")) { link.push_back("-flavor"); link.push_back("gnu"); } + for (auto const& a : {std::string("--compress-debug-sections=zlib"), std::string("-shared"), + obj.string(), std::string("-o"), out.string()}) + link.push_back(a); + auto linked = mcpp::run::execute(link, {.env = env, .host_tool = true, + .streams = mcpp::run::Streams::merged, + .report_spawn_failure = false}); + const bool yes = linked.exit_code == 0; + mcpp::log::verbose("toolchain", std::format("{} writes compressed debug sections: {}{}", + lld.string(), yes ? "yes" : "no", + yes ? std::string{} : " (" + first_line_of(linked.output) + ")")); + if (mcpp::effects::writes_global_cache()) std::ofstream(answer) << (yes ? "yes\n" : "no\n"); + return yes; + } + } + // An empty C file the driver may name: `-###` reads no input, but a driver + // checks that it exists. + const auto empty = mcpp::home::root() / "cache" / "probe" / "empty.c"; + std::error_code ec; + if (!std::filesystem::exists(empty, ec)) { + std::filesystem::create_directories(empty.parent_path(), ec); + std::ofstream(empty, std::ios::binary).flush(); + } + auto r = run_capture({compiler.string(), "-gz=zlib", "-###", "-c", "-x", "c", + empty.string(), "-o", (empty.parent_path() / "empty.o").string()}, + env, /*withErrors=*/true); + if (!r) return false; + return r->find("cannot compress debug sections") == std::string::npos + && r->find("-gz is not supported") == std::string::npos + && r->find("unsupported option '-gz") == std::string::npos; +} + std::expected run_capture(const std::vector& argv, const mcpp::run::Env& env, bool withErrors) { const auto key = probe_cache_key(argv, env, withErrors); diff --git a/src/toolchain/stdmod.cppm b/src/toolchain/stdmod.cppm index 564be3826..a5b60c968 100644 --- a/src/toolchain/stdmod.cppm +++ b/src/toolchain/stdmod.cppm @@ -32,6 +32,7 @@ export module mcpp.toolchain.stdmod; import std; import mcpp.home; +import mcpp.home.files; // mark_used import mcpp.libs.json; import mcpplibs.os.platform; import mcpp.build.command_line; @@ -477,6 +478,8 @@ std::expected ensure_built( && std::filesystem::exists(sm.objectPath) && metadata_matches(metaPath, metadata); bool rebuiltStd = false; + // A hit is a use, stated on the record (mcpp::home::mark_used). + if (std_cached) mcpp::home::mark_used(metaPath); if (!std_cached) { std::error_code ec; diff --git a/tests/e2e/1000_a_configuration_keeps_one_directory.sh b/tests/e2e/1000_a_configuration_keeps_one_directory.sh new file mode 100755 index 000000000..c4572b9e3 --- /dev/null +++ b/tests/e2e/1000_a_configuration_keeps_one_directory.sh @@ -0,0 +1,97 @@ +#!/usr/bin/env bash +# requires: gcc +# 1000 -- a configuration keeps one build directory over time (design +# 2026-10-31 §8). A new fingerprint of a lineage -- the compiler family, the +# target, the standard library's family, the standard and the configuration's +# flags -- takes over the directory of its predecessor, so an mcpp upgrade +# rebuilds the edges whose commands changed in place instead of opening a full +# copy; a predecessor of another toolchain is removed; and the build directories +# no configuration used for `[build] keep_days` are swept after a build. +# +# Criteria: +# A. A predecessor of the lineage is taken over: the build continues in it +# and compiles nothing again. +# B. A predecessor of another toolchain is removed ("Replaced"). +# C. A directory of another lineage is kept. +# D. The sweep after a build removes an unrecorded directory written longer +# ago than `keep_days`; `keep_days = 0` (`mcpp self config --keep-days 0`) +# keeps it. +# E. The configuration names the build-cache entries it used (`uses.list`), +# and a build that replays its graph without planning still marks them +# used: an entry last used 40 days ago is used today after the sweep. +# F. In a rooted workspace (`members = ["."]`), `clean --stale` keeps the +# configuration in use: the workspace's own directory is no member's +# leftover. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +export MCPP_HOME="$TMP/mcpp-home" +source "$(dirname "$0")/_inherit_toolchain.sh" +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } + +cd "$TMP" +mkdir -p app/src && cd app +printf '[package]\nname = "app"\nversion = "0.1.0"\n\n[dependencies]\n"compat.zlib" = "1.3.2"\n' > mcpp.toml +printf '#include \nint main() { return zlibVersion()[0] == 0; }\n' > src/main.cpp +"$MCPP" build > b0.log 2>&1 || fail "the first build failed" b0.log +dir=$(ls -d target/*/*/ | grep -E '/[0-9a-f]{16}/$' | head -1); dir=${dir%/} +triple=$(dirname "$dir") +[ -f "$dir/lineage" ] && [ -f "$dir/toolchain" ] || fail "the directory records no lineage" <(ls -a "$dir") +compiles() { grep -c 'main' "$1/.ninja_log"; } +before=$(compiles "$dir") + +# A +mv "$dir" "$triple/0000000000000000" +"$MCPP" build > a.log 2>&1 || fail "the build after the move failed" a.log +[ -d "$dir" ] && [ ! -d "$triple/0000000000000000" ] || fail "the predecessor was not taken over" <(ls "$triple") +[ "$(compiles "$dir")" = "$before" ] || fail "taking the predecessor over compiled again" "$dir/.ninja_log" + +# B +mv "$dir" "$triple/0000000000000000" +printf 'compiler another\n' > "$triple/0000000000000000/toolchain" +"$MCPP" build > b.log 2>&1 || fail "the build after a toolchain change failed" b.log +[ ! -d "$triple/0000000000000000" ] || fail "a predecessor of another toolchain was kept" <(ls "$triple") +grep -q 'Replaced' b.log || fail "the replacement was not said" b.log + +# C +mkdir -p "$triple/1111111111111111" +printf 'compiler other\n' > "$triple/1111111111111111/lineage" +touch "$triple/1111111111111111/build.ninja" +"$MCPP" build > c.log 2>&1 || fail "the build beside another lineage failed" c.log +[ -d "$triple/1111111111111111" ] || fail "a directory of another lineage was taken" <(ls "$triple") + +# D +old="$triple/2222222222222222" +mkdir -p "$old" && touch "$old/build.ninja" +touch -d '5 days ago' "$old/build.ninja" "$old" +"$MCPP" self config --keep-days 0 > dc.log 2>&1 || fail "self config --keep-days 0 failed" dc.log +rm -f target/.mcpp-swept +"$MCPP" build > d0.log 2>&1 || fail "the build with keep_days = 0 failed" d0.log +[ -d "$old" ] || fail "keep_days = 0 swept a directory" d0.log +"$MCPP" self config --keep-days 3 > dc.log 2>&1 || fail "self config --keep-days 3 failed" dc.log +rm -f target/.mcpp-swept +"$MCPP" build > d1.log 2>&1 || fail "the build that sweeps failed" d1.log +[ ! -d "$old" ] || fail "a directory no configuration used for 5 days was kept" d1.log <(ls "$triple") +grep -q 'Swept' d1.log || fail "the sweep was not said" d1.log +[ -d "$dir" ] || fail "the sweep removed the configuration in use" <(ls "$triple") + +# E +record=$(sed -n 's/^cache \(.*compat.zlib.*\)$/\1/p' "$dir/uses.list" | head -1) +[ -n "$record" ] && [ -f "$record" ] || fail "the configuration names no cache entry it used" "$dir/uses.list" +touch -d '40 days ago' "$record" +"$MCPP" build > e0.log 2>&1 || fail "the replayed build failed" e0.log +rm -f target/.mcpp-swept +"$MCPP" build > e1.log 2>&1 || fail "the build that sweeps failed" e1.log +[ -n "$(find "$record" -mtime -1)" ] || fail "a replayed build did not mark its cache entry used" <(stat "$record") + +# F +mkdir -p "$TMP/ws/src" && cd "$TMP/ws" +printf '[package]\nname = "ws"\nversion = "0.1.0"\n\n[workspace]\nmembers = ["."]\n' > mcpp.toml +printf 'int main() { return 0; }\n' > src/main.cpp +"$MCPP" build > f0.log 2>&1 || fail "the rooted workspace did not build" f0.log +current=$(ls -d target/*/*/ | grep -E '/[0-9a-f]{16}/$' | head -1) +"$MCPP" clean --stale > f1.log 2>&1 || fail "clean --stale failed" f1.log +[ -d "$current" ] || fail "clean --stale removed the rooted workspace's own build" f1.log + +echo "OK" diff --git a/tests/e2e/174_cache_modes_and_commands.sh b/tests/e2e/174_cache_modes_and_commands.sh index b3914bdff..c6336927c 100644 --- a/tests/e2e/174_cache_modes_and_commands.sh +++ b/tests/e2e/174_cache_modes_and_commands.sh @@ -15,7 +15,8 @@ # reclaim is a cache nobody can trust. `prune` in particular used to rank # entries by the directory's mtime, which only records when an entry was # WRITTEN — so a dependency that hit on every build looked as stale as one -# nobody had touched in a month. `gc` reads entry.json's `accessed` stamp. +# nobody had touched in a month. `gc` reads the later of entry.json's +# `accessed` stamp and entry.json's time, which a use sets (2026.10.31.3). set -e source "$(dirname "$0")/_host_path.sh" @@ -313,6 +314,7 @@ d = json.load(open(p)) d["accessed"] = "1000" # 1970-ish json.dump(d, open(p, "w"), indent=2) PYEOF +touch -d '@1000' "$entry" # and the record's time, which a use sets "$MCPP" cache gc --older-than 1s > gc.log 2>&1 || { cat gc.log; exit 1; } [[ "$(entry_count)" -eq 0 ]] || { echo "FAIL: gc did not collect a long-unused entry" @@ -350,9 +352,9 @@ rc=0 cat gcsize.log exit 1 } -grep -q 'package entries now' gcsize.log || { +grep -q 'package and tool entries now' gcsize.log || { cat gcsize.log - echo "FAIL: gc summary must scope its figure to package entries" + echo "FAIL: gc summary must scope its figure to package and tool entries" exit 1 } diff --git a/tests/e2e/730_bare_compat_dependency_is_not_a_refresh_miss.sh b/tests/e2e/730_bare_compat_dependency_is_not_a_refresh_miss.sh index 920e647a4..61cb92382 100755 --- a/tests/e2e/730_bare_compat_dependency_is_not_a_refresh_miss.sh +++ b/tests/e2e/730_bare_compat_dependency_is_not_a_refresh_miss.sh @@ -42,6 +42,10 @@ EOF echo 'int main() { return 0; }' > "$TMP/app/src/main.cpp" cd "$TMP/app" +# The package itself, installed online first: the criteria are about the +# refresh decision of an offline plan, not about which earlier test of the +# shard happened to install it (the shards follow the timings, and moved). +"$MCPP" emit build-database --format json > /dev/null 2> install.txt || true MCPP_OFFLINE=1 "$MCPP" emit build-database --format json -v > out.json 2> err.txt || true # What the shared home held when the rung was not taken: the test passes on a diff --git a/tests/e2e/996_an_ok_image_waits_for_a_prepare_action.sh b/tests/e2e/996_an_ok_image_waits_for_a_prepare_action.sh new file mode 100755 index 000000000..190741ca9 --- /dev/null +++ b/tests/e2e/996_an_ok_image_waits_for_a_prepare_action.sh @@ -0,0 +1,66 @@ +#!/usr/bin/env bash +# requires: llvm +# 996 -- an ok image whose graph declares a `prepare` action builds: its link +# edge waits for the action, and the edge that seals the image is not taken +# for a link (SPEC-013 §4.6; mcpp#702). The ordering guard read the edge that +# publishes `bin/.oke` as the link and refused the build as an emitter +# defect (mcppls-clangd's clangd.okb, 2026-10-31). +# +# Criteria: +# A. `mcpp run --target -openkal` builds and runs a program whose +# build.mcpp declares a `prepare` action writing a directory. +# B. In the manifest, the link edge waits for the package's action phony. +# C. The image's debug information is compressed by its `ld.lld` link and +# its unit's is beside the object (`split_debuginfo`, design 2026-10-31 +# §4.1): okld seals such an image and okrun runs it (A). +set -e + +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } + +case "$(uname -m)" in + x86_64|amd64) HOST_ARCH=x86_64 ;; + aarch64|arm64) HOST_ARCH=aarch64 ;; + *) echo "SKIP: no ok-image row for $(uname -m)"; exit 0 ;; +esac +T="$HOST_ARCH-openkal" + +mkdir -p "$TMP/app/src" +cd "$TMP/app" +printf '[package]\nname = "app"\nversion = "0.1.0"\n' > mcpp.toml +printf 'int main() { return 0; }\n' > src/main.cpp +cat > build.mcpp <<'CPP' +#include +import mcpp; +int main() { + const std::string dir = std::string(mcpp::out_dir()) + "/gen"; + mcpp::action a; + a.id = "gen:dir"; + a.role = mcpp::roles::prepare; + a.arg("sh").arg("-c").arg(("mkdir -p " + dir + " && : > " + dir + "/made && : > " + dir + ".stamp").c_str()) + .output((dir + ".stamp").c_str()) + .output_dir(dir.c_str()) + .submit(); + return 0; +} +CPP + +# A +"$MCPP" run --target "$T" > run.log 2>&1 || fail "A: the image did not build and run" run.log +grep -q "internal:" run.log && fail "A: the ordering guard refused the build" run.log + +# B +manifest=$(find target/"$T" -name build.ninja -not -path "*/.build-mcpp/*" | head -1) +grep -E "^build obj/platform/bin/app\.oke\.unsealed[ :].*\|\|.*mcpp-actions-app" "$manifest" > /dev/null \ + || fail "B: the link edge does not wait for the action" "$manifest" + +# C +image=$(find target/"$T" -name app.oke | head -1) +if command -v readelf > /dev/null; then + readelf -S -W "$image" | grep -E '\.debug_info' | grep -q ' C ' \ + || fail "C: the image's debug information is not compressed" <(readelf -S -W "$image") +fi +find target/"$T" -name 'main*.dwo' | grep -q . || fail "C: the image's unit has no .dwo" <(find target/"$T" -name '*.dwo') + +echo "OK" diff --git a/tests/e2e/997_a_members_dev_dependencies_stay_out_of_a_host_tool.sh b/tests/e2e/997_a_members_dev_dependencies_stay_out_of_a_host_tool.sh new file mode 100755 index 000000000..04b4b332a --- /dev/null +++ b/tests/e2e/997_a_members_dev_dependencies_stay_out_of_a_host_tool.sh @@ -0,0 +1,124 @@ +#!/usr/bin/env bash +# requires: gcc +# 997 -- a host tool has one store key whatever the command and the member +# (mcpp#874). The key's dependency closure was read from the graph the build +# had walked: empty where only a build dependency's `tools` reached the tool +# package (`mcpp test --workspace`), complete where it was a member being +# built (`mcpp build`), and holding the dev-dependencies of a member whose +# tests were built. The tool's path changed with the key, every action that +# ran it ran again, and its outputs rebuilt every importer. The closure is now +# the tool package's own manifests, as its build resolves them. +# +# Criteria: +# A. `mcpp test --workspace`, then `mcpp build`: the tool has one store entry, +# and it names `lib` (the tool's path dependency) and not `testing` +# (`lib`'s dev-dependency). +# B. A second `mcpp test --workspace` builds no host tool. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +cd "$TMP" +export MCPP_HOME="$TMP/mcpphome" +mkdir -p "$MCPP_HOME" +[ -d "$HOME/.mcpp/registry" ] && ln -s "$HOME/.mcpp/registry" "$MCPP_HOME/registry" +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } + +mkdir -p ws/testing/src ws/lib/src ws/lib/tests ws/toolpkg/src ws/app/src ws/app/tests +cd ws +cat > mcpp.toml <<'EOF' +[workspace] +members = ["testing", "lib", "toolpkg", "app"] +EOF +cat > testing/mcpp.toml <<'EOF' +[package] +name = "testing" +version = "0.1.0" + +[targets.testing] +kind = "lib" +EOF +echo 'int testing_ok() { return 0; }' > testing/src/testing.cpp +cat > lib/mcpp.toml <<'EOF' +[package] +name = "lib" +version = "0.1.0" + +[targets.lib] +kind = "lib" + +[dev-dependencies] +testing = { path = "../testing" } +EOF +echo 'int lib_value() { return 41; }' > lib/src/lib.cpp +cat > lib/tests/smoke.cpp <<'EOF' +int testing_ok(); +int lib_value(); +int main() { return testing_ok() + (lib_value() == 41 ? 0 : 1); } +EOF +cat > toolpkg/mcpp.toml <<'EOF' +[package] +name = "toolpkg" +version = "0.1.0" + +[dependencies] +lib = { path = "../lib" } + +[targets.gen] +kind = "bin" +main = "src/gen.cpp" +EOF +cat > toolpkg/src/gen.cpp <<'EOF' +#include +int lib_value(); +int main(int argc, char** argv) { + if (argc < 2) return 2; + FILE* f = std::fopen(argv[1], "w"); + if (!f) return 3; + std::fprintf(f, "int generated_answer() { return %d; }\n", lib_value() + 1); + return std::fclose(f); +} +EOF +cat > app/mcpp.toml <<'EOF' +[package] +name = "app" +version = "0.1.0" + +[build-dependencies] +toolpkg = { path = "../toolpkg", tools = ["gen"] } +EOF +cat > app/src/main.cpp <<'EOF' +#include +int generated_answer(); +int main() { std::printf("ANSWER=%d\n", generated_answer()); } +EOF +cat > app/tests/answer.cpp <<'EOF' +int generated_answer(); +int main() { return generated_answer() == 42 ? 0 : 1; } +EOF +cat > app/build.mcpp <<'EOF' +#include +#include +import mcpp; +int main() { + const std::string out = std::string(mcpp::out_dir()) + "/gen.cpp"; + const std::string cmd = std::string("\"") + mcpp::dep_bin("toolpkg", "gen") + "\" \"" + out + "\""; + if (std::system(cmd.c_str()) != 0) return 1; + mcpp::generated(out.c_str()); +} +EOF + +# A +"$MCPP" test --workspace > t1.log 2>&1 || fail "the first test run failed" t1.log +"$MCPP" build > b1.log 2>&1 || fail "the build failed" b1.log +keys=$(find "$MCPP_HOME/build-cache/v1/tool" -name entry.json -path "*toolpkg@*" | wc -l) +entries=$(find "$MCPP_HOME/build-cache/v1/tool" -name entry.json -path "*toolpkg@*") +[ "$keys" -eq 1 ] || fail "the tool has $keys store entries, expected 1" $entries +grep -q '"lib@' $entries || fail "the key does not name the tool's dependency lib" $entries +grep -q '"testing@' $entries && fail "the key names lib's dev-dependency testing" $entries + +# B +"$MCPP" test --workspace > t2.log 2>&1 || fail "the second test run failed" t2.log +grep -q "Building.*host tool" t2.log && fail "the second test run built a host tool again" t2.log + +echo "OK" diff --git a/tests/e2e/998_a_build_keeps_its_bytes_once.sh b/tests/e2e/998_a_build_keeps_its_bytes_once.sh new file mode 100755 index 000000000..8677b0407 --- /dev/null +++ b/tests/e2e/998_a_build_keeps_its_bytes_once.sh @@ -0,0 +1,240 @@ +#!/usr/bin/env bash +# requires: gcc +# 998 -- what a build writes is written once, and what no configuration uses +# is collected (mcpp#874: a workspace's target/ of 25 GB, 14x its objects). +# +# Criteria: +# A. A library's archive holds its own package's objects, not its +# dependencies' (docs/12). +# B. On ELF, a program's debug information is compressed. +# C. A shared library staged as an equal copy is placed again by a link. +# D. `mcpp clean --stale` removes a `.build-mcpp/out/` no configuration +# names in its `uses.list`, and keeps the one the build used. +# E. `mcpp clean --report` names the areas and the total. +# F. A path tool's edit supersedes its store entry: one entry remains, and +# `mcpp cache list` shows it as a tool. +# G. A generator that runs again and leaves its output as it was rebuilds +# nothing that includes the output (`restat` on the action's edge). +# H. A library of a build program that has a build program of its own is +# built in a group beside the first (`.1`), not in a copy of the +# host workspace under its own target/, and the program beside its record +# is a link to the group's binary. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +cd "$TMP" +export MCPP_HOME="$TMP/mcpphome" +mkdir -p "$MCPP_HOME" +[ -d "$HOME/.mcpp/registry" ] && ln -s "$HOME/.mcpp/registry" "$MCPP_HOME/registry" +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } + +mkdir -p dep/src tool/src app/src +cat > dep/mcpp.toml <<'EOF' +[package] +name = "dep" +version = "0.1.0" + +[targets.dep] +kind = "lib" +EOF +echo 'int dep_value() { return 41; }' > dep/src/dep.cpp +write_tool() { + cat > "$TMP/tool/mcpp.toml" <<'EOF' +[package] +name = "tool" +version = "0.1.0" + +[targets.gen] +kind = "bin" +main = "src/gen.cpp" +EOF + cat > "$TMP/tool/src/gen.cpp" < +int main(int argc, char** argv) { + if (argc < 2) return 2; + FILE* f = std::fopen(argv[1], "w"); + if (!f) return 3; + std::fprintf(f, "int app_extra() { return $1; }\n"); + return std::fclose(f); +} +EOF +} +write_tool 1 +cat > app/mcpp.toml <<'EOF' +[package] +name = "app" +version = "0.1.0" + +[targets.app] +kind = "lib" + +[targets.main] +kind = "bin" +main = "src/main.cpp" + +[dependencies] +dep = { path = "../dep" } + +[build-dependencies] +tool = { path = "../tool", tools = ["gen"] } +EOF +echo 'int dep_value(); int app_value() { return dep_value() + 1; }' > app/src/app.cpp +cat > app/src/main.cpp <<'EOF' +#include +int app_value(); +int app_extra(); +int main() { std::printf("%d\n", app_value() + app_extra()); } +EOF +cat > app/build.mcpp <<'EOF' +#include +#include +import mcpp; +int main() { + const std::string out = std::string(mcpp::out_dir()) + "/extra.cpp"; + const std::string cmd = std::string("\"") + mcpp::dep_bin("tool", "gen") + "\" \"" + out + "\""; + if (std::system(cmd.c_str()) != 0) return 1; + mcpp::generated(out.c_str()); +} +EOF +cd app +"$MCPP" build > b1.log 2>&1 || fail "the build failed" b1.log + +# A +archive=$(find target -name 'libapp.a' -not -path '*/.build-mcpp/*' | head -1) +[ -n "$archive" ] || fail "no libapp.a" b1.log +members=$(ar t "$archive") +echo "$members" | grep -q 'app' || fail "the archive lacks the package's own object: $members" +echo "$members" | grep -q 'dep' && fail "the archive holds its dependency's object: $members" + +# B +if [ "$(uname -s)" = Linux ] && command -v readelf > /dev/null; then + program=$(find target -path '*/bin/main' | head -1) + readelf -S -W "$program" | grep -E '\.debug_info' | grep -q ' C ' \ + || fail "the program's .debug_info is not compressed" <(readelf -S -W "$program") +fi + +# C +mkdir -p stage +printf 'not really a library' > stage/libx.so +cp stage/libx.so stage/placed.so +"$MCPP" stage --output stage/placed.so stage/libx.so > stage.log 2>&1 || fail "mcpp stage failed" stage.log +[ "$(stat -c %i stage/libx.so)" = "$(stat -c %i stage/placed.so)" ] \ + || fail "an equal copy of a shared library was left a copy" + +# D +used=$(sed -n 's/^out //p' target/*/*/uses.list | head -1) +[ -n "$used" ] || fail "the build names no generated-source directory" $(ls target/*/*/uses.list) +mkdir -p target/.build-mcpp/out/0000000000000000/app +touch -d '2 days ago' target/.build-mcpp/out/0000000000000000 +"$MCPP" clean --stale > clean.log 2>&1 || fail "clean --stale failed" clean.log +[ -d target/.build-mcpp/out/0000000000000000 ] && fail "an unused generated-source directory was kept" clean.log +[ -d "target/.build-mcpp/out/$used" ] || fail "the generated-source directory in use was removed" clean.log + +# E +"$MCPP" clean --report > report.log 2>&1 || fail "clean --report failed" report.log +grep -q '.build-mcpp/out' report.log && grep -q '^target/' report.log || fail "the report lacks its areas" report.log + +# F +sleep 1 +write_tool 2 +"$MCPP" build > b2.log 2>&1 || fail "the build after the tool's edit failed" b2.log +versions=$(find "$MCPP_HOME/build-cache/v1/tool" -mindepth 2 -maxdepth 2 -name 'tool@*' | wc -l) +[ "$versions" -eq 1 ] || fail "the superseded tool entry was kept ($versions)" \ + <(find "$MCPP_HOME/build-cache/v1/tool") +"$MCPP" cache list > cache.log 2>&1 || fail "cache list failed" cache.log +grep -E '\btool\b' cache.log | grep -q 'tool@' || fail "cache list does not show the tool entry" cache.log + +# G +mkdir -p "$TMP/gen/src" +cd "$TMP/gen" +cat > mcpp.toml <<'EOF' +[package] +name = "gen" +version = "0.1.0" +EOF +cat > gen.sh <<'EOF' +#!/bin/sh +printf '#define VALUE 7\n' > "$2.tmp" +if cmp -s "$2.tmp" "$2"; then rm "$2.tmp"; else mv "$2.tmp" "$2"; fi +EOF +chmod +x gen.sh +echo one > value.in +cat > build.mcpp <<'EOF' +#include +import mcpp; +int main() { + const std::string root = mcpp::manifest_dir(); + const std::string out = mcpp::out_dir(); + mcpp::action a; + a.id = "gen:value"; + a.role = "source"; + a.arg((root + "/gen.sh").c_str()).arg((root + "/value.in").c_str()) + .arg((out + "/value.h").c_str()) + .input((root + "/value.in").c_str()) + .output((out + "/value.h").c_str()) + .submit(); + mcpp::include_dir(out.c_str()); +} +EOF +cat > src/main.cpp <<'EOF' +#include +#include "value.h" +int main() { std::printf("%d\n", VALUE); } +EOF +"$MCPP" build > g1.log 2>&1 || fail "the generator's build failed" g1.log +log=$(find target -name .ninja_log -not -path '*/.build-mcpp/*' | head -1) +before=$(grep -c 'main' "$log") +sleep 1 +echo two > value.in +"$MCPP" build > g2.log 2>&1 || fail "the second build failed" g2.log +grep -q 'value.h' <(tail -n +2 "$log" | tail -5) || fail "the generator did not run again" "$log" +[ "$(grep -c 'main' "$log")" = "$before" ] || fail "an unchanged generated header rebuilt its includer" "$log" + +# H +mkdir -p "$TMP/genlib/src" "$TMP/user/src" +cd "$TMP/genlib" +cat > mcpp.toml <<'EOF' +[package] +name = "genlib" +version = "0.1.0" + +[targets.genlib] +kind = "lib" +EOF +printf 'export module genlib;\nexport int genlib_value();\n' > src/genlib.cppm +cat > build.mcpp <<'EOF' +#include +#include +import mcpp; +int main() { + const std::string out = std::string(mcpp::out_dir()) + "/genlib_impl.cpp"; + std::ofstream(out) << "module genlib;\nint genlib_value() { return 5; }\n"; + mcpp::generated(out.c_str()); +} +EOF +cd "$TMP/user" +cat > mcpp.toml <<'EOF' +[package] +name = "user" +version = "0.1.0" + +[build-dependencies] +genlib = { path = "../genlib" } +EOF +echo 'int main() { return 0; }' > src/main.cpp +cat > build.mcpp <<'EOF' +import mcpp; +import genlib; +int main() { return genlib_value() == 5 ? 0 : 1; } +EOF +"$MCPP" build > h1.log 2>&1 || fail "the build with a nested build program failed" h1.log +host=target/.build-mcpp/host +nested=$(find "$host" -path '*/target/.build-mcpp/host' | head -1) +[ -z "$nested" ] || fail "the host workspace holds a copy of itself: $nested" <(find "$host" -maxdepth 3) +ls -d "$host"/*.1 > /dev/null 2>&1 || fail "the library's build program has no group beside the first" <(ls "$host") +copy=target/.build-mcpp/build.mcpp.bin +binary=$(python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print(next(iter(d.values()))['binary'])" "$(ls "$host"/*/programs.json | grep -v '\.1/' | head -1)") +[ "$(stat -c %i "$copy")" = "$(stat -c %i "$binary")" ] || fail "the program beside its record is a copy" + +echo "OK" diff --git a/tests/e2e/999_debug_information_is_stored_once.sh b/tests/e2e/999_debug_information_is_stored_once.sh new file mode 100755 index 000000000..46645df3f --- /dev/null +++ b/tests/e2e/999_debug_information_is_stored_once.sh @@ -0,0 +1,63 @@ +#!/usr/bin/env bash +# requires: gcc +# 999 -- `split_debuginfo` (design 2026-10-31 §4.1 (b)): under `dev` the units +# compiled in the build directory keep their debug information beside their +# objects, and a program carries a skeleton that names it; an index package's +# units, which enter the global cache and are staged by other projects, keep +# theirs in the object (compressed). +# +# Criteria: +# A. A dev build: the root's unit has a `.dwo`, the index package's units +# have none, the program's `.debug_info` is compressed, and gdb reads the +# root's source through the skeleton. +# B. A release build writes no `.dwo`. +# C. `[profile.dev] split_debuginfo = "off"` writes no `.dwo`. +# D. A value other than "unpacked" and "off" is refused, naming both. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +cd "$TMP" +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } +[ "$(uname -s)" = Linux ] || { echo "SKIP: the split form is checked on Linux (ELF)"; exit 0; } + +mkdir -p app/src +cd app +write_manifest() { + printf '[package]\nname = "app"\nversion = "0.1.0"\n\n[dependencies]\n"compat.zlib" = "1.3.2"\n%s' "$1" > mcpp.toml +} +write_manifest "" +cat > src/main.cpp <<'EOF' +#include +#include +int main() { std::printf("%s\n", zlibVersion()); } +EOF + +# A +"$MCPP" build > a.log 2>&1 || fail "the dev build failed" a.log +dir=$(ls -d target/*/*/ | head -1) +find "$dir/obj" -name 'main*.dwo' | grep -q . || fail "the root's unit has no .dwo" <(find "$dir/obj") +[ -z "$(find "$dir/obj" -path '*compat_zlib*' -name '*.dwo')" ] || fail "an index package's unit was split" <(find "$dir/obj" -name '*.dwo') +program=$(find "$dir/bin" -name app -type f | head -1) +readelf -S -W "$program" | grep -E '\.debug_info' | grep -q ' C ' || fail "the program's .debug_info is not compressed" <(readelf -S -W "$program") +if command -v gdb > /dev/null; then + gdb -batch -ex 'list main' "$program" 2> /dev/null | grep -q zlibVersion || fail "gdb does not read the source through the skeleton" +fi + +# B +rm -rf target +"$MCPP" build --release > b.log 2>&1 || fail "the release build failed" b.log +[ -z "$(find target -name '*.dwo')" ] || fail "a release build wrote .dwo files" <(find target -name '*.dwo') + +# C +rm -rf target +write_manifest $'\n[profile.dev]\nsplit_debuginfo = "off"\n' +"$MCPP" build > c.log 2>&1 || fail "the build with split_debuginfo off failed" c.log +[ -z "$(find target -name '*.dwo')" ] || fail "split_debuginfo = \"off\" wrote .dwo files" <(find target -name '*.dwo') + +# D +write_manifest $'\n[profile.dev]\nsplit_debuginfo = "packed"\n' +"$MCPP" build > d.log 2>&1 && fail "an unknown split_debuginfo value was accepted" d.log +grep -q '"unpacked" or "off"' d.log || fail "the refusal does not name the values" d.log + +echo "OK" diff --git a/tests/e2e/timings/linux.tsv b/tests/e2e/timings/linux.tsv index f1fb5d647..a889e40bd 100644 --- a/tests/e2e/timings/linux.tsv +++ b/tests/e2e/timings/linux.tsv @@ -608,3 +608,8 @@ 993_a_test_describes_its_launch.sh 30000 994_the_contract_names_the_target_side.sh 60000 995_the_windows_launcher_builds_over_any_platform_release.sh 240000 +996_an_ok_image_waits_for_a_prepare_action.sh 15000 +997_a_members_dev_dependencies_stay_out_of_a_host_tool.sh 60000 +998_a_build_keeps_its_bytes_once.sh 120000 +999_debug_information_is_stored_once.sh 30000 +1000_a_configuration_keeps_one_directory.sh 40000 diff --git a/tests/unit/test_bmi_cache.cpp b/tests/unit/test_bmi_cache.cpp index c4716e30e..5a205c2b7 100644 --- a/tests/unit/test_bmi_cache.cpp +++ b/tests/unit/test_bmi_cache.cpp @@ -313,8 +313,10 @@ TEST(BmiCache, ProbeReportsNoLayoutMismatchWhenTheEntryItselfIsInvalid) { } // touch_accessed is what makes `cache gc` an LRU rather than "drop what was -// populated long ago". It must move the stamp and leave the artifacts alone — -// their mtimes participate in ninja's restat handling. +// populated long ago". A use sets the time of the entry's record (mcpp::home:: +// mark_used, design 2026-10-31 §4.6): the record's content and the artifacts +// stay as they are -- the artifacts' mtimes participate in ninja's restat +// handling -- and a second use within 12 hours writes nothing. TEST(BmiCache, TouchAccessedMovesTheStampAndNotTheArtifacts) { Tmp t; auto home = t.path / "home"; @@ -324,23 +326,27 @@ TEST(BmiCache, TouchAccessedMovesTheStampAndNotTheArtifacts) { auto k = makeKey(home); ASSERT_TRUE(populate_from(k, project, oneOfEach())); - auto j0 = readJson(k.entryFile()); - auto created0 = j0.value("created", std::string{}); + const auto content0 = readJson(k.entryFile()).dump(); auto objTime0 = std::filesystem::last_write_time(k.objDir() / "lib.m.o"); auto bmiTime0 = std::filesystem::last_write_time(k.bmiDir() / "lib.gcm"); - // Rewrite the stamp to something clearly old, then touch. - j0["accessed"] = "1000"; - std::ofstream(k.entryFile()) << j0.dump(2); + // The record last used two days ago, then a use. + const auto old = std::filesystem::file_time_type::clock::now() - std::chrono::hours(48); + std::filesystem::last_write_time(k.entryFile(), old); touch_accessed(k); - auto j1 = readJson(k.entryFile()); - EXPECT_NE(j1.value("accessed", std::string{}), "1000"); - EXPECT_EQ(j1.value("created", std::string{}), created0) - << "touch must not reset the creation stamp"; + const auto used = std::filesystem::last_write_time(k.entryFile()); + EXPECT_GT(used, old + std::chrono::hours(47)) << "a use must move the record's time"; + EXPECT_EQ(readJson(k.entryFile()).dump(), content0) << "a use must not rewrite the record"; EXPECT_EQ(std::filesystem::last_write_time(k.objDir() / "lib.m.o"), objTime0); EXPECT_EQ(std::filesystem::last_write_time(k.bmiDir() / "lib.gcm"), bmiTime0); EXPECT_TRUE(is_cached(k, oneOfEach())) << "touching must not invalidate the entry"; + + // Within 12 hours a second use writes nothing. + const auto recent = std::filesystem::file_time_type::clock::now() - std::chrono::hours(1); + std::filesystem::last_write_time(k.entryFile(), recent); + touch_accessed(k); + EXPECT_EQ(std::filesystem::last_write_time(k.entryFile()), recent); } TEST(BmiCache, RepopulatePreservesCreatedStamp) {