Skip to content

Fix CUDA driver resolution, CPU quota compatibility, and verifier packaging - #50

Merged
hanhainebula merged 1 commit into
mainfrom
fix/cuda-driver-library-path
Nov 6, 2026
Merged

hanhainebula merged 1 commit into
mainfrom
fix/cuda-driver-library-path

Conversation

@hanhainebula

Copy link
Copy Markdown
Collaborator

Fix three infrastructure failures that can prevent evaluation from starting or completing: incompatible CUDA driver libraries, CPU quotas lost through Compose translation, and a missing verifier entrypoint.

  • CUDA driver resolution: Remove the forced /usr/local/cuda/compat prefix from both Compose configurations and child_env() in spoken-query-search and document-page-search. These processes can then load the host-matched NVIDIA driver library. Previously, the image's compatibility library shadowed that library and caused CUDA error 803.
  • CPU quota compatibility: Preserve explicit CFS quotas that match Harbor's CPU budget and 100 ms period, and clear the duplicate NanoCPU setting with cpus: 0. The CFS quota continues to enforce the budget on Compose versions that ignore cpus. Mismatched budgets and unsupported periods still fail validation. This applies to both phases of ten tasks; configurations without explicit CFS quotas retain their existing handling. Update the runtime documentation and regression tests accordingly.
  • Verifier packaging: Add the existing judge_repeats.py to the Code Embedding verifier's COPY list. Its test.sh already invokes this file, but the image omitted it, preventing the required three-round audit from starting.

Task versions, resource budgets, scoring rules, models and input data are unchanged. The CUDA changes are task-local; the CPU compatibility change is in the shared runner.

Validation

  • Repository unit suite: 359 passed, 2 skipped, including configuration checks across all 94 task Compose files.
  • Document Page tests: 38 passed, 6 skipped; Spoken authoring tests: 31 passed; Code Embedding's packaged audit tests: 5 passed.
  • Native GPU A/B checks covered development root, verifier root and candidate UID 10001 for both CUDA-affected tasks. The original path returned CUDA 803; the updated path initialized successfully and completed matrix and convolution operations. Resource and candidate-permission checks passed.
  • Native Harbor/Docker checks confirmed 8-CPU and 16-CPU CFS limits with and without CPU affinity, unchanged memory limits, and preserved behavior for a configuration using only cpus.
  • Code Embedding packaging checks verified nine installed files against source hashes, imported the packaged helper/backend, and exercised three audit invocations offline.
  • All three task merge-ready checks, the repository release check and git diff --check passed.

GPU checks used the pinned shared base image and current task configuration/launcher code. Packaging validation reused cached verifier dependencies and recreated /tests using the production COPY instructions; a fresh full dependency build was not completed. The older Compose host was not retested; its recorded A/B evidence was reviewed. Full model evaluations and live judge API calls were not run.

@hanhainebula
hanhainebula merged commit 8f5c30a into main Nov 6, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant