Skip to content

Fix exception-masking bugs in Script.java and DefaultModuleDefinitionSet - #14498

Open
fermosan wants to merge 1 commit into
apache:mainfrom
fermosan:exception-masking
Open

fermosan wants to merge 1 commit into
apache:mainfrom
fermosan:exception-masking

Conversation

@fermosan

Copy link
Copy Markdown
Contributor

Description

Fixes two unrelated-looking bugs that share the same root defect: code that catches an
exception, then re-logs or re-throws it in a way that discards the real error detail, leaving
only a generic/misleading message in the logs.

1. Script.executeInternal() (two occurrences) — when a draining OutputInterpreter
(e.g. AllLinesParser) is used and the process exits non-zero, the failure path opened a
second BufferedReader on the same process stream a background Task had already consumed
and closed. Calling processError() on it always threw IOException: Stream closed instead of
the real process output — the actual stdout/stderr content was silently discarded even though
it had already been captured in task.getResult(). The success path a few lines above already
does this correctly; the failure path never applied the same check.

2. DefaultModuleDefinitionSet.loadContext() — the caught exception was passed to
logger.error() as an ordinary format-string argument instead of SLF4J's trailing-throwable
argument, so the stack trace was never actually logged — only a one-line toString() summary.
The re-thrown RuntimeException also didn't chain the original exception as its cause, so even
a caller further up the stack couldn't recover it via getCause().

Functional behavior change: none for the success/non-error paths. On failure, both now surface
the real underlying error instead of a misleading placeholder.

*(not tied to a filed issue — found during manual diagnosis, no tracking ticket exists yet)*

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

(Neither bug causes incorrect behavior or data loss by itself — both are observability bugs
that hide the real cause of an unrelated failure. Marked Minor on that basis)

Screenshots (if appropriate):

N/A — logging/exception-handling fix, no UI or visible behavior change.

How Has This Been Tested?

  • Both changes compile cleanly (mvn -pl :cloud-utils install -DskipTests and
    mvn -pl :cloud-framework-spring-module install -DskipTests, both BUILD SUCCESS).
  • DefaultModuleDefinitionSet fix verified end-to-end: reproduced a real module-load failure,
    confirmed the log previously showed only nested exception is java.lang.ExceptionInInitializerError with nothing further; after the fix, rebuilding and
    re-running shows the full cause chain down to the actual root exception.
  • Script.java fix has not been exercised against a live non-zero-exit drained-interpreter
    failure in this round of testing — the only real-world repro we had was a one-off incident
    (malformed SSH key rejected by OpenSSH) that isn't easily reproduced on demand. The fix is a
    direct structural mirror of the already-correct success-path logic a few lines above, in the
    same method.

How did you try to break this feature and the system with this change?

  • For Script.java: confirmed the non-draining-interpreter path (interpreter.drain() == false) is completely untouched by the ternary — processError(reader) still runs exactly as
    before for that case. task is only read when drain() is true, which is the same
    precondition the existing success path already relies on, so no new null-pointer path is
    introduced.
  • For DefaultModuleDefinitionSet: checked for any other code relying on this
    RuntimeException not having a cause set — found none. Setting the cause is purely
    additive; nothing downstream inspects or depends on getCause() being null.

  Two unrelated-looking bugs share the same root defect: code that catches
  an exception, then re-logs or re-throws it in a way that discards the
  actual error detail.

  - Script.executeInternal(): the non-zero-exit failure path opened a new
    BufferedReader on a stream a draining Task had already consumed and
    closed, so interpreter.processError() always threw "Stream closed"
    instead of surfacing the real process output. Fixed by reusing
    task.getResult() when interpreter.drain() is true, mirroring the
    success path's existing logic. Two occurrences in the same file.

  - DefaultModuleDefinitionSet.loadContext(): the caught exception was
    passed as an ordinary format-string argument to logger.error() instead
    of SLF4J's trailing-throwable argument, so the stack trace was never
    logged. The re-thrown RuntimeException also didn't chain the original
    cause. Fixed both: pass the exception as the throwable arg, and set it
    as the new exception's cause.

  Both were found while diagnosing real failures where the underlying
  error was invisible in the logs and had to be recovered by manually
  patching these exact lines.
@DaanHoogland

Copy link
Copy Markdown
Contributor

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@codecov

codecov Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 25.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 20.63%. Comparing base (fd6b755) to head (662b478).

Files with missing lines Patch % Lines
.../module/model/impl/DefaultModuleDefinitionSet.java 0.00% 2 Missing ⚠️
...s/src/main/java/com/cloud/utils/script/Script.java 50.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##               main   #14498   +/-   ##
=========================================
  Coverage     20.63%   20.63%           
- Complexity    21695    21697    +2     
=========================================
  Files          6489     6489           
  Lines        590136   590136           
  Branches      72361    72363    +2     
=========================================
+ Hits         121788   121802   +14     
+ Misses       455100   455081   -19     
- Partials      13248    13253    +5     
Flag Coverage Δ
uitests 3.74% <ø> (ø)
unittests 21.95% <25.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✖️ el8 ✖️ el9 ✔️ debian ✔️ suse15. SL-JID 19722

@DaanHoogland

Copy link
Copy Markdown
Contributor

@fermosan ,

the el build failed but with errors that I would then expect on other builds as well. I am re-running the subjob to verify if it is a fluke

FYI

20:28:57 [ERROR] Failures: 
20:28:57 [ERROR]   LazyCacheTest.testLoaderNotCalledIfPresent:63 
20:28:57 function.apply("key2");
20:28:57 Wanted 1 time:
20:28:57 -> at org.apache.cloudstack.utils.cache.LazyCacheTest.testLoaderNotCalledIfPresent(LazyCacheTest.java:63)
20:28:57 But was 2 times:
20:28:57 -> at com.github.benmanes.caffeine.cache.LocalLoadingCache.lambda$newMappingFunction$3(LocalLoadingCache.java:183)
20:28:57 -> at com.github.benmanes.caffeine.cache.LocalLoadingCache.lambda$newMappingFunction$3(LocalLoadingCache.java:183)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants