Skip to content

feat(workflow-operator): leave explanatory comments out of exported Python - #9154

Merged
aglinxinyuan merged 2 commits into
apache:mainfrom
kz930:feat/export-no-comments
Nov 1, 2026
Merged

aglinxinyuan merged 2 commits into
apache:mainfrom
kz930:feat/export-no-comments

Conversation

@kz930

@kz930 kz930 commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

The Python that "Export to Python" writes carried notes meant for whoever maintains the generator: why a frame is copied, why a row is dropped before scoring, why an empty cell keeps its row. Someone running the exported script does not need them, and they made the script longer to read. This PR takes those comments out of the exported code in 24 operators, and keeps a one or two line note in Scala above the string where the reason is worth keeping.

  • Machine learning: the sklearn estimators' shared training, testing and fit-and-score code, Linear Regression, and the Machine Learning Scorer.
  • Hugging Face: Iris Logistic Regression, Sentiment Analysis, Spam SMS Detection, Text Summarization.
  • Visualization: Bar, Box/Violin, Bubble, Bullet, Choropleth, Contour, Dendrogram, Gauge, Hierarchy, Histogram 2D, Icicle, Radar, Tree Plot, URL Visualizer.
  • Unnest String.

Only comments go. The translator's own markers stay: the # [Operator] heading over each section, # --- Output ---, and the # TODO it writes for an operator it cannot export yet. The Python that operators run inside the engine is not touched.

Any related issues, documentation, discussions?

The comments came in with the export PRs #8365, #8367, #8368, #8504 and #8340.

#9035 also changes Unnest String's export. Whichever of the two merges second keeps the other's change.

How was this PR tested?

No generated statement changes, only comment lines. Running the verification suite before and after, the 1,098 scripts it generates for 142 operators are identical once comment lines and temporary paths are set aside, and none of them has an explanatory comment left. All of WorkflowOperator/test passes (2,912 tests), as do scalafmtCheck and scalafixAll --check.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5)

🤖 Generated with Claude Code

…ython

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @aglinxinyuan
    You can notify them by mentioning @aglinxinyuan in a comment.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 4 better · 🔴 4 worse · ⚪ 7 noise (<±5%) · 0 without baseline

Compared against main d79ac16 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 382 0.233 26,144/32,621/32,621 us 🟢 -47.7% / 🔴 +131.6%
🔴 bs=100 sw=10 sl=64 820 0.501 117,980/142,586/142,586 us 🔴 +10.6% / 🔴 +47.9%
⚪ bs=1000 sw=10 sl=64 949 0.579 1,042,259/1,136,850/1,136,850 us ⚪ within ±5% / 🔴 +22.4%
Baseline details

Latest main d79ac16 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 382 tuples/sec 337 tuples/sec 888.52 tuples/sec +13.4% -57.0%
bs=10 sw=10 sl=64 MB/s 0.233 MB/s 0.206 MB/s 0.542 MB/s +13.1% -57.0%
bs=10 sw=10 sl=64 p50 26,144 us 22,168 us 11,467 us +17.9% +128.0%
bs=10 sw=10 sl=64 p95 32,621 us 62,431 us 14,087 us -47.7% +131.6%
bs=10 sw=10 sl=64 p99 32,621 us 62,431 us 17,519 us -47.7% +86.2%
bs=100 sw=10 sl=64 throughput 820 tuples/sec 864 tuples/sec 1,174 tuples/sec -5.1% -30.1%
bs=100 sw=10 sl=64 MB/s 0.501 MB/s 0.527 MB/s 0.716 MB/s -4.9% -30.1%
bs=100 sw=10 sl=64 p50 117,980 us 113,785 us 89,952 us +3.7% +31.2%
bs=100 sw=10 sl=64 p95 142,586 us 128,949 us 96,426 us +10.6% +47.9%
bs=100 sw=10 sl=64 p99 142,586 us 128,949 us 110,188 us +10.6% +29.4%
bs=1000 sw=10 sl=64 throughput 949 tuples/sec 971 tuples/sec 1,204 tuples/sec -2.3% -21.2%
bs=1000 sw=10 sl=64 MB/s 0.579 MB/s 0.593 MB/s 0.735 MB/s -2.4% -21.2%
bs=1000 sw=10 sl=64 p50 1,042,259 us 1,034,610 us 886,711 us +0.7% +17.5%
bs=1000 sw=10 sl=64 p95 1,136,850 us 1,095,979 us 928,644 us +3.7% +22.4%
bs=1000 sw=10 sl=64 p99 1,136,850 us 1,095,979 us 972,356 us +3.7% +16.9%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,523.98,200,128000,382,0.233,26144.18,32620.60,32620.60
1,100,10,64,20,2438.77,2000,1280000,820,0.501,117980.41,142586.02,142586.02
2,1000,10,64,20,21079.30,20000,12800000,949,0.579,1042259.42,1136849.99,1136849.99

@codecov-commenter

codecov-commenter commented Oct 26, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.04%. Comparing base (d79ac16) to head (89bd148).

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #9154      +/-   ##
============================================
- Coverage     92.05%   92.04%   -0.01%     
+ Complexity     5039     5038       -1     
============================================
  Files          1286     1286              
  Lines         55928    55917      -11     
  Branches       7138     7138              
============================================
- Hits          51483    51470      -13     
  Misses         2693     2693              
- Partials       1752     1754       +2     
Flag Coverage Δ *Carryforward flag
access-control-service 77.44% <ø> (ø)
agent-service 99.16% <ø> (ø) Carriedforward from b94d04d
amber 86.54% <100.00%> (-0.01%) ⬇️
computing-unit-managing-service 64.02% <ø> (ø)
config-service 87.61% <ø> (ø)
file-service 82.29% <ø> (ø)
frontend 96.47% <ø> (-0.01%) ⬇️ Carriedforward from b94d04d
notebook-migration-service 83.77% <ø> (ø)
pyamber 98.54% <ø> (ø) Carriedforward from b94d04d
workflow-compiling-service 74.25% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@aglinxinyuan aglinxinyuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@aglinxinyuan
aglinxinyuan added this pull request to the merge queue Oct 26, 2026
@kz930

kz930 commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

@carloea2 This removes the explanatory comments that the merged export PRs left in exported Python, and keeps a short note in Scala where the reason matters. The generated statements are unchanged across all 1,098 verification scripts. Could you take a look?

@aglinxinyuan
aglinxinyuan removed this pull request from the merge queue due to a manual request Oct 26, 2026

@carloea2 carloea2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all 24 changed files. A mechanical comparison after removing explanatory comment lines found no executable source changes. Required CI checks pass. I did not rerun the 1,098-script comparison.

@aglinxinyuan

Copy link
Copy Markdown
Contributor

Please resolve the conflicts.

…ents

# Conflicts:
#	common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/treeplot/TreePlotOpDesc.scala
@kz930

kz930 commented Nov 1, 2026

Copy link
Copy Markdown
Contributor Author

@aglinxinyuan Resolved by merging main. The only conflict was TreePlot's producesDataFrame override, which #9204 moved into the shared chart contract.

@aglinxinyuan
aglinxinyuan added this pull request to the merge queue Nov 1, 2026
Merged via the queue into apache:main with commit 1b515ad Nov 1, 2026
32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants