Repository navigation
feat(workflow-operator): leave explanatory comments out of exported Python - #9154
Conversation
…ython Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Automated Reviewer SuggestionsBased on the
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 382 | 0.233 | 26,144/32,621/32,621 us | 🟢 -47.7% / 🔴 +131.6% |
| 🔴 | bs=100 sw=10 sl=64 | 820 | 0.501 | 117,980/142,586/142,586 us | 🔴 +10.6% / 🔴 +47.9% |
| ⚪ | bs=1000 sw=10 sl=64 | 949 | 0.579 | 1,042,259/1,136,850/1,136,850 us | ⚪ within ±5% / 🔴 +22.4% |
Baseline details
Latest main d79ac16 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 382 tuples/sec | 337 tuples/sec | 888.52 tuples/sec | +13.4% | -57.0% |
| bs=10 sw=10 sl=64 | MB/s | 0.233 MB/s | 0.206 MB/s | 0.542 MB/s | +13.1% | -57.0% |
| bs=10 sw=10 sl=64 | p50 | 26,144 us | 22,168 us | 11,467 us | +17.9% | +128.0% |
| bs=10 sw=10 sl=64 | p95 | 32,621 us | 62,431 us | 14,087 us | -47.7% | +131.6% |
| bs=10 sw=10 sl=64 | p99 | 32,621 us | 62,431 us | 17,519 us | -47.7% | +86.2% |
| bs=100 sw=10 sl=64 | throughput | 820 tuples/sec | 864 tuples/sec | 1,174 tuples/sec | -5.1% | -30.1% |
| bs=100 sw=10 sl=64 | MB/s | 0.501 MB/s | 0.527 MB/s | 0.716 MB/s | -4.9% | -30.1% |
| bs=100 sw=10 sl=64 | p50 | 117,980 us | 113,785 us | 89,952 us | +3.7% | +31.2% |
| bs=100 sw=10 sl=64 | p95 | 142,586 us | 128,949 us | 96,426 us | +10.6% | +47.9% |
| bs=100 sw=10 sl=64 | p99 | 142,586 us | 128,949 us | 110,188 us | +10.6% | +29.4% |
| bs=1000 sw=10 sl=64 | throughput | 949 tuples/sec | 971 tuples/sec | 1,204 tuples/sec | -2.3% | -21.2% |
| bs=1000 sw=10 sl=64 | MB/s | 0.579 MB/s | 0.593 MB/s | 0.735 MB/s | -2.4% | -21.2% |
| bs=1000 sw=10 sl=64 | p50 | 1,042,259 us | 1,034,610 us | 886,711 us | +0.7% | +17.5% |
| bs=1000 sw=10 sl=64 | p95 | 1,136,850 us | 1,095,979 us | 928,644 us | +3.7% | +22.4% |
| bs=1000 sw=10 sl=64 | p99 | 1,136,850 us | 1,095,979 us | 972,356 us | +3.7% | +16.9% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,523.98,200,128000,382,0.233,26144.18,32620.60,32620.60
1,100,10,64,20,2438.77,2000,1280000,820,0.501,117980.41,142586.02,142586.02
2,1000,10,64,20,21079.30,20000,12800000,949,0.579,1042259.42,1136849.99,1136849.99
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9154 +/- ##
============================================
- Coverage 92.05% 92.04% -0.01%
+ Complexity 5039 5038 -1
============================================
Files 1286 1286
Lines 55928 55917 -11
Branches 7138 7138
============================================
- Hits 51483 51470 -13
Misses 2693 2693
- Partials 1752 1754 +2
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@carloea2 This removes the explanatory comments that the merged export PRs left in exported Python, and keeps a short note in Scala where the reason matters. The generated statements are unchanged across all 1,098 verification scripts. Could you take a look? |
carloea2
left a comment
There was a problem hiding this comment.
Reviewed all 24 changed files. A mechanical comparison after removing explanatory comment lines found no executable source changes. Required CI checks pass. I did not rerun the 1,098-script comparison.
|
Please resolve the conflicts. |
…ents # Conflicts: # common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/treeplot/TreePlotOpDesc.scala
|
@aglinxinyuan Resolved by merging main. The only conflict was TreePlot's |
What changes were proposed in this PR?
The Python that "Export to Python" writes carried notes meant for whoever maintains the generator: why a frame is copied, why a row is dropped before scoring, why an empty cell keeps its row. Someone running the exported script does not need them, and they made the script longer to read. This PR takes those comments out of the exported code in 24 operators, and keeps a one or two line note in Scala above the string where the reason is worth keeping.
Only comments go. The translator's own markers stay: the
# [Operator]heading over each section,# --- Output ---, and the# TODOit writes for an operator it cannot export yet. The Python that operators run inside the engine is not touched.Any related issues, documentation, discussions?
The comments came in with the export PRs #8365, #8367, #8368, #8504 and #8340.
#9035 also changes Unnest String's export. Whichever of the two merges second keeps the other's change.
How was this PR tested?
No generated statement changes, only comment lines. Running the verification suite before and after, the 1,098 scripts it generates for 142 operators are identical once comment lines and temporary paths are set aside, and none of them has an explanatory comment left. All of
WorkflowOperator/testpasses (2,912 tests), as doscalafmtCheckandscalafixAll --check.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
🤖 Generated with Claude Code