Skip to content

Backports/antalya 26.8/117009 - #2755

Draft
subkanthi wants to merge 5 commits into
antalya-26.8from
backports/antalya-26.8/117009
Draft

Backports/antalya 26.8/117009#2755
subkanthi wants to merge 5 commits into
antalya-26.8from
backports/antalya-26.8/117009

Conversation

@subkanthi

@subkanthi subkanthi commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

Changelog category (leave one):

  • New Feature
  • Experimental Feature
  • Improvement
  • Performance Improvement
  • Backward Incompatible Change
  • Build/Testing/Packaging Improvement
  • Documentation (changelog entry is not required)
  • Critical Bug Fix (crash, data loss, RBAC)
  • Bug Fix (user-visible misbehavior in an official stable release)
  • CI Fix or Improvement (changelog entry is not required)
  • Not for changelog (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

...

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

bobrik and others added 5 commits November 1, 2026 23:53
@bobrik @subkanthi
Speed up aggregation batches with non-null states
19cce13 
Aggregations with lots of buckets can get expensive. Here's an excerpt from a profile:

```
ROUTINE ======================== DB::IAggregateFunctionHelper<DB::AggregateFunctionSum<unsigned long, unsigned long, DB::AggregateFunctionSumData<unsigned long>, (DB::AggregateFunctionSumType)0> >::addBatch(unsigned long, unsigned long, char**, unsigned long, DB::IColumn const**, DB::Arena*, long) const in src/AggregateFunctions/IAggregateFunction.h
     1.79s      2.32s (flat, cum) 15.07% of Total
         .          .    549:            }
         .          .    550:        }
         .          .    551:        else
         .          .    552:        {
         .          .    553:            for (size_t i = row_begin; i < row_end; ++i)
     1.79s      1.79s    554:                if (places[i])
         .   526.32ms    555:                    static_cast<const Derived *>(this)->add(places[i] + place_offset, columns, i, arena);
         .          .    556:        }
         .          .    557:    }
         .          .    558:
         .          .    559:    void serializeBatch(const PaddedPODArray<AggregateDataPtr> & data, size_t start, size_t size, WriteBuffer & buf, std::optional<size_t> version) const final // NOLINT
         .          .    560:    {
```

We can get some of this back by specializing the common non-null version.

For a big query doing a very wide aggregation:

* 1 thread: 77.7s -> 74.3s (both wall and cpu)
* 64 threads: 17.0s -> 16.0s (wall), 99.5s -> 92.4s (cpu)

With a synthetic benchmark:

* Baseline: 1.24s

```
$ ../bench.py --clickhouse=/tmp/clickhouse-baseline --path=$HOME/clickhouse-no-nulls --runs=3 --query="SELECT toUInt16(bitAnd(number, 8191)) AS key, sum(number) FROM numbers(500000000) GROUP BY key FORMAT Null SETTINGS max_threads = 1"
wall:   1.24s, cpu:   1.23s, memory:   0.00GiB
wall:   1.25s, cpu:   1.24s, memory:   0.00GiB
wall:   1.23s, cpu:   1.23s, memory:   0.00GiB
```

* Patched: 1.10s

```
$ ../bench.py --clickhouse=/tmp/clickhouse-no-nulls --path=$HOME/clickhouse-no-nulls --runs=3 --query="SELECT toUInt16(bitAnd(number, 8191)) AS key, sum(number) FROM numbers(500000000) GROUP BY key FORMAT Null SETTINGS max_threads = 1"
wall:   1.10s, cpu:   1.09s, memory:   0.00GiB
wall:   1.10s, cpu:   1.09s, memory:   0.00GiB
wall:   1.11s, cpu:   1.10s, memory:   0.00GiB
```
@bobrik @subkanthi
Propagate non-null aggregate places through combinators
109856c 
Move the non-null batch implementation to `IAggregateFunctionHelper`
and propagate the guarantee through `AggregateFunctionIf`,
`AggregateFunctionOrFill`, and `AggregateFunctionTuple`.

Preserve specialized timeseries batching.

For `sumOrDefault` over 500 million rows and 8192 groups: 1.64s -> 1.5s.

No measurable change was observed for `sumIf`.
@subkanthi
Speed up aggregation batches with non-null states
b2cca28 
Aggregations with lots of buckets can get expensive. Here's an excerpt from a profile:

```
ROUTINE ======================== DB::IAggregateFunctionHelper<DB::AggregateFunctionSum<unsigned long, unsigned long, DB::AggregateFunctionSumData<unsigned long>, (DB::AggregateFunctionSumType)0> >::addBatch(unsigned long, unsigned long, char**, unsigned long, DB::IColumn const**, DB::Arena*, long) const in src/AggregateFunctions/IAggregateFunction.h
     1.79s      2.32s (flat, cum) 15.07% of Total
         .          .    549:            }
         .          .    550:        }
         .          .    551:        else
         .          .    552:        {
         .          .    553:            for (size_t i = row_begin; i < row_end; ++i)
     1.79s      1.79s    554:                if (places[i])
         .   526.32ms    555:                    static_cast<const Derived *>(this)->add(places[i] + place_offset, columns, i, arena);
         .          .    556:        }
         .          .    557:    }
         .          .    558:
         .          .    559:    void serializeBatch(const PaddedPODArray<AggregateDataPtr> & data, size_t start, size_t size, WriteBuffer & buf, std::optional<size_t> version) const final // NOLINT
         .          .    560:    {
```

We can get some of this back by specializing the common non-null version.

For a big query doing a very wide aggregation:

* 1 thread: 77.7s -> 74.3s (both wall and cpu)
* 64 threads: 17.0s -> 16.0s (wall), 99.5s -> 92.4s (cpu)

With a synthetic benchmark:

* Baseline: 1.24s

```
$ ../bench.py --clickhouse=/tmp/clickhouse-baseline --path=$HOME/clickhouse-no-nulls --runs=3 --query="SELECT toUInt16(bitAnd(number, 8191)) AS key, sum(number) FROM numbers(500000000) GROUP BY key FORMAT Null SETTINGS max_threads = 1"
wall:   1.24s, cpu:   1.23s, memory:   0.00GiB
wall:   1.25s, cpu:   1.24s, memory:   0.00GiB
wall:   1.23s, cpu:   1.23s, memory:   0.00GiB
```

* Patched: 1.10s

```
$ ../bench.py --clickhouse=/tmp/clickhouse-no-nulls --path=$HOME/clickhouse-no-nulls --runs=3 --query="SELECT toUInt16(bitAnd(number, 8191)) AS key, sum(number) FROM numbers(500000000) GROUP BY key FORMAT Null SETTINGS max_threads = 1"
wall:   1.10s, cpu:   1.09s, memory:   0.00GiB
wall:   1.10s, cpu:   1.09s, memory:   0.00GiB
wall:   1.11s, cpu:   1.10s, memory:   0.00GiB
```
@subkanthi
Propagate non-null aggregate places through combinators
e1e2501 
Move the non-null batch implementation to `IAggregateFunctionHelper`
and propagate the guarantee through `AggregateFunctionIf`,
`AggregateFunctionOrFill`, and `AggregateFunctionTuple`.

Preserve specialized timeseries batching.

For `sumOrDefault` over 500 million rows and 8192 groups: 1.64s -> 1.5s.

No measurable change was observed for `sumIf`.
@subkanthi
Merge branch 'backports/antalya-26.8/117009' of https://github.com/Al…
5671fe7 
…tinity/ClickHouse into backports/antalya-26.8/117009
@github-actions

github-actions Bot commented Nov 2, 2026 •
edited
Loading

Copy link
Copy Markdown

Workflow [PR], commit [5671fe7]

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@subkanthi @bobrik