Skip to content

[feature](be) Support binary conversion for Bitmap and HLL - #69913

Open
zghong wants to merge 10 commits into
apache:masterfrom
zghong:feat/read-bitmap-from-binary
Open

zghong wants to merge 10 commits into
apache:masterfrom
zghong:feat/read-bitmap-from-binary

Conversation

@zghong

@zghong zghong commented Oct 31, 2026

Copy link
Copy Markdown

What problem does this PR solve?

Issue Number: close #69912

Related PR: #68786, #57821

Problem Summary:

PR #68786 made external binary mapping permanently enabled, so Hive/HMS BINARY columns are exposed as Doris VARBINARY and enable.mapping.varbinary=false can no longer restore the old STRING mapping.

The Hive UDF documentation describes Bitmap/HLL UDF output that uses Doris serialized binary data, while the existing bitmap_from_base64 and hll_from_base64 functions require Base64 STRING values. Therefore, the documented Hive UDF consumption path cannot directly read the current VARBINARY result.

Add direct binary conversion functions:

bitmap_from_binary(VARBINARY)  -> BITMAP
bitmap_to_binary(BITMAP)       -> VARBINARY
hll_from_binary(VARBINARY)     -> HLL
hll_to_binary(HLL)             -> VARBINARY

The binary readers validate external input and return row-level NULL for malformed values. Bitmap deserialization additionally validates Roaring containers and consumed bytes to prevent incorrect parsing and backend memory-safety failures. hll_from_binary is also allowed in Stream Load and Broker Load mappings.

Release note

Add direct VARBINARY conversion functions for Doris Bitmap and HLL values, with safe handling of malformed binary input.

Check List (For Author)

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zghong

zghong commented Oct 31, 2026

Copy link
Copy Markdown
Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28347 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 75f5b1246d23ae2f9c78271424c71d6404c63b3c, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17685	5133	5020	5020
q2	2184	307	258	258
q3	10244	1377	811	811
q4	4686	466	334	334
q5	7446	806	533	533
q6	187	184	150	150
q7	746	791	581	581
q8	9257	1310	1179	1179
q9	5601	4205	4170	4170
q10	6812	1295	1027	1027
q11	435	244	223	223
q12	639	395	274	274
q13	18120	3602	3020	3020
q14	273	273	240	240
q15	q16	720	718	646	646
q17	1218	756	578	578
q18	6901	6169	6122	6122
q19	1123	968	598	598
q20	383	334	225	225
q21	4959	2112	2239	2112
q22	349	308	246	246
Total cold run time: 99968 ms
Total hot run time: 28347 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5561	5573	5570	5570
q2	670	569	513	513
q3	5159	5405	4952	4952
q4	1965	2129	1545	1545
q5	5352	5201	5536	5201
q6	276	216	154	154
q7	2316	2081	1827	1827
q8	3182	2694	2646	2646
q9	8111	7948	7964	7948
q10	3711	3935	3256	3256
q11	625	475	423	423
q12	709	697	524	524
q13	3548	3885	3327	3327
q14	310	334	295	295
q15	q16	686	710	640	640
q17	8090	7042	6819	6819
q18	13082	12158	12955	12158
q19	934	790	802	790
q20	2154	2164	1922	1922
q21	5541	4563	4659	4563
q22	487	476	425	425
Total cold run time: 72469 ms
Total hot run time: 65498 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151398 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 75f5b1246d23ae2f9c78271424c71d6404c63b3c, data reload: false

query5	4309	575	441	441
query6	426	205	193	193
query7	4799	450	231	231
query8	316	165	156	156
query9	8774	3187	3189	3187
query10	501	328	263	263
query11	5969	3824	3562	3562
query12	140	88	87	87
query13	1264	448	329	329
query14	6155	3907	3646	3646
query14_1	3433	3409	3400	3400
query15	205	191	187	187
query16	952	421	405	405
query17	865	677	529	529
query18	2416	424	309	309
query19	193	164	140	140
query20	83	85	81	81
query21	211	132	106	106
query22	13040	13030	12753	12753
query23	13485	12836	12450	12450
query23_1	12534	12561	12495	12495
query24	7396	824	475	475
query24_1	483	482	489	482
query25	539	410	364	364
query26	1251	260	141	141
query27	2757	449	265	265
query28	4609	2033	2034	2033
query29	1564	613	471	471
query30	300	223	189	189
query31	1000	930	820	820
query32	143	100	94	94
query33	544	310	242	242
query34	907	877	501	501
query35	749	747	648	648
query36	829	802	727	727
query37	134	100	84	84
query38	1789	1745	1670	1670
query39	801	801	745	745
query39_1	729	721	739	721
query40	217	128	94	94
query41	76	69	68	68
query42	90	82	83	82
query43	351	365	315	315
query44	1308	706	702	702
query45	190	182	173	173
query46	824	963	600	600
query47	2844	2925	2846	2846
query48	295	305	218	218
query49	576	386	290	290
query50	671	264	205	205
query51	10700	10605	10570	10570
query52	80	80	68	68
query53	186	211	149	149
query54	235	188	184	184
query55	72	66	63	63
query56	221	220	197	197
query57	1650	1627	1571	1571
query58	291	250	263	250
query59	2183	2218	1986	1986
query60	281	248	220	220
query61	143	142	138	138
query62	393	326	286	286
query63	189	155	157	155
query64	2726	971	757	757
query65	3327	3284	3282	3282
query66	1782	420	305	305
query67	20145	20104	20042	20042
query68	2982	988	604	604
query69	390	279	286	279
query70	851	795	801	795
query71	300	237	213	213
query72	2683	2545	2284	2284
query73	510	516	294	294
query74	4610	4498	4376	4376
query75	2340	2284	1895	1895
query76	2314	1010	593	593
query77	342	358	272	272
query78	9704	9733	9189	9189
query79	993	853	527	527
query80	981	418	336	336
query81	601	321	266	266
query82	572	132	106	106
query83	299	204	186	186
query84	318	120	102	102
query85	987	427	366	366
query86	353	242	231	231
query87	1974	1919	1818	1818
query88	3764	2773	2761	2761
query89	330	281	238	238
query90	1745	165	168	165
query91	161	151	127	127
query92	98	88	89	88
query93	988	995	569	569
query94	516	298	258	258
query95	578	397	300	300
query96	698	488	236	236
query97	2439	2439	2315	2315
query98	158	155	147	147
query99	759	764	634	634
Total cold run time: 234777 ms
Total hot run time: 151398 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.39 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 75f5b1246d23ae2f9c78271424c71d6404c63b3c, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.04
query3	0.26	0.08	0.08
query4	1.61	0.09	0.12
query5	0.24	0.24	0.25
query6	1.17	0.67	0.68
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.40	0.35	0.35
query10	0.57	0.57	0.58
query11	0.20	0.15	0.14
query12	0.18	0.15	0.15
query13	0.50	0.48	0.48
query14	1.03	1.02	1.02
query15	0.66	0.60	0.60
query16	0.36	0.34	0.36
query17	1.17	1.11	1.14
query18	0.20	0.19	0.19
query19	2.07	2.00	2.02
query20	0.02	0.02	0.01
query21	15.51	0.20	0.13
query22	4.83	0.05	0.06
query23	16.28	0.33	0.12
query24	2.96	0.56	0.44
query25	0.10	0.05	0.04
query26	1.26	0.21	0.16
query27	0.04	0.05	0.06
query28	3.17	0.68	0.39
query29	12.49	4.16	3.22
query30	0.29	0.13	0.10
query31	2.77	0.56	0.33
query32	3.22	0.59	0.50
query33	3.23	3.32	3.18
query34	15.41	4.03	3.39
query35	3.34	3.33	3.34
query36	0.55	0.48	0.46
query37	0.10	0.07	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.03
query40	0.17	0.14	0.14
query41	0.09	0.03	0.03
query42	0.03	0.04	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.82 s
Total hot run time: 24.39 s

@zghong

zghong commented Oct 31, 2026

Copy link
Copy Markdown
Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28295 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ec7d5db490f9b05307a146bdcee0991c37c30743, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17579	5124	5042	5042
q2	2117	301	250	250
q3	10261	1348	817	817
q4	4688	456	332	332
q5	7451	789	524	524
q6	179	189	151	151
q7	740	790	601	601
q8	9268	1322	1084	1084
q9	5514	4166	4161	4161
q10	6825	1284	1013	1013
q11	416	241	237	237
q12	644	398	277	277
q13	18093	3587	2987	2987
q14	273	288	253	253
q15	q16	719	707	660	660
q17	1202	754	585	585
q18	6942	6152	6095	6095
q19	1100	959	592	592
q20	368	323	231	231
q21	4716	2152	2237	2152
q22	342	298	251	251
Total cold run time: 99437 ms
Total hot run time: 28295 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5610	5557	5599	5557
q2	681	579	533	533
q3	5148	5398	4878	4878
q4	1982	2110	1550	1550
q5	5221	5131	5382	5131
q6	254	200	153	153
q7	2319	2062	1893	1893
q8	3030	2695	2617	2617
q9	8103	7837	8002	7837
q10	3718	3759	3201	3201
q11	635	462	430	430
q12	693	702	517	517
q13	3624	3878	3332	3332
q14	304	303	304	303
q15	q16	716	711	657	657
q17	8273	7070	6825	6825
q18	12976	12188	12918	12188
q19	915	799	776	776
q20	2137	2151	1910	1910
q21	5520	4701	4649	4649
q22	483	490	410	410
Total cold run time: 72342 ms
Total hot run time: 65347 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151589 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ec7d5db490f9b05307a146bdcee0991c37c30743, data reload: false

query5	4314	602	427	427
query6	428	209	210	209
query7	4802	457	222	222
query8	339	165	152	152
query9	8784	3176	3166	3166
query10	518	311	288	288
query11	6009	3863	3574	3574
query12	150	92	87	87
query13	1238	468	314	314
query14	6098	3918	3771	3771
query14_1	3414	3382	3417	3382
query15	198	199	194	194
query16	958	420	393	393
query17	837	652	535	535
query18	2407	420	294	294
query19	190	154	135	135
query20	83	84	80	80
query21	206	127	112	112
query22	12996	13046	12813	12813
query23	13491	12831	12336	12336
query23_1	12596	12489	12369	12369
query24	7465	865	469	469
query24_1	482	504	500	500
query25	543	426	368	368
query26	1270	262	147	147
query27	2918	463	316	316
query28	4905	2043	2038	2038
query29	1548	582	430	430
query30	298	221	189	189
query31	1010	945	829	829
query32	136	93	90	90
query33	513	295	232	232
query34	904	854	498	498
query35	720	742	637	637
query36	832	829	732	732
query37	123	98	79	79
query38	1774	1726	1672	1672
query39	787	790	740	740
query39_1	724	719	749	719
query40	222	121	95	95
query41	66	67	61	61
query42	86	82	81	81
query43	338	358	314	314
query44	1348	702	706	702
query45	177	174	166	166
query46	827	959	565	565
query47	2886	3033	2717	2717
query48	303	315	217	217
query49	572	398	285	285
query50	695	281	208	208
query51	10720	10670	10608	10608
query52	77	80	68	68
query53	187	203	150	150
query54	225	207	185	185
query55	68	65	65	65
query56	221	210	209	209
query57	1655	1631	1562	1562
query58	286	270	253	253
query59	2106	2156	1946	1946
query60	310	254	241	241
query61	177	169	165	165
query62	387	330	288	288
query63	193	172	161	161
query64	2693	910	741	741
query65	3305	3293	3272	3272
query66	1760	419	298	298
query67	20235	20205	20084	20084
query68	2974	1039	606	606
query69	391	285	240	240
query70	818	791	759	759
query71	299	241	217	217
query72	2736	2599	2445	2445
query73	518	539	303	303
query74	4615	4517	4380	4380
query75	2347	2309	1936	1936
query76	2338	1025	605	605
query77	345	383	280	280
query78	9603	9782	9181	9181
query79	906	853	544	544
query80	502	438	361	361
query81	525	332	281	281
query82	213	140	118	118
query83	201	205	188	188
query84	291	135	105	105
query85	847	476	435	435
query86	258	241	228	228
query87	2002	1949	1836	1836
query88	3780	2830	2770	2770
query89	318	282	242	242
query90	2103	176	164	164
query91	160	158	127	127
query92	102	85	90	85
query93	1004	987	567	567
query94	428	304	271	271
query95	580	327	355	327
query96	693	514	242	242
query97	2450	2421	2305	2305
query98	161	162	148	148
query99	754	774	656	656
Total cold run time: 234189 ms
Total hot run time: 151589 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.2 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ec7d5db490f9b05307a146bdcee0991c37c30743, data reload: false

query1	0.01	0.01	0.00
query2	0.09	0.05	0.05
query3	0.27	0.09	0.09
query4	1.60	0.10	0.10
query5	0.23	0.25	0.24
query6	1.17	0.69	0.66
query7	0.04	0.01	0.01
query8	0.05	0.04	0.03
query9	0.40	0.35	0.34
query10	0.58	0.57	0.59
query11	0.20	0.14	0.14
query12	0.18	0.16	0.14
query13	0.50	0.49	0.49
query14	1.04	1.01	1.00
query15	0.63	0.59	0.60
query16	0.35	0.35	0.34
query17	1.15	1.14	1.17
query18	0.21	0.19	0.20
query19	2.04	1.90	2.03
query20	0.02	0.01	0.02
query21	15.44	0.21	0.14
query22	4.82	0.05	0.06
query23	16.27	0.33	0.12
query24	3.09	0.60	0.36
query25	0.10	0.06	0.05
query26	1.26	0.21	0.14
query27	0.05	0.05	0.04
query28	3.49	0.79	0.36
query29	12.48	4.21	3.30
query30	0.28	0.12	0.09
query31	2.77	0.55	0.33
query32	3.22	0.60	0.50
query33	3.26	3.18	3.24
query34	15.30	3.97	3.39
query35	3.31	3.35	3.33
query36	0.57	0.44	0.43
query37	0.10	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.04	0.03
query40	0.18	0.15	0.14
query41	0.09	0.03	0.03
query42	0.03	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 96.99 s
Total hot run time: 24.2 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 8.00% (4/50) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 75.31% (180/239) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 67.47% (32917/48790)
Line Coverage 52.43% (349810/667134)
Region Coverage 47.76% (283898/594399)
Branch Coverage 49.45% (134168/271294)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 75.31% (180/239) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.71% (36745/47283)
Line Coverage 63.07% (416333/660088)
Region Coverage 59.37% (351616/592237)
Branch Coverage 60.33% (163263/270634)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 90.00% (45/50) 🎉
Increment coverage report
Complete coverage report

@zghong

zghong commented Nov 1, 2026

Copy link
Copy Markdown
Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28274 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d76177ee8a2c1d2f2a2ca551875f1a6cd6558c67, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17607	5229	5036	5036
q2	2100	292	243	243
q3	10294	1395	799	799
q4	4680	459	345	345
q5	7418	791	526	526
q6	183	186	153	153
q7	738	798	596	596
q8	9259	1290	1037	1037
q9	5515	4190	4210	4190
q10	6828	1282	1031	1031
q11	431	239	218	218
q12	635	399	288	288
q13	18086	3505	3008	3008
q14	272	273	253	253
q15	q16	722	705	670	670
q17	1200	766	575	575
q18	6980	6141	6108	6108
q19	1097	938	582	582
q20	382	327	221	221
q21	4655	2146	2245	2146
q22	347	284	249	249
Total cold run time: 99429 ms
Total hot run time: 28274 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5621	5609	5608	5608
q2	703	551	554	551
q3	5201	5363	4910	4910
q4	1980	2134	1544	1544
q5	5267	5244	5373	5244
q6	257	205	157	157
q7	2369	2029	1872	1872
q8	2991	2688	2640	2640
q9	8121	7955	8046	7955
q10	3738	3887	3223	3223
q11	601	452	421	421
q12	699	711	516	516
q13	3561	3896	3295	3295
q14	316	308	301	301
q15	q16	716	718	660	660
q17	8340	7045	6876	6876
q18	13020	12169	12793	12169
q19	894	767	782	767
q20	2130	2120	1913	1913
q21	5521	4550	4786	4550
q22	490	471	399	399
Total cold run time: 72536 ms
Total hot run time: 65571 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 150808 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d76177ee8a2c1d2f2a2ca551875f1a6cd6558c67, data reload: false

query5	4324	588	425	425
query6	423	207	188	188
query7	4797	448	225	225
query8	321	170	153	153
query9	8778	3146	3197	3146
query10	483	307	261	261
query11	6039	3844	3557	3557
query12	147	91	85	85
query13	1252	421	309	309
query14	6083	3902	3633	3633
query14_1	3444	3386	3395	3386
query15	199	196	177	177
query16	931	427	393	393
query17	828	636	495	495
query18	2417	404	295	295
query19	199	151	130	130
query20	80	82	80	80
query21	203	123	109	109
query22	12912	12989	12701	12701
query23	13501	12824	12532	12532
query23_1	12560	12545	12605	12545
query24	7364	813	471	471
query24_1	470	475	473	473
query25	513	398	335	335
query26	1252	256	134	134
query27	2762	440	271	271
query28	4686	2007	2001	2001
query29	1577	599	453	453
query30	311	210	186	186
query31	1010	936	810	810
query32	142	95	95	95
query33	521	309	269	269
query34	987	847	497	497
query35	722	751	637	637
query36	812	843	732	732
query37	124	97	87	87
query38	1763	1742	1680	1680
query39	790	788	733	733
query39_1	732	721	748	721
query40	223	115	97	97
query41	74	89	66	66
query42	88	88	83	83
query43	361	370	310	310
query44	1314	684	708	684
query45	184	188	167	167
query46	830	956	568	568
query47	2946	2817	2601	2601
query48	302	309	226	226
query49	588	404	310	310
query50	687	277	210	210
query51	10569	10692	10557	10557
query52	81	84	72	72
query53	188	213	153	153
query54	244	222	200	200
query55	80	71	68	68
query56	258	228	216	216
query57	1653	1599	1567	1567
query58	292	270	265	265
query59	2174	2207	2013	2013
query60	306	250	234	234
query61	168	163	159	159
query62	401	334	281	281
query63	188	168	164	164
query64	2833	1050	784	784
query65	3310	3263	3265	3263
query66	1775	416	300	300
query67	20183	20293	20012	20012
query68	3007	991	604	604
query69	391	287	249	249
query70	846	792	765	765
query71	289	237	214	214
query72	2707	2599	2268	2268
query73	505	523	289	289
query74	4605	4508	4323	4323
query75	2357	2290	1901	1901
query76	2301	1003	584	584
query77	333	364	263	263
query78	9770	9689	9179	9179
query79	988	850	520	520
query80	907	439	341	341
query81	577	316	276	276
query82	592	143	106	106
query83	308	193	174	174
query84	318	122	102	102
query85	853	427	376	376
query86	348	239	233	233
query87	2018	1952	1799	1799
query88	3753	2778	2765	2765
query89	331	282	235	235
query90	1798	170	174	170
query91	163	150	124	124
query92	102	88	87	87
query93	990	986	571	571
query94	538	301	254	254
query95	605	393	301	301
query96	690	481	245	245
query97	2453	2444	2318	2318
query98	164	153	147	147
query99	779	760	649	649
Total cold run time: 234889 ms
Total hot run time: 150808 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.24 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d76177ee8a2c1d2f2a2ca551875f1a6cd6558c67, data reload: false

query1	0.01	0.00	0.00
query2	0.10	0.05	0.05
query3	0.27	0.09	0.09
query4	1.61	0.09	0.09
query5	0.23	0.24	0.25
query6	1.16	0.68	0.67
query7	0.04	0.01	0.01
query8	0.05	0.04	0.04
query9	0.42	0.37	0.35
query10	0.58	0.60	0.62
query11	0.20	0.15	0.14
query12	0.19	0.15	0.15
query13	0.49	0.48	0.50
query14	1.04	1.02	1.00
query15	0.62	0.60	0.61
query16	0.33	0.35	0.35
query17	1.13	1.13	1.11
query18	0.21	0.22	0.22
query19	2.09	2.00	2.04
query20	0.02	0.02	0.01
query21	15.49	0.19	0.13
query22	4.87	0.06	0.05
query23	16.27	0.31	0.13
query24	3.04	0.55	0.31
query25	0.13	0.05	0.06
query26	1.16	0.21	0.15
query27	0.04	0.03	0.05
query28	3.47	0.80	0.39
query29	12.50	4.13	3.27
query30	0.28	0.13	0.10
query31	2.77	0.55	0.33
query32	3.23	0.60	0.50
query33	3.22	3.19	3.16
query34	15.39	4.04	3.37
query35	3.36	3.30	3.36
query36	0.56	0.47	0.43
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.03
query40	0.17	0.15	0.14
query41	0.09	0.02	0.02
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 97.1 s
Total hot run time: 24.24 s

@zghong
zghong force-pushed the feat/read-bitmap-from-binary branch from d76177e to 09cc06c Compare November 1, 2026 18:34
@zghong

zghong commented Nov 1, 2026

Copy link
Copy Markdown
Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28599 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 09cc06cf5f49fa1b00ed664b5b4b7ed6652a5cfd, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17623	5209	5092	5092
q2	2114	302	257	257
q3	10268	1375	821	821
q4	4666	456	337	337
q5	7439	818	533	533
q6	210	190	162	162
q7	737	794	604	604
q8	9246	1284	1143	1143
q9	5448	4171	4164	4164
q10	6825	1282	1033	1033
q11	439	248	229	229
q12	639	403	295	295
q13	18065	3533	3041	3041
q14	269	271	242	242
q15	q16	726	721	666	666
q17	1190	737	590	590
q18	6939	6129	6090	6090
q19	1101	938	617	617
q20	390	334	221	221
q21	5024	2206	2337	2206
q22	342	295	256	256
Total cold run time: 99700 ms
Total hot run time: 28599 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5619	5627	5657	5627
q2	693	573	536	536
q3	5208	5418	4926	4926
q4	1992	2139	1582	1582
q5	5227	5118	5458	5118
q6	248	209	166	166
q7	2420	2039	1850	1850
q8	3002	2745	2764	2745
q9	8010	7933	7929	7929
q10	3776	3924	3254	3254
q11	634	475	422	422
q12	706	703	545	545
q13	3434	3951	3265	3265
q14	305	324	307	307
q15	q16	703	729	645	645
q17	8453	7235	6992	6992
q18	12965	12135	12864	12135
q19	933	812	822	812
q20	2135	2172	1937	1937
q21	5607	4669	4935	4669
q22	493	482	418	418
Total cold run time: 72563 ms
Total hot run time: 65880 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151579 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 09cc06cf5f49fa1b00ed664b5b4b7ed6652a5cfd, data reload: false

query5	4307	592	444	444
query6	430	206	223	206
query7	4822	435	228	228
query8	315	164	150	150
query9	8789	3185	3183	3183
query10	485	317	262	262
query11	5955	3851	3575	3575
query12	147	93	88	88
query13	1247	441	334	334
query14	6117	3894	3607	3607
query14_1	3419	3522	3416	3416
query15	204	210	185	185
query16	942	438	410	410
query17	854	671	536	536
query18	2430	428	309	309
query19	200	169	138	138
query20	88	84	82	82
query21	209	131	107	107
query22	12884	12976	12927	12927
query23	13640	12788	12484	12484
query23_1	12664	12450	12554	12450
query24	7365	832	467	467
query24_1	503	479	486	479
query25	542	430	360	360
query26	1278	261	140	140
query27	2763	434	278	278
query28	4652	2044	2025	2025
query29	1583	618	463	463
query30	299	227	185	185
query31	1010	926	837	837
query32	156	97	97	97
query33	544	307	245	245
query34	950	849	515	515
query35	741	750	652	652
query36	831	796	734	734
query37	127	97	85	85
query38	1775	1733	1682	1682
query39	803	796	734	734
query39_1	740	737	746	737
query40	253	132	97	97
query41	74	72	70	70
query42	90	91	91	91
query43	349	362	308	308
query44	1341	703	708	703
query45	181	178	168	168
query46	867	991	606	606
query47	2823	2938	2835	2835
query48	309	315	225	225
query49	568	390	281	281
query50	711	277	218	218
query51	10485	10712	10647	10647
query52	82	83	71	71
query53	194	204	148	148
query54	250	188	197	188
query55	73	71	63	63
query56	237	225	199	199
query57	1650	1631	1545	1545
query58	274	265	261	261
query59	2128	2173	1995	1995
query60	288	242	225	225
query61	150	145	146	145
query62	384	327	309	309
query63	192	160	162	160
query64	2753	968	760	760
query65	3353	3273	3287	3273
query66	1795	413	291	291
query67	20134	20060	20011	20011
query68	2935	1016	598	598
query69	394	293	235	235
query70	862	771	784	771
query71	294	221	209	209
query72	2748	2590	2256	2256
query73	514	533	301	301
query74	4633	4502	4339	4339
query75	2304	2296	1933	1933
query76	2311	1012	589	589
query77	354	367	276	276
query78	9816	9667	9141	9141
query79	1022	858	537	537
query80	886	424	343	343
query81	574	321	272	272
query82	577	135	102	102
query83	303	193	176	176
query84	310	116	100	100
query85	874	432	384	384
query86	350	242	229	229
query87	2003	1950	1785	1785
query88	3838	2807	2778	2778
query89	325	279	253	253
query90	1821	172	175	172
query91	166	151	130	130
query92	99	87	88	87
query93	1012	1012	572	572
query94	514	334	273	273
query95	575	396	304	304
query96	699	491	242	242
query97	2431	2427	2322	2322
query98	159	155	146	146
query99	772	773	643	643
Total cold run time: 235033 ms
Total hot run time: 151579 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.4 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 09cc06cf5f49fa1b00ed664b5b4b7ed6652a5cfd, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.05
query3	0.26	0.09	0.08
query4	1.62	0.11	0.10
query5	0.24	0.25	0.24
query6	1.17	0.69	0.68
query7	0.04	0.01	0.01
query8	0.05	0.04	0.04
query9	0.41	0.35	0.35
query10	0.58	0.57	0.57
query11	0.20	0.14	0.14
query12	0.18	0.15	0.15
query13	0.48	0.49	0.48
query14	1.01	1.01	1.01
query15	0.65	0.62	0.60
query16	0.34	0.36	0.35
query17	1.18	1.15	1.12
query18	0.21	0.19	0.19
query19	2.09	2.00	1.99
query20	0.02	0.02	0.01
query21	15.50	0.19	0.13
query22	4.95	0.05	0.06
query23	16.28	0.32	0.12
query24	3.05	0.68	0.40
query25	0.12	0.05	0.04
query26	1.07	0.22	0.15
query27	0.04	0.04	0.03
query28	3.58	0.80	0.38
query29	12.50	4.19	3.24
query30	0.28	0.13	0.11
query31	2.77	0.56	0.32
query32	3.22	0.59	0.51
query33	3.12	3.24	3.26
query34	15.32	4.00	3.39
query35	3.36	3.33	3.34
query36	0.55	0.48	0.44
query37	0.09	0.07	0.06
query38	0.06	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.15	0.13
query41	0.09	0.03	0.03
query42	0.03	0.03	0.04
query43	0.05	0.03	0.03
Total cold run time: 97.08 s
Total hot run time: 24.4 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 8.00% (4/50) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Support reading Bitmap and HLL directly from VARBINARY

2 participants