Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

branch-3.0: [improve](mow) add observability on commit and publish cost time #46487 #46736

Open
wants to merge 1 commit into
base: branch-3.0
Choose a base branch
from

Conversation

github-actions[bot]
Copy link
Contributor

@github-actions github-actions bot commented Jan 9, 2025

Cherry-picked from #46487

)

Recording the time on commit and publish phase is helpful to know
whether mow cost too much time on this phase.
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Jan 9, 2025
@dataroaring dataroaring reopened this Jan 9, 2025
@hello-stephen
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40857 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 06f0284348348c744f67ee554ff3dd2fbe764b58, data reload: false

------ Round 1 ----------------------------------
q1	17604	7401	7304	7304
q2	2068	178	165	165
q3	10615	1086	1141	1086
q4	10467	770	722	722
q5	7764	2838	2804	2804
q6	233	145	145	145
q7	978	609	604	604
q8	9359	1923	2024	1923
q9	6589	6418	6356	6356
q10	6954	2318	2325	2318
q11	465	263	265	263
q12	412	216	207	207
q13	18055	3003	3028	3003
q14	237	224	205	205
q15	573	526	550	526
q16	696	602	623	602
q17	971	562	533	533
q18	7243	6843	6654	6654
q19	1405	1077	986	986
q20	476	205	196	196
q21	4074	3266	3277	3266
q22	1109	1011	989	989
Total cold run time: 108347 ms
Total hot run time: 40857 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7386	7442	7280	7280
q2	338	245	237	237
q3	3040	2985	3051	2985
q4	2131	1922	1842	1842
q5	5795	5777	5778	5777
q6	231	148	146	146
q7	2257	1810	1836	1810
q8	3380	3602	3550	3550
q9	9152	9123	9052	9052
q10	3648	3631	3618	3618
q11	629	504	515	504
q12	890	661	696	661
q13	23330	3155	3219	3155
q14	296	274	287	274
q15	588	543	525	525
q16	724	671	696	671
q17	1844	1632	1600	1600
q18	8433	7835	7687	7687
q19	1668	1648	1561	1561
q20	2135	1869	1874	1869
q21	5597	5514	5444	5444
q22	1126	1066	1013	1013
Total cold run time: 84618 ms
Total hot run time: 61261 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198173 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 06f0284348348c744f67ee554ff3dd2fbe764b58, data reload: false

query1	1231	928	926	926
query2	6231	2150	2119	2119
query3	10829	4182	4180	4180
query4	66093	29192	23366	23366
query5	4989	453	449	449
query6	433	198	178	178
query7	5635	315	310	310
query8	320	231	229	229
query9	9447	2698	2695	2695
query10	463	276	260	260
query11	17671	15102	15836	15102
query12	161	113	107	107
query13	1576	456	437	437
query14	10750	7680	7728	7680
query15	200	197	188	188
query16	7141	510	440	440
query17	1065	592	604	592
query18	2054	328	324	324
query19	229	176	173	173
query20	125	113	108	108
query21	213	102	107	102
query22	4906	4478	4548	4478
query23	34616	34214	34392	34214
query24	6190	2948	2913	2913
query25	555	421	437	421
query26	663	169	169	169
query27	1863	362	367	362
query28	4122	2463	2496	2463
query29	758	475	438	438
query30	250	165	165	165
query31	992	829	855	829
query32	71	56	58	56
query33	466	297	288	288
query34	929	504	519	504
query35	880	737	749	737
query36	1094	957	967	957
query37	120	71	76	71
query38	4063	4081	4040	4040
query39	1514	1488	1493	1488
query40	204	107	97	97
query41	49	48	47	47
query42	116	103	99	99
query43	541	499	500	499
query44	1183	833	822	822
query45	187	173	177	173
query46	1148	732	727	727
query47	2056	1910	1972	1910
query48	479	387	393	387
query49	737	396	383	383
query50	842	425	433	425
query51	7371	7344	7147	7147
query52	100	87	87	87
query53	261	179	183	179
query54	559	445	436	436
query55	77	73	76	73
query56	264	249	239	239
query57	1250	1113	1124	1113
query58	214	209	229	209
query59	3106	3151	2974	2974
query60	281	267	257	257
query61	107	105	129	105
query62	785	676	691	676
query63	218	193	187	187
query64	1381	665	676	665
query65	3309	3194	3190	3190
query66	709	305	306	305
query67	15796	15726	15808	15726
query68	4016	560	568	560
query69	421	276	263	263
query70	1210	1090	1057	1057
query71	361	255	252	252
query72	6395	3992	4054	3992
query73	761	340	354	340
query74	10405	9086	8944	8944
query75	3369	2636	2694	2636
query76	1869	1106	1166	1106
query77	479	265	264	264
query78	10891	9627	9599	9599
query79	1107	598	586	586
query80	853	438	439	438
query81	522	242	237	237
query82	205	116	109	109
query83	166	145	163	145
query84	273	81	78	78
query85	924	305	295	295
query86	364	294	293	293
query87	4414	4257	4291	4257
query88	4112	2382	2359	2359
query89	407	298	285	285
query90	2077	191	191	191
query91	188	151	149	149
query92	68	50	51	50
query93	1291	549	557	549
query94	836	302	296	296
query95	353	259	253	253
query96	609	281	277	277
query97	3387	3196	3133	3133
query98	220	206	192	192
query99	1575	1308	1293	1293
Total cold run time: 318366 ms
Total hot run time: 198173 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.78 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 06f0284348348c744f67ee554ff3dd2fbe764b58, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.02
query3	0.22	0.07	0.06
query4	1.63	0.10	0.10
query5	0.53	0.52	0.53
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.58	0.49	0.50
query10	0.54	0.55	0.56
query11	0.14	0.10	0.11
query12	0.15	0.12	0.10
query13	0.60	0.59	0.59
query14	2.96	3.04	2.92
query15	0.91	0.84	0.83
query16	0.39	0.38	0.38
query17	1.06	1.03	1.04
query18	0.24	0.22	0.22
query19	1.86	1.88	1.94
query20	0.01	0.01	0.01
query21	15.37	0.60	0.60
query22	3.12	2.40	1.92
query23	16.95	0.99	0.86
query24	3.32	0.65	1.67
query25	0.26	0.07	0.07
query26	0.50	0.13	0.14
query27	0.05	0.06	0.04
query28	10.48	1.11	1.08
query29	12.56	3.27	3.23
query30	0.24	0.06	0.07
query31	2.84	0.37	0.38
query32	3.27	0.47	0.47
query33	2.96	3.02	3.04
query34	17.22	4.44	4.46
query35	4.50	4.48	4.46
query36	0.67	0.49	0.49
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.03
query40	0.15	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 107.88 s
Total hot run time: 32.78 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants