congqixia
bd222e58eb
enhance: [2.4] Exclude L0 segment from readable snapshot ( #35510 )
...
Cherry-pick from master
pr: #35507
L0 segments now do not contain insert data and may cause confusion for
query hook optimizer if counted as sealed segment number.
This PR add segment level flag in segment entry and exclude L0 segments
while get readable segment snapshot
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-16 15:26:54 +08:00
congqixia
555b7a6aa7
enhance: [2.4] Remove duplicated schema helper creation in proxy ( #35489 ) ( #35502 )
...
Cherry-pick from master
pr: #35489
Related to PRs of #35415
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-16 14:32:54 +08:00
cai.zhang
09aea3fbf1
enhance: [cherry-pick] Optimize the use of locks and avoid double flush clustering buffer writer ( #35490 )
...
issue: #35436
master pr: #35486
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-16 02:24:59 +08:00
SimFG
4fae5407dd
enhance: [2.4] remove the invalid log ( #35473 )
...
/kind improvement
pr: #35465
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-15 14:14:53 +08:00
congqixia
c8926aee23
enhance: [2.4] Support dynamic field in SchemaHelper ( #35461 ) ( #35469 )
...
Cherry-pick from master
pr: #35461
Related to #35415
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-15 10:02:54 +08:00
XuanYang-cn
0fb3bf43cd
enhance: Init ChannelCP when creating a channel ( #35388 )
...
pr: #35387
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-14 10:16:58 +08:00
Bingyi Sun
acd3c53bda
fix: fix sparse float vector column warmup ( #35437 )
...
https://github.com/milvus-io/milvus/issues/35438
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-08-13 15:18:20 +08:00
aoiasd
a20cb727eb
enhance:[Cherry-pick] Check by proxy rate limiter when delete get data by query. ( #30891 ) ( #35262 )
...
relate: https://github.com/milvus-io/milvus/issues/30927
pr: https://github.com/milvus-io/milvus/pull/30891
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-08-13 14:32:21 +08:00
wei liu
8cd6718672
enhance: limit getSegmentInfo batch size to avoid excced grpc message limit ( #35432 )
...
issue: #35395
pr: #35394
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-13 11:42:19 +08:00
wei liu
b316040634
fix: force update next target if target can't be loaded ( #35366 )
...
issue: #35361
pr: #35365
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-13 10:24:20 +08:00
yihao.dai
b371467604
enhance: Remove the check for channel cp lag when generating compaction plan ( #35383 ) ( #35384 )
...
issue: https://github.com/milvus-io/milvus/issues/35382
pr: https://github.com/milvus-io/milvus/pull/35383
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-13 10:02:20 +08:00
smellthemoon
23052f6ac2
fix: upsert use the previous pk in insert when autoid( #34672 ) ( #35130 )
...
issue: #34668
pr: #34672
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-12 18:58:25 +08:00
zhagnlu
36e6cba771
enhance: optimize index interface ( #35363 )
...
pr: #35314
cherry-pick part code from commit
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-10 20:18:19 +08:00
Bingyi Sun
c47d039f1e
fix: fix empty indices of sparse float ( #35400 )
...
https://github.com/milvus-io/milvus/issues/35401
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-08-09 20:52:18 +08:00
jaime
623f9d3c53
fix: incorrect memory size and delta count in SegmentInfo ( #35370 )
...
issue: #35368
pr: #35369
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-09 20:34:19 +08:00
congqixia
e3cb994ca3
enhance: [2.4] Fix progress always zero due to integer divide ( #35393 )
...
Cherry-pick from master
pr: #35374
See also #35183
Signed-off-by: Congqi.Xia <congqi.xia@zilliz.com>
2024-08-09 17:20:18 +08:00
wei liu
a4fa9ced16
fix: delete by expr failed at retry progress ( #35241 ) ( #35268 )
...
issue: #35240
pr: #35241
delete by expr shard the same err object between channels, so if one
channel's query failed, it will fail all channel, which will break
channel level retry policy, and make delete operation failed.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-09 16:12:18 +08:00
wayblink
f7a3fd8bbf
enhance:[cherry-pick] improve ut cov of clustering compaction task ( #35243 )
...
issue: #34792
master pr: #35242
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-09 15:26:18 +08:00
Patrick Weizhi Xu
96c53bcf2b
fix: [2.4] rollback grpc to 1.50.1 ( #35386 )
...
issue: #35298
pr: #35385
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit faf151050a50f1565b9e8cc48c754e30ec5c65d8)
2024-08-09 14:06:18 +08:00
congqixia
537a817be9
fix: [2.4] Use k locations only for basic BF test location ( #35381 )
...
Cherry-pick from master
pr: #35380
Related to #35379
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-09 07:52:22 +08:00
XuanYang-cn
5cd462bc9e
enhance: Init channel cp metric when DataCoord recover ( #35352 )
...
pr: #35353
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-08 12:22:18 +08:00
yellow-shine
ddb52e71cb
enhance: pinpoint revision of 3thirdparty package ( #35316 )
...
https://github.com/milvus-io/milvus/pull/35325
---------
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-08 09:46:22 +08:00
zhagnlu
db9e4b898a
enhance: support bitmap index ( #35336 )
...
pr: #32902
cherry-pick bitmap index from master
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-07 20:12:18 +08:00
wei liu
b22f3a6276
fix: datacoord binlog size metrics include dropped segment ( #35339 )
...
issue: #35337
pr: #35338
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-07 15:12:17 +08:00
Patrick Weizhi Xu
5369ada091
fix: [2.4] init telemetry fails when set otlp as trace exporter ( #35299 )
...
issue: #35298
pr: #35301
2024-08-07 14:20:21 +08:00
wei liu
0201e00a2f
enhance: enable to set load config in cluster level ( #35293 )
...
issue: #35170
pr: #35169
This PR enable to set load configs in cluster level, such as replicas
and resource groups. then when load collections will use the load
config.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-07 12:38:21 +08:00
cai.zhang
4f0c1982d3
fix: [cherry-pick] Fix the issue of missing stats log after clustering compaction ( #35267 )
...
issue: #35265
master pr: #35266
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-07 12:08:16 +08:00
Jiquan Long
a00688c00a
fix: comparision operations between incompatible operands ( #35264 ) ( #35307 )
...
fix: https://github.com/milvus-io/milvus/issues/34139
pr: https://github.com/milvus-io/milvus/pull/35264
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-07 11:24:16 +08:00
Jiquan Long
d14d00b0cc
enhance: add metric to record disk quota ( #35305 ) ( #35306 )
...
https://github.com/milvus-io/milvus/issues/35177
pr: https://github.com/milvus-io/milvus/pull/35305
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-07 10:24:27 +08:00
Jiquan Long
b11db52160
fix: error message when input is not aligned ( #35322 ) ( #35323 )
...
fix: https://github.com/milvus-io/milvus/issues/35321
pr: https://github.com/milvus-io/milvus/pull/35322
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-07 09:56:28 +08:00
congqixia
e64d27aa51
fix: [2.4] Set corresponding DataScope for loadStreamDelete ( #35313 )
...
Cherry-pick from master
pr: #35312
Related to #35311
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-06 22:32:23 +08:00
yihao.dai
d0ce3f602e
enhance: Add import option to skip disk quota check ( #35274 ) ( #35275 )
...
Add an option to skip the disk quota check for backup-restore import.
issue: https://github.com/milvus-io/milvus/issues/33775
pr: https://github.com/milvus-io/milvus/pull/35274
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-06 15:54:17 +08:00
foxspy
6e8083d4a3
enhance: [2.4] Add disk filemananger parallel load control to reduce the memory consumption ( #35282 )
...
issue: #35280
pr: #35281
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-08-06 15:46:16 +08:00
wei liu
2ac1bf7532
enhance: Enable setting the replica number and resource group during collection creation ( #34403 ) ( #34561 )
...
issue: #30040
pr: #34403
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-06 15:06:17 +08:00
congqixia
2a1fc700a2
enhance: [2.4] Add in-memory cache for casbin enforcer result ( #35272 )
...
Cherry-pick from master
pr: #35271
See also #35270
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-06 12:14:17 +08:00
yihao.dai
7ef96ef648
enhance: Avoid unnecessary compaction ( #35148 ) ( #35149 )
...
Estimate the import segment size based on DiskSegmentMaxSize(2G) to
avoid unnecessary compaction after import completed.
issue: https://github.com/milvus-io/milvus/issues/35147
pr: https://github.com/milvus-io/milvus/pull/35148
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-06 10:30:20 +08:00
Jiquan Long
e598233b6a
feat: put inverted index into ram ( #35222 ) ( #35223 )
...
fix: https://github.com/milvus-io/milvus/issues/35224
pr: https://github.com/milvus-io/milvus/pull/35222
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-06 10:08:16 +08:00
congqixia
1b304e982a
enhance: [2.4] Resolve chunk cache conflict cpp lint ( #35278 )
...
Resolve cpp lint issue introduced in #35189
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-05 19:46:16 +08:00
yihao.dai
7f49476363
enhance: Refine import error log ( #35067 ) ( #35068 )
...
issue: https://github.com/milvus-io/milvus/issues/35060
pr: https://github.com/milvus-io/milvus/pull/35067
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-05 15:28:17 +08:00
wei liu
d48c690cb3
enhance: Avoid unnecesary syncTargetVersion func call after querycoord recover ( #34954 ) ( #35234 )
...
pr: #34954
before querycoord stop gracefully, we will save the current target to
meta store and recover it after querycoord start up, to speed the
querycoord's recovery time. but the target version hasn't been recovered
as expected, and it use latest timestamp as current target's version,
which has no effect to querycoord but an unnecessary syncTargetVersion
func call.
This PR recover the correct target version to avoid unnecessary
syncTargetVersion func call
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-05 10:18:16 +08:00
cai.zhang
2534b30e39
enhance: [cherry-pick] Add monitoring metrics for task execution time in datacoord ( #35141 )
...
issue: #35138
master pr: #35139
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-02 19:46:16 +08:00
Chun Han
58f7c35b75
enhance: add log for partition stats( #30376 ) ( #35220 )
...
related: #30376
pr: https://github.com/milvus-io/milvus/pull/35219
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-08-02 19:34:21 +08:00
cqy123456
d9c1990045
enhance:[cherry-pick]update mmapchunkmanager config in milvus.yaml ( #35186 )
...
issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/35180
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-08-02 19:26:48 +08:00
Chun Han
f8676178f7
fix: querynode hang when failing to allocate disk space for mmap( #35184 ) ( #35189 )
...
related: #35184
pr: https://github.com/milvus-io/milvus/pull/35187
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-08-02 19:20:16 +08:00
wei liu
2609e95cb5
enhance: add segment state on data coord bin log size metrics ( #35153 ) ( #35235 )
...
pr: #35153
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-02 18:28:14 +08:00
wei liu
11578772ef
fix: Set legacy level to l0 segment after qc restart ( #35197 ) ( #35211 )
...
issue: #35087
pr: #35197
after qc restarts, and target is not ready yet, if dist_handler try to
update segment dist, it will set legacy level to l0 segment, which may
cause l0 segment be moved to other node, cause search/query failed.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-02 18:22:15 +08:00
cai.zhang
756922ebec
fix: [cherry-pick] Maintain load idempotency even when building new indexes ( #35179 )
...
issue: #34404
master pr: #35178
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-02 17:28:15 +08:00
congqixia
b70ede4c9b
enhance: [2.4] Set WatchProgress for channel operation progress response ( #35183 ) ( #35231 )
...
Cherry-pick from master
pr: #35183
The watch progress is always zero in CheckChannelOperationProgress
response, which is meaningless and confusing. This PR set progress value
in rpc response to fix this problem.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-02 16:52:15 +08:00
wayblink
fc974c4db1
enhance: [cherry-pick] improve ut for compaction_policy_clustering ( #35206 )
...
#34792
master pr: #35205
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-02 16:38:15 +08:00
wei liu
d767f8977a
enhance: Refine param init for MmapDirPath ( #35181 ) ( #35214 )
...
pr: #35181
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-02 16:30:15 +08:00
