admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-05 02:12:48 +08:00
Code Issues Packages Projects Releases Wiki Activity
21,474 Commits 136 Branches 188 Tags
Commit Graph

9914 Commits

Author SHA1 Message Date
sre-ci-robot
ffd3c5d2f5
[automated] Update Knowhere Commit (#38542) 
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-12-18 01:24:47 +08:00
cai.zhang
7a05b5bbea
fix: Fix stats task wrong RootPath when upload binlog (#38539) 
issue: #38336

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-12-18 00:18:45 +08:00
yihao.dai
1aa31e2a9b
fix: Fix inaccurate general count (#38524) 
Checking general count should only count healthy collections and
partitions.

issue: https://github.com/milvus-io/milvus/issues/37630

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-12-17 23:24:45 +08:00
jaime
4d5572d7ed
enhance: [skip e2e] refine display format (#38537) 
issue: #36621

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-12-17 22:30:44 +08:00
Chun Han
decdfdae10
fix: growing-groupby-crush(#38533) (#38538) 
related: #38533

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-12-17 21:05:12 +08:00
Bingyi Sun
f0096ec292
fix: Fix IsMmapSupported for scalar index (#38135) 
https://github.com/milvus-io/milvus/issues/38134

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-12-17 20:30:44 +08:00
zhagnlu
9afcc5bc5c
fix:fix incorrect dir operations when create or load inverted index (#38359) 
#37944

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-12-17 20:06:45 +08:00
aoiasd
93fba1d5ab
fix: access log remove new log not the old and skip rotate empty log (#38309) 
relate: https://github.com/milvus-io/milvus/issues/38293

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-12-17 19:16:44 +08:00
jaime
29e620fa6d
fix: sync task still running after DataNode has stopped (#38377) 
issue: #38319

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-12-17 18:06:44 +08:00
zhagnlu
d0a7e98a27
fix:remove incorrect assert for delete query (#38509) 
#38472

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-12-17 17:48:44 +08:00
Zhen Ye
8916fbf122
fix: use new path for streamingnode recovery info (#38516) 
issue: #38399

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-12-17 17:10:45 +08:00
yihao.dai
d3c174b0f1
enhance: Accelerate observe collection (#38028) 
1. A collection should observe the channel only once.
2. A collection should check the CollectionLoadPercent for updates only
once.
3. Skip saving coll/partition meta if there are no changes, primarily to
accelerate collection observation after recovery.

issue: https://github.com/milvus-io/milvus/issues/37630

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-12-17 14:14:45 +08:00
yihao.dai
d4dab3c62f
enhance: Reduce segmentManager lock granularity (#37836) 
Use a channel level key lock for segments in segmentManager.

issue: https://github.com/milvus-io/milvus/issues/37633,
https://github.com/milvus-io/milvus/issues/37630

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-12-17 14:12:52 +08:00
Bingyi Sun
dd4f33ae19
fix: Fix chunked segment can not warmup using mmap (#38492) 
issue: #38410

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-12-17 13:42:45 +08:00
Ted Xu
33aecb0655
fix: build break on target test_cpp under OSX (#38479) 
See: #38434

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-12-17 13:38:45 +08:00
yihao.dai
01fa2f8da5
fix: Fix compile error (#38510) 
/kind improvement

related: #37559

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-12-17 12:16:19 +08:00
Zhen Ye
afac153c26
enhance: move the lifetime implementation out of server level lifetime (#38442) 
issue: #38399

- move the lifetime implementation of common code out of the server
level lifetime implementation

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-12-17 11:42:44 +08:00
jaime
28fdbc4e30
enhance: optimize CPU usage for CheckHealth requests (#35589) 
issue: #35563
1. Use an internal health checker to monitor the cluster's health state,
storing the latest state on the coordinator node. The CheckHealth
request retrieves the cluster's health from this latest state on the
proxy sides, which enhances cluster stability.
2. Each health check will assess all collections and channels, with
detailed failure messages temporarily saved in the latest state.
3. Use CheckHealth request instead of the heavy GetMetrics request on
the querynode and datanode

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-12-17 11:02:45 +08:00
SimFG
2afe2eaf3e
feat: support to replicate collection when the services contains the system tt msg (#37559) 
- issue: #37105

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-12-17 09:08:46 +08:00
Bingyi Sun
3e2a2f278b
enhance: Handle rust error in c++ (#38113) 
https://github.com/milvus-io/milvus/issues/37930

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-12-16 19:40:45 +08:00
wei liu
659847c11f
enhance: Remove load task limit in one round (#38436) 
the task limit in assignSegment/assignChannel will works for both load
task and balance task.

this PR remove the load task limit, only limit balance task num in one
round.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-12-16 19:30:43 +08:00
congqixia
9c8c1b3bb7
enhance: Remove levelZeroMut totally (#38473) 
The level zero mutex could be remove since all operations are guarded by
segment manager mutex

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-12-16 14:38:45 +08:00
wei liu
40f9db491e
fix: Fix SyncDistribution may cost too much time on retry (#38454) 
issue: #38428

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-12-16 11:38:44 +08:00
Ted Xu
4919ccf543
enhance: eliminate compile warnings (#38420) 
See: #38435

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-12-16 09:58:43 +08:00
zhagnlu
c3edc85359
fix: fix wrong size of arrow array for zero-copy mode (#38449) 
#37767

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-12-15 18:14:43 +08:00
zhagnlu
01de0afc4e
enhance: refactor delete mvcc function (#38066) 
#37413

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-12-15 18:02:43 +08:00
zhagnlu
6ea15265e1
enhance: add file info log when mmap failed. (#38386) 
#37944

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-12-15 17:36:43 +08:00
SimFG
fa8ac09550
fix: the issue of replicate message exception when the ttMsgEnable config is changed dynamically (#38178) 
- issue: #38177

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-12-14 23:24:51 +08:00
tinswzy
27229f7907
enhance: refine exists log print with ctx (#38080) 
issue: #35917 
Refines exists log print with ctx

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
 2024-12-14 22:36:44 +08:00
sre-ci-robot
1e274384cd
[automated] Update Knowhere Commit (#38458) 
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-12-14 00:54:43 +08:00
Xianhui Lin
b416ff8c6c
enhance: altercollectionfield support mmap_enable convert to mmap.enable (#38448) 
altercollectionfield support mmap_enable convert to mmap.enable 
issue: https://github.com/milvus-io/milvus/issues/37436

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2024-12-13 20:02:43 +08:00
Chun Han
c1f9158996
fix: search-group-by failed to get data from multi-chunked-segment(##… (#38383) 
related: #38343

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-12-13 16:54:43 +08:00
Ted Xu
3038383e36
fix: UT compile broken under osx (#38432) 
See: #38434

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-12-13 16:24:43 +08:00
zhagnlu
efbfa1cc3e
fix:fix ut failed for debug (#38384) 
#38382

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-12-13 14:38:43 +08:00
Zhen Ye
833c74aa66
enhance: add detail, replica count for resource group (#38314) 
issue: #30647

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-12-13 14:14:50 +08:00
congqixia
10460ed3f8
enhance: Simplify querynode tsafe & reduce goroutine number (#38416) 
Related to #37630

TSafe manager is too complex for current implementation and each
delegator need one goroutine waiting for tsafe update event.

Tsafe updating could be executed in pipeline. This PR remove tsafe
manager and simplify the entire logic of tsafe updating.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-12-13 10:56:43 +08:00
cai.zhang
6ffc57c8dc
fix: Fix sorting buffer in clustering compaction (#38417) 
issue: #28410

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-12-13 10:12:49 +08:00
sre-ci-robot
e404123e3e
[automated] Update Knowhere Commit (#38422) 
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-12-13 02:40:44 +08:00
Xianhui Lin
f2d79b3d3a
enhance: altercollection validation logic (#38419) 
altercollection validation logic
issue: https://github.com/milvus-io/milvus/issues/37436

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2024-12-12 21:42:42 +08:00
Xianhui Lin
d0a8110a7a
enhance: alterdatabase support delete property (#38380) 
alterdatabase support delete property
issue: https://github.com/milvus-io/milvus/issues/38379

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2024-12-12 17:04:43 +08:00
Buqian Zheng
75e64b993f
enhance: add metrics for counting number of nun-zeros/tokens of sparse/FTS search (#38329) 
sparse vectors may have arbitrary number of non zeros and it is hard to
optimize without knowing the actual distribution of nnz. this PR adds a
metric for analyzing that.

issue: https://github.com/milvus-io/milvus/issues/35853

comparing with https://github.com/milvus-io/milvus/pull/38328, this
includes also metric for FTS in query node delegator

also fixed a bug of sparse when searching by pk

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-12-12 16:22:43 +08:00
cqy123456
b14a0c4bf5
fix:GrowingDataGetter get the wrong string data (#38015) 
issue: https://github.com/milvus-io/milvus/issues/37994
2.4 pr: https://github.com/milvus-io/milvus/pull/37995

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-12-12 14:50:42 +08:00
yihao.dai
a514f839b2
enhance: Speed up meta recovery (#38285) 
Increase the batchSize in WalkWithPrefix operations to 10000.

issue: https://github.com/milvus-io/milvus/issues/37630

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-12-12 10:58:43 +08:00
congqixia
304cdc7783
fix: Check target partition load state only when dropping partition (#38375) 
Related to #38372

This PR make drop partition only check target partition load states only
in case of concurrent releasing other partition in same collection.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-12-11 21:06:42 +08:00
Gao
8977454311
enhance: support recall estimation (#38017) 
issue: #37899 
Only `search` api will be supported

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-12-11 20:40:48 +08:00
Ted Xu
dc85d8e968
enhance: improve mix compaction performance by removing max segment limitations (#38344) 
See #37234

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-12-11 20:38:42 +08:00
wei liu
e279ccf109
enhance: Enable score based balance channel policy (#38143) 
issue: #38142
current balance channel policy only consider current collection's
distribution, so if all collections has 1 channel, and all channels has
been loaded on same querynode, after querynode num increase, balance
channel won't be triggered.

This PR enable score based balance channel policy, to achieve:
1. distribute all channels evenly across multiple querynodes
2. distribute each collection's channel evenly across multiple
querynodes.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-12-11 17:20:43 +08:00
cai.zhang
0d7a89a4f8
fix: Use the correct RootPath when decompressing binlog in stats task (#38341) 
issue: #38336

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-12-11 16:16:42 +08:00
Zhen Ye
d3ae8e9232
fix: delay the wait other coord logic in query coord after query coord change into standby state (#38259) 
issue: https://github.com/milvus-io/milvus/issues/37764

- After removing rpc layer from mixcoord, the querycoord at standby mode
will be blocked forever of deployment rolling

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-12-11 15:48:42 +08:00
yihao.dai
43e0e2b7ed
fix: Fix empty import task result (#38316) 
Ensure the idempotency of import tasks to prevent duplicate tasks in
DataNode.

issue: https://github.com/milvus-io/milvus/issues/38313

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-12-11 15:42:49 +08:00