admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-02-02 01:06:41 +08:00
Code Issues Packages Projects Releases Wiki Activity
20,514 Commits 146 Branches 196 Tags
Commit Graph

9269 Commits

Author SHA1 Message Date
yihao.dai
d6adc62765
enhance: Support db for bulkinsert (#37012) (#37017) 
issue: https://github.com/milvus-io/milvus/issues/31273

pr: https://github.com/milvus-io/milvus/pull/37012

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-23 16:31:29 +08:00
congqixia
b958c56f48
fix: [2.4] Rectify delete buffer row count quota value (#37060) (#37068) 
Cherry pick from master
pr: #37060

Related to #37057

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-23 14:13:28 +08:00
congqixia
7eba3aa67e
fix: [2.4] Pass full field list when partial load enabled (#37053) (#37063) 
Cherry-pick from master
pr: #37053

Related to #37038

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-23 11:03:28 +08:00
Alexander Guzhva
415f2f47c0
enhance: [2.4][bitset] multiple 'and' and 'or' in a single op (#36973) 
issue: #34117
pr: #33345 

Refactoring:
* Added a capability to perform multiple bitwise and and or operations
in a single op
* AVX2, AVX512, ARM NEON, ARM SVE backed bitwise and, op, xor and sub
ops
* more unit tests for bitset
* fixed a bug in or_with_count for certain bitset sizes
* fixed a bug for certain offset values for inplace operations that take
two bitsets

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
 2024-10-22 16:25:33 +08:00
smellthemoon
d2a1db922f
enhance: refactor createCollection in RESTful API(#36790) (#36885) 
pr: #36790 
and pr:#36930

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-10-22 14:21:27 +08:00
congqixia
7acf1d53c1
enhance: [2.4] Preallocate delete data slice to avoid growslice (#37044) 
Rewritten based on master pr
pr: #37043

Related to #36887

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-22 14:15:28 +08:00
SimFG
55caf41169
enhance: [2.4] set the rpc error code to avoid the invalid retry (#37025) 
- pr: #37024

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-10-22 11:17:28 +08:00
yihao.dai
539f56220f
enhance: Remove bf from datanode (#36367) (#37027) 
Remove bf from datanode:
1. When watching vchannels, skip loading **flushed** segments's bf. For
generating merged bf, we need to keep loading **growing** segments's bf.
2. Bypass bloom filter checks for delete messages, directly writing to
L0 segments.
3. In version 2.4, when dropping a partition, marking segments as
dropped depends on having the full segment list in the DataNode. So, we
need to keep syncing the segments every 10 minutes.

issue: https://github.com/milvus-io/milvus/issues/34585

pr: https://github.com/milvus-io/milvus/pull/35902,
https://github.com/milvus-io/milvus/pull/36367,
https://github.com/milvus-io/milvus/pull/36592

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-22 11:15:28 +08:00
Chun Han
a2d4520dc7
enhance: avoid limiting ddl operations repeatedly(#37006) (#37011) 
related: #37006

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-10-22 10:41:28 +08:00
XuanYang-cn
6b68ec55e1
enhance: [24]Track complex delete rates (#36958) 
See also: #36953
pr: #36957

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-22 10:07:27 +08:00
wei liu
1dcc393e54
fix: Query node panic during sending rpc to worker (#36975) (#36988) 
issue: #36976
pr: #36975

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-10-21 10:15:26 +08:00
yihao.dai
4e0f5845a1
enhance: Limit import job number (#36891) (#36892) 
issue: https://github.com/milvus-io/milvus/issues/36890

pr: https://github.com/milvus-io/milvus/pull/36891

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-18 18:13:25 +08:00
XuanYang-cn
7847b662d4
enhance: [cp24]Log when limit writing by deletion (#36955) 
See also: #36953
pr: #36954

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-18 17:47:30 +08:00
yihao.dai
8923936c9a
enhance: Support memory mode chunk cache (#35347) (#35836) 
Chunk cache supports loading raw vectors into memory.

issue: https://github.com/milvus-io/milvus/issues/35273

pr: https://github.com/milvus-io/milvus/pull/35347

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-18 17:03:25 +08:00
Ted Xu
2742524508
enhance: enable parallel execution of L0 compactions (#36816) (#36985) 
pr: #36816

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-10-18 16:17:25 +08:00
wei liu
b88d610e42
fix: datacoord stuck at stopping progress (#36852) (#36961) 
issue: #36868
pr: #36852
if datacoord is syncing segments to datanode, and stop datacoord
happens, datacoord's stop progress will stuck until syncing segment
finished.

This PR add ctx to syncing segment, which will failed if stopping
datacoord happens.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-10-18 14:43:24 +08:00
Ted Xu
22838a8413
enhance: Datacoord to support prioritization of compaction tasks (#36979) 
See #36550

pr: #36547 
pr: #36956

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-10-18 14:15:25 +08:00
cai.zhang
2bfd22f217
fix: [cherry-pick] Fix clustering compaction task leak (#36803) 
issue: #36686
master pr: #36800 

bug reason:
- The clustering compaction tasks on the datanode were never cleaned up.
- The clustering compaction task contains a mapping from clustering key
to buffer, this caused a large memory leak.

fix:
- clean the tasks on datanode by datacoord when clustering compaction
finished.
- reset the mapping that from clustering key to buffer on datanode when
clustering finished.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-10-17 20:43:30 +08:00
cqy123456
6934e8da3a
enhance: [2.4]use growingMmapEnabled to control the behavior of interim index, not vectorField (#36391) 
issue: https://github.com/milvus-io/milvus/issues/36392
related pr: https://github.com/milvus-io/milvus/pull/36500

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-10-17 20:23:25 +08:00
cqy123456
ff662308ed
fix:[2.4]Chunk Id out of range in vector BF search, after the growing index removes the vec chunks. (#36938) 
issue: https://github.com/milvus-io/milvus/issues/36871
related pr: https://github.com/milvus-io/milvus/pull/36939

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-10-17 16:51:24 +08:00
congqixia
3252d7a64c
fix: [2.4] Load original key if ts is MaxTimestamp (#36934) (#36950) 
Cherry-pick from master
pr: #36934 

Related to #36933

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-17 16:05:29 +08:00
congqixia
dfe27ebf35
fix: [2.4] Direct forward delta exclude l0 segments (#36899) (#36914) 
Cherry-pick from master
pr: #36899

Related to #36887

Forward delete to L0 segment will return error and mark l0 segment
offline causing delegator unserviceable

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-17 14:11:29 +08:00
congqixia
877e9ad450
enhance: [2.4] Fill start pos & level for growing segment (#36888) (#36911) 
Cherry-pick from master
pr: #36888

Start position & level info is missing for growing segment loaded in
watch dml channel operation.

Level is important for metrics and start position is crucial for growing
exclude logic.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-17 14:09:32 +08:00
wei liu
7c35c51e15
fix: Rootcoord stuck at graceful stop progress (#36881) 
issue: #34553
pr: #36880
when rootcoord trigger graceful stop progress, it will block until all
rpc finished. for create collection request, rootcoord need to block
until datacoord finish to watch all channels, but datacoord need to call
`rootcoord.Alloc` during watch channel, and rootcoord doesn't respond to
new request anymore. which cause create collection stucks, and graceful
stop progress stucks.

This PR remove the func call `rootcoord.Alloc` to solve the logic dead
lock during graceful stop progress.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-10-17 12:13:37 +08:00
yihao.dai
604e346585
enhance: Enhance segment log (#36848) (#36849) 
/kind improvement

pr: https://github.com/milvus-io/milvus/pull/36848

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-15 20:43:30 +08:00
yihao.dai
f8d9c24256
enhance: Ignore db not found error in quota center (#36821) (#36850) 
In quota center, ignore the "DB not found error" to prevent it from
affecting the rate limiting of other databases.

/kind improvement

pr: https://github.com/milvus-io/milvus/pull/36821

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-15 20:41:24 +08:00
SimFG
6b9e28bc8f
enhance: [2.4] update the expr version to support automatic conversion of variable types (#36847) 
/kind improvement
- pr: #36832

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-10-15 10:55:23 +08:00
Xiaofan
44564f0403
fix: make sure alias is cached (#36808) 
see also #36806

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
 2024-10-13 17:39:22 +08:00
SimFG
0fe549adf1
enhance: [2.4] gc in the snapshot kv (#36793) 
- issue: #36770
- pr: #36792

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-10-11 22:13:21 +08:00
jaime
43b48a0f9b
enhance: skip alter operation when no change are detected (#36786) 
issue: #36784 
pr: #36785

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-10-11 21:43:21 +08:00
jaime
4110f35f1a
fix: may exceed max tnx in etcd operations (#36773) 
issue: #36772
pr: #36775

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-10-11 19:51:22 +08:00
congqixia
a57626afa0
fix: [2.4] Use metrics values instead of hardcode mock values (#36783) 
Cherry-pick from master
pr: #36782
Resolves #36781

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-11 19:49:21 +08:00
cqy123456
58b198c785
enhance: [2.4]support sparse vector mmap in growing segment type (#36565) 
issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/36566

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-10-11 17:23:21 +08:00
XuanYang-cn
e976b41f97
fix: Remove enableLevelZeroSegment config (#36507) 
See also: #36504
pr: #36535

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-11 16:41:21 +08:00
XuanYang-cn
28de6b86ba
fix: [24]fail to init fg clears flushTs so that slows flush (#36741) 
See also: #36709
pr: #36740

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-11 16:39:21 +08:00
yihao.dai
7ee46c582e
enhance: Support new RESTful URL to describe import job (#36753) (#36754) 
Support the new RESTful URL for retrieving/describing import progress:
`/v2/vectordb/jobs/import/describe`.
Deprecate the old URL: `/v2/vectordb/jobs/import/get_progress`.

issue: https://github.com/milvus-io/milvus/issues/36752

pr: https://github.com/milvus-io/milvus/pull/36753

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-11 16:03:22 +08:00
yihao.dai
a4ef93457d
enhance: Optimize import scheduling and add time cost metric (#36601) (#36684) 
1. Optimize import scheduling strategic:
a. Revise slot weights, calculating them based on the number of files
and segments for both import and pre-import tasks.
b. Ensure that the DN executes tasks in ascending order of task ID.
2. Add time cost metric and log.

issue: https://github.com/milvus-io/milvus/issues/36600,
https://github.com/milvus-io/milvus/issues/36518

pr: https://github.com/milvus-io/milvus/pull/36601

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-11 10:27:22 +08:00
congqixia
b24788b2c7
enhance: [2.4] Add balance report log for qc balancer (#36749) 
Cherry pick from master
pr: #36747 
Related to #36746

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-11 10:25:24 +08:00
SimFG
548f8e80c3
enhance: [2.4] the estimate method when loading the collection (#36728) 
- pr: #36307
- issue: #36530

Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-10-11 10:20:45 +08:00
XuanYang-cn
c69af44472
enhance: [24]Add more info in logs (#36732) 
pr: #36535

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-10 17:51:25 +08:00
aoiasd
eaa948752b
enhance: [Cherry-Pick] UpdateSegmentsInfo should update remaining segment info even if some one not exist (#36729) 
pr: https://github.com/milvus-io/milvus/pull/36726

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-10-10 15:11:20 +08:00
zhagnlu
02bd916fa4
fix: fix: fix json in [] expr bug (#36722) 
pr: #36721

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-10-09 21:53:20 +08:00
yihao.dai
9cb5396cf6
enhance: Use common gc config (#36668) (#36670) 
Use the GC config from `common` and remove the GC config from
`queryNode`.

issue: https://github.com/milvus-io/milvus/issues/36667

pr: https://github.com/milvus-io/milvus/pull/36668

related pr: https://github.com/milvus-io/milvus/pull/34949

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-09 19:49:20 +08:00
congqixia
3a80d1f602
enhance: [2.4] Add streaming forward policy switch for delegator (#36330) (#36712) 
Cherry pick from master
pr: #36330
Related to #35303

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-09 17:41:20 +08:00
yihao.dai
93a4574a35
Add buildIndex state for import job (#36705) 
issue: https://github.com/milvus-io/milvus/issues/36698

pr: https://github.com/milvus-io/milvus/pull/35868,
https://github.com/milvus-io/milvus/pull/36699

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-09 16:57:20 +08:00
XuanYang-cn
0ff8e13232
fix: [24]Remove neighbors if compactTo is unindexed (#36503) (#36694) 
See also: #36360
pr: #36503

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-09 15:43:20 +08:00
Patrick Weizhi Xu
6e93601b2c
enhance: [2.4] remove warnings when there is no partition key set (#36679) 
issue: #36673

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit db60f7cf2a56656b101067a2dbebd4dd32ed6ed2)
 2024-10-09 09:35:20 +08:00
cai.zhang
f7302a78fb
enhance: [cherry-pick]Convert unincode to ascii to improving expression parsing efficiency (#36676) 
issue: #36672 

master pr: #36675

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-10-09 09:23:24 +08:00
congqixia
9073f6281e
fix: [2.4] Add defer Unpin when error happens (#36620) (#36665) 
Cherry-pick from master
pr: #36620
Resolves: #36619

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-08 18:29:20 +08:00
Buqian Zheng
03fba95284
fix: [2.4] improve mmap related code in Column.h (#36521) 
pr: #36183
issue: #36182

---------

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-10-08 18:27:19 +08:00