Related to #37630
Previously the loaded collection metrics was calculated via scanning all
loaded segment in segment manager, which is slow and buggy
implementation.
This PR:
- Move collection num metrics to collection manager
- Remove deprecated loaded partition metrics update logic
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #38325
the old impl only to check grant in default db before drop role, which
may cause role be dropped when grant still exist.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
enhance :
1. alterindex delete properties
We have introduced a new parameter deleteKeys to the alterindex
functionality, which allows for the deletion of properties within an
index. This enhancement provides users with the flexibility to manage
index properties more effectively by removing specific keys as needed.
2. altercollection delete properties
We have introduced a new parameter deleteKeys to the altercollection
functionality, which allows for the deletion of properties within an
collection. This enhancement provides users with the flexibility to
manage collection properties more effectively by removing specific keys
as needed.
3.support altercollectionfield
We currently support modifying the fieldparams of a field in a
collection using altercollectionfield, which only allows changes to the
max-length attribute.
Key Points:
- New Parameter - deleteKeys: This new parameter enables the deletion of
specified properties from an index. By passing a list of keys to
deleteKeys, users can remove the corresponding properties from the
index.
- Mutual Exclusivity: The deleteKeys parameter cannot be used in
conjunction with the extraParams parameter. Users must choose one
parameter to pass based on their requirement. If deleteKeys is provided,
it indicates an intent to delete properties; if extraParams is provided,
it signifies the addition or update of properties.
issue: https://github.com/milvus-io/milvus/issues/37436
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
issue: #38237
this PR only use better compression level for proto msg which is larger
than 1MB, and use a lighter compression level for smaller proto msg,
which could get a better latency in most case.
this PR could reduce the latency from 22.7s to 4.7s with 10000
collctions and each collections has 1000 segments.
before this PR:
BenchmarkTargetManager-8 1 22781536357 ns/op 566407275088 B/op 11188282
allocs/op
after this PR:
BenchmarkTargetManager-8 1 4729566944 ns/op 36713248864 B/op 10963615
allocs/op
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #38201
leader task require to update delegator's distribution, and only success
after the distribution change has been applyed to delegator. but the
delegator will reject the distribution change if it's version is older
than current version in delegator. which cause the leader task stuck and
retry forever.
this PR remove the leader task finish check.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #37630
Add secondary index with vchannel to reduce `GetBy` rlock holding time
when segment number is large.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #38275
This PR move sync created partition step to proxy to avoid potential
logic deadlock when create partition happens with target segment change.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #38305
after we disable balance segment and balance channel happens at same
time, the constriant which require release segment must happens on
serviceable shard leader is unnessary.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #36456
Unify collection/partition number metrics to collection manager in case
of unwant missing modification
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #36621
- add filters to most of lists
- add more options to the existing list filter
- fix the error list column in channels and resource group list
Signed-off-by: mimo.oyn <ning.ouyang@zilliz.com>
The partition number has already been incremented in
`ChangePartitionState`, so there is no need to increment it again in
`AddPartition`.
issue: https://github.com/milvus-io/milvus/issues/37630
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
When there are too many key-value pairs, the etcd list operation may
times out. This PR replaces `LoadWithPrefix` in list operations, which
could involve many keys, with `WalkWithPrefix`.
issue: https://github.com/milvus-io/milvus/issues/37917
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
`File.Write` and `File.WriteInt` use `write`, which may be just direct
syscall in some systems. When mappding field data and write line by
line, this could cost lost of CPU time when the row number is large.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
