Cherry pick from master
pr: #29328
See also #29327
Change channel checkpoint metrics to unix seconds instead of checkpoint
timestamp lag value
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
related to #28549
pr: #28626
1. avoid duplicated sync segments under syncing states
2. add jitter to avoid sync segments at the same time
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
many growing segments may be created in a short time and there is no
restriction to the process, the CGO call will leave many threads
related: https://github.com/milvus-io/milvus/issues/29282
pr: #29306
Signed-off-by: yah01 <yah2er0ne@outlook.com>
issue: #23726
pr: #29231
This PR add control config to querycoord's background auto balance
channel operation
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #29243
only rootCoord read the configuration item `builtinRoles`, so proxy
never know whether the role to be deleted is builtin.
Signed-off-by: PowderLi <min.li@zilliz.com>
issue: #29004
master pr: #29055
add a new parameter: `params`, which is a map[string]float64;
but now only 2 valid item: radius + range_filter;
Signed-off-by: PowderLi <min.li@zilliz.com>
See also #29156
FlushTs need to to be reset to MaxUint64 after channel checkpoint is
after this timestamp. Otherwise, the segment will be shattered and flush
queue will be filled with tasks
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #28622
pr: #29216
After we support balance segment with growing segment count #28623, if
we balance segment and channel at same time, some segments need to be
rebalanced after balance channel finish.
This PR skip balance segment when channel need be balanced.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #23726
pr: #28469
1. enable auto balance channel between nodes in querycoord
2. make `genSegmentPlan` reuse the `AssignSegment` logic
3. make `genChannelPlan` reuse the `AssignChannel` logic
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Cherry-pick from master
pr: #29195
See also #29113
rand.Seed is deprecated and cost noticable CPU time during heavy payload
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
we found the load got stuck probably, and reviewed the logs.
the target observer seems not working, the reason is the taskDispatcher
removes the task in a goroutine, and modifies the task status after
committing the task into the goroutine pool, but this may happen after
the task removed, which leads to the task will never be removed
related #29086
pr: #29191
Signed-off-by: yah01 <yang.cen@zilliz.com>
issue: #28622
pr: #28623
query node with delegator will has more rows than other query node due
to delgator loads all growing rows.
This PR enable the balance segment which based on the num of growing
rows in leader view.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Cherry pick from master
pr: #29154
See also #29177
Add a config item for partition name as regexp feature and disable it by
default
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry pick from master
pr: #29143
See also #29113
Using zap.Stringer log field will evaluate log field value only when log
level meets the configuration, which could save some CPU time in search
route
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
This pull request enhances the logging functionality in the code related
to target updating. It adds more logs about the condition satisfying
when updating the target. The logs provide additional information about
the collection ID, replica number, channel readiness, segment readiness,
and leader view readiness. These logs will help in troubleshooting and
monitoring the target updating process.
pr: #29090
Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
issue: #29068
master pr: #29075
wait server to listen the http port
then check whether various urls can be accessed normally
Signed-off-by: PowderLi <min.li@zilliz.com>
issue: [milvus-proto
#212](https://github.com/milvus-io/milvus-proto/issues/212)
master pr: #28961
milvus can't use partition related privileges until upgrade
milvus-proto, even if them were added to milvus-proto
Signed-off-by: PowderLi <min.li@zilliz.com>
Cherry-pick from master
pr: #29117
See also #29113
This patch:
- Replace plain Enforcer with `casbin.SyncedEnforcer`
- Add implementation of persist.Adapter with `MetaCacheCasbinAdapter`
- Invoke enforcer.LoadPolicy when policy updated
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry-pick from master
pr: #29058
See also #29057
Add wrapper to maintain client&connection
When reset operation is needed, close method shall wait until all
on-going request return
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
In order to minimize the CPU usage of the coroutine and avoid frequent
execution of time-consuming operations in the flowgraph when the message
stream consists solely of "ttMsg," it is recommended to implement a
mechanism for quickly bypassing the subsequent flowgraph node processing
logic.
If "ttMsg" is continuously received for a certain period of time
(coldTime), the flowgraph enters skipMode. Once in skipMode, every
skipNum "ttMsg" messages are merged into one for processing. If a
non-"ttMsg" message is received while in skipMode, the flowgraph exits
skipMode.
pr: #28756
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
issue: #28960
master pr: #28961
add new configuration: builtinRoles
user can define roles in config file: milvus.yaml
there is an example:
db_ro, only have read privileges, include load
db_rw, read and write privileges, include create/drop/rename collection
db_admin, not only read and write privileges, but also user
administration
Signed-off-by: PowderLi <min.li@zilliz.com>
