https://github.com/milvus-io/milvus/issues/32321
PR merged to master -
[#32264](https://github.com/milvus-io/milvus/pull/32264)
Issue Description:
Tracing is an important means of identifying bottleneck points in a
system and is crucial for debugging production issues. Milvus(or any DB)
is generally the most downstream system for an user call -- a user call
can originate from UI and pass through multiple components, in
micro-services architecture, before reaching Milvus. So, when an user
experiences a glitch, one would debug the call trace via logs using a
common trace id. As of now, Milvus generates a new trace id for every
call and this request is to make sure client can pass the trace id which
will be used for all the logs across the Milvus sub-components so that
one can fetch logs for a user call across the components -- including
Milvus.
Signed-off-by: Shreesha Srinath Madogaran <smadogaran@salesforce.com>
Signed-off-by: Swechchha Sinha <swechchha.sinha@salesforce.com>
Co-authored-by: madogar <36537062+madogar@users.noreply.github.com>
Co-authored-by: Shreesha Srinath Madogaran <smadogaran@salesforce.com>
See also #29625
pr: #29626
This PR:
- Add a new implemention of `DeleteBuffer`: listDeleteBuffer
- holds cacheBlock slice
- `Put` method append new delete data into last block
- when a block is full, append a new block into the list
- Add `TryDiscard` method for `DeleteBuffer` interface
- For doubleCacheBuffer, do nothing
- For listDeleteBuffer, try to evict "old" blocks, which are blocks
before the first block whose start ts is behind provided ts
- Add checkpoint field for `UpdateVersion` sync action, which shall be
used to discard old cache delete block
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: congqixia <congqi.xia@zilliz.com>
issue: #34595
pr: #34596
When consuming insert data on the delegator node, QueryCoord will move
out some sealed segments to manage its memory usage. After the growing
segment gets flushed, some sealed segments from other workers will be
moved back to the delegator node. To avoid the frequent movement of
segments, we estimate the maximum growing row count and preserve a
fixed-size memory in the delegator node.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Cherry-pick from master
pr: #33123
See also #33122
This pr add param item `mq.ignoreBadPosition` to control behavior when
mq failed to parse message id from checkpoint
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Support get sdk type by user agent when we can't get sdk version by
connection in access log.
---------
pr: https://github.com/milvus-io/milvus/pull/30760
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Cherry-pick from master
pr: #31363
See also #31362
This PR make datacoord garbage collection scan operation using differet
interval than other opeartion.
This interval is a newly added param item, which default value is 7*24
hours.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
fix#32059
pr: #32089
this pr fix two issues:
1. offset is not handled correctly without specify a limit
2. reduceStopForBest doesn't guarantee to return limit result even if
there are more result when there is small segment
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
issue: #29261
pr: #29998
This PR Add restful api for devops to execute rolling upgrade, including
suspend/resume balance and manual transfer segments/channels.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
/kind improvement
pr: #31271
fix: https://github.com/milvus-io/milvus/issues/31272
This pr add more metrics, which are:
Slow query count, which the duration considered as slow can be
configurable;
Number of deleted entities;
Number of entities per collection;
Number of loaded entities per collection;
Number of indexed entities;
Number of indexed entities, per collection, per index and whether it's a
vetor index;
Quota states (LongTimeTickDelay, MemoryExhuasted, DiskQuotaExhuasted)
per database;
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
issue: #31222
pr: #31256
grpcclient's `call` func return a unrecoverable error, then the caller's
retry policy also breaks due to this unrecoverable error.
This PR introduce `retry.Handle`, the new func use `func() (bool,
error)` as input parameters, which return `shouldRetry` directly, to
avoid grpcclient return a unrecoverable error
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Cherry-pick from master
pr: #31388
See also #30806
`formatKey` may cost lots of CPU on string processing under high QPS
scenario, this PR adds a formattedKeys cache preventing string operation
in each param get value.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
