feat: Ignore reporting index metrics for non-existent indexes
Remove the reporting of index metrics for non-existent indexes in the
`getCollectionMetrics` function. This change improves the code by
skipping unnecessary operations and reduces log noise.
issue: https://github.com/milvus-io/milvus/issues/41280
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
enhance: update MixCoord registration in MilvusRoles
The `runMixCoord` function in `MilvusRoles` was updated to use the
`RegisterMixCoord` function from the `rootcoord_metrics` package instead
of `RegisterRootCoord`. This change aligns with the recent modifications
made to the `rootcoord_metrics` package.
issue:https://github.com/milvus-io/milvus/issues/41338
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
## What this PR does
This PR fixes an issue where the `PreCheck` function in DataCoord logs
unnecessary warnings
when attempting to retrieve 'dim' from non-vector fields.
The change adds a check to only call `GetDimFromParams` when the field
type is a vector type.
## Related issue
Fixes#41287
---------
Signed-off-by: 박상호 <sangho@rapportlabs.kr>
Signed-off-by: Sangho Park <hoyaspark@gmail.com>
support parallel loading sealed and growing segments with storage v2
format by async reading row groups.
related: #39173
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
Issue: #41210
After https://github.com/zilliztech/tantivy/pull/5, we can provide
milvus row id directly to tantivy rather than record it in the fast
field "doc_id".
So rather than search tantivy doc id and then get milvus row id from
"doc_id", now, the searched tantivy doc id is the milvus row id,
eliminating the expensive acquiring row id phase.
The following shows a simple benchmark where insert **1M** docs where
all rows are "hello", the latency is **segcore** level, CPU is 9900K:
**The latency is 2.02 and 2.1 times respectively.**
bench mark code:
```
TEST(TextMatch, TestPerf) {
auto schema = GenTestSchema({}, true);
auto seg = CreateSealedSegment(schema, empty_index_meta);
int64_t N = 1000000;
uint64_t seed = 19190504;
auto raw_data = DataGen(schema, N, seed);
auto str_col = raw_data.raw_->mutable_fields_data()
->at(1)
.mutable_scalars()
->mutable_string_data()
->mutable_data();
for (int64_t i = 0; i < N - 1; i++) {
str_col->at(i) = "hello";
}
SealedLoadFieldData(raw_data, *seg);
seg->CreateTextIndex(FieldId(101));
auto now = std::chrono::high_resolution_clock::now();
auto expr = GetMatchExpr(schema, "hello", OpType::TextMatch);
auto final = ExecuteQueryExpr(expr, seg.get(), N, MAX_TIMESTAMP);
auto end = std::chrono::high_resolution_clock::now();
auto duration =
std::chrono::duration_cast<std::chrono::microseconds>(end - now);
std::cout << "TextMatch query time: " << duration.count() << "ms"
<< std::endl;
}
```
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
Related to #41302
Previously wait for 200 milliseconds could cause unsable behavior of
this unittest. This PR make unittest wait for certain function call
instead of wait for some time.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Make DataNode use compaction parameters from request instead of
configuration.
issue: https://github.com/milvus-io/milvus/issues/41123
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Merge RootCoord, DataCoord And QueryCoord into MixCoord
Make Session into one
issue : https://github.com/milvus-io/milvus/issues/37764
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/40897
After this, the document add operations scheduling duration is decreased
roughly from 6s to 0.9s for the case in the issue.
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
issue: #41172
Elements with type int8 or int16 in Array is encoded using int32, so we
should parse it as int32 when creating index.
Signed-off-by: sunby <sunbingyi1992@gmail.com>
The restful API default timeout was hard-coded. This PR make this
timeout value configurable via paramtable.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #41194
- Refactor hasUnbalancedCollection flag handling to function scope
- Ensure tracking sets clearance when no balance needed
- Add deferred cleanup for both normal/stopping balance paths
- Add unit tests for collection tracking scenarios
The changes ensure tracking sets (normalBalanceCollectionsCurrentRound
and stoppingBalanceCollectionsCurrentRound) are properly cleared when:
- All collections in current round are balanced
- Balance checks return early due to unready targets
- Balance feature flags are disabled
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Ref: https://github.com/milvus-io/milvus/issues/40823
It does not make any sense to create single segment tantivy index for
old version such as 2.4 by using tantivy V7.
So, clean the relevant code.
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/40006
This PR make tantivy document add by batch. Add document by batch can
greately reduce the latency of scheduling the document add operation
(call tantivy `add_document` only schdules the add operation and it
returns immediately after scheduled) , because each call involes a tokio
block_on which is relatively heavy.
Reduce scheduling part not necessarily reduces the overall latency if
the index writer threads does not process indexing quickly enough.
But if scheduling itself is pretty slow, even the index writer threads
process indexing very fast (by increasing thread number), the overall
performance can still be limited.
The following codes bench the PR (Note, the duration only counts for
scheduling without commit)
```
fn test_performance() {
let field_name = "text";
let dir = TempDir::new().unwrap();
let mut index_wrapper = IndexWriterWrapper::create_text_writer(
field_name,
dir.path().to_str().unwrap(),
"default",
"",
1,
50_000_000,
false,
TantivyIndexVersion::V7,
)
.unwrap();
let mut batch = vec![];
for i in 0..1_000_000 {
batch.push(format!("hello{:04}", i));
}
let batch_ref = batch.iter().map(|s| s.as_str()).collect::<Vec<_>>();
let now = std::time::Instant::now();
index_wrapper
.add_data_by_batch(&batch_ref, Some(0))
.unwrap();
let elapsed = now.elapsed();
println!("add_data_by_batch elapsed: {:?}", elapsed);
}
```
Latency roughly reduces from 1.4s to 558ms.
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
