20695 Commits

Author SHA1 Message Date
congqixia
c0c12c6c5b
fix: Use SliceSetEqual to compare load field list (#36051)
Related to #36037

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-10 10:13:14 +08:00
wt
c916407f37
test: add query expr test cases (#36073)
1. query with expr under different scalar index types
2. test framework supports preparing one piece of data and multiple
parameter queries

Signed-off-by: wangting0128 <ting.wang@zilliz.com>
2024-09-09 19:55:06 +08:00
OxalisCu
3a381bc247
enhance: Bulkinsert supports null in csv formats (#35912)
see details in this issue
https://github.com/milvus-io/milvus/issues/35911

---------

Signed-off-by: OxalisCu <2127298698@qq.com>
2024-09-09 19:17:07 +08:00
congqixia
aaa8487590
enhance: Remove duplicated log in proxy service (#36110)
Remove duplicate identical log in proxy/service.go

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-09 17:19:05 +08:00
congqixia
51870aef08
enhance: Use stats Handler to record request/response size metrics (#36107)
Related to #36102

This PR use newly added `grpcSizeStatsHandler` to reduce calling
`proto.Size` since the request & response size info is recorded by grpc
framework.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-09 17:13:05 +08:00
yellow-shine
73a34a5a71
enhance: [skip e2e] add a rule to label ci-pass when go-sdk check status (#36105)
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-09-09 16:17:06 +08:00
congqixia
3123093dd7
enhance: Use MARISA_LABEL_ORDER when building trie index (#36034)
Related to #35941
Previous PR: #35943

This PR make `Trie` index using `MARISA_LABEL_ORDER`, which make
predictive search iterating in lexicographic order.

When trie index is build in label order, lexicographc could be utilized
accelerating `Range` operations.

However according to the official document, using `MARISA_LABEL_ORDER`
will make "exact match lookup, common prefix search, and predictive
search" slower.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-09 14:29:05 +08:00
XuanYang-cn
2687747278
fix: Set an empty segment if compaction deleted all inserts (#36044)
See also: #36038

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-09-09 14:23:05 +08:00
SimFG
99817953eb
fix: delay to start the metric server port (#36080)
- issue: #36083
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-09 14:01:05 +08:00
CharlesFeng
8787e65b1f
fix: lifeTime not released in time (#36093)
https://github.com/milvus-io/milvus/issues/36092

Signed-off-by: fengjun2016 <jornfeng@gmail.com>
2024-09-09 11:25:05 +08:00
wei liu
30a99b66c1
fix: Fix logic dead lock when delegator has high memory usage (#36065)
issue: #36064
when delegator has high memory usage, load l0 segment will failed. and
balance segment task will blocked by load segment task, then delegator
cann't free memory by moving out some segment, causes a logic dead lock.

this PR remove the limit for balance, we permit segment and balance
execute in parallel. which won't cause side effect due to:
1. one segment can only has one task in qc's scheduler, and load/release
task will replace balance task if necessary
2. balance speed has been limited, and it won't block load segment task.

3. if collection has load task and balance task at same time, load task
will be scheduled first due to high proirity.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-09 10:21:06 +08:00
zhagnlu
208c8a2328
fix:support config index offsetcache and fix create same index again (#35985)
#35971

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-08 18:23:05 +08:00
jaime
91d23ecbe1
fix: memory leak in proxy meta cache (#36075)
issue: #36074

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-08 17:33:05 +08:00
yihao.dai
6b4ae0c65e
enhance: Log warn on delayed compaction tasks (#36049)
/kind enhancement

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-08 17:23:05 +08:00
Chun Han
e480b103bd
feat: supporing hybrid search group_by (#35982)
related: #35096

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-08 17:09:04 +08:00
CharlesFeng
62f4a6a112
fix: binlog reader not released in time (#36078)
https://github.com/milvus-io/milvus/issues/36077

Signed-off-by: fengjun2016 <jornfeng@gmail.com>
2024-09-07 08:15:06 +08:00
sre-ci-robot
9a8ed9af5f Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-09-07 00:01:31 +00:00
Xiaofan
f4263fee55
Update README.md 2024-09-06 16:01:31 -07:00
XuanYang-cn
7859faf8ea
fix: Change deltalog memory estimation factor to one (#36033)
See also: #36031

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-09-06 18:21:05 +08:00
congqixia
a103dd5eb3
enhance: Fix SearchOnSealed clang-format lint (#36056)
Related to #36008

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-06 16:47:04 +08:00
smellthemoon
21b135c7c2
fix: not append valid data when transfer to insert record (#36027)
fix not append valid data when transfer to insert record and add a tiny
check when in groupBy field.
#35924

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-06 14:53:04 +08:00
SimFG
5247631289
fix: fill the metric type field in the LoadMetaInfo object (#35962)
- issue: #35960

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-05 20:50:23 -07:00
yellow-shine
a802aea13c
enhance: bump etcd version (#35978)
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-09-06 10:23:05 +08:00
sre-ci-robot
43a5523a8b
[automated] Update Pytest image changes (#35951)
Update Pytest image changes
See changes:
40d34f711e
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-09-06 09:41:05 +08:00
aoiasd
9871841fa0
fix: datacoord broker use querycoord broker config and get segment info timeout (#36026)
relate: https://github.com/milvus-io/milvus/issues/36025
Will case channel with large segment num watch failed.

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-06 08:15:05 +08:00
Jiquan Long
11325d9ed5
fix: binary arith expression on inverted index (#35945)
issue: https://github.com/milvus-io/milvus/issues/35946

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-09-05 20:01:05 +08:00
sre-ci-robot
8eb1daf362 Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-09-05 12:00:39 +00:00
yellow-shine
15ed76d1d6
enhance: [nightly pipeline] add more timeout for log and list all pods status (#36011)
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-09-05 19:31:47 +08:00
wei liu
75676fbd11
fix: Fix dynamic release partition may fail search/query request (#35919)
issue: #33550
cause concurrent issue may occur between remove parition in target
manager and sync segment list to delegator. when it happens, some
segment may be released in delegator, and those segment may also be
synced to delegator, which cause delegator become unserviceable due to
lack of necessary segments, then search/query fails.

this PR make sure that all write access to target_manager will be
executed in serial to avoid the concurrent issues.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-05 18:47:03 +08:00
congqixia
97ca65c732
enhance: [skip e2e] Add git safe dir for code check env (#36009)
Related to #36008

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-05 18:09:11 +08:00
cqy123456
560e8e70b0
enhance: reduce mmap_rss after chunkcache warmup (#35974)
related pr: https://github.com/milvus-io/milvus/pull/35965

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-09-05 18:07:05 +08:00
yihao.dai
8b043f58dc
fix: Fix DB limiter nodes are mistakenly cleaned up (#35991)
This issue only occurs for a short time right after a table is created.
To avoid this, we simply reduce the frequency of cleaning up invalid
limiter nodes.

issue: https://github.com/milvus-io/milvus/issues/35933

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-05 17:35:04 +08:00
congqixia
f985173da0
fix: Fill load field list from old version load info (#35993)
See also #35959

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-05 16:57:05 +08:00
congqixia
c61eea737b
enhance: Fix trace.cpp lint format issue (#36004)
Introduced by #35928

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-05 16:33:04 +08:00
Ted Xu
d9a40784a2
fix: fallback params may be overridden (#35972)
See #35756

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-09-05 16:19:04 +08:00
wei liu
32e55a02ea
fix: Fix privilege group hasn't been register for validate (#35937)
issue: #35471

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-05 15:35:04 +08:00
congqixia
7b21032d19
fix: Check all values for trie.predictive_search (#35943)
Related to #35941

For marisa trie `predictive_search` default behavior, it value iterated
is not in lexicographic order.

This PR is a brute force fix to make range operator returns correct
values.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-05 15:01:04 +08:00
congqixia
9e96ed4873
fix: Fix tracing config update logic (#35928)
Related to #35927

There are serveral issue this PR addresses:
- Use `ResetTraceConfig` method instead init one in update event handler
- Implement dynamic stats.Handler to receive tracing config update event
- Update `enable_trace` flag when `ResetTraceConfig` is invoked
- Change `enable_trace` to `std::atomic<bool>` in case of data race

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-05 14:27:04 +08:00
XuanYang-cn
5e3f700e5d
enhance: Remove too frequent logs in Delete (#35980)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-09-05 14:25:03 +08:00
congqixia
8593c4580a
enhance: Add delete buffer related quota logic (#35918)
See also #35303

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-05 11:39:03 +08:00
smellthemoon
80a9efd96d
fix: not check nullable and default value in pk field (#35987)
#35926

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-05 11:27:04 +08:00
congqixia
66ed289a85
enhance: Fix typo of clustering key not loaded msg (#35948)
Related to #35415

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-05 10:49:03 +08:00
cai.zhang
90bdb171ab
fix: Fix data race for clustering compaction writer (#35957)
issue: #35950

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-05 04:07:10 +08:00
CharlesFeng
6eb8b3f745
fix: err degenerated to a new variable (#35891)
https://github.com/milvus-io/milvus/issues/35890

Signed-off-by: fengjun2016 <jornfeng@gmail.com>
2024-09-04 14:57:04 +08:00
jaime
24fb10114b
enhance: remove cooling off in rate limiter for read requests (#35935)
issue: #35934

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-04 14:39:10 +08:00
jaime
919e96ac22
enhance: add IT for rate limit using db properties (#35930)
issue: #35929

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-04 14:37:04 +08:00
SimFG
ea36d13ab0
feat: add static view for the expr (#35887)
- issue: #35886
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-04 11:09:04 +08:00
nico
40d34f711e
test: update test cases (#35867)
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-09-04 10:37:03 +08:00
Abdullah Ahmed
cc02dc0a55
fix: Handle Input/Output Errors in vsnprintf and snprintf (#35898)
Fix for Issue: #35897
2024-09-04 08:15:04 +08:00
congqixia
69b1eea6f2
enhance: [GoSDK] Add load option for field partial load (#35920)
Related to #35415

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-03 19:27:04 +08:00