admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-02-02 01:06:41 +08:00
Code Issues Packages Projects Releases Wiki Activity
18,411 Commits 146 Branches 196 Tags
Commit Graph

7757 Commits

Author SHA1 Message Date
yah01
1ab538be43
enhance: enable assert method to format arguments (#28812) (#28907) 
for now the assert method in segcore could accept a string information,
too many codes don't print the value they assert.

make it happy
related #28811
pr: #28812

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-05 10:22:32 +08:00
congqixia
2873be9264
fix: [2.3] Reject compaction task with growing segments (#28927) 
See also #28924
The compaction task generated before datanode finish SaveBinlogPath grpc
call contains segments which are still in Growing state DataNode shall
verify each non-levelzero segments before submit compaction task to
executor

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-04 19:14:33 +08:00
congqixia
3a33afd1fb
enhance: [cherry-pick] Change const magic number in querycoord to param (#28819) (#28947) 
Cherry-pick from master
pr: #28819 
See also #28817

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-04 19:06:40 +08:00
wei liu
c650240f31
enhance: Change some frequency log to rated level (#28897) (#28934) 
pr: #28897
This pr change some frequency log's level to rated.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-12-04 18:52:32 +08:00
congqixia
ed550ebb7c
enhance: [cherry-pick] Resolve libunwind requirement conflict using 1.7.2 (#28935) 
Cherry-pick from master
pr: #28929
Try to resolve libunwind dependency requirement conflict between glog &
folly

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-04 14:06:40 +08:00
congqixia
f4468dfd15
fix: [cherry-pick] Check whether compaction plan exist before update(#28826) (#28892) 
pr: #28826 
Fix #28825

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-04 10:02:33 +08:00
wei liu
d2c171354f
fix: Balance channel may cause channel not availble error (#28829) (#28902) 
pr: #28829
issue: #28831
release old delegator before new delegator update it's distribution may
cause `channel not available` error
This PR will block release old delgator before new delegator finish
`syncDistribution`

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-12-04 09:40:32 +08:00
jaime
275e3ea456
fix: Fix missing target db name for RenameCollection (#28911) 
issue: #28908
pr: #28909

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-12-03 08:28:31 +08:00
PowderLi
96c8e8d587
fix: azure ListObjects (#27931) (#28894) 
issue: #27932
master pr: #27931

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-12-01 18:34:31 +08:00
yihao.dai
c936cc8a95
fix: Fix get binary vector from chunk cache (#28866) (#28884) 
The way of getting binary vector size is wrong. This PR will fix it.

issue: https://github.com/milvus-io/milvus/issues/28865

pr: https://github.com/milvus-io/milvus/pull/28866

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-12-01 17:50:32 +08:00
aoiasd
9bf33eb576
fix: [Cherry-Pick] accesslog method status not return failed when error in response (#28827) 
relate: https://github.com/milvus-io/milvus/issues/28086
pr: https://github.com/milvus-io/milvus/pull/28824

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-12-01 15:30:38 +08:00
Gao
b6cb9e90d2
fix: [2.3] update folly to resolve simd issue (#28879) 
related issue: https://github.com/milvus-io/milvus/issues/27552 , after
this, milvus could run successfully on sse4.2 only machine
pr: #28878

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-12-01 13:52:32 +08:00
congqixia
63e1ac0846
fix: [cherry-pick] schema->size() check logic with system field (#28802) (#28841) 
Cherry pick from master
pr: #28802

Now segcore load system field info as well, the growing segment
assertion shall not pass with "+ 2" value
This will cause all growing segments load failure
Fix #28801
Related to #28478
See also #28524

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-01 13:28:30 +08:00
yah01
ce8219f1a3
fix: bypass growing index if no index meta (#28791) (#28859) 
we shouldn't panic if no index meta, just skip building it
fix #28022
pr: #28791

---------

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-11-30 21:10:32 +08:00
aoiasd
880189e598
enhance: Delete by complex expression use partitionKey optimization (#27952) (#28128) 
relate: https://github.com/milvus-io/milvus/issues/27948
pr: https://github.com/milvus-io/milvus/pull/27952

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-11-30 17:56:31 +08:00
yihao.dai
d0a56ecca5
fix: Fix minio latency monitoring for get operation (#28510) (#28643) 
see also: https://github.com/milvus-io/milvus/issues/28509

pr: https://github.com/milvus-io/milvus/pull/28510

Currently Minio latency monitoring for get operation only collects the
duration of getting object (which just returns an io.Reader and does not
really read from minio), this pr will correct this behavior.

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-11-30 10:44:28 +08:00
cai.zhang
00461145b3
enhance: [Pick]Support importing data with parquet file (#28820) 
issue: https://github.com/milvus-io/milvus/issues/28272
master pr: #28608 

Numpy does not support array type import.
Array type data is imported through parquet.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-29 22:16:29 +08:00
cai.zhang
2062e5b368
enhance: [Pick]Update arrow version to v12 (#28425) (#28828) 
master pr: #28425 
issue: https://github.com/milvus-io/milvus/issues/28423

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-29 20:04:28 +08:00
congqixia
e3653af9c9
fix: data race on rootcoord.proxyClientManager.proxyClient (#28787) (#28798) 
Cherry-pick from master
pr: #28787
Fix #28786

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-29 18:00:27 +08:00
yihao.dai
a78ea4fea0
fix: Check ErrSegmentNotFound in delete node (#28371) (#28638) 
We have been check ErrSegmentNotFound in insert_buffer_node in datanode,
we should also check it in delete_node.

issue: https://github.com/milvus-io/milvus/issues/27145

pr: https://github.com/milvus-io/milvus/pull/28371

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-11-29 17:50:27 +08:00
congqixia
0688ec4d83
enhance: [cherry-pick] Add channel checkpoint ts lag metric (#28767) (#28776) 
Cherry-pick from master
pr: #28767
See also #28765

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-28 19:28:28 +08:00
Gao
ccca932cc6
fix: [2.3] correct autoindex segment num (#28429) 
issue: #28386 
pr: #28387

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-11-28 19:24:26 +08:00
cai.zhang
fda452ea4d
enhance: [Pick] Revert import support csv format (#28766) 
issue: #28778 
master pr: #28760

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-28 18:26:28 +08:00
aoiasd
224f2755c5
enhance: [Cherry-pick]Refine accesslog to support format by yaml and support more metric (#28740) 
relate: https://github.com/milvus-io/milvus/issues/28086
pr: https://github.com/milvus-io/milvus/issues/28086

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-11-28 15:34:28 +08:00
yah01
8ea1ef03ad
enhance: store system fields in segcore (#28524) (#28727) 
we need the system fields info for some usacase
fix: #28523

pr: #28524

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-28 15:32:31 +08:00
Enwei Jiao
7afd7553ed
feat: trigger compaction to handle index version (#28442) (#28754) 
issue: https://github.com/milvus-io/milvus/issues/28441
pr: https://github.com/milvus-io/milvus/pull/28442

---------

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-11-28 10:24:26 +08:00
wayblink
1ed92da414
enhance:[cherry-pick] Merge flowgraph goroutines into 1 (#28728) 
see also: #28654 #28233

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-11-27 21:04:27 +08:00
congqixia
5a962a631a
fix: [cherry-pick] Change schema to atomic.Pointer to avoid data race (#28739) (#28759) 
Cherry-pick from master
pr: #28739
See also #28738

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-27 19:50:27 +08:00
jaime
9378f78218
enhance: Add logs for each step during service initialization (#28687) 
/kind improvement
pr: #28624

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-11-27 17:54:26 +08:00
cai.zhang
ef115dd7df
enhance: Support Array DataType for bulk_insert (#28578) 
issue: https://github.com/milvus-io/milvus/issues/28272
master pr: #28341 
Support array DataType for bulk_insert with json, binlog files.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-27 16:56:27 +08:00
jaime
2e78b4de96
fix: cmux graceful shutdown on proxy service #28383 (#28680) 
issue: https://github.com/milvus-io/milvus/issues/28305
pr: #28383

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-11-27 16:28:34 +08:00
yah01
55fda49104
enhance: validate JSON data while inserting (#28602) (#28725) 
some SDKs doesn't check the JSON data validation,
add this in server.

pr: #28602

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-27 16:16:26 +08:00
yah01
c4be1ddf44
enhance: improve the error messages and logs (#28684) (#28726) 
- better name for log fields
- make the error and log consistent

pr: #28684

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-27 15:40:26 +08:00
yihao.dai
8520ee7552
enhance: Print nq (#28507) (#28639) 
Log nq in search path.

pr: https://github.com/milvus-io/milvus/pull/28507

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-11-27 10:18:26 +08:00
yah01
a1b861ed7a
enhance: improve load speed (#28518) (#28719) 
This check rejects load request if running out the pool workers, but
small segment would be loaded soon, another segments would been loading
again after a check interval, which leads to slow loading for collection

Block the request by go pool

pr: #28518

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-26 22:10:26 +08:00
cai.zhang
3b06db1d04
enhance: [Pick] Handle knowhere error for creare diskann index (#28691) 
master pr: #28690

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-26 18:58:25 +08:00
congqixia
6512b12fba
enhance: [cherry-pick] Make etcd kv request timeout configurable (#28661) (#28701) 
Cherry-pick from master
pr: #28661
See also #28660
This pr add request timeout config item for etcd kv request timeout
 Sync the default timeout value to same value for etcdKV & tikv config

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-24 21:16:26 +08:00
congqixia
e4ea148c0f
fix: [cherry-pick] Add IndexList check for load segment request (#28601) (#28700) 
Cherry-pick from master
pr: #28601
See also #28022 #28034
The load segment may reaches before watch dml channel, so the index meta
may be empty as well

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-24 15:02:39 +08:00
smellthemoon
288844e3cf
enhance: Reduce the goroutine in flowgraph to 2 (#28233) (#28545) 
each node in flow graph alloc a goroutine, but it is actually executed
sequentially and can be placed in one goroutine. InputNode will consume
msg form msgstream, alloc one goroutine.
issue: #24826 
pr: #28233

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-11-24 15:00:26 +08:00
chyezh
1b86efbafb
fix: [cherry-pick] error root path for catalog when using tikv (#28463) 
Related Issue: #28162 
Related pr: #28163

Signed-off-by: chyezh <ye.zhen@zilliz.com>
 2023-11-24 14:28:24 +08:00
yah01
c6ba4fa102
enhance: enhance the accuracy of memory usage (#28554) (#28559) 
before this, Milvus use container/system's memory info to get the memory
usage, which could be inaccurate.

we allocates the memory by private anon mmap,
then rss - shared would be the accurate memory usage

resolve https://github.com/milvus-io/milvus/issues/28553
pr: #28554

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-24 14:12:25 +08:00
cai.zhang
968d6fdcde
fix: Fix bug for parse expression that include quotes (#28418) 
master pr: #28416 
issue: #28365
Fix bug for parsing error when a string enclosed in single quotes in an
expression contains multiple double quotes.
such as:
```
expr = "tag == '\"blue\"'"
```

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-24 11:02:24 +08:00
yah01
5ca7851f4a
enhance: refine error messages (#28424) (#28614) 
- Split the simple reason and full detail
- Refine existing error messages related: #28422
related: https://github.com/milvus-io/milvus/issues/28422
pr: #28424

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-24 10:04:24 +08:00
zhagnlu
0bda17e97b
enhance: Add precheck when chunk manager init (#28593) 
#28329 
pr: https://github.com/milvus-io/milvus/pull/28330

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-11-23 20:34:25 +08:00
zhagnlu
41667f3227
enhance: Disable stdout buffer (#28597) 
#28390 
pr: https://github.com/milvus-io/milvus/pull/28391

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-11-23 20:20:24 +08:00
wei liu
33bbdf6c88
fix: Fix channel manager bg checker exit when disable auto balance (#28459) (#28606) 
issue: #28454
pr: #28459

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-23 19:18:29 +08:00
wayblink
a04b528ce7
fix: [cherry-pick] failed to release collection with more than 128 partitions (#28567) 
issue: #28343
pr: #28446

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-11-22 14:00:23 +08:00
smellthemoon
d2ebbe2317
enhance: create goroutine only once(#28594) (#28609) 
create goroutine only once when getOrCreateMergedTimeTickerSender
pr: #28594

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-11-22 10:26:28 +08:00
congqixia
0fbd610e89
fix: [cherry-pick] querynodev2 local worker failed to maintain collection ref (#28631) 
Cherry-pick from master
pr: #28590 #28598
See also #28589 #28596 
Increase ref for collection during load and unref after load completed.
Use the same logic protection from services.go `LoadSegments`
Perform `Unref` after release sealed segments

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-22 10:04:23 +08:00
wei liu
c7ec882033
enhance: Remove rpc during querycoord start (#28396) (#28604) 
issue: #28332
pr: #28396

during querycoord's recover, it try to call `DescribeCollection` and
`ShowPartitions` to root coord, to checker whether collection or
partition has been released in rootcoord. but if rootcoord isn't not
ready yet, the rpc will fail, the querycoord panic.

to fix this, we remove rpc call during querycoord's start

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-21 18:08:29 +08:00