8293 Commits

Author SHA1 Message Date
aoiasd
f3d1c75499
enhance: optimize access log formatter and add bench test (#30829)
relate: https://github.com/milvus-io/milvus/issues/28948
https://github.com/milvus-io/milvus/issues/30806

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-03-01 10:41:00 +08:00
yihao.dai
a5a4ca8459
enhance: Remove debug log (#30955)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-01 10:02:59 +08:00
MrPresent-Han
17a2fd048e
feat: support set up knowhere-build-pool-size on querynode(#29650) (#30922)
related: #29650

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-02-29 18:15:00 +08:00
chyezh
0c7474d7e8
enhance: add graceful stop timeout to avoid node stop hang under extreme cases (#30317)
1. add coordinator graceful stop timeout to 5s
2. change the order of datacoord component while stop
3. change querynode grace stop timeout to 900s, and we should
potentially change this to 600s when graceful stop is smooth

issue: #30310
also see pr: #30306

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-02-29 17:01:50 +08:00
Bingyi Sun
816ed671aa
fix: alter_index should return error if index not found (#30786)
issue: https://github.com/milvus-io/milvus/issues/30932

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-02-29 15:55:01 +08:00
Jiquan Long
dcfc35316f
fix: underscore matching not work (#30826)
issue: https://github.com/milvus-io/milvus/issues/30721

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-29 10:41:00 +08:00
chyezh
e59bc1e526
enhance: use latest milvus proto, add unimplemented server (#30889)
issue: #30647

Signed-off-by: chyezh <chyezh@outlook.com>
2024-02-29 10:13:00 +08:00
PowderLi
50a78b682e
fix: set proxy.http.acceptTypeAllowInt64: true as default (#30720)
issue: #30680

also let the parameter item to be refreshable

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-02-29 09:59:07 +08:00
yihao.dai
3775464b7c
enhance: Support varchar autoid for bulkinsertV1 (#30896)
This PR is a supplement to PR
https://github.com/milvus-io/milvus/pull/30377.

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-02-28 21:20:59 +08:00
XuanYang-cn
cdc5ce5d6f
fix: Donot set LogPath when executing compaction (#30537)
Compaction would copy logPaths from comapctFrom segA to compactTo segB,
and previous code would copy the logPath directly, causing there're
full-logPaths-of-segA in compactTo segB's meta. So, for the next
compaction of segB, if segA has been GCed, Download would report error
"The sperified key not found".

This PR makes sure compactTo segment's meta contains logID only. And
this PR also refines CompleteComapctionMutation, increasing some
readability and merge two methods into one.

See also: #30496

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-02-28 19:03:01 +08:00
congqixia
4082315bd0
enhance: Add ParseCTraceContext util function for tracing (#30883)
See also #29803

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-28 18:59:00 +08:00
aoiasd
81a2d9ced6
enhance: access log support get sdk type by user agent (#30760)
Support get sdk type by user agent when we can't get sdk version by
connection in access log.

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-02-28 18:47:00 +08:00
Jiquan Long
e2f35954d4
enhance: support pattern matching on json field (#30779)
issue: https://github.com/milvus-io/milvus/issues/30714

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-28 18:31:00 +08:00
Jiquan Long
16b785e149
enhance: optimize the memory usage and speed up loading variable length data (#30787)
/kind improvement
this removes the 1x copying while loading variable length data, also
avoids constructing std::string, which could lead to memory
fragmentation

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: yah01 <yah2er0ne@outlook.com>
2024-02-28 16:45:00 +08:00
Jiquan Long
4459078e0b
fix: wrong num_entities used when mmap variable length data (#30848)
https://github.com/milvus-io/milvus/issues/30728

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-28 16:38:56 +08:00
SimFG
229fc4f755
enhance: retry to read when the s3 get the unexpect eof error (#30861)
/kind improvement
issue: #30877

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-02-28 16:28:53 +08:00
congqixia
a115b731ed
enhance: fix old pr cpp format issue (#30894)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-28 16:28:20 +08:00
Buqian Zheng
f658dd5faa
enhance: update knowhere version to 60a5c9c (#30788)
/kind improvement

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-02-28 14:18:55 +08:00
wei liu
545e8de401
fix: promote leader task failed when segment only exist on current target (#30794)
issue: #30150

`checkLeaderTaskStale` will check segment whether exist on next current
for leaderTask's growing action, which will cause promote leader task
failed when segment only exist on current target

This PR will check segment for both current or next target.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-02-28 13:14:59 +08:00
wei liu
382b7485a6
enhance: Correct misleading nodeID in GetComponentStates's log (#30731)
This PR corrects the misleading nodeId in GetComponentStates's log

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-02-28 13:13:12 +08:00
Cai Yudong
8a219e0102
feat: Support knowhere trace using OpenTelemetry (#30750)
Issue: #21508

Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-02-28 12:29:00 +08:00
smellthemoon
a4f3e01a3a
fix: add the range search params check in proxy (#30423)
if check in Segcore, will not do the it when not insert data.
so, check "radius" and "range_filter" in proxy.
related with #30365

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-02-28 11:24:58 +08:00
sre-ci-robot
6e9f3ea531
[automated] Update Knowhere Commit (#30744)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-02-28 10:50:57 +08:00
chyezh
77477d6340
fix: wrong context passing into NewClient, error handling lost in session_util (#30817)
issue: #30799

Signed-off-by: chyezh <chyezh@outlook.com>
2024-02-28 10:40:09 +08:00
PowderLi
6548182a4a
fix: [restful]vector field cannot be empty while insert (#30871)
issue: #30870 

did a check whether the vector field is empty in advance

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-02-28 10:36:57 +08:00
congqixia
af315539d4
fix: Remove time tick delay metrics when nodes go offline (#30833)
See also #30832

This PR removes time tick delay metrics when rootcoord GetMetrics
response does not have previously existed querynode/datanode

Also add unit tests for this case

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi.Xia <congqi.xia@zilliz.com>
2024-02-28 10:10:56 +08:00
yah01
57397b1307
enhance: add new LRU cache impl (#30360)
- remove  the unused LRU cache
- add new LRU cache impl which wraps github.com/karlseguin/ccache

related #30361

---------

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-02-27 20:58:40 +08:00
PowderLi
19911a411f
fix: [restful v2] bug list (#30740)
bug list
1. element data type is needed while create a collection with an array
field #30638
2. spelling mistake about metricsType #30643
3. new collection enable dynamic field / auto id as user defined #30665
4. convert rowCount from string to int #30661
5. try it's best to create a new collection #30652
6. int64 percious #20415
7. insert into collection which has multi vector fields #30674
8. cannot rename a collection to other database #30700
9. update the request body of "indexes/create" #30769
10. got [] while list indexes of a collection which has no index #30722
11. restful need encode password before call
CreateCredential/UpdateCredential #30730
12. some parameters missed the required label #30737
13.define the field to be or not to be a partition key while create a
collection #30797

enhance: support opentelemetry
enhance: support dataType: Float16Vector & BFloat16Vector #22837
enhance: describe collection will show the field is partition key or not
#30789

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-02-27 16:48:56 +08:00
Xiaofan
0661d33879
enhance: change frequent log to debug (#30782)
change the "pipeline fetch insert msg" log to debug

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-02-27 10:30:56 +08:00
Jiquan Long
3e82d21ca1
enhance: reduce 1x memory copy when loading json (#30753)
/kind improvement

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-27 10:18:55 +08:00
congqixia
637dcffb6b
fix: Disk resource is not requested for index loaded with disk (#30757)
See also #30756

This PR:
- Request disk resource when index type, version loaded with disk
- Add attribute cache for index utility
- Add `typeutil.Pair`

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-27 08:50:56 +08:00
foxspy
e1e87d572b
fix: compatibility for diskann cache param (#30119)
patch search cache param from index configs when index meta could not
get the search cache size key
#issue: #30113

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-02-26 16:54:55 +08:00
zhagnlu
a0531b72aa
fix: reorder hybridsearch result for L2 metric (#30739)
#30694

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-02-26 14:18:55 +08:00
Bingyi Sun
ece9d273a7
enhance: some patches for #30636 (#30664)
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-02-26 11:42:55 +08:00
smellthemoon
9815cf50c9
fix: close rocksmq and natsmq when shutdown (#30706)
related: #30703 #30355

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-02-26 11:38:55 +08:00
wei liu
befe0e21fd
fix: Set indexInfo when try to set segment to leader view (#30758)
issue: #30150
see also: #30258

cause `SyncDataDistribution` will try to load delta for segment. if miss
indexInfo in request, sync action will failed due to lack of index info.

This PR set indexinfo when try to set segment to leader view

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-02-26 11:02:55 +08:00
Xiaofan
d970a9fc6c
enhance: refine error log of json parse error (#30671)
add more error info when json format does not match.
relate to https://github.com/milvus-io/milvus/issues/29837

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-02-24 12:32:14 +08:00
wei liu
6dd7297178
fix: Skip generate balance task when target not ready (#30724)
issue: #30723

This PR skip generate balance task when collection's target isn't ready.
also refine the check stale logic in query coord's scheduler, if channel
exist in current or next target, task won't be canceled.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-02-23 10:32:53 +08:00
chyezh
950624d8d3
fix: nil ptr is used as nil interface in grpc client (#30754)
issue: #30715

- Bug: Set nil struct pointer to describe nil interface.
Panic with segment violation when calling method on this nil struct
pointer.

Signed-off-by: chyezh <chyezh@outlook.com>
2024-02-23 10:26:53 +08:00
congqixia
536e082f5a
enhance: Use defer to clean legacy growing segment (#30770)
See also #30735 #30735

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-23 10:02:55 +08:00
congqixia
90e802e441
enhance: Add unit test for level-zero segment deltalog request resource (#30718)
See also #30670 #30704

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-23 10:00:54 +08:00
zhagnlu
c5363c70db
fix: fix upsert using wrong field to compute partition key (#30772)
#30607

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-02-23 09:54:53 +08:00
cai.zhang
16b4c9a79e
fix: Skip filling segmentID in indexBuildCh to prevent flush blocked (#30747)
issue: #30580

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-02-22 20:40:53 +08:00
Jiquan Long
e2330f02f8
fix: pattern match use incorrect raw data (#30764)
issue: https://github.com/milvus-io/milvus/issues/30687
We store all the varchar datas in an continuous address and use
string_view to quickly find them. In this case, using string_view.data()
directly will point to all rest varchar datas.

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-22 19:56:52 +08:00
aoiasd
e5353ad049
enhance: access log should get get client info by get method (#30502)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-02-22 14:58:52 +08:00
congqixia
f3b7a8892f
fix: Release loaded growing if WatchDmlChannel fail (#30735)
See also #30734

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-21 22:12:53 +08:00
Ted Xu
12acaf3e4f
enhance: Adding a generic stream payload reader (#30682)
See: #30404

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-02-21 17:10:52 +08:00
MrPresent-Han
77eb6defb1
feat: support groupby on growing and non-indexed sealed egment(#30307) (#30644)
related: #30308

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-02-21 14:02:53 +08:00
zhagnlu
18aac076de
fix: move test from NEON to X86 (#30324)
#26137

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-02-21 11:58:53 +08:00
yiwangdr
c6665c2a4c
test: support multiple data/querynodes in integration test (#30618)
issue: https://github.com/milvus-io/milvus/issues/29507

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-02-21 11:54:53 +08:00