270 Commits

Author SHA1 Message Date
PowderLi
2d5f674c78
fix: [restful]use default search parameter nq: 0 (#32362)
issue: #32225
master pr: #32355 #32485
v2.4 pr: #32356 #32486

1. v1 can only accept one vector, but v2 accept list of vectors
2. cannot get dbName from AliasReq #31978
3. parameters of create collection #31176

---------

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-04-23 17:42:19 +08:00
PowderLi
0f118e7083
fix: restful v2 (#32162)
issue: #31176
master pr: #32144
2.4 pr: #32160

1. cannot get dbName correctly while describe alias [Bug]: [restful
v2]The dbName parameter of the describe alias interface did not take
effect #31978
2. return a valid json string even if the user doesn't have the whole
privileges to describe collection [Bug]: [restful v2]When accessing the
interface with a user without permission, the response will contain
multiple results, some of which will report an error of lack of
permission, while one will return the correct result #31635
3. rename IndexParam.IndexConfig to IndexParam.Params
4. FieldSchema.ElementTypeParams, IndexParam.Params can not only accept
string

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-04-23 16:07:09 +08:00
cqy123456
47f767cf32
enhance: remove float16 in 2.3 branch (#31720)
issue: https://github.com/milvus-io/milvus/issues/31696

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-03-30 10:49:13 +08:00
PowderLi
f2f0d44a5d
feat: [cherry-pick] restful phase two (#30430)
issue: #28348 #29732

Support to trace the grpc request, pr: #28349
Support to trace restful request and request error, pr: #28685

restful phase two, pr: #29728 #30343
include: collections, entities, partitions, users, roles, indexes,
aliases, import jobs

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: PowderLi <min.li@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
2024-03-25 10:39:09 +08:00
jaime
5ddb0b435f
fix: revoke session may be ignored due to server context cancellation in advance (#31213)
issue: #31219
pr: #31220

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-03-14 19:05:04 +08:00
PowderLi
c93f127c7d
fix: [cherry-pick] [restful v1] bug list (#30873)
master pr: #30871 issue: #30870
fix: vector field cannot be empty while insert
did a check whether the vector field is empty in advance

master pr: #30740
fix:
1. spelling mistake about metricsType #30643
2. int64 percious #20415
3. insert into collection which has multi vector fields #30674

enhance: support dataType: Float16Vector & BFloat16Vector #22837
#30980(master pr: #30969)
enhance: describe collection will show the field is partition key or not
#30789

---------

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-03-03 17:56:59 +08:00
PowderLi
a4219cbb0f
fix: [cherry-pick] set proxy.http.acceptTypeAllowInt64: true as default (#30738)
issue: #30680
pr: #30720

also let the parameter item to be refreshable

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-02-29 09:59:07 +08:00
chyezh
a9625ec1ae
fix: nil ptr is used as nil interface in grpc client (#30755)
issue: #30715
pr: #30754

- Bug: Set nil struct pointer to describe nil interface.
Panic with segment violation when calling method on this nil struct
pointer.

Signed-off-by: chyezh <chyezh@outlook.com>
2024-02-23 10:08:54 +08:00
yah01
f50799b7fd
fix: proxy may never setup if the port binded (#30035) (#30416)
the proxy miss-returned nil while failed to listen the port, then the
server continues to run but we can't connect to service resolve #30034
pr: #30035

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-02-02 16:21:06 +08:00
chyezh
77e123762f
enhance: add graceful stop timeout to avoid node stop hang under extreme cases (#30320)
1. add coordinator and proxy graceful stop timeout to 5s.
3. add other work node graceful stop timeout to 900s, and we should
potentially change this to 600s when graceful stop is smooth
4. change the order of datacoord component while stop.
5. `LivenessCheck` do not perform graceful shutdown now. 

issue: https://github.com/milvus-io/milvus/issues/30310
pr: #30317
also see: https://github.com/milvus-io/milvus/pull/30306

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-01-27 08:45:02 +08:00
SimFG
95cd6f20d0
fix: [2.3] wrong format expr for the delete rest api (#30218)
/kind improvement
issue: https://github.com/milvus-io/milvus/issues/30092
pr: #30217

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-01-24 11:27:05 +08:00
PowderLi
3dc2585d9b
enhance: support dataType: array & json (#30077)
issue: #30075 
master pr: #30076

deal with the array<?> field data correctly

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-21 14:00:56 +08:00
zhenshan.cao
9aceff5a6e
fix: duplicate dynamic field data by mistake (#30043)
issue: #30000 
pr: https://github.com/milvus-io/milvus/pull/30042

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-01-17 00:20:55 +08:00
PowderLi
ff93e8b489
fix: [CHERRY-PICK] CollectionSchema.autoID is deprecated (#30011)
issue: [#30000](https://github.com/milvus-io/milvus/issues/30000)
related to: [milvus-proto
#202](https://github.com/milvus-io/milvus-proto/pull/202)
master pr: #30002

1. replace collSchema.AutoID with primaryField.AutoID
2. show `enableDynamic` & `enableDynamicField` at the same time
3. avoid data race about the access to metacache

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-16 14:32:53 +08:00
wayblink
e1446da83c
feat: [Cherry-pick] Implement DescribeAlias and ListAliases interfaces (#29896)
#22882
pr: #29641

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-01-12 16:30:51 +08:00
congqixia
0c83440f99
enhance: [Cherry-pick] cache collection schema attributes to reduce proxy cpu (#29668) (#29692)
Cherry-pick from master
pr: #29668

See also #29113

The collection schema is crucial when performing search/query but some
of the information is calculated for every request.

This PR change schema field of cached collection info into a utility
`schemaInfo` type to store some stable result, say pk field,
partitionKeyEnabled, etc. And provided field name to id map for
search/query services.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-07 22:36:48 +08:00
PowderLi
9e7974a2c6
fix:[CHERRY_PICK] link with install path's libblob-chunk-manager (#29731)
issue: #29494
pr: #29496


1. link with install path's libblob-chunk-manager
2. performance of `ShouldBindWith` is better than `ShouldBindBodyWith`
3. the middleware shouldn't read the unrefreshed parameter repeatly
4. diff from old restful, named HandlersV1

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-07 19:52:49 +08:00
wei liu
5fa63da4e3
fix: Rename invalid parameterutil package path (#29334) (#29391)
pr: #29334
This PR renames the invalid parameterutil package path

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-22 10:06:41 +08:00
PowderLi
4ab02da3ac
enhance: RESTFUL search api support range search (#29097)
issue: #29004
master pr: #29055

add a new parameter: `params`, which is a map[string]float64;
but now only 2 valid item: radius + range_filter;

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-17 23:14:45 +08:00
wei liu
7ef49adbd2
enhance: Use mockery to replace manual mock code (#29074) (#29182)
issue: #29043
pr: #29074
This PR remove mannul mock code for proxy and data coord

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-14 16:44:43 +08:00
PowderLi
92db41518d
enhance: upgrade proto to use partition related privileges (#29100)
issue: [milvus-proto
#212](https://github.com/milvus-io/milvus-proto/issues/212)
master pr: #28961

milvus can't use partition related privileges until upgrade
milvus-proto, even if them were added to milvus-proto

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-12 13:22:38 +08:00
shaoyue
ac0f000c29
enhance: Add proxy.ginLogSkipPaths (#29008)
pr: #28945

/cc @czs007

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2023-12-08 17:55:30 +08:00
MrPresent-Han
c21a1d59e4
fix: createCollection failed occasionally (#28592) (#28989)
related: #28592
pr:#https://github.com/milvus-io/milvus/pull/28712

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-08 11:34:39 +08:00
MrPresent-Han
5f4ac437b2
enhance: [Cherry-pick] Moving etcd client into session (#27069) (#28996)
relate: #26694
pr: https://github.com/milvus-io/milvus/pull/27069

Signed-off-by: Filip Haltmayer <filip.haltmayer@zilliz.com>
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Filip Haltmayer <81822489+filip-halt@users.noreply.github.com>
2023-12-07 16:22:34 +08:00
aoiasd
cb7b1d1d51
enhance: [Cherry-pick] pack proxy connection code and support accesslog print SDK_Version (#28844)
relate: https://github.com/milvus-io/milvus/issues/28086
pr: https://github.com/milvus-io/milvus/pull/28835

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-07 12:06:34 +08:00
PowderLi
cdd1305f26
enhance: find collection schema from cache (#28916)
issue: #28781 #28329
master pr: #28782

1. There is no need to call `DescribeCollection`, if the collection's
schema is found in the globalMetaCache
2. did `GetProperties` to check the access to Azure Blob Service

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-06 22:44:33 +08:00
aoiasd
224f2755c5
enhance: [Cherry-pick]Refine accesslog to support format by yaml and support more metric (#28740)
relate: https://github.com/milvus-io/milvus/issues/28086
pr: https://github.com/milvus-io/milvus/issues/28086

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-28 15:34:28 +08:00
jaime
2e78b4de96
fix: cmux graceful shutdown on proxy service #28383 (#28680)
issue: https://github.com/milvus-io/milvus/issues/28305
pr: #28383

Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-11-27 16:28:34 +08:00
yah01
5ca7851f4a
enhance: refine error messages (#28424) (#28614)
- Split the simple reason and full detail
- Refine existing error messages related: #28422
related: https://github.com/milvus-io/milvus/issues/28422
pr: #28424

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-24 10:04:24 +08:00
PowderLi
92e5f96064
fix: symbol 'GetStorageMetrics' and 'enableDynamicField' (#28581)
issue: #28579 #28504
pr: #28580

1. replace enableDynamic with enableDynamicField
2. cgo directly link to milvus_storage

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-21 15:30:23 +08:00
wei liu
f2e972abb3
Fix external grpc server doesn't graceful stop (#28310)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-09 22:12:22 +08:00
wei liu
d76c744646
Fix proxy not healthy before graceful stop (#28255)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-08 02:22:26 +08:00
PowderLi
43db873770
add a custom http header: Accept-Type-Allow-Int64 (#28125)
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-06 16:02:18 +08:00
jaime
bf7f32b025
Refine Proxy stop (#27919)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-10-26 10:24:09 +08:00
jaime
4640928280
Fix initialization and backward compatibility issue for GRPC compression (#27894)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-10-24 22:14:14 +08:00
SimFG
9b0ecbdca7
Support to replicate the mq message (#27240)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-10-20 14:26:09 +08:00
PowderLi
dcb7a89601
[restful] new interface: upsert (#27787)
interface: delete support expression

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-10-20 09:54:11 +08:00
SimFG
630636c4ec
Support the apikey authentication for the restful api (#27758)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-10-18 16:36:12 +08:00
SimFG
84d05b939b
Support the apikey authentication (#27723)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-10-17 21:00:11 +08:00
congqixia
2f201c25e2
Remove deprecated io/ioutil usage (#27747)
`io/ioutil` package is deprecated, use `io`,`os` package replacement
also added golangci-lint rule to block future reference

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: guoguangwu <guoguangwu@magic-shield.com>
2023-10-17 20:32:09 +08:00
PowderLi
09d8b76048
[restful] new context with grpc metadata (#27668)
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-10-17 20:00:14 +08:00
yah01
be980fbc38
Refine state check (#27541)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-11 21:01:35 +08:00
yah01
8394b3a1ec
Block creating new error from status reason (#27426)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-07 11:29:32 +08:00
yah01
63ac43a3b8
Refine errors for import (#27379)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-30 10:31:28 +08:00
wei liu
27046e242b
add timeout for graceful stop (#27326)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-09-26 10:15:25 +08:00
jaime
7f7c71ea7d
Decoupling client and server API in types interface (#27186)
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-09-26 09:57:25 +08:00
SimFG
26f06dd732
Format the code (#27275)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
congqixia
cc9974979f
Add staticcheck linter and fix existing problems (#27174)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-19 10:05:22 +08:00
yah01
168e82ee10
Fix panic while handling with the nil status (#27040)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-15 10:09:21 +08:00
congqixia
f0d0651989
Do not reset connection immediately if grpc code is Canceled or DeadlineExceeded (#27014)
We found lots of connection reset & canceled due to recent retry change
Current implementation resets connection no matter what the error code is
To sync behavior to previous retry, skip reset connection only if cancel error happens too much.

Also adds a config item for minResetInterval for grpc reset connection

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-13 15:01:18 +08:00