1254 Commits

Author SHA1 Message Date
yah01
a0e1a1eb31
feat: support enable/disable mmap for index (#29005)
support enable/disable mmap for index, the user could alter the index's
mode by `AlterIndex` method
related: https://github.com/milvus-io/milvus/issues/21866

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-21 18:07:24 +08:00
wei liu
fcba1c0d9e
fix: Rename invalid parameterutil package path (#29334)
This PR renames the invalid parameterutil package path

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-21 11:40:42 +08:00
yihao.dai
f457b9f7c9
fix: Return time tick delay error and refine quota error messages (#29289)
This pr:
1. Handles the time tick delay error when converting old error codes to
milvus errors.
2. Enhances quota error messages by eliminating "force deny" and
substituting it with "quota exceeded."

issue: https://github.com/milvus-io/milvus/issues/29288

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-12-20 22:52:44 +08:00
cai.zhang
1b4d2674b3
fix: Set the default index name to the name of the existing index (#29275)
issue: #29269

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-20 17:22:41 +08:00
congqixia
bcf8f27aa7
enhance: refine proxy meta cache partition logic (#29315)
See also #29113

- Unify partition info refresh logic
- Prevent parse partition names for each partition key search request

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-20 10:02:43 +08:00
PowderLi
bcd6865b29
enhance: add 3 builtin roles (#28961)
issue: #28960 [milvus-proto
#212](https://github.com/milvus-io/milvus-proto/issues/212)

add new configuration: builtinRoles
user can define roles in config file: `milvus.yaml`

there is an example:
1. db_ro, only have read privileges, include load
2. db_rw, read and write privileges, include create/drop/rename
collection
3. db_admin, not only read and write privileges, but also user
administration

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-18 14:28:41 +08:00
cai.zhang
2f7252b44e
enhance: Set default index name as field name (#29218)
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-15 12:06:39 +08:00
congqixia
8a63e53421
enhance: Add http method to control datacoord garbage collection (#29052)
See also #29051

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-14 19:26:39 +08:00
congqixia
264505b706
enhance: remove reset seed for every shuffle (#29195)
See also #29113
rand.Seed is deprecated and cost noticable CPU time during heavy payload

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-14 15:46:38 +08:00
congqixia
9407be6eb2
enhance: Add a config item for partition name as regexp feature and disable it by default (#29154)
See also #29177
Add a config item for partition name as regexp feature and disable it by
default

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-14 10:04:38 +08:00
aoiasd
b5ee563914
fix: accesslog can not print search expression (#28899)
relate: https://github.com/milvus-io/milvus/issues/28893

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-13 18:50:42 +08:00
congqixia
cb75e73c77
enhance: Use zap.Stringer for large log field (#29143)
See also #29113
Using zap.Stringer log field will evaluate log field value only when log
level meets the configuration, which could save some CPU time in search
route

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-12 22:52:39 +08:00
congqixia
d0bac9d0bb
enhance: Avoid initializing casbin enforcer for each request (#29117)
See also #29113

This patch:
- Replace plain Enforcer with `casbin.SyncedEnforcer`
- Add implementation of persist.Adapter with `MetaCacheCasbinAdapter`
- Invoke enforcer.LoadPolicy when policy updated

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-12 10:36:43 +08:00
jaime
bb82074937
fix: Fix missing target db name for RenameCollection (#28909)
issue: #28908

Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-12-05 17:12:37 +08:00
aoiasd
3cc4209d26
enhance: pack proxy connection code and support accesslog print SDK_Version (#28835)
relate: https://github.com/milvus-io/milvus/issues/28086
https://github.com/milvus-io/milvus/issues/28940

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-05 16:50:47 +08:00
PowderLi
20fc90c591
enhance: find collection schema from cache (#28782)
issue: #28781 #28329

1. There is no need to call `DescribeCollection`, if the collection's
schema is found in the globalMetaCache
2. did `GetProperties` to check the access to Azure Blob Service while
construct the ChunkManager

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-03 19:22:33 +08:00
aoiasd
24c565e37b
fix: accesslog method status not return failed when error in response (#28824)
relate: https://github.com/milvus-io/milvus/issues/28086

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-01 15:30:38 +08:00
yah01
bf633bb5d7
enhance: refine the retry error (#28573)
return the last error but not combining all errors, to improve
readability and erorr handling

resolve: #28572

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-30 18:34:32 +08:00
yihao.dai
4b8bc2798e
enhance: Print nq (#28507)
Log nq in search path.

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-11-30 10:42:27 +08:00
aoiasd
89d8ce2f73
enhance: refine access log to support format access log by yaml and print name info. (#28319)
relate: https://github.com/milvus-io/milvus/issues/28086

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-28 15:32:31 +08:00
SimFG
9c46788d87
enhance: Support to trace restful request and request error (#28685)
issue: #28348

Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-11-27 20:14:26 +08:00
jaime
b1e0a27f31
enhance: Add logs for each step during service initialization (#28624)
/kind improvement

Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-11-27 16:30:26 +08:00
SimFG
089e58dfbb
fix: Fix the unstable unit test TestReplicateManager (#28718)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-11-24 17:50:30 +08:00
yah01
3ea0129eb3
enhance: improve the error messages and logs (#28684)
- better name for log fields
- make the error and log consistent

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-24 15:08:24 +08:00
smellthemoon
79c0edb1d8
enhance:Remove msgbase unnecessary assignments (#28511)
remove some unnecessary assignments, for the reason that
commonpbutil.NewMsgBase has default value.

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-11-24 15:02:39 +08:00
SimFG
de13865769
enhance: Add load/release partitions to replicate msg stream (#28399)
/kind improvement
issue: #25655

Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-11-23 15:38:24 +08:00
smellthemoon
73f2bab454
enhance:add some log when create client and get component states (#28160)
/kind improvement

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-11-22 09:12:22 +08:00
yah01
d73dac52c7
enhance: validate JSON data while inserting (#28602)
some SDKs doesn't check the JSON data validation,
add this in server.

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-21 23:30:22 +08:00
congqixia
e47fe4e4d1
enhance: Reduce ListImportTasks log content (#28605)
In proxy `ListImportTasks` may print all task information from
rootcoord, this may cause log content too large to process
This PR make this log print taskID and importState only

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-21 20:58:23 +08:00
yah01
bfccfcd0ca
enhance: refine error messages (#28424)
- Split the simple reason and full detail
- Refine existing error messages
related: #28422

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-21 17:02:24 +08:00
SimFG
899a5a32cd
Hide the password info when failing to authorize (#28428)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-11-15 14:40:26 +08:00
XuanYang-cn
40d5c902b6
Enable getting multiple segments in plan result (#28350)
Compaction plan result contained one segment for one plan. For l0
compaction would write to multiple segments, this PR expand the segments
number in plan results and refactor some names for readibility.

- Name refactory: - CompactionStateResult -> CompactionPlanResult -
CompactionResult -> CompactionSegment

See also: #27606

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-14 15:56:19 +08:00
yah01
e9ff7ed13d
enhance: remove legacy error related code (#28385)
/kind improvement

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-14 15:02:18 +08:00
SimFG
cfb6edea61
Support to trace the grpc request (#28349)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-11-13 20:16:18 +08:00
SimFG
ef68680639
Fix the unstable resource manager testcase (#28033)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-11-03 11:00:17 +08:00
aoiasd
d2727cc0ff
Add write cache for access log (#27792)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-03 10:28:17 +08:00
yihao.dai
7bd44bd671
Fix unstable search task ut (#28101)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-11-02 12:52:15 +08:00
PowderLi
0252871d30
fix azure ListObjects (#27931)
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-01 11:34:14 +08:00
aoiasd
2aa330146a
Delete by complex expression use partitionKey optimization (#27952)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-01 11:26:16 +08:00
wei liu
0677d2623d
remove retry in getShards (#28011)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-10-31 16:16:14 +08:00
Ikko Eltociear Ashimine
ed4f20b0ed
Fix typo in util.go (#27975)
suppot -> support

Signed-off-by: Ikko Eltociear Ashimine <eltociear@gmail.com>
2023-10-30 14:40:27 +08:00
Jiquan Long
6913283716
Fill output fields for count request (#27967)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-10-30 10:00:12 +08:00
Filip Haltmayer
6b1a106a31
Moving etcd client into session (#27069)
Signed-off-by: Filip Haltmayer <filip.haltmayer@zilliz.com>
2023-10-27 07:36:12 +08:00
yihao.dai
f9c630247d
Construct plan directly when search with vector output (#27928)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-10-26 19:30:10 +08:00
shaoyue
977e508a0f
Propagate task ctx done err for proxy (#27892)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2023-10-26 19:06:13 +08:00
aoiasd
53246b1b38
Set accesslog default to close and use stdout (#27891)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-10-25 10:30:10 +08:00
aoiasd
9ba79a97e4
Add some log for delete task when it need query (#27317)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-10-24 14:54:12 +08:00
SimFG
9b0ecbdca7
Support to replicate the mq message (#27240)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-10-20 14:26:09 +08:00
congqixia
49516d44b4
Add ctx parameter and log tracer for watch and selectNodes (#27809)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-20 04:22:11 +08:00
zhenshan.cao
020ad9a6bc
Rectify wrong exception messages associated with Array datatype (#27769)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-10-19 17:24:07 +08:00