admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2025-12-06 17:18:35 +08:00
Code Issues Packages Projects Releases Wiki Activity
21,861 Commits 133 Branches 183 Tags
Commit Graph

80 Commits

Author SHA1 Message Date
SimFG
1263505808
feat: [2.5] support to deny dll according to database property (#40784) 
- issue: #40762
- pr: #40764

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2025-03-23 11:18:28 +08:00
SimFG
a3c60a03b5
feat: [2.5] add DDLDB rate type and related quota configurations (#40653) 
- issue: #40650
- pr: #40651

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2025-03-14 15:36:17 +08:00
congqixia
709594f158
enhance: [2.5] Use v2 package name for pkg module (#40117) 
Cherry-pick from master
pr: #39990
Related to #39095

https://go.dev/doc/modules/version-numbers

Update pkg version according to golang dep version convention

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-02-23 00:46:01 +08:00
yihao.dai
9b916f2cf8
enhance: [2.5] Skip creating partition rate limiters when not enable (#38911) 
issue: https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/38816

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-01-15 22:19:02 +08:00
Zhen Ye
95809ca767
enhance: make new go package to manage proto (#39128) 
issue: #39095
pr: #39114

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-01-10 10:53:01 +08:00
Xianhui Lin
d0a8110a7a
enhance: alterdatabase support delete property (#38380) 
alterdatabase support delete property
issue: https://github.com/milvus-io/milvus/issues/38379

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2024-12-12 17:04:43 +08:00
SimFG
1cc9cb49ad
enhance: allow to delete data when disk quota exhausted (#37134) 
- issue: #37133

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-10-25 16:47:29 +08:00
congqixia
30121a5a0d
fix: Rectify delete buffer row count quota value (#37060) 
Related to #37057

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-23 10:15:29 +08:00
XuanYang-cn
0516624ab0
enhance: Log when limit writing by deletion (#36954) 
See also: #36953

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-18 17:25:25 +08:00
yihao.dai
1bd3228635
enhance: Ignore db not found error in quota center (#36821) 
In quota center, ignore the "DB not found error" to prevent it from
affecting the rate limiting of other databases.

/kind improvement

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-15 15:55:22 +08:00
congqixia
116d7c18cc
fix: Use metrics values instead of hardcode mock values (#36782) 
Resolves #36781

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-11 20:09:20 +08:00
congqixia
8593c4580a
enhance: Add delete buffer related quota logic (#35918) 
See also #35303

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-05 11:39:03 +08:00
jaime
24fb10114b
enhance: remove cooling off in rate limiter for read requests (#35935) 
issue: #35934

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-09-04 14:39:10 +08:00
jaime
b0ac04d104
enhance: set database properties to restrict read access (#35745) 
issue: #35744

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-08-29 13:17:01 +08:00
yihao.dai
56b3490a98
fix: Fix rate wasn't limited to the expected value (#35699) 
Each time the rate is reset, the token bucket is fully refilled, causing
the rate wasn't limited to the expected value. This PR addresses the
issue by preventing the token reset.

issue: https://github.com/milvus-io/milvus/issues/35675,
https://github.com/milvus-io/milvus/issues/35702

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-28 14:13:00 +08:00
jaime
d8aa01bc1a
fix: inconsistent meta view causes rate limit invalid (#35665) 
issue: #35663

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-08-28 11:21:04 +08:00
Jiquan Long
b5022b0647
enhance: add metric to record disk quota (#35305) 
https://github.com/milvus-io/milvus/issues/35177

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-08-07 10:26:16 +08:00
Jiquan Long
4dbd023402
enhance: add metric to record maxInsertRate and queryNodeMemoryHighWaterLevel (#35185) 
fix: https://github.com/milvus-io/milvus/issues/35177

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-08-02 10:14:13 +08:00
Jiquan Long
b843c91bad
feat: record the number of times milvus turns into force-deny-writing states (#34960) 
/kind improvement
fix: #34961

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-07-25 11:39:46 +08:00
congqixia
67324eb809
enhance: Add l0 segment entry num quota (#34733) 
See also #34670

This PR add quota configuration for l0 segment entry number per
collection. If l0 compaction cannot keep up the insertion/upsertion
rate, this feature could back press the related rate.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-17 17:35:41 +08:00
jaime
60be454db0
enhance: add disk quota and max collections into db properties (#34368) 
issue: #34385

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-05 18:22:17 +08:00
jaime
0426390f06
enhance: improve check health (#33800) 
issue: #34264

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-01 10:16:06 +08:00
SimFG
44d7e03e56
fix: reset the RootCoordQuotaStates metric before recording this metric (#33553) 
- issue: #33539

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-06-03 21:09:47 +08:00
SimFG
1973ebbc42
fix: quotaCenter collect metrics failed because the wrong param (#33398) 
- issue: #33397

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-05-27 18:17:42 +08:00
SimFG
dd0c6d6980
fix: the panic when db isn't existed in the rate limit interceptor (#33244) 
issue: #33243

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-05-22 15:57:39 +08:00
SimFG
31a29a2451
enhance: avoid the repeated metric info in the proxy (#32380) 
issue: #30577

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-04-19 10:21:20 +08:00
Jiquan Long
7add7248e2
fix: remove useless log of quota center (#32307) 
/kind improvement
issue: #31272

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-04-17 16:17:19 +08:00
wei liu
0d849a6c0a
fix: fix collectionInfo leak in datacoord (#32175) 
issue: #32029

lack of logic to clean collection info in datacoord's meta, This PR
clean collection info after drop channel, to avoid collection info leak
in datacoord

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-04-15 16:33:19 +08:00
SimFG
c012e6786f
feat: support rate limiter based on db and partition levels (#31070) 
issue: https://github.com/milvus-io/milvus/issues/30577
co-author: @jaime0815

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-04-12 16:01:19 +08:00
wei liu
9111ecab82
enhance: Rootcoord's stop may block in quota_center's stop (#31447) 
this PR fixed that rootcoord's stop may block in quota_center' stop

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-04-01 21:09:18 +08:00
Jiquan Long
dc2cdbe387
enhance: add more metrics (#31271) 
/kind improvement
fix: #31272 

This pr add more metrics, which are:
- Slow query count, which the duration considered as slow can be
configurable;
- Number of deleted entities;
- Number of entities imported;
- Number of entities per collection;
- Number of loaded entities per collection;
- Number of indexed entities;
- Number of indexed entities, per collection, per index and whether it's
a vetor index;
- Quota states (LongTimeTickDelay, MemoryExhuasted, DiskQuotaExhuasted)
per database;

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-03-19 15:23:06 +08:00
congqixia
af315539d4
fix: Remove time tick delay metrics when nodes go offline (#30833) 
See also #30832

This PR removes time tick delay metrics when rootcoord GetMetrics
response does not have previously existed querynode/datanode

Also add unit tests for this case

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi.Xia <congqi.xia@zilliz.com>
 2024-02-28 10:10:56 +08:00
wei liu
e41fd6fbde
enhance: Move proxy client manager to util package (#28955) 
issue:  #28898

This PR move the `ProxyClientManager` to util package, in case of
reusing it's implementation in querycoord

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-12-20 19:22:42 +08:00
yihao.dai
c16fc854d1
Fix confusing quota log (#28262) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-11-13 15:38:18 +08:00
jaime
7f7c71ea7d
Decoupling client and server API in types interface (#27186) 
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-09-26 09:57:25 +08:00
SimFG
26f06dd732
Format the code (#27275) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-21 09:45:27 +08:00
congqixia
6ee5fedfef
Fix lock not applied in quota center (#27133) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-15 19:11:25 +08:00
yihao.dai
7624c2b949
Improve rated log in insertBufferNode (#26788) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-04 09:59:10 +08:00
yihao.dai
c6024a32f5
Add rate limit ratio monitoring (#26709) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-03 20:47:02 +08:00
yihao.dai
b6effd7345
Disable deny writing when the growing segment size exceeds the watermark (#26163) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-08-08 12:43:07 +08:00
yihao.dai
74ad7c4601
Fix monitoring metrics (#25549) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-07-16 18:48:33 +08:00
smellthemoon
d63323d117
Add rate limit and deny write in upsert (#25351) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-07-11 11:20:34 +08:00
jaime
18df2ba6fd
[Cherry-Pick] Support Database (#24769) 
Support Database(#23742)
Fix db nonexists error for FlushAll (#24222)
Fix check collection limits fails (#24235)
backward compatibility with empty DB name (#24317)
Fix GetFlushAllState with DB (#24347)
Remove db from global meta cache after drop database (#24474)
Fix db name is empty for describe collection response (#24603)
Add RBAC for Database API (#24653)
Fix miss load the same name collection during recover stage (#24941)

RBAC supports Database validation (#23609)
Fix to list grant with db return empty (#23922)
Optimize PrivilegeAll permission check (#23972)
Add the default db value for the rbac request (#24307)

Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-06-25 17:20:43 +08:00
wei liu
0d51f55c48
enable config each collection has different rate limit (#24139) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-06-21 15:02:42 +08:00
congqixia
41af0a98fa
Use go-api/v2 for milvus-proto (#24770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-09 01:28:37 +08:00
aoiasd
44e5daae3a
Fix querynode read quota (#24412) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-06-08 17:24:36 +08:00
yihao.dai
0fb9131973
Improve rate limiter log (#24459) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-05-30 11:23:28 +08:00
yihao.dai
1471da846d
Fix panic due to collection not exist in quota effect (#24320) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-05-24 20:53:30 +08:00
wei liu
6cfb4559d2
fix deny writing use wrong errcode (#24152) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-05-17 14:23:23 +08:00
yihao.dai
7384d83d2c
Support rate limit based on growing segments size (#24121) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-05-17 09:57:22 +08:00