sparknack
935160840c
enhance: add a disk quota for the loaded binlog size to prevent load failure of querynode ( #44893 )
...
issue: #41435
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
2025-10-19 19:44:01 +08:00
PjJinchen
64633cc5b3
fix: Metrics with collectionName but no databaseName label are causing name conflicts and confusion ( #43277 ) ( #43808 )
...
issue: https://github.com/milvus-io/milvus/issues/43277
---------
Signed-off-by: PjJinchen <6268414+pj1987111@users.noreply.github.com>
2025-08-15 01:37:44 +08:00
Zhen Ye
5aa7a116d2
fix: change maxTimeTickDelay from 5m into 20m ( #43377 )
...
issue: #43266
Signed-off-by: chyezh <chyezh@outlook.com>
2025-07-18 11:29:42 +08:00
Zhen Ye
15a6631147
enhance: add quota limit based on sn consuming lag ( #43105 )
...
issue: #42995
- The consuming lag at streaming node will be reported to coordinator.
- The consuming lag will trigger the write limit and deny by quota
center.
- Set the ttProtection by default.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-07-11 14:10:49 +08:00
cai.zhang
5566a85bcc
enhance: Add proxy task queue metrics ( #42156 )
...
issue: #42155
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-06-04 11:26:32 +08:00
Xianhui Lin
2278e9e03f
fix: set quota center metrics configuration before watch ( #41706 )
...
fix: set quota center metrics configuration before watch
issue: https://github.com/milvus-io/milvus/issues/35177
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-05-09 14:32:54 +08:00
Xianhui Lin
f9febe3bae
enhance: Merge RootCoord, DataCoord And QueryCoord into MixCoord ( #41006 )
...
Merge RootCoord, DataCoord And QueryCoord into MixCoord
Make Session into one
issue : https://github.com/milvus-io/milvus/issues/37764
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-04-11 16:36:30 +08:00
SimFG
16efcda5c4
feat: support to deny dll according to database property ( #40764 )
...
- issue: #40762
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-03-23 11:18:28 +08:00
SimFG
bf4fc6a8c6
feat: add DDLDB rate type and related quota configurations ( #40651 )
...
- issue: #40650
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-03-14 15:38:09 +08:00
congqixia
cb7f2fa6fd
enhance: Use v2 package name for pkg module ( #39990 )
...
Related to #39095
https://go.dev/doc/modules/version-numbers
Update pkg version according to golang dep version convention
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-22 23:15:58 +08:00
yihao.dai
154edd237d
enhance: Skip creating partition rate limiters when not enable ( #38816 )
...
issue: https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-23 14:25:17 +08:00
Zhen Ye
bb8d1ab3bf
enhance: make new go package to manage proto ( #39114 )
...
issue: #39095
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:49:01 +08:00
Xianhui Lin
d0a8110a7a
enhance: alterdatabase support delete property ( #38380 )
...
alterdatabase support delete property
issue: https://github.com/milvus-io/milvus/issues/38379
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2024-12-12 17:04:43 +08:00
SimFG
1cc9cb49ad
enhance: allow to delete data when disk quota exhausted ( #37134 )
...
- issue: #37133
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-10-25 16:47:29 +08:00
congqixia
30121a5a0d
fix: Rectify delete buffer row count quota value ( #37060 )
...
Related to #37057
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-23 10:15:29 +08:00
XuanYang-cn
0516624ab0
enhance: Log when limit writing by deletion ( #36954 )
...
See also: #36953
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-18 17:25:25 +08:00
yihao.dai
1bd3228635
enhance: Ignore db not found error in quota center ( #36821 )
...
In quota center, ignore the "DB not found error" to prevent it from
affecting the rate limiting of other databases.
/kind improvement
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-15 15:55:22 +08:00
congqixia
116d7c18cc
fix: Use metrics values instead of hardcode mock values ( #36782 )
...
Resolves #36781
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-11 20:09:20 +08:00
congqixia
8593c4580a
enhance: Add delete buffer related quota logic ( #35918 )
...
See also #35303
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-05 11:39:03 +08:00
jaime
24fb10114b
enhance: remove cooling off in rate limiter for read requests ( #35935 )
...
issue: #35934
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-04 14:39:10 +08:00
jaime
b0ac04d104
enhance: set database properties to restrict read access ( #35745 )
...
issue: #35744
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-29 13:17:01 +08:00
yihao.dai
56b3490a98
fix: Fix rate wasn't limited to the expected value ( #35699 )
...
Each time the rate is reset, the token bucket is fully refilled, causing
the rate wasn't limited to the expected value. This PR addresses the
issue by preventing the token reset.
issue: https://github.com/milvus-io/milvus/issues/35675 ,
https://github.com/milvus-io/milvus/issues/35702
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-28 14:13:00 +08:00
jaime
d8aa01bc1a
fix: inconsistent meta view causes rate limit invalid ( #35665 )
...
issue: #35663
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-28 11:21:04 +08:00
Jiquan Long
b5022b0647
enhance: add metric to record disk quota ( #35305 )
...
https://github.com/milvus-io/milvus/issues/35177
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-07 10:26:16 +08:00
Jiquan Long
4dbd023402
enhance: add metric to record maxInsertRate and queryNodeMemoryHighWaterLevel ( #35185 )
...
fix: https://github.com/milvus-io/milvus/issues/35177
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-02 10:14:13 +08:00
Jiquan Long
b843c91bad
feat: record the number of times milvus turns into force-deny-writing states ( #34960 )
...
/kind improvement
fix : #34961
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-25 11:39:46 +08:00
congqixia
67324eb809
enhance: Add l0 segment entry num quota ( #34733 )
...
See also #34670
This PR add quota configuration for l0 segment entry number per
collection. If l0 compaction cannot keep up the insertion/upsertion
rate, this feature could back press the related rate.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-17 17:35:41 +08:00
jaime
60be454db0
enhance: add disk quota and max collections into db properties ( #34368 )
...
issue: #34385
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-05 18:22:17 +08:00
jaime
0426390f06
enhance: improve check health ( #33800 )
...
issue: #34264
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-01 10:16:06 +08:00
SimFG
44d7e03e56
fix: reset the RootCoordQuotaStates metric before recording this metric ( #33553 )
...
- issue: #33539
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-06-03 21:09:47 +08:00
SimFG
1973ebbc42
fix: quotaCenter collect metrics failed because the wrong param ( #33398 )
...
- issue: #33397
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-27 18:17:42 +08:00
SimFG
dd0c6d6980
fix: the panic when db isn't existed in the rate limit interceptor ( #33244 )
...
issue: #33243
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-22 15:57:39 +08:00
SimFG
31a29a2451
enhance: avoid the repeated metric info in the proxy ( #32380 )
...
issue: #30577
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-19 10:21:20 +08:00
Jiquan Long
7add7248e2
fix: remove useless log of quota center ( #32307 )
...
/kind improvement
issue: #31272
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-04-17 16:17:19 +08:00
wei liu
0d849a6c0a
fix: fix collectionInfo leak in datacoord ( #32175 )
...
issue: #32029
lack of logic to clean collection info in datacoord's meta, This PR
clean collection info after drop channel, to avoid collection info leak
in datacoord
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-15 16:33:19 +08:00
SimFG
c012e6786f
feat: support rate limiter based on db and partition levels ( #31070 )
...
issue: https://github.com/milvus-io/milvus/issues/30577
co-author: @jaime0815
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-04-12 16:01:19 +08:00
wei liu
9111ecab82
enhance: Rootcoord's stop may block in quota_center's stop ( #31447 )
...
this PR fixed that rootcoord's stop may block in quota_center' stop
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-01 21:09:18 +08:00
Jiquan Long
dc2cdbe387
enhance: add more metrics ( #31271 )
...
/kind improvement
fix : #31272
This pr add more metrics, which are:
- Slow query count, which the duration considered as slow can be
configurable;
- Number of deleted entities;
- Number of entities imported;
- Number of entities per collection;
- Number of loaded entities per collection;
- Number of indexed entities;
- Number of indexed entities, per collection, per index and whether it's
a vetor index;
- Quota states (LongTimeTickDelay, MemoryExhuasted, DiskQuotaExhuasted)
per database;
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-03-19 15:23:06 +08:00
congqixia
af315539d4
fix: Remove time tick delay metrics when nodes go offline ( #30833 )
...
See also #30832
This PR removes time tick delay metrics when rootcoord GetMetrics
response does not have previously existed querynode/datanode
Also add unit tests for this case
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi.Xia <congqi.xia@zilliz.com>
2024-02-28 10:10:56 +08:00
wei liu
e41fd6fbde
enhance: Move proxy client manager to util package ( #28955 )
...
issue: #28898
This PR move the `ProxyClientManager` to util package, in case of
reusing it's implementation in querycoord
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-20 19:22:42 +08:00
yihao.dai
c16fc854d1
Fix confusing quota log ( #28262 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-11-13 15:38:18 +08:00
jaime
7f7c71ea7d
Decoupling client and server API in types interface ( #27186 )
...
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-09-26 09:57:25 +08:00
SimFG
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
congqixia
6ee5fedfef
Fix lock not applied in quota center ( #27133 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-15 19:11:25 +08:00
yihao.dai
7624c2b949
Improve rated log in insertBufferNode ( #26788 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-04 09:59:10 +08:00
yihao.dai
c6024a32f5
Add rate limit ratio monitoring ( #26709 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-03 20:47:02 +08:00
yihao.dai
b6effd7345
Disable deny writing when the growing segment size exceeds the watermark ( #26163 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-08-08 12:43:07 +08:00
yihao.dai
74ad7c4601
Fix monitoring metrics ( #25549 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-07-16 18:48:33 +08:00
smellthemoon
d63323d117
Add rate limit and deny write in upsert ( #25351 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-07-11 11:20:34 +08:00
jaime
18df2ba6fd
[Cherry-Pick] Support Database ( #24769 )
...
Support Database(#23742 )
Fix db nonexists error for FlushAll (#24222 )
Fix check collection limits fails (#24235 )
backward compatibility with empty DB name (#24317 )
Fix GetFlushAllState with DB (#24347 )
Remove db from global meta cache after drop database (#24474 )
Fix db name is empty for describe collection response (#24603 )
Add RBAC for Database API (#24653 )
Fix miss load the same name collection during recover stage (#24941 )
RBAC supports Database validation (#23609 )
Fix to list grant with db return empty (#23922 )
Optimize PrivilegeAll permission check (#23972 )
Add the default db value for the rbac request (#24307 )
Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-06-25 17:20:43 +08:00
