yihao.dai
c02fb64ad6
enhance: Allows proactive warming up of chunk cache ( #30182 )
...
Allows proactive warming up of chunk cache. Original vector data will be
asynchronously loaded into the chunk cache during the load process. It
has the potential to significantly reduce query/search latency for a
certain duration after the load, albeit with a concurrent increase in
disk usage.
issue: https://github.com/milvus-io/milvus/issues/30181
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-25 19:55:39 +08:00
Patrick Weizhi Xu
0907d76253
enhance: pass partition key scalar info if enabled when build vector index ( #29931 )
...
issue: #29892
Pass optional scalar IVF offsets to Cardinal
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-01-24 00:04:55 +08:00
congqixia
7cb6bebd96
enhance: replace magic number with ParamItem for dist handler ( #30020 )
...
See also #28817
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-16 17:33:03 +08:00
MrPresent-Han
2a0eb1d2e6
feat: support general capacity restrict for cloud-side resoure contro… ( #29845 )
...
related: #29844
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-16 16:32:53 +08:00
yah01
cda44c09f9
enhance: adjust the GPU pool size ( #29937 )
...
according to benchmark, the GPU pool size with 6 performs best
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-16 11:54:52 +08:00
Bingyi Sun
e1258b8cad
feat: integrate storagev2 into loading segment ( #29336 )
...
issue: #29335
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-01-12 18:10:51 +08:00
yah01
9e0163e12f
enhance: use GPU pool for gpu tasks ( #29678 )
...
- this much improve the performance for GPU index
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-04 17:50:46 +08:00
congqixia
da7c3cbd88
enhance: make delegator delete buffer holding all delete from cp ( #29626 )
...
See also #29625
This PR:
- Add a new implemention of `DeleteBuffer`: listDeleteBuffer
- holds cacheBlock slice
- `Put` method append new delete data into last block
- when a block is full, append a new block into the list
- Add `TryDiscard` method for `DeleteBuffer` interface
- For doubleCacheBuffer, do nothing
- For listDeleteBuffer, try to evict "old" blocks, which are blocks
before the first block whose start ts is behind provided ts
- Add checkpoint field for `UpdateVersion` sync action, which shall be
used to discard old cache delete block
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-04 17:02:46 +08:00
SimFG
d23f87a393
enhance: Add concurrency for datacoord segment GC ( #29561 )
...
issue: https://github.com/milvus-io/milvus/issues/29553
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-01-03 13:16:57 +08:00
MrPresent-Han
ed644983e2
enhance: add param for bloomfilter( #29388 ) ( #29490 )
...
related: #29388
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-28 18:10:46 +08:00
xige-16
0a70e8b601
enhance: Remove multiple vector field limit ( #27827 )
...
issue: https://github.com/milvus-io/milvus/issues/25639
/kind improvement
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-12-28 16:40:46 +08:00
wei liu
839a72129e
fix: Auto balance param can't be updated by dynamic ( #29501 )
...
This PR fixed that auto balance param can't be updated by dynamic
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-27 14:30:53 +08:00
MrPresent-Han
7c7003bff6
enhance:refine the range of chunk size config value( #29388 ) ( #29389 )
...
related: #29388
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-26 17:36:46 +08:00
cqy123456
4c979538a4
enhance: update cagra index params in config and add params check ( #29045 )
...
issue:https://github.com/milvus-io/milvus/issues/29230
this pr do two things about cagra index:
a.milvus yaml config support gpu memory settings
b.add cagra-params check
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
Co-authored-by: yusheng.ma <yusheng.ma@zilliz.com>
2023-12-26 11:04:47 +08:00
aoiasd
ac23e67f07
enhance: add default file path for access log ( #29460 )
...
relate:https://github.com/milvus-io/milvus/issues/29459
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-26 10:38:45 +08:00
aoiasd
46a937ac43
fix: msgstream memory leak caused by config event don't deregister ( #29266 )
...
relate: https://github.com/milvus-io/milvus/issues/28620
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-20 20:32:44 +08:00
XuanYang-cn
7d32861d53
enhance: Add channel check interval params ( #29311 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-20 15:06:42 +08:00
PowderLi
bcd6865b29
enhance: add 3 builtin roles ( #28961 )
...
issue: #28960 [milvus-proto
#212 ](https://github.com/milvus-io/milvus-proto/issues/212 )
add new configuration: builtinRoles
user can define roles in config file: `milvus.yaml`
there is an example:
1. db_ro, only have read privileges, include load
2. db_rw, read and write privileges, include create/drop/rename
collection
3. db_admin, not only read and write privileges, but also user
administration
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-18 14:28:41 +08:00
zhagnlu
a602171d06
enhance: Refactor runtime and expr framework ( #28166 )
...
#28165
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-18 12:04:42 +08:00
wei liu
820ee692fc
enhance: Add config for querycoord auto balance channel ( #29231 )
...
issue: #23726
This PR add control config to querycoord's background auto balance
channel operation
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-18 10:00:40 +08:00
congqixia
88b4b8b77c
enhance: make segments SQPool & LoadPool resizable ( #29239 )
...
See also #29223
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-15 20:22:39 +08:00
congqixia
4731c1b0d5
enhance: make SyncManager pool size refreshable ( #29224 )
...
See also #29223
This PR make `conc.Pool` resizable by adding `Resize` method for it.
Also make newly added datanode `MaxParallelSyncMgrTasks` config
refreshable
---------
Signed-off-by: Congqi.Xia <congqi.xia@zilliz.com>
2023-12-15 09:58:43 +08:00
congqixia
25a4525297
enhance: Change sync manager parallel config item ( #29216 )
...
Since the sync manager is global in datanode now, the old
`maxParallelSyncTaskNum` does not fit into current implementation
anymore.
This PR add a new param item for sync mgr parallel control and enlarge
default value
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-14 20:46:41 +08:00
congqixia
9407be6eb2
enhance: Add a config item for partition name as regexp feature and disable it by default ( #29154 )
...
See also #29177
Add a config item for partition name as regexp feature and disable it by
default
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-14 10:04:38 +08:00
aoiasd
b5ee563914
fix: accesslog can not print search expression ( #28899 )
...
relate: https://github.com/milvus-io/milvus/issues/28893
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-13 18:50:42 +08:00
wayblink
51f870da7e
feat: Introduce channelCheckpointUpdater to reduce goroutine use in ttNode ( #28570 )
...
/kind enhancement
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-12-12 13:48:42 +08:00
wei liu
69a71e11d1
enhance: Add config for auto upgrade segment index ( #29112 )
...
This PR add config for auto upgrade segment index to index engine's
version.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-12 11:56:37 +08:00
Enwei Jiao
0e65e90338
enhance: Support otlp with insecure ( #29115 )
...
issue: https://github.com/milvus-io/milvus/issues/28914
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-12-12 11:14:37 +08:00
yah01
0a87724f18
enhance: remove merger for load segments ( #29062 )
...
remove merger as now QueryNode could load segments concurrently
fix #29063
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-12 10:48:37 +08:00
cai.zhang
2b05460ef9
enhance: Make import-related error message clearer ( #28978 )
...
issue: #28976
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-08 10:12:38 +08:00
wayblink
6736f65345
feat: skip some empty ttMsg in Datanode flowgraph ( #28756 )
...
/kind feature
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-12-07 01:00:37 +08:00
shaoyue
4a067a4c8c
enhance: Add proxy.ginLogSkipPaths ( #28945 )
...
Fix #28944
/cc @xiaofan-luan
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2023-12-06 10:26:35 +08:00
shaoyue
8b2b0d412c
enhance: storeageType default value change to remote ( #28792 )
...
/kind enhancement
/cc @PowderLi
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2023-11-30 11:34:27 +08:00
congqixia
f9bb8e9648
enhance: Change const magic number in querycoord to param ( #28819 )
...
See also #28817
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-30 09:06:28 +08:00
cqy123456
3b1b14dd78
fix: update binlog index memory uasge before loading segments ( #28528 )
...
issue: #27678
when interimIndex = true, memory predict should be update with the
memory usage of binlog index build process.
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-11-29 16:42:27 +08:00
XuanYang-cn
606ec77b66
enhance: Unify levelzero segment config in DN ( #28720 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-28 18:04:26 +08:00
aoiasd
89d8ce2f73
enhance: refine access log to support format access log by yaml and print name info. ( #28319 )
...
relate: https://github.com/milvus-io/milvus/issues/28086
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-28 15:32:31 +08:00
Bingyi Sun
2843664a20
fix: channel name can not contains '-partition-' ( #28689 )
...
issue: https://github.com/milvus-io/milvus/issues/28675
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-24 10:14:24 +08:00
wei liu
4b8d3eaa41
enhance: Increase heartbeat timeout ( #28648 )
...
it's easy to trigger heartbeat timeout after 100ms when standalone cpu
usage reach 100%.
This PR increase the heartbeat timeout param to 2000ms
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-23 19:50:24 +08:00
Bingyi Sun
4fedff6d47
feat: integrate storage v2 into the write path ( #28440 )
...
#28378
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-23 17:26:24 +08:00
XuanYang-cn
e88bbaac24
feat: Add universal levelzero segment switch ( #28483 )
...
See also: #27349
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-20 19:04:22 +08:00
SimFG
cfb6edea61
Support to trace the grpc request ( #28349 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-11-13 20:16:18 +08:00
XuanYang-cn
f8aa46419a
Add LevelZeroCompaction configs ( #28190 )
...
See also: #27606
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-13 11:18:19 +08:00
SimFG
7dda2e8814
Change some log level in the pkg package ( #28181 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-11-08 23:34:22 +08:00
zhenshan.cao
2057fdce43
Change default storage type to minio ( #28236 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-11-07 21:15:31 +08:00
wei liu
5b45a138b1
disable auto balance when old node exists ( #28191 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-07 14:02:20 +08:00
Xiaofan
da19e49daf
Support purge old session for standalone ( #28184 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-11-06 21:21:42 +08:00
yah01
90e2c63d9e
Fix getting incorrect CPU num ( #28146 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-11-06 06:02:16 +08:00
congqixia
bf2f62c1e7
Add WriteBuffer to provide abstraction for delta policy ( #27874 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-04 12:10:17 +08:00
aoiasd
d2727cc0ff
Add write cache for access log ( #27792 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-03 10:28:17 +08:00
