Cai Yudong
26409d801e
enhance: Remove omp from segcore ( #29207 )
...
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-12-15 14:00:39 +08:00
cai.zhang
2f7252b44e
enhance: Set default index name as field name ( #29218 )
...
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-15 12:06:39 +08:00
congqixia
4731c1b0d5
enhance: make SyncManager pool size refreshable ( #29224 )
...
See also #29223
This PR make `conc.Pool` resizable by adding `Resize` method for it.
Also make newly added datanode `MaxParallelSyncMgrTasks` config
refreshable
---------
Signed-off-by: Congqi.Xia <congqi.xia@zilliz.com>
2023-12-15 09:58:43 +08:00
congqixia
25a4525297
enhance: Change sync manager parallel config item ( #29216 )
...
Since the sync manager is global in datanode now, the old
`maxParallelSyncTaskNum` does not fit into current implementation
anymore.
This PR add a new param item for sync mgr parallel control and enlarge
default value
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-14 20:46:41 +08:00
congqixia
8a63e53421
enhance: Add http method to control datacoord garbage collection ( #29052 )
...
See also #29051
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-14 19:26:39 +08:00
XuanYang-cn
cc727ace61
fix: Set compacted segments' level to level one ( #29190 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-14 18:46:40 +08:00
yah01
13beb5ccc0
fix: load gets stuck probably ( #29191 )
...
we found the load got stuck probably, and reviewed the logs.
the target observer seems not working, the reason is the taskDispatcher
removes the task in a goroutine, and modifies the task status after
committing the task into the goroutine pool, but this may happen after
the task removed, which leads to the task will never be removed
related #29086
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-14 18:28:38 +08:00
yah01
b8674811cf
fix: data race in ProxyClientManager ( #29206 )
...
this PR changed the ProxyClientManager to thread-safe
fix #29205
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-14 18:22:39 +08:00
sre-ci-robot
3e66e78508
[automated] Update Knowhere Commit ( #29178 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-12-14 17:16:39 +08:00
wei liu
008bae675d
enhance: Skip balance segment when channel need be balanced ( #29116 )
...
issue: #28622
After we support balance segment with growing segment count #28623 , if
we balance segment and channel at same time, some segments need to be
rebalanced after balance channel finish.
This PR skip balance segment when channel need be balanced.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-14 16:44:43 +08:00
SimFG
5a7fb97e07
fix: missing the grant info when using the SelectGrant api with the default db ( #29173 )
...
issue: #29172
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-12-14 16:20:43 +08:00
congqixia
264505b706
enhance: remove reset seed for every shuffle ( #29195 )
...
See also #29113
rand.Seed is deprecated and cost noticable CPU time during heavy payload
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-14 15:46:38 +08:00
wayblink
2274aa3b50
fix: bulkinsert binlog didn't consider ts order when processing delta data ( #29163 )
...
#29162
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-12-14 14:36:40 +08:00
yah01
58dbb7872a
fix: forbid balancing level zero segments ( #29168 )
...
related #29128
Signed-off-by: yah01 <yah2er0ne@outlook.com>
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-14 14:30:39 +08:00
congqixia
9407be6eb2
enhance: Add a config item for partition name as regexp feature and disable it by default ( #29154 )
...
See also #29177
Add a config item for partition name as regexp feature and disable it by
default
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-14 10:04:38 +08:00
aoiasd
b5ee563914
fix: accesslog can not print search expression ( #28899 )
...
relate: https://github.com/milvus-io/milvus/issues/28893
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-13 18:50:42 +08:00
XuanYang-cn
2c62bcbbf6
fix: Fix some left comments ( #29027 )
...
See also: #28814
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-13 18:48:39 +08:00
Bingyi Sun
ad866d2889
feat: integrate storagev2 into index build process ( #28995 )
...
issue: https://github.com/milvus-io/milvus/issues/28994
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-12-13 17:24:38 +08:00
wei liu
fe1eeae2aa
enhance: Use mockery to replace manual mock code ( #29074 )
...
issue: #29043
This PR remove mannul mock code for proxy and data coord
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-13 10:46:44 +08:00
congqixia
cb75e73c77
enhance: Use zap.Stringer for large log field ( #29143 )
...
See also #29113
Using zap.Stringer log field will evaluate log field value only when log
level meets the configuration, which could save some CPU time in search
route
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-12 22:52:39 +08:00
yah01
2f0c7a6544
fix: forbid balancing level zero segments ( #29130 )
...
we can't balance the L0 segments
related #29128
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-12 20:38:38 +08:00
PowderLi
cb5c06c718
fix: wait server to listen the http port ( #29075 )
...
issue: #29068
wait server to listen the http port
then check whether various urls can be accessed normally
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-12 16:22:50 +08:00
cai.zhang
49b8657f95
enhance: Support implicit type conversion for parquet ( #29046 )
...
issue: #29019
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-12 16:14:44 +08:00
Jiquan Long
e5f155612a
fix: txn control not applied to snapshot.Remove ( #29124 )
...
issue: #29123
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-12-12 14:48:38 +08:00
yah01
9819090247
enhance: add more logs for target updating ( #29090 )
...
- add more logs about the condition satisfying
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-12 14:06:43 +08:00
wayblink
51f870da7e
feat: Introduce channelCheckpointUpdater to reduce goroutine use in ttNode ( #28570 )
...
/kind enhancement
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-12-12 13:48:42 +08:00
wei liu
69a71e11d1
enhance: Add config for auto upgrade segment index ( #29112 )
...
This PR add config for auto upgrade segment index to index engine's
version.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-12 11:56:37 +08:00
Enwei Jiao
0e65e90338
enhance: Support otlp with insecure ( #29115 )
...
issue: https://github.com/milvus-io/milvus/issues/28914
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-12-12 11:14:37 +08:00
yah01
0a87724f18
enhance: remove merger for load segments ( #29062 )
...
remove merger as now QueryNode could load segments concurrently
fix #29063
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-12 10:48:37 +08:00
congqixia
d0bac9d0bb
enhance: Avoid initializing casbin enforcer for each request ( #29117 )
...
See also #29113
This patch:
- Replace plain Enforcer with `casbin.SyncedEnforcer`
- Add implementation of persist.Adapter with `MetaCacheCasbinAdapter`
- Invoke enforcer.LoadPolicy when policy updated
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-12 10:36:43 +08:00
wayblink
994b239161
fix: fix datarace in root_coord proxy_client_manager ( #28796 )
...
bug: #28797
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-12-12 10:18:39 +08:00
SimFG
ef18e6a532
enhance: Use a non-blocking method to trigger compaction when saveBinlogPath is executed ( #28941 )
...
/kind improvement
issue: #28924
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-12-12 10:16:39 +08:00
congqixia
a67fc08865
fix: balance_unstable_view unit test ( #29127 )
...
fix : #29126
Allow unstable output channel balance plan
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-12 10:02:39 +08:00
Xiaofan
9d54d6f590
fix: change Abseil to shared library to solve macos compilation issue ( #28986 )
...
fix the compilation error on macos 14.0 with x86 arch processor
related to #28985
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2023-12-11 21:54:37 +08:00
aoiasd
3c32ba2407
enhance: pack datacoord Cluster and SessionManager with interface and mock them ( #28869 )
...
relate: https://github.com/milvus-io/milvus/issues/28861
https://github.com/milvus-io/milvus/issues/28854
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-11 17:52:37 +08:00
congqixia
1fe5f12bd5
enhance: Add client connect wrapper to keep connection alive ( #29058 )
...
See also #29057
Add wrapper to maintain client&connection
When reset operation is needed, `Close` method shall wait until all
on-going request return
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-11 17:20:38 +08:00
wei liu
42e538b683
enhance: enable balance channel in querycoord ( #28469 )
...
issue: #23726
/kind improvement
1. enable auto balance channel between nodes in querycoord
2. make `genSegmentPlan` reuse the `AssignSegment` logic
3. make `genChannelPlan` reuse the `AssignChannel` logic
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-11 14:18:37 +08:00
cai.zhang
2b05460ef9
enhance: Make import-related error message clearer ( #28978 )
...
issue: #28976
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-08 10:12:38 +08:00
MrPresent-Han
464bc9e8f4
fix: fix reduce precision for search( #27325 ) ( #29031 )
...
related: #27325
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-08 10:04:37 +08:00
yah01
c4dda3c50f
fix: panic caused by type assert LocalSegment on Segment ( #29018 )
...
related #29017
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-08 01:50:38 +08:00
yah01
9b3e06ae86
enhance: add more metrics for level zero segments ( #29029 )
...
- Add SegmentNum metric for level zero segments
- Add level zero segments size metirc
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-07 14:34:35 +08:00
congqixia
dcb662d9ed
enhance: Refine C.NewSegment response and handle exception ( #28952 )
...
See also #28795
Orignal `C.NewSegment` may panic if some condition is not met, this pr
changes response struct to `CNewSegmentResult`, which contains
`C.CStatus` and may return catched exception
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-07 13:34:35 +08:00
congqixia
cb43647b9e
enhance: Log channel checkpoint source info in writebuffer ( #28993 )
...
See also #27675
Print channel checkpoint source with rated log will help debugging
system behavior
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-07 11:50:36 +08:00
wayblink
6736f65345
feat: skip some empty ttMsg in Datanode flowgraph ( #28756 )
...
/kind feature
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-12-07 01:00:37 +08:00
cai.zhang
fb089cda8b
enhance: Load raw data while scalar index doesn't have raw data ( #28888 )
...
issue: #28886
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-06 20:36:36 +08:00
congqixia
6a86ac0ac6
fix: Align minio object storage ut to new minio server behavior ( #29014 )
...
See also #29013
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-06 15:42:43 +08:00
shaoyue
4a067a4c8c
enhance: Add proxy.ginLogSkipPaths ( #28945 )
...
Fix #28944
/cc @xiaofan-luan
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2023-12-06 10:26:35 +08:00
XuanYang-cn
5bac7f7897
fix: Fix L0 compaction in datacoord ( #28814 )
...
See also: #27606
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-05 18:44:37 +08:00
congqixia
cb31016640
enhance: Write buffer time range when syncing logs ( #28970 )
...
Related to #27675
The timestamp from, to field is not field for new implementation of
writebuffer & sync manager
This pr fills these field for better log information
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-05 17:36:36 +08:00
yah01
fab52d167b
fix: may miss stream delta while loading ( #28871 )
...
we consume the delta data from the lastest channel checkpoint while
loading segment,
this works well without level 0 segments, but now it may lead to miss
some delta data,
so we have to consume from the current target's channel checkpoint
related: #27349
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-05 17:34:45 +08:00
