admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-02 00:45:30 +08:00
Code Issues Packages Projects Releases Wiki Activity
17,978 Commits 138 Branches 188 Tags
Commit Graph

7456 Commits

Author SHA1 Message Date
congqixia
edde3cf1c7
Add tracer for querycoord tasks (#27058) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-14 09:59:19 +08:00
Jiquan Long
af5619e73e
Fix misusing chunk manager (#27051) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-13 18:11:20 +08:00
XuanYang-cn
ea2b909c6d
Fix historm metrics buckets range for index (#26976) 
See also: #26743

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-13 17:21:18 +08:00
yah01
3d05ddf505
Reduce cpp test time (#27043) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-13 15:41:18 +08:00
congqixia
f0d0651989
Do not reset connection immediately if grpc code is Canceled or DeadlineExceeded (#27014) 
We found lots of connection reset & canceled due to recent retry change
Current implementation resets connection no matter what the error code is
To sync behavior to previous retry, skip reset connection only if cancel error happens too much.

Also adds a config item for minResetInterval for grpc reset connection

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-13 15:01:18 +08:00
PowderLi
c033580af4
show index info while GetSegmentInfo (#26981) 
according to QueryNode::GetSegmentInfo

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-09-13 11:37:18 +08:00
congqixia
394c898b4c
Discard SyncDistribution set action from legacy querycoord (#27027) 
Since Milvus in lower version (< 2.3.0), there is no load info in set action
which may corrupt data integrity and cause panicking

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-13 09:35:18 +08:00
yihao.dai
dd2cb1d44a
Flush by flush channels (#26859) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-12 21:07:19 +08:00
cqy123456
0ff4ddc76c
remove VectorMemNMIndex (#27000) 
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-09-12 17:13:18 +08:00
yah01
00c65fa0d7
Refine QueryNode errors (#27013) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-12 16:07:18 +08:00
yiwangdr
f85af0732c
remove MultiRemoveWithPrefix (#26924) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-09-12 15:07:17 +08:00
yah01
45d9fb5929
Fix wrong error codes & names (#26904) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-12 14:41:18 +08:00
Jiquan Long
802a6a86c0
Fix printing too many warnings (#27015) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-12 13:39:17 +08:00
aoiasd
e107d0794c
support complex delete expression (#25752) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-09-12 10:19:17 +08:00
sre-ci-robot
dde3cd2f93
Update knowhere commit (#26998) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-09-12 09:49:18 +08:00
congqixia
ac45af585b
Make segment loaded successful put in manager even ctx done (#26992) 
Leave segment loaded in manager even wait other segment failed
See also #26908
Fix error case in distributed scenario

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-11 21:33:18 +08:00
yah01
3203ce1654
Reduce copy while retrieving primary keys (#26616) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-09-11 21:31:18 +08:00
Enwei Jiao
c3f15c6b95
Refactor duplicate error class into one place (#26985) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-11 20:43:17 +08:00
congqixia
adfb5298c6
Refine TestSessionProcessActiveStandBy unit test logic (#26980) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-11 18:13:17 +08:00
congqixia
c45c32fad4
Set task reason for collection released (#26962) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-10 15:15:17 +08:00
MrPresent-Han
2101f2d289
fix unstable checker id due to go map iteration(#26943) (#26944) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-10 10:11:16 +08:00
congqixia
2a5d574a0d
Fix querynodev2 concurrent load logic (#26959) 
Fix logic error from #26926
function `waitSegmentLoadDone` shall return error when context is done

Make delegator control concurrency for each same segment
Related to #26908

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-10 07:41:18 +08:00
Futao Wei
ad1cc00a18
Fix typo in comment (#26951) 
Signed-off-by: unfode <forrest.futao.wei@gmail.com>
 2023-09-09 20:13:16 +08:00
congqixia
c6116d1819
Remove segment to LocalSegment type assertion (#26931) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-09 10:35:16 +08:00
congqixia
758aad705d
Fix checker using default interval after manual check (#26953) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-09 08:29:16 +08:00
Jiquan Long
9dae896395
Fix indexnode log (#26927) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-08 16:57:16 +08:00
congqixia
c8f9f22c4a
Fix segment loader return false success (#26926) 
`waitSegmentLoadDone` did not check waitCh result is success or failure
after load return without error, delegator will assume all segments are loaded

This PR changes waitCh to loadResult with `sync.Cond` with `atomic.Int32` to represent status

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-08 16:41:16 +08:00
Xu Tong
9166011c4a
Add float16 vector (#25852) 
Signed-off-by: Writer-X <1256866856@qq.com>
 2023-09-08 10:03:16 +08:00
XuanYang-cn
7f1ae35e72
Add timeout in dispatcher, AsConsumer and Seek (#26686) 
See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-08 09:51:17 +08:00
SimFG
0901b76732
Avoid the panic when the status of rpc response is nil (#26910) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-07 19:23:15 +08:00
congqixia
af5c01082b
Refine delegator lifetime control (#26881) 
- Add SafeChan interface in lifetime package
- Embed SafeChan into  interface
- Replace private lifetime struct in delegator package with
- Refine delegator on-going task lifetime control and wait all accepted task done
- Fix potential goroutine leakage from  if delegator closed concurrently

/kind improvement

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-07 10:11:15 +08:00
sre-ci-robot
56a6559fe7
Update knowhere commit (#26888) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-09-07 09:19:16 +08:00
yiwangdr
337edc321b
tikv integration (#26246) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-09-07 07:25:14 +08:00
cai.zhang
8b5b137207
Should set index task retry not failed when task canceled (#26878) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-09-06 19:01:14 +08:00
yah01
4a2377e669
Fix retrieving may return more result than limit (#26670) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-06 17:47:15 +08:00
SimFG
28681276e2
Improve the retry of the rpc client (#26795) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-06 17:43:14 +08:00
Jiquan Long
61c7b0990d
Workaround fix ChannelManager holding mutex too long (#26870) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-06 14:29:40 +08:00
zhagnlu
9b3e744b24
Fix embedded milvus log bug (#26864) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-09-06 14:27:14 +08:00
MrPresent-Han
528948559f
fix false load failure for long unserviable period(#26813) (#26818) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-06 12:57:15 +08:00
wei liu
0e2085b77f
fix dc standby to active (#26810) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-06 10:41:49 +08:00
wayblink
d7b6f3a9e8
Enhance FlushAll/GetFlushAllState API (#26802) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-09-06 10:35:48 +08:00
sre-ci-robot
78a2638fd4
Update knowhere commit (#26861) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-09-06 09:21:49 +08:00
bjzhjing
548c82eca5
Refactor storage.MergeInsertData() to optimize the merging process (#26839) 
Benchmark Milvus with https://github.com/qdrant/vector-db-benchmark and
specify the datasets as 'deep-image-96-angular'. Meanwhile, do perf
profiling during 'upload + index' stage of vector-db-benchmark and see
the following hot spots.

39.59%--github.com/milvus-io/milvus/internal/storage.MergeInsertData
        |
        |--21.43%--github.com/milvus-io/milvus/internal/storage.MergeFieldData
        |          |
        |          |--17.22%--runtime.memmove
        |                     |
        |                     |--1.53%--asm_exc_page_fault
        |                     ......
        |
        |--18.16%--runtime.memmove
                   |
                   |--1.66%--asm_exc_page_fault
                   ......

The hot code path is in storage.MergeInsertData() which updates
buffer.buffer by creating a new 'InsertData' instance and merging both
the old buffer.buffer and addedBuffer into it. When it calls golang
runtime.memmove to move buffer.buffer which is with big size (>1M), the
hot spots appear.

To avoid the above overhead, update storage.MergeInsertData() by
appending addedBuffer to buffer.buffer, instead of moving buffer.buffer
and addedBuffer to a new 'InsertData'. This change removes the hot spots
'runtime.memmove' from perf profiling output. Additionally, the 'upload
+ index' time, which is one performance metric of vector-db-benchmark,
is reduced around 60% with this change.

Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
 2023-09-05 21:41:48 +08:00
Enwei Jiao
73f4347ac6
Fix code checker error (#26844) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-05 15:51:48 +08:00
congqixia
fe7f7ea237
Add cpp-build and setenv for querynode mockery regen command (#26843) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-05 14:13:49 +08:00
Enwei Jiao
fb0705df1b
Decouple basetable and componentparam (#26725) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-05 10:31:48 +08:00
congqixia
4b58c71908
Add ctx parameter for organizeTask and GetWorker method (#26835) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-05 10:05:48 +08:00
sre-ci-robot
c132c53b1a
Update knowhere commit (#26840) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-09-05 09:05:49 +08:00
congqixia
1a8cf5c415
Organize all mockery generation commands in Makefile (#26826) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-04 21:19:48 +08:00
wei liu
1097776477
stop heartbeat if reach heartbeat limit (#26728) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-04 17:51:48 +08:00