admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-05 10:22:41 +08:00
Code Issues Packages Projects Releases Wiki Activity
17,966 Commits 136 Branches 188 Tags
Commit Graph

7451 Commits

Author SHA1 Message Date
PowderLi
c033580af4
show index info while GetSegmentInfo (#26981) 
according to QueryNode::GetSegmentInfo

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-09-13 11:37:18 +08:00
congqixia
394c898b4c
Discard SyncDistribution set action from legacy querycoord (#27027) 
Since Milvus in lower version (< 2.3.0), there is no load info in set action
which may corrupt data integrity and cause panicking

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-13 09:35:18 +08:00
yihao.dai
dd2cb1d44a
Flush by flush channels (#26859) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-12 21:07:19 +08:00
cqy123456
0ff4ddc76c
remove VectorMemNMIndex (#27000) 
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-09-12 17:13:18 +08:00
yah01
00c65fa0d7
Refine QueryNode errors (#27013) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-12 16:07:18 +08:00
yiwangdr
f85af0732c
remove MultiRemoveWithPrefix (#26924) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-09-12 15:07:17 +08:00
yah01
45d9fb5929
Fix wrong error codes & names (#26904) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-12 14:41:18 +08:00
Jiquan Long
802a6a86c0
Fix printing too many warnings (#27015) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-12 13:39:17 +08:00
aoiasd
e107d0794c
support complex delete expression (#25752) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-09-12 10:19:17 +08:00
sre-ci-robot
dde3cd2f93
Update knowhere commit (#26998) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-09-12 09:49:18 +08:00
congqixia
ac45af585b
Make segment loaded successful put in manager even ctx done (#26992) 
Leave segment loaded in manager even wait other segment failed
See also #26908
Fix error case in distributed scenario

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-11 21:33:18 +08:00
yah01
3203ce1654
Reduce copy while retrieving primary keys (#26616) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-09-11 21:31:18 +08:00
Enwei Jiao
c3f15c6b95
Refactor duplicate error class into one place (#26985) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-11 20:43:17 +08:00
congqixia
adfb5298c6
Refine TestSessionProcessActiveStandBy unit test logic (#26980) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-11 18:13:17 +08:00
congqixia
c45c32fad4
Set task reason for collection released (#26962) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-10 15:15:17 +08:00
MrPresent-Han
2101f2d289
fix unstable checker id due to go map iteration(#26943) (#26944) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-10 10:11:16 +08:00
congqixia
2a5d574a0d
Fix querynodev2 concurrent load logic (#26959) 
Fix logic error from #26926
function `waitSegmentLoadDone` shall return error when context is done

Make delegator control concurrency for each same segment
Related to #26908

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-10 07:41:18 +08:00
Futao Wei
ad1cc00a18
Fix typo in comment (#26951) 
Signed-off-by: unfode <forrest.futao.wei@gmail.com>
 2023-09-09 20:13:16 +08:00
congqixia
c6116d1819
Remove segment to LocalSegment type assertion (#26931) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-09 10:35:16 +08:00
congqixia
758aad705d
Fix checker using default interval after manual check (#26953) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-09 08:29:16 +08:00
Jiquan Long
9dae896395
Fix indexnode log (#26927) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-08 16:57:16 +08:00
congqixia
c8f9f22c4a
Fix segment loader return false success (#26926) 
`waitSegmentLoadDone` did not check waitCh result is success or failure
after load return without error, delegator will assume all segments are loaded

This PR changes waitCh to loadResult with `sync.Cond` with `atomic.Int32` to represent status

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-08 16:41:16 +08:00
Xu Tong
9166011c4a
Add float16 vector (#25852) 
Signed-off-by: Writer-X <1256866856@qq.com>
 2023-09-08 10:03:16 +08:00
XuanYang-cn
7f1ae35e72
Add timeout in dispatcher, AsConsumer and Seek (#26686) 
See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-08 09:51:17 +08:00
SimFG
0901b76732
Avoid the panic when the status of rpc response is nil (#26910) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-07 19:23:15 +08:00
congqixia
af5c01082b
Refine delegator lifetime control (#26881) 
- Add SafeChan interface in lifetime package
- Embed SafeChan into  interface
- Replace private lifetime struct in delegator package with
- Refine delegator on-going task lifetime control and wait all accepted task done
- Fix potential goroutine leakage from  if delegator closed concurrently

/kind improvement

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-07 10:11:15 +08:00
sre-ci-robot
56a6559fe7
Update knowhere commit (#26888) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-09-07 09:19:16 +08:00
yiwangdr
337edc321b
tikv integration (#26246) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-09-07 07:25:14 +08:00
cai.zhang
8b5b137207
Should set index task retry not failed when task canceled (#26878) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-09-06 19:01:14 +08:00
yah01
4a2377e669
Fix retrieving may return more result than limit (#26670) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-06 17:47:15 +08:00
SimFG
28681276e2
Improve the retry of the rpc client (#26795) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-06 17:43:14 +08:00
Jiquan Long
61c7b0990d
Workaround fix ChannelManager holding mutex too long (#26870) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-06 14:29:40 +08:00
zhagnlu
9b3e744b24
Fix embedded milvus log bug (#26864) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-09-06 14:27:14 +08:00
MrPresent-Han
528948559f
fix false load failure for long unserviable period(#26813) (#26818) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-06 12:57:15 +08:00
wei liu
0e2085b77f
fix dc standby to active (#26810) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-06 10:41:49 +08:00
wayblink
d7b6f3a9e8
Enhance FlushAll/GetFlushAllState API (#26802) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-09-06 10:35:48 +08:00
sre-ci-robot
78a2638fd4
Update knowhere commit (#26861) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-09-06 09:21:49 +08:00
bjzhjing
548c82eca5
Refactor storage.MergeInsertData() to optimize the merging process (#26839) 
Benchmark Milvus with https://github.com/qdrant/vector-db-benchmark and
specify the datasets as 'deep-image-96-angular'. Meanwhile, do perf
profiling during 'upload + index' stage of vector-db-benchmark and see
the following hot spots.

39.59%--github.com/milvus-io/milvus/internal/storage.MergeInsertData
        |
        |--21.43%--github.com/milvus-io/milvus/internal/storage.MergeFieldData
        |          |
        |          |--17.22%--runtime.memmove
        |                     |
        |                     |--1.53%--asm_exc_page_fault
        |                     ......
        |
        |--18.16%--runtime.memmove
                   |
                   |--1.66%--asm_exc_page_fault
                   ......

The hot code path is in storage.MergeInsertData() which updates
buffer.buffer by creating a new 'InsertData' instance and merging both
the old buffer.buffer and addedBuffer into it. When it calls golang
runtime.memmove to move buffer.buffer which is with big size (>1M), the
hot spots appear.

To avoid the above overhead, update storage.MergeInsertData() by
appending addedBuffer to buffer.buffer, instead of moving buffer.buffer
and addedBuffer to a new 'InsertData'. This change removes the hot spots
'runtime.memmove' from perf profiling output. Additionally, the 'upload
+ index' time, which is one performance metric of vector-db-benchmark,
is reduced around 60% with this change.

Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
 2023-09-05 21:41:48 +08:00
Enwei Jiao
73f4347ac6
Fix code checker error (#26844) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-05 15:51:48 +08:00
congqixia
fe7f7ea237
Add cpp-build and setenv for querynode mockery regen command (#26843) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-05 14:13:49 +08:00
Enwei Jiao
fb0705df1b
Decouple basetable and componentparam (#26725) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-05 10:31:48 +08:00
congqixia
4b58c71908
Add ctx parameter for organizeTask and GetWorker method (#26835) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-05 10:05:48 +08:00
sre-ci-robot
c132c53b1a
Update knowhere commit (#26840) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-09-05 09:05:49 +08:00
congqixia
1a8cf5c415
Organize all mockery generation commands in Makefile (#26826) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-04 21:19:48 +08:00
wei liu
1097776477
stop heartbeat if reach heartbeat limit (#26728) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-04 17:51:48 +08:00
cai.zhang
622077f9ad
Refine error message for json length exceed max length (#26676) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-09-04 17:47:48 +08:00
XuanYang-cn
b2e7cbdf4b
Remove TimeTravel in compactor (#26785) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-04 17:41:48 +08:00
cai.zhang
d7cd1f2a6d
Fix bug for get index state with compaction segment (#26822) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-09-04 17:29:48 +08:00
yihao.dai
4340cbfba2
Merge syncCPLagTooBehind policy into syncPeriodically policy (#26713) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-04 14:37:47 +08:00
chyezh
0530fd80c9
[Fixup] remove nats from default (#26791) 
Signed-off-by: chyezh <ye.zhen@zilliz.com>
 2023-09-04 10:01:04 +08:00