admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-02 17:05:33 +08:00
Code Issues Packages Projects Releases Wiki Activity
7,165 Commits 138 Branches 188 Tags
Commit Graph

182 Commits

Author SHA1 Message Date
neza2017
b2908c4780
Release collection (#5839) 
* relese collection

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* gen-proto

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* release collection

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* release collection

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-17 17:45:56 +08:00
Cai Yudong
ae3daff5e4
Rename Master to RootCoord (#5830) 
* rename master_service.go to root_coordinator.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename service to coordinator under cmd

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename service to coord under cmd

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename service to coord for metrics

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename service to coord for masterservice

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix metrics unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* roll back cmd

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-17 16:47:57 +08:00
congqixia
5ce06de3d1
Add context usage and fix defer issue (#5796) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-16 18:35:59 +08:00
sunby
d997dd20af
Modify GetRecoveryInfo logic (#5785) 
To satisfy QueryNode's request, GetRecoveryInfo interface should return
unflushed segments with start position. Because of using the same code
for getting seek position in QueryNode and DataNode before, we add a
flag to differentiate.

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 19:23:55 +08:00
sunby
9246c663fc Add start position in segment (#5753) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 16:39:01 +08:00
Cai Yudong
0f4bd50ce3 Support ETCD endpoints array (#5755) 2021-06-15 16:36:36 +08:00
sunby
ba9952e780 Fix unit test in dataservice (#5745) 
After a unit test, the msg stream will be closed. But in pulsar go
client, the connection between client and server will be reconnected.
Then we rebuild the msgstream immediately and produce a message, this
message will be sent to the previous connection which cause timeout of
test. We append the channelName wil a random value.

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 16:36:36 +08:00
congqixia
7d9f299ff6 Change default unregister policy (#5710) 
* Change default unassign policy

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix gofmt

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:06:13 +08:00
congqixia
a3788a99c5 Fix unregister node info has no channel (#5699) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:06:13 +08:00
congqixia
a3679d5540 Add vchannel buffer for cluster (#5691) 
* Add channel buffer for cluster

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Change default register policy

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:06:13 +08:00
XuanYang-cn
46699c4c2b fix datanode dropcollection error (#5690) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 16:06:13 +08:00
congqixia
c172af78bd Add dataservice state check (#5687) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:06:13 +08:00
sunby
189ac881f3 Fix bugs (#5676) 
* Remove redundant session startup

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Register datanode after start success

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix meta snap shot

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix datanode message stream channel

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix bugs when drop empty collection

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Fix bug of getting pchan statistics from task scheduler

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix i/dist/dataservice test code

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix epoch lifetime not applied

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* fix datanode flowgraph dd node

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix handle datanode timetick bug

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove repack function of dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix proxynode

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Apply extended seal policy

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add check for time tick

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix check

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix the repack function of dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix the bug when send statistics of pchan

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix the repack function when craete dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix bugs

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix describe collection

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix bug when send timestamp statistics

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix data node

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Add length check before flush request

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add log for data node

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix SaveBinlog bugs

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Add more log in datanode

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Put SegmentState.Flushing as the last one in enum to fit the client

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Fix params in GetInsertBinlogPaths

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Rename policy

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove unused ddl functions and fields

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Remove pchan when drop collection

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Add balanced assignment policy

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix master ut

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Add lock in session manager

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add log for debug

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix some logic bug and typo

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix recover bugs

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Get collection scheme of a specific timestamp

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Change CheckPoint to SegmentInfo in VchannelInfo

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Recover Unflushed segment numOfRows

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix dataservice unit tests

Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:06:11 +08:00
sunby
e9b282c0be Use ElementsMatch to test slice equality (#5641) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 16:04:48 +08:00
congqixia
e57e2f77de Add flush monitor and unit test (#5622) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:04:48 +08:00
sunby
f054fc9be2 Add GetRecoveryInfo in dataservice (#5629) 
When loading a partition, QueryNode fetches the binlogs need to be
loaded and the channels needed to be watched by GetRecoveryInfo

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 16:04:48 +08:00
sunby
3a5c8c4d3a Add seek position to WatchDmChannels response (#5601) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 16:04:48 +08:00
Cai Yudong
a948b71e06 Support to send dd msg to all dml channels (#5597) 
* send dd msg to dml channel

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add dml_channels.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix race

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-15 16:04:48 +08:00
sunby
e7521afed5 Change SaveBinlogPath (#5576) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 16:04:48 +08:00
congqixia
d0d845805b Change WatchDmChannelsRequest proto (#5577) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:04:48 +08:00
sunby
aa8a038305 Rename SegmentAllocator to SegmentManager (#5559) 
Add numRows to segmentStatus and Rename SegmentAllocator to
SegmentManager. Remove SegmentAllocStats.

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 16:04:48 +08:00
godchen
275881dbdb Remove address param (#5556) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-15 16:04:46 +08:00
congqixia
41794ec3a6 Change SaveBinLogPath proto & func (#5551) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby
f1ccbb8f9a Fix init session in dataservice (#5522) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
neza2017
5966076880 fix data service (#5535) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 15:59:04 +08:00
congqixia
62eaa0390a Add dataservice metrics with grpc prometheus middleware (#5523) 
* Add dataservice metrics with grpc prometheus middleware

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* register server metrics

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby
38b1f7dabe Change the logic of getting vchannel position (#5502) 
Datanode send `SaveBinlogPath` request after every segment flush finish
with the binlog paths and dml/ddl position. But the flush of segments is
not sorted. So we sort the segments according to segment id and find the
largest segment id with not nil dml position which is the position of
the msgstream to recover.

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby
9a54f1f72d Watch channel if needed when allocating segmnets (#5482) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
congqixia
a2dd16472d Implement random reassign and add test case (#5474) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby
43534ef63f Add unit tests in dataservice (#5478) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby
a14c35274f Fix bugs in dataservice (#5457) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
XuanYang-cn
55eac3ee7f Complete Flush procedure in DataNode (#5454) 
* Complete Flush proceduer in DataNode

Resolves: #5220

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* fix unittest

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 15:59:04 +08:00
congqixia
607a8ffa1f Update WatchDmChannels signature (#5447) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby
cfdd98b70f Fix recovery branch after rebasing master (#5444) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby
07c6a4a669 Add dataservice register and discovery (#5435) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
congqixia
e3956ad13f Change SaveBinLogPaths field2Path to repeated (#8) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby
c53afee616 DataNode scales flowgraph 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby
b9cf4c1095 Fix recovery (#5417) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby
81ac20143b Add persistency of datanode cluster (#5387) 
We save channels registered in datanode in etcd and restore the cluster info after
restarting.

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby
3662b3f0e5 Refactor dataservice (#5400) 
Pick up grpc calls to grpc_handler.go and refactor code format

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:01 +08:00
congqixia
ef7339ceda DataService stores flush msgstream pos at VChannel granularity (#5397) 
SaveBinLogPaths saves stream positions at VChannel granularity

resolves #5396

Signed-off-by: Congqi Xia congqi.xia@zilliz.com
 2021-06-15 15:54:57 +08:00
sunby
50b3d39370
Add GetRecoveryInfo interface and proto (#5598) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-07 14:16:36 +08:00
zhenshan.cao
03e3b4beb4
Add log to startup process (#5594) 
* Add log to startup process

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fixbug: avoid ctx exceed deadline

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fmt code

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fix bug: wrong time unit

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fixbug: datanservice client reconnect use time ctx

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-04 16:29:35 +08:00
Xiangyu Wang
f791ccb246 Update OWNERS file 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-06-03 20:01:24 +08:00
Xiangyu Wang
ca0993c274 Update OWNERS 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-05-30 18:58:47 +08:00
Cai Yudong
b414800d49
Update Seek interface (#5492) 
* update Seek

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update Seek for mqTtMsgStream

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* use Retry in Seek

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix static-check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-29 23:21:34 +08:00
neza2017
f80dbab6ec
watch proxynode (#5424) 
watch proxy node and,
send InvalidataCollectionMetaCache to each proxynode

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

Resolves: #5423
 2021-05-26 12:14:30 +00:00
godchen
5f0f840782
Remove datanode client code (#5427) 
Remove datanode client code.

Signed-off-by: godchen qingxiang.chen@zilliz.com
 2021-05-26 10:41:37 +00:00
godchen
17cabfbed3
Add datanode client retry (#5394) 
Add datanode client retry.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-25 07:47:08 +00:00
neza2017
c28c34e852
let master use session.ServerID as nodeID (#5385) 
let master use session.ServerID as nodeID

also see #5386 

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-05-25 07:06:05 +00:00