admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-07 19:31:51 +08:00
Code Issues Packages Projects Releases Wiki Activity
7,082 Commits 137 Branches 188 Tags
Commit Graph

130 Commits

Author SHA1 Message Date
Cai Yudong
a948b71e06 Support to send dd msg to all dml channels (#5597) 
* send dd msg to dml channel

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add dml_channels.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix race

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-15 16:04:48 +08:00
sunby
aa8a038305 Rename SegmentAllocator to SegmentManager (#5559) 
Add numRows to segmentStatus and Rename SegmentAllocator to
SegmentManager. Remove SegmentAllocStats.

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 16:04:48 +08:00
godchen
275881dbdb Remove address param (#5556) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-15 16:04:46 +08:00
sunby
f1ccbb8f9a Fix init session in dataservice (#5522) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby
38b1f7dabe Change the logic of getting vchannel position (#5502) 
Datanode send `SaveBinlogPath` request after every segment flush finish
with the binlog paths and dml/ddl position. But the flush of segments is
not sorted. So we sort the segments according to segment id and find the
largest segment id with not nil dml position which is the position of
the msgstream to recover.

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby
9a54f1f72d Watch channel if needed when allocating segmnets (#5482) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby
43534ef63f Add unit tests in dataservice (#5478) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby
a14c35274f Fix bugs in dataservice (#5457) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
XuanYang-cn
55eac3ee7f Complete Flush procedure in DataNode (#5454) 
* Complete Flush proceduer in DataNode

Resolves: #5220

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* fix unittest

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 15:59:04 +08:00
congqixia
607a8ffa1f Update WatchDmChannels signature (#5447) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby
cfdd98b70f Fix recovery branch after rebasing master (#5444) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby
07c6a4a669 Add dataservice register and discovery (#5435) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
congqixia
e3956ad13f Change SaveBinLogPaths field2Path to repeated (#8) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby
81ac20143b Add persistency of datanode cluster (#5387) 
We save channels registered in datanode in etcd and restore the cluster info after
restarting.

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby
3662b3f0e5 Refactor dataservice (#5400) 
Pick up grpc calls to grpc_handler.go and refactor code format

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:01 +08:00
congqixia
ef7339ceda DataService stores flush msgstream pos at VChannel granularity (#5397) 
SaveBinLogPaths saves stream positions at VChannel granularity

resolves #5396

Signed-off-by: Congqi Xia congqi.xia@zilliz.com
 2021-06-15 15:54:57 +08:00
sunby
50b3d39370
Add GetRecoveryInfo interface and proto (#5598) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-07 14:16:36 +08:00
zhenshan.cao
03e3b4beb4
Add log to startup process (#5594) 
* Add log to startup process

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fixbug: avoid ctx exceed deadline

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fmt code

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fix bug: wrong time unit

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fixbug: datanservice client reconnect use time ctx

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-04 16:29:35 +08:00
Cai Yudong
b414800d49
Update Seek interface (#5492) 
* update Seek

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update Seek for mqTtMsgStream

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* use Retry in Seek

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix static-check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-29 23:21:34 +08:00
neza2017
f80dbab6ec
watch proxynode (#5424) 
watch proxy node and,
send InvalidataCollectionMetaCache to each proxynode

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

Resolves: #5423
 2021-05-26 12:14:30 +00:00
godchen
5f0f840782
Remove datanode client code (#5427) 
Remove datanode client code.

Signed-off-by: godchen qingxiang.chen@zilliz.com
 2021-05-26 10:41:37 +00:00
godchen
17cabfbed3
Add datanode client retry (#5394) 
Add datanode client retry.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-25 07:47:08 +00:00
neza2017
c28c34e852
let master use session.ServerID as nodeID (#5385) 
let master use session.ServerID as nodeID

also see #5386 

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-05-25 07:06:05 +00:00
neza2017
e326eaad1e
master client,reconnect if grpc failed (#5360) 
master client,reconnect if grpc failed

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-05-24 01:44:49 +00:00
congqixia
4aa6a8fe18
Dataservice use lastest grpc design and saves ddl/dml pos pair (#5348) 
Dataservice use lastest grpc design and saves ddl/dml pos pair

* Update SaveBinlogPaths grpc definition
* Saves segment ddl & dml flush pos pair(start, end)

See also #5220 

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-05-21 11:38:58 +00:00
godchen
0eee7e5252
Change session (#5329) 
Change session.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-21 11:28:52 +00:00
sunby
02f2348500
Save segment allocation status to meta (#5184) 
To ensure segments being flushed correctly, we need to load 
the segment allocation status from meta after starting the service.
 2021-05-21 10:30:41 +00:00
congqixia
7a82462da8
DataService publishes flush-completed message into its channel (#5320) 
DataService publishes flush-completed message  into related msgstream

See also: #5220 

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-05-21 08:54:29 +00:00
sunby
c1efa53779
Remove redundant fields in SegmentInfo and related structure (#5280) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-05-21 06:51:25 +00:00
congqixia
38f5b0826f
Add SaveBinlogPaths grpc in dataservice (#5277) 
* Add SaveBinlogPaths grpc in dataservice

* fix merged compile error

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-05-20 11:34:45 +08:00
godchen
e224d1e725
Add SessionManager (#5288) 
Add SessionManager.
Resolves: #5174 

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-19 10:36:05 +00:00
congqixia
d5ab8ac345
Restore flush and stats stream pos (#5284) 
Restore segment flush stream & statistic stream to last success pos

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-05-19 06:13:53 +00:00
godchen
b74afd7a0e
Add service registration (#5189) 
Add service registration.
Part of Issue #5174.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-14 02:05:18 +00:00
dragondriver
64ab4d740b
Add row_count to partition statistics (#5162) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-05-10 09:39:08 +00:00
sunby
a00422bac2
Remove unused interfaces in dataservice (#5157) 
Remove unused interfaces in dataservice

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-05-10 09:03:24 +00:00
Cai Yudong
74f457bd14
add more testcase to improve code coverage for data service (#5041) 
* fix TestChannel run error

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* comment validateAllocRequest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add test APIs

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix static-check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-04-26 01:45:54 +00:00
Cai Yudong
5a75e23795
improve data service code coverage (#5020) 
* optimize TestAssignSegmentID

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update mockMasterService to improve code coverage

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename client to kvClient in Server

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update server.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add TestChannel

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update TestChannel

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update TestChannel

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix format

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update TestChannel

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-04-24 03:29:15 +00:00
Xiangyu Wang
82ccd4cec0
Rename module (#4988) 
* Rename module

Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-04-22 14:45:57 +08:00
sunby
0f2f46f39d Add copyright 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-04-19 11:35:38 +08:00
sunby
94c3fc9321 Add unit tests in dataservice 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-04-16 16:30:55 +08:00
sunby
a7dac818ee Refactor datanode register 
Dataservice should not be stalled during registering datanode.

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-04-13 09:47:02 +08:00
sunby
61edc53faf Refactor data service segment allocator 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-04-12 16:35:51 +08:00
sunby
751c9d4e81 Refactor meta.go 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-04-09 09:55:04 +08:00
zhenshan.cao
7bad0a01e9 Fix bug: unmarshal got unexpected EOF 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-03-31 13:55:51 +08:00
zhenshan.cao
efbf4452c5 Add log for port changed or proto unmarshal error 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-03-30 20:19:30 +08:00
sunby
a78cd3e2a7 Add proxy service timetick loop to expire allcoations 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-27 14:01:52 +08:00
quicksilver
7ad9b36207 Update reviewdog/action-hadolint github action version to v1.16.1 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2021-03-25 14:41:46 +08:00
XuanYang-cn
51f669f1cb Flush segments concurrently & change etcd meta 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-03-23 18:50:13 +08:00
紫晴
25f3e520f9 Add testcase for index 
Signed-off-by: 紫晴 <ting.wang@zilliz.com>
 2021-03-23 16:57:59 +08:00
zhenshan.cao
c2734fa55f Fix bug and enchance system 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-03-22 16:36:10 +08:00