wei liu
7d73032582
enhance: refactor leader_observer to leader_checker ( #29454 ) ( #29984 )
...
issue: #29453
pr: #29452
sync distribution by rpc will also call loadSegment/releaseSegment,
which may cause all kinds of concurrent case on same segment, such as
concurrent load and release on one segment.
This PR add leader_checker which generate load/release task to correct
the leader view, instead of calling sync distribution by rpc
---------
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-18 14:08:54 +08:00
wei liu
5520bfbb05
enhance: Change some frequency log to rated level ( #29720 ) ( #29903 )
...
pr: #29720
This PR change some frequency log to rated level
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-12 11:46:52 +08:00
congqixia
5ec79ab6f6
fix: [Cherry-pick] Add atomic method to get collection target ( #29580 )
...
Cherry pick from master
pr: #29577
Related to #29575
Add `getCollectionTarget` method which is atomic when scope is
`CurrentTargetFirst` or `NextTargetFirst`
Also return error when executor finds no channel in target manager
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-29 10:30:46 +08:00
wei liu
a13fc4d346
enhance: Remove useless log in collection observer ( #29555 )
...
pr: #29554
This PR removed useless log in collection observer
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-28 17:14:45 +08:00
yah01
4c0ca83928
enhance: speed up loading with many deletions ( #29455 ) ( #29520 )
...
the executor always fetches the latest segment info, so we could consume
from the latest checkpoint, which could save much time while deleted
many entities
pr: #29455
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-27 23:24:46 +08:00
yah01
5a8ddde92a
fix: load gets stuck probably ( #29191 ) ( #29192 )
...
we found the load got stuck probably, and reviewed the logs.
the target observer seems not working, the reason is the taskDispatcher
removes the task in a goroutine, and modifies the task status after
committing the task into the goroutine pool, but this may happen after
the task removed, which leads to the task will never be removed
related #29086
pr: #29191
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-14 16:56:38 +08:00
yah01
76757e53c4
enhance: Add more logs for target updating ( #29090 ) ( #29141 )
...
This pull request enhances the logging functionality in the code related
to target updating. It adds more logs about the condition satisfying
when updating the target. The logs provide additional information about
the collection ID, replica number, channel readiness, segment readiness,
and leader view readiness. These logs will help in troubleshooting and
monitoring the target updating process.
pr: #29090
Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-12 22:28:38 +08:00
aoiasd
8502037cff
fix: [Cherry-pick] sync action load segment with lack collection index info list ( #28956 )
...
relate: https://github.com/milvus-io/milvus/issues/28779
https://github.com/milvus-io/milvus/issues/28637
pr: https://github.com/milvus-io/milvus/pull/28788
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-07 14:14:42 +08:00
congqixia
3a33afd1fb
enhance: [cherry-pick] Change const magic number in querycoord to param ( #28819 ) ( #28947 )
...
Cherry-pick from master
pr: #28819
See also #28817
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-04 19:06:40 +08:00
wei liu
c650240f31
enhance: Change some frequency log to rated level ( #28897 ) ( #28934 )
...
pr: #28897
This pr change some frequency log's level to rated.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-04 18:52:32 +08:00
congqixia
d0f94f3d17
fix: make qcv2 observer dispatcher execute exactly once ( #28472 ) ( #28477 )
...
Cherry-pick from master
pr: #28472
See also #28466
In `taskDispatcher.schedule`, same task may be resubmitted if the
previous round did not finish
In this case, TaskObserver.check may set current target by mistake,
which may cause the random search/query failure
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-17 01:34:21 +08:00
yah01
385507ce47
Fix the target updated before version updated to cause data missing ( #28257 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-08 18:54:18 +08:00
wei liu
87e8d04ed7
fix sync distribution with wrong version ( #28130 ) ( #28170 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-06 11:38:18 +08:00
wei liu
e0222b2ce3
refine target manager code style ( #27883 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-10-25 00:44:12 +08:00
congqixia
323fc107a7
Fix taskDispatcher add multiple tasks will ignore following ones ( #27885 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-24 17:18:13 +08:00
congqixia
93a877f55e
Make qcv2 target&leader observer execute in parallel ( #27844 )
...
- Add `taskDispatcher` to submit and run task async safely
- Change `LeaderObeserver` and `TargetObserver` schedule and manual check action to submitting task into dispatcher
- Fix logic problem in collection observer when manual check return false
See also #27494
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-24 10:14:11 +08:00
wei liu
55e5f80e24
update collection target after observer start ( #27774 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-10-19 21:52:10 +08:00
yah01
be980fbc38
Refine state check ( #27541 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-11 21:01:35 +08:00
wei liu
42c475a0e0
remove useless log in querycoord ( #27362 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-10-11 10:13:34 +08:00
congqixia
eca79d149c
Add ctx control for observer manual check methods ( #27531 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-09 11:07:33 +08:00
yah01
a715165306
Set timeout for leader observer syncing ( #27504 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-08 16:55:31 +08:00
yah01
a8ce1b6686
Refine QueryCoord stopping ( #27371 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-27 16:27:27 +08:00
SimFG
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
yah01
b4f86ea55e
Construct all success status with merr ( #27226 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-20 10:57:23 +08:00
Enwei Jiao
fb0705df1b
Decouple basetable and componentparam ( #26725 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-05 10:31:48 +08:00
yah01
941a383019
Fix failed to load collection with more than 128 partitions ( #26763 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-02 00:09:01 +08:00
wei liu
949c320185
remove pull target from qc recover ( #26775 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-09-01 11:17:01 +08:00
Enwei Jiao
7d61355ab0
Refactor log for Query ( #26310 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-08-14 18:57:32 +08:00
wei liu
b47a72bfcf
fix set dirty segment distribution to leader view ( #26180 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-11 11:21:32 +08:00
Bingyi Sun
54c0e64059
Fix search on empty segments set bug ( #26136 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-08-08 11:17:08 +08:00
wei liu
6f89620a43
remove pull target rpc from lock ( #26054 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-04 10:31:06 +08:00
wei liu
518b6310a2
refine retry times on replica ( #26043 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-03 15:55:09 +08:00
Bingyi Sun
a3e22786ed
Move meta store to kv catalog ( #25915 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-07-31 13:57:04 +08:00
congqixia
0bc03ede0d
Add eventlog pkg and support grpc streaming event observation ( #25812 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-25 17:23:01 +08:00
wei liu
fc19b85a40
fix count(*）retrieve redundant growing segment ( #25825 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-24 14:09:00 +08:00
wei liu
b62c82af22
fix set target version in loading sealed segment ( #25603 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-14 20:00:31 +08:00
congqixia
2bed5bc461
Fix false load success for partition load timeout but collection remains ( #25379 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-06 19:10:25 +08:00
wei liu
2ae6def394
fix sync target version with wrong segment list ( #25337 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-06 10:30:26 +08:00
wei liu
68ae199a9f
load segment with target version, avoid read redundant segment ( #24929 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-06-27 11:48:45 +08:00
yiwangdr
4387f36897
make etcdKV private ( #24778 )
...
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2023-06-13 10:52:38 +08:00
congqixia
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
yah01
848ef7418b
Fix the refresh may be notified finished early ( #24438 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-05-26 18:43:26 +08:00
yihao.dai
7e0d1492c7
Load delete from channel checkpoint ( #23961 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-09 19:10:41 +08:00
congqixia
ed81eaa963
Make CollectionObserver trigger checker more frequently during load procedure ( #23928 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-08 14:06:41 +08:00
wei liu
cbfe7a45ef
fix pull target ( #23491 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-04-18 18:30:32 +08:00
congqixia
b5a73e6d1a
Add OWNERS files for querycoordv2 sub pkgs ( #23489 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-04-18 15:52:30 +08:00
yah01
296380d6e6
Support async refresh ( #23107 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-04-12 15:06:28 +08:00
MrPresent-Han
ba02e70301
fix bug when syncing distribution without schema and remove unused logic( #23085 ) ( #23216 )
...
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
2023-04-07 19:32:29 +08:00
jaime
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
congqixia
127867b873
Add ratedgroup for some info/warning log ( #23095 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-03-31 15:22:23 +08:00
