admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-06 19:02:18 +08:00
Code Issues Packages Projects Releases Wiki Activity
20,271 Commits 137 Branches 188 Tags
Commit Graph

1101 Commits

Author SHA1 Message Date
wayblink
358e9a10d2
enhance: Alter compactTo segments before compactFrom to avoid data loss if crash (#34513) 
#34512

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-12 00:55:34 +08:00
jaime
a08a0c831f
fix: encountering orphan channel-cp meta after DataCoord GC (#34612) 
issue: #34545

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-11 23:01:35 +08:00
XuanYang-cn
d7966f46ad
fix: Compaction task l0 state transfer wrong (#34597) 
This bug caused failed L0 compaction tasks never end.

See also: #34460

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-11 21:39:36 +08:00
Patrick Weizhi Xu
104d0966b7
feat: support partition key isolation (#34336) 
issue: #34332

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-07-11 19:01:35 +08:00
XuanYang-cn
d7a3697fb5
enhance: Add back compactionTaskNum metrics (#34583) 
Fix L0 compaction task recover unable to set segment not isCompacting

See also: #34460

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-11 17:45:37 +08:00
wayblink
fd3da90089
enhance: Refine clustering compaction latency metrics (#34528) 
#34527

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-11 17:43:43 +08:00
XuanYang-cn
e0b39d8bf4
fix: Milvus panic when compaction disabled and dropping a collection (#34103) 
See also: #31059

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-11 14:44:52 +08:00
congqixia
be92147723
enhance: Print reason when seal segments (#34575) 
See also #34574

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-11 11:40:51 +08:00
jaime
c332f69dec
enhance: skip orphan channel cp meta when checking cp lag (#34555) 
issue: # #34545

Print warn log instead of check health fail if orphan channel cp meta is
found in health check request.

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-11 09:36:56 +08:00
XuanYang-cn
314f4d995b
enhance: Tidy dc channel manager (#34515) 
See also: #34518

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-09 18:26:12 +08:00
wayblink
0c4d82c6d3
fix: fix compaction task startTime is 0 (#34496) 
fix bug: #34497 #34498

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-09 10:22:13 +08:00
congqixia
3333160b8d
enhance: Fix lint issues from recent PRs (#34482) 
See also #34483
Some lint issues are introduced due to lack of static check run. This PR
fixes these problems.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-09 10:06:24 +08:00
Gao
ae6d6f91e6
enhance: change autoindex default metric type (#34261) 
issue: #34304 
cosine is more widely used in float vectors, and cosine and hamming
distance are 'metrics' which have good geometric properties

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-07-08 19:52:24 +08:00
XuanYang-cn
467113deab
fix: Accidently exit the check loop (#34481) 
See also: #34460

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-08 16:46:13 +08:00
wayblink
efdaed4ac6
fix: fix clustering compaction can't enqueue when compaction queue is full (#34445) 
#30633

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-08 10:02:13 +08:00
yihao.dai
4e5f1d5f75
enhance: Pre-allocate ids for import (#33958) 
The import is dependent on syncTask, which in turn relies on the
allocator. This PR pre-allocate the necessary IDs for import syncTask.

issue: https://github.com/milvus-io/milvus/issues/33957

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-07 21:26:14 +08:00
jaime
21fc5f5d46
enhance: Remove datanode reporting TT based on MQ implementation (#34421) 
issue: #34420

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-05 15:48:09 +08:00
congqixia
439fb24b32
fix: Skip l0 segments when syncing segments to datanodes (#34388) 
See also #34387

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-04 18:02:14 +08:00
congqixia
d51d0954bd
enhance: Continue loop when reassign channel fails (#34331) 
Log will be confusing when `Reassign` channel operation failed for both
success & failure log will be printed in row. This PR continue the loop
to avoid this output.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-04 14:20:10 +08:00
cai.zhang
23e2a0e78b
fix: Remove unused code to fix compiling (#34406) 
issue: #34407

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-04 08:48:08 +08:00
jaime
d1f57aa4ba
enhance: remove deprecated code within channel manager (#34340) 
issue: https://github.com/milvus-io/milvus/issues/33994

only remove deprecated code, no additional changes.

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-03 19:46:09 +08:00
wayblink
da56880d0f
fix: Avoid datarace in clustering compaction (#34288) 
#34289

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-03 19:08:09 +08:00
wayblink
a501fa11c0
fix: remove isFull check in compaction.enqueue (#34338) 
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-03 18:10:10 +08:00
cai.zhang
feb13cdf07
enhance: Skip pick worker when task doesn't need to execute actually (#34348) 
issue: #34347

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-03 15:52:09 +08:00
cai.zhang
8165961223
enhance: Optimeize clustering compaction (#34313) 
issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-03 15:14:08 +08:00
wayblink
fb61344dc9
fix: Revert a optimize in clustering compaction (#34299) 
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-03 10:32:08 +08:00
cai.zhang
a5be322ab2
enhance: Don't set the timeout for SyncSegments (#34226) 
issue: #32809

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-02 14:28:07 +08:00
cai.zhang
4cf1a358ba
fix: Sync the sealed and flushed segments to datanode (#34301) 
issue: #33696

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-01 22:42:08 +08:00
congqixia
e083d99dd9
enhance: Use channel name map finding channel watcher (#34294) 
See also #33235

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-01 18:46:07 +08:00
jaime
d6afb31b94
enhance: make subfunctions of datanode component modular (#33992) 
issue: #33994

also remove deprecated channel manager based on the etcd implementation

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-01 14:46:07 +08:00
jaime
0426390f06
enhance: improve check health (#33800) 
issue: #34264

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-01 10:16:06 +08:00
wayblink
73ffc1b424
Refine clustering_compaction_task retry mechanism (#34194) 
#32939

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-30 20:22:09 +08:00
Chun Han
96dcee5dff
fix:load major compaction partial result(#34051) (#34052) 
related: #34051

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-06-28 14:04:05 +08:00
jaime
9630974fbb
enhance: move rocksmq from internal to pkg module (#33881) 
issue: #33956

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-25 21:18:15 +08:00
cai.zhang
c65f41dc60
fix: Only sync flushed segments to datanode (#34156) 
issue: #33540

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-25 20:46:07 +08:00
yihao.dai
940a0ac0a3
fix: Prevent import from generating orphaned files (#34070) 
When import failed, mark the import segment as dropped instead of drop
it directly to prevent generating orphaned files.

issue: https://github.com/milvus-io/milvus/issues/34068

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-25 16:58:02 +08:00
zhenshan.cao
d18c49013b
enhance: Refine compaction (#33982) 
issue : https://github.com/milvus-io/milvus/issues/32939

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-06-25 10:08:03 +08:00
Patrick Weizhi Xu
b961767005
enhance: support integral type for MV and skip MV if there is only one category (#33161) 
issue: #29892

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-06-24 10:20:01 +08:00
Bingyi Sun
622be36ad7
fix: fix l0 compaction may miss some sealed segments (#33980) 
issue: https://github.com/milvus-io/milvus/issues/33955

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-06-21 16:52:15 +08:00
XuanYang-cn
04edb07d82
enhance: Add deltaRowCount in l0 compaction (#33997) 
See also: #33998

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-20 17:46:01 +08:00
congqixia
6a15b6e2be
fix: Remove loopclosure issue in ChannelManagerImplV2 (#33989) 
See also #33987

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-19 21:25:59 +08:00
wayblink
5fac2fa1d2
fix: Panic if ProcessActiveStandBy returns error (#33369) 
#33368

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-19 11:16:00 +08:00
wayblink
5cb0760187
fix: Small fixs of major compaction (#33929) 
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-18 16:53:58 +08:00
cai.zhang
95148866ed
fix: Don't remove growing L0 segment in datanode metacache (#33829) 
issue: #33540 
1. gorwing L0 segments is invisible to datacoord.
2. flushed L0 segments need to clean by datacoord.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-17 10:09:57 +08:00
yihao.dai
8537f3daeb
enhance: Rename Compaction to CompactionV2 (#33858) 
Due to the removal of injection and syncSegments from the compaction, we
need to ensure that no compaction is successfully executed during the
rolling upgrade. This PR renames Compaction to CompactionV2, with the
following effects:
- New datacoord + old datanode: Utilizes the CompactionV2 interface,
resulting in the datanode error "CompactionV2 not implemented," causing
compaction to fail;
- Old datacoord + new datanode: Utilizes the CompactionV1 interface,
resulting in the datanode error "CompactionV1 not implemented," causing
compaction to fail.

issue: https://github.com/milvus-io/milvus/issues/32809

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-16 22:07:57 +08:00
wei liu
4987067375
enhance: Execute bloom filter apply in parallel to speed up segment predict (#33792) 
issue: #33610

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-14 11:37:56 +08:00
chyezh
b3ee1a4bd5
fix: lost DroppedAt field when DropCollection (#33766) 
issue: #33097

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-12 20:21:56 +08:00
Chun Han
f7af323d1e
fix: sync partitiion stats blocking balance task(#33741) (#33742) 
related: #33741

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-06-11 14:21:56 +08:00
yihao.dai
b1d46eb34b
fix: Fix multiple vector fields import (#33723) 
1. Fix dim mismatch with multi-vector fields and JSON import
2. Enhance: do not display file ID in GetImportResponse.

issue: https://github.com/milvus-io/milvus/issues/33681,
https://github.com/milvus-io/milvus/issues/33682

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-10 21:57:54 +08:00
yihao.dai
eb5d4de390
fix: Check if the import job exists (#33672) 
issue: https://github.com/milvus-io/milvus/issues/33671

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-10 21:51:55 +08:00