admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-04 18:02:08 +08:00
Code Issues Packages Projects Releases Wiki Activity
20,075 Commits 136 Branches 188 Tags
Commit Graph

8923 Commits

Author SHA1 Message Date
Jiquan Long
7b9462c0d3
enhance: fix copying hits of inverted index twice (#33968) 
issue: https://github.com/milvus-io/milvus/issues/29793
The custom `VecCollector` have already transformed the results into
vector of offsets, no need to copy them twice.

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-06-19 12:40:01 +08:00
wayblink
5fac2fa1d2
fix: Panic if ProcessActiveStandBy returns error (#33369) 
#33368

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-19 11:16:00 +08:00
wei liu
02945959d9
enhance: Avoid to iterate whole segment list for each task's process (#33943) 
when querycoord process segment task, it will try to iterate whole
segment list to checke whether segment is loaded, which cost too much
cpu if there has thousands of segments.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-19 10:19:58 +08:00
cqy123456
298e50b834
enhance: check index with data type (#33880) 
issue: https://github.com/milvus-io/milvus/issues/22837
related: https://github.com/milvus-io/milvus/pull/33878

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-19 10:15:59 +08:00
Gao
0d20303e54
fix: fix binary vector data size (#33750) 
issue: https://github.com/milvus-io/milvus/issues/22837

- fix byte size wrong for binary vectors
- fix the expect/actual error msg

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-06-18 21:39:59 +08:00
Gao
a789c60380
enhance: autoindex for multi data type (#33868) 
issue: #22837 

contain https://github.com/milvus-io/milvus/pull/33625
https://github.com/milvus-io/milvus/pull/33867
https://github.com/milvus-io/milvus/pull/33911 which already merged to
2.4 branch

Signed-off-by: chasingegg <chao.gao@zilliz.com>
Co-authored-by: foxspy <xianliang.li@zilliz.com>
 2024-06-18 21:34:01 +08:00
cqy123456
b460862537
fix: can't find Chunk struct after growing support mmap (#33951) 
issue: https://github.com/milvus-io/milvus/issues/32984

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-18 18:37:58 +08:00
wayblink
5cb0760187
fix: Small fixs of major compaction (#33929) 
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-18 16:53:58 +08:00
congqixia
3fdaae8792
fix: Return record with largest timestamp for entires with same PK (#33936) 
See also #33883

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-18 15:55:59 +08:00
congqixia
5772123ca2
fix: Check the correct return error in MultiRemove (#33926) 
See also #33925

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-18 15:23:59 +08:00
wei liu
5e906fd17f
enhance: Force to reset coord connection for unavailable error (#33908) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-18 14:53:59 +08:00
cqy123456
32f685ff12
enhance: growing segment support mmap (#32633) 
issue: https://github.com/milvus-io/milvus/issues/32984

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-18 14:42:00 +08:00
congqixia
ec64499536
fix: Check nodeID wildcard when removing pkOracle (#33895) 
See also #33894

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-18 14:11:58 +08:00
shaoting-huang
8cdc0e6233
fix: fix data codec writer close (#33818) 
issue:#33813

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
 2024-06-18 13:59:57 +08:00
presburger
e83ecd5074
enhance: fix not find arrow::parquet lib (#33928) 
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
 2024-06-18 00:22:01 +08:00
congqixia
2a04b0929a
fix: Prevent use captured iteration variable partitionID (#33906) 
See also #33902

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-17 19:11:59 +08:00
congqixia
f993b2913b
enhance: Reserve space of payload writer when serialize data (#33817) 
See also #33561 #33562

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-17 12:06:04 +08:00
wei liu
188ee433b9
enhance: Refine name rule check error msg (#33815) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-17 10:37:58 +08:00
cai.zhang
95148866ed
fix: Don't remove growing L0 segment in datanode metacache (#33829) 
issue: #33540 
1. gorwing L0 segments is invisible to datacoord.
2. flushed L0 segments need to clean by datacoord.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-17 10:09:57 +08:00
yihao.dai
1a9ab52f66
enhance: Ensure the idempotency of compaction task (#33872) 
/kind enhancement

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-16 22:09:57 +08:00
yihao.dai
8537f3daeb
enhance: Rename Compaction to CompactionV2 (#33858) 
Due to the removal of injection and syncSegments from the compaction, we
need to ensure that no compaction is successfully executed during the
rolling upgrade. This PR renames Compaction to CompactionV2, with the
following effects:
- New datacoord + old datanode: Utilizes the CompactionV2 interface,
resulting in the datanode error "CompactionV2 not implemented," causing
compaction to fail;
- Old datacoord + new datanode: Utilizes the CompactionV1 interface,
resulting in the datanode error "CompactionV1 not implemented," causing
compaction to fail.

issue: https://github.com/milvus-io/milvus/issues/32809

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-16 22:07:57 +08:00
zhagnlu
d43ec4db0b
enhance: support array bitmap index (#33527) 
#32900

---------

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-16 21:51:58 +08:00
zhagnlu
e422168f09
fix: readd timestamp index because segment timestamp not ordered (#33856) 
#33533

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-16 21:48:01 +08:00
XuanYang-cn
f67b6dc2b0
fix: DeleteData merge wrong data casuing data loss (#33820) 
See also: #33819

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-14 17:57:56 +08:00
chyezh
9b69601dfb
fix: load operation when segment is on releasing (#31340) 
issue: #30857

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-14 15:35:56 +08:00
yihao.dai
86a36b105a
enhance: Tidy compaction executor (#33778) 
Move compaction executor to compaction pacakge.

issue: https://github.com/milvus-io/milvus/issues/32451

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-14 14:34:01 +08:00
shaoting-huang
0ecd694305
enhance: legacy code clean up (#33838) 
issue: #33839

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
 2024-06-14 14:25:56 +08:00
wei liu
4987067375
enhance: Execute bloom filter apply in parallel to speed up segment predict (#33792) 
issue: #33610

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-14 11:37:56 +08:00
cai.zhang
b69e9093c8
fix: Fallback field type when it isn't in request (#33832) 
issue: #33432

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-14 09:55:56 +08:00
wayblink
08fcf3f62b
fix: Fix meta prefix overlap bug (#33830) 
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-13 19:27:57 +08:00
wei liu
ab93d9c23d
enhance: Use BatchPkExist to reduce bloom filter func call cost (#33611) 
issue:#33610

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-13 17:57:56 +08:00
ArenaSu
2dfa752527
doc: [skip-e2e] add comments for event manager (#33444) 
Add comments for event manager(internal/datanode/event_manager.go).

Signed-off-by: ArenaSu <704427617@qq.com>
 2024-06-13 17:56:06 +08:00
ArenaSu
db59249776
doc: [skip-e2e] add comments for meta table (#33403) 
Add commnets for meta table in internal/rootcoord/meta_table.go.

Signed-off-by: ArenaSu <704427617@qq.com>
 2024-06-13 17:54:09 +08:00
Yinzuo Jiang
7d74edd6dd
fix: update clang-tidy and clang-format from 10 to 12 (#33141) 
Default llvm toolchain version in Ubuntu 20.04 is 10, while Ubuntu 22.04
does not have `clang-tidy-10` or `clang-format-10` by default.

issue: #33142

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
 2024-06-13 15:27:58 +08:00
congqixia
512ea6be5f
enhance: Avoid merging insert data when buffering insert msgs (#33562) 
See also #33561

This PR:
- Use zero copy when buffering insert messages
- Make `storage.InsertCodec` support serialize multiple insert data
chunk into same batch binlog files

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-13 11:15:56 +08:00
congqixia
9ab3058da2
fix: Prevent restart timetick sender creating ut datanode (#33790) 
See also #33789

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-12 22:29:58 +08:00
congqixia
b39dfc25dc
enhance: Use fastjson lib for unmarshal delete log (#33787) 
```
goos: linux
goarch: amd64
GOMAXPROC=1
cpu: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
BenchmarkJsonSerdeStd             343872              3568 ns/op            1335 B/op         25 allocs/op
BenchmarkJsonSerdeFastjson       5124177               234.9 ns/op            16 B/op          1 allocs/op
```

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-12 20:41:57 +08:00
chyezh
b3ee1a4bd5
fix: lost DroppedAt field when DropCollection (#33766) 
issue: #33097

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-12 20:21:56 +08:00
yihao.dai
9a3e4080f1
enhance: Add comment for channel cp updater (#33759) 
/kind enhancement

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-12 20:01:55 +08:00
Cai Yudong
9d4535ce0b
enhance: Handle Float16Vector/BFloat16Vector numpy bulk insert as same as BinaryVector (#33760) 
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-06-12 17:17:55 +08:00
Buqian Zheng
47b04ea167
enhance: support sparse cardinal hnsw index (#33656) 
issue: #29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-06-12 16:57:55 +08:00
Buqian Zheng
8cb350598c
enhance: Improve GetVectorById of Sparse Float Vector (#33209) 
issue: #29419

* sparse float vector to support raw data mmap

For get vector from chunk cache, I added a unit test but marking it as
skipped due to a known issue. I have tested it locally.

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-06-12 10:09:55 +08:00
coldWater
6b9901c59f
enhance: add a semaphore for CompactionExecutor (#33558) 
#33182

---------

Signed-off-by: coldWater <254244460@qq.com>
 2024-06-11 17:25:55 +08:00
Chun Han
f7af323d1e
fix: sync partitiion stats blocking balance task(#33741) (#33742) 
related: #33741

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-06-11 14:21:56 +08:00
Jiquan Long
ecf2bcee42
enhance: speed up array-equal operator via inverted index (#33633) 
fix: #33632

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-06-11 14:13:54 +08:00
chyezh
2b7ee1968f
enhance: new messsage interface for log service (#33286) 
issue: #33285

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-11 10:38:01 +08:00
chyezh
8ca5ced821
fix: async warmup will be blocked by state lock (#33686) 
issue: #33685

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-10 21:59:53 +08:00
yihao.dai
b1d46eb34b
fix: Fix multiple vector fields import (#33723) 
1. Fix dim mismatch with multi-vector fields and JSON import
2. Enhance: do not display file ID in GetImportResponse.

issue: https://github.com/milvus-io/milvus/issues/33681,
https://github.com/milvus-io/milvus/issues/33682

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-10 21:57:54 +08:00
yihao.dai
eb5d4de390
fix: Check if the import job exists (#33672) 
issue: https://github.com/milvus-io/milvus/issues/33671

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-10 21:51:55 +08:00
wayblink
a1232fafda
feat: Major compaction (#33620) 
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
 2024-06-10 21:34:08 +08:00