admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-04 18:02:08 +08:00
Code Issues Packages Projects Releases Wiki Activity
20,059 Commits 136 Branches 188 Tags
Commit Graph

8909 Commits

Author SHA1 Message Date
presburger
e83ecd5074
enhance: fix not find arrow::parquet lib (#33928) 
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
 2024-06-18 00:22:01 +08:00
congqixia
2a04b0929a
fix: Prevent use captured iteration variable partitionID (#33906) 
See also #33902

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-17 19:11:59 +08:00
congqixia
f993b2913b
enhance: Reserve space of payload writer when serialize data (#33817) 
See also #33561 #33562

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-17 12:06:04 +08:00
wei liu
188ee433b9
enhance: Refine name rule check error msg (#33815) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-17 10:37:58 +08:00
cai.zhang
95148866ed
fix: Don't remove growing L0 segment in datanode metacache (#33829) 
issue: #33540 
1. gorwing L0 segments is invisible to datacoord.
2. flushed L0 segments need to clean by datacoord.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-17 10:09:57 +08:00
yihao.dai
1a9ab52f66
enhance: Ensure the idempotency of compaction task (#33872) 
/kind enhancement

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-16 22:09:57 +08:00
yihao.dai
8537f3daeb
enhance: Rename Compaction to CompactionV2 (#33858) 
Due to the removal of injection and syncSegments from the compaction, we
need to ensure that no compaction is successfully executed during the
rolling upgrade. This PR renames Compaction to CompactionV2, with the
following effects:
- New datacoord + old datanode: Utilizes the CompactionV2 interface,
resulting in the datanode error "CompactionV2 not implemented," causing
compaction to fail;
- Old datacoord + new datanode: Utilizes the CompactionV1 interface,
resulting in the datanode error "CompactionV1 not implemented," causing
compaction to fail.

issue: https://github.com/milvus-io/milvus/issues/32809

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-16 22:07:57 +08:00
zhagnlu
d43ec4db0b
enhance: support array bitmap index (#33527) 
#32900

---------

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-16 21:51:58 +08:00
zhagnlu
e422168f09
fix: readd timestamp index because segment timestamp not ordered (#33856) 
#33533

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-16 21:48:01 +08:00
XuanYang-cn
f67b6dc2b0
fix: DeleteData merge wrong data casuing data loss (#33820) 
See also: #33819

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-14 17:57:56 +08:00
chyezh
9b69601dfb
fix: load operation when segment is on releasing (#31340) 
issue: #30857

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-14 15:35:56 +08:00
yihao.dai
86a36b105a
enhance: Tidy compaction executor (#33778) 
Move compaction executor to compaction pacakge.

issue: https://github.com/milvus-io/milvus/issues/32451

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-14 14:34:01 +08:00
shaoting-huang
0ecd694305
enhance: legacy code clean up (#33838) 
issue: #33839

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
 2024-06-14 14:25:56 +08:00
wei liu
4987067375
enhance: Execute bloom filter apply in parallel to speed up segment predict (#33792) 
issue: #33610

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-14 11:37:56 +08:00
cai.zhang
b69e9093c8
fix: Fallback field type when it isn't in request (#33832) 
issue: #33432

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-14 09:55:56 +08:00
wayblink
08fcf3f62b
fix: Fix meta prefix overlap bug (#33830) 
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-13 19:27:57 +08:00
wei liu
ab93d9c23d
enhance: Use BatchPkExist to reduce bloom filter func call cost (#33611) 
issue:#33610

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-13 17:57:56 +08:00
ArenaSu
2dfa752527
doc: [skip-e2e] add comments for event manager (#33444) 
Add comments for event manager(internal/datanode/event_manager.go).

Signed-off-by: ArenaSu <704427617@qq.com>
 2024-06-13 17:56:06 +08:00
ArenaSu
db59249776
doc: [skip-e2e] add comments for meta table (#33403) 
Add commnets for meta table in internal/rootcoord/meta_table.go.

Signed-off-by: ArenaSu <704427617@qq.com>
 2024-06-13 17:54:09 +08:00
Yinzuo Jiang
7d74edd6dd
fix: update clang-tidy and clang-format from 10 to 12 (#33141) 
Default llvm toolchain version in Ubuntu 20.04 is 10, while Ubuntu 22.04
does not have `clang-tidy-10` or `clang-format-10` by default.

issue: #33142

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
 2024-06-13 15:27:58 +08:00
congqixia
512ea6be5f
enhance: Avoid merging insert data when buffering insert msgs (#33562) 
See also #33561

This PR:
- Use zero copy when buffering insert messages
- Make `storage.InsertCodec` support serialize multiple insert data
chunk into same batch binlog files

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-13 11:15:56 +08:00
congqixia
9ab3058da2
fix: Prevent restart timetick sender creating ut datanode (#33790) 
See also #33789

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-12 22:29:58 +08:00
congqixia
b39dfc25dc
enhance: Use fastjson lib for unmarshal delete log (#33787) 
```
goos: linux
goarch: amd64
GOMAXPROC=1
cpu: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
BenchmarkJsonSerdeStd             343872              3568 ns/op            1335 B/op         25 allocs/op
BenchmarkJsonSerdeFastjson       5124177               234.9 ns/op            16 B/op          1 allocs/op
```

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-12 20:41:57 +08:00
chyezh
b3ee1a4bd5
fix: lost DroppedAt field when DropCollection (#33766) 
issue: #33097

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-12 20:21:56 +08:00
yihao.dai
9a3e4080f1
enhance: Add comment for channel cp updater (#33759) 
/kind enhancement

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-12 20:01:55 +08:00
Cai Yudong
9d4535ce0b
enhance: Handle Float16Vector/BFloat16Vector numpy bulk insert as same as BinaryVector (#33760) 
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-06-12 17:17:55 +08:00
Buqian Zheng
47b04ea167
enhance: support sparse cardinal hnsw index (#33656) 
issue: #29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-06-12 16:57:55 +08:00
Buqian Zheng
8cb350598c
enhance: Improve GetVectorById of Sparse Float Vector (#33209) 
issue: #29419

* sparse float vector to support raw data mmap

For get vector from chunk cache, I added a unit test but marking it as
skipped due to a known issue. I have tested it locally.

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-06-12 10:09:55 +08:00
coldWater
6b9901c59f
enhance: add a semaphore for CompactionExecutor (#33558) 
#33182

---------

Signed-off-by: coldWater <254244460@qq.com>
 2024-06-11 17:25:55 +08:00
Chun Han
f7af323d1e
fix: sync partitiion stats blocking balance task(#33741) (#33742) 
related: #33741

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-06-11 14:21:56 +08:00
Jiquan Long
ecf2bcee42
enhance: speed up array-equal operator via inverted index (#33633) 
fix: #33632

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-06-11 14:13:54 +08:00
chyezh
2b7ee1968f
enhance: new messsage interface for log service (#33286) 
issue: #33285

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-11 10:38:01 +08:00
chyezh
8ca5ced821
fix: async warmup will be blocked by state lock (#33686) 
issue: #33685

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-10 21:59:53 +08:00
yihao.dai
b1d46eb34b
fix: Fix multiple vector fields import (#33723) 
1. Fix dim mismatch with multi-vector fields and JSON import
2. Enhance: do not display file ID in GetImportResponse.

issue: https://github.com/milvus-io/milvus/issues/33681,
https://github.com/milvus-io/milvus/issues/33682

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-10 21:57:54 +08:00
yihao.dai
eb5d4de390
fix: Check if the import job exists (#33672) 
issue: https://github.com/milvus-io/milvus/issues/33671

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-10 21:51:55 +08:00
wayblink
a1232fafda
feat: Major compaction (#33620) 
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
 2024-06-10 21:34:08 +08:00
chyezh
f53ab54c5d
enhance: async cgo utility (#33133) 
issue: #30926, #33132

- implement future-based cgo utility.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-09 22:55:53 +08:00
yihao.dai
3540eee977
enhance: Support L0 import (#33514) 
issue: https://github.com/milvus-io/milvus/issues/33157

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-07 14:17:20 +08:00
zhagnlu
6ce9df913f
fix: clean vector memory (#33692) 
#33533

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-07 10:37:54 +08:00
wei liu
0a3d456688
enhance: add restful api to trigger component stop (#32076) 
issue: #32698
This PR add two rest api for component stop and status check:
1. `/management/stop?role=querynode` can stop the specified component
2. `/management/check/ready?role=rootcoord` can check whether the target
component is serviceable

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-07 10:35:54 +08:00
smellthemoon
c61fb1eff5
enhance: do check when add not empty logpath (#33640) 
meta only store logid

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-06-07 10:19:51 +08:00
SimFG
ecee7d90d4
enhance: try to speed up the loading of small collections (#33570) 
- issue: #33569

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-06-07 08:25:53 +08:00
cai.zhang
27cc9f2630
enhance: Support analyze data (#33651) 
issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
 2024-06-06 17:37:51 +08:00
cai.zhang
cfea3f43cf
fix: Don't sync L0 segments to channel watcher (#33664) 
issue: #33540

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-06 15:59:50 +08:00
XuanYang-cn
4dd0c54ca0
fix: Fix l0 compactor may cause DN from OOM (#33554) 
See also: #33547

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-06 14:33:52 +08:00
congqixia
f6e251514f
fix: Write back dbid modification for nonDB id collection (#33641) 
See also #33608

Make `fixDefaultDBIDConsistency` also write back collection dbid
modification when nonDB id collection is found.

This fix shall prevent dropped collections of this kind show up again
after dropping and restart.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-06 14:29:53 +08:00
wei liu
b69740c8f3
enhance: Remove unnecessary log info during load segment (#33663) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-06 14:13:50 +08:00
XuanYang-cn
68c9e7db8c
fix: Sync dropped segment for dropped partition (#33331) 
See also: #33330

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-06 10:25:52 +08:00
sre-ci-robot
fd191dd7db
[automated] Update Knowhere Commit (#33655) 
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-06-06 01:53:50 +08:00
cai.zhang
feeb869ff9
enhance: Remove compaction plans on the datanode (#33548) 
issue: #33546

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-05 20:27:51 +08:00