admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-02-02 01:06:41 +08:00
Code Issues Packages Projects Releases Wiki Activity
19,978 Commits 146 Branches 196 Tags
Commit Graph

8854 Commits

Author SHA1 Message Date
yihao.dai
b1e74dc7cb
enhance: [cherry-pick] Decouple compaction from shard (#34157) 
This PR cherry-picks the following commits:

- Implement task limit control logic in datanode.
https://github.com/milvus-io/milvus/pull/32881
- Load bf from storage instead of memory during L0 compaction.
https://github.com/milvus-io/milvus/pull/32913
- Remove dependencies on shards (e.g. SyncSegments, injection).
https://github.com/milvus-io/milvus/pull/33138
- Rename Compaction interface to CompactionV2.
https://github.com/milvus-io/milvus/pull/33858
- Remove the unused residual compaction logic.
https://github.com/milvus-io/milvus/pull/33932

issue: https://github.com/milvus-io/milvus/issues/32809

pr: https://github.com/milvus-io/milvus/pull/32881,
https://github.com/milvus-io/milvus/pull/32913,
https://github.com/milvus-io/milvus/pull/33138,
https://github.com/milvus-io/milvus/pull/33858,
https://github.com/milvus-io/milvus/pull/33932

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-25 20:22:03 +08:00
yihao.dai
3ae1865ffb
fix: Prevent import from generating orphaned files (#34070) (#34071) 
When import failed, mark the import segment as dropped instead of drop
it directly to prevent generating orphaned files.

issue: https://github.com/milvus-io/milvus/issues/34068

pr: https://github.com/milvus-io/milvus/pull/34070

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-25 17:00:04 +08:00
cai.zhang
65050b3117
fix: [pick] Fallback field type when it isn't in request (#33832) (#34133) 
issue: #33432
pr: #33832

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-25 14:56:04 +08:00
cqy123456
f5344abdaf
enhance: [cherry-pick]growing segment support mmap (#34110) 
issue: issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/32633,
https://github.com/milvus-io/milvus/pull/33951,
https://github.com/milvus-io/milvus/pull/33993

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-25 14:52:07 +08:00
XuanYang-cn
a33b68678d
enhance: [cherry-pick] Move compactor into sub package (#34098) 
This PR consists of the following commits:

- enhance: Tidy compactor and remove dup codes (#32198)
- fix: Fix l0 compactor may cause DN from OOM (#33554)
- enhance: Add deltaRowCount in l0 compaction (#33997)
- enhance: enable stream writer in compactions (#32612)
- fix: turn on compression on stream writers (#34067)
- fix: adding blob memory size in binlog serde (#33324)

See also: #32451, #33547, #33998, #31679
pr: #32198, #33554, #33997, #32612

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
 2024-06-25 11:16:02 +08:00
Patrick Weizhi Xu
1168e8c107
enhance: [2.4] support integral type for MV and skip MV if there is only one category (#34005) 
pr: #33161
issue: #29892

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-06-24 19:12:02 +08:00
Jiquan Long
22e6807e9a
feat: support inverted index for array (#33452) (#34053) 
pr: https://github.com/milvus-io/milvus/pull/33184
pr: https://github.com/milvus-io/milvus/pull/33452
pr: https://github.com/milvus-io/milvus/pull/33633
issue: https://github.com/milvus-io/milvus/issues/27704
Co-authored-by: xiaocai2333 <cai.zhang@zilliz.com>

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
 2024-06-24 10:50:03 +08:00
yihao.dai
2fda43e49f
fix: Do compressBinlog to fix logID 0 (#34060) (#34062) 
Do compressBinlog to ensure that reloadFromKV will fill binlogs' logID
after datacoord restarts.

issue: https://github.com/milvus-io/milvus/issues/34059

pr: https://github.com/milvus-io/milvus/pull/34060

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-22 19:42:02 +08:00
wei liu
061a00c58f
enhance: Enable database level replica num and resource groups for loading collection (#33052) (#33981) 
pr: #33052

issue: #30040

This PR introduce two database level props:
1. database.replica.number
2. database.resource_groups

User can set those two database props by AlterDatabase API, then can
load collection without specified replica_num and resource groups. then
it will use database level load param when try to load collections.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-21 16:56:02 +08:00
congqixia
e02a95e3c2
fix: [2.4] Return record with largest timestamp for entires with same PK (#33936) (#34024) 
Cherry-pick from master
pr: #33936
See also #33883

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-21 14:14:01 +08:00
congqixia
891a94ad9e
fix: [2.4] Check nodeID wildcard when removing pkOracle (#33895) (#34020) 
Cherry-pick from master
pr: #33895
See also #33894

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-21 12:04:00 +08:00
jaime
c219dca001
fix: metrics database_num is 0 after restarting rootcoord (#34010) 
issue: https://github.com/milvus-io/milvus/issues/34041

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-21 11:46:02 +08:00
wei liu
7d1d5a838a
fix: Fix GetReplicas API return nil status (#33715) (#34019) 
issue: #33702
pr: #33715

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-21 10:26:02 +08:00
wei liu
fbc8fb3cb2
enhance: Skip return data distribution if no change happen (#32814) (#33985) 
issue: #32813
pr: #32814

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-21 10:24:12 +08:00
XuanYang-cn
e55fee6b04
enhance: Add deltaRowCount in l0 compaction (#33843) 
See also: #33998 
pr: #33997

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-20 19:32:02 +08:00
congqixia
4424c9e5e7
fix: [2.4] Remove loopclosure issue in ChannelManagerImplV2 (#33989) (#34004) 
Cherry-pick from master
pr: #33989
See also #33987

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-20 14:42:00 +08:00
wei liu
a7ae45c91c
enhance: Add trace for bf cost in l0 compactor (#33898) 
pr: #33860

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-20 14:40:15 +08:00
congqixia
c28b69384a
fix: [2.4] Check the correct return error in MultiRemove (#33926) (#33966) 
Cherry-pick from master
pr: #33926
See also #33925

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-20 11:28:02 +08:00
Jiquan Long
032685239f
enhance: fix copying hits of inverted index twice (#33968) (#33975) 
issue: https://github.com/milvus-io/milvus/issues/29793
pr: https://github.com/milvus-io/milvus/pull/33968
The custom `VecCollector` have already transformed the results into
vector of offsets, no need to copy them twice.

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-06-20 10:06:06 +08:00
wei liu
87508c3390
enhance: Avoid to iterate whole segment list for each task's process(#33943) (#33976) 
pr: #33943

when querycoord process segment task, it will try to iterate whole
segment list to checke whether segment is loaded, which cost too much
cpu if there has thousands of segments.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-20 10:00:05 +08:00
presburger
30b70b05d0
fix: fix milvus gpu compile error (#33990) 
NVIDIA RAFT uses customized versions of spdlog and fmt, which conflict
with the versions provided by Conan. Therefore, it is necessary to
isolate the build environment for knowhere to avoid these conflicts.
issue: #33991

Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
 2024-06-20 09:20:00 +08:00
wei liu
2d3c5191b8
enhance: Refine name rule check error msg (#33815) (#33900) 
pr: #33815

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-19 14:26:01 +08:00
presburger
79546a6c25
fix: fix not find arrow::parquet lib (#33941) 
Upgrade the Arrow version and remove unnecessary library link
dependencies.
issue: #33942
pr: https://github.com/milvus-io/milvus/pull/33928

Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
 2024-06-18 13:09:59 +08:00
wei liu
aafa86095f
enhance: Force to reset coord connection for unavailable error (#33910) 
pr: #33908

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-18 11:15:58 +08:00
wei liu
4513569207
enhance: add restful api to trigger component stop (#32076) (#33799) 
issue: #32698
pr: #32076
This PR add two rest api for component stop and status check:
1. `/management/stop?role=querynode` can stop the specified component
2. `/management/check/ready?role=rootcoord` can check whether the target
component is serviceable

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-17 21:53:59 +08:00
Gao
08c096cf55
enhance: Use primitive type for vectorType (#33911) 
issue: #22837 
pr: #33868 

Use primitive type instead of proto enum type for queryHook to recognize

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-06-17 19:05:58 +08:00
congqixia
e8071830fa
fix: [2.4] Prevent use captured iteration variable partitionID (#33907) 
Cherry-pick from master
pr: #33906 
See also #33902

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-17 19:02:00 +08:00
Gao
5fc1370f6f
enhance: [2.4] autoindex for multi data type (#33867) 
issue: #22837 
pr: https://github.com/milvus-io/milvus/pull/33868

- opensource autoindex support
- metric type check for different data types
- autoindex data type for search param

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-06-14 23:26:00 +08:00
cqy123456
562369627d
enhance: [cherry-pick]check index with data type in knowhere api (#33878) 
issue: https://github.com/milvus-io/milvus/issues/22837
related: https://github.com/milvus-io/milvus/pull/33880

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-14 19:45:58 +08:00
jaime
fd1c7b1a1c
enhance: enable flush rate limiter of collection level (#33864) 
pr: #33837

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-14 16:27:57 +08:00
zhagnlu
b071ebe49b
fix: readd timestamp index because segment timestamp not ordered (#33857) 
pr: #33856

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-14 14:34:01 +08:00
chyezh
dd6c982bdb
fix: load operation when segment is on releasing (#33699) 
issue: #30857
pr: #31340

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-14 11:47:57 +08:00
XuanYang-cn
dff7ec7bb2
enhance: Donot mark compaction tasks to be timeouted (#33833) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-14 10:45:58 +08:00
XuanYang-cn
a446e754b4
fix: [2.4]DeleteData merge wrong data casuing data loss (#33821) 
See also: #33819
pr: #33820

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-13 16:07:56 +08:00
wei liu
25d8b74f71
enhance: Execute bloom filter apply in parallel to speed up segment predict (#33793) 
issue: #33610
pr: #33792

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-13 14:14:04 +08:00
Cai Yudong
ebd0af14f4
enhance: Handle Float16Vector/BFloat16Vector numpy bulk insert as same as BinaryVector (#33760) (#33788) 
pr: #33760
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-06-13 10:49:57 +08:00
congqixia
86f3433053
enhance: [2.4]Use fastjson lib for unmarshal delete log (#33787) (#33802) 
Cherry-pick from master
pr: #33878
```
goos: linux
goarch: amd64
GOMAXPROC=1
cpu: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
BenchmarkJsonSerdeStd             343872              3568 ns/op            1335 B/op         25 allocs/op
BenchmarkJsonSerdeFastjson       5124177               234.9 ns/op            16 B/op          1 allocs/op
```

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-13 10:27:57 +08:00
congqixia
efd1fa8b8a
fix: [2.4] Prevent restart timetick sender creating ut datanode (#33790) (#33801) 
Cherry-pick from master
pr: #33790
See also #33789

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-13 10:03:57 +08:00
chyezh
74b4bd1c77
fix: lost DroppedAt field when DropCollection (#33767) 
issue: #33097
pr: #33766

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-12 20:19:56 +08:00
Gao
7ef28927e6
fix: [2.4] fix binary vector data size (#33751) 
issue #22837 
pr: #33750

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-06-12 20:11:55 +08:00
wei liu
54feef30e7
enhance: Use BatchPkExist to reduce bloom filter func call cost (#33752) 
issue: #33610
pr: #33611

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-12 17:45:58 +08:00
SimFG
f664b51ebe
enhance: [2.4] try to speed up the loading of small collections (#33746) 
- issue: #33569
- pr: #33570

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-06-11 15:07:55 +08:00
SimFG
c331aa4ad3
enhance: [2.4] add the includeCurrentMsg param for the Seek method (#33743) 
/kind improvement

- issue: #33325
- pr: #33326

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-06-11 15:01:55 +08:00
congqixia
ee22750104
fix: [2.4] Write back dbid modification for nonDB id collection (#33641) (#33694) 
Cherry-pick from master
pr: #33641
See also #33608

Make `fixDefaultDBIDConsistency` also write back collection dbid
modification when nonDB id collection is found.

This fix shall prevent dropped collections of this kind show up again
after dropping and restart.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-11 11:25:55 +08:00
yihao.dai
396f8608dd
fix: Fix multiple vector fields import (#33723) (#33724) 
1. Fix dim mismatch with multi-vector fields and JSON import
2. Enhance: do not display file ID in GetImportResponse.

issue: https://github.com/milvus-io/milvus/issues/33681,
https://github.com/milvus-io/milvus/issues/33682

pr: https://github.com/milvus-io/milvus/pull/33723

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-10 21:55:55 +08:00
yihao.dai
b71a404776
fix: Check if the import job exists (#33672) (#33673) 
issue: https://github.com/milvus-io/milvus/issues/33671

pr: https://github.com/milvus-io/milvus/pull/33672

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-10 21:50:29 +08:00
yihao.dai
ed1dee9e38
enhance: Support L0 import (#33514) (#33712) 
issue: https://github.com/milvus-io/milvus/issues/33157

pr: https://github.com/milvus-io/milvus/pull/33514

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-08 11:17:52 +08:00
Buqian Zheng
39e341e83a
fix: [2.4] update check for sparse hnsw index (#33714) 
issue: #29419
pr: #33713

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-06-07 18:36:07 +08:00
zhagnlu
729001c3a8
fix: clean vector memory (#33692) (#33708) 
pr: #33692

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-07 18:17:53 +08:00
chyezh
93348af5c0
fix: async warmup will be blocked by state lock (#33687) 
issue: #33685
pr: #33686

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-07 14:23:54 +08:00