11 Commits

Author SHA1 Message Date
yihao.dai
a61668c77e
feat: Introduce stats task for import (#35868)
This PR introduce stats task for import:
1. Define new `Stats` and `IndexBuilding` states for importJob
2. Add new stats step to the import process: trigger the stats task and
wait for its completion
3. Abort stats task if import job failed

issue: https://github.com/milvus-io/milvus/issues/33744

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-15 15:17:08 +08:00
OxalisCu
ed4eaffc9d
enhance: add csv support for bulkinsert (#34938)
See this issue for details: #34937

---------

Signed-off-by: OxalisCu <2127298698@qq.com>
2024-08-21 17:47:01 +08:00
wei liu
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 (#34394)
issue: #34252

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
Cai Yudong
9d4535ce0b
enhance: Handle Float16Vector/BFloat16Vector numpy bulk insert as same as BinaryVector (#33760)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-06-12 17:17:55 +08:00
yihao.dai
3540eee977
enhance: Support L0 import (#33514)
issue: https://github.com/milvus-io/milvus/issues/33157

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-07 14:17:20 +08:00
yihao.dai
35532a3e7d
fix: Fill stats log id and check validity (#33477)
1. Fill log ID of stats log from import
2. Add a check to validate the log ID before writing to meta

issue: https://github.com/milvus-io/milvus/issues/33476

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-05 11:17:56 +08:00
Cai Yudong
4ef163fb70
enhance: Support readable JSON file import for Float16/BFloat16/SparseFloat (#33064)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-05-16 14:47:35 +08:00
Cai Yudong
bcdbd1966e
feat: Support sparse float vector bulk insert for binlog/json/parquet (#32649)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-05-07 18:43:30 +08:00
Cai Yudong
5fc439c600
feat: Bulk insert support fp16/bf16 (#32157)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-22 10:05:22 +08:00
yihao.dai
c408a32db6
feat: Add disk quota checks for import V2 (#31131)
Return quota error when the files to be imported exceed the disk quota.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-15 14:43:03 +08:00
yihao.dai
a434d33e75
feat: Add import scheduler and manager (#29367)
This PR introduces novel managerial roles for importv2:
1. ImportMeta: To manage all the import tasks;
2. ImportScheduler: To process tasks and modify their states;
3. ImportChecker: To ascertain the completion of all tasks and instigate
relevant operations.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-01 18:31:02 +08:00