10 Commits

Author SHA1 Message Date
yihao.dai
3540eee977
enhance: Support L0 import (#33514)
issue: https://github.com/milvus-io/milvus/issues/33157

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-07 14:17:20 +08:00
Cai Yudong
4004e4c545
enhance: Optimize bulk insert unittest (#33224)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-05-24 10:23:41 +08:00
yihao.dai
9ff023ee35
fix: Fix filtering by partition key fails for importing data (#33274)
Before executing the import, partition IDs should be reordered according
to partition names. Otherwise, the data might be hashed to the wrong
partition during import. This PR corrects this error.

issue: https://github.com/milvus-io/milvus/issues/33237

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-23 11:13:40 +08:00
Cai Yudong
b560602885
enhance: Store SparseFloatVector into parquet as JSON string (#33101)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-05-17 15:01:37 +08:00
Cai Yudong
4ef163fb70
enhance: Support readable JSON file import for Float16/BFloat16/SparseFloat (#33064)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-05-16 14:47:35 +08:00
Cai Yudong
dc89c6f810
enhance: remove duplicated data generation APIs for bulk insert test (#32889)
Issue: #22837

including following changes:
1. Add API CreateInsertData() and BuildArrayData() in
internal/util/testutil
2. Remove duplicated test APIs from importutilv2 unittest and bulk
insert integration test

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-05-10 15:27:31 +08:00
Cai Yudong
bcdbd1966e
feat: Support sparse float vector bulk insert for binlog/json/parquet (#32649)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-05-07 18:43:30 +08:00
Cai Yudong
5fc439c600
feat: Bulk insert support fp16/bf16 (#32157)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-22 10:05:22 +08:00
Bingyi Sun
5c0bb40549
fix: merge index params when creating index (#31127)
issue: https://github.com/milvus-io/milvus/issues/31102

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-11 17:31:03 +08:00
yihao.dai
a434d33e75
feat: Add import scheduler and manager (#29367)
This PR introduces novel managerial roles for importv2:
1. ImportMeta: To manage all the import tasks;
2. ImportScheduler: To process tasks and modify their states;
3. ImportChecker: To ascertain the completion of all tasks and instigate
relevant operations.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-01 18:31:02 +08:00