6 Commits

Author	SHA1	Message	Date
yihao.dai	31cf849f68	enhance: Support retriving file size from importutilv2.Reader (#31533) ... To reduce the overhead caused by listing the S3 objects, add an interface to importutil.Reader to retrieve file sizes. issue: https://github.com/milvus-io/milvus/issues/31532, https://github.com/milvus-io/milvus/issues/28521 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-03-25 20:29:07 +08:00
yihao.dai	776709e5ff	fix: Fix binlog import (#31310) ... Fix binlog import functionality by removing the existing check and refining the size retrieval process. issue: https://github.com/milvus-io/milvus/issues/31221, https://github.com/milvus-io/milvus/issues/28521 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-03-17 20:59:04 +08:00
yihao.dai	811316d2ba	fix: Fix binlog import and refine error reporting (#31241) ... 1. Fix binlog import with partition key. 2. Refine binlog import error reportins. 3. Avoid division by zero when retrieving import progress. issue: https://github.com/milvus-io/milvus/issues/31221, https://github.com/milvus-io/milvus/issues/28521 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-03-15 10:55:05 +08:00
yihao.dai	b5c67948b7	enhance: Enhance and modify the return content of ImportV2 (#31192) ... 1. The Import APIs now provide detailed progress information for each imported file, including details such as file name, file size, progress, and more. 2. The APIs now return the collection name and the completion time. 3. Other modifications include changing jobID to jobId and other similar adjustments. issue: https://github.com/milvus-io/milvus/issues/28521 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-03-13 19:51:03 +08:00
yihao.dai	a434d33e75	feat: Add import scheduler and manager (#29367) ... This PR introduces novel managerial roles for importv2: 1. ImportMeta: To manage all the import tasks; 2. ImportScheduler: To process tasks and modify their states; 3. ImportChecker: To ascertain the completion of all tasks and instigate relevant operations. issue: https://github.com/milvus-io/milvus/issues/28521 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-03-01 18:31:02 +08:00
yihao.dai	c5918290e6	feat: Add import executor and manager for datanode (#29438) ... This PR introduces novel importv2 roles for datanode: 1. Executor: To execute tasks, a import task will be divided into the following steps: read data -> hash data -> sync data; 2. Manager: To manage all the tasks; issue: https://github.com/milvus-io/milvus/issues/28521 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-01-31 20:45:04 +08:00