sthuang
90acc8a58f
enhance: upgrade go arrow version from 12.0.1 to 17.0.0 ( #39916 )
...
related: https://github.com/milvus-io/milvus/issues/39915
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-02-25 10:30:02 +08:00
congqixia
cb7f2fa6fd
enhance: Use v2 package name for pkg module ( #39990 )
...
Related to #39095
https://go.dev/doc/modules/version-numbers
Update pkg version according to golang dep version convention
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-22 23:15:58 +08:00
yihao.dai
b45cf2d49f
enhance: Add max length check for csv import ( #37077 )
...
1. Add max length check for csv import.
2. Tidy import options.
3. Tidy common import util functions.
issue: https://github.com/milvus-io/milvus/issues/34150
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-25 14:37:29 +08:00
smellthemoon
80a7c78f28
enhance: import supports null in parquet and json formats ( #35558 )
...
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-20 16:50:55 +08:00
yihao.dai
4de063ae14
fix: Make the dynamic column optional in parquet import ( #32738 )
...
issue: https://github.com/milvus-io/milvus/issues/32729
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-07 11:21:29 +08:00
yihao.dai
31cf849f68
enhance: Support retriving file size from importutilv2.Reader ( #31533 )
...
To reduce the overhead caused by listing the S3 objects, add an
interface to importutil.Reader to retrieve file sizes.
issue: https://github.com/milvus-io/milvus/issues/31532 ,
https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-25 20:29:07 +08:00
yihao.dai
c5918290e6
feat: Add import executor and manager for datanode ( #29438 )
...
This PR introduces novel importv2 roles for datanode:
1. Executor: To execute tasks, a import task will be divided into the
following steps: read data -> hash data -> sync data;
2. Manager: To manage all the tasks;
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-31 20:45:04 +08:00
yihao.dai
156a0dd450
feat: Add import reader for Parquet ( #29618 )
...
This PR implements a Parquet reader for import.
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-07 19:38:49 +08:00
