congqixia
cb7f2fa6fd
enhance: Use v2 package name for pkg module ( #39990 )
...
Related to #39095
https://go.dev/doc/modules/version-numbers
Update pkg version according to golang dep version convention
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-22 23:15:58 +08:00
Cai Yudong
7476eb3625
feat: Support bulk insert for Int8Vector ( #39499 )
...
Issue: #38666
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2025-01-23 10:19:06 +08:00
congqixia
b0bd290a6e
enhance: Use internal json(sonic) to replace std json lib ( #37708 )
...
Related to #35020
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-18 10:46:31 +08:00
yihao.dai
b45cf2d49f
enhance: Add max length check for csv import ( #37077 )
...
1. Add max length check for csv import.
2. Tidy import options.
3. Tidy common import util functions.
issue: https://github.com/milvus-io/milvus/issues/34150
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-25 14:37:29 +08:00
Buqian Zheng
82c5cf2fa2
feat: add bulk insert support for Functions ( #36715 )
...
issue: https://github.com/milvus-io/milvus/issues/35853 and
https://github.com/milvus-io/milvus/issues/35856
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-10-12 17:19:20 +08:00
smellthemoon
fc1bdd4c84
fix: to forbid bulk insert with nullable field in numpy files ( #36246 )
...
#36241
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-14 15:35:07 +08:00
smellthemoon
80a7c78f28
enhance: import supports null in parquet and json formats ( #35558 )
...
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-20 16:50:55 +08:00
nish112022
3948bd4e79
fix: Added check for validating varchar,array max length ( #35499 )
...
issue : https://github.com/milvus-io/milvus/issues/34150
This is for numpy,parquet,json readers.
---------
Signed-off-by: Nischay Yadav <nischay.yadav@ibm.com>
2024-08-20 11:42:55 +08:00
congqixia
3333160b8d
enhance: Fix lint issues from recent PRs ( #34482 )
...
See also #34483
Some lint issues are introduced due to lack of static check run. This PR
fixes these problems.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-09 10:06:24 +08:00
Cai Yudong
9d4535ce0b
enhance: Handle Float16Vector/BFloat16Vector numpy bulk insert as same as BinaryVector ( #33760 )
...
Issue: #22837
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-06-12 17:17:55 +08:00
Cai Yudong
4004e4c545
enhance: Optimize bulk insert unittest ( #33224 )
...
Issue: #22837
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-05-24 10:23:41 +08:00
Cai Yudong
dc89c6f810
enhance: remove duplicated data generation APIs for bulk insert test ( #32889 )
...
Issue: #22837
including following changes:
1. Add API CreateInsertData() and BuildArrayData() in
internal/util/testutil
2. Remove duplicated test APIs from importutilv2 unittest and bulk
insert integration test
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-05-10 15:27:31 +08:00
Cai Yudong
bcdbd1966e
feat: Support sparse float vector bulk insert for binlog/json/parquet ( #32649 )
...
Issue: #22837
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-05-07 18:43:30 +08:00
yihao.dai
4de063ae14
fix: Make the dynamic column optional in parquet import ( #32738 )
...
issue: https://github.com/milvus-io/milvus/issues/32729
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-07 11:21:29 +08:00
yihao.dai
1594122c0a
enhance: Make the dynamic field file optional during numpy import ( #32596 )
...
1. Make the dynamic field file optional during numpy import
2. Add integration importing test with dynamic
3. Disallow file of pk when autoID=true during numpy import
issue: https://github.com/milvus-io/milvus/issues/32542
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-28 19:39:25 +08:00
Cai Yudong
5fc439c600
feat: Bulk insert support fp16/bf16 ( #32157 )
...
Issue: #22837
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-22 10:05:22 +08:00
yihao.dai
31cf849f68
enhance: Support retriving file size from importutilv2.Reader ( #31533 )
...
To reduce the overhead caused by listing the S3 objects, add an
interface to importutil.Reader to retrieve file sizes.
issue: https://github.com/milvus-io/milvus/issues/31532 ,
https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-25 20:29:07 +08:00
cai.zhang
de2c95d00c
enhance: Constraint dynamic field as key-value format ( #31183 )
...
issue: #31051
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-12 12:45:03 +08:00
yihao.dai
c5918290e6
feat: Add import executor and manager for datanode ( #29438 )
...
This PR introduces novel importv2 roles for datanode:
1. Executor: To execute tasks, a import task will be divided into the
following steps: read data -> hash data -> sync data;
2. Manager: To manage all the tasks;
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-31 20:45:04 +08:00
yihao.dai
3d07b6682c
feat: Add import reader for numpy ( #29253 )
...
This PR implements a new numpy reader for import.
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-08 19:42:49 +08:00
