yihao.dai
5e525eb3bf
enhance: Retry reads from object storage on rate limit error ( #46455 )
...
This PR improves the robustness of object storage operations by retrying
both explicit throttling errors (e.g. HTTP 429, SlowDown, ServerBusy).
These errors commonly occur under high concurrency and are typically
recoverable with bounded retries.
issue: https://github.com/milvus-io/milvus/issues/44772
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Configurable retry support for reads from object storage and improved
mapping of transient/rate-limit errors.
* Added a retryable reader wrapper used by CSV/JSON/Parquet/Numpy import
paths.
* **Configuration**
* New parameter to control storage read retry attempts.
* **Tests**
* Expanded unit tests covering error mapping and retry behaviors across
storage backends.
* Standardized mock readers and test initialization to simplify test
setups.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-12-23 11:03:18 +08:00
groot
a545ebc702
fix: Fix a bug that bulkimport cannot handle empty struct list ( #45693 )
...
issue: https://github.com/milvus-io/milvus/issues/42148
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2025-11-25 17:21:06 +08:00
zhenshan.cao
a3b8bcb198
fix: correct default value backfill during AddField ( #45634 )
...
issue: https://github.com/milvus-io/milvus/issues/44585
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2025-11-18 23:05:42 +08:00
groot
e48fe7f820
fix: Fix bulkimport bug for Struct field ( #45474 )
...
issue: https://github.com/milvus-io/milvus/issues/45006
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2025-11-13 11:31:41 +08:00
zhenshan.cao
490a618c30
fix: Handle timestamptz import errors ( #45287 )
...
issue: https://github.com/milvus-io/milvus/issues/44585
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2025-11-05 15:05:33 +08:00
cai.zhang
c33d221536
fix: Fix bug for importing Geometry data ( #45089 )
...
issue: #44787 , #45012
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-10-27 20:34:11 +08:00
Spade A
d8591f9548
fix: csv/json import with STRUCT adapts concatenated struct name ( #45000 )
...
After https://github.com/milvus-io/milvus/pull/44557 , the field name in
STRUCT field becomes STRUCT_NAME[FIELD_NAME]
This PR make import consider the change.
issue: https://github.com/milvus-io/milvus/issues/45006
ref: https://github.com/milvus-io/milvus/issues/42148
TODO: parquet is much more complex than csv/json, and I will leave it to
a separate PR.
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
2025-10-24 10:22:15 +08:00
cai.zhang
d5ecb63f53
enhance: Support import geometry data by json/csv ( #44826 )
...
issue: #44787
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-10-17 17:08:02 +08:00
Bingyi Sun
c25166a202
fix: Fix bulk import with autoid ( #44604 )
...
issue: #44424
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-10-09 12:09:56 +08:00
Bingyi Sun
96e1de4e22
feat: allow users to write pk field when autoid is enabled ( #44424 )
...
https://github.com/milvus-io/milvus/issues/44425
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-09-23 16:10:04 +08:00
Spade A
eb793531b9
feat: impl StructArray -- support import for CSV/JSON/PARQUET/BINLOG ( #44201 )
...
Ref https://github.com/milvus-io/milvus/issues/42148
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
2025-09-15 20:41:59 +08:00
Tianx
c0d62268ac
feat: add timesatmptz data type ( #44005 )
...
issue: https://github.com/milvus-io/milvus/issues/27467
>
https://github.com/milvus-io/milvus/issues/27467#issuecomment-3092211420
> * [x] M1 Create collection with timestamptz field
> * [x] M2 Insert timestamptz field data
> * [x] M3 Retrieve timestamptz field data
> * [x] M4 Implement handoff[ ]
The second PR of issue:
https://github.com/milvus-io/milvus/issues/27467 , which completes M1-M4
described above.
---------
Signed-off-by: xtx <xtianx@smail.nju.edu.cn>
2025-08-26 15:59:53 +08:00
groot
1ee8cea35b
enhance: bulkinsert handle nullable/defaultValue/functionOutput fields ( #42956 )
...
issue: https://github.com/milvus-io/milvus/issues/42173
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2025-07-04 14:20:44 +08:00
yihao.dai
6c1a37fca1
fix: Fix import reader goroutine leak ( #41869 )
...
Close the chunk manager's reader after the import completes to prevent
goroutine leaks.
issues: https://github.com/milvus-io/milvus/issues/41868
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-05-16 10:18:35 +08:00
SimFG
91d40fa558
fix: Update logging context and upgrade dependencies ( #41318 )
...
- issue: #41291
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-04-23 10:52:38 +08:00
yihao.dai
b4cb8a4b13
enhance: Add UTF-8 string validation for import ( #40694 )
...
issue: https://github.com/milvus-io/milvus/issues/40684
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-04-01 19:04:21 +08:00
yihao.dai
bab30a41bf
enhance: Improve import error msgs ( #40567 )
...
issue: https://github.com/milvus-io/milvus/issues/40208
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-13 21:02:07 +08:00
Xiaofan
fb48b3c7ac
fix: empty sparse row in importer ( #40585 )
...
fix #40584
parquet bulk writer can not finish 0 dim sparse vector.
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2025-03-13 01:29:41 +08:00
jaime
c8a96377bb
enhance: move object storage client creation to pkg package ( #40440 )
...
issue: #40439
Signed-off-by: jaime <yun.zhang@zilliz.com>
2025-03-12 20:38:07 +08:00
congqixia
cb7f2fa6fd
enhance: Use v2 package name for pkg module ( #39990 )
...
Related to #39095
https://go.dev/doc/modules/version-numbers
Update pkg version according to golang dep version convention
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-22 23:15:58 +08:00
Cai Yudong
7476eb3625
feat: Support bulk insert for Int8Vector ( #39499 )
...
Issue: #38666
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2025-01-23 10:19:06 +08:00
congqixia
b0bd290a6e
enhance: Use internal json(sonic) to replace std json lib ( #37708 )
...
Related to #35020
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-18 10:46:31 +08:00
yihao.dai
b45cf2d49f
enhance: Add max length check for csv import ( #37077 )
...
1. Add max length check for csv import.
2. Tidy import options.
3. Tidy common import util functions.
issue: https://github.com/milvus-io/milvus/issues/34150
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-25 14:37:29 +08:00
Buqian Zheng
82c5cf2fa2
feat: add bulk insert support for Functions ( #36715 )
...
issue: https://github.com/milvus-io/milvus/issues/35853 and
https://github.com/milvus-io/milvus/issues/35856
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-10-12 17:19:20 +08:00
OxalisCu
3a381bc247
enhance: Bulkinsert supports null in csv formats ( #35912 )
...
see details in this issue
https://github.com/milvus-io/milvus/issues/35911
---------
Signed-off-by: OxalisCu <2127298698@qq.com>
2024-09-09 19:17:07 +08:00
Xiaofan
50fcfe8ef1
enhance: add nan and inf check ( #35683 )
...
fix #35594
add float check on files
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-08-25 15:22:57 +08:00
OxalisCu
ed4eaffc9d
enhance: add csv support for bulkinsert ( #34938 )
...
See this issue for details: #34937
---------
Signed-off-by: OxalisCu <2127298698@qq.com>
2024-08-21 17:47:01 +08:00
