yihao.dai
5e525eb3bf
enhance: Retry reads from object storage on rate limit error ( #46455 )
...
This PR improves the robustness of object storage operations by retrying
both explicit throttling errors (e.g. HTTP 429, SlowDown, ServerBusy).
These errors commonly occur under high concurrency and are typically
recoverable with bounded retries.
issue: https://github.com/milvus-io/milvus/issues/44772
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Configurable retry support for reads from object storage and improved
mapping of transient/rate-limit errors.
* Added a retryable reader wrapper used by CSV/JSON/Parquet/Numpy import
paths.
* **Configuration**
* New parameter to control storage read retry attempts.
* **Tests**
* Expanded unit tests covering error mapping and retry behaviors across
storage backends.
* Standardized mock readers and test initialization to simplify test
setups.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-12-23 11:03:18 +08:00
Spade A
d8591f9548
fix: csv/json import with STRUCT adapts concatenated struct name ( #45000 )
...
After https://github.com/milvus-io/milvus/pull/44557 , the field name in
STRUCT field becomes STRUCT_NAME[FIELD_NAME]
This PR make import consider the change.
issue: https://github.com/milvus-io/milvus/issues/45006
ref: https://github.com/milvus-io/milvus/issues/42148
TODO: parquet is much more complex than csv/json, and I will leave it to
a separate PR.
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
2025-10-24 10:22:15 +08:00
cai.zhang
d5ecb63f53
enhance: Support import geometry data by json/csv ( #44826 )
...
issue: #44787
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-10-17 17:08:02 +08:00
Spade A
eb793531b9
feat: impl StructArray -- support import for CSV/JSON/PARQUET/BINLOG ( #44201 )
...
Ref https://github.com/milvus-io/milvus/issues/42148
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
2025-09-15 20:41:59 +08:00
groot
1ee8cea35b
enhance: bulkinsert handle nullable/defaultValue/functionOutput fields ( #42956 )
...
issue: https://github.com/milvus-io/milvus/issues/42173
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2025-07-04 14:20:44 +08:00
SimFG
91d40fa558
fix: Update logging context and upgrade dependencies ( #41318 )
...
- issue: #41291
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-04-23 10:52:38 +08:00
yihao.dai
b4cb8a4b13
enhance: Add UTF-8 string validation for import ( #40694 )
...
issue: https://github.com/milvus-io/milvus/issues/40684
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-04-01 19:04:21 +08:00
yihao.dai
bab30a41bf
enhance: Improve import error msgs ( #40567 )
...
issue: https://github.com/milvus-io/milvus/issues/40208
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-13 21:02:07 +08:00
congqixia
cb7f2fa6fd
enhance: Use v2 package name for pkg module ( #39990 )
...
Related to #39095
https://go.dev/doc/modules/version-numbers
Update pkg version according to golang dep version convention
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-22 23:15:58 +08:00
smellthemoon
92a2d608ac
fix: Bulk insert failed when the nullable/default_value field is not exist ( #39063 )
...
#39036
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-01-09 19:27:03 +08:00
congqixia
b0bd290a6e
enhance: Use internal json(sonic) to replace std json lib ( #37708 )
...
Related to #35020
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-18 10:46:31 +08:00
smellthemoon
80a7c78f28
enhance: import supports null in parquet and json formats ( #35558 )
...
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-20 16:50:55 +08:00
nish112022
3948bd4e79
fix: Added check for validating varchar,array max length ( #35499 )
...
issue : https://github.com/milvus-io/milvus/issues/34150
This is for numpy,parquet,json readers.
---------
Signed-off-by: Nischay Yadav <nischay.yadav@ibm.com>
2024-08-20 11:42:55 +08:00
yihao.dai
1b5554c8cb
enhance: Support $meta key for json import ( #32013 )
...
During JSON import:
1. Allow the specification of the $meta key
2. Prohibit duplicated keys within the $meta field, for instance,
`{"id": 1, "vector": [], "x": 6, "$meta": {"x": 8}}`
issue: https://github.com/milvus-io/milvus/issues/31835
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-10 17:27:17 +08:00
