wei liu 7aed88113c

enhance: Deduplicate primary keys in upsert request batch (#45249) ...

issue: #44320

This change adds deduplication logic to handle duplicate primary keys
within a single upsert batch, keeping the last occurrence of each
primary key.

Key changes:
- Add DeduplicateFieldData function to remove duplicate PKs from field
data, supporting both Int64 and VarChar primary keys
- Refactor fillFieldPropertiesBySchema into two separate functions:
validateFieldDataColumns for validation and fillFieldPropertiesOnly for
property filling, improving code clarity and reusability
- Integrate deduplication logic in upsertTask.PreExecute to
automatically deduplicate data before processing
- Add comprehensive unit tests for deduplication with various PK types
(Int64, VarChar) and field types (scalar, vector)
- Add Python integration tests to verify end-to-end behavior

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>

2025-11-17 21:35:40 +08:00

..

accesslog

enhance: Add accesslog field for template value length info (#44723)

2025-10-11 18:23:57 +08:00

connection

fix: Update logging context and upgrade dependencies (#41318)

2025-04-23 10:52:38 +08:00

privilege

enhance: Refactor privilege management by extracting privilege cache into separate package (#44762)

2025-10-13 11:15:58 +08:00

replicate

fix: wrong execution order of DDL/DCL on secondary (#44886)

2025-10-21 22:38:05 +08:00

shardclient

enhance: forbid build analyzer at proxy (#44067)

2025-10-23 10:58:12 +08:00

.mockery.yaml

enhance: standardize mockery config for proxy package (#45030)

2025-10-22 16:00:05 +08:00

authentication_interceptor_test.go

enhance: skip check source id (#45377)

2025-11-07 15:19:34 +08:00

authentication_interceptor.go

enhance: skip check source id (#45377)

2025-11-07 15:19:34 +08:00

cgo_util_test.go

feat: impl StructArray -- support more types of vector in STRUCT (#44736)

2025-10-15 10:25:59 +08:00

cgo_util.go

feat: impl StructArray -- support create index for vector array (embedding list) and search on it (#43726)

2025-08-20 10:27:46 +08:00

channels_mgr_test.go

enhance: remove old arch non-streaming arch code (#43651)

2025-08-06 14:41:40 +08:00

channels_mgr.go

enhance: remove old arch non-streaming arch code (#43651)

2025-08-06 14:41:40 +08:00

channels_time_ticker_test.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

channels_time_ticker.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

condition_test.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

condition.go

Propagate task ctx done err for proxy (#27892)

2023-10-26 19:06:13 +08:00

count_reducer_test.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

count_reducer.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

data_coord_mock_test.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

database_interceptor_test.go

feat:add BatchDescribeCollection interface (#43786)

2025-08-18 01:23:45 +08:00

database_interceptor.go

feat:add BatchDescribeCollection interface (#43786)

2025-08-18 01:23:45 +08:00

default_limit_reducer.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

dummyreq_test.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

dummyreq.go

enhance: Use internal json(sonic) to replace std json lib (#37708)

2024-11-18 10:46:31 +08:00

function_task_test.go

feat: Support add_function/alter_function/drop_function (#44895)

2025-11-13 20:53:39 +08:00

function_task.go

feat: Support add_function/alter_function/drop_function (#44895)

2025-11-13 20:53:39 +08:00

hook_interceptor_test.go

enhance: set the rpc error code to avoid the invalid retry (#37024)

2024-10-22 10:33:26 +08:00

hook_interceptor.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

http_req_impl_test.go

fix: Hide credential of embedding/reranking providers in WebUI (#44275)

2025-09-10 16:39:57 +08:00

http_req_impl.go

fix: Hide credential of embedding/reranking providers in WebUI (#44275)

2025-09-10 16:39:57 +08:00

impl_test.go

feat: Support add_function/alter_function/drop_function (#44895)

2025-11-13 20:53:39 +08:00

impl.go

feat: Support add_function/alter_function/drop_function (#44895)

2025-11-13 20:53:39 +08:00

interface_def.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

management_test.go

enhance: Merge RootCoord, DataCoord And QueryCoord into MixCoord (#41006)

2025-04-11 16:36:30 +08:00

management.go

enhance: Merge RootCoord, DataCoord And QueryCoord into MixCoord (#41006)

2025-04-11 16:36:30 +08:00

meta_cache_test.go

enhance: extract shard client logic into dedicated package (#45018)

2025-10-22 10:22:04 +08:00

meta_cache_testonly.go

enhance: extract shard client logic into dedicated package (#45018)

2025-10-22 10:22:04 +08:00

meta_cache.go

enhance: extract shard client logic into dedicated package (#45018)

2025-10-22 10:22:04 +08:00

metrics_info_test.go

enhance: Add proxy task queue metrics (#42156)

2025-06-04 11:26:32 +08:00

metrics_info.go

enhance: Add proxy task queue metrics (#42156)

2025-06-04 11:26:32 +08:00

mock_cache.go

enhance: standardize mockery config for proxy package (#45030)

2025-10-22 16:00:05 +08:00

mock_channels_manager.go

enhance: remove old arch non-streaming arch code (#43651)

2025-08-06 14:41:40 +08:00

mock_msgstream_test.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

mock_test.go

feat: impl StructArray -- create schema, insert, and retrieve data (#42855)

2025-07-27 01:30:55 +08:00

mock_tso_test.go

fix: mockery too unavailable after upgrade golang version (#41481)

2025-04-24 10:46:43 +08:00

msg_pack_test.go

enhance: extract shard client logic into dedicated package (#45018)

2025-10-22 10:22:04 +08:00

msg_pack.go

fix: empty internal InsertMsg caues panic (#44903)

2025-10-17 10:28:01 +08:00

OWNERS

[skip ci]Update OWNERS files (#11898)

2021-11-16 15:41:11 +08:00

privilege_interceptor_test.go

enhance: extract shard client logic into dedicated package (#45018)

2025-10-22 10:22:04 +08:00

privilege_interceptor.go

enhance: Refactor privilege management by extracting privilege cache into separate package (#44762)

2025-10-13 11:15:58 +08:00

proxy_rpc_test.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

proxy_test.go

fix: csv/json import with STRUCT adapts concatenated struct name (#45000)

2025-10-24 10:22:15 +08:00

proxy.go

enhance: extract shard client logic into dedicated package (#45018)

2025-10-22 10:22:04 +08:00

rate_limit_interceptor_test.go

feat: add DDLDB rate type and related quota configurations (#40651)

2025-03-14 15:38:09 +08:00

rate_limit_interceptor.go

feat: add DDLDB rate type and related quota configurations (#40651)

2025-03-14 15:38:09 +08:00

reducer_test.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

reducer.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

repack_func_test.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

repack_func.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

rootcoord_mock_test.go

feat: Support add_function/alter_function/drop_function (#44895)

2025-11-13 20:53:39 +08:00

rpc_msg.go

Add slow log for read (#23272)

2023-04-14 15:30:29 +08:00

search_pipeline_test.go

feat: Support zilliz models (#45168)

2025-11-13 12:55:37 +08:00

search_pipeline.go

enhance: extract shard client logic into dedicated package (#45018)

2025-10-22 10:22:04 +08:00

search_reduce_util_test.go

fix: Handle empty FieldsData in reduce/rerank for requery scenario (#44917)

2025-10-21 19:42:03 +08:00

search_reduce_util.go

fix: Group value is nil(#45418) (#45422)

2025-11-08 10:29:33 +08:00

search_util.go

fix: Fix bugs related to TimestampTz (#45111)

2025-11-04 16:51:33 +08:00

service_provider_test.go

enhance: extract shard client logic into dedicated package (#45018)

2025-10-22 10:22:04 +08:00

service_provider.go

feat: impl StructArray -- support same names in different STRUCT (#44557)

2025-10-10 15:53:56 +08:00

simple_rate_limiter_test.go

feat: support to deny dll according to database property (#40764)

2025-03-23 11:18:28 +08:00

simple_rate_limiter.go

feat: support to deny dll according to database property (#40764)

2025-03-23 11:18:28 +08:00

task_alias_test.go

enhance: Merge RootCoord, DataCoord And QueryCoord into MixCoord (#41006)

2025-04-11 16:36:30 +08:00

task_alias.go

enhance: remove old arch non-streaming arch code (#43651)

2025-08-06 14:41:40 +08:00

task_database_test.go

enhance: Merge RootCoord, DataCoord And QueryCoord into MixCoord (#41006)

2025-04-11 16:36:30 +08:00

task_database.go

fix: Fix bugs related to TimestampTz (#45111)

2025-11-04 16:51:33 +08:00

task_delete_streaming.go

feat: support use cipher hook in streaming node (#40562)

2025-08-19 10:41:44 +08:00

task_delete_test.go

enhance: extract shard client logic into dedicated package (#45018)

2025-10-22 10:22:04 +08:00

task_delete.go

fix: Fix bugs related to TimestampTz (#45111)

2025-11-04 16:51:33 +08:00

task_flush_all_streaming_test.go

enhance: Add granular flush targets support for FlushAll operation (#44234)

2025-09-19 18:38:01 +08:00

task_flush_all_streaming.go

enhance: Add granular flush targets support for FlushAll operation (#44234)

2025-09-19 18:38:01 +08:00

task_flush_all_test.go

enhance: remove old arch non-streaming arch code (#43651)

2025-08-06 14:41:40 +08:00

task_flush_all.go

enhance: Add granular flush targets support for FlushAll operation (#44234)

2025-09-19 18:38:01 +08:00

task_flush_streaming.go

enhance: Add granular flush targets support for FlushAll operation (#44234)

2025-09-19 18:38:01 +08:00

task_flush.go

enhance: remove old arch non-streaming arch code (#43651)

2025-08-06 14:41:40 +08:00

task_import.go

enhance: Merge RootCoord, DataCoord And QueryCoord into MixCoord (#41006)

2025-04-11 16:36:30 +08:00

task_index_test.go

fix: fix index compatibility after upgrade (#45373)

2025-11-13 12:59:38 +08:00

task_index.go

fix: remove validateFieldName in dropIndex (#45460)

2025-11-14 10:17:37 +08:00

task_insert_streaming.go

feat: support use cipher hook in streaming node (#40562)

2025-08-19 10:41:44 +08:00

task_insert_test.go

feat: Geospatial Data Type and GIS Function support for milvus (#44547)

2025-09-28 19:43:05 +08:00

task_insert.go

enhance: Deduplicate primary keys in upsert request batch (#45249)

2025-11-17 21:35:40 +08:00

task_policies_test.go

enhance: extract shard client logic into dedicated package (#45018)

2025-10-22 10:22:04 +08:00

task_policies.go

enhance: extract shard client logic into dedicated package (#45018)

2025-10-22 10:22:04 +08:00

task_query_namespace_test.go

enhance: add namespace for query and search request (#44343)

2025-10-16 17:52:01 +08:00

task_query_test.go

enhance: extract shard client logic into dedicated package (#45018)

2025-10-22 10:22:04 +08:00

task_query.go

fix: timezone parameter ingored in Search/Query (#45320)

2025-11-05 19:19:33 +08:00

task_scheduler_test.go

enhance: remove old arch non-streaming arch code (#43651)

2025-08-06 14:41:40 +08:00

task_scheduler.go

enhance: remove old arch non-streaming arch code (#43651)

2025-08-06 14:41:40 +08:00

task_search_namespace_test.go

enhance: add namespace for query and search request (#44343)

2025-10-16 17:52:01 +08:00

task_search_test.go

enhance: standardize mockery config for proxy package (#45030)

2025-10-22 16:00:05 +08:00

task_search.go

feat: Support zilliz models (#45168)

2025-11-13 12:55:37 +08:00

task_statistic_test.go

enhance: extract shard client logic into dedicated package (#45018)

2025-10-22 10:22:04 +08:00

task_statistic.go

enhance: extract shard client logic into dedicated package (#45018)

2025-10-22 10:22:04 +08:00

task_test.go

enhance: extract shard client logic into dedicated package (#45018)

2025-10-22 10:22:04 +08:00

task_upsert_streaming.go

feat: support use cipher hook in streaming node (#40562)

2025-08-19 10:41:44 +08:00

task_upsert_test.go

enhance: Deduplicate primary keys in upsert request batch (#45249)

2025-11-17 21:35:40 +08:00

task_upsert.go

enhance: Deduplicate primary keys in upsert request batch (#45249)

2025-11-17 21:35:40 +08:00

task_validator_test.go

enhance: disabling max entry limit by default (#43166)

2025-07-08 10:10:46 +08:00

task_validator.go

enhance: disabling max entry limit by default (#43166)

2025-07-08 10:10:46 +08:00

task.go

feat: Support add_function/alter_function/drop_function (#44895)

2025-11-13 20:53:39 +08:00

timestamp_test.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

timestamp.go

fix: Update logging context and upgrade dependencies (#41318)

2025-04-23 10:52:38 +08:00

trace_log_interceptor_test.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

trace_log_interceptor.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

type_def.go

[skip e2e] Add comment for pChan (#14098)

2021-12-23 21:44:12 +08:00

util_test.go

enhance: Deduplicate primary keys in upsert request batch (#45249)

2025-11-17 21:35:40 +08:00

util.go

enhance: Deduplicate primary keys in upsert request batch (#45249)

2025-11-17 21:35:40 +08:00

validate_util_test.go

enhance: Add nullable support for Geometry and Timestamptz types (#44846)

2025-10-15 14:04:00 +08:00

validate_util.go

fix: Fix bugs related to TimestampTz (#45111)

2025-11-04 16:51:33 +08:00