XuanYang-cn ca7ec23198

enhance: Use partitionID when delete by partitionKey (#38231) ...

When delete by partition_key, Milvus will generates L0 segments
globally. During L0 Compaction, those L0 segments will touch all
partitions collection wise. Due to the false-positive rate of segment
bloomfilters, L0 compactions will append false deltalogs to completed
irrelevant partitions, which causes *partition deletion amplification.

This PR uses partition_key to set targeted partitionID when producing
deleteMsgs into MsgStreams. This'll narrow down L0 segments scope to
partition level, and remove the false-positive influence
collection-wise.

However, due to DeleteMsg structure, we can only label one partition to
one deleteMsg, so this enhancement fails if user wants to delete over 2
partition_keys in one deletion.

See also: #34665

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

2024-12-20 11:18:46 +08:00

..

accesslog

fix: access log remove new log not the old and skip rotate empty log (#38309)

2024-12-17 19:16:44 +08:00

connection

enhance: timer instead of ticker (#36133)

2024-09-10 15:23:06 +08:00

authentication_interceptor_test.go

fix: Unify hook singleton implementation in proxy (#34887)

2024-07-26 18:07:53 +08:00

authentication_interceptor.go

enhance: report restful info to hook (#38112)

2024-12-05 19:02:41 +08:00

cgo_util_test.go

enhance: add vector index mgr to remove vector index type dependency (#36843)

2024-10-17 22:15:25 +08:00

cgo_util.go

enhance: optimize milvus core building (#35610)

2024-08-23 12:35:02 +08:00

channels_mgr_test.go

enhance: refine pular related mq interfaces (#38007)

2024-12-04 20:50:39 +08:00

channels_mgr.go

enhance: refine pular related mq interfaces (#38007)

2024-12-04 20:50:39 +08:00

channels_time_ticker_test.go

Format the code (#27275)

2023-09-21 09:45:27 +08:00

channels_time_ticker.go

Format the code (#27275)

2023-09-21 09:45:27 +08:00

condition_test.go

Format the code (#27275)

2023-09-21 09:45:27 +08:00

condition.go

Propagate task ctx done err for proxy (#27892)

2023-10-26 19:06:13 +08:00

count_reducer_test.go

Format the code (#27275)

2023-09-21 09:45:27 +08:00

count_reducer.go

feat: add more operation detail info for better allocation (#30438)

2024-03-28 06:33:11 +08:00

data_coord_mock_test.go

enhance: Remove import v1 (#31403)

2024-03-22 15:29:09 +08:00

database_interceptor_test.go

fix: grant ManualCompact api doesn't work (#38096)

2024-12-03 10:36:38 +08:00

database_interceptor.go

fix: grant ManualCompact api doesn't work (#38096)

2024-12-03 10:36:38 +08:00

default_limit_reducer.go

Improve the error message for getting all indexes of collection (#27389)

2023-10-08 21:23:32 +08:00

dummyreq_test.go

enhance: Use internal json(sonic) to replace std json lib (#37708)

2024-11-18 10:46:31 +08:00

dummyreq.go

enhance: Use internal json(sonic) to replace std json lib (#37708)

2024-11-18 10:46:31 +08:00

hook_interceptor_test.go

enhance: set the rpc error code to avoid the invalid retry (#37024)

2024-10-22 10:33:26 +08:00

hook_interceptor.go

enhance: set the rpc error code to avoid the invalid retry (#37024)

2024-10-22 10:33:26 +08:00

http_req_impl_test.go

enhance: add list index and segment index retrieval API for WebUI (#37861)

2024-11-22 16:58:34 +08:00

http_req_impl.go

enhance: [skip e2e] refine display format (#38537)

2024-12-17 22:30:44 +08:00

impl_test.go

feat: support to replicate collection when the services contains the system tt msg (#37559)

2024-12-17 09:08:46 +08:00

impl.go

feat: support to replicate collection when the services contains the system tt msg (#37559)

2024-12-17 09:08:46 +08:00

interface_def.go

Decoupling client and server API in types interface (#27186)

2023-09-26 09:57:25 +08:00

lb_balancer.go

enhance: Decouple shard client manager from shard cache (#37371)

2024-11-12 10:30:28 +08:00

lb_policy_test.go

fix: Get shard client failed by client is closed (#37729)

2024-11-18 18:22:30 +08:00

lb_policy.go

enhance: Optimize param cost in search (#37738)

2024-11-19 10:14:40 +08:00

look_aside_balancer_test.go

fix: Get shard client failed by client is closed (#37729)

2024-11-18 18:22:30 +08:00

look_aside_balancer.go

fix: Get shard client failed by client is closed (#37729)

2024-11-18 18:22:30 +08:00

management_test.go

enhance: add restful api to trigger component stop (#32076)

2024-06-07 10:35:54 +08:00

management.go

enhance: Use internal json(sonic) to replace std json lib (#37708)

2024-11-18 10:46:31 +08:00

meta_cache_adapter_test.go

enhance: Avoid initializing casbin enforcer for each request (#29117)

2023-12-12 10:36:43 +08:00

meta_cache_adapter.go

fix: the panic when db isn't existed in the rate limit interceptor (#33244)

2024-05-22 15:57:39 +08:00

meta_cache_test.go

feat: support to replicate collection when the services contains the system tt msg (#37559)

2024-12-17 09:08:46 +08:00

meta_cache.go

feat: support to replicate collection when the services contains the system tt msg (#37559)

2024-12-17 09:08:46 +08:00

metrics_info_test.go

feat: management WebUI homepage (#36822)

2024-10-23 11:29:28 +08:00

metrics_info.go

enhance: add disk stats within system metrics (#38033)

2024-12-06 16:32:41 +08:00

mock_cache.go

fix: Add version to the proxy cache to resolve concurrency issues (#38067)

2024-12-04 18:06:39 +08:00

mock_channels_manager.go

enhance: refine pular related mq interfaces (#38007)

2024-12-04 20:50:39 +08:00

mock_lb_balancer.go

enhance: Decouple shard client manager from shard cache (#37371)

2024-11-12 10:30:28 +08:00

mock_lb_policy.go

fix: fix make generate-mockery panic with go1.22 (#36830)

2024-10-17 12:11:31 +08:00

mock_msgstream_test.go

feat: support to replicate collection when the services contains the system tt msg (#37559)

2024-12-17 09:08:46 +08:00

mock_shardclient_manager.go

fix: Get shard client failed by client is closed (#37729)

2024-11-18 18:22:30 +08:00

mock_test.go

feat: support to replicate collection when the services contains the system tt msg (#37559)

2024-12-17 09:08:46 +08:00

mock_tso_test.go

fix: fix make generate-mockery panic with go1.22 (#36830)

2024-10-17 12:11:31 +08:00

msg_pack_test.go

feat: Support field partial load collection (#35416)

2024-08-20 16:49:02 +08:00

msg_pack.go

enhance: refine exists log print with ctx (#38080)

2024-12-14 22:36:44 +08:00

OWNERS

[skip ci]Update OWNERS files (#11898)

2021-11-16 15:41:11 +08:00

privilege_cache_test.go

enhance: Add in-memory cache for casbin enforcer result (#35271)

2024-08-05 18:48:15 +08:00

privilege_cache.go

fix: Re-read value after once initialization (#35642)

2024-08-22 17:24:58 +08:00

privilege_interceptor_test.go

enhance: RBAC new grant/revoke privilege (#37785)

2024-11-21 22:20:34 +08:00

privilege_interceptor.go

enhance: RBAC new grant/revoke privilege (#37785)

2024-11-21 22:20:34 +08:00

proxy_rpc_test.go

Refine Proxy errors (#27499)

2023-10-09 10:09:33 +08:00

proxy_test.go

enhance: RBAC new grant/revoke privilege (#37785)

2024-11-21 22:20:34 +08:00

proxy.go

fix: the issue of replicate message exception when the ttMsgEnable config is changed dynamically (#38178)

2024-12-14 23:24:51 +08:00

rate_limit_interceptor_test.go

fix: use the default partition for the limit quota when the request partition name is empty (#38005)

2024-11-27 11:00:36 +08:00

rate_limit_interceptor.go

fix: use the default partition for the limit quota when the request partition name is empty (#38005)

2024-11-27 11:00:36 +08:00

reducer_test.go

Format the code (#27275)

2023-09-21 09:45:27 +08:00

reducer.go

Fill output fields for count request (#27967)

2023-10-30 10:00:12 +08:00

repack_func_test.go

Format the code (#27275)

2023-09-21 09:45:27 +08:00

repack_func.go

Support to replicate the mq message (#27240)

2023-10-20 14:26:09 +08:00

replicate_stream_manager_test.go

fix: the issue of replicate message exception when the ttMsgEnable config is changed dynamically (#38178)

2024-12-14 23:24:51 +08:00

replicate_stream_manager.go

fix: the issue of replicate message exception when the ttMsgEnable config is changed dynamically (#38178)

2024-12-14 23:24:51 +08:00

reScorer_test.go

enhance: refine exists log print with ctx (#38080)

2024-12-14 22:36:44 +08:00

reScorer.go

enhance: refine exists log print with ctx (#38080)

2024-12-14 22:36:44 +08:00

rootcoord_mock_test.go

enhance: alterindex & altercollection supports altering properties (#37437)

2024-12-11 10:20:42 +08:00

roundrobin_balancer_test.go

enhance: Enable dynamic update replica selection policy (#35860)

2024-09-13 17:05:15 +08:00

roundrobin_balancer.go

enhance: Decouple shard client manager from shard cache (#37371)

2024-11-12 10:30:28 +08:00

rpc_msg.go

Add slow log for read (#23272)

2023-04-14 15:30:29 +08:00

search_reduce_util_test.go

enhance: simplify reduction on single search result (#36334)

2024-09-20 11:59:10 +08:00

search_reduce_util.go

enhance: refine exists log print with ctx (#38080)

2024-12-14 22:36:44 +08:00

search_util.go

enhance: Use partitionID when delete by partitionKey (#38231)

2024-12-20 11:18:46 +08:00

segment_test.go

fix: wrong context passing into NewClient, error handling lost in session_util (#30817)

2024-02-28 10:40:09 +08:00

segment.go

enhance: refine exists log print with ctx (#38080)

2024-12-14 22:36:44 +08:00

shard_client_test.go

fix: Get shard client failed by client is closed (#37729)

2024-11-18 18:22:30 +08:00

shard_client.go

fix: Get shard client failed by client is closed (#37729)

2024-11-18 18:22:30 +08:00

simple_rate_limiter_test.go

enhance: enable flush rate limiter of collection level (#33837)

2024-06-24 10:52:03 +08:00

simple_rate_limiter.go

enhance: refine exists log print with ctx (#38080)

2024-12-14 22:36:44 +08:00

task_alias_test.go

fix: execute method should return an error when result is a failure (#34813)

2024-07-20 23:13:40 +08:00

task_alias.go

enhance: Enable linux code checker (#35084)

2024-07-30 15:53:51 +08:00

task_database_test.go

feat: support to replicate collection when the services contains the system tt msg (#37559)

2024-12-17 09:08:46 +08:00

task_database.go

feat: support to replicate collection when the services contains the system tt msg (#37559)

2024-12-17 09:08:46 +08:00

task_delete_streaming.go

enhance: refine exists log print with ctx (#38080)

2024-12-14 22:36:44 +08:00

task_delete_test.go

enhance: Use partitionID when delete by partitionKey (#38231)

2024-12-20 11:18:46 +08:00

task_delete.go

enhance: Use partitionID when delete by partitionKey (#38231)

2024-12-20 11:18:46 +08:00

task_flush_streaming.go

enhance: refine exists log print with ctx (#38080)

2024-12-14 22:36:44 +08:00

task_flush.go

enhance: using streaming service in insert/upsert/flush/delete/querynode (#35406)

2024-08-29 10:03:08 +08:00

task_index_test.go

enhance: refine exists log print with ctx (#38080)

2024-12-14 22:36:44 +08:00

task_index.go

enhance: refine exists log print with ctx (#38080)

2024-12-14 22:36:44 +08:00

task_insert_streaming.go

enhance: refine exists log print with ctx (#38080)

2024-12-14 22:36:44 +08:00

task_insert_test.go

fix: Fix conan gdal building problem (#37338)

2024-10-31 21:04:16 +08:00

task_insert.go

feat: support to replicate collection when the services contains the system tt msg (#37559)

2024-12-17 09:08:46 +08:00

task_policies_test.go

enhance: Decouple shard client manager from shard cache (#37371)

2024-11-12 10:30:28 +08:00

task_policies.go

enhance: refine exists log print with ctx (#38080)

2024-12-14 22:36:44 +08:00

task_query_test.go

enhance: remove the collectionBasicInfo class in the proxy metacache (#37874)

2024-11-22 16:10:33 +08:00

task_query.go

enhance: refine exists log print with ctx (#38080)

2024-12-14 22:36:44 +08:00

task_scheduler_test.go

enhance: remove the collectionBasicInfo class in the proxy metacache (#37874)

2024-11-22 16:10:33 +08:00

task_scheduler.go

enhance: refine exists log print with ctx (#38080)

2024-12-14 22:36:44 +08:00

task_search_test.go

enhance: refine exists log print with ctx (#38080)

2024-12-14 22:36:44 +08:00

task_search.go

enhance: refine exists log print with ctx (#38080)

2024-12-14 22:36:44 +08:00

task_statistic_test.go

fix: Get shard client failed by client is closed (#37729)

2024-11-18 18:22:30 +08:00

task_statistic.go

enhance: refine exists log print with ctx (#38080)

2024-12-14 22:36:44 +08:00

task_test.go

feat: support to replicate collection when the services contains the system tt msg (#37559)

2024-12-17 09:08:46 +08:00

task_upsert_streaming.go

fix: add the db information in the dml message (#37969)

2024-11-27 10:02:35 +08:00

task_upsert_test.go

feat: support to replicate collection when the services contains the system tt msg (#37559)

2024-12-17 09:08:46 +08:00

task_upsert.go

feat: support to replicate collection when the services contains the system tt msg (#37559)

2024-12-17 09:08:46 +08:00

task.go

feat: support to replicate collection when the services contains the system tt msg (#37559)

2024-12-17 09:08:46 +08:00

timestamp_test.go

Format the code (#27275)

2023-09-21 09:45:27 +08:00

timestamp.go

enhance:Remove msgbase unnecessary assignments (#28511)

2023-11-24 15:02:39 +08:00

trace_log_interceptor_test.go

enhance: Support to trace restful request and request error (#28685)

2023-11-27 20:14:26 +08:00

trace_log_interceptor.go

enhance: refine access log to support format access log by yaml and print name info. (#28319)

2023-11-28 15:32:31 +08:00

type_def.go

[skip e2e] Add comment for pChan (#14098)

2021-12-23 21:44:12 +08:00

util_test.go

enhance: support recall estimation (#38017)

2024-12-11 20:40:48 +08:00

util.go

enhance: allow hyphen in partition name (#38494)

2024-12-18 15:04:45 +08:00

validate_util_test.go

fix: Check if the dynamic fields contain any static fields (#38025)

2024-12-01 18:06:37 +08:00

validate_util.go

fix: Check if the dynamic fields contain any static fields (#38025)

2024-12-01 18:06:37 +08:00