mirror of https://gitee.com/milvus-io/milvus.git synced 2026-02-04 11:18:44 +08:00

History

enhance: Handoff growing segment after sorted (#37385 )

issue: #33744 

1. Segments generated from inserts will be loaded as growing until they
are sorted by primary key.
2. This PR may increase memory pressure on the delegator, but we need to
test the performance of stats. In local testing, the speed of stats is
greater than the insert speed.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>

2024-11-07 16:08:24 +08:00

allocator

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

broker

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

session

fix: datacoord stuck at stopping progress (#36852 )

2024-10-17 12:13:37 +08:00

analyze_meta_test.go

fix: Record the nodeID before assigning tasks (#36371 )

2024-09-28 17:21:15 +08:00

analyze_meta.go

fix: Record the nodeID before assigning tasks (#36371 )

2024-09-28 17:21:15 +08:00

build_index_policy.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

channel_manager_factory.go

fix: drop collection failed if enable streaming service (#37444 )

2024-11-07 10:26:26 +08:00

channel_manager_test.go

feat: add segment,pipeline, replica and resourcegroup api for WebUI (#37344 )

2024-11-07 11:52:25 +08:00

channel_manager.go

feat: add segment,pipeline, replica and resourcegroup api for WebUI (#37344 )

2024-11-07 11:52:25 +08:00

channel_store_test.go

fix: drop collection failed if enable streaming service (#37444 )

2024-11-07 10:26:26 +08:00

channel_store.go

fix: drop collection failed if enable streaming service (#37444 )

2024-11-07 10:26:26 +08:00

channel.go

fix: Fill nil schema so that Milvus can watch channel for those upgraded from 2.2 to 2.4 #35695 (#35694 )

2024-08-27 10:36:59 +08:00

cluster_test.go

enhance: Move datanode/indexnode manager to session pkg (#35634 )

2024-08-22 16:02:56 +08:00

cluster.go

enhance: Move datanode/indexnode manager to session pkg (#35634 )

2024-08-22 16:02:56 +08:00

compaction_l0_view_test.go

enhance: Add deltaRowCount in l0 compaction (#33997 )

2024-06-20 17:46:01 +08:00

compaction_l0_view.go

enhance: Add deltaRowCount in l0 compaction (#33997 )

2024-06-20 17:46:01 +08:00

compaction_policy_clustering_test.go

enhance: Separate allocator pkg in datacoord (#35622 )

2024-08-22 10:06:56 +08:00

compaction_policy_clustering.go

enhance: Remove pre-marking segments as L2 during clustering compaction (#36799 )

2024-10-23 17:15:28 +08:00

compaction_policy_l0_test.go

enhance: Separate allocator pkg in datacoord (#35622 )

2024-08-22 10:06:56 +08:00

compaction_policy_l0.go

fix: Remove enableLevelZeroSegment config (#36535 )

2024-10-17 11:59:24 +08:00

compaction_policy_single_test.go

enhance: Separate allocator pkg in datacoord (#35622 )

2024-08-22 10:06:56 +08:00

compaction_policy_single.go

enhance: Remove pre-marking segments as L2 during clustering compaction (#36799 )

2024-10-23 17:15:28 +08:00

compaction_queue_test.go

fix: Refine compactionTask to avoid data race (#36936 )

2024-10-24 09:55:28 +08:00

compaction_queue.go

fix: Refine compactionTask to avoid data race (#36936 )

2024-10-24 09:55:28 +08:00

compaction_task_clustering_test.go

fix: Refine compactionTask to avoid data race (#36936 )

2024-10-24 09:55:28 +08:00

compaction_task_clustering.go

enhance: Handoff growing segment after sorted (#37385 )

2024-11-07 16:08:24 +08:00

compaction_task_l0_test.go

fix: Exlude L0 compaction when clustering is executing (#37141 )

2024-10-29 06:28:24 +08:00

compaction_task_l0.go

fix: Exlude L0 compaction when clustering is executing (#37141 )

2024-10-29 06:28:24 +08:00

compaction_task_meta_test.go

feat: add tasks page into management WebUI (#37002 )

2024-10-28 10:13:29 +08:00

compaction_task_meta.go

feat: add tasks page into management WebUI (#37002 )

2024-10-28 10:13:29 +08:00

compaction_task_mix_test.go

fix: Skip mark compaction timeout for mix and l0 compaction (#37118 )

2024-10-28 14:33:29 +08:00

compaction_task_mix.go

fix: Skip mark compaction timeout for mix and l0 compaction (#37118 )

2024-10-28 14:33:29 +08:00

compaction_task.go

fix: Refine compactionTask to avoid data race (#36936 )

2024-10-24 09:55:28 +08:00

compaction_test.go

fix: drop collection failed if enable streaming service (#37444 )

2024-11-07 10:26:26 +08:00

compaction_trigger_test.go

fix: Refine compactionTask to avoid data race (#36936 )

2024-10-24 09:55:28 +08:00

compaction_trigger_v2_test.go

enhance: Separate allocator pkg in datacoord (#35622 )

2024-08-22 10:06:56 +08:00

compaction_trigger_v2.go

enhance: Add PreallocatedSegmentIDs for the compaction task (#36734 )

2024-10-13 17:59:21 +08:00

compaction_trigger.go

enhance: Remove pre-marking segments as L2 during clustering compaction (#36799 )

2024-10-23 17:15:28 +08:00

compaction_view.go

enhance: Add deltaRowCount in l0 compaction (#33997 )

2024-06-20 17:46:01 +08:00

compaction.go

fix: drop collection failed if enable streaming service (#37444 )

2024-11-07 10:26:26 +08:00

const.go

enhance: pass partition key scalar info if enabled when build vector index (#29931 )

2024-01-24 00:04:55 +08:00

errors_test.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

errors.go

Refine DataCoord status (#27262 )

2023-09-26 17:15:27 +08:00

garbage_collector_test.go

feat: add tasks page into management WebUI (#37002 )

2024-10-28 10:13:29 +08:00

garbage_collector.go

enhance: enable manual compaction for collections without indexes (#36577 )

2024-10-08 19:57:18 +08:00

go_channel_singleton.go

fix: Fix improper use of offset in HybridSearch (#36244 )

2024-09-13 22:05:15 +08:00

handler_test.go

enhance: Handoff growing segment after sorted (#37385 )

2024-11-07 16:08:24 +08:00

handler.go

enhance: Handoff growing segment after sorted (#37385 )

2024-11-07 16:08:24 +08:00

import_checker_test.go

enhance: Optimize import scheduling and add time cost metric (#36601 )

2024-10-09 14:41:20 +08:00

import_checker.go

enhance: Optimize import scheduling and add time cost metric (#36601 )

2024-10-09 14:41:20 +08:00

import_job.go

enhance: Support db for bulkinsert (#37012 )

2024-10-25 14:31:39 +08:00

import_meta_test.go

feat: add tasks page into management WebUI (#37002 )

2024-10-28 10:13:29 +08:00

import_meta.go

feat: add tasks page into management WebUI (#37002 )

2024-10-28 10:13:29 +08:00

import_scheduler_test.go

enhance: Optimize import scheduling and add time cost metric (#36601 )

2024-10-09 14:41:20 +08:00

import_scheduler.go

enhance: Optimize import scheduling and add time cost metric (#36601 )

2024-10-09 14:41:20 +08:00

import_task.go

feat: add tasks page into management WebUI (#37002 )

2024-10-28 10:13:29 +08:00

import_util_test.go

feat: add tasks page into management WebUI (#37002 )

2024-10-28 10:13:29 +08:00

import_util.go

feat: add tasks page into management WebUI (#37002 )

2024-10-28 10:13:29 +08:00

index_engine_version_manager_test.go

Optimize the codec code of session (#27360 )

2023-10-01 10:33:30 +08:00

index_engine_version_manager.go

enhance: Add go-deadlock as unittest only dependency (#33063 )

2024-05-15 16:33:34 +08:00

index_meta_test.go

feat: add tasks page into management WebUI (#37002 )

2024-10-28 10:13:29 +08:00

index_meta.go

feat: add segment,pipeline, replica and resourcegroup api for WebUI (#37344 )

2024-11-07 11:52:25 +08:00

index_service_test.go

feat: add tasks page into management WebUI (#37002 )

2024-10-28 10:13:29 +08:00

index_service.go

enhance: add an unify vector index config checker (#36844 )

2024-10-28 10:11:37 +08:00

job_manager_test.go

feat: add tasks page into management WebUI (#37002 )

2024-10-28 10:13:29 +08:00

job_manager.go

feat: Introduce stats task for import (#35868 )

2024-09-15 15:17:08 +08:00

meta_test.go

feat: add segment,pipeline, replica and resourcegroup api for WebUI (#37344 )

2024-11-07 11:52:25 +08:00

meta_util.go

Make VChannelInfo read compatible with 2.0.2 (#17995 )

2022-07-07 21:58:22 +08:00

meta.go

enhance: Handoff growing segment after sorted (#37385 )

2024-11-07 16:08:24 +08:00

metrics_info_test.go

feat: add segment,pipeline, replica and resourcegroup api for WebUI (#37344 )

2024-11-07 11:52:25 +08:00

metrics_info.go

feat: add segment,pipeline, replica and resourcegroup api for WebUI (#37344 )

2024-11-07 11:52:25 +08:00

mock_channel_store.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

mock_channelmanager.go

feat: add segment,pipeline, replica and resourcegroup api for WebUI (#37344 )

2024-11-07 11:52:25 +08:00

mock_cluster.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

mock_compaction_meta.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

mock_compaction_plan_context.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

mock_handler.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

mock_index_engine_version_manager.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

mock_job_manager.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

mock_segment_manager.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

mock_subcluster.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

mock_test.go

enhance: Add PreallocatedSegmentIDs for the compaction task (#36734 )

2024-10-13 17:59:21 +08:00

mock_trigger_manager.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

OWNERS

[skip ci]Update OWNERS files (#11898 )

2021-11-16 15:41:11 +08:00

partition_stats_meta_test.go

fix: Set current partition stats version to 0 by default when not present (#37299 )

2024-10-31 12:48:21 +08:00

partition_stats_meta.go

fix: Set current partition stats version to 0 by default when not present (#37299 )

2024-10-31 12:48:21 +08:00

policy_test.go

fix: Fill nil schema so that Milvus can watch channel for those upgraded from 2.2 to 2.4 #35695 (#35694 )

2024-08-27 10:36:59 +08:00

policy.go

fix: drop collection failed if enable streaming service (#37444 )

2024-11-07 10:26:26 +08:00

README.md

[skip ci]Change etcd to lowercase (#9983 )

2021-10-15 18:58:37 +08:00

segment_allocation_policy_test.go

enhance: Change the fixed value to a ratio for clustering segment size (#35076 )

2024-08-01 22:04:14 +08:00

segment_allocation_policy.go

fix: use binlog counter to trigger flush but not stats log (#37037 )

2024-10-23 15:07:29 +08:00

segment_info_test.go

fix: Return all compactTo segments after support split (#36361 )

2024-09-20 14:11:11 +08:00

segment_info.go

fix: Remove neighbors if compactTo is unindexed (#36503 )

2024-10-08 14:15:19 +08:00

segment_manager_test.go

fix: use binlog counter to trigger flush but not stats log (#37037 )

2024-10-23 15:07:29 +08:00

segment_manager.go

fix: Fix import segments leak in segment manager (#36602 )

2024-10-08 10:11:22 +08:00

segment_operator_test.go

fix: Prevent clone when selecting segments from meta (#30928 )

2024-03-01 13:59:02 +08:00

segment_operator.go

enhance: Update stats task to optional (#35947 )

2024-09-12 20:37:08 +08:00

server_test.go

fix: drop collection failed if enable streaming service (#37444 )

2024-11-07 10:26:26 +08:00

server.go

enhance: Handoff growing segment after sorted (#37385 )

2024-11-07 16:08:24 +08:00

services_test.go

fix: drop collection failed if enable streaming service (#37444 )

2024-11-07 10:26:26 +08:00

services.go

fix: drop collection failed if enable streaming service (#37444 )

2024-11-07 10:26:26 +08:00

stats_task_meta_test.go

fix: Record the nodeID before assigning tasks (#36371 )

2024-09-28 17:21:15 +08:00

stats_task_meta.go

feat: add segment,pipeline, replica and resourcegroup api for WebUI (#37344 )

2024-11-07 11:52:25 +08:00

sync_segments_scheduler_test.go

fix: datacoord stuck at stopping progress (#36852 )

2024-10-17 12:13:37 +08:00

sync_segments_scheduler.go

fix: datacoord stuck at stopping progress (#36852 )

2024-10-17 12:13:37 +08:00

task_analyze.go

fix: Record the nodeID before assigning tasks (#36371 )

2024-09-28 17:21:15 +08:00

task_index.go

enhance: add unify vector index config management (#36846 )

2024-11-01 06:18:21 +08:00

task_scheduler_test.go

feat: add tasks page into management WebUI (#37002 )

2024-10-28 10:13:29 +08:00

task_scheduler.go

feat: add tasks page into management WebUI (#37002 )

2024-10-28 10:13:29 +08:00

task_stats_test.go

enhance: Remove pre-marking segments as L2 during clustering compaction (#36799 )

2024-10-23 17:15:28 +08:00

task_stats.go

feat: support bm25 logs mixcompaction (#36072 )

2024-10-14 16:57:22 +08:00

types.go

fix: Record the nodeID before assigning tasks (#36371 )

2024-09-28 17:21:15 +08:00

util_test.go

enhance: Separate allocator pkg in datacoord (#35622 )

2024-08-22 10:06:56 +08:00

util.go

enhance: add vector index mgr to remove vector index type dependency (#36843 )

2024-10-17 22:15:25 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
Root Coordinator: timestamp, id and meta source.
Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.