congqixia e71b7c7cc9

enhance: Reduce datanode metacache frequent scan range (#33400) ...

See also #32165

There were some frequent scan in metacache:
- List all segments whose start positions not synced
- List compacted segments

Those scan shall cause lots of CPU time when flushed segment number is
large meanwhile `Flushed` segments can be skipped in those two scenarios

This PR make:
- Add segment state shortcut in metacache
- List start positions state before `Flushed`
- Make compacted segments state to be `Dropped` and use `Dropped` state
while scanning them

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

2024-05-28 14:19:42 +08:00

..

bf_write_buffer_test.go

enhance: Reduce datanode metacache frequent scan range (#33400)

2024-05-28 14:19:42 +08:00

bf_write_buffer.go

enhance: Reduce datanode metacache frequent scan range (#33400)

2024-05-28 14:19:42 +08:00

delta_buffer_test.go

fix: Correct Size calculation of DeleteData (#30397)

2024-02-02 10:47:04 +08:00

delta_buffer.go

fix: Correct Size calculation of DeleteData (#30397)

2024-02-02 10:47:04 +08:00

insert_buffer_test.go

enhance: only buffer delete if it match insert has smaller timestamp (#30122)

2024-01-19 17:28:53 +08:00

insert_buffer.go

enhance: Tidy compactor and remove dup codes (#32198)

2024-05-23 09:53:40 +08:00

l0_write_buffer_test.go

enhance: Reduce datanode metacache frequent scan range (#33400)

2024-05-28 14:19:42 +08:00

l0_write_buffer.go

enhance: Use pre-built logger for write buffer frequent ops (#33273)

2024-05-22 21:11:40 +08:00

manager_test.go

enhance: Make write buffer memory check do until safe (#32172)

2024-04-12 10:55:18 +08:00

manager.go

enhance: Make write buffer memory check do until safe (#32172)

2024-04-12 10:55:18 +08:00

mock_mananger.go

enhance: Add MemoryHighSyncPolicy back to write buffer manager (#29997)

2024-01-31 19:03:04 +08:00

mock_write_buffer.go

enhance: Add MemoryHighSyncPolicy back to write buffer manager (#29997)

2024-01-31 19:03:04 +08:00

options.go

fix: prevent segments got flushed multiple times (#30240)

2024-01-24 14:19:00 +08:00

segment_buffer.go

enhance: Tidy compactor and remove dup codes (#32198)

2024-05-23 09:53:40 +08:00

sync_policy_test.go

enhance: Add MemoryHighSyncPolicy back to write buffer manager (#29997)

2024-01-31 19:03:04 +08:00

sync_policy.go

fix: Make FlushTs Sync Policy apply to all buffers (#31839)

2024-04-03 11:47:13 +08:00

write_buffer_test.go

enhance: Use struct{} for sync task future result (#32673)

2024-04-29 10:59:26 +08:00

write_buffer.go

enhance: Reduce datanode metacache frequent scan range (#33400)

2024-05-28 14:19:42 +08:00