mirror of
https://gitee.com/milvus-io/milvus.git
synced 2026-01-07 19:31:51 +08:00
ddf19de7f7
### **User description** add a new config to control meta batch to avoid too large fix: https://github.com/milvus-io/milvus/issues/44569 pr: https://github.com/milvus-io/milvus/pull/44645 ___ ### **PR Type** Enhancement ___ ### **Description** - Replace hardcoded `MaxEtcdTxnNum` constant with configurable parameter - Add new `maxEtcdTxnNum` configuration to `MetaStoreConfig` with default value 64 - Update all metastore catalog implementations to use dynamic config value - Remove hardcoded constant from `util/constant.go` and update test expectations ___ ### Diagram Walkthrough ```mermaid flowchart LR A["Hardcoded MaxEtcdTxnNum<br/>constant 128"] -->|Replace with| B["MetaStoreConfig<br/>maxEtcdTxnNum param"] B -->|Default value| C["64 operations<br/>per transaction"] B -->|Used by| D["DataCoord<br/>Catalog"] B -->|Used by| E["RootCoord<br/>Catalog"] B -->|Used by| F["StreamingCoord<br/>Catalog"] B -->|Used by| G["StreamingNode<br/>Catalog"] B -->|Used by| H["SuffixSnapshot<br/>Catalog"] ``` <details><summary><h3>File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>5 files</summary><table> <tr> <td><strong>kv_catalog.go</strong><dd><code>Use configurable MaxEtcdTxnNum in batch operations</code> </dd></td> <td><a href="https://github.com/milvus-io/milvus/pull/46514/files#diff-21f10b97df37f264c572a5bea752c442a1933f1441f658c90c546740d529d536">+4/-2</a> </td> </tr> <tr> <td><strong>kv_catalog.go</strong><dd><code>Replace hardcoded constant with dynamic config parameter</code> </dd></td> <td><a href="https://github.com/milvus-io/milvus/pull/46514/files#diff-ca605f818c1903caba7e8fdd022856403889ed63703161028a6cc0005418aa0b">+6/-6</a> </td> </tr> <tr> <td><strong>suffix_snapshot.go</strong><dd><code>Use dynamic config for etcd transaction batch limits</code> </dd></td> <td><a href="https://github.com/milvus-io/milvus/pull/46514/files#diff-41ca5f1e7439335fbd3c198a612f93fb12268bd94e3dc988117f669e45fe462a">+4/-3</a> </td> </tr> <tr> <td><strong>kv_catalog.go</strong><dd><code>Replace util constant with paramtable configuration</code> </dd></td> <td><a href="https://github.com/milvus-io/milvus/pull/46514/files#diff-5ddd0cfcc47a07c1f0a5246b63928f018e8267176a6d1d5780712fa986f508c4">+3/-2</a> </td> </tr> <tr> <td><strong>kv_catalog.go</strong><dd><code>Use configurable MaxEtcdTxnNum for batch operations</code> </dd></td> <td><a href="https://github.com/milvus-io/milvus/pull/46514/files#diff-4cc762324a9c223f7276776ec29d0476bd70e94eca0e194a1b9e6ee67c7c15f5">+4/-3</a> </td> </tr> </table></details></td></tr><tr><td><strong>Tests</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>kv_catalog_test.go</strong><dd><code>Update test expectations for batch operation counts</code> </dd></td> <td><a href="https://github.com/milvus-io/milvus/pull/46514/files#diff-52a66a32833546c9f5a39c02bf3ee2bd58099a1027e84179021c3e1e91afd6e6">+2/-2</a> </td> </tr> <tr> <td><strong>kv_catalog_test.go</strong><dd><code>Update tests to use configurable MaxEtcdTxnNum parameter</code> </dd></td> <td><a href="https://github.com/milvus-io/milvus/pull/46514/files#diff-bbf08f8d1c8410ed63e07719efe937402b111a27a841f0098552a8a5d8d4574f">+4/-3</a> </td> </tr> </table></details></td></tr><tr><td><strong>Configuration changes</strong></td><td><details><summary>3 files</summary><table> <tr> <td><strong>constant.go</strong><dd><code>Remove hardcoded MaxEtcdTxnNum constant definition</code> </dd></td> <td><a href="https://github.com/milvus-io/milvus/pull/46514/files#diff-9a2143fe538a654bdd5e1e0967e4e547faea75726e569376a6055bb837c6c683">+0/-3</a> </td> </tr> <tr> <td><strong>service_param.go</strong><dd><code>Add MaxEtcdTxnNum parameter to MetaStoreConfig</code> </dd></td> <td><a href="https://github.com/milvus-io/milvus/pull/46514/files#diff-9ef00df6bb7232974dc09d15a0ce2719d977163f41789918e0e4ac7fa4742bf0">+10/-0</a> </td> </tr> <tr> <td><strong>milvus.yaml</strong><dd><code>Add maxEtcdTxnNum configuration with default value</code> </dd></td> <td><a href="https://github.com/milvus-io/milvus/pull/46514/files#diff-6e254e06f0f065af33ea15a45c6538bdb785c064a70f2cc9c7c7369d80065a06">+1/-0</a> </td> </tr> </table></details></td></tr></tbody></table> </details> ___ --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com> Co-authored-by: Xiaofan <83447078+xiaofan-luan@users.noreply.github.com> Co-authored-by: xiaofanluan <xiaofan.luan@zilliz.com>