admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-07 19:31:51 +08:00
Code Issues Packages Projects Releases Wiki Activity
20,466 Commits 137 Branches 188 Tags
Commit Graph

9208 Commits

Author SHA1 Message Date
wei liu
344dc6a9f8
enhance: enable to set load config in cluster level (#35169) 
issue: #35170
This PR enable to set load configs in cluster level, such as replicas
and resource groups. then when load collections will use the load
config.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-07 12:38:21 +08:00
chyezh
c725416288
enhance: move streaming proto into pkg (#35284) 
issue: #33285

- move streaming related proto into pkg.
- add v2 message type and change flush message into v2 message.

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-08-07 10:34:16 +08:00
Jiquan Long
b5022b0647
enhance: add metric to record disk quota (#35305) 
https://github.com/milvus-io/milvus/issues/35177

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-08-07 10:26:16 +08:00
Jiquan Long
976ceb4a46
fix: comparision operations between incompatible operands (#35264) 
fix: #34139

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-08-07 10:24:27 +08:00
Jiquan Long
f13b21c041
fix: error message when input is not aligned (#35322) 
fix: https://github.com/milvus-io/milvus/issues/35321

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-08-07 09:56:28 +08:00
zhagnlu
c19fe95154
fix: support string match for hybrid and bitmap index (#35294) 
#34841

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-08-07 09:54:22 +08:00
sre-ci-robot
e75efa7359
[automated] Update Knowhere Commit (#35328) 
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-08-07 01:00:16 +08:00
congqixia
6ff238e88a
fix: Set corresponding DataScope for loadStreamDelete (#35312) 
Related to #35311

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-06 22:32:23 +08:00
Cai Yudong
3c9a47c8db
feat: Encode traceID and spanID as hex string (#34807) 
Issue: https://github.com/zilliztech/knowhere/pull/714

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-08-06 15:20:16 +08:00
Jiquan Long
91df03afe8
feat: put inverted index into ram (#35222) 
fix: https://github.com/milvus-io/milvus/issues/35224

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-08-06 11:54:16 +08:00
yihao.dai
678018d9ca
enhance: Avoid unnecessary compaction (#35148) 
Estimate the import segment size based on DiskSegmentMaxSize(2G) to
avoid unnecessary compaction after import completed.

issue: https://github.com/milvus-io/milvus/issues/35147

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-06 10:30:21 +08:00
chyezh
14051fed7d
enhance: streaming service client (#34656) 
issue: #33285

- implement streaming service client.
- implement producing and consuming service client by streaming coord
client and streaming node client.

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-08-05 21:38:15 +08:00
congqixia
a2b517523d
enhance: Add in-memory cache for casbin enforcer result (#35271) 
See also #35270

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-05 18:48:15 +08:00
shaoyue
1db099973f
enhance: Gin & restfulv1 handler use sonic json (#35020) 
/cc @czs007 @congqixia 
/kind enhancement

Signed-off-by: haorenfsa <haorenfsa@gmail.com>
 2024-08-05 16:44:17 +08:00
yihao.dai
b71e058bc5
enhance: Add import option to skip disk quota check (#35274) 
Add an option to skip the disk quota check for backup-restore import.

issue: https://github.com/milvus-io/milvus/issues/33775

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-05 16:40:16 +08:00
cai.zhang
6542c1ab0e
enhance: Add monitoring metrics for task execution time in datacoord (#35139) 
issue: #35138

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-08-05 16:26:17 +08:00
cai.zhang
d905713724
fix: Maintain load idempotency even when building new indexes (#35178) 
issue: #34404

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-08-05 16:22:17 +08:00
yihao.dai
678c0a708b
enhance: Refine import error log (#35067) 
issue: https://github.com/milvus-io/milvus/issues/35060

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-05 15:30:16 +08:00
wei liu
c22b0a636e
fix: delete by expr failed at retry progress (#35241) 
issue: #35240
delete by expr shard the same err object between channels, so if one
channel's query failed, it will fail all channel, which will break
channel level retry policy, and make delete operation failed.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-05 11:18:16 +08:00
zhagnlu
4b553b0333
enhance: revert remove duplicated pk function (#35103) 
issue: #34778
 Revert "fix: fix query count(*) concurrently"
 Revert "enhance: mark duplicated pk as deleted "

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-08-05 10:48:17 +08:00
chyezh
9871966415
enhance: segment alloc interceptor (#34996) 
#33285

- add segment alloc interceptor for streamingnode.
- add add manual alloc segment rpc for datacoord.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-08-04 07:40:15 +08:00
Chun Han
3faef63a25
enhance: add log for partition stats( #30376) (#35219) 
related:  #30376

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-08-02 19:34:22 +08:00
zhagnlu
16dd53e7cf
enhance: remove timestamp_filter after retrieve (#35207) 
#35226

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-08-02 19:32:46 +08:00
cqy123456
b99a554827
enhance:update mmap_chunk_manager config in milvus.yaml (#35180) 
issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/35186

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-08-02 19:28:15 +08:00
yihao.dai
a4439cc911
enhance: Implement flusher in streamingNode (#34942) 
- Implement flusher to:
  - Manage the pipelines (creation, deletion, etc.)
  - Manage the segment write buffer
  - Manage sync operation (including receive flushMsg and execute flush)
- Add a new `GetChannelRecoveryInfo` RPC in DataCoord.
- Reorganize packages: `flushcommon` and `datanode`.

issue: https://github.com/milvus-io/milvus/issues/33285

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-02 18:30:23 +08:00
jaime
fcec4c21b9
fix: check collection health(queryable) fail for releasing collection (#34947) 
issue: #34946

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-08-02 17:20:15 +08:00
Bingyi Sun
3641ae6611
fix: Fix index memory estimation (#35225) 
issue: https://github.com/milvus-io/milvus/issues/35229

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-08-02 16:24:15 +08:00
wayblink
81773bfadf
enhance: add commit time in partitionStats proto (#35125) 
fix: #35110

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-08-02 16:16:14 +08:00
smellthemoon
475c333fa2
enhance: add valid_data in span (#35030) 
#31728

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-08-02 15:40:14 +08:00
wei liu
f466129924
enhance: add segment state on data coord bin log size metrics (#35153) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-02 14:48:14 +08:00
wei liu
3dd3749f0b
enhance: Avoid unnecesary syncTargetVersion func call after querycoord recover (#34954) 
before querycoord stop gracefully, we will save the current target to
meta store and recover it after querycoord start up, to speed the
querycoord's recovery time. but the target version hasn't been recovered
as expected, and it use latest timestamp as current target's version,
which has no effect to querycoord but an unnecessary syncTargetVersion
func call.

This PR recover the correct target version to avoid unnecessary
syncTargetVersion func call

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-02 14:46:13 +08:00
congqixia
095d77269b
enhance: Set WatchProgress for channel operation progress response (#35183) 
The watch progress is always zero in CheckChannelOperationProgress
response, which is meaningless and confusing. This PR set progress value
in rpc response to fix this problem.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-02 12:06:14 +08:00
congqixia
c64a078458
enhance: Support proxy/delegator qn client pooling (#35194) 
See also #35196

Add param item for proxy/delegator query node client pooling and
implement pooling logic

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-02 11:24:19 +08:00
wei liu
3b735b4b02
enhance: Refine param init for MmapDirPath (#35181) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-02 11:12:14 +08:00
wayblink
95462668ca
enhance: unify time in clustering compaction task to unix (#35167) 
#34495

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-08-02 10:30:19 +08:00
wei liu
27b6d58981
fix: Set legacy level to l0 segment after qc restart (#35197) 
issue: #35087
after qc restarts, and target is not ready yet, if dist_handler try to
update segment dist, it will set legacy level to l0 segment, which may
cause l0 segment be moved to other node, cause search/query failed.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-02 10:18:13 +08:00
Jiquan Long
4dbd023402
enhance: add metric to record maxInsertRate and queryNodeMemoryHighWaterLevel (#35185) 
fix: https://github.com/milvus-io/milvus/issues/35177

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-08-02 10:14:13 +08:00
cai.zhang
196a7986b3
enhance: Change the fixed value to a ratio for clustering segment size (#35076) 
issue: #34495

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-08-01 22:04:14 +08:00
congqixia
a642a26ed4
enhance: Resolve ChunkFileWriter lint issue (#35166) 
See also #34483

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-01 16:52:13 +08:00
wayblink
5bbb1c201c
enhance:support l2 single compaction (#34935) 
#34928

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-08-01 14:36:13 +08:00
zhagnlu
f8c1b138a8
fix:fix get array error for int type (#35154) 
#35055

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-08-01 14:30:12 +08:00
zhenshan.cao
aa247f192d
enhance: remove unused code for StorageV2 (#35132) 
issue: https://github.com/milvus-io/milvus/issues/34168

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-08-01 12:08:13 +08:00
cai.zhang
9412002d7d
fix: Fix data race for clustering buffer writer (#35145) 
issue: #34495

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-08-01 11:20:13 +08:00
cai.zhang
089fb2b5fb
enhance: Skip updating the level that are already at the specified level (#35094) 
issue: #34495

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-08-01 11:18:12 +08:00
Bingyi Sun
f229f244d2
enhance: add chunk basic impl (#34634) 
https://github.com/milvus-io/milvus/issues/35112
This pr would not affect milvus functionality by now.
It implments a Chunk memory layout that looks like 
```
VariableColumn
|offset|offset|offset|
|data|data|data|

```
We maybe move offsets to the beginning and add null bitmaps later but
not in this PR.
And mmap test will also be added in another PR.

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-08-01 10:29:51 +08:00
wei liu
e9d61daa3f
enhance: Reduce delegator memory overloaded factor to 0.1 (#35092) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-01 10:21:50 +08:00
sre-ci-robot
c15d074b21
[automated] Update Knowhere Commit (#35157) 
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-08-01 01:17:49 +08:00
Gao
6695c6d0a3
enhance: add channel num for queryHook optimization (#35104) 
At most cases, data in each channel is almost evenly distributed, we
could utilize the channel num info to optimize searh param in queryHook

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-07-31 18:59:49 +08:00
wei liu
03912a8788
enhance: Avoid balance stuck after segment list become stable (#34728) 
issue: #34715
if collection's segment list doesn't changes anymore, then the next
target will be empty at most time, and balance segment will check
whether segment exist in both current and next target, so the balance
cloud be blocked due to next target is empty.

This PR permit segment to be moved if next target is empty, to avoid
balance stuck.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-31 18:09:48 +08:00
Patrick Weizhi Xu
a9744d62b1
enhance: improve error msg when expr invalid during isolation (#35127) 
issue: #34336 

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit b1d7b24972e9414ab9488c7611aca9975e674cd4)
 2024-07-31 17:23:51 +08:00