admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-03 17:31:58 +08:00
Code Issues Packages Projects Releases Wiki Activity
20,634 Commits 138 Branches 188 Tags
Commit Graph

9323 Commits

Author SHA1 Message Date
Zhen Ye
b2eb9fe2a7
fix: memory leak in unittest and open the USE_ASAN option when build unittest (#35855) 
issue: #35854

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-09-02 15:59:04 +08:00
yihao.dai
6fd33285e1
fix: Fix compile error (#35901) 
/kind improvement

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-02 14:50:35 +08:00
cai.zhang
2c9bb4dfa3
feat: Support stats task to sort segment by PK (#35054) 
issue: #33744 

This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-02 14:19:03 +08:00
congqixia
9d80137698
fix: Check clustering key skip load behavior (#35865) 
feature issue: #35415
See also #35861

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-02 11:17:02 +08:00
zhagnlu
576ac2bbed
fix: Fix the reference to a variable after it has been moved (#35875) 
#35607

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-09-02 10:05:02 +08:00
Jiquan Long
5ea2454fdf
feat: tantivy tokenizer binding (#35801) 
fix: #35800

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-09-01 17:13:03 +08:00
yihao.dai
1413ffe9b1
enhance: Rename preAllocatedSegments (#35871) 
Rename `preAllocatedSegments` to `preAllocatedSegmentIDs` to avoid
confusion.

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-01 17:09:01 +08:00
yihao.dai
2e090b2426
enhance: Refine flusher channel management (#35870) 
Change the ChannelTask to ChannelLifetime, only removing it upon
unregistering.

issue: https://github.com/milvus-io/milvus/issues/33285

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-01 17:07:10 +08:00
Yinzuo Jiang
407fc933e7
fix: bump bytedance/sonic to v1.12.2 to fix compilation error with go 1.23.0 (#35879) 
fixes: #35878

Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
 2024-09-01 13:43:01 +08:00
SimFG
311f860676
enhance: support to drop the role which is related the privilege list (#35727) 
- issue: #35545

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-08-30 15:17:00 +08:00
XuanYang-cn
323400c190
enhance: Enable to write multiple segments in mix compactor (#35705) 
Prevent segments to be written larger than maxSize * expansionRate

See also: #35584

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-08-30 11:29:01 +08:00
yihao.dai
1c3abd1aea
fix: Fix unstable TestFlusher ut (#35831) 
issue: https://github.com/milvus-io/milvus/issues/35830

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-30 10:25:01 +08:00
Zhen Ye
f9be01ce89
fix: mockery config (#35834) 
issue: #33285

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-08-30 09:49:00 +08:00
zhagnlu
671112d17b
enhance: add more info to hybrid index log (#35808) 
#32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-08-29 21:07:04 +08:00
smellthemoon
a3f2f044d6
fix: not set nullable when stream writer write headers (#35799) 
#35802

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-08-29 20:59:00 +08:00
congqixia
3c8941a88e
enhance: Print log only when rate limit updates (#35806) 
The debug log for "RateLimiter register for rateType" is too frequent
and in e2e cases, the may print 18M times in one run.

This PR make the log be printed only when rate limit is updated.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-29 15:07:00 +08:00
congqixia
09ef3f1b4f
fix: Make sure querycoord observers started once (#35811) 
Related to #35809

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-29 14:45:00 +08:00
Patrick Weizhi Xu
b3089b5bdc
feat: support range search pagination retains order (#35738) 
issue: #35464

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-08-29 14:09:00 +08:00
jaime
b0ac04d104
enhance: set database properties to restrict read access (#35745) 
issue: #35744

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-08-29 13:17:01 +08:00
smellthemoon
b51b4a2838
fix: try get not exist file after upgrade (#35740) 
https://github.com/milvus-io/milvus/issues/35741

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-08-29 11:09:01 +08:00
congqixia
86691656f3
enhance: Change frequent balancer debug log to rated one (#35749) 
"skip balance" log is too frequent in debug level. This PR changes it
into rated on.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-29 10:07:00 +08:00
congqixia
985d84d3ee
enhance: Add skip load validation for create collection task (#35737) 
Related to #35415

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-29 10:05:08 +08:00
Zhen Ye
99dff06391
enhance: using streaming service in insert/upsert/flush/delete/querynode (#35406) 
issue: #33285

- using streaming service in insert/upsert/flush/delete/querynode
- fixup flusher bugs and refactor the flush operation
- enable streaming service for dml and ddl
- pass the e2e when enabling streaming service
- pass the integration tst when enabling streaming service

---------

Signed-off-by: chyezh <chyezh@outlook.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-29 10:03:08 +08:00
congqixia
c5c9ef9876
fix: Check response size before add to counter (#35778) 
Related to #35767

prometheus counter cannot add negative value
when response is not written(say timeout/network broken) panicking may
happen if not check

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-29 00:05:00 +08:00
Zhen Ye
9b96841ae9
fix: wrong construction in evalctx (#35772) 
issue: #35771

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-08-28 19:21:00 +08:00
congqixia
91223deb91
fix: Use SafeChan as TxnManager.closed (#35763) 
Resolves: #35762

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-28 19:06:59 +08:00
yihao.dai
56b3490a98
fix: Fix rate wasn't limited to the expected value (#35699) 
Each time the rate is reset, the token bucket is fully refilled, causing
the rate wasn't limited to the expected value. This PR addresses the
issue by preventing the token reset.

issue: https://github.com/milvus-io/milvus/issues/35675,
https://github.com/milvus-io/milvus/issues/35702

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-28 14:13:00 +08:00
Jiquan Long
a52ba3d09d
enhance: allow many segments for inverted index (#35616) 
fix: https://github.com/milvus-io/milvus/issues/35615

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-08-28 11:30:59 +08:00
jaime
d8aa01bc1a
fix: inconsistent meta view causes rate limit invalid (#35665) 
issue: #35663

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-08-28 11:21:04 +08:00
jaime
b7ea1defd3
fix: mistaken deletions may occur during GC channel checkpoints (#35707) 
issue: #35706

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-08-28 10:11:05 +08:00
Zhen Ye
98866205fa
fix: munmap deallocate too much memory (#35725) 
issue: #35693

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-08-27 17:18:59 +08:00
yihao.dai
9868fe4e6c
fix: Fix panic due to empty candidate import segments (#35673) 
issue: https://github.com/milvus-io/milvus/issues/35662

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-27 17:08:59 +08:00
zhagnlu
4d2f96c760
enhance: support bitmap mmap (#35399) 
#32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-08-27 16:34:59 +08:00
Chun Han
bfd9d86fe9
feat: support groupby size on go-layer(#33544) (#33845) 
related: #33544

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-08-27 14:21:00 +08:00
aoiasd
fe83805d56
fix: loss data bug for deprecated querynode DoubleBuffer (#35128) 
relate: https://github.com/milvus-io/milvus/issues/31548

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-08-27 14:10:59 +08:00
XuanYang-cn
f12e368a76
fix: Fill nil schema so that Milvus can watch channel for those upgraded from 2.2 to 2.4 #35695 (#35694) 
See also: [#35701 ](https://github.com/milvus-io/milvus/issues/35701)

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-08-27 10:36:59 +08:00
SimFG
3e1052f889
enhance: use the msg position obj when getting replicate channel position (#35606) 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-08-27 10:28:59 +08:00
jaime
7d3c0d748c
fix: the incorrect number of partitions in rootcoord meta (#35600) 
issue: #35698

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-08-26 15:20:58 +08:00
XuanYang-cn
82743c5c50
fix: Clear channelcp meta and metrics ASAP (#35658) 
See also: #35588

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-08-26 12:22:57 +08:00
sre-ci-robot
6ddfd02f01
[automated] Update Knowhere Commit (#35688) 
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-08-26 01:04:57 +08:00
Patrick Weizhi Xu
4236c3bafb
enhance: enable MV on binary vectors (#35668) 
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-08-25 16:27:09 +08:00
smellthemoon
098c10922b
enhance: add some check when create collection (#35629) 
not support nullable==true in vector type. check it when create
collection.

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-08-25 16:23:05 +08:00
Xiaofan
0dc5e89007
enhance: reduce the log level of frequent log (#35652) 
fix #35651

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
 2024-08-25 16:20:57 +08:00
cai.zhang
615a653988
fix: Fix offset out of range for creating Trie index (#35553) 
issue: #35550

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-08-25 15:50:57 +08:00
yihao.dai
f2b83d316b
enhance: Support memory mode chunk cache (#35347) 
Chunk cache supports loading raw vectors into memory.

issue: https://github.com/milvus-io/milvus/issues/35273

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-25 15:42:58 +08:00
Xiaofan
50fcfe8ef1
enhance: add nan and inf check (#35683) 
fix #35594
add float check on files

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
 2024-08-25 15:22:57 +08:00
zhagnlu
42f7800b5b
enhance: add bitmap offset cache to speed up retrieve raw data (#35498) 
#35458

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-08-24 01:40:58 +08:00
Zhen Ye
75da36d1aa
enhance: enable asan for milvus (#35627) 
issue: #35626

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-08-23 21:06:58 +08:00
congqixia
59387f0b3a
fix: Use correct release func when datanode unwatch a channel (#35655) 
See also #35654

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-23 14:09:03 +08:00
Gao
e8e3544a11
enhance: add hit segment num metrics for queryHook (#35577) 
issue: #35576

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-08-23 12:49:02 +08:00