340 Commits

Author SHA1 Message Date
aoiasd
4da759fa4e
Upsert compatible with 2.3.3 status log (#28124)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-06 10:12:18 +08:00
zhagnlu
506ec41470
force use DNS for aliyunoss because of sdk bug (#26176)
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-08-08 12:07:08 +08:00
zhagnlu
caf62033cb
Upgrade minio-go and add region and virtual host config for segcore chunk manager (#25811)
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-08-02 14:47:07 +08:00
congqixia
48e30aa1dd
Fix querynode always retry on corrupted statslog (#25104)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-19 20:16:58 +08:00
xige-16
bd162f4484
Write the cache file to the cacheStorage.rootpath dir (#25714)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-07-19 12:02:58 +08:00
yah01
7c1d35dcd6
Fix delete record may be re-applied (#24858)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-03 15:04:25 +08:00
Xiaofan
83367219bb
Fix frequent stats log (#25094)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-06-25 20:35:46 +08:00
yah01
11f7e38c10
Add go payload writer (#24656)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-08 15:52:36 +08:00
congqixia
5973f5aeb9
Use go-api/v2 for milvus-proto (#24723)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-07 21:32:37 +08:00
shaoyue
d9e255f1b5
Support aliyun oss as object storage with ak or IAM (#23949)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2023-05-10 15:57:21 +08:00
yah01
62cabbfce2
Support JSON data type (#23839)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-05-09 22:11:24 +08:00
Xiaofan
42a78f7d27
Fix LRU Concurrency (#23310)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-04-10 10:18:29 +08:00
congqixia
e946f9de71
Use singleflight to limit readWithCache concurrent operation (#23036)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-04-03 15:40:24 +08:00
cai.zhang
87b1698130
Fix bug for batch delete files on gcp of minio (#23052) (#23083) (#23091)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-03-31 14:14:23 +08:00
xige-16
0d85f023b9
Fix querynode panic when binlog ts wrong (#23127)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-03-30 15:16:23 +08:00
XuanYang-cn
21cd958383
Remove annoying timerecorder print for metrics (#22464)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-03-01 10:23:49 +08:00
jaime
eac4523c79
Refine Read method of MinioChunkManager (#22257)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-02-21 10:34:26 +08:00
xige-16
50739e6c6a
Add more metrics (#21975)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-02-09 16:06:33 +08:00
Xiaofan
375b2b355d
Fix Memory calculation (#21798)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-28 11:09:51 +08:00
xige-16
7108e2d15f
Remove arrow uasge in FieldData (#21523)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-01-10 20:37:41 +08:00
cai.zhang
50c7a80b2b
Add ut for ListWithPrefix of minio chunk manager (#21105)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-12-11 16:37:22 +08:00
Xiaofan
714782fce1
Add more test for file reader (#21083)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-12-09 11:53:21 +08:00
cai.zhang
6041e5b805
Fix bug for ListWithPrefix from minio with recursive is false (#21040)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-12-07 17:11:19 +08:00
congqixia
fd1a67ef64
Fix compaction target segment rowNum is always 0 (#20937) (#20941)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-12-01 22:11:17 +08:00
yah01
004b443288
Change LRU to a generic type (#20744) (#20759)
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-22 16:13:11 +08:00
groot
e1e2323fdd
Fix a regression of local storage (#20631)
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-11-21 10:31:12 +08:00
Xiaofan
279cf6f5e5
Revert Index Codec (#20652)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-11-16 19:47:08 +08:00
Xiaofan
d01092a119
Fix Index Codec memory and storage usage (#20622)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-11-16 11:45:08 +08:00
shaoyue
e32b5e3711
Fix gcp oauth token not cached (#20396)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-11-08 15:15:03 +08:00
Xiaofan
766957e75a
Recuce memory consumption in MinioChunkManager (#20284)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-11-03 12:19:35 +08:00
shaoyue
9fac1476f3
Add support for GCS(GoogleCloudStorage) with IAM (#20164)
go mod tidy

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-11-01 11:07:35 +08:00
Xiaofan
2bfecf5b4e
Refine bloomfilter and memory usage (#20168)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-31 17:41:34 +08:00
SimFG
a55f739608
Separate public proto files (#19782)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
yah01
89c9cb3680
Fix scan may break GC limitation (#19670)
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-15 15:13:24 +08:00
jaime
1919353f02
Fix parse segment id error from delta log (#19566)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-09-30 14:18:55 +08:00
congqixia
838a633584
Add ctx parameter for ChunkManager methods (#19546)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-29 16:18:56 +08:00
Ten Thousand Leaves
b30c9d4f60
Support Bulk Load in Milvus 2.2 (#18982)
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-09-26 18:06:54 +08:00
Xiaofan
928a213e31
Fix Chunk Manager Path mis use (#19141)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-09-23 14:40:51 +08:00
congqixia
f24e16a66f
Handle multiple parquet RowGroup/Column (#19283)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-21 09:12:50 +08:00
SimFG
d7f38a803d
Separate some proto files (#19218)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
groot
b161aec95e
Support input empty string (#19111) (#19144)
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-09-13 13:36:29 +08:00
xige-16
4de1bfe5bc
Add cpp data codec (#18538)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: zhagnlu lu.zhang@zilliz.com

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-09-09 22:12:34 +08:00
congqixia
ce9ba0dd02
Remove debug log in minio chunk manager (#19134)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-09 17:32:37 +08:00
congqixia
db23a256c2
Make minioChunkManager ListObject level by level when recursive is true (#19096)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-08 14:58:34 +08:00
congqixia
68a6587374
Set insert&stats binlog timestamp range (#19005)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-04 09:05:09 +08:00
congqixia
c91bb0b015
Unify rootPath in configs and ChunkManager (#18808)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-08-25 19:32:53 +08:00
codeman
70254c3ec5
Unified catalog interface for segment (#18289) (#18290)
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
2022-08-20 10:24:51 +08:00
xige-16
e40061b864
Update binlog event format (#18347)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-08-11 14:06:38 +08:00
jaime
efa5dfaa7b
Fix Parse binlog path failure (#18583)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-08-10 17:04:37 +08:00
jaime
6e7831470c
Revert settings of Pulsar address and refine config code (#18494)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-08-02 21:26:33 +08:00