Spade A 52c7d7dd80

fix: offset combined with term should be based on Token positions in phrase match (#39931) ...

fix: #39711

Unlike English sentence where each words are parsed exactly once and one
after one with position length 1, one Chinese word may be parsed to
multiple words with position length larger than 1.

For example, "badminton and skiing" will be parsed to Token{ start: 0,
length: 1, text: "badminton" }, Token{ start: 1, length: 1, text: "and"
}, and Token{ start: 2, length: 1, text: "tennis" }.

While for exmaple for Chinsese: "羽毛球和滑雪" may be parsed to Token{ start:
0, length: 2, text: "羽毛" }, Token{ start: 0, length: 3, text: "羽毛球" },
Token{ start: 3, length: 1, text: "和" }, and Token{ start: 4, length: 2,
text: "滑雪" }.

This PR fix that the code not recognizes this situation.

---------

Signed-off-by: SpadeA <tangchenjie1210@gmail.com>

2025-02-18 20:38:51 +08:00

..

bench

enhance: Rename API GenDataset to GenFieldData in unittest (#39386)

2025-01-17 15:55:03 +08:00

test_utils

feat: support random sample (#39532)

2025-02-18 12:40:50 +08:00

CMakeLists.txt

feat: support random sample (#39532)

2025-02-18 12:40:50 +08:00

init_gtest.cpp

enhance: async search and retrieve in cgo (#33228)

2024-06-22 09:38:02 +08:00

test_always_true_expr.cpp

enhance: support iterative filter execution (#37363)

2024-12-11 11:32:44 +08:00

test_array_bitmap_index.cpp

enhance: record memory size (uncompressed) item for index (#38770)

2025-01-14 10:33:06 +08:00

test_array_expr.cpp

feat: support phrase match query (#38869)

2025-01-12 20:24:58 +08:00

test_array_inverted_index.cpp

fix: enable to build index with single segment (#39233)

2025-01-16 11:01:06 +08:00

test_array.cpp

enhance: refine array view to optimize memory usage(#38736) (#38808)

2025-01-07 13:26:55 +08:00

test_azure_chunk_manager.cpp

enhance: optimize milvus core building (#35610)

2024-08-23 12:35:02 +08:00

test_bf_sparse.cpp

enhance: speed up search iterator stage 1 (#37947)

2024-12-26 10:32:49 +08:00

test_bf.cpp

enhance: reduce copy of bitset and id conversion of brurtforce search (#37675)

2024-11-19 15:48:40 +08:00

test_binary.cpp

add growing segment index (#23615)

2023-04-26 10:14:41 +08:00

test_binlog_index.cpp

enhance: use growingMmapEnabled to control the behavior of interim index, not vectorField (#36500)

2024-10-17 20:25:24 +08:00

test_bitmap_index.cpp

enhance: record memory size (uncompressed) item for index (#38770)

2025-01-14 10:33:06 +08:00

test_bitmap.cpp

Support SIMD of several Expr (#23715) (#23717)

2023-05-12 14:11:20 +08:00

test_bitset.cpp

enhance: [bitset] extend op_find() to be able to search both 0 and 1 (#39176)

2025-01-14 09:50:58 +08:00

test_bool_index.cpp

Support SIMD of several Expr (#23715) (#23717)

2023-05-12 14:11:20 +08:00

test_build_inverted_index_with_single_segment.cpp

fix: enable to build index with single segment (#39233)

2025-01-16 11:01:06 +08:00

test_c_api.cpp

fix:fix GetValueFromConfig for bool type (#39526)

2025-01-24 16:17:05 +08:00

test_c_stream_reduce.cpp

enhance: Use template to remove unittest duplication (#39144)

2025-01-13 09:58:57 +08:00

test_c_tokenizer.cpp

enhance: rename tokenizer to analyzer and check analyzer params (#37478)

2024-11-10 16:12:26 +08:00

test_cached_search_iterator.cpp

enhance: speed up search iterator stage 1 (#37947)

2024-12-26 10:32:49 +08:00

test_chunk_cache.cpp

enhance: eliminate compile warnings (part2) (#38535)

2024-12-25 15:30:50 +08:00

test_chunk_vector.cpp

feat: Enable more VECTOR_INT8 unittest (#39569)

2025-01-24 17:03:07 +08:00

test_chunk.cpp

fix: get correct size when sealed segment chunked (#37062)

2024-10-25 12:01:31 +08:00

test_chunked_column.cpp

enhance: Optimize GetChunkIDByOffset and add ut (#37704)

2024-11-15 14:16:31 +08:00

test_chunked_segment.cpp

enhance: add unit test for string pk (#39329)

2025-01-20 19:03:04 +08:00

test_common.cpp

enhance: add valid_data in span (#35030)

2024-08-02 15:40:14 +08:00

test_concurrent_vector.cpp

feat: [Sparse Float Vector] segcore basics and index building (#30357)

2024-03-11 14:45:02 +08:00

test_data_codec.cpp

fix: Fix conan gdal building problem (#37338)

2024-10-31 21:04:16 +08:00

test_delete_record.cpp

enhance: optimize delete init construct code (#39327)

2025-02-17 21:05:26 +08:00

test_disk_file_manager_test.cpp

enhance: remove unused code for StorageV2 (#35132)

2024-08-01 12:08:13 +08:00

test_exec.cpp

feat: support phrase match query (#38869)

2025-01-12 20:24:58 +08:00

test_expr_materialized_view.cpp

enhance: refactor executor framework V2 (#35251)

2024-09-13 20:57:09 +08:00

test_expr.cpp

feat: support json index (#36750)

2025-02-15 14:06:15 +08:00

test_float16.cpp

enhance: Use template to remove unittest duplication (#39144)

2025-01-13 09:58:57 +08:00

test_function.cpp

feat: Implement custom function module in milvus expr (#36560)

2024-10-25 15:25:30 +08:00

test_futures.cpp

enhance: async search and retrieve in cgo (#33228)

2024-06-22 09:38:02 +08:00

test_gcp_native_chunk_manager.cpp

enhance: GCS native support (GCS implemented using Google Cloud Storage libraries) (#36214)

2024-09-30 13:23:32 +08:00

test_group_by.cpp

enhance:refine group_strict_size parameter(#37482) (#37483)

2024-11-12 09:56:28 +08:00

test_growing_index.cpp

enhance: support null in c data_datacodec and load null value (#32183)

2024-07-23 16:07:51 +08:00

test_growing.cpp

enhance: refactor delete mvcc function (#38066)

2024-12-15 18:02:43 +08:00

test_hybrid_index.cpp

enhance: record memory size (uncompressed) item for index (#38770)

2025-01-14 10:33:06 +08:00

test_index_c_api.cpp

feat: Update segcore for VECTOR_INT8 (#39415)

2025-01-21 11:03:03 +08:00

test_index_wrapper.cpp

enhance: Rename API GenDataset to GenFieldData in unittest (#39386)

2025-01-17 15:55:03 +08:00

test_indexing.cpp

enhance: Rename API GenDataset to GenFieldData in unittest (#39386)

2025-01-17 15:55:03 +08:00

test_init.cpp

enhance: update cagra index params in config and add params check (#29045)

2023-12-26 11:04:47 +08:00

test_integer_overflow.cpp

enhance: refactor executor framework V2 (#35251)

2024-09-13 20:57:09 +08:00

test_inverted_index.cpp

enhance: record memory size (uncompressed) item for index (#38770)

2025-01-14 10:33:06 +08:00

test_iterative_filter.cpp

fix: set iterative filter hint to false when no expr specified (#39033)

2025-01-08 12:56:56 +08:00

test_kmeans_clustering.cpp

enhance: support null in c data_datacodec and load null value (#32183)

2024-07-23 16:07:51 +08:00

test_loading.cpp

feat: Enable more VECTOR_INT8 unittest (#39569)

2025-01-24 17:03:07 +08:00

test_local_chunk_manager.cpp

Migrate the ability to upload and download binlog to cpp (#22984)

2023-06-25 14:38:44 +08:00

test_minio_chunk_manager.cpp

enhance: Support MinIO TLS connection (#31311)

2024-03-21 11:15:20 +08:00

test_mmap_chunk_manager.cpp

enhance: use segment id and type to register in MmapChunkManager and opt malloc in variableChunk (#33993)

2024-06-20 17:42:02 +08:00

test_monitor.cpp

enhance: add scalar filtering and vector search latency metrics (#34785)

2024-07-19 14:01:39 +08:00

test_offset_ordered_array.cpp

enhance: remove timestamp_filter after retrieve (#35207)

2024-08-02 19:32:46 +08:00

test_offset_ordered_map.cpp

enhance: remove timestamp_filter after retrieve (#35207)

2024-08-02 19:32:46 +08:00

test_plan_proto.cpp

enhance: refactor executor framework V2 (#35251)

2024-09-13 20:57:09 +08:00

test_query.cpp

enhance: Use template to remove unittest duplication (#39144)

2025-01-13 09:58:57 +08:00

test_random_sample.cpp

feat: support random sample (#39532)

2025-02-18 12:40:50 +08:00

test_range_search_sort.cpp

feat: [Sparse Float Vector] segcore basics and index building (#30357)

2024-03-11 14:45:02 +08:00

test_reduce_c.cpp

Fix search binary pagination failure (#22477)

2023-03-08 11:03:51 +08:00

test_reduce.cpp

Fix cpplint (#22657)

2023-03-10 09:47:54 +08:00

test_regex_query_util.cpp

fix: regex query can't handle text with newline (#32569)

2024-04-26 12:01:26 +08:00

test_regex_query.cpp

fix: enable to build index with single segment (#39233)

2025-01-16 11:01:06 +08:00

test_relational.cpp

Fix cpplint (#22657)

2023-03-10 09:47:54 +08:00

test_remote_chunk_manager.cpp

enhance: GCS native support (GCS implemented using Google Cloud Storage libraries) (#36214)

2024-09-30 13:23:32 +08:00

test_retrieve.cpp

feat: support phrase match query (#38869)

2025-01-12 20:24:58 +08:00

test_rust_result.cpp

enhance: Handle rust error in c++ (#38113)

2024-12-16 19:40:45 +08:00

test_scalar_index_creator.cpp

feat: [Sparse Float Vector] segcore to support sparse vector search and get raw vector by id (#30629)

2024-03-12 09:16:30 -07:00

test_scalar_index.cpp

enhance: support bitmap mmap (#35399)

2024-08-27 16:34:59 +08:00

test_sealed.cpp

feat: Enable more VECTOR_INT8 unittest (#39569)

2025-01-24 17:03:07 +08:00

test_segcore.cpp

enhance: growing segment support mmap (#32633)

2024-06-18 14:42:00 +08:00

test_similarity_corelation.cpp

Add binary metric types SUBSTRUCTURE/SUPERSTRUCTURE back (#26766)

2023-08-31 20:07:00 +08:00

test_span.cpp

fix: to use the correct offset in span (#37780)

2024-11-18 21:56:30 +08:00

test_storage.cpp

enhance: enable asan for cpp unittest (#37041)

2024-10-23 17:21:27 +08:00

test_string_expr.cpp

enhance: support null expr (#38772)

2025-01-02 14:16:54 +08:00

test_string_index.cpp

fix: enable to build index with single segment (#39233)

2025-01-16 11:01:06 +08:00

test_text_match.cpp

fix: offset combined with term should be based on Token positions in phrase match (#39931)

2025-02-18 20:38:51 +08:00

test_timestamp_index.cpp

Fix cpplint (#22657)

2023-03-10 09:47:54 +08:00

test_tracer.cpp

enhance: add trace_id into segcore logs (#37656)

2024-11-18 10:20:30 +08:00

test_utils.cpp

enhance: optimize delete init construct code (#39327)

2025-02-17 21:05:26 +08:00