milvus/internal/core/src/query/PlanProto.h
cai.zhang 19346fa389
feat: Geospatial Data Type and GIS Function support for milvus (#44547)
issue: #43427

This pr's main goal is merge #37417 to milvus 2.5 without conflicts.

# Main Goals

1. Create and describe collections with geospatial type
2. Insert geospatial data into the insert binlog
3. Load segments containing geospatial data into memory
4. Enable query and search can display  geospatial data
5. Support using GIS funtions like ST_EQUALS in query
6. Support R-Tree index for geometry type

# Solution

1. **Add Type**: Modify the Milvus core by adding a Geospatial type in
both the C++ and Go code layers, defining the Geospatial data structure
and the corresponding interfaces.
2. **Dependency Libraries**: Introduce necessary geospatial data
processing libraries. In the C++ source code, use Conan package
management to include the GDAL library. In the Go source code, add the
go-geom library to the go.mod file.
3. **Protocol Interface**: Revise the Milvus protocol to provide
mechanisms for Geospatial message serialization and deserialization.
4. **Data Pipeline**: Facilitate interaction between the client and
proxy using the WKT format for geospatial data. The proxy will convert
all data into WKB format for downstream processing, providing column
data interfaces, segment encapsulation, segment loading, payload
writing, and cache block management.
5. **Query Operators**: Implement simple display and support for filter
queries. Initially, focus on filtering based on spatial relationships
for a single column of geospatial literal values, providing parsing and
execution for query expressions.Now only support brutal search
7. **Client Modification**: Enable the client to handle user input for
geospatial data and facilitate end-to-end testing.Check the modification
in pymilvus.

---------

Signed-off-by: Yinwei Li <yinwei.li@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: ZhuXi <150327960+Yinwei-Yu@users.noreply.github.com>
2025-09-28 19:43:05 +08:00

152 lines
4.5 KiB
C++

// Copyright (C) 2019-2020 Zilliz. All rights reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software distributed under the License
// is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
// or implied. See the License for the specific language governing permissions and limitations under the License
#pragma once
#include <boost/dynamic_bitset.hpp>
#include <memory>
#include "Plan.h"
#include "PlanNode.h"
#include "common/Schema.h"
#include "pb/plan.pb.h"
#include "plan/PlanNode.h"
namespace milvus::query {
class ProtoParser {
public:
using TypeCheckFunction = std::function<bool(const DataType)>;
static bool
TypeIsBool(const DataType type) {
return type == DataType::BOOL;
}
static bool
TypeIsAny(const DataType) {
return true;
}
public:
explicit ProtoParser(SchemaPtr schema) : schema(std::move(schema)) {
}
std::unique_ptr<VectorPlanNode>
PlanNodeFromProto(const proto::plan::PlanNode& plan_node_proto);
std::unique_ptr<RetrievePlanNode>
RetrievePlanNodeFromProto(const proto::plan::PlanNode& plan_node_proto);
std::unique_ptr<Plan>
CreatePlan(const proto::plan::PlanNode& plan_node_proto);
std::unique_ptr<RetrievePlan>
CreateRetrievePlan(const proto::plan::PlanNode& plan_node_proto);
expr::TypedExprPtr
ParseExprs(const proto::plan::Expr& expr_pb,
TypeCheckFunction type_check = TypeIsBool);
std::shared_ptr<rescores::Scorer>
ParseScorer(const proto::plan::ScoreFunction& function);
private:
expr::TypedExprPtr
CreateAlwaysTrueExprs();
expr::TypedExprPtr
ParseBinaryExprs(const proto::plan::BinaryExpr& expr_pb);
expr::TypedExprPtr
ParseBinaryArithOpEvalRangeExprs(
const proto::plan::BinaryArithOpEvalRangeExpr& expr_pb);
expr::TypedExprPtr
ParseBinaryRangeExprs(const proto::plan::BinaryRangeExpr& expr_pb);
expr::TypedExprPtr
ParseCallExprs(const proto::plan::CallExpr& expr_pb);
expr::TypedExprPtr
ParseColumnExprs(const proto::plan::ColumnExpr& expr_pb);
expr::TypedExprPtr
ParseCompareExprs(const proto::plan::CompareExpr& expr_pb);
expr::TypedExprPtr
ParseExistExprs(const proto::plan::ExistsExpr& expr_pb);
expr::TypedExprPtr
ParseNullExprs(const proto::plan::NullExpr& expr_pb);
expr::TypedExprPtr
ParseJsonContainsExprs(const proto::plan::JSONContainsExpr& expr_pb);
expr::TypedExprPtr
ParseGISFunctionFilterExprs(
const proto::plan::GISFunctionFilterExpr& expr_pb);
expr::TypedExprPtr
ParseTermExprs(const proto::plan::TermExpr& expr_pb);
expr::TypedExprPtr
ParseUnaryExprs(const proto::plan::UnaryExpr& expr_pb);
expr::TypedExprPtr
ParseUnaryRangeExprs(const proto::plan::UnaryRangeExpr& expr_pb);
expr::TypedExprPtr
ParseTimestamptzArithCompareExprs(
const proto::plan::TimestamptzArithCompareExpr& expr_pb);
expr::TypedExprPtr
ParseValueExprs(const proto::plan::ValueExpr& expr_pb);
void
PlanOptionsFromProto(const proto::plan::PlanOption& plan_option_proto,
PlanOptions& plan_options);
private:
const SchemaPtr schema;
};
} // namespace milvus::query
template <>
struct fmt::formatter<milvus::proto::plan::GenericValue::ValCase>
: formatter<string_view> {
auto
format(milvus::proto::plan::GenericValue::ValCase c,
format_context& ctx) const {
string_view name = "unknown";
switch (c) {
case milvus::proto::plan::GenericValue::ValCase::kBoolVal:
name = "kBoolVal";
break;
case milvus::proto::plan::GenericValue::ValCase::kInt64Val:
name = "kInt64Val";
break;
case milvus::proto::plan::GenericValue::ValCase::kFloatVal:
name = "kFloatVal";
break;
case milvus::proto::plan::GenericValue::ValCase::kStringVal:
name = "kStringVal";
break;
case milvus::proto::plan::GenericValue::ValCase::kArrayVal:
name = "kArrayVal";
break;
case milvus::proto::plan::GenericValue::ValCase::VAL_NOT_SET:
name = "VAL_NOT_SET";
break;
}
return formatter<string_view>::format(name, ctx);
}
};