OIiveirra commented on code in PR #69032:
URL: https://github.com/apache/doris/pull/69032#discussion_r4303140255
##########
be/src/exec/common/hash_table/hash_key_type.h:
##########
@@ -125,7 +125,11 @@ inline HashKeyType get_hash_key_type(const
std::vector<DataTypePtr>& data_types)
// fixed-width or string-key specialization even when it is the only
grouping/join key.
return HashKeyType::serialized;
}
- // serialized cannot be used in the case of single column, because the
join operator will have some processing of column nullable, resulting in
incorrect serialized results.
+ // Spatial columns store StringViews, not STRING offsets/chars. Use their
column serializer
+ // for both values and nullable markers, as for the existing single-column
complex keys.
+ if (t->get_primitive_type() == TYPE_GEOMETRY || t->get_primitive_type() ==
TYPE_GEOGRAPHY) {
Review Comment:
Fixed in b490cd866af. I used the analysis-side alternative to keep this PR
bounded: reject spatial source/probe runtime-filter contexts both at initial
admission and after probe rewrites at the scan, and reject constant-list
spatial IN/NOT IN before coercion. Serialized spatial hash joins remain
supported.
FE tests pass 93/93, including scalar controls and rewritten-probe checks.
The real Iceberg regression passes in both generation and non-generation modes:
bare geom/geog joins with pruning disabled and runtime-filter flags 1/2/8 all
return 5, and their plans contain no RF. Both IN operators reject explicitly
for both kinds.
##########
be/src/format/transformer/viceberg_parquet_writer.cpp:
##########
@@ -18,15 +18,128 @@
#include "format/transformer/viceberg_parquet_writer.h"
#include <parquet/api/reader.h>
+#include <parquet/arrow/schema.h>
+#include <parquet/file_writer.h>
#include <parquet/schema.h>
#include <algorithm>
+#include <numeric>
+#include <string>
+#include "exprs/function/geo/functions_geo.h"
#include "format/table/iceberg/iceberg_arrow_block_convertor.h"
#include "format/table/parquet_utils.h"
+#include "runtime/exec_env.h"
#include "runtime/runtime_state.h"
namespace doris {
+namespace {
+
+::parquet::schema::NodePtr clone_parquet_node(const ::parquet::schema::Node&
node) {
+ if (node.is_primitive()) {
+ const auto& primitive = static_cast<const
::parquet::schema::PrimitiveNode&>(node);
+ return ::parquet::schema::PrimitiveNode::Make(
+ primitive.name(), primitive.repetition(),
primitive.logical_type(),
+ primitive.physical_type(), primitive.type_length(),
primitive.field_id());
+ }
+
+ const auto& group = static_cast<const ::parquet::schema::GroupNode&>(node);
+ ::parquet::schema::NodeVector fields;
+ fields.reserve(group.field_count());
+ for (int index = 0; index < group.field_count(); ++index) {
+ fields.push_back(clone_parquet_node(*group.field(index)));
+ }
+ return ::parquet::schema::GroupNode::Make(
+ group.name(), group.repetition(), fields,
+ group.logical_type()->is_none() ? nullptr : group.logical_type(),
group.field_id());
+}
+
+arrow::Result<::parquet::LogicalType::EdgeInterpolationAlgorithm>
geography_algorithm(
+ const std::string& algorithm) {
+ using Algorithm = ::parquet::LogicalType::EdgeInterpolationAlgorithm;
+ if (algorithm == "spherical") {
+ return Algorithm::SPHERICAL;
+ }
+ if (algorithm == "vincenty") {
+ return Algorithm::VINCENTY;
+ }
+ if (algorithm == "thomas") {
+ return Algorithm::THOMAS;
+ }
+ if (algorithm == "andoyer") {
+ return Algorithm::ANDOYER;
+ }
+ if (algorithm == "karney") {
+ return Algorithm::KARNEY;
+ }
+ return arrow::Status::Invalid("Unsupported Iceberg geography edge
algorithm: ", algorithm);
+}
+
+bool contains_spatial_type(iceberg::Type* type) {
+ switch (type->type_id()) {
+ case iceberg::TypeID::GEOMETRY:
+ case iceberg::TypeID::GEOGRAPHY:
+ return true;
+ case iceberg::TypeID::STRUCT:
+ for (const auto& field : type->as_struct_type()->fields()) {
+ if (contains_spatial_type(field.field_type())) {
+ return true;
+ }
+ }
+ return false;
+ case iceberg::TypeID::LIST:
+ return
contains_spatial_type(type->as_list_type()->element_field().field_type());
+ case iceberg::TypeID::MAP:
+ return
contains_spatial_type(type->as_map_type()->key_field().field_type()) ||
+
contains_spatial_type(type->as_map_type()->value_field().field_type());
+ default:
+ return false;
+ }
+}
+
+arrow::Result<std::shared_ptr<::parquet::schema::GroupNode>>
make_iceberg_parquet_schema(
+ const std::shared_ptr<arrow::Schema>& arrow_schema, const
iceberg::Schema& iceberg_schema,
+ const std::shared_ptr<::parquet::WriterProperties>& writer_properties,
+ const std::shared_ptr<::parquet::ArrowWriterProperties>&
arrow_properties) {
+ std::shared_ptr<::parquet::SchemaDescriptor> parquet_schema;
+ ARROW_RETURN_NOT_OK(::parquet::arrow::ToParquetSchema(arrow_schema.get(),
*writer_properties,
+ *arrow_properties,
&parquet_schema));
+
+ const auto* source_root = parquet_schema->group_node();
+ ::parquet::schema::NodeVector fields;
+ fields.reserve(source_root->field_count());
+ for (int index = 0; index < source_root->field_count(); ++index) {
+ const auto& source_field = source_root->field(index);
+ const auto& iceberg_field = iceberg_schema.columns()[index];
+ if (iceberg_field.field_type()->type_id() ==
iceberg::TypeID::GEOMETRY) {
+ const auto& geometry =
+ static_cast<const
iceberg::GeometryType&>(*iceberg_field.field_type());
+ fields.push_back(::parquet::schema::PrimitiveNode::Make(
+ source_field->name(), source_field->repetition(),
+ ::parquet::LogicalType::Geometry(geometry.crs()),
::parquet::Type::BYTE_ARRAY,
Review Comment:
Fixed in b490cd866af using the permitted rejection alternative. The current
sink does not transport referenced table-property definitions, so the Parquet
writer rejects projjson: CRS references before ParquetFileWriter::Open for both
spatial kinds, including case-equivalent prefixes. Ordinary nonreferenced CRS
writes and raw custom-CRS reads remain available; no protocol or
metadata-transport expansion was added.
The BE writer/schema tests pass 19/19. The real Iceberg regression supplies
a valid table-property CRS definition and matching source/target identities;
both writes fail explicitly, and both target row counts and committed-file
counts remain zero. Full suite generation and independent non-generation
verification pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
