jqcc opened a new issue, #605: URL: https://github.com/apache/doris-flink-connector/issues/605
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description When we use Flink-SQL write a doris-sink-sql, if we define nested types in array, we may get null value in doris. <img width="1102" alt="Image" src="https://github.com/user-attachments/assets/ac7b67b6-bd79-4622-afb0-266c003740a0"; /> ### Doris DDL ``` sql CREATE TABLE `sunjq_doris_connector_debug` ( `id` bigint NULL, `array_int` array<int> NULL, `array_array` array<array<text>> NULL, `array_struct` array<struct<a:text,start:text,end:text>> NULL, `array_map` array<map<int,text>> NULL, `dt` date NULL, `array_dt` array<date> NULL ) ENGINE=OLAP DUPLICATE KEY(`id`) DISTRIBUTED BY RANDOM BUCKETS 2 PROPERTIES ( "replication_allocation" = "tag.location.default: 3", "min_load_replica_num" = "-1", "is_being_synced" = "false", "storage_medium" = "ssd", "storage_format" = "V2", "inverted_index_storage_format" = "V1", "light_schema_change" = "true", "disable_auto_compaction" = "false", "enable_single_replica_compaction" = "false", "group_commit_interval_ms" = "10000", "group_commit_data_bytes" = "134217728" ); ``` ### Flink-SQL ``` sql CREATE TABLE source ( id BIGINT, array_int array<int>, array_array array<array<string>>, array_struct array<row<a string,`start` string,`end` string>>, array_map array<map<int,string>>, dt date, array_dt array<date> ) with ( 'connector' = 'datagen', 'number-of-rows' = '10', 'fields.id.min' = '1', 'fields.id.max' = '10' ); CREATE TABLE doris_test_sink ( id BIGINT, array_int array<int>, array_array array<array<string>>, array_struct array<row<a string,`start` string,`end` string>>, array_map array<map<int,string>>, dt date, array_dt array<date> ) WITH ( 'connector' = 'doris', 'fenodes' = 'FE_IP:8030', 'table.identifier' = 'tmp.test_table', 'username' = 'root', 'password' = 'passwd', 'sink.properties.format' = 'json', 'sink.buffer-count' = '4', 'sink.buffer-size' = '4086', 'sink.label-prefix' = 'doris_label_sunjq_20250616_001', 'sink.properties.read_json_by_line' = 'true' ); INSERT INTO doris_test_sink select * from source; ``` <img width="2061" alt="Image" src="https://github.com/user-attachments/assets/cf50d4ba-98df-4960-9d3a-2f3b46f317e0"; /> ### Solution When serializing, recursively process the data in ArrayData ``` java // org.apache.doris.flink.deserialization.converter.DorisRowConverter#convertArrayData(org.apache.flink.table.data.ArrayData, org.apache.flink.table.types.logical.LogicalType) private static List<Object> convertArrayData(ArrayData array, LogicalType type) { LogicalType elementType = ((ArrayType) type).getElementType(); List<Object> values; if (array instanceof GenericArrayData) { values = Arrays.asList(((GenericArrayData) array).toObjectArray()); } else if (array instanceof BinaryArrayData) { values = Arrays.asList(((BinaryArrayData) array).toObjectArray(elementType)); } else { throw new UnsupportedOperationException("Unsupported array data: " + array.getClass()); } if (LogicalTypeRoot.DATE.equals(elementType.getTypeRoot())) { return values.stream() .map(date -> Date.valueOf(LocalDate.ofEpochDay((Integer) date))) .collect(Collectors.toList()); } if (LogicalTypeRoot.ARRAY.equals(elementType.getTypeRoot())) { return values.stream() .map(arr -> convertArrayData((ArrayData) arr, elementType)) .collect(Collectors.toList()); } if (LogicalTypeRoot.MAP.equals(elementType.getTypeRoot())) { return values.stream() .map(arr -> writeValueAsString(convertMapData((MapData) arr, elementType))) .collect(Collectors.toList()); } if (LogicalTypeRoot.ROW.equals(elementType.getTypeRoot())) { return values.stream() .map( arr -> writeValueAsString( convertRowData(GenericRowData.of(arr), 0, elementType))) .collect(Collectors.toList()); } return values; } ``` ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
