uros-b commented on code in PR #17537: URL: https://github.com/apache/iceberg/pull/17537#discussion_r3735215755
########## site/docs/blog/posts/2026-08-10-variant-in-apache-iceberg.md: ########## @@ -0,0 +1,144 @@ +--- +date: 2026-08-10 +title: "Semi-Structured Data in Apache Iceberg: Meet the Variant Type" +slug: variant-in-apache-iceberg +authors: + - nssalian +categories: + - blog +--- + +<!-- + - Licensed to the Apache Software Foundation (ASF) under one or more + - contributor license agreements. See the NOTICE file distributed with + - this work for additional information regarding copyright ownership. + - The ASF licenses this file to You under the Apache License, Version 2.0 + - (the "License"); you may not use this file except in compliance with + - the License. You may obtain a copy of the License at + - + - http://www.apache.org/licenses/LICENSE-2.0 + - + - Unless required by applicable law or agreed to in writing, software + - distributed under the License is distributed on an "AS IS" BASIS, + - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + - See the License for the specific language governing permissions and + - limitations under the License. + --> + +Semi-structured data, such as JSON-like documents whose fields differ from row to row, has always been a poor fit for table formats built around fixed schemas. Iceberg v3 adds the Variant type for exactly this data: a single column can hold values of arbitrary, evolving shape, stored in a compact binary form that engines read and write consistently. + +This is the first post in a series on Variant in Apache Iceberg. It covers what Variant is, why it exists, and how it fits into an Iceberg table. Later posts cover shredding, the technique that stores frequently accessed Variant fields as typed, columnar data. Variant is stored in Parquet, Avro, and ORC; shredding is currently available only in Parquet. + +<!-- more --> + +## What is Variant? + +Variant is a self-describing type for semi-structured data. A single Variant column can hold objects, arrays, and primitive values. + +## Why Variant? + +Consider event data whose shape changes over time: + +```json +{"timestamp": "2026-01-15T10:30:00Z", "user": 5, "event": "login"} +{"timestamp": "2026-01-15T11:45:00Z", "user": 5, "event": "purchase", "amount": 99.99} +{"timestamp": "2026-01-15T12:00:00Z", "user": 7, "event": "login", "device": "mobile"} +``` + +Two traditional approaches handle this, and both have drawbacks: + +- **JSON stored as a string.** This is flexible, but reading a single field means parsing the whole text. JSON's type system is also thin: a timestamp is just a string, and a number's precision is ambiguous. +- **A rigid, flattened schema.** This is fast to query, but every new field is a schema migration, and sparse or one-off fields waste space. + +Binary encodings such as BSON store values in binary form, but they still repeat every field name (`"timestamp"`, `"user"`, `"event"`) in every row. + +Variant is as flexible as JSON but stores data in a compact, typed binary form. Values keep their native types: a timestamp stays a timestamp and a decimal keeps its precision, instead of collapsing to JSON's strings and numbers. Field names are stored once in binary, so there is none of the per-row string overhead of JSON or BSON. No schema is declared up front, so documents of different shapes coexist in one column and a new field needs no migration. + +## Variant in Apache Iceberg + +Variant was added to the Iceberg type system in the v3 spec, alongside other v3 extensions such as nanosecond timestamps and the `geometry` and `geography` types. The spec places it in its own category: `variant` is "neither a primitive type nor a nested type." It is richer than a primitive, yet has no fixed, declared shape the way a struct or list does. + +A Variant value is similar to JSON, but with a wider set of primitives including `date`, `timestamp`, `timestamptz`, `binary`, and `decimal`. It can also nest: + +1. A **Variant array** is an ordered collection of Variant values. Unlike an Iceberg list, its elements are not constrained to a single element type. +2. A **Variant object** is a collection of string-keyed fields whose values are themselves Variant values. Unlike an Iceberg struct, its fields are not a fixed, named set of typed columns. + +### How it is stored + +Iceberg does not define its own binary encoding for Variant. The type and its encoding come from the [Apache Parquet project](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md), and Iceberg uses that encoding as-is, so a `variant` column maps to a Parquet `group` with two binary fields: + +```parquet +optional group event_data (VARIANT(1)) { + required binary metadata; + required binary value; +} +``` + +- `metadata` holds the dictionary of field names shared across the value, so names are stored once instead of per row. +- `value` holds the data itself in compact binary form: primitives, arrays, and objects. + +Both fields are accessed by name and carry no Iceberg field IDs. Variant is stored as this `metadata` + `value` pair in Parquet, Avro, and ORC. One optimization sets Parquet apart: shredding is supported only there. In Avro and ORC, a Variant is always the single unshredded pair. + +### Reading and writing across engines + +- **Apache Spark 4.0 and 4.1** create tables with `VARIANT` columns and read and write them. Reading landed in Iceberg 1.10.0 on Spark 4.0; Iceberg 1.11.0 added Spark 4.1 and writing shredded Variant on both Spark 4.0 and 4.1. +- **Apache Flink 2.1** added Variant support in Iceberg 1.11.0. Review Comment: +1 with @Guosmilesmile's comment. Shredded-write support for Flink (PR #15596) was merged to main after 1.11.0 and is milestoned for 1.12.0, so not yet released. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
