[I] v3 initial-default rows are silently dropped when a query filters on the defaulted column [iceberg]

Fri, 05 Jun 2026 01:48:21 -0700 


cbb330 opened a new issue, #16690:
URL: https://github.com/apache/iceberg/issues/16690

   ### Apache Iceberg version
   
   1.11.0 (latest release)
   
   ### Query engine
   
   Spark
   
   ### Please describe the bug 🐞
   
   <h3 id="apache-iceberg-version" style="margin-top: 1.4em; font-size: 1.15em; 
color: rgb(31, 35, 40); font-family: -apple-system, &quot;system-ui&quot;, 
&quot;Segoe UI&quot;, Helvetica, Arial, sans-serif; font-style: normal; 
font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: 
normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; 
widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: 
normal; background-color: rgb(253, 253, 253); text-decoration-thickness: 
initial; text-decoration-style: initial; text-decoration-color: 
initial;">Apache Iceberg version</h3><p style="margin: 1em 0px; color: rgb(31, 
35, 40); font-family: -apple-system, &quot;system-ui&quot;, &quot;Segoe 
UI&quot;, Helvetica, Arial, sans-serif; font-size: medium; font-style: normal; 
font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; 
letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; 
text-transform: none; wid
 ows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: 
normal; background-color: rgb(253, 253, 253); text-decoration-thickness: 
initial; text-decoration-style: initial; text-decoration-color: initial;">main 
(development)</p><h3 id="query-engine" style="margin-top: 1.4em; font-size: 
1.15em; color: rgb(31, 35, 40); font-family: -apple-system, 
&quot;system-ui&quot;, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif; 
font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; 
letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; 
text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 
0px; white-space: normal; background-color: rgb(253, 253, 253); 
text-decoration-thickness: initial; text-decoration-style: initial; 
text-decoration-color: initial;">Query engine</h3><p style="margin: 1em 0px; 
color: rgb(31, 35, 40); font-family: -apple-system, &quot;system-ui&quot;, 
&quot;Segoe UI&quot;, Helvetica, Arial, sans-se
 rif; font-size: medium; font-style: normal; font-variant-ligatures: normal; 
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 
2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; 
word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; 
background-color: rgb(253, 253, 253); text-decoration-thickness: initial; 
text-decoration-style: initial; text-decoration-color: initial;">Spark</p><h3 
id="please-describe-the-bug" style="margin-top: 1.4em; font-size: 1.15em; 
color: rgb(31, 35, 40); font-family: -apple-system, &quot;system-ui&quot;, 
&quot;Segoe UI&quot;, Helvetica, Arial, sans-serif; font-style: normal; 
font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: 
normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; 
widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: 
normal; background-color: rgb(253, 253, 253); text-decoration-thickness: 
initial; text-decoratio
 n-style: initial; text-decoration-color: initial;">Please describe the bug 
🐞</h3><p style="margin: 1em 0px; color: rgb(31, 35, 40); font-family: 
-apple-system, &quot;system-ui&quot;, &quot;Segoe UI&quot;, Helvetica, Arial, 
sans-serif; font-size: medium; font-style: normal; font-variant-ligatures: 
normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; 
orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 
2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; 
background-color: rgb(253, 253, 253); text-decoration-thickness: initial; 
text-decoration-style: initial; text-decoration-color: initial;">A column added 
by schema evolution with an<span> </span><code style="font-family: 
ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 13.6px; margin: 0px; 
hyphens: manual; white-space: pre-wrap; background: rgb(239, 241, 243); 
padding: 0.2em 0.4em; border-radius: 6px;">initial-default</code><span> 
</span>is correctl
 y backfilled on a<span> </span><strong>full scan</strong>, but is<span> 
</span><strong>silently dropped from results whenever a query filters on that 
column</strong>. No exception is raised — it returns silent wrong results 
(missing rows).</p><p style="margin: 1em 0px; color: rgb(31, 35, 40); 
font-family: -apple-system, &quot;system-ui&quot;, &quot;Segoe UI&quot;, 
Helvetica, Arial, sans-serif; font-size: medium; font-style: normal; 
font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; 
letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; 
text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 
0px; white-space: normal; background-color: rgb(253, 253, 253); 
text-decoration-thickness: initial; text-decoration-style: initial; 
text-decoration-color: initial;">Reproduced on<span> </span><code 
style="font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 
13.6px; margin: 0px; hyphens: manual; white-space:
  pre-wrap; background: rgb(239, 241, 243); padding: 0.2em 0.4em; 
border-radius: 6px;">main</code><span> </span>@<span> </span><code 
style="font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 
13.6px; margin: 0px; hyphens: manual; white-space: pre-wrap; background: 
rgb(239, 241, 243); padding: 0.2em 0.4em; border-radius: 
6px;">c00669fde</code><span> </span>(and release<span> </span><code 
style="font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 
13.6px; margin: 0px; hyphens: manual; white-space: pre-wrap; background: 
rgb(239, 241, 243); padding: 0.2em 0.4em; border-radius: 
6px;">1.10.1</code>),<span> </span><strong>Spark 3.5, 4.0, and 
4.1</strong><span> </span>(identical result on all three), Parquet, 
format-version 3. The root cause is in shared<span> </span><code 
style="font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 
13.6px; margin: 0px; hyphens: manual; white-space: pre-wrap; background: 
rgb(239, 241, 243); padd
 ing: 0.2em 0.4em; border-radius: 6px;">core</code>/<code style="font-family: 
ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 13.6px; margin: 0px; 
hyphens: manual; white-space: pre-wrap; background: rgb(239, 241, 243); 
padding: 0.2em 0.4em; border-radius: 6px;">parquet</code><span> </span>read 
code, so it is not Spark-version-specific; the same per-file record filter is 
used by the Flink and generic readers as well.</p><p style="margin: 1em 0px; 
color: rgb(31, 35, 40); font-family: -apple-system, &quot;system-ui&quot;, 
&quot;Segoe UI&quot;, Helvetica, Arial, sans-serif; font-size: medium; 
font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; 
font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; 
text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; 
-webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(253, 
253, 253); text-decoration-thickness: initial; text-decoration-style: initial; 
text-deco
 ration-color: initial;">The defaulted column has to be<span> </span><em>added 
to an existing table</em><span> </span>to be absent from older files, which 
today is only reachable through the schema-evolution API — Spark SQL<span> 
</span><code style="font-family: ui-monospace, SFMono-Regular, Menlo, 
monospace; font-size: 13.6px; margin: 0px; hyphens: manual; white-space: 
pre-wrap; background: rgb(239, 241, 243); padding: 0.2em 0.4em; border-radius: 
6px;">ALTER TABLE ... ADD COLUMN ... DEFAULT</code><span> </span>is currently 
rejected (<code style="font-family: ui-monospace, SFMono-Regular, Menlo, 
monospace; font-size: 13.6px; margin: 0px; hyphens: manual; white-space: 
pre-wrap; background: rgb(239, 241, 243); padding: 0.2em 0.4em; border-radius: 
6px;">UnsupportedOperationException: setting default values in Spark is 
currently unsupported</code>), which is part of why this filtered-read case is 
untested.</p><h4 id="repro" style="margin-top: 1.4em; color: rgb(31, 35, 40); 
font-fam
 ily: -apple-system, &quot;system-ui&quot;, &quot;Segoe UI&quot;, Helvetica, 
Arial, sans-serif; font-size: medium; font-style: normal; 
font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: 
normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; 
widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: 
normal; background-color: rgb(253, 253, 253); text-decoration-thickness: 
initial; text-decoration-style: initial; text-decoration-color: 
initial;">Repro</h4><p style="margin: 1em 0px; color: rgb(31, 35, 40); 
font-family: -apple-system, &quot;system-ui&quot;, &quot;Segoe UI&quot;, 
Helvetica, Arial, sans-serif; font-size: medium; font-style: normal; 
font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; 
letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; 
text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 
0px; white-space: normal; background-color: rgb(253, 25
 3, 253); text-decoration-thickness: initial; text-decoration-style: initial; 
text-decoration-color: initial;"><code style="font-family: ui-monospace, 
SFMono-Regular, Menlo, monospace; font-size: 13.6px; margin: 0px; hyphens: 
manual; white-space: pre-wrap; background: rgb(239, 241, 243); padding: 0.2em 
0.4em; border-radius: 6px;">id=1</code><span> </span>is written before 
column<span> </span><code style="font-family: ui-monospace, SFMono-Regular, 
Menlo, monospace; font-size: 13.6px; margin: 0px; hyphens: manual; white-space: 
pre-wrap; background: rgb(239, 241, 243); padding: 0.2em 0.4em; border-radius: 
6px;">c</code><span> </span>exists (so<span> </span><code style="font-family: 
ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 13.6px; margin: 0px; 
hyphens: manual; white-space: pre-wrap; background: rgb(239, 241, 243); 
padding: 0.2em 0.4em; border-radius: 6px;">c</code><span> </span>is physically 
absent from that file);<span> </span><code style="font-family: ui-monospa
 ce, SFMono-Regular, Menlo, monospace; font-size: 13.6px; margin: 0px; hyphens: 
manual; white-space: pre-wrap; background: rgb(239, 241, 243); padding: 0.2em 
0.4em; border-radius: 6px;">c</code><span> </span>is then added with<span> 
</span><code style="font-family: ui-monospace, SFMono-Regular, Menlo, 
monospace; font-size: 13.6px; margin: 0px; hyphens: manual; white-space: 
pre-wrap; background: rgb(239, 241, 243); padding: 0.2em 0.4em; border-radius: 
6px;">initial-default 'US'</code>;<span> </span><code style="font-family: 
ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 13.6px; margin: 0px; 
hyphens: manual; white-space: pre-wrap; background: rgb(239, 241, 243); 
padding: 0.2em 0.4em; border-radius: 6px;">id=2</code><span> </span>is written 
with<span> </span><code style="font-family: ui-monospace, SFMono-Regular, 
Menlo, monospace; font-size: 13.6px; margin: 0px; hyphens: manual; white-space: 
pre-wrap; background: rgb(239, 241, 243); padding: 0.2em 0.4em; border-radius: 
 6px;">c='US'</code>.</p><div class="sourceCode" id="cb2" 
style="background-color: rgb(253, 253, 253); overflow: auto; margin: 1em 0px; 
color: rgb(31, 35, 40); font-family: -apple-system, &quot;system-ui&quot;, 
&quot;Segoe UI&quot;, Helvetica, Arial, sans-serif; font-size: medium; 
font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; 
font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; 
text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; 
-webkit-text-stroke-width: 0px; white-space: normal; text-decoration-thickness: 
initial; text-decoration-style: initial; text-decoration-color: initial;"><pre 
class="sourceCode java" style="margin: 0px; overflow: visible; background: 
transparent; padding: 14px; border-radius: 8px;"><code class="sourceCode java" 
style="font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 
13.6px; margin: 0px; hyphens: manual; white-space: pre; background: none 
transparent; padding: 0px; bo
 rder-radius: 6px; overflow: visible; overflow-wrap: normal; position: 
relative;"><span id="cb2-1" style="color: inherit; text-decoration: inherit; 
display: inline-block; line-height: 1.25;"><a href="#cb2-1" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a><span 
class="co" style="color: rgb(96, 160, 176); font-style: italic;">// 
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestDefaultFilteredRead.java</span></span>
   <span id="cb2-2" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-2" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a><span 
class="kw" style="color: rgb(0, 112, 32); font-weight: bold;">public</span> 
<span class="kw" style="color: rgb(0, 112, 32); font-weight: 
bold;">class</span> TestDefaultFilteredRead <span class="kw" style="color: 
rgb(0, 112, 32); font-weight: bold;">extends</span> CatalogTestBase <span 
class="op" style="color: rgb(102, 102, 102);">{</span></span>
   <span id="cb2-3" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-3" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: 
none;"></a></span>
   <span id="cb2-4" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-4" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>  
<span class="at" style="color: rgb(125, 144, 41);">@Parameters</span><span 
class="op" style="color: rgb(102, 102, 102);">(</span>name <span class="op" 
style="color: rgb(102, 102, 102);">=</span> <span class="st" style="color: 
rgb(64, 112, 160);">"catalogName = {0}, implementation = {1}, config = 
{2}"</span><span class="op" style="color: rgb(102, 102, 102);">)</span></span>
   <span id="cb2-5" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-5" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>  
<span class="kw" style="color: rgb(0, 112, 32); font-weight: 
bold;">protected</span> <span class="dt" style="color: rgb(144, 32, 
0);">static</span> <span class="bu" style="color: rgb(0, 128, 
0);">Object</span><span class="op" style="color: rgb(102, 102, 
102);">[][]</span> <span class="fu" style="color: rgb(6, 40, 
126);">parameters</span><span class="op" style="color: rgb(102, 102, 
102);">()</span> <span class="op" style="color: rgb(102, 102, 
102);">{</span></span>
   <span id="cb2-6" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-6" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>    
<span class="cf" style="color: rgb(0, 112, 32); font-weight: 
bold;">return</span> <span class="kw" style="color: rgb(0, 112, 32); 
font-weight: bold;">new</span> <span class="bu" style="color: rgb(0, 128, 
0);">Object</span><span class="op" style="color: rgb(102, 102, 
102);">[][]</span> <span class="op" style="color: rgb(102, 102, 
102);">{</span></span>
   <span id="cb2-7" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-7" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>      
<span class="op" style="color: rgb(102, 102, 
102);">{</span>SparkCatalogConfig<span class="op" style="color: rgb(102, 102, 
102);">.</span><span class="fu" style="color: rgb(6, 40, 
126);">HADOOP</span><span class="op" style="color: rgb(102, 102, 
102);">.</span><span class="fu" style="color: rgb(6, 40, 
126);">catalogName</span><span class="op" style="color: rgb(102, 102, 
102);">(),</span></span>
   <span id="cb2-8" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-8" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>      
 SparkCatalogConfig<span class="op" style="color: rgb(102, 102, 
102);">.</span><span class="fu" style="color: rgb(6, 40, 
126);">HADOOP</span><span class="op" style="color: rgb(102, 102, 
102);">.</span><span class="fu" style="color: rgb(6, 40, 
126);">implementation</span><span class="op" style="color: rgb(102, 102, 
102);">(),</span></span>
   <span id="cb2-9" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-9" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>      
 SparkCatalogConfig<span class="op" style="color: rgb(102, 102, 
102);">.</span><span class="fu" style="color: rgb(6, 40, 
126);">HADOOP</span><span class="op" style="color: rgb(102, 102, 
102);">.</span><span class="fu" style="color: rgb(6, 40, 
126);">properties</span><span class="op" style="color: rgb(102, 102, 
102);">()}</span></span>
   <span id="cb2-10" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-10" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>    
<span class="op" style="color: rgb(102, 102, 102);">};</span></span>
   <span id="cb2-11" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-11" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>  
<span class="op" style="color: rgb(102, 102, 102);">}</span></span>
   <span id="cb2-12" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-12" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: 
none;"></a></span>
   <span id="cb2-13" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-13" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>  
<span class="at" style="color: rgb(125, 144, 41);">@AfterEach</span></span>
   <span id="cb2-14" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-14" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>  
<span class="kw" style="color: rgb(0, 112, 32); font-weight: 
bold;">public</span> <span class="dt" style="color: rgb(144, 32, 
0);">void</span> <span class="fu" style="color: rgb(6, 40, 
126);">dropTable</span><span class="op" style="color: rgb(102, 102, 
102);">()</span> <span class="op" style="color: rgb(102, 102, 
102);">{</span></span>
   <span id="cb2-15" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-15" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>    
<span class="fu" style="color: rgb(6, 40, 126);">sql</span><span class="op" 
style="color: rgb(102, 102, 102);">(</span><span class="st" style="color: 
rgb(64, 112, 160);">"DROP TABLE IF EXISTS %s"</span><span class="op" 
style="color: rgb(102, 102, 102);">,</span> tableName<span class="op" 
style="color: rgb(102, 102, 102);">);</span></span>
   <span id="cb2-16" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-16" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>  
<span class="op" style="color: rgb(102, 102, 102);">}</span></span>
   <span id="cb2-17" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-17" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: 
none;"></a></span>
   <span id="cb2-18" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-18" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>  
<span class="at" style="color: rgb(125, 144, 41);">@TestTemplate</span></span>
   <span id="cb2-19" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-19" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>  
<span class="kw" style="color: rgb(0, 112, 32); font-weight: 
bold;">public</span> <span class="dt" style="color: rgb(144, 32, 
0);">void</span> <span class="fu" style="color: rgb(6, 40, 
126);">filteredReadOverAbsentDefaultColumn</span><span class="op" style="color: 
rgb(102, 102, 102);">()</span> <span class="op" style="color: rgb(102, 102, 
102);">{</span></span>
   <span id="cb2-20" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-20" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>    
<span class="fu" style="color: rgb(6, 40, 126);">sql</span><span class="op" 
style="color: rgb(102, 102, 102);">(</span><span class="st" style="color: 
rgb(64, 112, 160);">"CREATE TABLE %s (id bigint, name string) USING iceberg 
"</span></span>
   <span id="cb2-21" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-21" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>      
  <span class="op" style="color: rgb(102, 102, 102);">+</span> <span class="st" 
style="color: rgb(64, 112, 160);">"TBLPROPERTIES 
('format-version'='3','write.format.default'='parquet')"</span><span class="op" 
style="color: rgb(102, 102, 102);">,</span> tableName<span class="op" 
style="color: rgb(102, 102, 102);">);</span></span>
   <span id="cb2-22" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-22" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>    
<span class="fu" style="color: rgb(6, 40, 126);">sql</span><span class="op" 
style="color: rgb(102, 102, 102);">(</span><span class="st" style="color: 
rgb(64, 112, 160);">"INSERT INTO %s VALUES (1, 'Alice')"</span><span class="op" 
style="color: rgb(102, 102, 102);">,</span> tableName<span class="op" 
style="color: rgb(102, 102, 102);">);</span>          <span class="co" 
style="color: rgb(96, 160, 176); font-style: italic;">// F_old: column c 
absent</span></span>
   <span id="cb2-23" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-23" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: 
none;"></a></span>
   <span id="cb2-24" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-24" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>    
Table table <span class="op" style="color: rgb(102, 102, 102);">=</span> 
validationCatalog<span class="op" style="color: rgb(102, 102, 
102);">.</span><span class="fu" style="color: rgb(6, 40, 
126);">loadTable</span><span class="op" style="color: rgb(102, 102, 
102);">(</span>tableIdent<span class="op" style="color: rgb(102, 102, 
102);">);</span></span>
   <span id="cb2-25" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-25" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>    
table<span class="op" style="color: rgb(102, 102, 102);">.</span><span 
class="fu" style="color: rgb(6, 40, 126);">updateSchema</span><span class="op" 
style="color: rgb(102, 102, 102);">().</span><span class="fu" style="color: 
rgb(6, 40, 126);">addColumn</span><span class="op" style="color: rgb(102, 102, 
102);">(</span><span class="st" style="color: rgb(64, 112, 
160);">"c"</span><span class="op" style="color: rgb(102, 102, 102);">,</span> 
<span class="bu" style="color: rgb(0, 128, 0);">Types</span><span class="op" 
style="color: rgb(102, 102, 102);">.</span><span class="fu" style="color: 
rgb(6, 40, 126);">StringType</span><span class="op" style="color: rgb(102, 102, 
102);">.</span><span class="fu" style="color: rgb(6, 40, 126);">get</span><span 
class="op" style=
 "color: rgb(102, 102, 102);">(),</span> Expressions<span class="op" 
style="color: rgb(102, 102, 102);">.</span><span class="fu" style="color: 
rgb(6, 40, 126);">lit</span><span class="op" style="color: rgb(102, 102, 
102);">(</span><span class="st" style="color: rgb(64, 112, 
160);">"US"</span><span class="op" style="color: rgb(102, 102, 
102);">)).</span><span class="fu" style="color: rgb(6, 40, 
126);">commit</span><span class="op" style="color: rgb(102, 102, 
102);">();</span></span>
   <span id="cb2-26" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-26" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>    
<span class="fu" style="color: rgb(6, 40, 126);">sql</span><span class="op" 
style="color: rgb(102, 102, 102);">(</span><span class="st" style="color: 
rgb(64, 112, 160);">"REFRESH TABLE %s"</span><span class="op" style="color: 
rgb(102, 102, 102);">,</span> tableName<span class="op" style="color: rgb(102, 
102, 102);">);</span></span>
   <span id="cb2-27" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-27" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>    
<span class="fu" style="color: rgb(6, 40, 126);">sql</span><span class="op" 
style="color: rgb(102, 102, 102);">(</span><span class="st" style="color: 
rgb(64, 112, 160);">"INSERT INTO %s VALUES (2, 'Bob', 'US')"</span><span 
class="op" style="color: rgb(102, 102, 102);">,</span> tableName<span 
class="op" style="color: rgb(102, 102, 102);">);</span>      <span class="co" 
style="color: rgb(96, 160, 176); font-style: italic;">// F_new: c = 
'US'</span></span>
   <span id="cb2-28" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-28" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: 
none;"></a></span>
   <span id="cb2-29" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-29" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>    
<span class="co" style="color: rgb(96, 160, 176); font-style: italic;">// full 
scan is correct -&gt; the default is materialized for id=1</span></span>
   <span id="cb2-30" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-30" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>    
<span class="fu" style="color: rgb(6, 40, 126);">assertThat</span><span 
class="op" style="color: rgb(102, 102, 102);">(</span><span class="fu" 
style="color: rgb(6, 40, 126);">sql</span><span class="op" style="color: 
rgb(102, 102, 102);">(</span><span class="st" style="color: rgb(64, 112, 
160);">"SELECT id FROM %s ORDER BY id"</span><span class="op" style="color: 
rgb(102, 102, 102);">,</span> tableName<span class="op" style="color: rgb(102, 
102, 102);">))</span></span>
   <span id="cb2-31" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-31" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>      
  <span class="op" style="color: rgb(102, 102, 102);">.</span><span class="fu" 
style="color: rgb(6, 40, 126);">containsExactly</span><span class="op" 
style="color: rgb(102, 102, 102);">(</span><span class="fu" style="color: 
rgb(6, 40, 126);">row</span><span class="op" style="color: rgb(102, 102, 
102);">(</span><span class="dv" style="color: rgb(64, 160, 
112);">1L</span><span class="op" style="color: rgb(102, 102, 102);">),</span> 
<span class="fu" style="color: rgb(6, 40, 126);">row</span><span class="op" 
style="color: rgb(102, 102, 102);">(</span><span class="dv" style="color: 
rgb(64, 160, 112);">2L</span><span class="op" style="color: rgb(102, 102, 
102);">));</span></span>
   <span id="cb2-32" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-32" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>    
<span class="co" style="color: rgb(96, 160, 176); font-style: italic;">// 
</span><span class="al" style="color: rgb(255, 0, 0); font-weight: 
bold;">BUG</span><span class="co" style="color: rgb(96, 160, 176); font-style: 
italic;">: filtering on c drops id=1 (the backfilled row). Expected [1, 2], 
actual [2].</span></span>
   <span id="cb2-33" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-33" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>    
<span class="fu" style="color: rgb(6, 40, 126);">assertThat</span><span 
class="op" style="color: rgb(102, 102, 102);">(</span><span class="fu" 
style="color: rgb(6, 40, 126);">sql</span><span class="op" style="color: 
rgb(102, 102, 102);">(</span><span class="st" style="color: rgb(64, 112, 
160);">"SELECT id FROM %s WHERE c = 'US' ORDER BY id"</span><span class="op" 
style="color: rgb(102, 102, 102);">,</span> tableName<span class="op" 
style="color: rgb(102, 102, 102);">))</span></span>
   <span id="cb2-34" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-34" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>      
  <span class="op" style="color: rgb(102, 102, 102);">.</span><span class="fu" 
style="color: rgb(6, 40, 126);">containsExactly</span><span class="op" 
style="color: rgb(102, 102, 102);">(</span><span class="fu" style="color: 
rgb(6, 40, 126);">row</span><span class="op" style="color: rgb(102, 102, 
102);">(</span><span class="dv" style="color: rgb(64, 160, 
112);">1L</span><span class="op" style="color: rgb(102, 102, 102);">),</span> 
<span class="fu" style="color: rgb(6, 40, 126);">row</span><span class="op" 
style="color: rgb(102, 102, 102);">(</span><span class="dv" style="color: 
rgb(64, 160, 112);">2L</span><span class="op" style="color: rgb(102, 102, 
102);">));</span>        <span class="co" style="color: rgb(96, 160, 176); 
font-style: italic;">// fails today
 : only [2]</span></span>
   <span id="cb2-35" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-35" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>    
<span class="fu" style="color: rgb(6, 40, 126);">assertThat</span><span 
class="op" style="color: rgb(102, 102, 102);">(</span><span class="fu" 
style="color: rgb(6, 40, 126);">sql</span><span class="op" style="color: 
rgb(102, 102, 102);">(</span><span class="st" style="color: rgb(64, 112, 
160);">"SELECT id FROM %s WHERE c IS NOT NULL ORDER BY id"</span><span 
class="op" style="color: rgb(102, 102, 102);">,</span> tableName<span 
class="op" style="color: rgb(102, 102, 102);">))</span></span>
   <span id="cb2-36" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-36" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>      
  <span class="op" style="color: rgb(102, 102, 102);">.</span><span class="fu" 
style="color: rgb(6, 40, 126);">containsExactly</span><span class="op" 
style="color: rgb(102, 102, 102);">(</span><span class="fu" style="color: 
rgb(6, 40, 126);">row</span><span class="op" style="color: rgb(102, 102, 
102);">(</span><span class="dv" style="color: rgb(64, 160, 
112);">1L</span><span class="op" style="color: rgb(102, 102, 102);">),</span> 
<span class="fu" style="color: rgb(6, 40, 126);">row</span><span class="op" 
style="color: rgb(102, 102, 102);">(</span><span class="dv" style="color: 
rgb(64, 160, 112);">2L</span><span class="op" style="color: rgb(102, 102, 
102);">));</span>        <span class="co" style="color: rgb(96, 160, 176); 
font-style: italic;">// fails today
 : only [2]</span></span>
   <span id="cb2-37" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-37" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a>  
<span class="op" style="color: rgb(102, 102, 102);">}</span></span>
   <span id="cb2-38" style="color: inherit; text-decoration: inherit; display: 
inline-block; line-height: 1.25;"><a href="#cb2-38" aria-hidden="true" 
tabindex="-1" style="color: rgb(9, 105, 218); text-decoration: none;"></a><span 
class="op" style="color: rgb(102, 102, 
102);">}</span></span></code></pre></div><h4 id="expected-behavior" 
style="margin-top: 1.4em; color: rgb(31, 35, 40); font-family: -apple-system, 
&quot;system-ui&quot;, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif; 
font-size: medium; font-style: normal; font-variant-ligatures: normal; 
font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: 
start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; 
-webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(253, 
253, 253); text-decoration-thickness: initial; text-decoration-style: initial; 
text-decoration-color: initial;">Expected behavior</h4>
   Query | Expected
   -- | --
   SELECT id, c | (1,US) (2,US)
   WHERE c = 'US' | [1, 2]
   WHERE upper(c) = 'US' | [1, 2]
   WHERE c IS NOT NULL | [1, 2]
   WHERE c IS NULL | []
   
   <p style="margin: 1em 0px; color: rgb(31, 35, 40); font-family: 
-apple-system, &quot;system-ui&quot;, &quot;Segoe UI&quot;, Helvetica, Arial, 
sans-serif; font-size: medium; font-style: normal; font-variant-ligatures: 
normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; 
orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 
2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; 
background-color: rgb(253, 253, 253); text-decoration-thickness: initial; 
text-decoration-style: initial; text-decoration-color: initial;">The full scan 
proves the default is materialized for<span> </span><code style="font-family: 
ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 13.6px; margin: 0px; 
hyphens: manual; white-space: pre-wrap; background: rgb(239, 241, 243); 
padding: 0.2em 0.4em; border-radius: 6px;">id=1</code>; the filtered queries 
drop it whenever the predicate references<span> </span><code 
style="font-family: ui-m
 onospace, SFMono-Regular, Menlo, monospace; font-size: 13.6px; margin: 0px; 
hyphens: manual; white-space: pre-wrap; background: rgb(239, 241, 243); 
padding: 0.2em 0.4em; border-radius: 6px;">c</code>.</p><h4 id="root-cause" 
style="margin-top: 1.4em; color: rgb(31, 35, 40); font-family: -apple-system, 
&quot;system-ui&quot;, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif; 
font-size: medium; font-style: normal; font-variant-ligatures: normal; 
font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: 
start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; 
-webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(253, 
253, 253); text-decoration-thickness: initial; text-decoration-style: initial; 
text-decoration-color: initial;">Root cause</h4><p style="margin: 1em 0px; 
color: rgb(31, 35, 40); font-family: -apple-system, &quot;system-ui&quot;, 
&quot;Segoe UI&quot;, Helvetica, Arial, sans-serif; font-size: medium; 
font-style: norma
 l; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 
400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; 
text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 
0px; white-space: normal; background-color: rgb(253, 253, 253); 
text-decoration-thickness: initial; text-decoration-style: initial; 
text-decoration-color: initial;">The default is injected in the per-format 
reader (<code style="font-family: ui-monospace, SFMono-Regular, Menlo, 
monospace; font-size: 13.6px; margin: 0px; hyphens: manual; white-space: 
pre-wrap; background: rgb(239, 241, 243); padding: 0.2em 0.4em; border-radius: 
6px;">BaseParquetReaders</code>, via<span> </span><code style="font-family: 
ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 13.6px; margin: 0px; 
hyphens: manual; white-space: pre-wrap; background: rgb(239, 241, 243); 
padding: 0.2em 0.4em; border-radius: 
6px;">NestedField.initialDefault()</code>)<span> </span><strong>afte
 r</strong><span> </span>record-level filtering. The read applies the residual 
as a record filter (<code style="font-family: ui-monospace, SFMono-Regular, 
Menlo, monospace; font-size: 13.6px; margin: 0px; hyphens: manual; white-space: 
pre-wrap; background: rgb(239, 241, 243); padding: 0.2em 0.4em; border-radius: 
6px;">Parquet.ReadBuilder.filterRecords=true</code><span> </span>→<span> 
</span><code style="font-family: ui-monospace, SFMono-Regular, Menlo, 
monospace; font-size: 13.6px; margin: 0px; hyphens: manual; white-space: 
pre-wrap; background: rgb(239, 241, 243); padding: 0.2em 0.4em; border-radius: 
6px;">.useRecordFilter(...)</code>,<span> </span><code style="font-family: 
ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 13.6px; margin: 0px; 
hyphens: manual; white-space: pre-wrap; background: rgb(239, 241, 243); 
padding: 0.2em 0.4em; border-radius: 6px;">.filter(residual)</code><span> 
</span>in<span> </span><code style="font-family: ui-monospace, SFMono-Regular, M
 enlo, monospace; font-size: 13.6px; margin: 0px; hyphens: manual; white-space: 
pre-wrap; background: rgb(239, 241, 243); padding: 0.2em 0.4em; border-radius: 
6px;">BaseRowReader</code>). For a file physically missing<span> </span><code 
style="font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 
13.6px; margin: 0px; hyphens: manual; white-space: pre-wrap; background: 
rgb(239, 241, 243); padding: 0.2em 0.4em; border-radius: 6px;">c</code>, the 
record filter reads<span> </span><code style="font-family: ui-monospace, 
SFMono-Regular, Menlo, monospace; font-size: 13.6px; margin: 0px; hyphens: 
manual; white-space: pre-wrap; background: rgb(239, 241, 243); padding: 0.2em 
0.4em; border-radius: 6px;">c</code><span> </span>as null, so<span> 
</span><code style="font-family: ui-monospace, SFMono-Regular, Menlo, 
monospace; font-size: 13.6px; margin: 0px; hyphens: manual; white-space: 
pre-wrap; background: rgb(239, 241, 243); padding: 0.2em 0.4em; border-radius: 
6px;">c = 'U
 S'</code><span> </span>— and the<span> </span><code style="font-family: 
ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 13.6px; margin: 0px; 
hyphens: manual; white-space: pre-wrap; background: rgb(239, 241, 243); 
padding: 0.2em 0.4em; border-radius: 6px;">IsNotNull(c)</code><span> 
</span>Spark infers for any null-intolerant predicate (which is why even the 
un-pushable<span> </span><code style="font-family: ui-monospace, 
SFMono-Regular, Menlo, monospace; font-size: 13.6px; margin: 0px; hyphens: 
manual; white-space: pre-wrap; background: rgb(239, 241, 243); padding: 0.2em 
0.4em; border-radius: 6px;">upper(c) = 'US'</code><span> </span>drops rows) — 
matches nothing, and every record is dropped<span> </span><em>before</em><span> 
</span>the default is applied. This mirrors why partition columns are immune: 
those are folded out of the per-file residual by<span> </span><code 
style="font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 
13.6px; margin
 : 0px; hyphens: manual; white-space: pre-wrap; background: rgb(239, 241, 243); 
padding: 0.2em 0.4em; border-radius: 6px;">ResidualEvaluator</code>;<span> 
</span><code style="font-family: ui-monospace, SFMono-Regular, Menlo, 
monospace; font-size: 13.6px; margin: 0px; hyphens: manual; white-space: 
pre-wrap; background: rgb(239, 241, 243); padding: 0.2em 0.4em; border-radius: 
6px;">initial-default</code><span> </span>columns are not.</p><p style="margin: 
1em 0px; color: rgb(31, 35, 40); font-family: -apple-system, 
&quot;system-ui&quot;, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif; 
font-size: medium; font-style: normal; font-variant-ligatures: normal; 
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 
2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; 
word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; 
background-color: rgb(253, 253, 253); text-decoration-thickness: initial; 
text-decoration-style: initial; tex
 t-decoration-color: initial;">Manifest pruning is not the cause (a column with 
no file metrics returns<span> </span><code style="font-family: ui-monospace, 
SFMono-Regular, Menlo, monospace; font-size: 13.6px; margin: 0px; hyphens: 
manual; white-space: pre-wrap; background: rgb(239, 241, 243); padding: 0.2em 
0.4em; border-radius: 6px;">ROWS_MIGHT_MATCH</code>, so the file is kept). The 
drop is the reader-side record filter.</p><h4 id="environment" 
style="margin-top: 1.4em; color: rgb(31, 35, 40); font-family: -apple-system, 
&quot;system-ui&quot;, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif; 
font-size: medium; font-style: normal; font-variant-ligatures: normal; 
font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: 
start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; 
-webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(253, 
253, 253); text-decoration-thickness: initial; text-decoration-style: initial; 
text-decor
 ation-color: initial;">Environment</h4><ul style="padding-left: 1.7em; 
margin-top: 1em; color: rgb(31, 35, 40); font-family: -apple-system, 
&quot;system-ui&quot;, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif; 
font-size: medium; font-style: normal; font-variant-ligatures: normal; 
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 
2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; 
word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; 
background-color: rgb(253, 253, 253); text-decoration-thickness: initial; 
text-decoration-style: initial; text-decoration-color: 
initial;"><li>Iceberg:<span> </span><code style="font-family: ui-monospace, 
SFMono-Regular, Menlo, monospace; font-size: 13.6px; margin: 0px; hyphens: 
manual; white-space: pre-wrap; background: rgb(239, 241, 243); padding: 0.2em 
0.4em; border-radius: 6px;">main</code><span> </span>@<span> </span><code 
style="font-family: ui-monospace, SFMono-Regular, Men
 lo, monospace; font-size: 13.6px; margin: 0px; hyphens: manual; white-space: 
pre-wrap; background: rgb(239, 241, 243); padding: 0.2em 0.4em; border-radius: 
6px;">c00669fde</code><span> </span>(also<span> </span><code 
style="font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 
13.6px; margin: 0px; hyphens: manual; white-space: pre-wrap; background: 
rgb(239, 241, 243); padding: 0.2em 0.4em; border-radius: 
6px;">1.10.1</code>)</li><li>Spark: 3.5, 4.0, 4.1 (all reproduce), Parquet, 
Hadoop catalog, format-version 3</li></ul><h4 id="possible-fix" 
style="margin-top: 1.4em; color: rgb(31, 35, 40); font-family: -apple-system, 
&quot;system-ui&quot;, &quot;Segoe UI&quot;, Helvetica, Arial, sans-serif; 
font-size: medium; font-style: normal; font-variant-ligatures: normal; 
font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: 
start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; 
-webkit-text-stroke-width: 0px; white-space: normal; 
 background-color: rgb(253, 253, 253); text-decoration-thickness: initial; 
text-decoration-style: initial; text-decoration-color: initial;">Possible 
fix</h4><p style="margin: 1em 0px; color: rgb(31, 35, 40); font-family: 
-apple-system, &quot;system-ui&quot;, &quot;Segoe UI&quot;, Helvetica, Arial, 
sans-serif; font-size: medium; font-style: normal; font-variant-ligatures: 
normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; 
orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 
2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; 
background-color: rgb(253, 253, 253); text-decoration-thickness: initial; 
text-decoration-style: initial; text-decoration-color: initial;">Fold an 
absent-with-default column out of the per-file residual the way partition 
constants already are — substitute the<span> </span><code style="font-family: 
ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 13.6px; margin: 0px; 
hyphens: ma
 nual; white-space: pre-wrap; background: rgb(239, 241, 243); padding: 0.2em 
0.4em; border-radius: 6px;">initialDefault</code><span> </span>literal for any 
field absent from the file’s physical schema and constant-fold (<code 
style="font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 
13.6px; margin: 0px; hyphens: manual; white-space: pre-wrap; background: 
rgb(239, 241, 243); padding: 0.2em 0.4em; border-radius: 
6px;">c='US'</code>→true,<span> </span><code style="font-family: ui-monospace, 
SFMono-Regular, Menlo, monospace; font-size: 13.6px; margin: 0px; hyphens: 
manual; white-space: pre-wrap; background: rgb(239, 241, 243); padding: 0.2em 
0.4em; border-radius: 6px;">c='CA'</code>→false,<span> </span><code 
style="font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 
13.6px; margin: 0px; hyphens: manual; white-space: pre-wrap; background: 
rgb(239, 241, 243); padding: 0.2em 0.4em; border-radius: 
6px;">IsNotNull(c)</code>→true,<span> </sp
 an><code style="font-family: ui-monospace, SFMono-Regular, Menlo, monospace; 
font-size: 13.6px; margin: 0px; hyphens: manual; white-space: pre-wrap; 
background: rgb(239, 241, 243); padding: 0.2em 0.4em; border-radius: 
6px;">IsNull(c)</code>→false). Done in the format read builders (where the file 
schema is known) this is engine-agnostic and preserves manifest pruning for 
files that contain the column. Happy to put up the PR.</p>
   
   ### Willingness to contribute
   
   - [x] I can contribute a fix for this bug independently
   - [ ] I would be willing to contribute a fix for this bug with guidance from 
the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to