Revanth14 commented on code in PR #1213: URL: https://github.com/apache/iceberg-go/pull/1213#discussion_r3471684837
########## table/scan_planning.go: ########## @@ -0,0 +1,138 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +// This file is a PROPOSED public API surface for REST server-side scan +// planning (apache/iceberg-go#1178). It defines the table-side seam — the +// option, request/result types, and the ScanPlanner interface (implemented by +// catalog/rest) — so it can be reviewed as Go rather than prose. +// WithScanPlanningMode records the requested mode on the Scan, but nothing reads +// it until scanner delegation lands, so nothing here changes existing behavior. + +package table + +import ( + "context" + + "github.com/apache/iceberg-go" + icebergio "github.com/apache/iceberg-go/io" +) + +// ScanPlanningMode is the user-facing scan option: three values +// (local/remote/auto) selecting how (*Scan).PlanFiles plans a scan. Local +// planning remains the default; remote is opt-in via WithScanPlanningMode. +// +// This is deliberately distinct from the REST table-config key +// `scan-planning-mode` (values `client`/`server`), which is a server directive +// resolved separately (OQ4): a `client` table forces local planning, a `server` +// table forces remote planning, and explicit conflicting scan options fail +// fast. There is intentionally no fourth `server` value here; the directive +// lives in the table config, not the user option. +type ScanPlanningMode string + +const ( + // ScanPlanningLocal always plans locally by reading manifests through the + // table's FileIO. This is the default and current behavior. + ScanPlanningLocal ScanPlanningMode = "local" + // ScanPlanningRemote requires a planner that advertises remote capability + // and fails loudly if remote planning is unavailable. + ScanPlanningRemote ScanPlanningMode = "remote" + // ScanPlanningAuto uses remote planning when available and allowed by the + // table config, otherwise falls back to local. + ScanPlanningAuto ScanPlanningMode = "auto" +) + +// WithScanPlanningMode sets the scan-planning mode for a scan. The default is +// ScanPlanningLocal unless the REST table config requires server planning. +func WithScanPlanningMode(mode ScanPlanningMode) ScanOption { + return func(scan *Scan) { scan.planningMode = mode } +} + +// ScanPlanningRequest is the input a Scan hands to a ScanPlanner. It carries +// the resolved scan state a planner needs without depending on catalog/rest. +// +// Open question (epic OQ4): when the table has evolved, UseSnapshotSchema must +// pin which schema binds a returned residual and the partition decode: the +// snapshot's schema (via schema-id), kept separate from each file's partition +// spec-id. Incremental scans (start/end snapshot) are deferred to a later +// phase; point-in-time SnapshotID lands first. +type ScanPlanningRequest struct { + Identifier Identifier + // Metadata is the full table metadata. This likely over-specifies the + // contract: a planner needs only schema(s), partition specs, and snapshot + // resolution; narrowing to a smaller interface is an open refinement. + Metadata Metadata + MetadataLocation string + SnapshotID *int64 + SelectedFields []string + RowFilter iceberg.BooleanExpression + MinRowsRequested *int64 + StatsFields []string + // CaseSensitive must carry the Scan's value (which defaults to true), not + // Go's false zero value, or the wire request would flip the spec default. + // Nil means use the scan default. + CaseSensitive *bool + // UseSnapshotSchema is a pointer to distinguish the spec default from an + // explicit false when the scanner-delegation phase binds it to table config. + UseSnapshotSchema *bool +} + +// PlanIO lazily loads the FileIO that should be used to read a planned scan. +// Nil means the scan should keep using the table's normal FileIO. Remote +// planners may return a PlanIO backed by plan-scoped storage credentials. +type PlanIO interface { Review Comment: Agreed, addressed by choosing the interface + scan-owned lifecycle path. `PlanIO` now has `Close() error`, and its doc names the owner: a returned `ScanPlanningResult.IO` is stored on the `Scan` that planned it. `ReadTasks` then loads from that `PlanIO` instead of the table FileIO and closes it after reading; setup errors close it immediately, and the Arrow iterator path wraps the iterator so it closes when iteration ends or stops early. The doc also states the concurrency contract explicitly: plan-scoped IO ties the `PlanFiles -> ReadTasks` sequence to the same `Scan`, and a `Scan` carrying plan-scoped IO is not safe for concurrent `PlanFiles` / `ReadTasks`. On the delivery mechanism specifically: I stored the IO on the planning `Scan` rather than swapping `ioF` on a copy. `PlanFiles` returns only `[]FileScanTask`, so the receiver is the only place `ReadTasks` can pick the IO up without a breaking signature change. `PlanFiles` already mutates its receiver (`snapshotID` / `asOfTimestamp`), so the `Scan` is already single-use across the `PlanFiles -> ReadTasks` sequence. A copy would not add safety the existing contract does not already assume, so I made that sequence/concurrency contract explicit in the doc instead of forking `Scan` state. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
