MisterRaindrop opened a new pull request, #2071:
URL: https://github.com/apache/cloudberry/pull/2071
### What does this PR do?
The contract between `datalake_fdw` and `datalake_agent`: the proto files
both sides will build from, before either has code that uses them. Nothing
here is compiled into the extension yet, and the PostgreSQL build is
unchanged.
Closes #2010. Part of #2008 (B0).
- `contrib/datalake_agent/proto/`: `common.proto`, `iceberg_catalog.proto`
(`IcebergCatalogService`), `catalog_mgmt.proto`
(`CatalogManagementService`), and a README.
- `contrib/datalake_fdw/src/meta/fragment.proto`: fragments, written-file
reports and pushed-down predicates -- what the C side reads and writes
directly, so it lives with the C side, in the same package.
- `.github/workflows/datalake-proto.yml`: runs the C++ and the Java gRPC
generators on every change to these files.
Decisions worth a look:
- **One RPC per `IcebergMetaEngine` entry**, and the README maps them in both
directions. `alter_table` and `truncate_table` have none yet: an engine
without them leaves the capability bits unset and the central dispatch
answers `DL_ERR_NOT_SUPPORTED` without calling it. Each gets its RPC in the
change that implements it.
- **What a statement acts on is fixed in the request, not resolved on
arrival.** `LoadTable` and `CreateTable` return the exact metadata location
and the table UUID; `GetFragment` plans against that location; every write
and `DropTable` must name the UUID, so a table dropped and recreated under
the same name is never the one written to or purged; UPDATE, DELETE and
rewrite commits name the snapshot they were planned against, so conflict
detection starts there and not at whatever is current at commit time.
- **Commit state has one authority**, the three-way `CommitOutcome`. A commit
whose answer was lost is neither applied nor not; a boolean would have to
lie one way or the other.
- **`CommitFileGroups` streams both ways**, one group per message, so a
rewrite of any size stays under gRPC's message limit.
- **Versions are checked by both sides.** The client sends
`x-cloudberry-client-version`; every response -- streamed batches included
-- carries the server's version and the oldest client it accepts. The
server refuses a client that is too old, and the client refuses a server
that is too old for it: a field an older server does not know arrives as
zero, and for several fields here zero means "none".
- **Errors** end in a `google.rpc.Status` with an `ErrorDetail` whose
`business_code` a client branches on; no stack trace leaves the server.
- **Temporal literals count from the Unix epoch**, as Iceberg's do. The field
comments give the offset from PostgreSQL's 2000-01-01, because a predicate
off by 10957 days prunes the wrong files and no recheck can bring them
back.
- **Nothing generated is committed.** Each side runs `protoc` at build time
with both directories as include roots.
### Type of Change
- [ ] Bug fix (non-breaking change)
- [x] New feature (non-breaking change)
- [ ] Breaking change (fix or feature with breaking changes)
- [ ] Documentation update
### Test Plan
- [x] The workflow's steps, extracted from the YAML and run unchanged in
`ubuntu:24.04`: C++ with the distribution's `protoc` 3.21.12 and
`grpc_cpp_plugin`, Java with `protoc` 3.25.5 and `protoc-gen-grpc-java`
1.81.0 -- the versions the agent's Maven build will pin -- all with
`--fatal_warnings`. Clean on both.
- [x] The checks fail when they should: an unused import fails the run under
`--fatal_warnings`, and a file without `java_package` fails the
package assertion (its classes land in `cloudberry/datalake/v1/`).
- [x] Apache RAT: every new file is AL, 0 unknown.
- [ ] Unit tests added/updated (nothing to run yet; B1 and B2 bring them)
- [ ] Passed `make installcheck` (no C change)
### Impact
**Dependencies:** none for the PostgreSQL build. The workflow downloads two
generators from Maven Central, pinned by version and checked against SHA-256
digests that match Maven Central's published SHA-1.
**User-facing changes:** none.
### Checklist
- [x] Followed [contribution
guide](https://cloudberry.apache.org/contribute/code)
- [x] Added/updated documentation
- [x] Reviewed code for security implications
- [ ] This PR contains AI-assisted code generation
- [ ] Requested review from [cloudberry
committers](https://github.com/orgs/apache/teams/cloudberry-committers)
### Additional Context
Known limit, in the README: `AppendResponse` and `UpdateResponse` carry
`outcome` and `committed_metadata_location`, which only a commit can
truthfully report. `Append` and `Update` stage; the table changes at
`CommitAppend` and `CommitUpdate`, and commit state is read from those.
B1 (#2011, the Java service framework) builds on this and follows as its own
PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
