Doris-Breakwater commented on issue #67331: URL: https://github.com/apache/doris/issues/67331#issuecomment-5475118653
Initial triage: this is a confirmed FE metadata-image scalability defect, with high confidence in the immediate cause. It is not a conventional "heap is too small" OOM. ### Verified facts - In the `4.1.2` source, `Database.writeTables()` serializes every table with `GsonUtils.GSON.toJson(table)` and then passes the complete result to `Text.writeString()`. - `Gson.toJson(Object)` uses a `StringWriter`, so the stack trace is consistent with one table's JSON growing past the JVM's maximum array-backed `String`/builder capacity. Increasing `-Xmx` cannot raise that per-array limit. - There is a second hard boundary immediately afterward: `Text.writeString()` converts the complete `String` to another UTF-8 byte array and writes its size with a signed 32-bit `int`. On load, `Text.readString()` allocates the complete byte array and returns a complete `String`. Therefore, changing only the outer Gson call to write to a `Writer`, or changing only the length field, would not be a complete fix. - `RuntimeTypeAdapterFactory` also calls `delegate.toJsonTree()` on write and `Streams.parse()` on read. A genuinely bounded-memory solution must avoid this whole-object tree materialization for an oversized table as well. - The image container itself uses `long` offsets/counts, so an image larger than 2 GB is supported at the module/file level; the limiting unit here is the individual `Text` record for one table. - This is the remaining boundary of the earlier fix in #37222: that change explicitly stopped serializing an entire database as one Gson object because Gson could not handle sizes beyond 2 GB, and split the database into per-table JSON records. This report demonstrates that a single table can now cross the same boundary. - The retry behavior is also confirmed by `Checkpoint.doCheckpoint()`: while the last image version is behind the finalized journal id, each daemon cycle loads the last image, replays through the target journal id, and attempts `saveImage()` again. The failure is caught and the checkpoint catalog is destroyed, but journal/image cleanup occurs only after a successful checkpoint. Thus the image/journal gap and replay cost can continue growing. - The reported `write meta module: db size in bytes: 1140220014` is the number of DB-module bytes written before the exception, not proof that the offending table's complete serialized size is 1.14 GB. - The current master code still has the same `toJson(table)` + `Text.writeString()` table path, so the general single-table limit is not fixed there. ### Scope that is not yet proven - The stack proves the single-table serialization boundary, but it does not identify which persisted field(s) account for most of the table JSON. The reported replica count is a strong scale indicator, not by itself proof that ordinary replica fields are the only contributor. - If this is a shared-data/Cloud deployment, #66984 (with the branch-4.1 backport in #67122 at the time of this triage) removes stale per-replica compute-group route entries that can massively inflate an image. That fix may reduce the affected metadata below the limit when that specific stale-route pattern is present, but it does **not** remove the general per-table serialization limit. - The claim that the FE is later killed externally while producing a heap dump needs separate runtime evidence. The checkpoint exception and repeated allocation pressure are verified; the eventual killer and its exact trigger are not. ### Recommended code direction 1. Introduce a versioned table-image encoding that never requires one complete table JSON `String`, one complete UTF-8 byte array, or one complete Gson `JsonElement` tree. Logical records below table level (for example table header plus partition/index/tablet/replica records) are preferable because they give bounded serialization units; alternatively, use chunked framing plus truly streaming adapters on both write and read. 2. Preserve the old reader for existing images and gate the new encoding with the image/meta version, following the compatibility pattern used by #37222. Verify checkpoint creation, immediate reload validation, follower image loading, upgrade from an old image, and re-dump in the new format. 3. Do not treat a `long` length prefix or `Gson.toJson(..., Writer)` alone as the fix: `Text`, the reader, and `RuntimeTypeAdapterFactory` must be addressed together. 4. Add tests with a deliberately small configurable/test-only chunk threshold so boundary behavior can be exercised without allocating 2 GB. Also include a synthetic large OLAP-table metadata test and assert bounded peak memory plus byte-for-byte/read-back correctness. 5. Add per-table checkpoint diagnostics (table id/name, object counts, bytes written, and elapsed time) so the dominant table is visible before an OOM rather than only the enclosing `db` module. ### Information requested - Exact FE `BuildHash` from `SHOW FRONTENDS`, plus whether this is shared-nothing or shared-data/Cloud mode. - For the largest table: table id/name (name may be anonymized), partition count, materialized-index/rollup count, tablet count, and replica count. In Cloud mode, also provide the number of persisted route-map entries/distinct compute-group keys per replica or evidence that the stale-route pattern from #66984 is absent. - The complete log from the first failed checkpoint through cleanup, including last successful image id, target finalized journal id, subsequent retry ids, and the full first stack trace. - A live heap histogram near the failure, or a histogram derived from the heap dump, showing the leading Doris metadata classes and Gson/String/array objects. - For the reported external kill: container exit reason/code, cgroup `memory.events`, kernel/runtime OOM logs, and whether the heap dump completed (including dump size and free disk space). ### Operational guidance until a fix is available - Reducing the offending table's metadata below the boundary (for example by retiring unnecessary partitions/tablets/rollups where operationally safe) is the only direct workaround demonstrated by this mechanism. Increasing `-Xmx` does not remove the structural limit. - Preserve the latest successful image and all required journals; do not manually delete or edit image/BDB metadata. Monitor the latest image id versus finalized journal id and metadata-disk growth, because recovery time/risk increases as the gap grows. - If Cloud mode is confirmed, evaluate #66984/#67122 against the actual metadata before relying on it as mitigation. The issue currently has no labels, assignee, or milestone. `kind/fix` and `area/metadata` appear appropriate. Impact is high once triggered because checkpoints cannot advance, although the trigger requires extreme per-table metadata scale. Breakwater-GitHub-Analysis-Slot: slot_f7ccb533b7fb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
