zghong opened a new issue, #67331: URL: https://github.com/apache/doris/issues/67331
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.2 ### What's Wrong? When FE generates a checkpoint image, it serializes each table using `Gson.toJson(table)` and then writes the resulting JSON string to the image. For a table with extremely large metadata (for example, a very high number of partitions, tablets, and replicas), the serialized JSON for a single table can exceed Java's maximum array/string size limit (approximately 2 GB). This causes checkpoint creation to fail with: ```text Caused by: java.lang.OutOfMemoryError: Required array length 2147483642 + 26 is too large at java.base/jdk.internal.util.ArraysSupport.hugeLength(ArraysSupport.java:649) at java.base/jdk.internal.util.ArraysSupport.newLength(ArraysSupport.java:642) at java.base/java.lang.AbstractStringBuilder.newCapacity(AbstractStringBuilder.java:257) at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:229) at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:582) at java.base/java.lang.StringBuffer.append(StringBuffer.java:313) at java.base/java.io.StringWriter.write(StringWriter.java:106) at java.base/java.io.StringWriter.append(StringWriter.java:150) at java.base/java.io.StringWriter.append(StringWriter.java:41) at com.google.gson.stream.JsonWriter.value(JsonWriter.java:585) at com.google.gson.internal.bind.TypeAdapters$28.write(TypeAdapters.java:828) at com.google.gson.internal.bind.TypeAdapters$28.write(TypeAdapters.java:846) at com.google.gson.internal.bind.TypeAdapters$28.write(TypeAdapters.java:838) at com.google.gson.internal.bind.TypeAdapters$28.write(TypeAdapters.java:846) at com.google.gson.internal.bind.TypeAdapters$28.write(TypeAdapters.java:838) at com.google.gson.internal.bind.TypeAdapters$28.write(TypeAdapters.java:846) at com.google.gson.internal.bind.TypeAdapters$28.write(TypeAdapters.java:846) at com.google.gson.internal.bind.TypeAdapters$28.write(TypeAdapters.java:846) at com.google.gson.internal.bind.TypeAdapters$28.write(TypeAdapters.java:846) at com.google.gson.internal.bind.TypeAdapters$28.write(TypeAdapters.java:725) at com.google.gson.internal.Streams.write(Streams.java:73) at org.apache.doris.persist.gson.RuntimeTypeAdapterFactory$1.write(RuntimeTypeAdapterFactory.java:361) at com.google.gson.TypeAdapter$1.write(TypeAdapter.java:196) at com.google.gson.Gson.toJson(Gson.java:842) at com.google.gson.Gson.toJson(Gson.java:812) at com.google.gson.Gson.toJson(Gson.java:759) at com.google.gson.Gson.toJson(Gson.java:736) at org.apache.doris.catalog.Database.writeTables(Database.java:696) at org.apache.doris.catalog.Database.write(Database.java:748) at org.apache.doris.datasource.InternalCatalog.saveDb(InternalCatalog.java:3979) at org.apache.doris.catalog.Env.saveDb(Env.java:2645) ... 13 more ``` After a checkpoint failure, FE continues running and retries checkpoint generation. Since no new image is created, each retry must replay a large number of edit logs and then fails again while serializing the same oversized table. Repeated retries create substantial allocation pressure and may eventually cause the FE process to be killed externally during heap dump generation. Relevant stack/log messages: ```text start to save image to .../image.ckpt Required array length 2147483642 + 26 is too large write meta module: db size in bytes: 1140220014 Save image failed failed to do checkpoint ``` The issue is not resolved by increasing `-Xmx`, because it is caused by the JVM's single-array/string size limit rather than insufficient total heap memory. ### What You Expected? No erros. ### How to Reproduce? This issue requires a table whose serialized metadata exceeds approximately 2 GB, so it may be difficult to reproduce in a small test environment. #### General reproduction scenario 1. Deploy an Apache Doris cluster with FE configured with a large heap, for example: ```text -Xms90G -Xmx90G -XX:+UseZGC -XX:+HeapDumpOnOutOfMemoryError ``` 2. Create a database and a very large OLAP table. 3. Continuously add partitions and load data so that the table has an extremely large number of tablets and replicas. The observed production environment had approximately: ```text Replica metadata objects: ~12.19 million ``` 4. Wait for or manually trigger FE checkpoint generation. 5. Observe FE logs. Checkpoint generation fails when serializing the oversized table: ```text Required array length 2147483642 + 26 is too large Save image failed failed to do checkpoint ``` 6. Because the checkpoint image is not updated, subsequent checkpoint attempts replay a large number of journals and fail again. Eventually, allocation stalls increase and the FE process may be terminated by the container/runtime/OS. #### Example table-building pattern A reproduction table would need a very large number of partitions and buckets, for example: ```sql CREATE TABLE large_metadata_table ( k1 DATE NOT NULL, k2 BIGINT NOT NULL, v1 BIGINT SUM NOT NULL ) AGGREGATE KEY(k1, k2) PARTITION BY RANGE(k1) () DISTRIBUTED BY HASH(k2) BUCKETS <large_bucket_count> PROPERTIES ( "replication_num" = "3" ); ``` Then repeatedly add a very large number of partitions: ```sql ALTER TABLE large_metadata_table ADD PARTITION pYYYYMMDD VALUES [('YYYY-MM-DD'), ('YYYY-MM-DD')); ``` The exact threshold depends on the table schema, partition count, bucket count, replication factor, and metadata size. The key condition is that the JSON generated for **one table** exceeds the JVM’s maximum single-array size. ### Anything Else? Observed environment: - FE JVM heap: `-Xms90G -Xmx90G` - GC: ZGC - Available machine/container memory: approximately 128 GB - Replica metadata objects: approximately 12.19 million - The issue occurred during FE checkpoint image generation. - The image had fallen behind by millions of journals because previous checkpoint attempts had already failed. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
