PeterZaitsev opened a new issue, #69303: URL: https://github.com/apache/doris/issues/69303
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version Doris 4.1.5 (`doris-4.1.5-release-917b20be838`), single FE + single BE, bundled `cdc-client.jar`. Source: Percona Server for MySQL 9.7.2 (the same applies to any MySQL 8.0.23+ table with an INVISIBLE column), JDBC driver mysql-connector-j 8.4.0. ### What's Wrong? A MySQL streaming job (`CREATE JOB ... ON STREAMING FROM MYSQL ... TO DATABASE`, auto table creation, `"offset" = "initial"`) pauses during the full snapshot of any table whose primary key is an **INVISIBLE** column. The Doris table itself is created correctly, with the invisible column as its UNIQUE KEY, but every snapshot split fails and the job goes to `PAUSED`: ``` ErrorMsg: {"code":"INTERNAL_ERR","msg":"ConnectException: Data row is smaller than a column index, internal schema representation is probably out of sync with real database schema"} ``` `be/log/cdc-client.log`: ``` ERROR io.debezium.relational.TableSchemaBuilder - Error requesting a row value, row: 8, requested index: 8 at position 0 ERROR org.apache.flink.cdc.connectors.mysql.debezium.reader.SnapshotSplitReader - Execute snapshot read task for mysql split MySqlSnapshotSplit{tableId=db.history, splitId='db.history:0', splitKeyType=[`id` BIGINT NOT NULL], ... io.debezium.DebeziumException: org.apache.kafka.connect.errors.ConnectException: Data row is smaller than a column index, internal schema representation is probably out of sync with real database schema at org.apache.flink.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask.execute(MySqlSnapshotSplitReadTask.java:134) ... Caused by: org.apache.kafka.connect.errors.ConnectException: Data row is smaller than a column index, internal schema representation is probably out of sync with real database schema at io.debezium.relational.TableSchemaBuilder.validateIncomingRowToInternalMetadata(TableSchemaBuilder.java:254) at io.debezium.relational.TableSchemaBuilder.lambda$createKeyGenerator$3(TableSchemaBuilder.java:216) at io.debezium.relational.TableSchema.keyFromColumnData(TableSchema.java:130) at io.debezium.relational.RelationalChangeRecordEmitter.emitReadRecord(RelationalChangeRecordEmitter.java:110) at io.debezium.pipeline.EventDispatcher.dispatchSnapshotEvent(EventDispatcher.java:155) at org.apache.flink.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask.createDataEventsForTable(MySqlSnapshotSplitReadTask.java:292) ``` The table schema the reader builds has 9 columns (the 8 visible ones plus the invisible key), but the snapshot rows carry 8 values. `SELECT *` does not return INVISIBLE columns in MySQL, so the snapshot query appears to read without the key column, and the key generator then asks for index 8 of an 8-value row. The whole job stays paused, not only that table. The same applies to tables with a **generated invisible primary key** (`sql_generate_invisible_primary_key = ON`, MySQL 8.0.30+), whose key `my_row_id` is invisible by definition. Those tables have a primary key, so they pass the "primary key tables only" rule of the streaming job, and then fail in the snapshot. ### What You Expected? The snapshot reads the invisible column too (an explicit column list rather than `SELECT *`), and the table syncs like any other primary key table. If invisible columns are not supported, the job should refuse the table when it is created (or skip it with a clear message) rather than create it and then pause the whole job in the snapshot. ### How to Reproduce? On the MySQL source (binlog ROW / FULL, GTID on): ```sql CREATE DATABASE cdc_inv; CREATE TABLE cdc_inv.t ( a INT, b VARCHAR(20), id BIGINT NOT NULL AUTO_INCREMENT INVISIBLE, PRIMARY KEY (id) ); INSERT INTO cdc_inv.t (a, b) VALUES (1, 'x'), (2, 'y'), (3, 'z'); ``` (or, equivalently, `SET sql_generate_invisible_primary_key = ON; CREATE TABLE cdc_inv.t (a INT, b VARCHAR(20));`) On Doris 4.1.5: ```sql CREATE DATABASE cdc_inv; USE cdc_inv; CREATE JOB inv_cdc ON STREAMING FROM MYSQL ( "jdbc_url" = "jdbc:mysql://<mysql-host>:3306", "driver_url" = "mysql-connector-j-8.4.0.jar", "driver_class" = "com.mysql.cj.jdbc.Driver", "user" = "<user>", "password" = "<password>", "database" = "cdc_inv", "offset" = "initial" ) TO DATABASE cdc_inv ("table.create.properties.replication_num" = "1"); SELECT Name, Status, ErrorMsg FROM jobs("type"="insert") WHERE ExecuteType = "STREAMING"; -- Status: PAUSED, ErrorMsg: ConnectException: Data row is smaller than a column index ... ``` ### Anything Else? - Every other table in the same database (all with ordinary visible primary keys, about 114 M rows) synced fine; the job ran normally after the table with the invisible key was left out with `include_tables`. That is the workaround we use; the other one is making the column visible on the source (`ALTER TABLE ... ALTER COLUMN id SET VISIBLE`), which may not be acceptable when an application relies on `INSERT ... VALUES` without a column list. - The error comes from the Flink CDC MySQL snapshot reader / Debezium bundled in `cdc-client.jar`, so the fix may belong upstream; reporting it here because the Doris streaming job is where users meet it and where the table is accepted. - Only the snapshot phase was exercised; we did not test whether binlog changes for such a table are applied correctly (binlog rows with a FULL row image do include invisible columns). ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
