Jerome786 commented on issue #64393:
URL: https://github.com/apache/doris/issues/64393#issuecomment-4725198528
2026-06-16 15:55:45.361 ERROR - execute sql error: errCode = 2,
detailMessage = (10.123.8.19)[E-256]failed to read from
subods_4403.db/ods_pxf_db_pm_proj_progress_item_fd/bucket-0/data-5af33d94-c36f-413a-b1e3-1c21c20acb31-0.parquet:
AccessDenied code=FORBIDDEN, type=15,
request_id=tx000000000000016b941b8-006a310181-4eeae0a0-publicfailed to read.
cur path:
s3://paimon/subods_4403.db/ods_pxf_db_pm_proj_progress_item_fd/bucket-0/data-5af33d94-c36f-413a-b1e3-1c21c20acb31-0.parquet
2026-06-16 15:55:45.362 ERROR - sql task error
java.sql.SQLException: errCode = 2, detailMessage =
(10.123.8.19)[E-256]failed to read from
subods_4403.db/ods_pxf_db_pm_proj_progress_item_fd/bucket-0/data-5af33d94-c36f-413a-b1e3-1c21c20acb31-0.parquet:
AccessDenied code=FORBIDDEN, type=15,
request_id=tx000000000000016b941b8-006a310181-4eeae0a0-publicfailed to read.
cur path:
s3://paimon/subods_4403.db/ods_pxf_db_pm_proj_progress_item_fd/bucket-0/data-5af33d94-c36f-413a-b1e3-1c21c20acb31-0.parquet
at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
at
com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:955)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1094)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1042)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1345)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1027)
at sun.reflect.GeneratedMethodAccessor203.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.mysql.cj.jdbc.ha.MultiHostConnectionProxy$JdbcInterfaceProxy.invoke(MultiHostConnectionProxy.java:105)
at com.sun.proxy.$Proxy165.executeUpdate(Unknown Source)
at
org.apache.dolphinscheduler.plugin.task.sql.SqlTask.executeUpdate(SqlTask.java:342)
at
org.apache.dolphinscheduler.plugin.task.sql.SqlTask.executeFuncAndSql(SqlTask.java:208)
at
org.apache.dolphinscheduler.plugin.task.sql.SqlTask.handle(SqlTask.java:155)
at
org.apache.dolphinscheduler.server.worker.executor.PhysicalTaskExecutor.doTriggerTaskPlugin(PhysicalTaskExecutor.java:77)
at
org.apache.dolphinscheduler.task.executor.AbstractTaskExecutor.start(AbstractTaskExecutor.java:81)
at
org.apache.dolphinscheduler.task.executor.worker.TaskExecutorWorker.start(TaskExecutorWorker.java:65)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
2026-06-16 15:55:45.362 ERROR - PhysicalTaskExecutor{id=2914046,
name=tb_gd_bvalue_details, stat=RUNNING} execute failed
org.apache.dolphinscheduler.plugin.task.api.TaskException: Execute sql task
failed
at
org.apache.dolphinscheduler.plugin.task.sql.SqlTask.handle(SqlTask.java:166)
at
org.apache.dolphinscheduler.server.worker.executor.PhysicalTaskExecutor.doTriggerTaskPlugin(PhysicalTaskExecutor.java:77)
at
org.apache.dolphinscheduler.task.executor.AbstractTaskExecutor.start(AbstractTaskExecutor.java:81)
at
org.apache.dolphinscheduler.task.executor.worker.TaskExecutorWorker.start(TaskExecutorWorker.java:65)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
自动创建表 默认 CREATE_SCHEMA_WHEN_NOT_EXIST data_save_mode:APPEND_DATA
sink {
Paimon {
warehouse = "s3a://paimon/"
database = "subods_4403"
table = "ods_pxf_db_pm_proj_progress_item_fd"
paimon.hadoop.conf = {
fs.s3a.access-key=******
fs.s3a.secret-key=*****
fs.s3a.endpoint="******"
fs.s3a.path.style.access=true
fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
}
}
}
0:VPAIMON_SCAN_NODE(74)
table: lake.subods_4403.ods_pxf_db_pm_proj_progress_item_fd
inputSplitNum=2, totalFileSize=930459, scanRanges=2
partition=1/0
backends:
19331
s3://paimon/subods_4403.db/ods_pxf_db_pm_proj_progress_item_fd/bucket-0/data-25398118-26d1-4683-acbd-7fc971da387b-0.parquet
start: 0 length: 464981
dataFileNum=1, deleteFileNum=0, deleteSplitNum=0
136692382
s3://paimon/subods_4403.db/ods_pxf_db_pm_proj_progress_item_fd/bucket-0/data-5af33d94-c36f-413a-b1e3-1c21c20acb31-0.parquet
start: 0 length: 465478
dataFileNum=1, deleteFileNum=0, deleteSplitNum=0
cardinality=20614, numNodes=5
pushdown agg=NONE
paimonNativeReadSplits=2/2
predicatesFromPaimon: NONE
PaimonSplitStats:
SplitStat [type=NATIVE, rowCount=20614, mergedRowCount=NONE,
rawFileConvertable=true, hasDeletionVector=false]
limit: 10
tuple ids: 0
0:VPAIMON_SCAN_NODE(74)
table: lake.subods_4403.ods_pxf_db_pm_prj_manager_fd
inputSplitNum=2, totalFileSize=0, scanRanges=2
partition=1/0
backends:
19330
/data-191167ee-c336-4642-90a9-7b777c637169-0.parquet start: 0
length: 0
dataFileNum=1, deleteFileNum=0, deleteSplitNum=0
136692382
/data-87203413-3efe-44de-bb00-56a5a4bd1d7c-0.parquet start: 0
length: 0
dataFileNum=1, deleteFileNum=0, deleteSplitNum=0
cardinality=8291, numNodes=5
pushdown agg=NONE
paimonNativeReadSplits=0/2
predicatesFromPaimon: NONE
PaimonSplitStats:
SplitStat [type=JNI, rowCount=4235, mergedRowCount=NONE,
rawFileConvertable=false, hasDeletionVector=false]
SplitStat [type=JNI, rowCount=4056, mergedRowCount=NONE,
rawFileConvertable=false, hasDeletionVector=false]
limit: 10
tuple ids: 0
CREATE CATALOG `lake` PROPERTIES (
"warehouse" = "s3a://paimon/",
"type" = "paimon",
"s3.use_path_style" = "true",
"s3.secret_key" = "*XXX",
"s3.region" = "cn-hangzhou",
"s3.endpoint" = "*******",
"s3.access_key" = "*XXX",
"paimon.catalog.type" = "filesystem",
"fs.s3a.secret.key" = "*******",
"fs.s3a.path.style.access" = "true",
"fs.s3a.impl" = "org.apache.hadoop.fs.s3a.S3AFileSystem",
"fs.s3a.endpoint.region" = "cn-hangzhou",
"fs.s3a.endpoint" = "*******",
"fs.s3a.connection.ssl.enabled" = "false",
"fs.s3a.aws.credentials.provider" =
"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider",
"fs.s3a.access.key" = "********"
);
version ： doris version doris-4.0.6-rc02-1663f25c16f
> Breakwater-GitHub-Analysis-Slot: slot_f855cb78fc4c
>
> Initial analysis:
>
> I checked the Doris 4.0.5 Paimon scan path. This does not look like a
general catalog credential failure, because the same catalog works through the
JNI scanner and some tables also work with the native path.
>
> The relevant difference is the reader path:
>
> * With `force_jni_scanner=false`, Doris can convert a Paimon `DataSplit`
to raw ORC/Parquet files and read them through the native file scanner. In that
path, FE normalizes each raw file path through `LocationPath` and passes
backend S3 properties such as endpoint, region, credentials/token, and
`use_path_style` to BE. BE then reads the object through Doris' native S3
client.
> * If the split has a Paimon deletion file/deletion vector, the native
Paimon reader opens that deletion file separately through the same BE file
reader before applying row filters.
> * With the JNI scanner, Doris passes Hadoop-style properties to
`PaimonJniScanner`, and Paimon Java reads the serialized Paimon split itself.
That path does not exercise the same BE native S3 object-access path.
>
> So the issue is actionable as a likely Paimon native-reader compatibility
problem for this S3-compatible storage path, or a real object-permission
difference on the specific raw/deletion files used by
`ods_pxf_db_cm_contract_pay_info_fd`. The current issue is not enough to
distinguish those two cases.
>
> Useful next checks:
>
> 1. Please attach the BE log around the failure, including the full Doris
error line with `code=FORBIDDEN`, request id, and the object path/key that
failed. It is especially important to know whether the failed object is a data
file or a Paimon deletion-vector/deletion file.
> 2. Please run `EXPLAIN VERBOSE` for the failing table with `set
force_jni_scanner=false` and include the `paimonNativeReadSplits`,
`PaimonSplitStats`, `dataFileNum`, `deleteFileNum`, and `deleteSplitNum` lines.
The same output for one working table in the same catalog would make the
difference clear.
> 3. Please provide the catalog properties with secrets masked, but keep
endpoint, region, URI/warehouse, storage scheme (`s3://`, `s3a://`, etc.),
`use_path_style` / `s3.path-style-access`, session-token usage, and any
`paimon.*` or `fs.s3a.*` settings visible.
> 4. Please provide a minimal failing query, the exact build hash for
`doris-4.0.5`, and whether this table has deletes/updates/deletion vectors
while the working tables are append-only.
>
> Temporary workaround: keep using `set force_jni_scanner=true` for the
affected table until the native path can be confirmed and fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
