morningman opened a new issue, #70122:
URL: https://github.com/apache/doris/issues/70122
### Background
Some deployments must audit every query: compliance rules require a record
of each statement in `fe.audit.log` and in `__internal_schema.audit_log`, and
require that the database's administrators cannot quietly switch that off.
Doris treats ADMIN as trusted, and today query audit is something ADMIN, and
partly any user, can turn off or weaken at runtime.
### Current behavior
- `SET GLOBAL enable_audit_plugin = false` stops the audit table from
receiving events.
- `ADMIN SET FRONTEND CONFIG ("audit_event_log_queue_size" = "0")`, or
`/api/_set_config` with `persist=true`, drops every event for both
`fe.audit.log` and the table.
- `audit_plugin_max_sql_length = 0` reduces each recorded statement to a
truncation marker, and `audit_plugin_max_insert_stmt_length = 0` drops even the
table name of `INSERT ... VALUES`. A negative `audit_plugin_max_sql_length`
drops every event except `INSERT ... VALUES`, because the truncation throws.
- The batching variables (`audit_plugin_max_batch_bytes`,
`audit_plugin_max_batch_interval_sec`), `audit_plugin_load_timeout` and the
dispatch timeouts (`query_audit_log_timeout_ms`,
`be_report_query_statistics_timeout_ms`,
`workload_runtime_status_thread_interval_ms`) can hold events in memory or make
loads drop them.
- `enable_prepared_stmt_audit_log` is a session variable, off by default, so
any user's server-side prepared executions (`COM_STMT_EXECUTE`) leave no record.
- In fe.conf, `skip_audit_user_list` drops the events of the users in it,
and an `audit_log_modules` without `query` stops `fe.audit.log` from recording
queries.
### Expected behavior
An operator can enable a mode in which no database account, `root` included,
can turn query audit off or hollow it out through SQL, the FE HTTP config
interface or session settings; only someone who can edit the FE's configuration
files and restart it can.
### Proposed design
A new FE config `query_audit_always_on`, default false, not runtime-mutable.
While it is on, every statement the FE receives is audited (failed statements
and server-side `COM_STMT_EXECUTE` included) to both sinks, and no protected
parameter may be weaker than its factory default: the switches stay on, a limit
whose smaller values drop events has its default as the minimum, a wait whose
larger values hold events back has it as the maximum, and the batching knobs
are fixed.
- `QueryAuditPolicy` (fe-common) defines the rules once. Readers use the
effective value of each variable, since old values also come from the image,
the edit log, forwarded sessions and existing sessions.
- `SET`, `SET_VAR`, `UNSET` and runtime config changes reject a weaker value
with an error, so the attempt fails and is audited itself; `UNSET VARIABLE ALL`
keeps such a variable instead of failing.
- At startup, conflicting values from fe.conf and fe_custom.conf are
replaced in memory with a WARN (`skip_audit_user_list` is ignored, `query` is
added to `audit_log_modules`, out-of-bound configs get their bound) instead of
refusing to start.
- A master that takes over rewrites persisted non-compliant global values
through the global variable edit log, so `SHOW VARIABLES` agrees on every FE.
- A prepared execution forwarded to the master is audited only by the FE
that received it, as `COM_QUERY` is.
Alternatives considered: making only `enable_audit_plugin` read-only, which
leaves the other knobs and persisted values in place; enforcing on read only,
under which a rejected change appears to succeed and leaves no record; and
smaller floors such as 4096 bytes for the SQL length, for which there is no
objective threshold.
### Impact
Users who need audit that administrators cannot turn off, in both
shared-nothing and cloud mode. FE only. With the switch off nothing changes,
except that a negative `audit_plugin_max_sql_length` counts as 0 and is
rejected by `SET`.
### Reproduction or evidence
```sql
SET GLOBAL enable_audit_plugin = false;
ADMIN SET FRONTEND CONFIG ("audit_event_log_queue_size" = "0");
SET GLOBAL audit_plugin_max_sql_length = -1; -- drops every event except
INSERT ... VALUES
```
### Scope and non-goals
In scope: the parameters above, and audit of failed prepared executions. Out
of scope: code execution inside the FE (`INSTALL PLUGIN`, JDBC `driver_url`),
tampering with records already written (for example `TRUNCATE` on the audit
table), reliable delivery (a full queue, a failed load or an FE crash can still
lose events), `enable_bdbje_debug_mode`, load and slow-query audit, and
external audit plugins.
### Acceptance criteria
- With the switch on, each change above is rejected with an error naming
`query_audit_always_on`, and the attempt appears in the audit table as a failed
statement.
- Every `COM_STMT_EXECUTE` is audited once, also when forwarded to the
master.
- Conflicting values in the config files are overridden at startup with a
WARN.
- `ADMIN SET FRONTEND CONFIG ("query_audit_always_on" = "false")` fails with
`is not mutable`.
- With the switch off, behavior is unchanged except the negative-length fix.
### Risks and compatibility
No metadata, image, Thrift or Protobuf change; the master's rewrite uses the
existing global variable edit log. Each FE enforces its own setting, so every
FE must be configured. With the switch on, clients that set
`enable_prepared_stmt_audit_log = false` get an error, prepared executions
raise audit volume, and clusters that had shortened the recorded statements
record up to 2 MB per statement.
### Related
- The threat model (`threat-model.md`) gains a property for this mode (ยง4.8
property 13, M20), pending maintainer agreement.
- #38419 and #38865 added `enable_prepared_stmt_audit_log`, #38310 added
`skip_audit_user_list`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
