featzhang created FLINK-39368:
---------------------------------
Summary: [connector-http] Add User-Agent configuration support for
HTTP sink and lookup
Key: FLINK-39368
URL: https://issues.apache.org/jira/browse/FLINK-39368
Project: Flink
Issue Type: Improvement
Components: Connectors / HTTP
Reporter: featzhang
h2. Summary
Currently, the HTTP connector does not set a {{User-Agent}} header by default,
and there is no dedicated configuration option to control it. Users who need
to identify their Flink jobs to downstream HTTP servers must resort to
manually
configuring a raw header property, which is not discoverable and lacks a
sensible default.
This issue tracks adding first-class {{User-Agent}} configuration support to
both the HTTP sink and HTTP lookup source.
h2. Motivation
Many HTTP servers use the {{User-Agent}} header for:
- Request logging and traffic attribution
- Rate limiting policies per client
- Conditional routing or access control
A dedicated option with a meaningful default ({{flink-http-connector}}) makes
the connector self-identifying out of the box, while still allowing users to
override it per job.
h2. Proposed Changes
h3. New configuration options
|| Option key || Component || Default value || Description ||
| {{http.user.agent}} | HTTP Lookup Source | {{flink-http-connector}} |
User-Agent header value for lookup requests |
| {{http.user.agent}} | HTTP Async Sink | {{flink-http-connector}} |
User-Agent header value for sink requests |
h3. Implementation
* Add {{ConfigOption<String> SOURCE_LOOKUP_USER_AGENT}} to
{{HttpLookupConnectorOptions}} with key {{http.user.agent}} and default
{{flink-http-connector}}.
* Add {{ConfigOption<String> USER_AGENT}} to
{{HttpDynamicSinkConnectorOptions}}
with key {{http.user.agent}} and default {{flink-http-connector}}.
* Register both options in their respective factory {{optionalOptions()}}
methods
({{HttpLookupTableSourceFactory}} and {{HttpDynamicTableSinkFactory}}).
* Apply the configured value in {{RequestFactoryBase}} (lookup) and
{{JavaNetSinkHttpClient}} (sink) when building request headers.
h3. Example usage (Flink SQL)
{code:sql}
-- Lookup source with custom User-Agent
CREATE TABLE http_lookup (
id BIGINT,
name STRING
) WITH (
'connector' = 'http',
'url' = 'http://example.com/api',
'format' = 'json',
'http.user.agent' = 'my-flink-job/1.0'
);
-- Async sink with custom User-Agent
CREATE TABLE http_sink (
id BIGINT,
name STRING
) WITH (
'connector' = 'http-async-sink',
'url' = 'http://example.com/api',
'format' = 'json',
'http.user.agent' = 'my-flink-job/1.0'
);
{code}
h2. Testing
* {{JavaNetSinkHttpClientTest}}: updated existing tests to assert default
{{User-Agent}} header; added {{shouldBuildClientWithUserAgentHeader()}} and
{{shouldUseDefaultUserAgentWhenNotConfigured()}}.
* {{JavaNetHttpPollingClientTest}}: same additions for the lookup client side.
h2. Related
* PR: https://github.com/apache/flink-connector-http/pull/34
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
