GitHub user mlevkov created a discussion: Proposal: HTTP Source Connector (Webhook Gateway)
Hi team, Following up on the HTTP sink connector ([#2925](https://github.com/apache/iggy/pull/2925)), we'd like to propose its inverse — an HTTP **source** connector that acts as a webhook gateway. It embeds an HTTP server inside the Source plugin, accepts incoming POST requests, and produces messages to Iggy topics. The connector ecosystem has 3 sources today, all poll-based (random, PostgreSQL, Elasticsearch). There's no push-based source for accepting inbound webhooks — the most common pattern for receiving real-time events from SaaS integrations (GitHub, Stripe, Slack), IoT devices, CI/CD pipelines, and inter-service communication. The design addresses several challenges unique to a server-style source: - **Push-to-pull bridge**: HTTP server enqueues to a lock-free bounded buffer (`crossbeam::ArrayQueue`), `poll()` drains it — no mutex on the hot path - **Event-sourced endpoint registry**: A dedicated Iggy config topic holds endpoint registrations/revocations, enabling hot reload and multi-instance coordination without restart - **Ephemeral endpoints**: Cryptographically random URL paths (`/e/{random_id}`) that act as bearer tokens — individually revocable, rotatable, with optional per-endpoint HMAC validation - **Structured concurrency**: Three tasks (HTTP server, config consumer, poll loop) coordinated via `CancellationToken` tree for clean shutdown within the SDK's 5-second timeout Full design document attached. We'd especially appreciate feedback on: 1. **Own `IggyClient` inside the connector** — the config consumer needs to consume an Iggy topic independently of the runtime's producer pipeline. No existing connector does this. Is this acceptable, or would you prefer a runtime-level mechanism? 2. **Server-style source** — this is architecturally novel for the connector ecosystem. Any concerns about the push-to-pull bridge pattern? 3. **Config topic convention** — we propose `platform/http-source-config`. Is there an established naming convention? 4. **Multi-stream production** — the runtime creates one producer for the last `[[streams]]` entry. Should we address this limitation separately? Looking forward to your feedback. --- [IDEA-013-iggy-http-source-github-discussion.md — Full design document attached below] GitHub link: https://github.com/apache/iggy/discussions/3039 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
