jnioche opened a new pull request, #1861: URL: https://github.com/apache/stormcrawler/pull/1861
## Summary - Adds a configurable hard timeout (`fetcher.thread.timeout`) that wraps `protocol.getProtocolOutput()` calls in both `FetcherBolt` and `SimpleFetcherBolt` - When a fetch exceeds the timeout, the protocol thread is interrupted, the URL is marked as `FETCH_ERROR`, and the fetcher thread moves on to the next item - Disabled by default (`-1`), relying on protocol-level socket timeouts only Previously, if the protocol layer hung beyond socket timeouts (due to client library bugs, misconfigured timeouts, or misbehaving proxies), fetcher threads would block indefinitely with no recovery mechanism. Fixes #996 ## Test plan - [x] New `testThreadTimeout` test in `AbstractFetcherBoltTest` (inherited by both `FetcherBoltTest` and `SimpleFetcherBoltTest`) — uses WireMock with a 10s delay and a 2s bolt timeout to verify the thread is reclaimed and a `FETCH_ERROR` is emitted - [x] All 221 core tests pass 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
