This is an automated email from the ASF dual-hosted git repository.
bneradt pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/trafficserver.git
The following commit(s) were added to refs/heads/master by this push:
new 7cf94a61f1 Add documentation for graceful shutdown (#12666)
7cf94a61f1 is described below
commit 7cf94a61f12af2e3f641d9379f787cc1e619c7de
Author: Brian Neradt <[email protected]>
AuthorDate: Mon Nov 17 11:54:37 2025 -0600
Add documentation for graceful shutdown (#12666)
This adds documentation describing how to gracefully shutdown ATS. It
documents the traffic_ctl server drain command explaining what it does
and doesn't do. It also provides some suggested best practices for
graceful shutdown.
Fixes: #4491
---
doc/admin-guide/graceful-shutdown.en.rst | 173 +++++++++++++++++++++++++
doc/admin-guide/index.en.rst | 1 +
doc/appendices/command-line/traffic_ctl.en.rst | 55 +++++++-
doc/developer-guide/jsonrpc/jsonrpc-api.en.rst | 4 +-
4 files changed, 228 insertions(+), 5 deletions(-)
diff --git a/doc/admin-guide/graceful-shutdown.en.rst
b/doc/admin-guide/graceful-shutdown.en.rst
new file mode 100644
index 0000000000..91ba1b6641
--- /dev/null
+++ b/doc/admin-guide/graceful-shutdown.en.rst
@@ -0,0 +1,173 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. include:: ../common.defs
+
+.. _admin-graceful-shutdown:
+
+Graceful Shutdown
+*****************
+
+Overview
+========
+
+|TS| provides a graceful shutdown feature that allows active client connections
+to complete naturally while signaling to clients that they should close their
+connections. This is useful when preparing a server for maintenance, removing
it
+from a load balancer rotation, or performing other operations where you want to
+reduce active connections without abruptly terminating traffic.
+
+The :ref:`traffic_ctl server drain <traffic-control-command-server-drain>`
+command enables this draining mode.
+
+.. _drain-command:
+
+The Drain Command
+=================
+
+Basic Usage
+-----------
+
+To enable drain mode:
+
+.. code-block:: bash
+
+ traffic_ctl server drain
+
+To disable drain mode and return to normal operation:
+
+.. code-block:: bash
+
+ traffic_ctl server drain --undo
+
+How It Works
+------------
+
+When you run :ref:`traffic_ctl server drain
+<traffic-control-command-server-drain>`, the following occurs:
+
+1. A global drain flag is set internally (``TSSystemState::draining = true``).
+2. The ``proxy.process.proxy.draining`` metric is set to 1.
+3. All subsequent client responses are affected as described below.
+
+Behavior by Protocol
+--------------------
+
+HTTP/1.1 Sessions
+~~~~~~~~~~~~~~~~~
+
+For HTTP/1.1 connections, |TS| automatically adds a ``Connection: close``
header
+to all responses. When clients receive this header, compliant clients will:
+
+- Complete the current request/response.
+- Close the connection after receiving the response.
+- Establish a new connection if additional requests are needed.
+
+HTTP/2 Sessions
+~~~~~~~~~~~~~~~
+
+For HTTP/2 connections, |TS| initiates the HTTP/2 graceful shutdown procedure:
+
+- Sends a GOAWAY frame to clients.
+- Clients stop creating new streams on the connection.
+- Existing streams are allowed to complete.
+- Clients close the connection when all active streams finish.
+
+Advantages Over Manual Methods
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The drain command is simpler and more complete than manually setting
+``Connection: close`` headers (for example, via the header_rewrite plugin)
+because it:
+
+- Works automatically without requiring configuration file changes.
+- Handles both HTTP/1.1 and HTTP/2 protocols properly.
+- Can be easily toggled on and off with the ``--undo`` flag.
+- Requires no server restart or configuration reload.
+
+What Drain Does NOT Do
+-----------------------
+
+It's important to understand the limitations of drain mode. Namely, the drain
+mode is something of a soft request to clients to drain current connections
+without other explicit action by |TS|. Specifically:
+
+- **Does NOT close listening sockets**: |TS| continues accepting new
connections
+ from clients.
+- **Does NOT terminate existing connections**: Connections remain open until
+ clients close them or they time out naturally.
+- **Does NOT reject new requests**: New connections are accepted and processed
+ normally, but they receive the drain signals described above.
+
+Monitoring Drain Status
+========================
+
+You can check whether |TS| is in drain mode using:
+
+.. code-block:: bash
+
+ traffic_ctl server status
+
+This will show a JSON response including:
+
+.. code-block:: json
+
+ {
+ "initialized_done": "true",
+ "is_ssl_handshaking_stopped": "false",
+ "is_draining": "true",
+ "is_event_system_shut_down": "false"
+ }
+
+You can also monitor the ``proxy.process.proxy.draining`` metric, which will
+be 1 when drain mode is active and 0 otherwise.
+
+Best Practices
+==============
+
+When using drain mode for maintenance or other operations:
+
+1. Remove the server from your load balancer rotation (or otherwise reduce
traffic to it).
+2. Run :program:`traffic_ctl server drain`.
+3. Monitor active connections (via metrics, the use of ``ss``, ``top``, etc.).
+4. Wait for connections to drain naturally or until you reach an acceptable
level.
+5. Proceed with maintenance or other operations.
+
+Additional recommendations:
+
+- **Combine with load balancer controls**: Use drain mode in conjunction with
+ your load balancer's session persistence and health check features for the
+ smoothest transitions.
+
+- **Monitor connection counts**: Watch your connection metrics to see the
effect
+ of drain mode and determine when enough connections have closed.
+
+- **Use** ``--undo`` **if needed**: If you need to return a server to service
+ quickly, use :program:`traffic_ctl server drain --undo` to disable drain
mode.
+
+- **Test in non-production first**: Test your drain procedures in a staging
+ environment to understand timing and behavior with your specific traffic
+ patterns.
+
+
+See Also
+========
+
+- :ref:`traffic_ctl server drain <traffic-control-command-server-drain>`
+- :ref:`traffic_ctl server status <traffic-control-command-server-status>`
+- :ts:cv:`proxy.config.stop.shutdown_timeout`
+
diff --git a/doc/admin-guide/index.en.rst b/doc/admin-guide/index.en.rst
index ac0cc95969..03632c0d51 100644
--- a/doc/admin-guide/index.en.rst
+++ b/doc/admin-guide/index.en.rst
@@ -31,6 +31,7 @@ Table of Contents:
installation/index.en
configuration/index.en
interaction/index.en
+ graceful-shutdown.en
security/index.en
storage/index.en
plugins/index.en
diff --git a/doc/appendices/command-line/traffic_ctl.en.rst
b/doc/appendices/command-line/traffic_ctl.en.rst
index 6a5b75c5bf..1b5444ef67 100644
--- a/doc/appendices/command-line/traffic_ctl.en.rst
+++ b/doc/appendices/command-line/traffic_ctl.en.rst
@@ -378,14 +378,63 @@ traffic_ctl server
.. program:: traffic_ctl server
+.. _traffic-control-command-server-drain:
+
.. program:: traffic_ctl server
.. option:: drain
- :ref:`admin_server_start_drain`
+ Enable graceful connection draining mode. When enabled, |TS| signals clients
+ to close their connections gracefully by:
+
+ - Adding ``Connection: close`` headers to HTTP/1.1 responses
+ - Sending GOAWAY frames to HTTP/2 clients for graceful shutdown
+
+ This allows active connections to complete naturally while encouraging
clients
+ to establish new connections. |TS| continues accepting new connections while
+ in drain mode.
+
+ .. note::
+
+ Drain mode does NOT close listening sockets or terminate existing
connections.
+ It only signals to clients that they should close their connections.
+
+ .. option:: --undo, -U
+
+ Disable drain mode and return to normal operation.
+
+ .. option:: --no-new-connection, -N
+
+ This option is accepted but not yet implemented. It has no effect in the
+ current version.
+
+ Examples:
+
+ Enable drain mode:
+
+ .. code-block:: bash
+
+ $ traffic_ctl server drain
+
+ Disable drain mode:
+
+ .. code-block:: bash
+
+ $ traffic_ctl server drain --undo
+
+ Check drain status:
+
+ .. code-block:: bash
+
+ $ traffic_ctl server status | jq '.is_draining'
+ "false"
+
+ For detailed information about drain behavior and best practices, see
+ :ref:`admin-graceful-shutdown`.
- :ref:`admin_server_stop_drain`
+ See also:
- Drop the number of active client connections.
+ - :ref:`admin_server_start_drain` (JSONRPC API)
+ - :ref:`admin_server_stop_drain` (JSONRPC API)
.. _traffic-control-command-server-status:
diff --git a/doc/developer-guide/jsonrpc/jsonrpc-api.en.rst
b/doc/developer-guide/jsonrpc/jsonrpc-api.en.rst
index 1a966da87b..e232a09e39 100644
--- a/doc/developer-guide/jsonrpc/jsonrpc-api.en.rst
+++ b/doc/developer-guide/jsonrpc/jsonrpc-api.en.rst
@@ -1120,7 +1120,7 @@ admin_server_stop_drain
Description
~~~~~~~~~~~
-Stop the drain requests process. Recover server from the drain mode
+Stop the process of draining connections. See :ref:`admin-graceful-shutdown`
for more information.
Parameters
~~~~~~~~~~
@@ -1157,7 +1157,7 @@ admin_server_start_drain
Description
~~~~~~~~~~~
-Drain TS requests.
+Drain TS connections. See :ref:`admin-graceful-shutdown` for more information.
Parameters
~~~~~~~~~~
