joggerjoel opened a new issue, #12580:
URL: https://github.com/apache/apisix/issues/12580
### Current Behavior
APISIX consistently hangs during the `init_etcd` initialization phase across
multiple versions (3.13.0, 3.12.0, 3.9.1) and configuration approaches. The
system:
1. **Starts successfully** - Docker container launches, APISIX process begins
2. **Hangs on `init_etcd`** - Gets stuck at "trying to initialize the data
of etcd"
3. **Never completes initialization** - Process remains in hanging state
indefinitely
4. **Admin API partially responds** - Can accept requests but returns
incomplete data
5. **Routes cannot be synchronized** - Created routes never become accessible
6. **Etcd remains healthy** - Etcd service is fully operational and
accessible
**Specific symptoms:**
- Container logs show: `/usr/local/openresty//luajit/bin/luajit
./apisix/cli/apisix.lua init_etcd`
- Process hangs at: `trying to initialize the data of etcd`
- No error messages or timeout errors
- No progress beyond this point
- System resources remain consumed by hanging process
### Expected Behavior
1. **Complete initialization** - Successfully finish the `init_etcd` phase
2. **Establish etcd connection** - Connect to and validate etcd service
3. **Load configuration** - Successfully load routes and plugin configuration
4. **Start nginx workers** - Complete worker process initialization
5. **Enable route management** - Allow routes to be created and synchronized
6. **Provide full Admin API** - Complete Admin API functionality
**Normal startup sequence should be:**
```
/usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua init
/usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua init_etcd
[etcd connection established]
[configuration loaded]
[nginx workers started]
[APISIX ready]
```
## **Environment Details**
- **APISIX Versions Tested**: 3.13.0-debian, 3.12.0-debian, 3.9.1-debian
- **Docker Environment**: Docker Compose with custom Dockerfile
- **Base Images**: All versions from official `apache/apisix` repository
- **Configuration Approaches Tested**: Etcd-based, YAML file-based, minimal
config
- **Plugin Configurations**: Full custom plugins, minimal plugins, built-in
only
- **Etcd Service**: Bitnami etcd:latest (healthy and accessible)
- **OS**: Linux 6.8.0-60-generic
- **Docker Version**: Latest Docker Compose
## **Reproduction Steps**
1. Use any APISIX version (3.9.1, 3.12.0, or 3.13.0)
2. Configure with either etcd or file-based config provider
3. Start APISIX container
4. Observe logs - process will hang at `init_etcd` phase
5. Wait indefinitely - no progress or completion
## **Configuration Files**
### Minimal config.yaml (tested and failed):
```yaml
deployment:
role: traditional
role_traditional:
config_provider: etcd
etcd:
host:
- "http://etcd:2379";
prefix: "/apisix"
timeout: 30
admin:
admin_listen:
port: 9180
ip: 0.0.0.0
admin_key:
- name: admin
key: [ADMIN_KEY]
role: admin
enable_admin_ui: true
allow_admin:
- 127.0.0.0/24
- 172.16.0.0/12
- 192.168.0.0/16
admin_key_required: true
admin_api_version: v1
enable_admin_cors: true
nginx_config:
worker_processes: 1
apisix:
node_listen: 80
enable_admin: true
ssl:
enable: false
plugins:
- serverless-pre-function
```
### Docker Compose (relevant section):
```yaml
services:
etcd:
image: bitnami/etcd:latest
environment:
- ALLOW_NONE_AUTHENTICATION=yes
- ETCD_ADVERTISE_CLIENT_URLS=http://etcd:2379
- ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
ports:
- "2379:2379"
- "2380:2380"
volumes:
- etcd_data:/bitnami/etcd
apisix:
build: .
ports:
- "80:80"
- "9180:9180"
volumes:
- ./conf/config.yaml:/usr/local/apisix/conf/config.yaml
- ./logs:/usr/local/apisix/logs
depends_on:
- etcd
```
## **Logs and Evidence**
### Container logs showing hanging:
```
apisix-1 | /usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua
init
apisix-1 | /usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua
init_etcd
apisix-1 | trying to initialize the data of etcd
```
### Etcd health check (working):
```bash
curl "http://localhost:2379/health";
# Result: {"health": "true", "reason": ""}
```
### Admin API response (incomplete):
```bash
curl -H "X-API-KEY: [KEY]" "http://localhost:9180/apisix/admin/routes";
# Result:
{"action":"get","node":{"key":"/apisix/routes","dir":true,"nodes":[]}}
```
## **Impact**
- **High Severity**: APISIX cannot function as an API gateway
- **Route Management**: Impossible to create or manage routes
- **Production Use**: Cannot be used in production environments
- **Development Blocking**: Prevents development and testing workflows
## **Additional Context**
This issue appears to be **environment-specific** rather than a general
APISIX bug, as:
- Multiple versions exhibit identical behavior
- Multiple configuration approaches fail
- Etcd service is healthy and accessible
- Issue persists across different plugin configurations
The problem likely lies in the Docker environment, system configuration, or
resource constraints rather than APISIX code itself.
## **What We've Tried (All Failed)**
1. **Version downgrading**: 3.13.0 → 3.12.0 → 3.9.1 (same issue)
2. **Configuration simplification**: Minimal config, no custom plugins (same
issue)
3. **Etcd configuration changes**: Timeout adjustments, prefix changes (same
issue)
4. **File-based configuration**: YAML config provider (same issue)
5. **Plugin isolation**: Built-in plugins only (same issue)
6. **Clean restarts**: Cleared logs, cache, etcd data (same issue)
### Error Logs
_No response_
### Steps to Reproduce
1. Use any APISIX version (3.9.1, 3.12.0, or 3.13.0)
2. Configure with either etcd or file-based config provider
3. Start APISIX container
4. Observe logs - process will hang at `init_etcd` phase
5. Wait indefinitely - no progress or completion
### Environment
### **System Information**
- **OS**: Ubuntu 24.04.2 LTS (Noble Numbat)
- **Kernel**: Linux 6.8.0-60-generic #63-Ubuntu SMP PREEMPT_DYNAMIC
- **Architecture**: x86_64
- **Memory**: 187GB RAM (179GB available)
- **Storage**: 3.6TB NVMe SSD (3.4TB available)
- **CPU**: 32 cores
### **Docker Environment**
- **Docker Version**: 27.5.1, build 27.5.1-0ubuntu3~24.04.2
- **Docker Compose Version**: v2.39.2
- **Docker OS**: Ubuntu 24.04.2 LTS
- **Docker Kernel**: 6.8.0-60-generic
- **Docker Memory**: 187.9GiB
- **Docker CPUs**: 32
### **APISIX Configuration**
- **APISIX Versions Tested**: 3.13.0-debian, 3.12.0-debian, 3.9.1-debian
- **Base Images**: All versions from official `apache/apisix` repository
- **Configuration Approaches Tested**: Etcd-based, YAML file-based, minimal
config
- **Plugin Configurations**: Full custom plugins, minimal plugins, built-in
only
- **Custom Dockerfile**: Yes (extends official image with additional
packages)
### **Etcd Service**
- **Image**: bitnami/etcd:latest
- **Authentication**: Disabled (ALLOW_NONE_AUTHENTICATION=yes)
- **Client URLs**: http://etcd:2379
- **Listen URLs**: http://0.0.0.0:2379
- **Ports**: 2379 (client), 2380 (peer)
- **Status**: Healthy and accessible
- **Volume Persistence**: etcd_data:/bitnami/etcd
### **Network Configuration**
- **Docker Network**: apisix_apisix-network (bridge driver)
- **Network Mode**: Bridge
- **Port Mappings**: 80:80, 443:443, 9180:9180, 2379:2379, 2380:2380
- **Service Dependencies**: APISIX depends on etcd and mariadb
### **Additional Services**
- **MariaDB**: 10.11 (external connection to 192.168.1.74:3306)
- **APISIX Dashboard**: 3.0.1-alpine
- **APISIX Manager**: Custom curl-based management container
### **Volume Mounts**
- **Config**: ./conf/config.yaml → /usr/local/apisix/conf/config.yaml
- **Logs**: ./logs → /usr/local/apisix/logs
- **Custom Plugins**: Multiple plugin files mounted
- **Etcd Data**: etcd_data volume for persistence
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
