relayd: incomplete response from a TLS-accelerated apache

[Maxim Bourmistrov]

Hey,
I investigate a problem were TLS-asselerated machine response is incomplete.
I was able to reproduce this on OpenBSD 5.9, 6.0 and 6.1. Test on 5.8 is about 
to be.

Following env I have:

relay1: relayd machine
web1: apache 2.2.31 serving the request
client1: requester

relay1 is configured following way (relevant lines):

http protocol http_relay {
        tcp { nodelay, sack, socket buffer 65536, backlog 1024 }
        match header append "X-Forwarded-For" value "$REMOTE_ADDR"
        match header set "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"
        match header set "Keep-Alive" value "$TIMEOUT"
        match request header remove "Proxy"
}

http protocol tls_accel {
        tcp { nodelay, sack, socket buffer 65536, backlog 1024 }
        match header append "X-Forwarded-For" value "$REMOTE_ADDR"
        match header set "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"
        match header set "X-Forwarded-Proto" value "https"
        match header set "X-Forwarded-Port" value "443"
        match header set "Keep-Alive" value "$TIMEOUT"
        match request header remove "Proxy"

        tls { tlsv1, \
                ciphers "AES:!AES256:!aNULL" \
          }
}

table <webpool> { 172.16.1.111 }

relay int_test_tls {
        listen on 172.16.1.99 port 443 tls
        protocol tls_accel
        forward to <webpool> port 80 mode roundrobin check http "/" code 200
}

relay int_test_http {
        listen on 172.16.1.99 port 80
        protocol http_relay
        forward to <webpool> port 80 mode roundrobin check http "/" code 200
}

web1 is a std Apache 2.2.31 with enabled deflate for the following

AddOutputFilterByType DEFLATE application/json
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE application/x-javascript
AddOutputFilterByType DEFLATE application/javascript

and serving a JS file.

client1 is running PHP code from CLI to reproduce this problem.


Following is observed:

1. Client1 requests web1 directly on port 80 and gets full response

shell$ php client3.php
Expected length: 547204
Received length: 547204

[Response Headers]
HTTP/1.1 200 OK
Date: Mon, 08 May 2017 11:08:27 GMT
Server: Apache/2.2.31 (Unix) mod_ssl/2.2.31 OpenSSL/1.0.1e-fips
Last-Modified: Mon, 08 May 2017 07:22:43 GMT
ETag: "60319-85984-54efe1ae42be3"
Accept-Ranges: bytes
Content-Length: 547204
Vary: Accept-Encoding
Connection: close
Content-Type: application/javascript 

2. Client1 requests web1 directly on port 80 WITH GZIP enabled and gets full 
response back
I see gzipped stream on the screen and then it gets decoded to a complete file. 
File I get is not cut.

Expected length: Content-Length not recieved
Received length: 165454

[Response Headers]
HTTP/1.1 200 OK
Date: Mon, 08 May 2017 11:10:18 GMT
Server: Apache/2.2.31 (Unix) mod_ssl/2.2.31 OpenSSL/1.0.1e-fips
Last-Modified: Mon, 08 May 2017 07:22:43 GMT
ETag: "60319-85984-54efe1ae42be3"
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Connection: close
Content-Type: application/javascript

3. and 4. Clien1 requests relay1 on port 80 (with and without GZIP) and gets 
complete response

5. Client1 requests relay1 on port 443 without GZIP - response is incomplete

Expected length: 547204
Received length: 396424

[Response Headers]
HTTP/1.1 200 OK
Accept-Ranges: bytes
Connection: close
Content-Length: 547204
Content-Type: application/javascript
Date: Mon, 08 May 2017 11:14:59 GMT
ETag: "60319-85984-54efe1ae42be3"
Last-Modified: Mon, 08 May 2017 07:22:43 GMT
Server: Apache/2.2.31 (Unix) mod_ssl/2.2.31 OpenSSL/1.0.1e-fips
Vary: Accept-Encoding

6. Client1 requests relay1 on port 443 with GZIP - response is complete.


So non-gzipped response from behind the relay1 is incomplete while doing TLS 
termination.
Files server.js and client.php can be provided upon request.

Any ideas?

Br