Re: JVM crashing with caCertificatePath in server.xml

2024-05-17 Thread Michael Osipov
On 2024/05/16 21:37:34 Christopher Schultz wrote:
> Michael,
> 
> On 5/16/24 12:00, Michael Osipov wrote:
> > On 2024/05/16 15:55:04 Andy Arismendi wrote:
> >> Ok great! Thank you for taking the time and making the effort to look into 
> >> this Michael, much appreciated!
> > 
> > Here is a dynamically linked, patched version until there is an official 
> > release: http://home.apache.org/~michaelo/issues/tomcat/openssl-crash/
> > 
> > Please give it a try.
> 
> Since you have produced a debug build of tcnative (and other 
> components?) could you post the debug trace of the native stack?

Unfortunately I can't. While I have the files with debug symbols I am limited 
by https://github.com/mturk/cmsc?tab=readme-ov-file#warning. I don't have a 
full blown Visual Studio installed.

M

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Is there any way to add "connection: close" response header via http/1.1 connection after graceful shutdown begin by outside library

2024-05-17 Thread SG H
Hello all.
I have a question about connection close header in http/1.1 protocol.

While configure k8s environment run Spring Boot Server with Application
Load Balancer(a.k.a ALB)
I heard there are chance to client can get 503 from ALB.

ALB are communicating with a pod using HTTP/1.1 in my case.
When I rolling update pods with new image in k8s, there are must be newly
created pods and terminating pods.
And in terminating pods, Spring Boot are in graceful shutdown phase, so new
connection to these will got RST packet by tomcat embed I guess.
In additional, after graceful shutdown phase, processed requests are not
include connection response header.

Because of above

1. ALB may try to connect Spring Boot in graceful shutdown phase, got RST
and ALB response to client with 503
2. ALB may got response from Spring Boot in graceful shutdown phase which
no connection header.
So, if next request fired, ALB try to reuse current connection because
before connection is recognized as keep-alive, then got RST and ALB
response to client with 503

This may solved by make ALB retry to connect to another instance several
times until got success packet not RST.
But this behavior seems not good because ALB may got RST by another reason.

If I use HTTP/2, I got GOAWAY frame at graceful shutdown phase begun from
connection by async.
And this behavior done by tomcat.
But I can't use HTTP/2 in my environment now.

Seems Golang had a issue similar this before, so it looks like have a code
that add connection: close response header on shutdown phase.

https://cs.opensource.google/go/go/+/refs/tags/go1.22.3:src/net/http/server.go;l=1512
https://cs.opensource.google/go/go/+/refs/tags/go1.22.3:src/net/http/server.go;l=1355
https://cs.opensource.google/go/go/+/refs/tags/go1.22.3:src/net/http/server.go;l=1266
https://cs.opensource.google/go/go/+/refs/tags/go1.22.3:src/net/http/server.go;l=3385

Is there any good solution I can try?

https://github.com/spring-projects/spring-boot/issues/40802

I raised PR on spring boot github about this, and I heard this behavior
might be modified in tomcat side.

I used Embed Tomcat 10.1.17 (in spring boot 3.2.1)
Run on macbook pro m1

Best regards, SaeGon-Heo


Re: JVM crashing with caCertificatePath in server.xml

2024-05-17 Thread Christopher Schultz

Michael,

On 5/17/24 03:42, Michael Osipov wrote:

On 2024/05/16 21:37:34 Christopher Schultz wrote:

Michael,

On 5/16/24 12:00, Michael Osipov wrote:

On 2024/05/16 15:55:04 Andy Arismendi wrote:

Ok great! Thank you for taking the time and making the effort to look into this 
Michael, much appreciated!


Here is a dynamically linked, patched version until there is an official 
release: http://home.apache.org/~michaelo/issues/tomcat/openssl-crash/

Please give it a try.


Since you have produced a debug build of tcnative (and other
components?) could you post the debug trace of the native stack?


Unfortunately I can't. While I have the files with debug symbols I am limited 
by https://github.com/mturk/cmsc?tab=readme-ov-file#warning. I don't have a 
full blown Visual Studio installed.


Okay. If you did build with VS, can you get a debug build with a backtrace?

I guess you already tracked the crash to openssl_fopen. When I did a 
decompile of the official binary, I can see the code but it's very 
difficult to read:


void FUN_1800cccd0(char *param_1,char *param_2)

{
  char cVar1;
  longlong lVar2;
  int iVar3;
  DWORD DVar4;
  char *pcVar5;
  FILE *pFVar6;
  int *piVar7;
  ulonglong uVar8;
  uint uVar9;
  int cbMultiByte;
  undefined *puVar10;
  undefined *puVar11;
  uint uVar12;
  undefined8 uStackY_80;
  undefined auStackY_78 [32];
  wchar_t local_48 [8];
  ulonglong local_38;
  undefined8 uStack_30;

  uStack_30 = 0x1800ccce3;
  local_38 = DAT_18033f868 ^ (ulonglong)local_48;
  cVar1 = *param_1;
  uVar12 = 0;
  pcVar5 = param_1;
  for (uVar9 = uVar12; (cVar1 != '\0' && (uVar9 < 0x8000)); uVar9 = 
uVar9 + 1) {

pcVar5 = pcVar5 + 1;
cVar1 = *pcVar5;
  }
  cbMultiByte = (uVar9 & 0x7fff) + 1;
  uStackY_80 = 0x1800ccd50;
  iVar3 = MultiByteToWideChar(0xfde9,8,param_1,cbMultiByte,(LPWSTR)0x0,0);
  DVar4 = 8;
  if (iVar3 < 1) {
uStackY_80 = 0x1800ccd5d;
DVar4 = GetLastError();
if (DVar4 == 0x3ec) {
  uStackY_80 = 0x1800ccd84;
  iVar3 = 
MultiByteToWideChar(0xfde9,0,param_1,cbMultiByte,(LPWSTR)0x0,0);

  DVar4 = 0;
  if (0 < iVar3) goto LAB_1800ccdac;
}
uStackY_80 = 0x1800ccd91;
DVar4 = GetLastError();
puVar10 = auStackY_78;
puVar11 = auStackY_78;
if (DVar4 != 0x459) goto LAB_1800cce89;
  }
  else {
LAB_1800ccdac:
uVar8 = (longlong)iVar3 * 2 + 0xf;
if (uVar8 <= (ulonglong)((longlong)iVar3 * 2)) {
  uVar8 = 0xff0;
}
uStackY_80 = 0x1800ccdd1;
lVar2 = -(uVar8 & 0xfff0);
*(int *)(&stack0xffb0 + lVar2) = iVar3;
*(wchar_t **)(&stack0xffa8 + lVar2) = (wchar_t 
*)((longlong)local_48 + lVar2);

*(undefined8 *)(auStackY_78 + lVar2 + -8) = 0x1800ccdf7;
iVar3 = MultiByteToWideChar(0xfde9,DVar4,param_1,cbMultiByte,
*(LPWSTR *)(&stack0xffa8 + 
lVar2),
*(int *)(&stack0xffb0 + 
lVar2));

puVar11 = auStackY_78 + lVar2;
if (iVar3 == 0) goto LAB_1800cce89;
cVar1 = *param_2;
pcVar5 = param_2;
for (; (cVar1 != '\0' && (uVar12 < 0x8000)); uVar12 = uVar12 + 1) {
  pcVar5 = pcVar5 + 1;
  cVar1 = *pcVar5;
}
*(undefined4 *)(&stack0xffb0 + lVar2) = 8;
*(wchar_t **)(&stack0xffa8 + lVar2) = local_48;
*(undefined8 *)(auStackY_78 + lVar2 + -8) = 0x1800cce4d;
iVar3 = MultiByteToWideChar(0xfde9,0,param_2,(uVar12 & 0x7fff) + 1,
*(LPWSTR *)(&stack0xffa8 + 
lVar2),
*(int *)(&stack0xffb0 + 
lVar2));

puVar11 = auStackY_78 + lVar2;
if (iVar3 == 0) goto LAB_1800cce89;
*(undefined8 *)(auStackY_78 + lVar2 + -8) = 0x1800cce5d;
pFVar6 = _wfopen((wchar_t *)((longlong)local_48 + lVar2),local_48);
puVar11 = auStackY_78 + lVar2;
if (pFVar6 != (FILE *)0x0) goto LAB_1800cce89;
*(undefined8 *)(auStackY_78 + lVar2 + -8) = 0x1800cce6a;
piVar7 = _errno();
puVar10 = auStackY_78 + lVar2;
if (*piVar7 != 2) {
  *(undefined8 *)(auStackY_78 + lVar2 + -8) = 0x1800cce78;
  piVar7 = _errno();
  puVar10 = auStackY_78 + lVar2;
  puVar11 = auStackY_78 + lVar2;
  if (*piVar7 != 9) goto LAB_1800cce89;
}
  }
  *(undefined8 *)(puVar10 + -8) = 0x1800ccda7;
  fopen(param_1,param_2);
  puVar11 = puVar10;
LAB_1800cce89:
  uVar8 = local_38 ^ (ulonglong)local_48;
  *(undefined8 *)(puVar11 + -8) = 0x1800cce95;
  FUN_180263660(uVar8);
  return;
}

Thanks for helping to at least link it to this openssl source:

https://github.com/openssl/openssl/blob/45f5d51b72a262bf85c4461fbded91485ce6b9da/crypto/o_fopen.c#L38

Since libtcnative.dll is statically-linked, it doesn't even need a 
symbol table for internal calls so the openssl_fopen token is completely 
lost. Also, libtcnative contains all of TCN, APR, and OpenSSL. TCN 
doesn't make direct Win32 calls so that leaves ... all of APR and 
OpenSSL to search for this patt