Hi Pádraig, Moving to bug-gnulib from coreutils discussion [1].
It is a bit hard to tell if OpenSSL sets errno on failure. The source code is a bit of a mess involving many custom allocaters (CRYPTO_malloc, CRYPTO_secure_malloc, OPENSSL_malloc (*)) and the system malloc. Therefore, I think we should just avoid the EVP error handling mess and just assume ENOMEM/EINVAL. These should be correct 99% of the time in my estimate. WDYT? Collin [1] https://lists.gnu.org/archive/html/coreutils/2025-09/msg00019.html (*) Much like the multiple ways to create an EVP_MD_CTX, annoyingly.
>From ae13f814ee25b1701301b6c9e1c676eec7cc55a2 Mon Sep 17 00:00:00 2001 Message-ID: <ae13f814ee25b1701301b6c9e1c676eec7cc55a2.1757189535.git.collin.fu...@gmail.com> From: Collin Funk <[email protected]> Date: Sat, 6 Sep 2025 13:11:14 -0700 Subject: [PATCH] crypto/sha3-buffer: Set errno when OpenSSL functions fail. * lib/sha3.c: Include <errno.h> (DEFINE_SHA3_INIT_CTX): Set errno to ENOMEM if function fails. (sha3_finish_ctx, sha3_process_block): Set errno to EINVAL on failure. --- ChangeLog | 5 +++++ lib/sha3.c | 15 +++++++++++++-- 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/ChangeLog b/ChangeLog index b8a4ced928..5133c402b3 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,5 +1,10 @@ 2025-09-06 Collin Funk <[email protected]> + crypto/sha3-buffer: Set errno when OpenSSL functions fail. + * lib/sha3.c: Include <errno.h> + (DEFINE_SHA3_INIT_CTX): Set errno to ENOMEM if function fails. + (sha3_finish_ctx, sha3_process_block): Set errno to EINVAL on failure. + crypto/sha3, crypto/sha3-buffer: Don't leak memory when using OpenSSL. Reported by Pádraig Brady in: <https://lists.gnu.org/archive/html/bug-gnulib/2025-09/msg00058.html>. diff --git a/lib/sha3.c b/lib/sha3.c index e7b3b9c7c6..93b53c8368 100644 --- a/lib/sha3.c +++ b/lib/sha3.c @@ -328,6 +328,9 @@ sha3_process_block (const void *buffer, size_t len, struct sha3_ctx *ctx) #else /* OpenSSL implementation. */ +/* We avoid using all of EVP error strings. Just guess a reasonable errno. */ +#include <errno.h> + #define DEFINE_SHA3_INIT_CTX(SIZE) \ bool \ sha3_##SIZE##_init_ctx (struct sha3_ctx *ctx) \ @@ -335,11 +338,15 @@ sha3_process_block (const void *buffer, size_t len, struct sha3_ctx *ctx) int result; \ ctx->evp_ctx = EVP_MD_CTX_create (); \ if (ctx->evp_ctx == NULL) \ - return false; \ + { \ + errno = ENOMEM; \ + return false; \ + } \ result = EVP_DigestInit_ex (ctx->evp_ctx, EVP_sha3_##SIZE (), \ NULL); \ if (result == 0) \ { \ + errno = ENOMEM; \ sha3_free_ctx (ctx); \ return false; \ } \ @@ -370,7 +377,10 @@ sha3_finish_ctx (struct sha3_ctx *ctx, void *resbuf) int result = EVP_DigestFinal_ex (ctx->evp_ctx, resbuf, NULL); sha3_free_ctx (ctx); if (result == 0) - return NULL; + { + errno = EINVAL; + return NULL; + } return resbuf; } @@ -397,6 +407,7 @@ sha3_process_bytes (const void *buffer, size_t len, struct sha3_ctx *ctx) int result = EVP_DigestUpdate (ctx->evp_ctx, buffer, len); if (result == 0) { + errno = EINVAL; sha3_free_ctx (ctx); return false; } -- 2.51.0
