Re: crypto/sha3-buffer: Set errno when OpenSSL functions fail.

[Pádraig Brady]

On 06/09/2025 22:37, Collin Funk wrote:

Hi Paul,

Paul Eggert <egg...@cs.ucla.edu> writes:

On 2025-09-06 13:19, Collin Funk wrote:

Moving to bug-gnulib from coreutils discussion [1].
we should just avoid the EVP error handling mess and
just assume ENOMEM/EINVAL.
[1]https://lists.gnu.org/archive/html/coreutils/2025-09/msg00019.html

Sounds good to me.

Thanks, I pushed it.

This is good.

Given the amount of operations that EVP_DigestInit_ex() does,
reducing to ENOMEM is a bit of a stretch, but I can't think of
anything better TBH. Perhaps ENOTSUP, but it's fine as is.

Note openssl has facilities for recording system errors,
and for kicks I played around with extracting them with:
  static int openssl_errno (int fallback) {
    unsigned long err;
    int last_errno = fallback;
    while ((err = ERR_get_error()))
      if (ERR_GET_LIB(err) == ERR_LIB_SYS)
        last_errno = ERR_GET_REASON(err);
    return last_errno;
  }
But that showed no system errors were recorded
under memory pressure at least, so that's not worth worrying about.

I tested your patch against coreutils and it works well:

$ (ulimit -v 7800 && src/cksum -a sha2 -l 256 /dev/null >/dev/null)
$ (ulimit -v 7800 && src/cksum -a sha3 -l 256 /dev/null >/dev/null)
cksum: /dev/null: Cannot allocate memory

thanks!
Padraig

p.s. Setting OPENSSL_DEBUG=1 in the environment doesn't help
to debug issues on Fedora at least, as I presume that's disabled
at build time for perf reasons.