Hi, Jon.
I seem to have found a possible culprit: CONFIG_HZ. Starting with 6.8, Ubuntu
kernels set this config value to 1000. We previously set this to 250 (some
context can be found here LP: #2051342) while it seems like AL2023/6.1 has this
set to 100, according to
https://docs.aws.amazon.com/linux/al2023/ug/compare-with-al2-kernel.html#CONFIG_HZ.
I have compiled the Noble 6.8 linux-aws kernel with CONFIG_HZ set to 100 here
https://launchpad.net/~magalilemes/+archive/ubuntu/test-kernels-6 and set to
250 here https://launchpad.net/~magalilemes/+archive/ubuntu/test-kernels-5, for
reference. When running the linux-aws kernel with CONFIG_HZ=100, I am able to
obtain similar results to what you got using Amazon Linux 2023 with SEV-SNP
enabled and disabled. For reference:
```
sign verify sign/s verify/s
rsa 512 bits 0.000032s 0.000002s 31368.0 520064.5
rsa 1024 bits 0.000082s 0.000005s 12215.9 197941.9
rsa 2048 bits 0.000569s 0.000017s 1756.3 59392.2
rsa 3072 bits 0.001787s 0.000036s 559.5 27753.8
rsa 4096 bits 0.004100s 0.000063s 243.9 15958.0
rsa 7680 bits 0.036875s 0.000215s 27.1 4640.8
rsa 15360 bits 0.199327s 0.000846s 5.0 1182.1
ubuntu@ip-172-31-0-140:~$ uname -a
Linux ip-172-31-0-140 6.8.0-1029-aws #31+hz100-Ubuntu SMP Sat Apr 26 01:21:16
UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@ip-172-31-0-140:~$ sudo dmesg | grep -i snp
[ 5.070724] Memory Encryption Features active: AMD SEV SEV-ES SEV-SNP
[ 5.870059] SEV: Using SNP CPUID table, 64 entries present.
[ 9.146872] SEV: SNP guest platform device initialized.
```
I confess I don't truly understand how this particular config option is
influencing the performance when SEV-SNP is enabled. For comparison, SEV-SNP
instances in other cloud environments don't seem suffer from this issue, and
their kernels also have CONFIG_HZ=1000. I was wondering if
a) this is perhaps an issue with the hypervisor.
b) you could test the AL 2023 kernel with CONFIG_HZ set to 1000, like we have
in our kernels, to have a better confirmation this is really the cause.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws in Ubuntu.
https://bugs.launchpad.net/bugs/2095017
Title:
SEV-SNP Causes Significant RSA Performance Degradation on Ubuntu 24.04
Status in linux-aws package in Ubuntu:
New
Bug description:
Description
When SEV-SNP is enabled on Ubuntu 24.04, RSA operations experience a
significant performance degradation (approximately 66% slower)
compared to when SEV-SNP is disabled. This appears to be caused by the
loss of crucial CPU instruction sets when SEV-SNP is enabled. The same
performance degradation is not observed on Amazon Linux 2023 with SEV-
SNP enabled.
Environment
• OS Version: Ubuntu 24.04.1 LTS (Noble Numbat)
• Kernel Version: 6.8.0-1018-aws
• CPU: AMD EPYC 7R13 Processor
• Instance Type: AWS EC2 (SEV-SNP capable instance)
• AMI ID: ami-036841078a4b68e14
Performance Comparisons
Performance Comparison: SEV-SNP Enabled vs Disabled
Test Environment
OS: Ubuntu 24.04.1 LTS (Noble Numbat)
Kernel: 6.8.0-1018-aws
Test Command: openssl speed -multi 2 rsa
Results
RSA Key Size SEV-SNP Enabled (sign/s) SEV-SNP Disabled (sign/s)
Performance Impact
512 bits 11,365.3 33,597.3 -66.2%
1024 bits 4,334.5 12,942.3 -66.5%
2048 bits 637.5 1,854.3 -65.6%
3072 bits 203.1 592.6 -65.7%
4096 bits 89.4 260.2 -65.6%
7680 bits 9.9 28.7 -65.5%
15360 bits 1.8 5.3 -66.0%
Analysis
• SEV-SNP significantly impacts RSA performance, with a consistent decrease
of about 65-66% across all key sizes.
==================================================================
Performance Comparison: SEV-SNP Enabled vs Disabled on Amazon Linux
2023
Test Environment
• OS: Amazon Linux 2023.6.20241212
• Kernel: 6.1.119-129.201.amzn2023.x86_64
• Test Command: openssl speed -multi 2 rsa
• AMI: ami-0b4624933067d393a
Results
RSA Key Size SEV-SNP Enabled (sign/s) SEV-SNP Disabled (sign/s)
PerformImpact
512 bits 33,067.8 35,244.7
-6.2%
1024 bits 12,338.7 13,159.0
-6.2%
2048 bits 1,755.9 1,881.4
-6.7%
3072 bits 560.0 599.6
-6.6%
4096 bits 245.6 262.4
-6.4%
7680 bits 27.2 29.0
-6.2%
15360 bits 5.0 5.4
-7.4%
Analysis
• The performance impact of SEV-SNP on Amazon Linux 2023 is significantly
lower compared to the previous Ubuntu test.
• The decrease in performance is consistently around 6-7% across all RSA key
sizes.
• Both SEV-SNP enabled and disabled instances show much higher performance
compared to the Ubuntu instances in the previous test.
Comparison with Ubuntu Results
RSA Key Size AL2023 Impact Ubuntu Impact
512 bits -6.2% -66.2%
1024 bits -6.2% -66.5%
2048 bits -6.7% -65.6%
3072 bits -6.6% -65.7%
4096 bits -6.4% -65.6%
7680 bits -6.2% -65.5%
15360 bits -7.4% -66.0%
==========================================================
Missing CPU Flags with SEV-SNP Enabled:
AVX
AVX2
VAES
VPCLMULQDQ
FMA
XSAVE
XSAVEOPT
XSAVEC
XGETBV1
RDPRU
* Update: FYI Same flags are also missing when we launch a AL2023
instance with SEV-SNP enabled
Impact on Crypto Subsystem:
• Loss of optimized implementations:
sha256-avx2
sha512-avx2
sha1-avx2
polyval-clmulni
Various VAES-based implementations
Expected Behavior
SEV-SNP should maintain access to these CPU instruction sets while providing
memory encryption and integrity protection, as demonstrated by Amazon Linux
2023's implementation of SEV-SNP, which shows only a 6-7% performance impact.
Additional Evidence
1. The performance impact is consistent across different key sizes
2. The same hardware and configuration on Amazon Linux 2023 shows only 6-7%
performance degradation with SEV-SNP enabled
3. The missing CPU flags directly correlate with the performance degradation
observed
Steps to Reproduce
1. Launch an SEV-SNP capable instance using Ubuntu 24.04 AMI
2. Enable SEV-SNP
3. Run openssl speed -multi 2 rsa
4. Compare results with SEV-SNP disabled instance
5. Verify CPU flags using cat /proc/cpuinfo
Supporting Data
• Complete CPU flags comparison
• Crypto subsystem configuration from both environments
• Full OpenSSL benchmark results
• Sosreport data available if needed
Impact
This issue significantly impacts workloads that rely on RSA operations in
SEV-SNP enabled environments, potentially affecting:
• TLS/SSL performance
• Certificate operations
• Secure communication systems
• Authentication systems
Workaround
Currently, the only known workaround is to use Amazon Linux 2023 instead of
Ubuntu 24.04 for workloads requiring both SEV-SNP and optimal RSA performance.
Additional Notes
1. This behavior appears to be specific to Ubuntu's implementation of SEV-SNP
2. The issue affects all RSA operations consistently
3. Similar performance impact is not observed with Amazon Linux 2023
4. The missing CPU instructions are available when SEV-SNP is disabled
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/2095017/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
