On 7/14/25 4:31 PM, Gustavo Romero wrote:
Hi folks,
Richard, thanks for v8. Pierrick, thanks for testing it. :)
On 7/14/25 14:09, Pierrick Bouvier wrote:
On 7/14/25 8:58 AM, Richard Henderson wrote:
Changes for v8:
- Re-order SCTLR2 and TCR2 so that they are independent of MEC.
- Enable the SCTLR2 and TCR2 enable bits.
- Squash 3 smaller MEC patches together.
This still fails the RME tests, because we still need TF-A rebuilt
with ENABLE_FEAT_SCTLR2 and ENABLE_FEAT_TCR2. Pierrick, since you
have just done such a build, could you re-test with this series?
I tested that on my local Realm enabled setup and I can confirm this solved the
issue and current series works.
Both flags are needed in TF-A. ENABLE_FEAT_TCR2 is needed to boot host, and
ENABLE_FEAT_SCTLR2 is needed to boot nested guest.
I'm a bit confused because the QEMU RME tests, afaics, uses OP-TEE, not TF-A.
I've built TF-A
using the scripts in [0], enabling ENABLE_FEAT_TCR2 and ENABLE_FEAT_SCTLR2, but
no way to get
it booting. I understand we can embed a OP-TEE into the TF_A via
BL32=<optee_image> when
building TF-A. Is that what you're using?
I agree it's confusing. In short, no, OP-TEE is not used anywhere for
Realms, only TF-A and RMM are used in our images. It seems that OP-TEE
is a term used generically to represent any firmware running in secure
mode, but it's a *totally* different software than TF-A + RMM. Naming
OP-TEE like this is like if Linux would have been named "OP-kernel".
The RME tests we have are based on this excellent tutorial [0], and
build is automated with 'qemu-rme-stack' [1], that simply follows those
instructions.
[0]
https://linaro.atlassian.net/wiki/spaces/QEMU/pages/30128767027/Device+Assignment+Enabled+RME+Stack+on+QEMU
[1] https://github.com/pbo-linaro/qemu-rme-stack
To add to the confusion (*get ready*), [0] uses OP-TEE build, which is
simply a build system for op-tee + other things, but in the variant it
generates, OP-TEE itself is not included. Yes, that's utterly confusing
and took me quite some time to discover it, after talking with Mathieu
himself. They removed it explicitely to make the stack more simple. In
short, for Realms, forget about OP-TEE.
If you want to see it by yourself:
https://git.codelinaro.org/linaro/dcap/op-tee-4.2.0/build/-/blob/cca/v8/qemu_v8_cca.mk?ref_type=heads#L172
...
TF_A_FLAGS ?= \
BL33=$(BL33_BIN) \
PLAT=qemu \
QEMU_USE_GIC_DRIVER=$(TFA_GIC_DRIVER) \
DEBUG=$(TF_A_DEBUG) \
LOG_LEVEL=$(TF_A_LOGLVL) \
ENABLE_RME=1 \
RMM=$(RMM_BIN)
...
TF_A_FLAGS_BL32_OPTEE = BL32=$(OPTEE_OS_HEADER_V2_BIN)
TF_A_FLAGS_BL32_OPTEE += BL32_EXTRA1=$(OPTEE_OS_PAGER_V2_BIN)
TF_A_FLAGS_BL32_OPTEE += BL32_EXTRA2=$(OPTEE_OS_PAGEABLE_V2_BIN)
TF_A_FLAGS_SPMC_AT_EL_n = $(TF_A_FLAGS_BL32_OPTEE) SPD=opteed
...
#TF_A_FLAGS += $(TF_A_FLAGS_SPMC_AT_EL_$(SPMC_AT_EL))
The last line shows that OP_TEE flags are **NOT** added to TF_A_FLAGS
build flags.
The qemu_v8_cca.mk build file was copied from qemu_v8.mk, which itself
has the OP-TEE inclusion.
---
Recently, I had to generate a custom rootfs, and I experimented
generating it directly from docker images, to avoid rebuilding the world
using Buildroot. Once it worked, I realized it was a good opportunity to
rebuild the rest of the stack too. The result is 'qemu-linux-stack' [2].
Master branch has only tf-a + uboot (no Realm support), while rme branch
[3], which supports Realm, uses tf-a + rmm + edk2 instead. I removed
u-boot as I couldn't get it to boot, and I knew that edk2 worked.
One branch is only one configuration (and it will stay this way).
[2] https://github.com/pbo-linaro/qemu-linux-stack
[3] https://github.com/pbo-linaro/qemu-linux-stack/tree/rme
Beyond the personal knowledge I got through that, I hope it can be used
for others for who it's confusing about what runs before start_kernel,
and I guess I'm not the only one who didn't know about that. In the end,
things are not too complicated, but as most of the build systems out
there (OP-TEE build, shrinkwrap, ...) try to be "generic and versatile",
it ends up being complicated. I prefer basic and straightforward script
shells to lenghty documentation and wiki pages, but it's a personal choice.
---
Now, coming to the change introduced by this series, and supporting
FEAT_SCTRL2 and FEAT_TCR2, all those images need to be updated [0], [1],
[2], [3], because essentially, TF-A itself must be patched to support
this. I was about to mention that, and mention that I can send a PR
directly to it once we have this merged on QEMU side.
The change in arm-trusted-firmware is quite simple:
+diff --git a/plat/qemu/common/common.mk b/plat/qemu/common/common.mk
+index 751511cf8..6bc108492 100644
+--- a/plat/qemu/common/common.mk
++++ b/plat/qemu/common/common.mk
+@@ -122,6 +122,10 @@ ENABLE_FEAT_FGT := 2
+ # 8.7
+ ENABLE_FEAT_HCX := 2
+
++# 8.8
++ENABLE_FEAT_TCR2 := 2
++ENABLE_FEAT_SCTLR2 := 2
++
I'll push all that tomorrow on master, and rebase rmm and
device_passthrough on top of it.
I'll add this to original RME images, it's less direct though, as it's
needed to update the tf-a fork, and then the optee build system.
Sorry for this very long email. At least, all is there.
Thanks.
Cheers,
Gustavo
[0] https://github.com/pbo-linaro/qemu-linux-stack.git
As I'm off today, I'll update that properly tomorrow when I have time, and not
rush things. I'll update RME images for sbsa and virt tests + device
passthrough test, and post associated patches.
Thanks,
Pierrick
