This is tremendously helpful. Thanks a lot. My work is focusing on kernel, base on arm architecture, I want have whole view of the running kernel.
With qemu & qemu monitor & gdb & crash tools, I can stop kernel anywhere, read phy/virt address, registers even system register(with last qemu release). Thanks for sharing your experience, It does greate help. Hope to hear that you successfully run last AOSP on upstream QEMU. 在 2020年7月4日星期六 UTC+8上午12:35:32，Julien Robin写道： > > Hi 郁进, > > Recently I focused on making the untouched Android code base working with > standard/upstream QEMU (which is a little bit different from the AOSP's > emulator, which is a QEMU derivative), focusing on x86_64 builds and > "android-x86" project build configurations; the work is still in progress. > But from what I know : > > When building aosp_XXXX-eng project, the "emulator" command line is made > available so that just typing "emulator" into your terminal is (normally) > enough to get it starting with the correct parameters set. Most of the time > I use "emulator -verbose -show-kernel -no-snapshot" when running the AOSP's > emulator. So that you have all the default parameters shown into your > terminal, then kernel debug information, then Android's "init" output into > your terminal. The kernel used to run those builds is called > "kernel-ranchu" : it's a customised and prebuilt version of the 4.14 > kernel, intended for use with the aosp's emulator virtual hardware and > drivers (called goldfish in its original version, then ranchu). > > If you build aosp then close the terminal window, and you want the > "emulator" command to be available again, you should just type "source > build/envsetup.sh && lunch" commands again (no need to rebuild everything). > > But for arm64 builds its horribly slow, like several minutes for booting > (upstream qemu is making the following option "-accel tcg,thread=multi" > available so that the performance using an AMD 3700X is almost similar to > using KVM on a native arm64 Raspberry Pi 4. But unfortunately this accel > option does not seem to be used by the "emulator" command, and seems not to > work on AOSP's emulator, at least last time I tried). Be aware that > emulator is a derivative of QEMU, recognizing some AOSP specific commands, > and not recognizing some orginal QEMU commands. > > *About having the kernel finding and starting "init" :* > > Lot of things can be confusing about having your emulator or qemu finding > and running "init" process. I will give you everything I discovered so that > hopefully you will be able to understand and debug anything that happens to > you > > On really old versions of Android, "init" executable was only available > into the ramdisk, and "system" partition was just containing the /system > subfolder (this is the opposite of "system as root"). In this case, Linux > kernel absolutely needed to load the ramdisk, because init was into this > ramdisk. I guess fstab file was into the ramdisk too, in order to find and > mount partitions once init started. > > Some more recent version of android using "system as root" (8 or 9 maybe) > gave some ramdisk file as parameter to the emulator, despite the fact this > ramdisk wasn't actually used anymore by the emulator : the content of the > ramdisk was already into system.img. And the kernel was already able to > access the system.img partition without the need of anything into the > ramdisk - so the ramdisk wasn't used anymore. "init" file was at the root > of the system.img partition image. In this case, you should ensure your > machine and kernel are able to find your ext4 root (when in trouble, look > into the kernel messages to find an ext4 file system at /dev/sda, or > /dev/vda, or even sda1, vda1, etc, so that you can be sure about the kernel > ability to see the partition you need). > > But Android 10 introduced a new partition/image format that can be > dynamically expanded and physically fragmented (so that devices sold with a > too small system partition for being updated, can be updated anyway). This > is cool feature, but a little bit more complex to handle. When using this > kind of system images, android requires a brand new specific ramdisk with a > specific "init" executable in order to read and mount this new system > partition format created by android. I believe emulator is using it with > Android 10 (to be confirmed). But in order to focus on my work, I'm still > using the previous way : an ext4 system.img with everything into it ;) so > that qemu boots it directly. > > > For example, on my x86_64 custom builds using "system as root", and having > the vendor files included into system.img, as a standard ext4 image, I run > the following command to start it using QEMU : > > kvm -m 4096 -smp 4 -kernel bzImage -append "root=/dev/sda rootfstype=ext4 > ro init=/init selinux=1 checkreqprot=1 androidboot.selinux=permissive > console=ttyS0 androidboot.hardware=luoss loglevel=8" -serial mon:stdio > -drive format=raw,file=system.img -vga std > -m 4096 ask for 4GB RAM > bzImage is a custom 5.4 kernel > root=/dev/sda rootfstype=ext4 ro init=/init is telling my kernel to mount > what it sees as "/dev/sda" and to run the "init" executable which is > located into it > selinux=1 checkreqprot=1 androidboot.selinux=permissive are selinux > parameters : when you want to disable selinux, you can't (init absolutely > wants your kernel to have selinux running) but you set it to > androidboot.selinux=permissive, so that no operation is actually blocked. > console=ttyS0 ask the kernel to print its messages into the serial port > androidboot.hardware=HW_NAME is used to tell init process to load > init.HW_NAME.rc configuration file (that should be into /init.HW_NAME.rc > or /vendor/etc/init/hw/init.HW_NAME.rc). For aosp emulator builds, it is > /vendor/etc/init/hw/init.ranchu.rc > loglevel=8 ask for the kernel, then init, to be as verbose as possible > > -serial mon:stdio is a qemu parameter I use to redirect the virtual serial > port (on which the kernel talks) to the terminal in which qemu is running. > So that I can use this terminal as a read/write terminal with the virtual > machine. > > > The system image i'm using to do so is an ext4 partition image, on which i > placed everything needed, and an init.luoss.rc and fstab.luoss files - > mainly just to mount a /data tmpfs instead of using a real userdata.img > partiton - yes, I'm cheating ;) normally system.img is not enough, and in > most case, /vendor subfolder is into a vendor.img partition. > > I guess that when I'll be trying to run arm64 builds on qemu, some > parameters will need to be changed. > > I hope that all this information will help you to successfully dig, > understand and solve your issues! > > Best regards > > Julien > > Le lundi 29 juin 2020 06:41:01 UTC+2, 郁进 a écrit : >> >> Hi, >> >> I found another tickets you submitted to run arm64 android via qemu. >> >> I tried to lunch aosp_arm64-eng project, run with android emulator, with >> qemu args: >> >> emulator -kernel >> ../Android_kernel/out/android-mainline/common/arch/arm64/boot/Image -system >> xxx ...... *-qemu **--monitor tcp:127.0.0.1:2222 >> <http://127.0.0.1:2222>,server,nowait -gdb tcp::4444 -S* >> >> But the Android kernel panic occurred since no init found, and I try to >> dump guest memory via qemu monitor, can't parsed by crash tools. >> >> Have you build and run arm64 android via qemu successfully? How ? >> >> Looking forward to your reply, many thanks. >> >> >> 在 2020年3月3日星期二 UTC+8下午11:18:37，Julien ROBIN写道： >>> >>> Many thanks for your answers >>> >>> In fact what I'm trying to do is using QEMU files into AOSP source code >>> to get Android (with GUI) built and running on a standard/upstream QEMU >>> version. >>> >>> This would give the ability of running Android and its future releases >>> without real "porting", on any device running a Linux kernel (be it >>> upstream or not) and having KVM working on it. Which would be extremely >>> interesting. >>> >>> From what you write, it seems qemu-trusty-arm64 is not really for me. So >>> what I'm trying to do is probably more about using device/generic/qemu/ >>> qemu_arm64.mk (and others archs) files included into AOSP. So I'll >>> continue into this other subject : >>> https://groups.google.com/forum/?hl=bn#!topic/android-building/kbiEg5OxBGU >>> >>> >>> While talking with you, I noticed that AVD Emulator is different from >>> upstream QEMU, probably about goldfish and ranchu related things. Things >>> working fine with the emulator (aosp_x86_64-eng for example and >>> android-goldfish-4.14-dev which seems mandatory to get Android 10 working >>> with AVD Emulator) does not seems to be working with upstream QEMU (at >>> least I can't get the GUI). Therefore the documentation about AVD Emulator >>> builds https://source.android.com/setup/create/avd didn't really help >>> me for upstream QEMU. >>> >>> Can you confirm it's possible to use modern versions of AOSP on upstream >>> QEMU? Someone did succeed in the past with Android 6 / untouched AOSP and a >>> extended compatibility kernel (I have one based on >>> android-goldfish-4.14-dev), this is why I'm trying with newer versions of >>> AOSP. This was documented here, but it's not up to date anymore : >>> https://www.collabora.com/news-and-blog/blog/2016/09/02/building-android-for-qemu-a-step-by-step-guide/ >>> >>> Thank you in advance >>> >>> Julien >>> >>> >>> On 3/2/20 10:51 PM, 'Matthew Maurer' via Android Building wrote: >>> >>> This target is a testing target used for evaluating our reference >>> TrustZone implementation. "storageproxyd" is continuously resetting there >>> because it is not being launched by our test script >>> <https://android.googlesource.com/trusty/device/arm/generic-arm64/+/refs/heads/master/project/qemu/qemu.py> >>> which >>> would attach a virtual RPMB resource. >>> >>> A few notes about this target: >>> * It does not have a GUI >>> * Running Java is not supported and unlikely to work >>> * It requires a custom EL3 and Trusty running alongside it to work >>> >>> The manifest for these components is at >>> https://android.googlesource.com/trusty/manifest, and that manifest >>> contains a checked in prebuilt of the qemu_trusty_arm64-userdebug target >>> that we are currently testing against. >>> >>> If you tell me more about what you're trying to make the target do, I >>> may be able to give you further help with getting it to do what you want. >>> If you aren't trying to use or inspect the Trusty TZ implementation, this >>> target probably isn't the one you're looking for. >>> >>> On Mon, Mar 2, 2020 at 1:34 PM Dan Willemsen <[email protected]> >>> wrote: >>> >>>> +Matthew Maurer Do we have any documentation on this target? I think >>>> the last time we talked about this target it was only minimally functional. >>>> >>>> - Dan >>>> >>>> On Mon, Mar 2, 2020 at 12:51 PM Julien Robin <[email protected]> >>>> wrote: >>>> >>>>> I succeeded to find a workaround for the issue I signaled here, >>>>> editing device/generic/trusty/BoardConfig.mk and putting >>>>> BUILD_QEMU_IMAGES >>>>> to false, which still gives vendor.img, system.img and userdata.img >>>>> >>>>> I then found this kernel >>>>> https://android.googlesource.com/kernel/common/+/refs/heads/android-trusty-4.19, >>>>> >>>>> and using arch/arm64/configs/trusty_qemu_defconfig, I successfully built >>>>> it. >>>>> >>>>> Also, I'm able to start most of this Android build on Debian 10 >>>>> provided qemu-system-aarch64 the following way : >>>>> >>>>> qemu-system-aarch64 >>>>> -M virt >>>>> -cpu cortex-a57 >>>>> -smp 1 >>>>> -m 2048 >>>>> -monitor none >>>>> -parallel none >>>>> -serial mon:stdio >>>>> -kernel Image -append "root=/dev/vda rootfstype=ext4 ro >>>>> init=/init loglevel=8 selinux=1 checkreqprot=1 >>>>> androidboot.selinux=permissive androidboot.hardware=qemu_trusty >>>>> console=ttyAMA0" >>>>> -device virtio-gpu-pci >>>>> -drive format=raw,file=system-custom.img >>>>> -drive format=raw,file=userdata-custom.img >>>>> >>>>> userdata-custom.img is just a bigger userdata.img (1024MB instead of >>>>> 4) in which I used resize2fs, system-custom.img is just a system.img in >>>>> which the placeholder folder "vendor" now embeds the content of >>>>> vendor.img, >>>>> copied by "cp -a". Finally, still into system-custom.img, >>>>> fstab.qemu_trusty >>>>> only contains >>>>> LABEL=data /data ext4 noatime,nosuid, >>>>> nodev,errors=panic wait >>>>> So that /data (available at /dev/vdb) is mounted by its label, while >>>>> system-custom.img embeding root system and vendor is already mounted by >>>>> kernel cmdline (by /dev/vda). ramdisk.img is not used as system.img >>>>> already >>>>> contains everything needed to start init. >>>>> >>>>> However, I guess I should use a slightly modified version of qemu, >>>>> available here >>>>> https://android.googlesource.com/trusty/external/qemu/+/refs/heads/master >>>>> Because using Debian 10 qemu version, I get the following output, and >>>>> no display : >>>>> >>>>> init: init first stage started! >>>>> [...] >>>>> init: Loading SELinux policy >>>>> [...] >>>>> selinux: SELinux: Loaded policy from /vendor/etc/selinux/ >>>>> precompiled_sepolicy >>>>> >>>>> selinux: SELinux: Loaded file_contexts >>>>> >>>>> random: init: uninitialized urandom read (40 bytes read) >>>>> random: init: uninitialized urandom read (40 bytes read) >>>>> init: init second stage started! >>>>> [...] >>>>> ueventd: Coldboot took 1.344 seconds >>>>> [...] >>>>> init: starting service 'storageproxyd'... >>>>> libprocessgroup: CgroupMap::FindController called for [1] failed, RC >>>>> file was not initialized properly >>>>> libprocessgroup: Failed to make and chown /uid_0: Read-only file >>>>> system >>>>> libprocessgroup: CgroupMap::FindController called for [829] failed, >>>>> RC file was not initialized properly >>>>> init: createProcessGroup(0, 829) failed for service 'storageproxyd': >>>>> Read-only file system >>>>> init: cpuset cgroup controller is not mounted! >>>>> init: Service 'storageproxyd' (pid 829) exited with status 1 >>>>> init: Sending signal 9 to service 'storageproxyd' (pid 829) process >>>>> group... >>>>> libprocessgroup: CgroupMap::FindController called for [1] failed, RC >>>>> file was not initialized properly >>>>> libprocessgroup: CgroupMap::FindController called for [1] failed, RC >>>>> file was not initialized properly >>>>> >>>>> >>>>> init: starting service 'storageproxyd'... >>>>> libprocessgroup: CgroupMap::FindController called for [1] failed, RC >>>>> file was not initialized properly >>>>> libprocessgroup: Failed to make and chown /uid_0: Read-only file >>>>> system >>>>> init: createProcessGroup(0, 830) failed for service 'storageproxyd': >>>>> Read-only file system >>>>> libprocessgroup: CgroupMap::FindController called for [830] failed, >>>>> RC file was not initialized properly >>>>> init: cpuset cgroup controller is not mounted! >>>>> init: Service 'storageproxyd' (pid 830) exited with status 1 >>>>> init: Sending signal 9 to service 'storageproxyd' (pid 830) process >>>>> group... >>>>> >>>>> ...and so on in loop >>>>> >>>>> However I'm still not sure to proceed correctly, as I'm doing >>>>> everything alone with no documentation (I may have missed it?) >>>>> >>>>> Is here some documentation available? If there isn't, is someone able >>>>> to provide an example of working qemu command line (even old or >>>>> unoptimized), or just few explanations so that I can be placed on the >>>>> right >>>>> track? >>>>> >>>>> Thank you very much in advance >>>>> >>>>> >>>>> On 3/1/20 2:14 PM, Julien Robin wrote: >>>>>> >>>>>> Hi there, >>>>>> >>>>>> I see that starting from branch android-10.0.0_r1, lunch offers a >>>>>> choice called qemu_trusty_arm64-userdebug, in which I'm interested >>>>>> >>>>>> However, I got the following error, no matter which OS I'm using for >>>>>> building (be it Ubuntu 18.048 or Debian 10). I have this error on r1, >>>>>> r20 >>>>>> and r29. >>>>>> >>>>>> [ 62% 16150/26047] Create system-qemu.img now >>>>>> FAILED: out/target/product/trusty/system-qemu.img >>>>>> /bin/bash -c "(export SGDISK=out/host/linux-x86/bin/sgdisk >>>>>> SIMG2IMG=out/host/linux-x86/bin/simg2img; >>>>>> device/generic/goldfish/tools/mk_combined_img.py -i >>>>>> out/target/product/trusty/system-qemu-config.txt -o >>>>>> out/target/product/trusty/system-qemu.img)" >>>>>> 1+0 records in >>>>>> 2048+0 records out >>>>>> 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0063128 s, 166 MB/s >>>>>> Traceback (most recent call last): >>>>>> File "device/generic/goldfish/tools/mk_combined_img.py", line 163, >>>>>> in <module> >>>>>> main() >>>>>> File "device/generic/goldfish/tools/mk_combined_img.py", line 135, >>>>>> in main >>>>>> if check_sparse(partition["path"]): >>>>>> File "device/generic/goldfish/tools/mk_combined_img.py", line 12, >>>>>> in check_sparse >>>>>> with open(filename, 'rb') as i: >>>>>> IOError: [Errno 2] No such file or directory: >>>>>> 'out/target/product/trusty/vbmeta.img' >>>>>> 13:44:11 ninja failed with: exit status 1 >>>>>> >>>>>> #### failed to build some targets (19:41 (mm:ss)) #### >>>>>> >>>>>> What should I do to get qemu_trusty_arm64-userdebug building >>>>>> successfully? >>>>>> >>>>>> Thank you in advance! >>>>>> >>>>>> Best regards, >>>>>> Julien ROBIN >>>>>> >>>>> -- >>>>> -- >>>>> You received this message because you are subscribed to the "Android >>>>> Building" mailing list. >>>>> To post to this group, send email to [email protected] >>>>> To unsubscribe from this group, send email to >>>>> [email protected] >>>>> For more options, visit this group at >>>>> http://groups.google.com/group/android-building?hl=en >>>>> >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Android Building" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/android-building/a30c5a05-6134-43a5-8f3b-32313e32bac0%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/android-building/a30c5a05-6134-43a5-8f3b-32313e32bac0%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>> -- >>> You received this message because you are subscribed to the "Android >>> Building" mailing list. >>> To post to this group, send email to [email protected] >>> To unsubscribe from this group, send email to >>> [email protected] >>> For more options, visit this group at >>> http://groups.google.com/group/android-building?hl=en >>> >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Android Building" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/android-building/fY-gdZQ-67M/unsubscribe >>> . >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/android-building/CAGSQo01HoD%2B8b1LdOrqaqmnQK5Fa1Ys9REVZ9Rt-85E4WrYRaw%40mail.gmail.com >>> >>> <https://groups.google.com/d/msgid/android-building/CAGSQo01HoD%2B8b1LdOrqaqmnQK5Fa1Ys9REVZ9Rt-85E4WrYRaw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> -- -- You received this message because you are subscribed to the "Android Building" mailing list. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/android-building?hl=en --- You received this message because you are subscribed to the Google Groups "Android Building" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/android-building/9ffaf8c5-828c-40ef-8dd1-54479ba99ac6o%40googlegroups.com.
