I mentioned this to Marcus via irc yesterday and I think it is useful context (leaving out parts that John already mentioned):
"07:48 <marcustomlinson> so question: when we have that slow boot, what screen is everyone left on to wait 07:48 <marcustomlinson> blank? spinning ubuntu icom 07:49 <jdstrand> blank 07:49 <jdstrand> it is after the image is flashed and is comin up. there is no feedback ... 07:54 <marcustomlinson> I think we can hold back on Michi's MP as worst case scenario ... 07:55 <jdstrand> basically, the way it is all framed is that these sorts of updates should typically only happen when jumping base releases, since a policy recompile has to happen anyway 07:55 <jdstrand> so, vivid to xenial 07:55 <jdstrand> I can push this change into xenial no problem 07:55 <jdstrand> then whenever that ota comes out, it is there 07:55 <marcustomlinson> ah ok 07:55 <jdstrand> but we try to reduce the pain of interim updates 07:56 <jdstrand> we don't want every ota to have a slow boot 07:56 <marcustomlinson> do you see these updates as that rarely required? 07:56 <jdstrand> but it is possible to batch them or make a decision that this is important enough to make everyone wait 07:56 <jdstrand> yes 07:56 <jdstrand> we've maybe only had to recompile all policy once since vivid base ... 07:57 <jdstrand> we do make other profile changes, but that might be in say, the calendar policy 07:57 <jdstrand> the calendar policy is only used by a few apps-- so no big deal ... 07:57 <jdstrand> but something that affects all scopes is a different thing 07:58 <marcustomlinson> ok, thanks for the info! 07:58 <jdstrand> np 07:58 <jdstrand> if you are interested in the nitty gritty details, do see the apparmor bug 07:58 <marcustomlinson> will do 07:59 <jdstrand> it should be noted that since that bug has been filed, we have made many policy compile improvements 07:59 <jdstrand> but the problem can never go away 07:59 <jdstrand> it was at 2.5 seconds 07:59 <jdstrand> per profile 08:00 <jdstrand> I think we are under 1.5 now 08:00 <marcustomlinson> nice 08:00 <jdstrand> but even if we got it to 1, if you have 200 apps installed, that is almost 3.5 minutes of waiting 08:00 <marcustomlinson> like I said, I think we actually "make this go away" by showing people a progress bar or something ... 08:00 <jdstrand> I agree. the pieces are there on click. something different would have to be done for snappy " In a nutshell, John has done some amazing stuff already with policy compile times, and more improvements will come and in fact, we are already considerably faster than IOS (aiui) and Android's app update process after reboot even with full policy recompiles, but the problem is we don't give people any feedback what is happening (I discuss ways for unity/the platform to deal with this above). Do keep in mind that because of the sheer volume of apps that can be installed on a phone and because the compile time won't ever be '0', we will always want to be judicious about policy updates that affect all apps-- that's where visual feedback can help. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to apparmor in Ubuntu. https://bugs.launchpad.net/bugs/1350598 Title: AppArmor policy compile improvements Status in AppArmor: Triaged Status in Canonical System Image: Confirmed Status in apparmor package in Ubuntu: Triaged Bug description: apparmor_parser can take a long time to compile policy especially when there is a lot of policy, so we want to utilize compiled cache profile as much as possible. Cache files will have to be regenerated in the following cases: * the kernel .features file is updated (eg, new features are added to apparmor in the new kernel) * apparmor itself is updated * on devices with click packages, apparmor-easyprof-ubuntu and/or click-apparmor is updated As of 2014-10-02, what can be expected is: - Systems with system-image updates (eg, Ubuntu Touch): - First boot will use the precompiled cache files in the rootfs or custom tarball and be fast - Reboots will use the cache files on the device and be fast - First boots after upgrades will use the cache files on the device if the above conditions are not met and be fast - Production devices will not meet any of those conditions except under exceptional and rare circumstances (eg, major OS upgrades like 14.10 to 15.04) and be fast - First boots after upgrades that meet one of the conditions will need to regenerate the cache. This can happen on development releases where the kernel features file, apparmor, apparmor-easyprof-ubuntu or click-apparmor are still under development and getting updates - Systems with apt updates (eg, current Ubuntu Desktop and Server): - First boot will compile cache files - Reboots will use the cache files on the machine and be fast - First boots after upgrades will use the cache files on the machine if the above conditions are not met and be fast - Stable releases of Ubuntu will not meet any of those conditions except under exceptional and rare circumstances (eg, major OS upgrades like 14.10 to 15.04) and be fast - First boots after upgrades that meet one of the above conditions will need to regenerate the cache. This can happen on development releases where the kernel features file, apparmor, apparmor-easyprof-ubuntu or click-apparmor are still under development and getting updates In addition to the above, updates to only apparmor-easyprof-ubuntu will regenerate the cache files for only the policy that is affected (eg, if there is a change to the location policy group in policy version 1.2, only apps using this policy version and this policy group will need to be recompiled). Planned improvements (in order of most likely to be done first): 1. Finetuning the checks to invalidate the cache (eg, .md5sums could only be for /etc/apparmor.d/abstractions, ...): WONTFIX (will want an md5sum on apparmor_parser since it could change the cache and the md5sum will always change. Furthermore, apparmor-easyprof-ubuntu is all policy so there is no gain there. click-apparmor could possibly benefit, but it doesn't change often and when it does, it is typically for policy) 2. Investigate ways to utilize the custom tarball and rootfs precompiled cache files on upgrades when apparmor, apparmor-easyprof-ubuntu and click-apparmor are updated: DONE 3. Improve cache handling for app store apps (eg, having the app store server precompile them so that the device can download them when it needs to rather than having to regenerate them itself): WONTFIX (doesn't scale) 4. For systems with apt upgrades, compile the policy either during install or on kernel upgrade rather than on boot. For systems with read-only fs-style upgrades, compile the policy prior to reboot rather than on boot. 5. Support cache files per kernel .features file (or kernel version). This will allow people to boot into a previous kernel without having to recompile policy 6. Improve parser compile time 7. Investigate how to utilize profile composition and profile stacking to decrease compile and load times (eg, one idea is that the policy template is compiled once and each policy group once such that the parser need only stitch the already compiled bits together) 8. For Ubuntu Touch/Personal system-image based systems, investigate ways to utilize the update tarball and compile policy before rebooting to improve the user experience Work is planned for the medium term for '1-3' with '4' and '5' coming as needed. '6' will of course help, but it is already very optimized and compile average ~2 seconds on armhf per profile (note: already faster than Android's 'optimizing apps' per app on a nexus 7) -- if we cut that in half a typical user with 300 apps would still have to wait 300 seconds so other techniques like '2' should be employed. '6' and '7' will be handled in the long term. '8' can be implemented now to improve the user experience: " > Sorry for not being clear. The idea is that when the phone says that > there is an update, the user has to tap 'Install and Reboot'. The idea > is that before reboot (and therefore still in the unity8 session), we > look inside what is downloaded, see if there are any policy changes. If > there are, we extract them and then compile policy with a progress > meter. The question I posed to you is how hard is it to look inside (or > provide a manifest of changed packages) and extract what is needed to > compile policy? Ok. The update is available as a set of tarballs, available in a fixed directory. It should be straightforward to check whether any of those tarballs contains files matching a particular path. If you want to know whether particular packages have changed, that would be a matter of extracting the dpkg database and comparing. (We don't otherwise track the packagewise delta between the images.) A partial extraction of the tarball based on particular filenames is a simple matter of tar arguments." Original description: Just updated my Nexus 7 2013 from #160 to #161. It's been sat at the Google logo for 15 minutes now. It looks and feels like it's hung. As a user I'd be rebooting it thinking it had crashed by now. I shell in and find apparmor_parser using a lot of cpu for a long time. top - 00:14:01 up 15 min, 2 users, load average: 5.12, 4.85, 3.21 Tasks: 202 total, 2 running, 200 sleeping, 0 stopped, 0 zombie %Cpu(s): 50.5 us, 0.8 sy, 0.0 ni, 48.5 id, 0.2 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 1848024 total, 787400 used, 1060624 free, 54216 buffers KiB Swap: 32764 total, 0 used, 32764 free. 579228 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1970 root 20 0 4976 3652 852 R 99.8 0.2 14:31.04 apparmor_parser 2596 phablet 20 0 5996 1264 824 R 1.3 0.1 0:08.79 top 914 root 0 -20 7572 552 396 S 0.7 0.0 0:05.02 mpdecision 21 root 20 0 0 0 0 S 0.3 0.0 0:00.92 kworker/0:1 229 root 20 0 0 0 0 S 0.3 0.0 0:00.10 jbd2/mmcblk0p30 982 root 20 0 38856 1164 868 S 0.3 0.1 0:01.77 adbd 2570 phablet 20 0 10540 1456 692 S 0.3 0.1 0:02.30 sshd 1 root 20 0 3884 2648 1068 S 0.0 0.1 0:05.98 init 2 root -2 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:00.04 ksoftirqd/0 ... it eventually finished after 18 minutes. To manage notifications about this bug go to: https://bugs.launchpad.net/apparmor/+bug/1350598/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
