Re: [Bug 1636847] Re: unexpectedly large memory usage of mounted snaps

2021-03-23 Thread Seth Forshee
On Thu, Mar 18, 2021 at 01:36:55AM -, SaveTheRbtz wrote: > Is it possible (pun intended) to compromise on > `CONFIG_SQUASHFS_DECOMP_MULTI` (which uses num_online_cpus) instead of > `CONFIG_SQUASHFS_DECOMP_MULTI_PERCPU` (which uses `num_possible_cpus`)? It's better, but still not that great of

[Bug 1636847] Re: unexpectedly large memory usage of mounted snaps

2021-03-17 Thread SaveTheRbtz
Is it possible (pun intended) to compromise on `CONFIG_SQUASHFS_DECOMP_MULTI` (which uses num_online_cpus) instead of `CONFIG_SQUASHFS_DECOMP_MULTI_PERCPU` (which uses `num_possible_cpus`)? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubunt

[Bug 1636847] Re: unexpectedly large memory usage of mounted snaps

2021-03-17 Thread Seth Forshee
@SaveTheRbtz I did get to the cause, and added a lot of notes about it in the comments for this bug. I even wrote some code to provide a more balanced compromise between memory usage and performance, but once the memory issues were alleviated it seems no one cared about it enough to even try out th

[Bug 1636847] Re: unexpectedly large memory usage of mounted snaps

2021-03-17 Thread SaveTheRbtz
The log message[1] says that this issues is under investigation: > For the moment turn it off while it can be investigated. Did we find the root cause? Seems like Debian is successfully uses it nowadays: ``` tar -xOzf linux_5.10.19-1.debian.tar.xz ./debian/config/config | fgrep CONFIG_SQUASHFS_D

[Bug 1636847] Re: unexpectedly large memory usage of mounted snaps

2017-01-17 Thread Michael Vogt
** Changed in: snappy Status: In Progress => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1636847 Title: unexpectedly large memory usage of mounted snaps To manage notification

[Bug 1636847] Re: unexpectedly large memory usage of mounted snaps

2016-12-06 Thread Launchpad Bug Tracker
This bug was fixed in the package linux - 4.8.0-30.32 --- linux (4.8.0-30.32) yakkety; urgency=low * CVE-2016-8655 (LP: #1646318) - packet: fix race condition in packet_set_ring -- Brad Figg Thu, 01 Dec 2016 08:02:53 -0800 ** Changed in: linux (Ubuntu Zesty) Status:

[Bug 1636847] Re: unexpectedly large memory usage of mounted snaps

2016-11-30 Thread Michael Vogt
** Changed in: snappy Status: New => In Progress ** Changed in: snappy Importance: Undecided => Critical -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1636847 Title: unexpectedly large me

[Bug 1636847] Re: unexpectedly large memory usage of mounted snaps

2016-11-29 Thread Launchpad Bug Tracker
This bug was fixed in the package linux - 4.8.0-28.30 --- linux (4.8.0-28.30) yakkety; urgency=low [ Luis Henriques ] * Release Tracking Bug - LP: #1641083 * lxc-attach to malicious container allows access to host (LP: #1639345) - Revert "UBUNTU: SAUCE: (noup) ptrace:

[Bug 1636847] Re: unexpectedly large memory usage of mounted snaps

2016-11-28 Thread Seth Forshee
Thanks for the responses, that explains the strange results on the single CPU system. Have you had a chance to try the new kernels yet? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1636847 Title: u

[Bug 1636847] Re: unexpectedly large memory usage of mounted snaps

2016-11-22 Thread Zygmunt Krynicki
Hey Seth. To reply to your earlier question: @zyga: I'm honestly very surprised that the config change had that drastic an impact on the single-CPU system. Can you tell me what 'cat /sys/devices/system/cpu/possible' says on that system? This was in a virtual machine with one CPU and the file lis

[Bug 1636847] Re: unexpectedly large memory usage of mounted snaps

2016-11-16 Thread Luis Henriques
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- yakkety' to 'verification-done-yakkety'. If verification is not done by 5 working days from

[Bug 1636847] Re: unexpectedly large memory usage of mounted snaps

2016-11-14 Thread Seth Forshee
Observations from some very unscientific testing. Testing was done with fio using 8 parallel jobs doing random reads in an amd64 VM with 8 cores. * The kernel in comment #10 and 4.4.0-45 (with SQUASHFS_DECOMP_MULTI_PERCPU) performed comparably for the most part. 4.4.0-47 (with CONFIG_SQUASHFS_DECO

[Bug 1636847] Re: unexpectedly large memory usage of mounted snaps

2016-11-10 Thread Seth Forshee
Regarding #2 in comment #8 - I found that we can more or less do this with few simple modifications to SQUASHFS_DECOMP_MULTI. The config options are upper bounds on the number of decompressors and data cache blocks. I tested this with the mounted-fs-memory-checker for comparison, limiting squashfs

[Bug 1636847] Re: unexpectedly large memory usage of mounted snaps

2016-11-09 Thread Launchpad Bug Tracker
This bug was fixed in the package linux - 4.4.0-47.68 --- linux (4.4.0-47.68) xenial; urgency=low [ Kamal Mostafa ] * Release Tracking Bug - LP: #1636941 * Add a driver for Amazon Elastic Network Adapters (ENA) (LP: #1635721) - lib/bitmap.c: conversion routines to/from

[Bug 1636847] Re: unexpectedly large memory usage of mounted snaps

2016-11-08 Thread Seth Forshee
Adding some notes and ideas based on looking more at the squashfs implementation. One idea I had was to reduce the data cache (or maybe even eliminate it) by making squashfs decompress directly into the page cache. It turns out that squashfs already does this if CONFIG_SQUASHFS_FILE_DIRECT=y, whic

[Bug 1636847] Re: unexpectedly large memory usage of mounted snaps

2016-11-07 Thread Seth Forshee
Test build with CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=1 is done. http://people.canonical.com/~sforshee/lp1636847/linux-4.4.0-46.67+lp1636847v201611071220/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1

[Bug 1636847] Re: unexpectedly large memory usage of mounted snaps

2016-11-07 Thread Seth Forshee
@zyga: I'm honestly very surprised that the config change had that drastic an impact on the single-CPU system. Can you tell me what 'cat /sys/devices/system/cpu/possible' says on that system? Can I assume that the "size-1M" implies a 1MB block size? Let's start with the simplest options for reduc

[Bug 1636847] Re: unexpectedly large memory usage of mounted snaps

2016-10-26 Thread Tim Gardner
** Changed in: linux (Ubuntu Xenial) Status: Confirmed => Fix Committed ** Changed in: linux (Ubuntu Yakkety) Status: Confirmed => Fix Committed ** Changed in: linux (Ubuntu) Status: Confirmed => Fix Committed ** Also affects: linux (Ubuntu Zesty) Importance: Medium

[Bug 1636847] Re: unexpectedly large memory usage of mounted snaps

2016-10-26 Thread Andy Whitcroft
** Also affects: linux (Ubuntu Yakkety) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Xenial) Importance: Undecided Status: New ** Changed in: linux (Ubuntu Xenial) Status: New => Confirmed ** Changed in: linux (Ubuntu Yakkety) Status: New =>

[Bug 1636847] Re: unexpectedly large memory usage of mounted snaps

2016-10-26 Thread Zygmunt Krynicki
After using the CONFING_SQUASHFS_DECOMP_SINGLE=y option in an experimental kernel the memory usage dropped significantly. The traces below are for -45 and the experimental -46 kernel. The 1 and 4 are the number of CPUs on the system. A single CPU system used to consumer 131MB per mounted snap, thi

[Bug 1636847] Re: unexpectedly large memory usage of mounted snaps

2016-10-26 Thread Andy Whitcroft
Ok test kernels at the URL below, please report testing results back here: people.canonical.com/~apw/lp1636847-xenial/ Thanks. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1636847 Title: unex

[Bug 1636847] Re: unexpectedly large memory usage of mounted snaps

2016-10-26 Thread Andy Whitcroft
Evidence suggests this is triggered by the parallel decompressor. Will spin a kernel with that switched out: CONFIG_SQUASHFS_DECOMP_SINGLE=y -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1636847

[Bug 1636847] Re: unexpectedly large memory usage of mounted snaps

2016-10-26 Thread Jamie Strandboge
Marking as 'confirmed' so the bot doesn't auto-close it. ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1636847 Title: unexpected