On Tue, Sep 22, 2026 at 03:27:44PM +0530, Arun George/Arun George wrote: > Hi Gregory, > > Thanks for sharing the working branch. > > Some of us at Samsung are testing the series (both v4 & v5) on a > compression capable CXL expander and are observing encouraging results. >
This is really cool, thank you so much for spending the cycles to test. I'm working on getting a v6 out soon. Glad to see the progress! > Good things first! We feel that the isolation using private nodes and > the capability selection using NODE_PRIVATE_CAP_* work functionally well > for the compressed memory use cases. > > On improvement points, we observed much higher 'page_faults', > 'allocation_stalls' etc. which contributed to the higher tail latencies > in some tests. I hope these are already part of the optimization plans. > The higher latencies might have resulted from the write_protection > applied to the private node, I believe. > Yes, this is expected, and I think there are questions about how it should be optimized both in a provisioning sense and a software sense. For example, exposing a full memory expander as 100% compressible may not make sense - given that this could many a lot of memory. I have some data that shows it may make more sense to split a memory expander into two regions to better fit the compressible region to meet the actual capacity the cold-tail is capable of consuming. On the software side there may be some ability to adjust this at runtime with hotplug instead of requiring a reboot, but I think that needs to be experimented with. > On the tools side, we used MLC and Taobench for the tests. > Great! I also have some TaoBench and fio results i'm looking to share, glad to have some additional peer review using Tao. > Note that our intention for this phase of experiments was to test only > the 'private node' layer based isolation, and not the cram and below > layers for the compressed memory. Therefore we did not enable/utilize > the compression capability in the hardware for these set of experiments > (enabling compression would require the cram level ballooning/memory > shrinking and cxl level interfacing driver to the compression device). > That would be different set of experiments where we would be testing the > cram balloon shrinkers and our alternate algorithms (upstream targeted). > So cram and below layers were used only for enumeration of private node > regions for these tests and not for the run-time memory shrinkers. > Agreed, i'm looking forward to getting past the isolation bits and get more focused on the compression implementation details. > ======= Test Methodology ============== > > We explored these 3 cases: > > 1) (Baseline – existing CXL infra): DRAM 16GB + CXL 32GB. Here we > used the existing CXL driver infra to enable the device. No private node > is involved. And compression is disabled on device. > > 2) (private node in default write protected path): DRAM 16GB + CXL > Private Node 32GB. Here private node infra is used to enable the device. > And compression is disabled on device. > > 3) (private node with no write protection): DRAM 16GB + CXL Private > Node 32GB. Here private node infra is used to enable the device without > the write protection/fencing enabled. Compression is disabled on device. > We could not complete these runs as they resulted in kernel panics. > Guess the code path is not stable yet for this (We had hoped that case 1 > and case 3 results would be similar). We also observed some unmovable > page warnings logs in 'dmesg' before crash (might be related to the > panic). Adding the log snippets at the end. > Very similar to my test setup, seems like good signal we are of similar mind on the use case. (For readers: there was no coordination here). I personally did not test private-node without write protection, although it makes sense to test the throughput of a demotion-only node as a baseline. Smart. My guess is there was probably a reclaim throughput issue, or there may have been a bug in the v4/v5 code. I will look at adding some pressure tests to my suite that test demotion-only explicitly. > ========= Results summary =================== > > Private (CRAM) node performance is lower compared to a normal CXL memory > allocation path. VM stats shows more page_faults and allocation_stalls > on the private node case. Could it be the allocator waits for migration > path to demote pages to private node? Or the actual hot pages in dram > got demoted to private node to make space during overwrites? We will try > further analysis on this. > - allocator waits for migration? Yes. And worse, once the node is full, migration will fail and you'll swap directly from the top tier and reduce your reclaim behavior on the lower node. It becomes very important to push proactive reclaim on a demotion-only tier to ensure there is sufficient headroom to receive demotions in the future, since direct-reclaim basically never targets a lower-tier on the first pass. There's a big discussion about whether tiered systems need to invert their reclaim behavior (reclaim from the lowest node first to find progress there, then do demotion - rather than the other way around). - Hot pages in dram Also yes. The comparison to make isn't only against a raw CXL node, but also against Zswap or Zram or Swap. How does the workload fair on a system with 16GB RAM and 32GB Zswap/Swap? This is important context. > ========== MLC Experiments ==================== > > CMD: > $./mlc --loaded_latency -j0 -c0 -b1g -k1-15 -W5 -r > > Experiment Results: > > 1. (Baseline) DRAM 16GB + CXL 32GB > > Inject Latency Bandwidth > Delay (ns) MB/sec > ================ > 00000 686.18 40137.5 > 00002 686.69 40124.0 > 00008 709.74 40362.7 ... > 2. (private node in default write protected path) > DRAM 16GB + CXL Private Node(Uncompressed) 32GB > > Inject Latency Bandwidth > Delay (ns) MB/sec > ================ > 00000 320.09 12186.2 > 00002 543.42 22726.1 > 00008 482.11 28385.2 ... > Takeaway: Our MLC tests show a trade-off between the two configurations. > During low inject delay, the private Node is faster. But when inject > delay goes high, the baseline shows better latency. Could it be the TLB > cache effects? > These latency results suggest to me that the memory being tested was always DRAM, though the initial memory fault would have been CXL. i.e. even if the memory was faulted directly onto the node via mempolicy, the very first write to it would have caused promotion. If you dropped the write protection you'd probably see it equals the baseline. The bandwidth numbers support this. Higher stall, but faster latency means the migration was more beneficial for latency than letting the memory sit on CXL. Neat. I'm not sure these results are particularly useful in describing how a workload would react. > -------------------------------------------------------------------- > Case 3 dmesg log snippet: > > [ 82.525741] page: refcount:1 mapcount:0 mapping:0000000000000000 > index:0x0 pfn:0x5f5800 > [ 82.525753] flags: > 0x17ffffc0002000(reserved|node=0|zone=2|lastcpupid=0x1fffff) > [ 82.525762] raw: 0017ffffc0002000 ffefc85717d60008 ffefc85717d60008 > 0000000000000000 > [ 82.525764] raw: 0000000000000000 0000000000000000 00000001ffffffff > 0000000000000000 > [ 82.525766] page dumped because: unmovable page > .............. > Was this memory hotplugged via cram.c or did you replicate the code and hotplug it another way? If so, did you hotplug it as ZONE_NORMAL or ZONE_MOVABLE? For a compressed tier, it must always be hotplugged as ZONE_MOVABLE, without exception, because otherwise it can allow a GUP pin or kernel allocation that would eventually become permanently stuck there - and by definition of a compression tier must be 100% movable memory. ~Gregory
