On 2/22/19 7:02 PM, Alexander Duyck wrote: > On Mon, Feb 4, 2019 at 1:47 PM Nitesh Narayan Lal <[email protected]> wrote: >> The following patch-set proposes an efficient mechanism for handing freed >> memory between the guest and the host. It enables the guests with no page >> cache to rapidly free and reclaims memory to and from the host respectively. >> >> Benefit: >> With this patch-series, in our test-case, executed on a single system and >> single NUMA node with 15GB memory, we were able to successfully launch >> atleast 5 guests >> when page hinting was enabled and 3 without it. (Detailed explanation of the >> test procedure is provided at the bottom). >> >> Changelog in V8: >> In this patch-series, the earlier approach [1] which was used to capture and >> scan the pages freed by the guest has been changed. The new approach is >> briefly described below: >> >> The patch-set still leverages the existing arch_free_page() to add this >> functionality. It maintains a per CPU array which is used to store the pages >> freed by the guest. The maximum number of entries which it can hold is >> defined by MAX_FGPT_ENTRIES(1000). When the array is completely filled, it >> is scanned and only the pages which are available in the buddy are stored. >> This process continues until the array is filled with pages which are part >> of the buddy free list. After which it wakes up a kernel per-cpu-thread. >> This kernel per-cpu-thread rescans the per-cpu-array for any re-allocation >> and if the page is not reallocated and present in the buddy, the kernel >> thread attempts to isolate it from the buddy. If it is successfully >> isolated, the page is added to another per-cpu array. Once the entire >> scanning process is complete, all the isolated pages are reported to the >> host through an existing virtio-balloon driver. >> >> Known Issues: >> * Fixed array size: The problem with having a fixed/hardcoded array >> size arises when the size of the guest varies. For example when the guest >> size increases and it starts making large allocations fixed size limits this >> solution's ability to capture all the freed pages. This will result in less >> guest free memory getting reported to the host. >> >> Known code re-work: >> * Plan to re-use Wei's work, which communicates the poison value to >> the host. >> * The nomenclatures used in virtio-balloon needs to be changed so >> that the code can easily be distinguished from Wei's Free Page Hint code. >> * Sorting based on zonenum, to avoid repetitive zone locks for the >> same zone. >> >> Other required work: >> * Run other benchmarks to evaluate the performance/impact of this >> approach. >> >> Test case: >> Setup: >> Memory-15837 MB >> Guest Memory Size-5 GB >> Swap-Disabled >> Test Program-Simple program which allocates 4GB memory via malloc, touches >> it via memset and exits. >> Use case-Number of guests that can be launched completely including the >> successful execution of the test program. >> Procedure: >> The first guest is launched and once its console is up, the test allocation >> program is executed with 4 GB memory request (Due to this the guest occupies >> almost 4-5 GB of memory in the host in a system without page hinting). Once >> this program exits at that time another guest is launched in the host and >> the same process is followed. We continue launching the guests until a guest >> gets killed due to low memory condition in the host. >> >> Result: >> Without Hinting-3 Guests >> With Hinting-5 to 7 Guests(Based on the amount of memory freed/captured). >> >> [1] https://www.spinics.net/lists/kvm/msg170113.html > So I tried reproducing your test and I am not having much luck. > According to the sysctl in the guest I am seeing > "vm.guest-page-hinting = 1" which is supposed to indicate that the > hinting is enabled in both QEMU and the guest right? That is correct. If your guest has the balloon driver enabled it will also enable the hinting. > I'm just wanting > to verify that this is the case before I start doing any debugging. > > I'm assuming you never really ran any multi-threaded tests on a > multi-CPU guest did you? This is correct. I forgot to mention this as another todo item for me in the cover email. I will test multiple vcpus, once I finalize the design changes which I am doing right now. Thanks for pointing this out. > With the patches applied I am seeing > stability issues. If I enable a VM with multiple CPUs and run > something like the page_fault1 test from the will-it-scale suite I am > seeing multiple traces being generated by the guest kernel and it > ultimately just hangs. As I am done with the changes on which I am currently working. I will look into this as well. > > I have included the traces below. There end up being 3 specific > issues, a double free that is detected, the RCU stall, and then starts > complaining about a soft lockup. > > Thanks. > > - Alex > > -- This looks like a page complaining about a double add when added to > the LRU -- > [ 50.479635] list_add double add: new=fffff64480000008, > prev=ffffa000fffd50c0, next=fffff64480000008. > [ 50.481066] ------------[ cut here ]------------ > [ 50.481753] kernel BUG at lib/list_debug.c:31! > [ 50.482448] invalid opcode: 0000 [#1] SMP PTI > [ 50.483108] CPU: 1 PID: 852 Comm: hinting/1 Not tainted > 5.0.0-rc7-next-20190219-baseline+ #50 > [ 50.486362] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > BIOS Bochs 01/01/2011 > [ 50.487881] RIP: 0010:__list_add_valid+0x4b/0x70 > [ 50.488623] Code: 00 00 c3 48 89 c1 48 c7 c7 d8 70 10 9e 31 c0 e8 > 4f db c8 ff 0f 0b 48 89 c1 48 89 fe 31 c0 48 c7 c7 88 71 10 9e e8 39 > db c8 ff <0f> 0b 48 89 d1 48 c7 c7 30 71 10 9e 48 89 f2 48 89 c6 31 c0 > e8 20 > [ 50.492626] RSP: 0018:ffffb9a8c3b4bdf0 EFLAGS: 00010246 > [ 50.494189] RAX: 0000000000000058 RBX: ffffa000fffd50c0 RCX: > 0000000000000000 > [ 50.496308] RDX: 0000000000000000 RSI: ffffa000df85e6c8 RDI: > ffffa000df85e6c8 > [ 50.497876] RBP: ffffa000fffd50c0 R08: 0000000000000273 R09: > 0000000000000005 > [ 50.498981] R10: 0000000000000000 R11: ffffb9a8c3b4bb70 R12: > fffff64480000008 > [ 50.500077] R13: fffff64480000008 R14: fffff64480000000 R15: > ffffa000fffd5000 > [ 50.501184] FS: 0000000000000000(0000) GS:ffffa000df840000(0000) > knlGS:0000000000000000 > [ 50.502432] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 50.503325] CR2: 00007ffff6e47000 CR3: 000000080f76c002 CR4: > 0000000000160ee0 > [ 50.504431] Call Trace: > [ 50.505464] free_one_page+0x2b5/0x470 > [ 50.506070] hyperlist_ready+0xa9/0xc0 > [ 50.506662] hinting_fn+0x1db/0x3c0 > [ 50.507220] smpboot_thread_fn+0x10e/0x160 > [ 50.507868] kthread+0xf8/0x130 > [ 50.508371] ? sort_range+0x20/0x20 > [ 50.508934] ? kthread_bind+0x10/0x10 > [ 50.509520] ret_from_fork+0x35/0x40 > [ 50.510098] Modules linked in: ip6t_rpfilter ip6t_REJECT > nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat > ebtable_broute bridge stp llc ip6table_nat nf_nat_ipv6 ip6table_mangle > ip6table_raw ip6table_security iptable_nat nf_nat_ipv4 nf_nat > nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw > iptable_security ebtable_filter ebtables ip6table_filter ip6_tables > sunrpc sb_edac crct10dif_pclmul crc32_pclmul ghash_clmulni_intel > kvm_intel kvm ppdev irqbypass parport_pc joydev virtio_balloon > pcc_cpufreq i2c_piix4 pcspkr parport xfs libcrc32c cirrus > drm_kms_helper ttm drm e1000 crc32c_intel virtio_blk ata_generic > floppy serio_raw pata_acpi qemu_fw_cfg > [ 50.519202] ---[ end trace 141fe2acdf2e3818 ]--- > [ 50.519935] RIP: 0010:__list_add_valid+0x4b/0x70 > [ 50.520675] Code: 00 00 c3 48 89 c1 48 c7 c7 d8 70 10 9e 31 c0 e8 > 4f db c8 ff 0f 0b 48 89 c1 48 89 fe 31 c0 48 c7 c7 88 71 10 9e e8 39 > db c8 ff <0f> 0b 48 89 d1 48 c7 c7 30 71 10 9e 48 89 f2 48 89 c6 31 c0 > e8 20 > [ 50.523570] RSP: 0018:ffffb9a8c3b4bdf0 EFLAGS: 00010246 > [ 50.524399] RAX: 0000000000000058 RBX: ffffa000fffd50c0 RCX: > 0000000000000000 > [ 50.525516] RDX: 0000000000000000 RSI: ffffa000df85e6c8 RDI: > ffffa000df85e6c8 > [ 50.526634] RBP: ffffa000fffd50c0 R08: 0000000000000273 R09: > 0000000000000005 > [ 50.527754] R10: 0000000000000000 R11: ffffb9a8c3b4bb70 R12: > fffff64480000008 > [ 50.528872] R13: fffff64480000008 R14: fffff64480000000 R15: > ffffa000fffd5000 > [ 50.530004] FS: 0000000000000000(0000) GS:ffffa000df840000(0000) > knlGS:0000000000000000 > [ 50.531276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 50.532189] CR2: 00007ffff6e47000 CR3: 000000080f76c002 CR4: > 0000000000160ee0 > > -- This appears to be a deadlock on the zone lock -- > [ 156.436784] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: > [ 156.439195] rcu: 0-...0: (0 ticks this GP) > idle=6ca/1/0x4000000000000000 softirq=10718/10718 fqs=2546 > [ 156.440810] rcu: 1-...0: (1 GPs behind) > idle=8f2/1/0x4000000000000000 softirq=8233/8235 fqs=2547 > [ 156.442320] rcu: 2-...0: (0 ticks this GP) > idle=ae2/1/0x4000000000000002 softirq=6779/6779 fqs=2547 > [ 156.443910] rcu: 3-...0: (0 ticks this GP) > idle=456/1/0x4000000000000000 softirq=1616/1616 fqs=2547 > [ 156.445454] rcu: (detected by 14, t=60109 jiffies, g=17493, q=31) > [ 156.446545] Sending NMI from CPU 14 to CPUs 0: > [ 156.448330] NMI backtrace for cpu 0 > [ 156.448331] CPU: 0 PID: 1308 Comm: page_fault1_pro Tainted: G > D 5.0.0-rc7-next-20190219-baseline+ #50 > [ 156.448331] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > BIOS Bochs 01/01/2011 > [ 156.448332] RIP: 0010:queued_spin_lock_slowpath+0x21/0x1f0 > [ 156.448332] Code: c0 75 ec c3 90 90 90 90 90 0f 1f 44 00 00 0f 1f > 44 00 00 ba 01 00 00 00 8b 07 85 c0 75 0a f0 0f b1 17 85 c0 75 f2 f3 > c3 f3 90 <eb> ec 81 fe 00 01 00 00 0f 84 44 01 00 00 81 e6 00 ff ff ff > 75 3e > [ 156.448333] RSP: 0000:ffffb9a8c3e83c10 EFLAGS: 00000002 > [ 156.448339] RAX: 0000000000000001 RBX: 0000000000000007 RCX: > 0000000000000001 > [ 156.448340] RDX: 0000000000000001 RSI: 0000000000000001 RDI: > ffffa000fffd6240 > [ 156.448340] RBP: 0000000000000000 R08: 0000000000000000 R09: > 00000000006f36aa > [ 156.448341] R10: 0000000000000000 R11: 0000000000000000 R12: > 0000000000000081 > [ 156.448341] R13: 0000000000100dca R14: 0000000000000000 R15: > ffffa000fffd5d00 > [ 156.448342] FS: 00007ffff7fec440(0000) GS:ffffa000df800000(0000) > knlGS:0000000000000000 > [ 156.448342] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 156.448342] CR2: 00007fffefe2d000 CR3: 0000000695904004 CR4: > 0000000000160ef0 > [ 156.448343] Call Trace: > [ 156.448343] get_page_from_freelist+0x50f/0x1280 > [ 156.448343] ? get_page_from_freelist+0xa44/0x1280 > [ 156.448344] __alloc_pages_nodemask+0x141/0x2e0 > [ 156.448344] alloc_pages_vma+0x73/0x180 > [ 156.448344] __handle_mm_fault+0xd59/0x14e0 > [ 156.448345] handle_mm_fault+0xfa/0x210 > [ 156.448345] __do_page_fault+0x207/0x4c0 > [ 156.448345] do_page_fault+0x32/0x140 > [ 156.448346] ? async_page_fault+0x8/0x30 > [ 156.448346] async_page_fault+0x1e/0x30 > [ 156.448346] RIP: 0033:0x401840 > [ 156.448347] Code: 00 00 45 31 c9 31 ff 41 b8 ff ff ff ff b9 22 00 > 00 00 ba 03 00 00 00 be 00 00 00 08 e8 d9 f5 ff ff 48 83 f8 ff 74 2b > 48 89 c2 <c6> 02 00 48 01 ea 48 83 03 01 48 89 d1 48 29 c1 48 81 f9 ff > ff ff > [ 156.448347] RSP: 002b:00007fffffffc0a0 EFLAGS: 00010293 > [ 156.448348] RAX: 00007fffeee48000 RBX: 00007ffff7ff7000 RCX: > 0000000000fe5000 > [ 156.448348] RDX: 00007fffefe2d000 RSI: 0000000008000000 RDI: > 0000000000000000 > [ 156.448349] RBP: 0000000000001000 R08: ffffffffffffffff R09: > 0000000000000000 > [ 156.448349] R10: 0000000000000022 R11: 0000000000000246 R12: > 00007fffffffc240 > [ 156.448349] R13: 0000000000000000 R14: 0000000000610710 R15: > 0000000000000005 > [ 156.448355] Sending NMI from CPU 14 to CPUs 1: > [ 156.489676] NMI backtrace for cpu 1 > [ 156.489677] CPU: 1 PID: 1309 Comm: page_fault1_pro Tainted: G > D 5.0.0-rc7-next-20190219-baseline+ #50 > [ 156.489677] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > BIOS Bochs 01/01/2011 > [ 156.489678] RIP: 0010:queued_spin_lock_slowpath+0x21/0x1f0 > [ 156.489678] Code: c0 75 ec c3 90 90 90 90 90 0f 1f 44 00 00 0f 1f > 44 00 00 ba 01 00 00 00 8b 07 85 c0 75 0a f0 0f b1 17 85 c0 75 f2 f3 > c3 f3 90 <eb> ec 81 fe 00 01 00 00 0f 84 44 01 00 00 81 e6 00 ff ff ff > 75 3e > [ 156.489679] RSP: 0000:ffffb9a8c3b4bc10 EFLAGS: 00000002 > [ 156.489679] RAX: 0000000000000001 RBX: 0000000000000007 RCX: > 0000000000000001 > [ 156.489680] RDX: 0000000000000001 RSI: 0000000000000001 RDI: > ffffa000fffd6240 > [ 156.489680] RBP: 0000000000000000 R08: 0000000000000000 R09: > 00000000006f36aa > [ 156.489680] R10: 0000000000000000 R11: 0000000000000000 R12: > 0000000000000081 > [ 156.489681] R13: 0000000000100dca R14: 0000000000000000 R15: > ffffa000fffd5d00 > [ 156.489681] FS: 00007ffff7fec440(0000) GS:ffffa000df840000(0000) > knlGS:0000000000000000 > [ 156.489682] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 156.489682] CR2: 00007ffff4608000 CR3: 000000081ddf6003 CR4: > 0000000000160ee0 > [ 156.489682] Call Trace: > [ 156.489683] get_page_from_freelist+0x50f/0x1280 > [ 156.489683] ? get_page_from_freelist+0xa44/0x1280 > [ 156.489683] __alloc_pages_nodemask+0x141/0x2e0 > [ 156.489683] alloc_pages_vma+0x73/0x180 > [ 156.489684] __handle_mm_fault+0xd59/0x14e0 > [ 156.489684] handle_mm_fault+0xfa/0x210 > [ 156.489684] __do_page_fault+0x207/0x4c0 > [ 156.489685] do_page_fault+0x32/0x140 > [ 156.489685] ? async_page_fault+0x8/0x30 > [ 156.489685] async_page_fault+0x1e/0x30 > [ 156.489686] RIP: 0033:0x401840 > [ 156.489686] Code: 00 00 45 31 c9 31 ff 41 b8 ff ff ff ff b9 22 00 > 00 00 ba 03 00 00 00 be 00 00 00 08 e8 d9 f5 ff ff 48 83 f8 ff 74 2b > 48 89 c2 <c6> 02 00 48 01 ea 48 83 03 01 48 89 d1 48 29 c1 48 81 f9 ff > ff ff > [ 156.489687] RSP: 002b:00007fffffffc0a0 EFLAGS: 00010293 > [ 156.489687] RAX: 00007fffeee48000 RBX: 00007ffff7ff7080 RCX: > 00000000057c0000 > [ 156.489692] RDX: 00007ffff4608000 RSI: 0000000008000000 RDI: > 0000000000000000 > [ 156.489693] RBP: 0000000000001000 R08: ffffffffffffffff R09: > 0000000000000000 > [ 156.489693] R10: 0000000000000022 R11: 0000000000000246 R12: > 00007fffffffc240 > [ 156.489694] R13: 0000000000000000 R14: 000000000060f870 R15: > 0000000000000005 > [ 156.489696] Sending NMI from CPU 14 to CPUs 2: > [ 156.530601] NMI backtrace for cpu 2 > [ 156.530602] CPU: 2 PID: 858 Comm: hinting/2 Tainted: G D > 5.0.0-rc7-next-20190219-baseline+ #50 > [ 156.530602] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > BIOS Bochs 01/01/2011 > [ 156.530603] RIP: 0010:queued_spin_lock_slowpath+0x21/0x1f0 > [ 156.530603] Code: c0 75 ec c3 90 90 90 90 90 0f 1f 44 00 00 0f 1f > 44 00 00 ba 01 00 00 00 8b 07 85 c0 75 0a f0 0f b1 17 85 c0 75 f2 f3 > c3 f3 90 <eb> ec 81 fe 00 01 00 00 0f 84 44 01 00 00 81 e6 00 ff ff ff > 75 3e > [ 156.530604] RSP: 0018:ffffa000df883e38 EFLAGS: 00000002 > [ 156.530604] RAX: 0000000000000001 RBX: fffff644a05a0ec8 RCX: > dead000000000200 > [ 156.530605] RDX: 0000000000000001 RSI: 0000000000000001 RDI: > ffffa000fffd6240 > [ 156.530605] RBP: ffffa000df8af340 R08: ffffa000da2b2000 R09: > 0000000000000100 > [ 156.530606] R10: 0000000000000004 R11: 0000000000000005 R12: > fffff6449fb5fb08 > [ 156.530606] R13: ffffa000fffd5d00 R14: 0000000000000001 R15: > 0000000000000001 > [ 156.530606] FS: 0000000000000000(0000) GS:ffffa000df880000(0000) > knlGS:0000000000000000 > [ 156.530607] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 156.530607] CR2: 00007ffff6e47000 CR3: 0000000813b34003 CR4: > 0000000000160ee0 > [ 156.530607] Call Trace: > [ 156.530608] <IRQ> > [ 156.530608] free_pcppages_bulk+0x1af/0x6d0 > [ 156.530608] free_unref_page+0x54/0x70 > [ 156.530608] tlb_remove_table_rcu+0x23/0x40 > [ 156.530609] rcu_core+0x2b0/0x470 > [ 156.530609] __do_softirq+0xde/0x2bf > [ 156.530609] irq_exit+0xd5/0xe0 > [ 156.530610] smp_apic_timer_interrupt+0x74/0x140 > [ 156.530610] apic_timer_interrupt+0xf/0x20 > [ 156.530610] </IRQ> > [ 156.530611] RIP: 0010:_raw_spin_lock+0x10/0x20 > [ 156.530611] Code: b8 01 00 00 00 c3 48 8b 3c 24 be 00 02 00 00 e8 > f6 cf 77 ff 31 c0 c3 0f 1f 00 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 > 0f b1 17 <0f> 94 c2 84 d2 74 02 f3 c3 89 c6 e9 d0 e8 7c ff 0f 1f 44 00 > 00 65 > [ 156.530612] RSP: 0018:ffffb9a8c3bf3df0 EFLAGS: 00000246 ORIG_RAX: > ffffffffffffff13 > [ 156.530612] RAX: 0000000000000000 RBX: 0000000000000001 RCX: > 0000000000000000 > [ 156.530613] RDX: 0000000000000001 RSI: fffff6449fd4aec0 RDI: > ffffa000fffd6240 > [ 156.530613] RBP: 0000000000000000 R08: 0000000000000001 R09: > 0000000000000002 > [ 156.530613] R10: 0000000000000000 R11: 0000000000003bf3 R12: > 00000000007f52bb > [ 156.530614] R13: 00000000007ecca4 R14: fffff6449fd4aec0 R15: > ffffa000fffd5d00 > [ 156.530614] free_one_page+0x32/0x470 > [ 156.530614] ? __switch_to_asm+0x40/0x70 > [ 156.530615] hyperlist_ready+0xa9/0xc0 > [ 156.530615] hinting_fn+0x1db/0x3c0 > [ 156.530615] smpboot_thread_fn+0x10e/0x160 > [ 156.530616] kthread+0xf8/0x130 > [ 156.530616] ? sort_range+0x20/0x20 > [ 156.530616] ? kthread_bind+0x10/0x10 > [ 156.530616] ret_from_fork+0x35/0x40 > [ 156.530619] Sending NMI from CPU 14 to CPUs 3: > [ 156.577112] NMI backtrace for cpu 3 > [ 156.577113] CPU: 3 PID: 1311 Comm: page_fault1_pro Tainted: G > D 5.0.0-rc7-next-20190219-baseline+ #50 > [ 156.577113] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > BIOS Bochs 01/01/2011 > [ 156.577114] RIP: 0010:queued_spin_lock_slowpath+0x21/0x1f0 > [ 156.577114] Code: c0 75 ec c3 90 90 90 90 90 0f 1f 44 00 00 0f 1f > 44 00 00 ba 01 00 00 00 8b 07 85 c0 75 0a f0 0f b1 17 85 c0 75 f2 f3 > c3 f3 90 <eb> ec 81 fe 00 01 00 00 0f 84 44 01 00 00 81 e6 00 ff ff ff > 75 3e > [ 156.577115] RSP: 0000:ffffb9a8c407fc10 EFLAGS: 00000002 > [ 156.577115] RAX: 0000000000000001 RBX: 0000000000000007 RCX: > 0000000000000001 > [ 156.577116] RDX: 0000000000000001 RSI: 0000000000000001 RDI: > ffffa000fffd6240 > [ 156.577116] RBP: 0000000000000000 R08: 0000000000000000 R09: > 00000000006f36aa > [ 156.577121] R10: 0000000000000000 R11: 0000000000000000 R12: > 0000000000000081 > [ 156.577122] R13: 0000000000100dca R14: 0000000000000000 R15: > ffffa000fffd5d00 > [ 156.577122] FS: 00007ffff7fec440(0000) GS:ffffa000df8c0000(0000) > knlGS:0000000000000000 > [ 156.577122] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 156.577123] CR2: 00007ffff398a000 CR3: 000000081aa00003 CR4: > 0000000000160ee0 > [ 156.577123] Call Trace: > [ 156.577123] get_page_from_freelist+0x50f/0x1280 > [ 156.577124] ? get_page_from_freelist+0xa44/0x1280 > [ 156.577124] ? try_charge+0x637/0x860 > [ 156.577124] __alloc_pages_nodemask+0x141/0x2e0 > [ 156.577125] alloc_pages_vma+0x73/0x180 > [ 156.577125] __handle_mm_fault+0xd59/0x14e0 > [ 156.577125] handle_mm_fault+0xfa/0x210 > [ 156.577126] __do_page_fault+0x207/0x4c0 > [ 156.577126] do_page_fault+0x32/0x140 > [ 156.577126] ? async_page_fault+0x8/0x30 > [ 156.577127] async_page_fault+0x1e/0x30 > [ 156.577127] RIP: 0033:0x401840 > [ 156.577128] Code: 00 00 45 31 c9 31 ff 41 b8 ff ff ff ff b9 22 00 > 00 00 ba 03 00 00 00 be 00 00 00 08 e8 d9 f5 ff ff 48 83 f8 ff 74 2b > 48 89 c2 <c6> 02 00 48 01 ea 48 83 03 01 48 89 d1 48 29 c1 48 81 f9 ff > ff ff > [ 156.577128] RSP: 002b:00007fffffffc0a0 EFLAGS: 00010293 > [ 156.577129] RAX: 00007fffeee48000 RBX: 00007ffff7ff7180 RCX: > 0000000004b42000 > [ 156.577129] RDX: 00007ffff398a000 RSI: 0000000008000000 RDI: > 0000000000000000 > [ 156.577130] RBP: 0000000000001000 R08: ffffffffffffffff R09: > 0000000000000000 > [ 156.577130] R10: 0000000000000022 R11: 0000000000000246 R12: > 00007fffffffc240 > [ 156.577130] R13: 0000000000000000 R14: 000000000060db00 R15: > 0000000000000005 > > -- After the above two it starts spitting this one out every 10 - 30 > seconds or so -- > [ 183.788386] watchdog: BUG: soft lockup - CPU#14 stuck for 23s! > [kworker/14:1:121] > [ 183.790003] Modules linked in: ip6t_rpfilter ip6t_REJECT > nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat > ebtable_broute bridge stp llc ip6table_nat nf_nat_ipv6 ip6table_mangle > ip6table_raw ip6table_security iptable_nat nf_nat_ipv4 nf_nat > nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw > iptable_security ebtable_filter ebtables ip6table_filter ip6_tables > sunrpc sb_edac crct10dif_pclmul crc32_pclmul ghash_clmulni_intel > kvm_intel kvm ppdev irqbypass parport_pc joydev virtio_balloon > pcc_cpufreq i2c_piix4 pcspkr parport xfs libcrc32c cirrus > drm_kms_helper ttm drm e1000 crc32c_intel virtio_blk ata_generic > floppy serio_raw pata_acpi qemu_fw_cfg > [ 183.799984] CPU: 14 PID: 121 Comm: kworker/14:1 Tainted: G D > 5.0.0-rc7-next-20190219-baseline+ #50 > [ 183.801674] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > BIOS Bochs 01/01/2011 > [ 183.803078] Workqueue: events netstamp_clear > [ 183.803873] RIP: 0010:smp_call_function_many+0x206/0x260 > [ 183.804847] Code: e8 0f 97 7c 00 3b 05 bd d1 1e 01 0f 83 7c fe ff > ff 48 63 d0 48 8b 4d 00 48 03 0c d5 80 28 18 9e 8b 51 18 83 e2 01 74 > 0a f3 90 <8b> 51 18 83 e2 01 75 f6 eb c7 0f b6 4c 24 0c 48 83 c4 10 89 > ef 5b > [ 183.808273] RSP: 0018:ffffb9a8c35a3d38 EFLAGS: 00000202 ORIG_RAX: > ffffffffffffff13 > [ 183.809662] RAX: 0000000000000000 RBX: ffffa000dfba9d88 RCX: > ffffa000df8301c0 > [ 183.810971] RDX: 0000000000000001 RSI: 0000000000000100 RDI: > ffffa000dfba9d88 > [ 183.812268] RBP: ffffa000dfba9d80 R08: 0000000000000000 R09: > 0000000000003fff > [ 183.813582] R10: 0000000000000000 R11: 000000000000000f R12: > ffffffff9d02f690 > [ 183.814884] R13: 0000000000000000 R14: ffffa000dfba9da8 R15: > 0000000000000100 > [ 183.816195] FS: 0000000000000000(0000) GS:ffffa000dfb80000(0000) > knlGS:0000000000000000 > [ 183.817673] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 183.818729] CR2: 00007ffff704b080 CR3: 0000000814c48001 CR4: > 0000000000160ee0 > [ 183.820038] Call Trace: > [ 183.820510] ? netif_receive_skb_list+0x68/0x4a0 > [ 183.821367] ? poke_int3_handler+0x40/0x40 > [ 183.822126] ? netif_receive_skb_list+0x69/0x4a0 > [ 183.822975] on_each_cpu+0x28/0x60 > [ 183.823611] ? netif_receive_skb_list+0x68/0x4a0 > [ 183.824467] text_poke_bp+0x68/0xe0 > [ 183.825126] ? netif_receive_skb_list+0x68/0x4a0 > [ 183.825983] __jump_label_transform+0x101/0x140 > [ 183.826829] arch_jump_label_transform+0x26/0x40 > [ 183.827687] __jump_label_update+0x56/0xc0 > [ 183.828456] static_key_enable_cpuslocked+0x57/0x80 > [ 183.829358] static_key_enable+0x16/0x20 > [ 183.830085] process_one_work+0x16c/0x380 > [ 183.830831] worker_thread+0x49/0x3e0 > [ 183.831516] kthread+0xf8/0x130 > [ 183.832106] ? rescuer_thread+0x340/0x340 > [ 183.832848] ? kthread_bind+0x10/0x10 > [ 183.833532] ret_from_fork+0x35/0x40 -- Regards Nitesh
signature.asc
Description: OpenPGP digital signature
