• Michal Hocko's avatar
    mm, memory_hotplug: make has_unmovable_pages more robust · 15c30bc0
    Michal Hocko authored
    Oscar has reported:
    : Due to an unfortunate setting with movablecore, memblocks containing bootmem
    : memory (pages marked by get_page_bootmem()) ended up marked in zone_movable.
    : So while trying to remove that memory, the system failed in do_migrate_range
    : and __offline_pages never returned.
    :
    : This can be reproduced by running
    : qemu-system-x86_64 -m 6G,slots=8,maxmem=8G -numa node,mem=4096M -numa node,mem=2048M
    : and movablecore=4G kernel command line
    :
    : linux kernel: BIOS-provided physical RAM map:
    : linux kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
    : linux kernel: BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
    : linux kernel: BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
    : linux kernel: BIOS-e820: [mem 0x0000000000100000-0x00000000bffdffff] usable
    : linux kernel: BIOS-e820: [mem 0x00000000bffe0000-0x00000000bfffffff] reserved
    : linux kernel: BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
    : linux kernel: BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
    : linux kernel: BIOS-e820: [mem 0x0000000100000000-0x00000001bfffffff] usable
    : linux kernel: NX (Execute Disable) protection: active
    : linux kernel: SMBIOS 2.8 present.
    : linux kernel: DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.0.0-prebuilt.qemu-project.org
    : linux kernel: Hypervisor detected: KVM
    : linux kernel: e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
    : linux kernel: e820: remove [mem 0x000a0000-0x000fffff] usable
    : linux kernel: last_pfn = 0x1c0000 max_arch_pfn = 0x400000000
    :
    : linux kernel: SRAT: PXM 0 -> APIC 0x00 -> Node 0
    : linux kernel: SRAT: PXM 1 -> APIC 0x01 -> Node 1
    : linux kernel: ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff]
    : linux kernel: ACPI: SRAT: Node 0 PXM 0 [mem 0x00100000-0xbfffffff]
    : linux kernel: ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x13fffffff]
    : linux kernel: ACPI: SRAT: Node 1 PXM 1 [mem 0x140000000-0x1bfffffff]
    : linux kernel: ACPI: SRAT: Node 0 PXM 0 [mem 0x1c0000000-0x43fffffff] hotplug
    : linux kernel: NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0xbfffffff] -> [mem 0x0
    : linux kernel: NUMA: Node 0 [mem 0x00000000-0xbfffffff] + [mem 0x100000000-0x13fffffff] -> [mem 0
    : linux kernel: NODE_DATA(0) allocated [mem 0x13ffd6000-0x13fffffff]
    : linux kernel: NODE_DATA(1) allocated [mem 0x1bffd3000-0x1bfffcfff]
    :
    : zoneinfo shows that the zone movable is placed into both numa nodes:
    : Node 0, zone  Movable
    :   pages free     160140
    :         min      1823
    :         low      2278
    :         high     2733
    :         spanned  262144
    :         present  262144
    :         managed  245670
    : Node 1, zone  Movable
    :   pages free     448427
    :         min      3827
    :         low      4783
    :         high     5739
    :         spanned  524288
    :         present  524288
    :         managed  515766
    
    Note how only Node 0 has a hutplugable memory region which would rule it
    out from the early memblock allocations (most likely memmap).  Node1
    will surely contain memmaps on the same node and those would prevent
    offlining to succeed.  So this is arguably a configuration issue.
    Although one could argue that we should be more clever and rule early
    allocations from the zone movable.  This would be correct but probably
    not worth the effort considering what a hack movablecore is.
    
    Anyway, We could do better for those cases though.  We rely on
    start_isolate_page_range resp.  has_unmovable_pages to do their job.
    The first one isolates the whole range to be offlined so that we do not
    allocate from it anymore and the later makes sure we are not stumbling
    over non-migrateable pages.
    
    has_unmovable_pages is overly optimistic, however.  It doesn't check all
    the pages if we are withing zone_movable because we rely that those
    pages will be always migrateable.  As it turns out we are still not
    perfect there.  While bootmem pages in zonemovable sound like a clear
    bug which should be fixed let's remove the optimization for now and warn
    if we encounter unmovable pages in zone_movable in the meantime.  That
    should help for now at least.
    
    Btw.  this wasn't a real problem until commit 72b39cfc ("mm,
    memory_hotplug: do not fail offlining too early") because we used to
    have a small number of retries and then failed.  This turned out to be
    too fragile though.
    
    Link: http://lkml.kernel.org/r/20180523125555.30039-2-mhocko@kernel.org
    
    Signed-off-by: default avatarMichal Hocko <mhocko@suse.com>
    Reported-by: default avatarOscar Salvador <osalvador@techadventures.net>
    Tested-by: default avatarOscar Salvador <osalvador@techadventures.net>
    Reviewed-by: default avatarPavel Tatashin <pasha.tatashin@oracle.com>
    Cc: Vlastimil Babka <vbabka@suse.cz>
    Cc: Reza Arbab <arbab@linux.vnet.ibm.com>
    Cc: Igor Mammedov <imammedo@redhat.com>
    Cc: Vitaly Kuznetsov <vkuznets@redhat.com>
    Cc: Anshuman Khandual <khandual@linux.vnet.ibm.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    15c30bc0