Skip to content
  • Dave Hansen's avatar
    mm/slub.c: fix page->_count corruption (again) · a0320865
    Dave Hansen authored
    Commit abca7c49 ("mm: fix slab->page _count corruption when using
    slub") notes that we can not _set_ a page->counters directly, except
    when using a real double-cmpxchg.  Doing so can lose updates to
    That is an absolute rule:
            You may not *set* page->counters except via a cmpxchg.
    Commit abca7c49
     fixed this for the folks who have the slub
    cmpxchg_double code turned off at compile time, but it left the bad case
    alone.  It can still be reached, and the same bug triggered in two
    1. Turning on slub debugging at runtime, which is available on
       the distro kernels that I looked at.
    2. On 64-bit CPUs with no CMPXCHG16B (some early AMD x86-64
       cpus, evidently)
    There are at least 3 ways we could fix this:
    1. Take all of the exising calls to cmpxchg_double_slab() and
       __cmpxchg_double_slab() and convert them to take an old, new
       and target 'struct page'.
    2. Do (1), but with the newly-introduced 'slub_data'.
    3. Do some magic inside the two cmpxchg...slab() functions to
       pull the counters out of new_counters and only set those
       fields in page->{inuse,frozen,objects}.
    I've done (2) as well, but it's a bunch more code.  This patch is an
    attempt at (3).  This was the most straightforward and foolproof way
    that I could think to do this.
    This would also technically allow us to get rid of the ugly
    #if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \
    in 'struct page', but leaving it alone has the added benefit that
    'counters' stays 'unsigned' instead of 'unsigned long', so all the
    copies that the slub code does stay a bit smaller.
    Signed-off-by: default avatarDave Hansen <>
    Cc: Christoph Lameter <>
    Cc: Pekka Enberg <>
    Cc: Matt Mackall <>
    Cc: Pravin B Shelar <>
    Signed-off-by: default avatarAndrew Morton <>
    Signed-off-by: default avatarLinus Torvalds <>