1. 02 Jul, 2011 4 commits
  2. 03 Jun, 2011 1 commit
  3. 26 May, 2011 1 commit
  4. 25 May, 2011 1 commit
  5. 21 May, 2011 1 commit
  6. 17 May, 2011 3 commits
  7. 07 May, 2011 1 commit
    • Christoph Lameter's avatar
      slub: Remove CONFIG_CMPXCHG_LOCAL ifdeffery · 1759415e
      Christoph Lameter authored
      
      
      Remove the #ifdefs. This means that the irqsafe_cpu_cmpxchg_double() is used
      everywhere.
      
      There may be performance implications since:
      
      A. We now have to manage a transaction ID for all arches
      
      B. The interrupt holdoff for arches not supporting CONFIG_CMPXCHG_LOCAL is reduced
      to a very short irqoff section.
      
      There are no multiple irqoff/irqon sequences as a result of this change. Even in the fallback
      case we only have to do one disable and enable like before.
      
      Signed-off-by: default avatarChristoph Lameter <cl@linux.com>
      Signed-off-by: default avatarPekka Enberg <penberg@kernel.org>
      1759415e
  8. 04 May, 2011 1 commit
    • Thomas Gleixner's avatar
      slub: Fix the lockless code on 32-bit platforms with no 64-bit cmpxchg · 30106b8c
      Thomas Gleixner authored
      
      
      The SLUB allocator use of the cmpxchg_double logic was wrong: it
      actually needs the irq-safe one.
      
      That happens automatically when we use the native unlocked 'cmpxchg8b'
      instruction, but when compiling the kernel for older x86 CPUs that do
      not support that instruction, we fall back to the generic emulation
      code.
      
      And if you don't specify that you want the irq-safe version, the generic
      code ends up just open-coding the cmpxchg8b equivalent without any
      protection against interrupts or preemption.  Which definitely doesn't
      work for SLUB.
      
      This was reported by Werner Landgraf <w.landgraf@ru.ru>, who saw
      instability with his distro-kernel that was compiled to support pretty
      much everything under the sun.  Most big Linux distributions tend to
      compile for PPro and later, and would never have noticed this problem.
      
      This also fixes the prototypes for the irqsafe cmpxchg_double functions
      to use 'bool' like they should.
      
      [ Btw, that whole "generic code defaults to no protection" design just
        sounds stupid - if the code needs no protection, there is no reason to
        use "cmpxchg_double" to begin with.  So we should probably just remove
        the unprotected version entirely as pointless.   - Linus ]
      
      Signed-off-by: default avatarThomas Gleixner <tglx@linutronix.de>
      Reported-and-tested-by: default avatarwerner <w.landgraf@ru.ru>
      Acked-and-tested-by: default avatarIngo Molnar <mingo@elte.hu>
      Acked-by: default avatarChristoph Lameter <cl@linux.com>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Jens Axboe <axboe@kernel.dk>
      Cc: Tejun Heo <tj@kernel.org>
      Link: http://lkml.kernel.org/r/alpine.LFD.2.02.1105041539050.3005@ionos
      
      
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      30106b8c
  9. 17 Apr, 2011 5 commits
  10. 12 Apr, 2011 1 commit
  11. 31 Mar, 2011 1 commit
  12. 24 Mar, 2011 2 commits
  13. 22 Mar, 2011 2 commits
  14. 20 Mar, 2011 1 commit
  15. 11 Mar, 2011 4 commits
    • Lai Jiangshan's avatar
      slub,rcu: don't assume the size of struct rcu_head · da9a638c
      Lai Jiangshan authored
      
      
      The size of struct rcu_head may be changed. When it becomes larger,
      it will pollute the page array.
      
      We reserve some some bytes for struct rcu_head when a slab
      is allocated in this situation.
      
      Changed from V1:
      	use VM_BUG_ON instead BUG_ON
      
      Acked-by: default avatarChristoph Lameter <cl@linux.com>
      Signed-off-by: default avatarLai Jiangshan <laijs@cn.fujitsu.com>
      Signed-off-by: default avatarPekka Enberg <penberg@kernel.org>
      da9a638c
    • Lai Jiangshan's avatar
      slub: automatically reserve bytes at the end of slab · ab9a0f19
      Lai Jiangshan authored
      
      
      There is no "struct" for slub's slab, it shares with struct page.
      But struct page is very small, it is insufficient when we need
      to add some metadata for slab.
      
      So we add a field "reserved" to struct kmem_cache, when a slab
      is allocated, kmem_cache->reserved bytes are automatically reserved
      at the end of the slab for slab's metadata.
      
      Changed from v1:
      	Export the reserved field via sysfs
      
      Acked-by: default avatarChristoph Lameter <cl@linux.com>
      Signed-off-by: default avatarLai Jiangshan <laijs@cn.fujitsu.com>
      Signed-off-by: default avatarPekka Enberg <penberg@kernel.org>
      ab9a0f19
    • Christoph Lameter's avatar
      Lockless (and preemptless) fastpaths for slub · 8a5ec0ba
      Christoph Lameter authored
      
      
      Use the this_cpu_cmpxchg_double functionality to implement a lockless
      allocation algorithm on arches that support fast this_cpu_ops.
      
      Each of the per cpu pointers is paired with a transaction id that ensures
      that updates of the per cpu information can only occur in sequence on
      a certain cpu.
      
      A transaction id is a "long" integer that is comprised of an event number
      and the cpu number. The event number is incremented for every change to the
      per cpu state. This means that the cmpxchg instruction can verify for an
      update that nothing interfered and that we are updating the percpu structure
      for the processor where we picked up the information and that we are also
      currently on that processor when we update the information.
      
      This results in a significant decrease of the overhead in the fastpaths. It
      also makes it easy to adopt the fast path for realtime kernels since this
      is lockless and does not require the use of the current per cpu area
      over the critical section. It is only important that the per cpu area is
      current at the beginning of the critical section and at the end.
      
      So there is no need even to disable preemption.
      
      Test results show that the fastpath cycle count is reduced by up to ~ 40%
      (alloc/free test goes from ~140 cycles down to ~80). The slowpath for kfree
      adds a few cycles.
      
      Sadly this does nothing for the slowpath which is where the main issues with
      performance in slub are but the best case performance rises significantly.
      (For that see the more complex slub patches that require cmpxchg_double)
      
      Kmalloc: alloc/free test
      
      Before:
      
      10000 times kmalloc(8)/kfree -> 134 cycles
      10000 times kmalloc(16)/kfree -> 152 cycles
      10000 times kmalloc(32)/kfree -> 144 cycles
      10000 times kmalloc(64)/kfree -> 142 cycles
      10000 times kmalloc(128)/kfree -> 142 cycles
      10000 times kmalloc(256)/kfree -> 132 cycles
      10000 times kmalloc(512)/kfree -> 132 cycles
      10000 times kmalloc(1024)/kfree -> 135 cycles
      10000 times kmalloc(2048)/kfree -> 135 cycles
      10000 times kmalloc(4096)/kfree -> 135 cycles
      10000 times kmalloc(8192)/kfree -> 144 cycles
      10000 times kmalloc(16384)/kfree -> 754 cycles
      
      After:
      
      10000 times kmalloc(8)/kfree -> 78 cycles
      10000 times kmalloc(16)/kfree -> 78 cycles
      10000 times kmalloc(32)/kfree -> 82 cycles
      10000 times kmalloc(64)/kfree -> 88 cycles
      10000 times kmalloc(128)/kfree -> 79 cycles
      10000 times kmalloc(256)/kfree -> 79 cycles
      10000 times kmalloc(512)/kfree -> 85 cycles
      10000 times kmalloc(1024)/kfree -> 82 cycles
      10000 times kmalloc(2048)/kfree -> 82 cycles
      10000 times kmalloc(4096)/kfree -> 85 cycles
      10000 times kmalloc(8192)/kfree -> 82 cycles
      10000 times kmalloc(16384)/kfree -> 706 cycles
      
      Kmalloc: Repeatedly allocate then free test
      
      Before:
      
      10000 times kmalloc(8) -> 211 cycles kfree -> 113 cycles
      10000 times kmalloc(16) -> 174 cycles kfree -> 115 cycles
      10000 times kmalloc(32) -> 235 cycles kfree -> 129 cycles
      10000 times kmalloc(64) -> 222 cycles kfree -> 120 cycles
      10000 times kmalloc(128) -> 343 cycles kfree -> 139 cycles
      10000 times kmalloc(256) -> 827 cycles kfree -> 147 cycles
      10000 times kmalloc(512) -> 1048 cycles kfree -> 272 cycles
      10000 times kmalloc(1024) -> 2043 cycles kfree -> 528 cycles
      10000 times kmalloc(2048) -> 4002 cycles kfree -> 571 cycles
      10000 times kmalloc(4096) -> 7740 cycles kfree -> 628 cycles
      10000 times kmalloc(8192) -> 8062 cycles kfree -> 850 cycles
      10000 times kmalloc(16384) -> 8895 cycles kfree -> 1249 cycles
      
      After:
      
      10000 times kmalloc(8) -> 190 cycles kfree -> 129 cycles
      10000 times kmalloc(16) -> 76 cycles kfree -> 123 cycles
      10000 times kmalloc(32) -> 126 cycles kfree -> 124 cycles
      10000 times kmalloc(64) -> 181 cycles kfree -> 128 cycles
      10000 times kmalloc(128) -> 310 cycles kfree -> 140 cycles
      10000 times kmalloc(256) -> 809 cycles kfree -> 165 cycles
      10000 times kmalloc(512) -> 1005 cycles kfree -> 269 cycles
      10000 times kmalloc(1024) -> 1999 cycles kfree -> 527 cycles
      10000 times kmalloc(2048) -> 3967 cycles kfree -> 570 cycles
      10000 times kmalloc(4096) -> 7658 cycles kfree -> 637 cycles
      10000 times kmalloc(8192) -> 8111 cycles kfree -> 859 cycles
      10000 times kmalloc(16384) -> 8791 cycles kfree -> 1173 cycles
      
      Signed-off-by: default avatarChristoph Lameter <cl@linux.com>
      Signed-off-by: default avatarPekka Enberg <penberg@kernel.org>
      8a5ec0ba
    • Christoph Lameter's avatar
      slub: Get rid of slab_free_hook_irq() · d3f661d6
      Christoph Lameter authored
      
      
      The following patch will make the fastpaths lockless and will no longer
      require interrupts to be disabled. Calling the free hook with irq disabled
      will no longer be possible.
      
      Move the slab_free_hook_irq() logic into slab_free_hook. Only disable
      interrupts if the features are selected that require callbacks with
      interrupts off and reenable after calls have been made.
      
      Signed-off-by: default avatarChristoph Lameter <cl@linux.com>
      Signed-off-by: default avatarPekka Enberg <penberg@kernel.org>
      d3f661d6
  16. 27 Feb, 2011 1 commit
  17. 23 Feb, 2011 1 commit
  18. 23 Jan, 2011 1 commit
  19. 14 Jan, 2011 1 commit
  20. 11 Jan, 2011 1 commit
  21. 07 Jan, 2011 1 commit
  22. 04 Dec, 2010 2 commits
    • Tero Roponen's avatar
      slub: Fix a crash during slabinfo -v · 37d57443
      Tero Roponen authored
      Commit f7cb1933
      
       ("SLUB: Pass active
      and inactive redzone flags instead of boolean to debug functions")
      missed two instances of check_object(). This caused a lot of warnings
      during 'slabinfo -v' finally leading to a crash:
      
        BUG ext4_xattr: Freepointer corrupt
        ...
        BUG buffer_head: Freepointer corrupt
        ...
        BUG ext4_alloc_context: Freepointer corrupt
        ...
        ...
        BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
        IP: [<ffffffff810a291f>] file_sb_list_del+0x1c/0x35
        PGD 79d78067 PUD 79e67067 PMD 0
        Oops: 0002 [#1] SMP
        last sysfs file: /sys/kernel/slab/:t-0000192/validate
      
      This patch fixes the problem by converting the two missed instances.
      
      Acked-by: default avatarChristoph Lameter <cl@linux.com>
      Signed-off-by: default avatarTero Roponen <tero.roponen@gmail.com>
      Signed-off-by: default avatarPekka Enberg <penberg@kernel.org>
      37d57443
    • Tero Roponen's avatar
      slub: Fix a crash during slabinfo -v · 8165984a
      Tero Roponen authored
      Commit f7cb1933
      
       ("SLUB: Pass active
      and inactive redzone flags instead of boolean to debug functions")
      missed two instances of check_object(). This caused a lot of warnings
      during 'slabinfo -v' finally leading to a crash:
      
        BUG ext4_xattr: Freepointer corrupt
        ...
        BUG buffer_head: Freepointer corrupt
        ...
        BUG ext4_alloc_context: Freepointer corrupt
        ...
        ...
        BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
        IP: [<ffffffff810a291f>] file_sb_list_del+0x1c/0x35
        PGD 79d78067 PUD 79e67067 PMD 0
        Oops: 0002 [#1] SMP
        last sysfs file: /sys/kernel/slab/:t-0000192/validate
      
      This patch fixes the problem by converting the two missed instances.
      
      Acked-by: default avatarChristoph Lameter <cl@linux.com>
      Signed-off-by: default avatarTero Roponen <tero.roponen@gmail.com>
      Signed-off-by: default avatarPekka Enberg <penberg@kernel.org>
      8165984a
  23. 14 Nov, 2010 1 commit
  24. 06 Nov, 2010 2 commits
    • Pavel Emelyanov's avatar
      slub: Fix slub_lock down/up imbalance · 98072e4d
      Pavel Emelyanov authored
      There are two places, that do not release the slub_lock.
      
      Respective bugs were introduced by sysfs changes ab4d5ed5 (slub: Enable
      sysfs support for !CONFIG_SLUB_DEBUG) and 2bce6485
      
       ( slub: Allow removal
      of slab caches during boot).
      
      Acked-by: default avatarChristoph Lameter <cl@linux.com>
      Signed-off-by: default avatarPavel Emelyanov <xemul@openvz.org>
      Signed-off-by: default avatarPekka Enberg <penberg@kernel.org>
      98072e4d
    • Richard Kennedy's avatar
      slub tracing: move trace calls out of always inlined functions to reduce kernel code size · 4a92379b
      Richard Kennedy authored
      
      
      Having the trace calls defined in the always inlined kmalloc functions
      in include/linux/slub_def.h causes a lot of code duplication as the
      trace functions get instantiated for each kamalloc call site. This can
      simply be removed by pushing the trace calls down into the functions in
      slub.c.
      
      On my x86_64 built this patch shrinks the code size of the kernel by
      approx 36K and also shrinks the code size of many modules -- too many to
      list here ;)
      
      size vmlinux (2.6.36) reports
             text        data     bss     dec     hex filename
          5410611	 743172	 828928	6982711	 6a8c37	vmlinux
          5373738	 744244	 828928	6946910	 6a005e	vmlinux + patch
      
      The resulting kernel has had some testing & kmalloc trace still seems to
      work.
      
      This patch
      - moves trace_kmalloc out of the inlined kmalloc() and pushes it down
      into kmem_cache_alloc_trace() so this it only get instantiated once.
      
      - rename kmem_cache_alloc_notrace()  to kmem_cache_alloc_trace() to
      indicate that now is does have tracing. (maybe this would better being
      called something like kmalloc_kmem_cache ?)
      
      - adds a new function kmalloc_order() to handle allocation and tracing
      of large allocations of page order.
      
      - removes tracing from the inlined kmalloc_large() replacing them with a
      call to kmalloc_order();
      
      - move tracing out of inlined kmalloc_node() and pushing it down into
      kmem_cache_alloc_node_trace
      
      - rename kmem_cache_alloc_node_notrace() to
      kmem_cache_alloc_node_trace()
      
      - removes the include of trace/events/kmem.h from slub_def.h.
      
      v2
      - keep kmalloc_order_trace inline when !CONFIG_TRACE
      
      Signed-off-by: default avatarRichard Kennedy <richard@rsk.demon.co.uk>
      Signed-off-by: default avatarPekka Enberg <penberg@kernel.org>
      4a92379b