1. 14 Jun, 2018 1 commit
  2. 24 May, 2018 1 commit
    • Joonsoo Kim's avatar
      Revert "mm/cma: manage the memory of the CMA area by using the ZONE_MOVABLE" · d883c6cf
      Joonsoo Kim authored
      This reverts the following commits that change CMA design in MM.
       3d2054ad ("ARM: CMA: avoid double mapping to the CMA area if CONFIG_HIGHMEM=y")
       1d47a3ec ("mm/cma: remove ALLOC_CMA")
       ("mm/cma: manage the memory of the CMA area by using the ZONE_MOVABLE")
      Ville reported a following error on i386.
        Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
        microcode: microcode updated early to revision 0x4, date = 2013-06-28
        Initializing CPU#0
        Initializing HighMem for node 0 (000377fe:00118000)
        Initializing Movable for node 0 (00000001:00118000)
        BUG: Bad page state in process swapper  pfn:377fe
        page:f53effc0 count:0 mapcount:-127 mapping:00000000 index:0x0
        flags: 0x80000000()
        raw: 80000000 00000000 00000000 ffffff80 00000000 00000100 00000200 00000001
        page dumped because: nonzero mapcount
        Modules linked in:
        CPU: 0 PID: 0 Comm: swapper Not tainted 4.17.0-rc5-elk+ #145
        Hardware name: Dell Inc. Latitude E5410/03VXMC, BIOS A15 07/11/2013
        Call Trace:
      The reason for this error is that the span of MOVABLE_ZONE is extended
      to whole node span for future CMA initialization, and, normal memory is
      wrongly freed here.  I submitted the fix and it seems to work, but,
      another problem happened.
      It's so late time to fix the later problem so I decide to reverting the
      Reported-by: default avatarVille Syrjälä <ville.syrjala@linux.intel.com>
      Acked-by: default avatarLaura Abbott <labbott@redhat.com>
      Acked-by: default avatarMichal Hocko <mhocko@suse.com>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarJoonsoo Kim <iamjoonsoo.kim@lge.com>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
  3. 11 Apr, 2018 2 commits
  4. 06 Apr, 2018 2 commits
    • Mike Rapoport's avatar
    • David Rientjes's avatar
      mm, compaction: drain pcps for zone when kcompactd fails · bc3106b2
      David Rientjes authored
      It's possible for free pages to become stranded on per-cpu pagesets
      (pcps) that, if drained, could be merged with buddy pages on the zone's
      free area to form large order pages, including up to MAX_ORDER.
      Consider a verbose example using the tools/vm/page-types tool at the
      beginning of a ZONE_NORMAL ('B' indicates a buddy page and 'S' indicates
      a slab page).  Pages on pcps do not have any page flags set.
        109954  1       _______S________________________________________________________
        109955  2       __________B_____________________________________________________
        109957  1       ________________________________________________________________
        109958  1       __________B_____________________________________________________
        109959  7       ________________________________________________________________
        109960  1       __________B_____________________________________________________
        109961  9       ________________________________________________________________
        10996a  1       __________B_____________________________________________________
        10996b  3       ________________________________________________________________
        10996e  1       __________B_____________________________________________________
        10996f  1       ________________________________________________________________
        109f8c  1       __________B_____________________________________________________
        109f8d  2       ________________________________________________________________
        109f8f  2       __________B_____________________________________________________
        109f91  f       ________________________________________________________________
        109fa0  1       __________B_____________________________________________________
        109fa1  7       ________________________________________________________________
        109fa8  1       __________B_____________________________________________________
        109fa9  1       ________________________________________________________________
        109faa  1       __________B_____________________________________________________
        109fab  1       _______S________________________________________________________
      The compaction migration scanner is attempting to defragment this memory
      since it is at the beginning of the zone.  It has done so quite well,
      all movable pages have been migrated.  From pfn [0x109955, 0x109fab),
      there are only buddy pages and pages without flags set.
      These pages may be stranded on pcps that could otherwise allow this
      memory to be coalesced if freed back to the zone free area.  It is
      possible that some of these pages may not be on pcps and that something
      has called alloc_pages() and used the memory directly, but we rely on
      the absence of __GFP_MOVABLE in these cases to allocate from
      MIGATE_UNMOVABLE pageblocks to try to keep these MIGRATE_MOVABLE
      pageblocks as free as possible.
      These buddy and pcp pages, spanning 1,621 pages, could be coalesced and
      allow for three transparent hugepages to be dynamically allocated.
      Running the numbers for all such spans on the system, it was found that
      there were over 400 such spans of only buddy pages and pages without
      flags set at the time this /proc/kpageflags sample was collected.
      Without this support, there were _no_ order-9 or order-10 pages free.
      When kcompactd fails to defragment memory such that a cc.order page can
      be allocated, drain all pcps for the zone back to the buddy allocator so
      this stranding cannot occur.  Compaction for that order will
      subsequently be deferred, which acts as a ratelimit on this drain.
      Link: http://lkml.kernel.org/r/alpine.DEB.2.20.1803010340100.88270@chino.kir.corp.google.com
      Signed-off-by: default avatarDavid Rientjes <rientjes@google.com>
      Acked-by: default avatarVlastimil Babka <vbabka@suse.cz>
      Cc: Mel Gorman <mgorman@techsingularity.net>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
  5. 01 Feb, 2018 1 commit
  6. 18 Nov, 2017 5 commits
  7. 02 Nov, 2017 1 commit
    • Greg Kroah-Hartman's avatar
      License cleanup: add SPDX GPL-2.0 license identifier to files with no license · b2441318
      Greg Kroah-Hartman authored
      Many source files in the tree are missing licensing information, which
      makes it harder for compliance tools to determine the correct license.
      By default all files without license information are under the default
      license of the kernel, which is GPL version 2.
      Update the files which contain no license information with the 'GPL-2.0'
      SPDX license identifier.  The SPDX identifier is a legally binding
      shorthand, which can be used instead of the full boiler plate text.
      This patch is based on work done by Thomas Gleixner and Kate Stewart and
      Philippe Ombredanne.
      How this work was done:
      Patches were generated and checked against linux-4.14-rc6 for a subset of
      the use cases:
       - file had no licensing information it it.
       - file was a */uapi/* one with no licensing information in it,
       - file was a */uapi/* one with existing licensing information,
      Further patches will be generated in subsequent months to fix up cases
      where non-standard license headers were used, and references to license
      had to be inferred by heuristics based on keywords.
      The analysis to determine which SPDX License Identifier to be applied to
      a file was done in a spreadsheet of side by side results from of the
      output of two independent scanners (ScanCode & Windriver) producing SPDX
      tag:value files created by Philippe Ombredanne.  Philippe prepared the
      base worksheet, and did an initial spot review of a few 1000 files.
      The 4.13 kernel was the starting point of the analysis with 60,537 files
      assessed.  Kate Stewart did a file by file comparison of the scanner
      results in the spreadsheet to determine which SPDX license identifier(s)
      to be applied to the file. She confirmed any determination that was not
      immediately clear with lawyers working with the Linux Foundation.
      Criteria used to select files for SPDX license identifier tagging was:
       - Files considered eligible had to be source code files.
       - Make and config files were included as candidates if they contained >5
         lines of source
       - File already had some variant of a license header in it (even if <5
      All documentation files were explicitly excluded.
      The following heuristics were used to determine which SPDX license
      identifiers to apply.
       - when both scanners couldn't find any license traces, file was
         considered to have no license information in it, and the top level
         COPYING file license applied.
         For non */uapi/* files that summary was:
         SPDX license identifier                            # files
         GPL-2.0                                              11139
         and resulted in the first patch in this series.
         If that file was a */uapi/* path one, it was "GPL-2.0 WITH
         Linux-syscall-note" otherwise it was "GPL-2.0".  Results of that was:
         SPDX license identifier                            # files
         GPL-2.0 WITH Linux-syscall-note                        930
         and resulted in the second patch in this series.
       - if a file had some form of licensing information in it, and was one
         of the */uapi/* ones, it was denoted with the Linux-syscall-note if
         any GPL family license was found in the file or had no licensing in
         it (per prior point).  Results summary:
         SPDX license identifier                            # files
         GPL-2.0 WITH Linux-syscall-note                       270
         GPL-2.0+ WITH Linux-syscall-note                      169
         ((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause)    21
         ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause)    17
         LGPL-2.1+ WITH Linux-syscall-note                      15
         GPL-1.0+ WITH Linux-syscall-note                       14
         ((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause)    5
         LGPL-2.0+ WITH Linux-syscall-note                       4
         LGPL-2.1 WITH Linux-syscall-note                        3
         ((GPL-2.0 WITH Linux-syscall-note) OR MIT)              3
         ((GPL-2.0 WITH Linux-syscall-note) AND MIT)             1
         and that resulted in the third patch in this series.
       - when the two scanners agreed on the detected license(s), that became
         the concluded license(s).
       - when there was disagreement between the two scanners (one detected a
         license but the other didn't, or they both detected different
         licenses) a manual inspection of the file occurred.
       - In most cases a manual inspection of the information in the file
         resulted in a clear resolution of the license that should apply (and
         which scanner probably needed to revisit its heuristics).
       - When it was not immediately clear, the license identifier was
         confirmed with lawyers working with the Linux Foundation.
       - If there was any question as to the appropriate license identifier,
         the file was flagged for further research and to be revisited later
         in time.
      In total, over 70 hours of logged manual review was done on the
      spreadsheet to determine the SPDX license identifiers to apply to the
      source files by Kate, Philippe, Thomas and, in some cases, confirmation
      by lawyers working with the Linux Foundation.
      Kate also obtained a third independent scan of the 4.13 code base from
      FOSSology, and compared selected files where the other two scanners
      disagreed against that SPDX file, to see if there was new insights.  The
      Windriver scanner is based on an older version of FOSSology in part, so
      they are related.
      Thomas did random spot checks in about 500 files from the spreadsheets
      for the uapi headers and agreed with SPDX license identifier in the
      files he inspected. For the non-uapi files Thomas did random spot checks
      in about 15000 files.
      In initial set of patches against 4.14-rc6, 3 files were found to have
      copy/paste license identifier errors, and have been fixed to reflect the
      correct identifier.
      Additionally Philippe spent 10 hours this week doing a detailed manual
      inspection and review of the 12,461 patched files from the initial patch
      version early this week with:
       - a full scancode scan run, collecting the matched texts, detected
         license ids and scores
       - reviewing anything where there was a license detected (about 500+
         files) to ensure that the applied SPDX license was correct
       - reviewing anything where there was no detection but the patch license
         was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
         SPDX license was correct
      This produced a worksheet with 20 files needing minor correction.  This
      worksheet was then exported into 3 different .csv files for the
      different types of files to be modified.
      These .csv files were then reviewed by Greg.  Thomas wrote a script to
      parse the csv files and add the proper SPDX tag to the file, in the
      format that the file expected.  This script was further refined by Greg
      based on the output to detect more types of files automatically and to
      distinguish between header and source .c files (which need different
      comment types.)  Finally Greg ran the script using the .csv files to
      generate the patches.
      Reviewed-by: default avatarKate Stewart <kstewart@linuxfoundation.org>
      Reviewed-by: default avatarPhilippe Ombredanne <pombredanne@nexb.com>
      Reviewed-by: default avatarThomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarGreg Kroah-Hartman <gregkh@linuxfoundation.org>
  8. 04 Oct, 2017 1 commit
  9. 06 Jul, 2017 1 commit
  10. 09 May, 2017 5 commits
  11. 03 May, 2017 1 commit
  12. 02 Mar, 2017 1 commit
  13. 25 Feb, 2017 1 commit
  14. 23 Feb, 2017 2 commits
  15. 15 Dec, 2016 1 commit
    • Michal Hocko's avatar
      mm, compaction: allow compaction for GFP_NOFS requests · 73e64c51
      Michal Hocko authored
      compaction has been disabled for GFP_NOFS and GFP_NOIO requests since
      the direct compaction was introduced by commit 56de7263 ("mm:
      compaction: direct compact when a high-order allocation fails").  The
      main reason is that the migration of page cache pages might recurse back
      to fs/io layer and we could potentially deadlock.  This is overly
      conservative because all the anonymous memory is migrateable in the
      GFP_NOFS context just fine.  This might be a large portion of the memory
      in many/most workkloads.
      Remove the GFP_NOFS restriction and make sure that we skip all fs pages
      (those with a mapping) while isolating pages to be migrated.  We cannot
      consider clean fs pages because they might need a metadata update so
      only isolate pages without any mapping for nofs requests.
      The effect of this patch will be probably very limited in many/most
      workloads because higher order GFP_NOFS requests are quite rare,
      although different configurations might lead to very different results.
      David Chinner has mentioned a heavy metadata workload with 64kB block
      which to quote him:
      : Unfortunately, there was an era of cargo cult configuration tweaks in the
      : Ceph community that has resulted in a large number of production machines
      : with XFS filesystems configured this way.  And a lot of them store large
      : numbers of small files and run under significant sustained memory
      : pressure.
      : I slowly working towards getting rid of these high order allocations and
      : replacing them with the equivalent number of single page allocations, but
      : I haven't got that (complex) change working yet.
      We can do the following to simulate that workload:
      $ mkfs.xfs -f -n size=64k <dev>
      $ mount <dev> /mnt/scratch
      $ time ./fs_mark  -D  10000  -S0  -n  100000  -s  0  -L  32 \
              -d  /mnt/scratch/0  -d  /mnt/scratch/1 \
              -d  /mnt/scratch/2  -d  /mnt/scratch/3 \
              -d  /mnt/scratch/4  -d  /mnt/scratch/5 \
              -d  /mnt/scratch/6  -d  /mnt/scratch/7 \
              -d  /mnt/scratch/8  -d  /mnt/scratch/9 \
              -d  /mnt/scratch/10  -d  /mnt/scratch/11 \
              -d  /mnt/scratch/12  -d  /mnt/scratch/13 \
              -d  /mnt/scratch/14  -d  /mnt/scratch/15
      and indeed is hammers the system with many high order GFP_NOFS requests as
      per a simle tracepoint during the load:
      $ echo '!(gfp_flags & 0x80) && (gfp_flags &0x400000)' > $TRACE_MNT/events/kmem/mm_page_alloc/filter
      I am getting
      5287609 order=0
           37 order=1
      1594905 order=2
      3048439 order=3
      6699207 order=4
        66645 order=5
      My testing was done in a kvm guest so performance numbers should be
      taken with a grain of salt but there seems to be a difference when the
      patch is applied:
      * Original kernel
      FSUse%        Count         Size    Files/sec     App Overhead
           1      1600000            0       4300.1         20745838
           3      3200000            0       4239.9         23849857
           5      4800000            0       4243.4         25939543
           6      6400000            0       4248.4         19514050
           8      8000000            0       4262.1         20796169
           9      9600000            0       4257.6         21288675
          11     11200000            0       4259.7         19375120
          13     12800000            0       4220.7         22734141
          14     14400000            0       4238.5         31936458
          16     16000000            0       4231.5         23409901
          18     17600000            0       4045.3         23577700
          19     19200000            0       2783.4         58299526
          21     20800000            0       2678.2         40616302
          23     22400000            0       2693.5         83973996
      and xfs complaining about memory allocation not making progress
      [ 2304.372647] XFS: fs_mark(3289) possible memory allocation deadlock size 65624 in kmem_alloc (mode:0x2408240)
      [ 2304.443323] XFS: fs_mark(3285) possible memory allocation deadlock size 65728 in kmem_alloc (mode:0x2408240)
      [ 4796.772477] XFS: fs_mark(3424) possible memory allocation deadlock size 46936 in kmem_alloc (mode:0x2408240)
      [ 4796.775329] XFS: fs_mark(3423) possible memory allocation deadlock size 51416 in kmem_alloc (mode:0x2408240)
      [ 4797.388808] XFS: fs_mark(3424) possible memory allocation deadlock size 65728 in kmem_alloc (mode:0x2408240)
      * Patched kernel
      FSUse%        Count         Size    Files/sec     App Overhead
           1      1600000            0       4289.1         19243934
           3      3200000            0       4241.6         32828865
           5      4800000            0       4248.7         32884693
           6      6400000            0       4314.4         19608921
           8      8000000            0       4269.9         24953292
           9      9600000            0       4270.7         33235572
          11     11200000            0       4346.4         40817101
          13     12800000            0       4285.3         29972397
          14     14400000            0       4297.2         20539765
          16     16000000            0       4219.6         18596767
          18     17600000            0       4273.8         49611187
          19     19200000            0       4300.4         27944451
          21     20800000            0       4270.6         22324585
          22     22400000            0       4317.6         22650382
          24     24000000            0       4065.2         22297964
      So the dropdown at Count 19200000 didn't happen and there was only a
      single warning about allocation not making progress
      [ 3063.815003] XFS: fs_mark(3272) possible memory allocation deadlock size 65624 in kmem_alloc (mode:0x2408240)
      This suggests that the patch has helped even though there is not all that
      much of anonymous memory as the workload mostly generates fs metadata.  I
      assume the success rate would be higher with more anonymous memory which
      should be the case in many workloads.
      [akpm@linux-foundation.org: fix comment]
      Link: http://lkml.kernel.org/r/20161012114721.31853-1-mhocko@kernel.org
      Signed-off-by: default avatarMichal Hocko <mhocko@suse.com>
      Acked-by: default avatarVlastimil Babka <vbabka@suse.cz>
      Cc: Mel Gorman <mgorman@techsingularity.net>
      Cc: Joonsoo Kim <js1304@gmail.com>
      Cc: Dave Chinner <david@fromorbit.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
  16. 13 Dec, 2016 1 commit
    • Ming Ling's avatar
      mm, compaction: fix NR_ISOLATED_* stats for pfn based migration · 6afcf8ef
      Ming Ling authored
      Since commit bda807d4 ("mm: migrate: support non-lru movable page
      migration") isolate_migratepages_block) can isolate !PageLRU pages which
      would acct_isolated account as NR_ISOLATED_*.  Accounting these non-lru
      pages NR_ISOLATED_{ANON,FILE} doesn't make any sense and it can misguide
      heuristics based on those counters such as pgdat_reclaimable_pages resp.
      too_many_isolated which would lead to unexpected stalls during the
      direct reclaim without any good reason.  Note that
      __alloc_contig_migrate_range can isolate a lot of pages at once.
      On mobile devices such as 512M ram android Phone, it may use a big zram
      swap.  In some cases zram(zsmalloc) uses too many non-lru but
      migratedable pages, such as:
            MemTotal: 468148 kB
            Normal free:5620kB
            Free swap:4736kB
            Total swap:409596kB
            ZRAM: 164616kB(zsmalloc non-lru pages)
      Fix this by only accounting lru pages to NR_ISOLATED_* in
      isolate_migratepages_block right after they were isolated and we still
      know they were on LRU.  Drop acct_isolated because it is called after
      the fact and we've lost that information.  Batching per-cpu counter
      doesn't make much improvement anyway.  Also make sure that we uncharge
      only LRU pages when putting them back on the LRU in
      putback_movable_pages resp.  when unmap_and_move migrates the page.
      [mhocko@suse.com: replace acct_isolated() with direct counting]
      Fixes: bda807d4 ("mm: migrate: support non-lru movable page migration")
      Link: http://lkml.kernel.org/r/20161019080240.9682-1-mhocko@kernel.org
      Signed-off-by: default avatarMing Ling <ming.ling@spreadtrum.com>
      Signed-off-by: default avatarMichal Hocko <mhocko@suse.com>
      Acked-by: default avatarMinchan Kim <minchan@kernel.org>
      Acked-by: default avatarVlastimil Babka <vbabka@suse.cz>
      Cc: Mel Gorman <mgorman@suse.de>
      Cc: Joonsoo Kim <js1304@gmail.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
  17. 01 Dec, 2016 1 commit
  18. 08 Oct, 2016 12 commits