1. 11 Jan, 2018 1 commit
  2. 02 Nov, 2017 1 commit
    • Greg Kroah-Hartman's avatar
      License cleanup: add SPDX GPL-2.0 license identifier to files with no license · b2441318
      Greg Kroah-Hartman authored
      
      
      Many source files in the tree are missing licensing information, which
      makes it harder for compliance tools to determine the correct license.
      
      By default all files without license information are under the default
      license of the kernel, which is GPL version 2.
      
      Update the files which contain no license information with the 'GPL-2.0'
      SPDX license identifier.  The SPDX identifier is a legally binding
      shorthand, which can be used instead of the full boiler plate text.
      
      This patch is based on work done by Thomas Gleixner and Kate Stewart and
      Philippe Ombredanne.
      
      How this work was done:
      
      Patches were generated and checked against linux-4.14-rc6 for a subset of
      the use cases:
       - file had no licensing information it it.
       - file was a */uapi/* one with no licensing information in it,
       - file was a */uapi/* one with existing licensing information,
      
      Further patches will be generated in subsequent months to fix up cases
      where non-standard license headers were used, and references to license
      had to be inferred by heuristics based on keywords.
      
      The analysis to determine which SPDX License Identifier to be applied to
      a file was done in a spreadsheet of side by side results from of the
      output of two independent scanners (ScanCode & Windriver) producing SPDX
      tag:value files created by Philippe Ombredanne.  Philippe prepared the
      base worksheet, and did an initial spot review of a few 1000 files.
      
      The 4.13 kernel was the starting point of the analysis with 60,537 files
      assessed.  Kate Stewart did a file by file comparison of the scanner
      results in the spreadsheet to determine which SPDX license identifier(s)
      to be applied to the file. She confirmed any determination that was not
      immediately clear with lawyers working with the Linux Foundation.
      
      Criteria used to select files for SPDX license identifier tagging was:
       - Files considered eligible had to be source code files.
       - Make and config files were included as candidates if they contained >5
         lines of source
       - File already had some variant of a license header in it (even if <5
         lines).
      
      All documentation files were explicitly excluded.
      
      The following heuristics were used to determine which SPDX license
      identifiers to apply.
      
       - when both scanners couldn't find any license traces, file was
         considered to have no license information in it, and the top level
         COPYING file license applied.
      
         For non */uapi/* files that summary was:
      
         SPDX license identifier                            # files
         ---------------------------------------------------|-------
         GPL-2.0                                              11139
      
         and resulted in the first patch in this series.
      
         If that file was a */uapi/* path one, it was "GPL-2.0 WITH
         Linux-syscall-note" otherwise it was "GPL-2.0".  Results of that was:
      
         SPDX license identifier                            # files
         ---------------------------------------------------|-------
         GPL-2.0 WITH Linux-syscall-note                        930
      
         and resulted in the second patch in this series.
      
       - if a file had some form of licensing information in it, and was one
         of the */uapi/* ones, it was denoted with the Linux-syscall-note if
         any GPL family license was found in the file or had no licensing in
         it (per prior point).  Results summary:
      
         SPDX license identifier                            # files
         ---------------------------------------------------|------
         GPL-2.0 WITH Linux-syscall-note                       270
         GPL-2.0+ WITH Linux-syscall-note                      169
         ((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause)    21
         ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause)    17
         LGPL-2.1+ WITH Linux-syscall-note                      15
         GPL-1.0+ WITH Linux-syscall-note                       14
         ((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause)    5
         LGPL-2.0+ WITH Linux-syscall-note                       4
         LGPL-2.1 WITH Linux-syscall-note                        3
         ((GPL-2.0 WITH Linux-syscall-note) OR MIT)              3
         ((GPL-2.0 WITH Linux-syscall-note) AND MIT)             1
      
         and that resulted in the third patch in this series.
      
       - when the two scanners agreed on the detected license(s), that became
         the concluded license(s).
      
       - when there was disagreement between the two scanners (one detected a
         license but the other didn't, or they both detected different
         licenses) a manual inspection of the file occurred.
      
       - In most cases a manual inspection of the information in the file
         resulted in a clear resolution of the license that should apply (and
         which scanner probably needed to revisit its heuristics).
      
       - When it was not immediately clear, the license identifier was
         confirmed with lawyers working with the Linux Foundation.
      
       - If there was any question as to the appropriate license identifier,
         the file was flagged for further research and to be revisited later
         in time.
      
      In total, over 70 hours of logged manual review was done on the
      spreadsheet to determine the SPDX license identifiers to apply to the
      source files by Kate, Philippe, Thomas and, in some cases, confirmation
      by lawyers working with the Linux Foundation.
      
      Kate also obtained a third independent scan of the 4.13 code base from
      FOSSology, and compared selected files where the other two scanners
      disagreed against that SPDX file, to see if there was new insights.  The
      Windriver scanner is based on an older version of FOSSology in part, so
      they are related.
      
      Thomas did random spot checks in about 500 files from the spreadsheets
      for the uapi headers and agreed with SPDX license identifier in the
      files he inspected. For the non-uapi files Thomas did random spot checks
      in about 15000 files.
      
      In initial set of patches against 4.14-rc6, 3 files were found to have
      copy/paste license identifier errors, and have been fixed to reflect the
      correct identifier.
      
      Additionally Philippe spent 10 hours this week doing a detailed manual
      inspection and review of the 12,461 patched files from the initial patch
      version early this week with:
       - a full scancode scan run, collecting the matched texts, detected
         license ids and scores
       - reviewing anything where there was a license detected (about 500+
         files) to ensure that the applied SPDX license was correct
       - reviewing anything where there was no detection but the patch license
         was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
         SPDX license was correct
      
      This produced a worksheet with 20 files needing minor correction.  This
      worksheet was then exported into 3 different .csv files for the
      different types of files to be modified.
      
      These .csv files were then reviewed by Greg.  Thomas wrote a script to
      parse the csv files and add the proper SPDX tag to the file, in the
      format that the file expected.  This script was further refined by Greg
      based on the output to detect more types of files automatically and to
      distinguish between header and source .c files (which need different
      comment types.)  Finally Greg ran the script using the .csv files to
      generate the patches.
      
      Reviewed-by: default avatarKate Stewart <kstewart@linuxfoundation.org>
      Reviewed-by: default avatarPhilippe Ombredanne <pombredanne@nexb.com>
      Reviewed-by: default avatarThomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarGreg Kroah-Hartman <gregkh@linuxfoundation.org>
      b2441318
  3. 01 Oct, 2017 1 commit
    • Theodore Ts'o's avatar
      ext4: retry allocations conservatively · 68fd9750
      Theodore Ts'o authored
      
      
      Now that we no longer try to reserve metadata blocks for delayed
      allocations (which tended to overestimate the required number of
      blocks significantly), we really don't need retry allocations when the
      disk is very full as aggressively any more.
      
      The only time when it makes sense to retry an allocation is if we have
      freshly deleted blocks that will only become available after a
      transaction commit.  And if we lose that race, it's not worth it to
      try more than once.
      
      Signed-off-by: default avatarTheodore Ts'o <tytso@mit.edu>
      68fd9750
  4. 06 Jul, 2016 1 commit
    • Theodore Ts'o's avatar
      ext4: validate s_reserved_gdt_blocks on mount · 5b9554dc
      Theodore Ts'o authored
      
      
      If s_reserved_gdt_blocks is extremely large, it's possible for
      ext4_init_block_bitmap(), which is called when ext4 sets up an
      uninitialized block bitmap, to corrupt random kernel memory.  Add the
      same checks which e2fsck has --- it must never be larger than
      blocksize / sizeof(__u32) --- and then add a backup check in
      ext4_init_block_bitmap() in case the superblock gets modified after
      the file system is mounted.
      
      Reported-by: default avatarVegard Nossum <vegard.nossum@oracle.com>
      Signed-off-by: default avatarTheodore Ts'o <tytso@mit.edu>
      Cc: stable@vger.kernel.org
      5b9554dc
  5. 26 Jun, 2016 1 commit
  6. 07 Jun, 2016 1 commit
  7. 13 May, 2016 1 commit
    • Jan Kara's avatar
      ext4: fix race in transient ENOSPC detection · dbc427ce
      Jan Kara authored
      
      
      When there are blocks to free in the running transaction, block
      allocator can return ENOSPC although the filesystem has some blocks to
      free. We use ext4_should_retry_alloc() to force commit of the current
      transaction and return whether anything was committed so that it makes
      sense to retry the allocation. However the transaction may get committed
      after block allocation fails but before we call
      ext4_should_retry_alloc(). So ext4_should_retry_alloc() returns false
      because there is nothing to commit and we wrongly return ENOSPC.
      
      Fix the race by unconditionally returning 1 from ext4_should_retry_alloc()
      when we tried to commit a transaction. This should not add any
      unnecessary retries since we had a transaction running a while ago when
      trying to allocate blocks and we want to retry the allocation once that
      transaction has committed anyway.
      
      Signed-off-by: default avatarJan Kara <jack@suse.cz>
      Signed-off-by: default avatarTheodore Ts'o <tytso@mit.edu>
      dbc427ce
  8. 12 Feb, 2016 1 commit
  9. 18 Oct, 2015 1 commit
  10. 17 Oct, 2015 2 commits
  11. 08 Jun, 2015 1 commit
    • Lukas Czerner's avatar
      ext4: verify block bitmap even after fresh initialization · 41e5b7ed
      Lukas Czerner authored
      
      
      If we want to rely on the buffer_verified() flag of the block bitmap
      buffer, we have to set it consistently. However currently if we're
      initializing uninitialized block bitmap in
      ext4_read_block_bitmap_nowait() we're not going to set buffer verified
      at all.
      
      We can do this by simply setting the flag on the buffer, but I think
      it's actually better to run ext4_validate_block_bitmap() to make sure
      that what we did in the ext4_init_block_bitmap() is right.
      
      So run ext4_validate_block_bitmap() even after the block bitmap
      initialization. Also bail out early from ext4_validate_block_bitmap() if
      we see corrupt bitmap, since we already know it's corrupt and we do not
      need to verify that.
      
      Signed-off-by: default avatarLukas Czerner <lczerner@redhat.com>
      Signed-off-by: default avatarTheodore Ts'o <tytso@mit.edu>
      41e5b7ed
  12. 03 Apr, 2015 2 commits
  13. 13 Oct, 2014 1 commit
    • Dmitry Monakhov's avatar
      ext4: move error report out of atomic context in ext4_init_block_bitmap() · aef4885a
      Dmitry Monakhov authored
      
      
      Error report likely result in IO so it is bad idea to do it from
      atomic context.
      
      This patch should fix following issue:
      
      BUG: sleeping function called from invalid context at include/linux/buffer_head.h:349
      in_atomic(): 1, irqs_disabled(): 0, pid: 137, name: kworker/u128:1
      5 locks held by kworker/u128:1/137:
       #0:  ("writeback"){......}, at: [<ffffffff81085618>] process_one_work+0x228/0x4d0
       #1:  ((&(&wb->dwork)->work)){......}, at: [<ffffffff81085618>] process_one_work+0x228/0x4d0
       #2:  (jbd2_handle){......}, at: [<ffffffff81242622>] start_this_handle+0x712/0x7b0
       #3:  (&ei->i_data_sem){......}, at: [<ffffffff811fa387>] ext4_map_blocks+0x297/0x430
       #4:  (&(&bgl->locks[i].lock)->rlock){......}, at: [<ffffffff811f3180>] ext4_read_block_bitmap_nowait+0x5d0/0x630
      CPU: 3 PID: 137 Comm: kworker/u128:1 Not tainted 3.17.0-rc2-00184-g82752e4 #165
      Hardware name: Intel Corporation W2600CR/W2600CR, BIOS SE5C600.86B.99.99.x028.061320111235 06/13/2011
      Workqueue: writeback bdi_writeback_workfn (flush-1:0)
       0000000000000411 ffff880813777288 ffffffff815c7fdc ffff880813777288
       ffff880813a8bba0 ffff8808137772a8 ffffffff8108fb30 ffff880803e01e38
       ffff880803e01e38 ffff8808137772c8 ffffffff811a8d53 ffff88080ecc6000
      Call Trace:
       [<ffffffff815c7fdc>] dump_stack+0x51/0x6d
       [<ffffffff8108fb30>] __might_sleep+0xf0/0x100
       [<ffffffff811a8d53>] __sync_dirty_buffer+0x43/0xe0
       [<ffffffff811a8e03>] sync_dirty_buffer+0x13/0x20
       [<ffffffff8120f581>] ext4_commit_super+0x1d1/0x230
       [<ffffffff8120fa03>] save_error_info+0x23/0x30
       [<ffffffff8120fd06>] __ext4_error+0xb6/0xd0
       [<ffffffff8120f260>] ? ext4_group_desc_csum+0x140/0x190
       [<ffffffff811f2d8c>] ext4_read_block_bitmap_nowait+0x1dc/0x630
       [<ffffffff8122e23a>] ext4_mb_init_cache+0x21a/0x8f0
       [<ffffffff8113ae95>] ? lru_cache_add+0x55/0x60
       [<ffffffff8112e16c>] ? add_to_page_cache_lru+0x6c/0x80
       [<ffffffff8122eaa0>] ext4_mb_init_group+0x190/0x280
       [<ffffffff8122ec51>] ext4_mb_good_group+0xc1/0x190
       [<ffffffff8123309a>] ext4_mb_regular_allocator+0x17a/0x410
       [<ffffffff8122c821>] ? ext4_mb_use_preallocated+0x31/0x380
       [<ffffffff81233535>] ? ext4_mb_new_blocks+0x205/0x8e0
       [<ffffffff8116ed5c>] ? kmem_cache_alloc+0xfc/0x180
       [<ffffffff812335b0>] ext4_mb_new_blocks+0x280/0x8e0
       [<ffffffff8116f2c4>] ? __kmalloc+0x144/0x1c0
       [<ffffffff81221797>] ? ext4_find_extent+0x97/0x320
       [<ffffffff812257f4>] ext4_ext_map_blocks+0xbc4/0x1050
       [<ffffffff811fa387>] ? ext4_map_blocks+0x297/0x430
       [<ffffffff811fa3ab>] ext4_map_blocks+0x2bb/0x430
       [<ffffffff81200e43>] ? ext4_init_io_end+0x23/0x50
       [<ffffffff811feb44>] ext4_writepages+0x564/0xaf0
       [<ffffffff815cde3b>] ? _raw_spin_unlock+0x2b/0x40
       [<ffffffff810ac7bd>] ? lock_release_non_nested+0x2fd/0x3c0
       [<ffffffff811a009e>] ? writeback_sb_inodes+0x10e/0x490
       [<ffffffff811a009e>] ? writeback_sb_inodes+0x10e/0x490
       [<ffffffff811377e3>] do_writepages+0x23/0x40
       [<ffffffff8119c8ce>] __writeback_single_inode+0x9e/0x280
       [<ffffffff811a026b>] writeback_sb_inodes+0x2db/0x490
       [<ffffffff811a0664>] wb_writeback+0x174/0x2d0
       [<ffffffff810ac359>] ? lock_release_holdtime+0x29/0x190
       [<ffffffff811a0863>] wb_do_writeback+0xa3/0x200
       [<ffffffff811a0a40>] bdi_writeback_workfn+0x80/0x230
       [<ffffffff81085618>] ? process_one_work+0x228/0x4d0
       [<ffffffff810856cd>] process_one_work+0x2dd/0x4d0
       [<ffffffff81085618>] ? process_one_work+0x228/0x4d0
       [<ffffffff81085c1d>] worker_thread+0x35d/0x460
       [<ffffffff810858c0>] ? process_one_work+0x4d0/0x4d0
       [<ffffffff810858c0>] ? process_one_work+0x4d0/0x4d0
       [<ffffffff8108a885>] kthread+0xf5/0x100
       [<ffffffff810990e5>] ? local_clock+0x25/0x30
       [<ffffffff8108a790>] ? __init_kthread_worker+0x70/0x70
       [<ffffffff815ce2ac>] ret_from_fork+0x7c/0xb0
       [<ffffffff8108a790>] ? __init_kthread_work
      
      Signed-off-by: default avatarDmitry Monakhov <dmonakhov@openvz.org>
      Signed-off-by: default avatarTheodore Ts'o <tytso@mit.edu>
      Cc: stable@vger.kernel.org
      aef4885a
  14. 04 Sep, 2014 1 commit
    • Theodore Ts'o's avatar
      ext4: prepare to drop EXT4_STATE_DELALLOC_RESERVED · e3cf5d5d
      Theodore Ts'o authored
      
      
      The EXT4_STATE_DELALLOC_RESERVED flag was originally implemented
      because it was too hard to make sure the mballoc and get_block flags
      could be reliably passed down through all of the codepaths that end up
      calling ext4_mb_new_blocks().
      
      Since then, we have mb_flags passed down through most of the code
      paths, so getting rid of EXT4_STATE_DELALLOC_RESERVED isn't as tricky
      as it used to.
      
      This commit plumbs in the last of what is required, and then adds a
      WARN_ON check to make sure we haven't missed anything.  If this passes
      a full regression test run, we can then drop
      EXT4_STATE_DELALLOC_RESERVED.
      
      Signed-off-by: default avatarTheodore Ts'o <tytso@mit.edu>
      Reviewed-by: default avatarJan Kara <jack@suse.cz>
      e3cf5d5d
  15. 15 Jul, 2014 1 commit
  16. 26 Jun, 2014 1 commit
  17. 12 May, 2014 3 commits
  18. 15 Apr, 2014 1 commit
    • Azat Khuzhin's avatar
      ext4: fix ext4_count_free_clusters() with EXT4FS_DEBUG and bigalloc enabled · 036acea2
      Azat Khuzhin authored
      
      
      With bigalloc enabled we must use EXT4_CLUSTERS_PER_GROUP() instead of
      EXT4_BLOCKS_PER_GROUP() otherwise we will go beyond the allocated buffer.
      
      $ mount -t ext4 /dev/vde /vde
      [   70.573993] EXT4-fs DEBUG (fs/ext4/mballoc.c, 2346): ext4_mb_alloc_groupinfo:
      [   70.575174] allocated s_groupinfo array for 1 meta_bg's
      [   70.576172] EXT4-fs DEBUG (fs/ext4/super.c, 2092): ext4_check_descriptors:
      [   70.576972] Checking group descriptorsBUG: unable to handle kernel paging request at ffff88006ab56000
      [   72.463686] IP: [<ffffffff81394eb9>] __bitmap_weight+0x2a/0x7f
      [   72.464168] PGD 295e067 PUD 2961067 PMD 7fa8e067 PTE 800000006ab56060
      [   72.464738] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
      [   72.465139] Modules linked in:
      [   72.465402] CPU: 1 PID: 3560 Comm: mount Tainted: G        W    3.14.0-rc2-00069-ge57bce1 #60
      [   72.466079] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
      [   72.466505] task: ffff88007ce6c8a0 ti: ffff88006b7f0000 task.ti: ffff88006b7f0000
      [   72.466505] RIP: 0010:[<ffffffff81394eb9>]  [<ffffffff81394eb9>] __bitmap_weight+0x2a/0x7f
      [   72.466505] RSP: 0018:ffff88006b7f1c00  EFLAGS: 00010206
      [   72.466505] RAX: 0000000000000000 RBX: 000000000000050a RCX: 0000000000000040
      [   72.466505] RDX: 0000000000000000 RSI: 0000000000080000 RDI: 0000000000000000
      [   72.466505] RBP: ffff88006b7f1c28 R08: 0000000000000002 R09: 0000000000000000
      [   72.466505] R10: 000000000000babe R11: 0000000000000400 R12: 0000000000080000
      [   72.466505] R13: 0000000000000200 R14: 0000000000002000 R15: ffff88006ab55000
      [   72.466505] FS:  00007f43ba1fa840(0000) GS:ffff88007f800000(0000) knlGS:0000000000000000
      [   72.466505] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
      [   72.466505] CR2: ffff88006ab56000 CR3: 000000006b7e6000 CR4: 00000000000006e0
      [   72.466505] Stack:
      [   72.466505]  ffff88006ab65000 0000000000000000 0000000000000000 0000000000010000
      [   72.466505]  ffff88006ab6f400 ffff88006b7f1c58 ffffffff81396bb8 0000000000010000
      [   72.466505]  0000000000000000 ffff88007b869a90 ffff88006a48a000 ffff88006b7f1c70
      [   72.466505] Call Trace:
      [   72.466505]  [<ffffffff81396bb8>] memweight+0x5f/0x8a
      [   72.466505]  [<ffffffff811c3b19>] ext4_count_free+0x13/0x21
      [   72.466505]  [<ffffffff811c396c>] ext4_count_free_clusters+0xdb/0x171
      [   72.466505]  [<ffffffff811e3bdd>] ext4_fill_super+0x117c/0x28ef
      [   72.466505]  [<ffffffff81391569>] ? vsnprintf+0x1c7/0x3f7
      [   72.466505]  [<ffffffff8114d8dc>] mount_bdev+0x145/0x19c
      [   72.466505]  [<ffffffff811e2a61>] ? ext4_calculate_overhead+0x2a1/0x2a1
      [   72.466505]  [<ffffffff811dab1d>] ext4_mount+0x15/0x17
      [   72.466505]  [<ffffffff8114e3aa>] mount_fs+0x67/0x150
      [   72.466505]  [<ffffffff811637ea>] vfs_kern_mount+0x64/0xde
      [   72.466505]  [<ffffffff81165d19>] do_mount+0x6fe/0x7f5
      [   72.466505]  [<ffffffff81126cc8>] ? strndup_user+0x3a/0xd9
      [   72.466505]  [<ffffffff8116604b>] SyS_mount+0x85/0xbe
      [   72.466505]  [<ffffffff81619e90>] tracesys+0xdd/0xe2
      [   72.466505] Code: c3 89 f0 b9 40 00 00 00 55 99 48 89 e5 41 57 f7 f9 41 56 49 89 ff 41 55 45 31 ed 41 54 41 89 f4 53 31 db 41 89 c6 45 39 ee 7e 10 <4b> 8b 3c ef 49 ff c5 e8 bf ff ff ff 01 c3 eb eb 31 c0 45 85 f6
      [   72.466505] RIP  [<ffffffff81394eb9>] __bitmap_weight+0x2a/0x7f
      [   72.466505]  RSP <ffff88006b7f1c00>
      [   72.466505] CR2: ffff88006ab56000
      [   72.466505] ---[ end trace 7d051a08ae138573 ]---
      Killed
      
      Signed-off-by: default avatar"Theodore Ts'o" <tytso@mit.edu>
      036acea2
  19. 31 Oct, 2013 1 commit
  20. 28 Aug, 2013 4 commits
    • Darrick J. Wong's avatar
      ext4: mark group corrupt on group descriptor checksum · bdfb6ff4
      Darrick J. Wong authored
      
      
      If the group descriptor fails validation, mark the whole blockgroup
      corrupt so that the inode/block allocators skip this group.  The
      previous approach takes the risk of writing to a damaged group
      descriptor; hopefully it was never the case that the [ib]bitmap fields
      pointed to another valid block and got dirtied, since the memset would
      fill the page with 1s.
      
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Signed-off-by: default avatar"Theodore Ts'o" <tytso@mit.edu>
      bdfb6ff4
    • Darrick J. Wong's avatar
      ext4: mark block group as corrupt on block bitmap error · 163a203d
      Darrick J. Wong authored
      
      
      When we notice a block-bitmap corruption (because of device failure or
      something else), we should mark this group as corrupt and prevent
      further block allocations/deallocations from it. Currently, we end up
      generating one error message for every block in the bitmap. This
      potentially could make the system unstable as noticed in some
      bugs. With this patch, the error will be printed only the first time
      and mark the entire block group as corrupted. This prevents future
      access allocations/deallocations from it.
      
      Also tested by corrupting the block
      bitmap and forcefully introducing the mb_free_blocks error:
      (1) create a largefile (2Gb)
      $ dd if=/dev/zero of=largefile oflag=direct bs=10485760 count=200
      (2) umount filesystem. use dumpe2fs to see which block-bitmaps
      are in use by largefile and note their block numbers
      (3) use dd to zero-out the used block bitmaps
      $ dd if=/dev/zero of=/dev/hdc4 bs=4096 seek=14 count=8 oflag=direct
      (4) mount the FS and delete the largefile.
      (5) recreate the largefile. verify that the new largefile does not
      get any blocks from the groups marked as bad.
      Without the patch, we will see mb_free_blocks error for each bit in
      each zero'ed out bitmap at (4). With the patch, we only see the error
      once per blockgroup:
      [  309.706803] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 15: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
      [  309.720824] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 14: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
      [  309.732858] EXT4-fs error (device sdb4) in ext4_free_blocks:4802: IO failure
      [  309.748321] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 13: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
      [  309.760331] EXT4-fs error (device sdb4) in ext4_free_blocks:4802: IO failure
      [  309.769695] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 12: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
      [  309.781721] EXT4-fs error (device sdb4) in ext4_free_blocks:4802: IO failure
      [  309.798166] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 11: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
      [  309.810184] EXT4-fs error (device sdb4) in ext4_free_blocks:4802: IO failure
      [  309.819532] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 10: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
      
      Google-Bug-Id: 7258357
      
      [darrick.wong@oracle.com]
      Further modifications (by Darrick) to make more obvious that this corruption
      bit applies to blocks only.  Set the corruption flag if the block group bitmap
      verification fails.
      
      Original-author: Aditya Kali <adityakali@google.com>
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Signed-off-by: default avatar"Theodore Ts'o" <tytso@mit.edu>
      163a203d
    • Darrick J. Wong's avatar
      ext4: fix type declaration of ext4_validate_block_bitmap · dbde0abe
      Darrick J. Wong authored
      
      
      The block_group parameter to ext4_validate_block_bitmap is both used
      as a ext4_group_t inside the function and the same type is passed in
      by all callers.  We might as well use the typedef consistently instead
      of open-coding the 'unsigned int'.
      
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Signed-off-by: default avatar"Theodore Ts'o" <tytso@mit.edu>
      dbde0abe
    • Darrick J. Wong's avatar
      ext4: error out if verifying the block bitmap fails · 48d9eb97
      Darrick J. Wong authored
      
      
      The block bitmap verification code assumes that calling ext4_error()
      either panics the system or makes the fs readonly.  However, this is
      not always true: when 'errors=continue' is specified, an error is
      printed but we don't return any indication of error to the caller,
      which is (probably) the block allocator, which pretends that the crud
      we read in off the disk is a usable bitmap.  Yuck.
      
      A block bitmap that fails the check should at least return no bitmap
      to the caller.  The block allocator should be told to go look in a
      different group, but that's a separate issue.
      
      The easiest way to reproduce this is to modify bg_block_bitmap (on a
      ^flex_bg fs) to point to a block outside the block group; or you can
      create a metadata_csum filesystem and zero out the block bitmaps.
      
      Signed-off-by: default avatarDarrick J. Wong <darrick.wong@oracle.com>
      Signed-off-by: default avatar"Theodore Ts'o" <tytso@mit.edu>
      48d9eb97
  21. 06 Jul, 2013 1 commit
    • Theodore Ts'o's avatar
      ext4: fix ext4_get_group_number() · 960fd856
      Theodore Ts'o authored
      The function ext4_get_group_number() was introduced as an optimization
      in commit bd86298e.  Unfortunately, this commit incorrectly
      calculate the group number for file systems with a 1k block size (when
      s_first_data_block is 1 instead of zero).  This could cause the
      following kernel BUG:
      
      [  568.877799] ------------[ cut here ]------------
      [  568.877833] kernel BUG at fs/ext4/mballoc.c:3728!
      [  568.877840] Oops: Exception in kernel mode, sig: 5 [#1]
      [  568.877845] SMP NR_CPUS=32 NUMA pSeries
      [  568.877852] Modules linked in: binfmt_misc
      [  568.877861] CPU: 1 PID: 3516 Comm: fs_mark Not tainted 3.10.0-03216-g7c6809ff-dirty #1
      [  568.877867] task: c0000001fb0b8000 ti: c0000001fa954000 task.ti: c0000001fa954000
      [  568.877873] NIP: c0000000002f42a4 LR: c0000000002f4274 CTR: c000000000317ef8
      [  568.877879] REGS: c0000001fa956ed0 TRAP: 0700   Not tainted  (3.10.0-03216-g7c6809ff
      
      -dirty)
      [  568.877884] MSR: 8000000000029032 <SF,EE,ME,IR,DR,RI>  CR: 24000428  XER: 00000000
      [  568.877902] SOFTE: 1
      [  568.877905] CFAR: c0000000002b5464
      [  568.877908]
      GPR00: 0000000000000001 c0000001fa957150 c000000000c6a408 c0000001fb588000
      GPR04: 0000000000003fff c0000001fa9571c0 c0000001fa9571c4 000138098c50625f
      GPR08: 1301200000000000 0000000000000002 0000000000000001 0000000000000000
      GPR12: 0000000024000422 c00000000f33a300 0000000000008000 c0000001fa9577f0
      GPR16: c0000001fb7d0100 c000000000c29190 c0000000007f46e8 c000000000a14672
      GPR20: 0000000000000001 0000000000000008 ffffffffffffffff 0000000000000000
      GPR24: 0000000000000100 c0000001fa957278 c0000001fdb2bc78 c0000001fa957288
      GPR28: 0000000000100100 c0000001fa957288 c0000001fb588000 c0000001fdb2bd10
      [  568.877993] NIP [c0000000002f42a4] .ext4_mb_release_group_pa+0xec/0x1c0
      [  568.877999] LR [c0000000002f4274] .ext4_mb_release_group_pa+0xbc/0x1c0
      [  568.878004] Call Trace:
      [  568.878008] [c0000001fa957150] [c0000000002f4274] .ext4_mb_release_group_pa+0xbc/0x1c0 (unreliable)
      [  568.878017] [c0000001fa957200] [c0000000002fb070] .ext4_mb_discard_lg_preallocations+0x394/0x444
      [  568.878025] [c0000001fa957340] [c0000000002fb45c] .ext4_mb_release_context+0x33c/0x734
      [  568.878032] [c0000001fa957440] [c0000000002fbcf8] .ext4_mb_new_blocks+0x4a4/0x5f4
      [  568.878039] [c0000001fa957510] [c0000000002ef56c] .ext4_ext_map_blocks+0xc28/0x1178
      [  568.878047] [c0000001fa957640] [c0000000002c1a94] .ext4_map_blocks+0x2c8/0x490
      [  568.878054] [c0000001fa957730] [c0000000002c536c] .ext4_writepages+0x738/0xc60
      [  568.878062] [c0000001fa957950] [c000000000168a78] .do_writepages+0x5c/0x80
      [  568.878069] [c0000001fa9579d0] [c00000000015d1c4] .__filemap_fdatawrite_range+0x88/0xb0
      [  568.878078] [c0000001fa957aa0] [c00000000015d23c] .filemap_write_and_wait_range+0x50/0xfc
      [  568.878085] [c0000001fa957b30] [c0000000002b8edc] .ext4_sync_file+0x220/0x3c4
      [  568.878092] [c0000001fa957be0] [c0000000001f849c] .vfs_fsync_range+0x64/0x80
      [  568.878098] [c0000001fa957c70] [c0000000001f84f0] .vfs_fsync+0x38/0x4c
      [  568.878105] [c0000001fa957d00] [c0000000001f87f4] .do_fsync+0x54/0x90
      [  568.878111] [c0000001fa957db0] [c0000000001f8894] .SyS_fsync+0x28/0x3c
      [  568.878120] [c0000001fa957e30] [c000000000009c88] syscall_exit+0x0/0x7c
      [  568.878125] Instruction dump:
      [  568.878130] 60000000 813d0034 81610070 38000000 7f8b4800 419e001c 813f007c 7d2bfe70
      [  568.878144] 7d604a78 7c005850 54000ffe 7c0007b4 <0b000000> e8a10076 e87f0090 7fa4eb78
      [  568.878160] ---[ end trace 594d911d9654770b ]---
      
      In addition fix the STD_GROUP optimization so that it works for
      bigalloc file systems as well.
      
      Signed-off-by: default avatar"Theodore Ts'o" <tytso@mit.edu>
      Reported-by: default avatarLi Zhong <lizhongfs@gmail.com>
      Reviewed-by: default avatarLukas Czerner <lczerner@redhat.com>
      Cc: stable@vger.kernel.org  # 3.10
      960fd856
  22. 06 Jun, 2013 1 commit
    • Theodore Ts'o's avatar
      ext4: optimize test_root() · f4afb4f4
      Theodore Ts'o authored
      
      
      The test_root() function could potentially loop forever due to
      overflow issues.  So rewrite test_root() to avoid this issue; as a
      bonus, it is 38% faster when benchmarked via a test loop:
      
      int main(int argc, char **argv)
      {
      	int  i;
      
      	for (i = 0; i < 1 << 24; i++) {
      		if (test_root(i, 7))
      			printf("%d\n", i);
      	}
      }
      
      Signed-off-by: default avatar"Theodore Ts'o" <tytso@mit.edu>
      f4afb4f4
  23. 20 Apr, 2013 1 commit
  24. 10 Apr, 2013 1 commit
    • Lukas Czerner's avatar
      ext4: introduce reserved space · 27dd4385
      Lukas Czerner authored
      
      
      Currently in ENOSPC condition when writing into unwritten space, or
      punching a hole, we might need to split the extent and grow extent tree.
      However since we can not allocate any new metadata blocks we'll have to
      zero out unwritten part of extent or punched out part of extent, or in
      the worst case return ENOSPC even though use actually does not allocate
      any space.
      
      Also in delalloc path we do reserve metadata and data blocks for the
      time we're going to write out, however metadata block reservation is
      very tricky especially since we expect that logical connectivity implies
      physical connectivity, however that might not be the case and hence we
      might end up allocating more metadata blocks than previously reserved.
      So in future, metadata reservation checks should be removed since we can
      not assure that we do not under reserve.
      
      And this is where reserved space comes into the picture. When mounting
      the file system we slice off a little bit of the file system space (2%
      or 4096 clusters, whichever is smaller) which can be then used for the
      cases mentioned above to prevent costly zeroout, or unexpected ENOSPC.
      
      The number of reserved clusters can be set via sysfs, however it can
      never be bigger than number of free clusters in the file system.
      
      Note that this patch fixes the failure of xfstest 274 as expected.
      
      Signed-off-by: default avatarLukas Czerner <lczerner@redhat.com>
      Signed-off-by: default avatar"Theodore Ts'o" <tytso@mit.edu>
      Reviewed-by: default avatarCarlos Maiolino <cmaiolino@redhat.com>
      27dd4385
  25. 04 Apr, 2013 2 commits
    • Lukas Czerner's avatar
      ext4: introduce ext4_get_group_number() · bd86298e
      Lukas Czerner authored
      
      
      Currently on many places in ext4 we're using
      ext4_get_group_no_and_offset() even though we're only interested in
      knowing the block group of the particular block, not the offset within
      the block group so we can use more efficient way to compute block
      group.
      
      This patch introduces ext4_get_group_number() which computes block
      group for a given block much more efficiently. Use this function
      instead of ext4_get_group_no_and_offset() everywhere where we're only
      interested in knowing the block group.
      
      Signed-off-by: default avatarLukas Czerner <lczerner@redhat.com>
      Signed-off-by: default avatar"Theodore Ts'o" <tytso@mit.edu>
      bd86298e
    • Lukas Czerner's avatar
      ext4: make ext4_block_in_group() much more efficient · 68911009
      Lukas Czerner authored
      
      
      Currently in when getting the block group number for a particular
      block in ext4_block_in_group() we're using
      ext4_get_group_no_and_offset() which uses do_div() to get the block
      group and the remainer which is offset within the group.
      
      We don't need all of that in ext4_block_in_group() as we only need to
      figure out the group number.
      
      This commit changes ext4_block_in_group() to calculate group number
      directly. This shows as a big improvement with regards to cpu
      utilization. Measuring fallocate -l 15T on fresh file system with perf
      showed that 23% of cpu time was spend in the
      ext4_get_group_no_and_offset(). With this change it completely
      disappears from the list only bumping the occurrence of
      ext4_init_block_bitmap() which is the biggest user of
      ext4_block_in_group() by 4%. As the result of this change on my system
      the fallocate call was approx. 10% faster.
      
      However since there is '-g' option in mkfs which allow us setting
      different groups size (mostly for developers) I've introduced new per
      file system flag whether we have a standard block group size or
      not. The flag is used to determine whether we can use the bit shift
      optimization or not.
      
      Signed-off-by: default avatarLukas Czerner <lczerner@redhat.com>
      Signed-off-by: default avatar"Theodore Ts'o" <tytso@mit.edu>
      68911009
  26. 02 Mar, 2013 1 commit
  27. 22 Feb, 2013 1 commit
    • Lukas Czerner's avatar
      ext4: fix free clusters calculation in bigalloc filesystem · 304e220f
      Lukas Czerner authored
      
      
      ext4_has_free_clusters() should tell us whether there is enough free
      clusters to allocate, however number of free clusters in the file system
      is converted to blocks using EXT4_C2B() which is not only wrong use of
      the macro (we should have used EXT4_NUM_B2C) but it's also completely
      wrong concept since everything else is in cluster units.
      
      Moreover when calculating number of root clusters we should be using
      macro EXT4_NUM_B2C() instead of EXT4_B2C() otherwise the result might be
      off by one. However r_blocks_count should always be a multiple of the
      cluster ratio so doing a plain bit shift should be enough here. We
      avoid using EXT4_B2C() because it's confusing.
      
      As a result of the first problem number of free clusters is much bigger
      than it should have been and ext4_has_free_clusters() would return 1 even
      if there is really not enough free clusters available.
      
      Fix this by removing the EXT4_C2B() conversion of free clusters and
      using bit shift when calculating number of root clusters. This bug
      affects number of xfstests tests covering file system ENOSPC situation
      handling. With this patch most of the ENOSPC problems with bigalloc file
      system disappear, especially the errors caused by delayed allocation not
      having enough space when the actual allocation is finally requested.
      
      Signed-off-by: default avatarLukas Czerner <lczerner@redhat.com>
      Signed-off-by: default avatar"Theodore Ts'o" <tytso@mit.edu>
      Cc: stable@vger.kernel.org
      304e220f
  28. 12 Jan, 2013 1 commit
    • Eryu Guan's avatar
      ext4: check bh in ext4_read_block_bitmap() · 15b49132
      Eryu Guan authored
      
      
      Validate the bh pointer before using it, since
      ext4_read_block_bitmap_nowait() might return NULL.
      
      I've seen this in fsfuzz testing.
      
       EXT4-fs error (device loop0): ext4_read_block_bitmap_nowait:385: comm touch: Cannot get buffer for block bitmap - block_group = 0, block_bitmap = 3925999616
       BUG: unable to handle kernel NULL pointer dereference at           (null)
       IP: [<ffffffff8121de25>] ext4_wait_block_bitmap+0x25/0xe0
       ...
       Call Trace:
        [<ffffffff8121e1e5>] ext4_read_block_bitmap+0x35/0x60
        [<ffffffff8125e9c6>] ext4_free_blocks+0x236/0xb80
        [<ffffffff811d0d36>] ? __getblk+0x36/0x70
        [<ffffffff811d0a5f>] ? __find_get_block+0x8f/0x210
        [<ffffffff81191ef3>] ? kmem_cache_free+0x33/0x140
        [<ffffffff812678e5>] ext4_xattr_release_block+0x1b5/0x1d0
        [<ffffffff812679be>] ext4_xattr_delete_inode+0xbe/0x100
        [<ffffffff81222a7c>] ext4_free_inode+0x7c/0x4d0
        [<ffffffff812277b8>] ? ext4_mark_inode_dirty+0x88/0x230
        [<ffffffff8122993c>] ext4_evict_inode+0x32c/0x490
        [<ffffffff811b8cd7>] evict+0xa7/0x1c0
        [<ffffffff811b8ed3>] iput_final+0xe3/0x170
        [<ffffffff811b8f9e>] iput+0x3e/0x50
        [<ffffffff812316fd>] ext4_add_nondir+0x4d/0x90
        [<ffffffff81231d0b>] ext4_create+0xeb/0x170
        [<ffffffff811aae9c>] vfs_create+0xac/0xd0
        [<ffffffff811ac845>] lookup_open+0x185/0x1c0
        [<ffffffff8129e3b9>] ? selinux_inode_permission+0xa9/0x170
        [<ffffffff811acb54>] do_last+0x2d4/0x7a0
        [<ffffffff811af743>] path_openat+0xb3/0x480
        [<ffffffff8116a8a1>] ? handle_mm_fault+0x251/0x3b0
        [<ffffffff811afc49>] do_filp_open+0x49/0xa0
        [<ffffffff811bbaad>] ? __alloc_fd+0xdd/0x150
        [<ffffffff8119da28>] do_sys_open+0x108/0x1f0
        [<ffffffff8119db51>] sys_open+0x21/0x30
        [<ffffffff81618959>] system_call_fastpath+0x16/0x1b
      
      Also fix comment for ext4_read_block_bitmap_nowait()
      
      Signed-off-by: default avatarEryu Guan <guaneryu@gmail.com>
      Signed-off-by: default avatar"Theodore Ts'o" <tytso@mit.edu>
      Cc: stable@vger.kernel.org
      15b49132
  29. 22 Oct, 2012 1 commit
  30. 17 Aug, 2012 1 commit
  31. 30 Jun, 2012 1 commit
  32. 07 Jun, 2012 1 commit
    • Theodore Ts'o's avatar
      ext4: fix the free blocks calculation for ext3 file systems w/ uninit_bg · b0dd6b70
      Theodore Ts'o authored
      Ext3 filesystems that are converted to use as many ext4 file system
      features as possible will enable uninit_bg to speed up e2fsck times.
      These file systems will have a native ext3 layout of inode tables and
      block allocation bitmaps (as opposed to ext4's flex_bg layout).
      Unfortunately, in these cases, when first allocating a block in an
      uninitialized block group, ext4 would incorrectly calculate the number
      of free blocks in that block group, and then errorneously report that
      the file system was corrupt:
      
      EXT4-fs error (device vdd): ext4_mb_generate_buddy:741: group 30, 32254 clusters in bitmap, 32258 in gd
      
      This problem can be reproduced via:
      
          mke2fs -q -t ext4 -O ^flex_bg /dev/vdd 5g
          mount -t ext4 /dev/vdd /mnt
          fallocate -l 4600m /mnt/test
      
      The problem was caused by a bone headed mistake in the check to see if a
      particular metadata block was part of the block group.
      
      Many thanks to Kees Cook for finding and bisecting the buggy commit
      which introduced this bug (commit fd034a84
      
      , present since v3.2).
      
      Reported-by: default avatarSander Eikelenboom <linux@eikelenboom.it>
      Reported-by: default avatarKees Cook <keescook@chromium.org>
      Signed-off-by: default avatar"Theodore Ts'o" <tytso@mit.edu>
      Tested-by: default avatarKees Cook <keescook@chromium.org>
      Cc: stable@kernel.org
      b0dd6b70