1. 13 Dec, 2019 1 commit
    • Anand Jain's avatar
      btrfs: send: remove WARN_ON for readonly mount · fbd54297
      Anand Jain authored
      We log warning if root::orphan_cleanup_state is not set to
      ORPHAN_CLEANUP_DONE in btrfs_ioctl_send(). However if the filesystem is
      mounted as readonly we skip the orphan item cleanup during the lookup
      and root::orphan_cleanup_state remains at the init state 0 instead of
      ORPHAN_CLEANUP_DONE (2). So during send in btrfs_ioctl_send() we hit the
      warning as below.
      
        WARN_ON(send_root->orphan_cleanup_state != ORPHAN_CLEANUP_DONE);
      
      WARNING: CPU: 0 PID: 2616 at /Volumes/ws/btrfs-devel/fs/btrfs/send.c:7090 btrfs_ioctl_send+0xb2f/0x18c0 [btrfs]
      ::
      RIP: 0010:btrfs_ioctl_send+0xb2f/0x18c0 [btrfs]
      ::
      Call Trace:
      ::
      _btrfs_ioctl_send+0x7b/0x110 [btrfs]
      btrfs_ioctl+0x150a/0x2b00 [btrfs]
      ::
      do_vfs_ioctl+0xa9/0x620
      ? __fget+0xac/0xe0
      ksys_ioctl+0x60/0x90
      __x64_sys_ioctl+0x16/0x20
      do_syscall_64+0x49/0x130
      entry_SYSCALL_64_after_hwframe+0x44/0xa9
      
      Reproducer:
        mkfs.btrfs -fq /dev/sdb
        mount /dev/sdb /btrfs
        btrfs subvolume create /btrfs/sv1
        btrfs subvolume snapshot -r /btrfs/sv1 /btrfs/ss1
        umount /btrfs
        mount -o ro /dev/sdb /btrfs
        btrfs send /btrfs/ss1 -f /tmp/f
      
      The warning exists because having orphan inodes could confuse send and
      cause it to fail or produce incorrect streams.  The two cases that would
      cause such send failures, which are already fixed are:
      
      1) Inodes that were unlinked - these are orphanized and remain with a
         link count of 0. These caused send operations to fail because it
         expected to always find at least one path for an inode. However this
         is no longer a problem since send is now able to deal with such
         inodes since commit 46b2f459 ("Btrfs: fix send failure when root
         has deleted files still open") and treats them as having been
         completely removed (the state after an orphan cleanup is performed).
      
      2) Inodes that were in the process of being truncated. These resulted in
         send not knowing about the truncation and potentially issue write
         operations full of zeroes for the range from the new file size to the
         old file size. This is no longer a problem because we no longer
         create orphan items for truncation since commit f7e9e8fc
      
       ("Btrfs:
         stop creating orphan items for truncate").
      
      As such before these commits, the WARN_ON here provided a clue in case
      something went wrong. Instead of being a warning against the
      root::orphan_cleanup_state value, it could have been more accurate by
      checking if there were actually any orphan items, and then issue a
      warning only if any exists, but that would be more expensive to check.
      Since orphanized inodes no longer cause problems for send, just remove
      the warning.
      Reported-by: default avatarChristoph Anton Mitterer <calestyo@scientia.net>
      Link: https://lore.kernel.org/linux-btrfs/21cb5e8d059f6e1496a903fa7bfc0a297e2f5370.camel@scientia.net/
      
      
      CC: stable@vger.kernel.org # 4.19+
      Suggested-by: default avatarFilipe Manana <fdmanana@suse.com>
      Reviewed-by: default avatarFilipe Manana <fdmanana@suse.com>
      Signed-off-by: default avatarAnand Jain <anand.jain@oracle.com>
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      fbd54297
  2. 18 Nov, 2019 3 commits
    • Filipe Manana's avatar
      Btrfs: send, skip backreference walking for extents with many references · fd0ddbe2
      Filipe Manana authored
      
      
      Backreference walking, which is used by send to figure if it can issue
      clone operations instead of write operations, can be very slow and use
      too much memory when extents have many references. This change simply
      skips backreference walking when an extent has more than 64 references,
      in which case we fallback to a write operation instead of a clone
      operation. This limit is conservative and in practice I observed no
      signicant slowdown with up to 100 references and still low memory usage
      up to that limit.
      
      This is a temporary workaround until there are speedups in the backref
      walking code, and as such it does not attempt to add extra interfaces or
      knobs to tweak the threshold.
      Reported-by: default avatarAtemu <atemu.main@gmail.com>
      Link: https://lore.kernel.org/linux-btrfs/CAE4GHgkvqVADtS4AzcQJxo0Q1jKQgKaW3JGp3SGdoinVo=C9eQ@mail.gmail.com/T/#me55dc0987f9cc2acaa54372ce0492c65782be3fa
      
      
      CC: stable@vger.kernel.org # 4.4+
      Reviewed-by: default avatarQu Wenruo <wqu@suse.com>
      Signed-off-by: default avatarFilipe Manana <fdmanana@suse.com>
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      fd0ddbe2
    • Filipe Manana's avatar
      Btrfs: send, allow clone operations within the same file · 11f2069c
      Filipe Manana authored
      For send we currently skip clone operations when the source and
      destination files are the same. This is so because clone didn't support
      this case in its early days, but support for it was added back in May
      2013 by commit a96fbc72
      
       ("Btrfs: allow file data clone within a
      file"). This change adds support for it.
      
      Example:
      
        $ mkfs.btrfs -f /dev/sdd
        $ mount /dev/sdd /mnt/sdd
      
        $ xfs_io -f -c "pwrite -S 0xab -b 64K 0 64K" /mnt/sdd/foobar
        $ xfs_io -c "reflink /mnt/sdd/foobar 0 64K 64K" /mnt/sdd/foobar
      
        $ btrfs subvolume snapshot -r /mnt/sdd /mnt/sdd/snap
      
        $ mkfs.btrfs -f /dev/sde
        $ mount /dev/sde /mnt/sde
      
        $ btrfs send /mnt/sdd/snap | btrfs receive /mnt/sde
      
      Without this change file foobar at the destination has a single 128Kb
      extent:
      
        $ filefrag -v /mnt/sde/snap/foobar
        Filesystem type is: 9123683e
        File size of /mnt/sde/snap/foobar is 131072 (32 blocks of 4096 bytes)
         ext:     logical_offset:        physical_offset: length:   expected: flags:
           0:        0..      31:          0..        31:     32:             last,unknown_loc,delalloc,eof
        /mnt/sde/snap/foobar: 1 extent found
      
      With this we get a single 64Kb extent that is shared at file offsets 0
      and 64K, just like in the source filesystem:
      
        $ filefrag -v /mnt/sde/snap/foobar
        Filesystem type is: 9123683e
        File size of /mnt/sde/snap/foobar is 131072 (32 blocks of 4096 bytes)
         ext:     logical_offset:        physical_offset: length:   expected: flags:
           0:        0..      15:       3328..      3343:     16:             shared
           1:       16..      31:       3328..      3343:     16:       3344: last,shared,eof
        /mnt/sde/snap/foobar: 2 extents found
      Reviewed-by: default avatarJosef Bacik <josef@toxicpanda.com>
      Signed-off-by: default avatarFilipe Manana <fdmanana@suse.com>
      Reviewed-by: default avatarDavid Sterba <dsterba@suse.com>
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      11f2069c
    • David Sterba's avatar
      btrfs: drop unused parameter is_new from btrfs_iget · 4c66e0d4
      David Sterba authored
      The parameter is now always set to NULL and could be dropped. The last
      user was get_default_root but that got reworked in 05dbe683
      
       ("Btrfs:
      unify subvol= and subvolid= mounting") and the parameter became unused.
      Reviewed-by: default avatarAnand Jain <anand.jain@oracle.com>
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      4c66e0d4
  3. 08 Oct, 2019 1 commit
    • Austin Kim's avatar
      btrfs: silence maybe-uninitialized warning in clone_range · 431d3988
      Austin Kim authored
      
      
      GCC throws warning message as below:
      
      ‘clone_src_i_size’ may be used uninitialized in this function
      [-Wmaybe-uninitialized]
       #define IS_ALIGNED(x, a)  (((x) & ((typeof(x))(a) - 1)) == 0)
                             ^
      fs/btrfs/send.c:5088:6: note: ‘clone_src_i_size’ was declared here
       u64 clone_src_i_size;
         ^
      The clone_src_i_size is only used as call-by-reference
      in a call to get_inode_info().
      
      Silence the warning by initializing clone_src_i_size to 0.
      
      Note that the warning is a false positive and reported by older versions
      of GCC (eg. 7.x) but not eg 9.x. As there have been numerous people, the
      patch is applied. Setting clone_src_i_size to 0 does not otherwise make
      sense and would not do any action in case the code changes in the future.
      Signed-off-by: default avatarAustin Kim <austindh.kim@gmail.com>
      Reviewed-by: default avatarDavid Sterba <dsterba@suse.com>
      [ add note ]
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      431d3988
  4. 09 Sep, 2019 2 commits
    • Nikolay Borisov's avatar
      btrfs: Relinquish CPUs in btrfs_compare_trees · 6af112b1
      Nikolay Borisov authored
      When doing any form of incremental send the parent and the child trees
      need to be compared via btrfs_compare_trees. This  can result in long
      loop chains without ever relinquishing the CPU. This causes softlockup
      detector to trigger when comparing trees with a lot of items. Example
      report:
      
      watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [snapperd:16153]
      CPU: 0 PID: 16153 Comm: snapperd Not tainted 5.2.9-1-default #1 openSUSE Tumbleweed (unreleased)
      Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
      pstate: 40000005 (nZcv daif -PAN -UAO)
      pc : __ll_sc_arch_atomic_sub_return+0x14/0x20
      lr : btrfs_release_extent_buffer_pages+0xe0/0x1e8 [btrfs]
      sp : ffff00001273b7e0
      Call trace:
       __ll_sc_arch_atomic_sub_return+0x14/0x20
       release_extent_buffer+0xdc/0x120 [btrfs]
       free_extent_buffer.part.0+0xb0/0x118 [btrfs]
       free_extent_buffer+0x24/0x30 [btrfs]
       btrfs_release_path+0x4c/0xa0 [btrfs]
       btrfs_free_path.part.0+0x20/0x40 [btrfs]
       btrfs_free_path+0x24/0x30 [btrfs]
       get_inode_info+0xa8/0xf8 [btrfs]
       finish_inode_if_needed+0xe0/0x6d8 [btrfs]
       changed_cb+0x9c/0x410 [btrfs]
       btrfs_compare_trees+0x284/0x648 [btrfs]
       send_subvol+0x33c/0x520 [btrfs]
       btrfs_ioctl_send+0x8a0/0xaf0 [btrfs]
       btrfs_ioctl+0x199c/0x2288 [btrfs]
       do_vfs_ioctl+0x4b0/0x820
       ksys_ioctl+0x84/0xb8
       __arm64_sys_ioctl+0x28/0x38
       el0_svc_common.constprop.0+0x7c/0x188
       el0_svc_handler+0x34/0x90
       el0_svc+0x8/0xc
      
      Fix this by adding a call to cond_resched at the beginning of the main
      loop in btrfs_compare_trees.
      
      Fixes: 7069830a
      
       ("Btrfs: add btrfs_compare_trees function")
      CC: stable@vger.kernel.org # 4.4+
      Reviewed-by: default avatarJohannes Thumshirn <jthumshirn@suse.de>
      Signed-off-by: default avatarNikolay Borisov <nborisov@suse.com>
      Reviewed-by: default avatarDavid Sterba <dsterba@suse.com>
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      6af112b1
    • David Sterba's avatar
      btrfs: move functions for tree compare to send.c · 18d0f5c6
      David Sterba authored
      
      
      Send is the only user of tree_compare, we can move it there along with
      the other helpers and definitions.
      Reviewed-by: default avatarJohannes Thumshirn <jthumshirn@suse.de>
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      18d0f5c6
  5. 30 Jul, 2019 1 commit
    • Filipe Manana's avatar
      Btrfs: fix incremental send failure after deduplication · b4f9a1a8
      Filipe Manana authored
      When doing an incremental send operation we can fail if we previously did
      deduplication operations against a file that exists in both snapshots. In
      that case we will fail the send operation with -EIO and print a message
      to dmesg/syslog like the following:
      
        BTRFS error (device sdc): Send: inconsistent snapshot, found updated \
        extent for inode 257 without updated inode item, send root is 258, \
        parent root is 257
      
      This requires that we deduplicate to the same file in both snapshots for
      the same amount of times on each snapshot. The issue happens because a
      deduplication only updates the iversion of an inode and does not update
      any other field of the inode, therefore if we deduplicate the file on
      each snapshot for the same amount of time, the inode will have the same
      iversion value (stored as the "sequence" field on the inode item) on both
      snapshots, therefore it will be seen as unchanged between in the send
      snapshot while there are new/updated/deleted extent items when comparing
      to the parent snapshot. This makes the send operation return -EIO and
      print an error message.
      
      Example reproducer:
      
        $ mkfs.btrfs -f /dev/sdb
        $ mount /dev/sdb /mnt
      
        # Create our first file. The first half of the file has several 64Kb
        # extents while the second half as a single 512Kb extent.
        $ xfs_io -f -s -c "pwrite -S 0xb8 -b 64K 0 512K" /mnt/foo
        $ xfs_io -c "pwrite -S 0xb8 512K 512K" /mnt/foo
      
        # Create the base snapshot and the parent send stream from it.
        $ btrfs subvolume snapshot -r /mnt /mnt/mysnap1
        $ btrfs send -f /tmp/1.snap /mnt/mysnap1
      
        # Create our second file, that has exactly the same data as the first
        # file.
        $ xfs_io -f -c "pwrite -S 0xb8 0 1M" /mnt/bar
      
        # Create the second snapshot, used for the incremental send, before
        # doing the file deduplication.
        $ btrfs subvolume snapshot -r /mnt /mnt/mysnap2
      
        # Now before creating the incremental send stream:
        #
        # 1) Deduplicate into a subrange of file foo in snapshot mysnap1. This
        #    will drop several extent items and add a new one, also updating
        #    the inode's iversion (sequence field in inode item) by 1, but not
        #    any other field of the inode;
        #
        # 2) Deduplicate into a different subrange of file foo in snapshot
        #    mysnap2. This will replace an extent item with a new one, also
        #    updating the inode's iversion by 1 but not any other field of the
        #    inode.
        #
        # After these two deduplication operations, the inode items, for file
        # foo, are identical in both snapshots, but we have different extent
        # items for this inode in both snapshots. We want to check this doesn't
        # cause send to fail with an error or produce an incorrect stream.
      
        $ xfs_io -r -c "dedupe /mnt/bar 0 0 512K" /mnt/mysnap1/foo
        $ xfs_io -r -c "dedupe /mnt/bar 512K 512K 512K" /mnt/mysnap2/foo
      
        # Create the incremental send stream.
        $ btrfs send -p /mnt/mysnap1 -f /tmp/2.snap /mnt/mysnap2
        ERROR: send ioctl failed with -5: Input/output error
      
      This issue started happening back in 2015 when deduplication was updated
      to not update the inode's ctime and mtime and update only the iversion.
      Back then we would hit a BUG_ON() in send, but later in 2016 send was
      updated to return -EIO and print the error message instead of doing the
      BUG_ON().
      
      A test case for fstests follows soon.
      
      Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=203933
      Fixes: 1c919a5e
      
       ("btrfs: don't update mtime/ctime on deduped inodes")
      CC: stable@vger.kernel.org # 4.4+
      Signed-off-by: default avatarFilipe Manana <fdmanana@suse.com>
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      b4f9a1a8
  6. 02 Jul, 2019 1 commit
    • Filipe Manana's avatar
      Btrfs: prevent send failures and crashes due to concurrent relocation · 9e967495
      Filipe Manana authored
      
      
      Send always operates on read-only trees and always expected that while it
      is in progress, nothing changes in those trees. Due to that expectation
      and the fact that send is a read-only operation, it operates on commit
      roots and does not hold transaction handles. However relocation can COW
      nodes and leafs from read-only trees, which can cause unexpected failures
      and crashes (hitting BUG_ONs). while send using a node/leaf, it gets
      COWed, the transaction used to COW it is committed, a new transaction
      starts, the extent previously used for that node/leaf gets allocated,
      possibly for another tree, and the respective extent buffer' content
      changes while send is still using it. When this happens send normally
      fails with EIO being returned to user space and messages like the
      following are found in dmesg/syslog:
      
        [ 3408.699121] BTRFS error (device sdc): parent transid verify failed on 58703872 wanted 250 found 253
        [ 3441.523123] BTRFS error (device sdc): did not find backref in send_root. inode=63211, offset=0, disk_byte=5222825984 found extent=5222825984
      
      Other times, less often, we hit a BUG_ON() because an extent buffer that
      send is using used to be a node, and while send is still using it, it
      got COWed and got reused as a leaf while send is still using, producing
      the following trace:
      
       [ 3478.466280] ------------[ cut here ]------------
       [ 3478.466282] kernel BUG at fs/btrfs/ctree.c:1806!
       [ 3478.466965] invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC PTI
       [ 3478.467635] CPU: 0 PID: 2165 Comm: btrfs Not tainted 5.0.0-btrfs-next-46 #1
       [ 3478.468311] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014
       [ 3478.469681] RIP: 0010:read_node_slot+0x122/0x130 [btrfs]
       (...)
       [ 3478.471758] RSP: 0018:ffffa437826bfaa0 EFLAGS: 00010246
       [ 3478.472457] RAX: ffff961416ed7000 RBX: 000000000000003d RCX: 0000000000000002
       [ 3478.473151] RDX: 000000000000003d RSI: ffff96141e387408 RDI: ffff961599b30000
       [ 3478.473837] RBP: ffffa437826bfb8e R08: 0000000000000001 R09: ffffa437826bfb8e
       [ 3478.474515] R10: ffffa437826bfa70 R11: 0000000000000000 R12: ffff9614385c8708
       [ 3478.475186] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
       [ 3478.475840] FS:  00007f8e0e9cc8c0(0000) GS:ffff9615b6a00000(0000) knlGS:0000000000000000
       [ 3478.476489] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
       [ 3478.477127] CR2: 00007f98b67a056e CR3: 0000000005df6005 CR4: 00000000003606f0
       [ 3478.477762] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
       [ 3478.478385] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
       [ 3478.479003] Call Trace:
       [ 3478.479600]  ? do_raw_spin_unlock+0x49/0xc0
       [ 3478.480202]  tree_advance+0x173/0x1d0 [btrfs]
       [ 3478.480810]  btrfs_compare_trees+0x30c/0x690 [btrfs]
       [ 3478.481388]  ? process_extent+0x1280/0x1280 [btrfs]
       [ 3478.481954]  btrfs_ioctl_send+0x1037/0x1270 [btrfs]
       [ 3478.482510]  _btrfs_ioctl_send+0x80/0x110 [btrfs]
       [ 3478.483062]  btrfs_ioctl+0x13fe/0x3120 [btrfs]
       [ 3478.483581]  ? rq_clock_task+0x2e/0x60
       [ 3478.484086]  ? wake_up_new_task+0x1f3/0x370
       [ 3478.484582]  ? do_vfs_ioctl+0xa2/0x6f0
       [ 3478.485075]  ? btrfs_ioctl_get_supported_features+0x30/0x30 [btrfs]
       [ 3478.485552]  do_vfs_ioctl+0xa2/0x6f0
       [ 3478.486016]  ? __fget+0x113/0x200
       [ 3478.486467]  ksys_ioctl+0x70/0x80
       [ 3478.486911]  __x64_sys_ioctl+0x16/0x20
       [ 3478.487337]  do_syscall_64+0x60/0x1b0
       [ 3478.487751]  entry_SYSCALL_64_after_hwframe+0x49/0xbe
       [ 3478.488159] RIP: 0033:0x7f8e0d7d4dd7
       (...)
       [ 3478.489349] RSP: 002b:00007ffcf6fb4908 EFLAGS: 00000202 ORIG_RAX: 0000000000000010
       [ 3478.489742] RAX: ffffffffffffffda RBX: 0000000000000105 RCX: 00007f8e0d7d4dd7
       [ 3478.490142] RDX: 00007ffcf6fb4990 RSI: 0000000040489426 RDI: 0000000000000005
       [ 3478.490548] RBP: 0000000000000005 R08: 00007f8e0d6f3700 R09: 00007f8e0d6f3700
       [ 3478.490953] R10: 00007f8e0d6f39d0 R11: 0000000000000202 R12: 0000000000000005
       [ 3478.491343] R13: 00005624e0780020 R14: 0000000000000000 R15: 0000000000000001
       (...)
       [ 3478.493352] ---[ end trace d5f537302be4f8c8 ]---
      
      Another possibility, much less likely to happen, is that send will not
      fail but the contents of the stream it produces may not be correct.
      
      To avoid this, do not allow send and relocation (balance) to run in
      parallel. In the long term the goal is to allow for both to be able to
      run concurrently without any problems, but that will take a significant
      effort in development and testing.
      Signed-off-by: default avatarFilipe Manana <fdmanana@suse.com>
      Reviewed-by: default avatarDavid Sterba <dsterba@suse.com>
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      9e967495
  7. 01 Jul, 2019 1 commit
  8. 28 May, 2019 2 commits
    • Filipe Manana's avatar
      Btrfs: incremental send, fix emission of invalid clone operations · 3c850b45
      Filipe Manana authored
      When doing an incremental send we can now issue clone operations with a
      source range that ends at the source's file eof and with a destination
      range that ends at an offset smaller then the destination's file eof.
      If the eof of the source file is not aligned to the sector size of the
      filesystem, the receiver will get a -EINVAL error when trying to do the
      operation or, on older kernels, silently corrupt the destination file.
      The corruption happens on kernels without commit ac765f83
      ("Btrfs: fix data corruption due to cloning of eof block"), while the
      failure to clone happens on kernels with that commit.
      
      Example reproducer:
      
        $ mkfs.btrfs -f /dev/sdb
        $ mount /dev/sdb /mnt/sdb
      
        $ xfs_io -f -c "pwrite -S 0xb1 0 2M" /mnt/sdb/foo
        $ xfs_io -f -c "pwrite -S 0xc7 0 2M" /mnt/sdb/bar
        $ xfs_io -f -c "pwrite -S 0x4d 0 2M" /mnt/sdb/baz
        $ xfs_io -f -c "pwrite -S 0xe2 0 2M" /mnt/sdb/zoo
      
        $ btrfs subvolume snapshot -r /mnt/sdb /mnt/sdb/base
      
        $ btrfs send -f /tmp/base.send /mnt/sdb/base
      
        $ xfs_io -c "reflink /mnt/sdb/bar 1560K 500K 100K" /mnt/sdb/bar
        $ xfs_io -c "reflink /mnt/sdb/bar 1560K 0 100K" /mnt/sdb/zoo
        $ xfs_io -c "truncate 550K" /mnt/sdb/bar
      
        $ btrfs subvolume snapshot -r /mnt/sdb /mnt/sdb/incr
      
        $ btrfs send -f /tmp/incr.send -p /mnt/sdb/base /mnt/sdb/incr
      
        $ mkfs.btrfs -f /dev/sdc
        $ mount /dev/sdc /mnt/sdc
      
        $ btrfs receive -f /tmp/base.send /mnt/sdc
        $ btrfs receive -vv -f /tmp/incr.send /mnt/sdc
        (...)
        truncate bar size=563200
        utimes bar
        clone zoo - source=bar source offset=512000 offset=0 length=51200
        ERROR: failed to clone extents to zoo
        Invalid argument
      
      The failure happens because the clone source range ends at the eof of file
      bar, 563200, which is not aligned to the filesystems sector size (4Kb in
      this case), and the destination range ends at offset 0 + 51200, which is
      less then the size of the file zoo (2Mb).
      
      So fix this by detecting such case and instead of issuing a clone
      operation for the whole range, do a clone operation for smaller range
      that is sector size aligned followed by a write operation for the block
      containing the eof. Here we will always be pessimistic and assume the
      destination filesystem of the send stream has the largest possible sector
      size (64Kb), since we have no way of determining it.
      
      This fixes a recent regression introduced in kernel 5.2-rc1.
      
      Fixes: 040ee612
      
       ("Btrfs: send, improve clone range")
      Signed-off-by: default avatarFilipe Manana <fdmanana@suse.com>
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      3c850b45
    • Filipe Manana's avatar
      Btrfs: incremental send, fix file corruption when no-holes feature is enabled · 6b1f72e5
      Filipe Manana authored
      When using the no-holes feature, if we have a file with prealloc extents
      with a start offset beyond the file's eof, doing an incremental send can
      cause corruption of the file due to incorrect hole detection. Such case
      requires that the prealloc extent(s) exist in both the parent and send
      snapshots, and that a hole is punched into the file that covers all its
      extents that do not cross the eof boundary.
      
      Example reproducer:
      
        $ mkfs.btrfs -f -O no-holes /dev/sdb
        $ mount /dev/sdb /mnt/sdb
      
        $ xfs_io -f -c "pwrite -S 0xab 0 500K" /mnt/sdb/foobar
        $ xfs_io -c "falloc -k 1200K 800K" /mnt/sdb/foobar
      
        $ btrfs subvolume snapshot -r /mnt/sdb /mnt/sdb/base
      
        $ btrfs send -f /tmp/base.snap /mnt/sdb/base
      
        $ xfs_io -c "fpunch 0 500K" /mnt/sdb/foobar
      
        $ btrfs subvolume snapshot -r /mnt/sdb /mnt/sdb/incr
      
        $ btrfs send -p /mnt/sdb/base -f /tmp/incr.snap /mnt/sdb/incr
      
        $ md5sum /mnt/sdb/incr/foobar
        816df6f64deba63b029ca19d880ee10a   /mnt/sdb/incr/foobar
      
        $ mkfs.btrfs -f /dev/sdc
        $ mount /dev/sdc /mnt/sdc
      
        $ btrfs receive -f /tmp/base.snap /mnt/sdc
        $ btrfs receive -f /tmp/incr.snap /mnt/sdc
      
        $ md5sum /mnt/sdc/incr/foobar
        cf2ef71f4a9e90c2f6013ba3b2257ed2   /mnt/sdc/incr/foobar
      
          --> Different checksum, because the prealloc extent beyond the
              file's eof confused the hole detection code and it assumed
              a hole starting at offset 0 and ending at the offset of the
              prealloc extent (1200Kb) instead of ending at the offset
              500Kb (the file's size).
      
      Fix this by ensuring we never cross the file's size when issuing the
      write operations for a hole.
      
      Fixes: 16e7549f
      
       ("Btrfs: incompatible format change to remove hole extents")
      CC: stable@vger.kernel.org # 3.14+
      Signed-off-by: default avatarFilipe Manana <fdmanana@suse.com>
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      6b1f72e5
  9. 29 Apr, 2019 3 commits
    • Filipe Manana's avatar
      Btrfs: fix race between send and deduplication that lead to failures and crashes · 62d54f3a
      Filipe Manana authored
      Send operates on read only trees and expects them to never change while it
      is using them. This is part of its initial design, and this expection is
      due to two different reasons:
      
      1) When it was introduced, no operations were allowed to modifiy read-only
         subvolumes/snapshots (including defrag for example).
      
      2) It keeps send from having an impact on other filesystem operations.
         Namely send does not need to keep locks on the trees nor needs to hold on
         to transaction handles and delay transaction commits. This ends up being
         a consequence of the former reason.
      
      However the deduplication feature was introduced later (on September 2013,
      while send was introduced in July 2012) and it allowed for deduplication
      with destination files that belong to read-only trees (subvolumes and
      snapshots).
      
      That means that having a send operation (either full or incremental) running
      in parallel with a deduplication that has the destination inode in one of
      the trees used by the send operation, can result in tree nodes and leaves
      getting freed and reused while send is using them. This problem is similar
      to the problem solved for the root nodes getting freed and reused when a
      snapshot is made against one tree that is currenly being used by a send
      operation, fixed in commits [1] and [2]. These commits explain in detail
      how the problem happens and the explanation is valid for any node or leaf
      that is not the root of a tree as well. This problem was also discussed
      and explained recently in a thread [3].
      
      The problem is very easy to reproduce when using send with large trees
      (snapshots) and just a few concurrent deduplication operations that target
      files in the trees used by send. A stress test case is being sent for
      fstests that triggers the issue easily. The most common error to hit is
      the send ioctl return -EIO with the following messages in dmesg/syslog:
      
       [1631617.204075] BTRFS error (device sdc): did not find backref in send_root. inode=63292, offset=0, disk_byte=5228134400 found extent=5228134400
       [1631633.251754] BTRFS error (device sdc): parent transid verify failed on 32243712 wanted 24 found 27
      
      The first one is very easy to hit while the second one happens much less
      frequently, except for very large trees (in that test case, snapshots
      with 100000 files having large xattrs to get deep and wide trees).
      Less frequently, at least one BUG_ON can be hit:
      
       [1631742.130080] ------------[ cut here ]------------
       [1631742.130625] kernel BUG at fs/btrfs/ctree.c:1806!
       [1631742.131188] invalid opcode: 0000 [#6] SMP DEBUG_PAGEALLOC PTI
       [1631742.131726] CPU: 1 PID: 13394 Comm: btrfs Tainted: G    B D W         5.0.0-rc8-btrfs-next-45 #1
       [1631742.132265] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014
       [1631742.133399] RIP: 0010:read_node_slot+0x122/0x130 [btrfs]
       (...)
       [1631742.135061] RSP: 0018:ffffb530021ebaa0 EFLAGS: 00010246
       [1631742.135615] RAX: ffff93ac8912e000 RBX: 000000000000009d RCX: 0000000000000002
       [1631742.136173] RDX: 000000000000009d RSI: ffff93ac564b0d08 RDI: ffff93ad5b48c000
       [1631742.136759] RBP: ffffb530021ebb7d R08: 0000000000000001 R09: ffffb530021ebb7d
       [1631742.137324] R10: ffffb530021eba70 R11: 0000000000000000 R12: ffff93ac87d0a708
       [1631742.137900] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000001
       [1631742.138455] FS:  00007f4cdb1528c0(0000) GS:ffff93ad76a80000(0000) knlGS:0000000000000000
       [1631742.139010] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
       [1631742.139568] CR2: 00007f5acb3d0420 CR3: 000000012be3e006 CR4: 00000000003606e0
       [1631742.140131] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
       [1631742.140719] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
       [1631742.141272] Call Trace:
       [1631742.141826]  ? do_raw_spin_unlock+0x49/0xc0
       [1631742.142390]  tree_advance+0x173/0x1d0 [btrfs]
       [1631742.142948]  btrfs_compare_trees+0x268/0x690 [btrfs]
       [1631742.143533]  ? process_extent+0x1070/0x1070 [btrfs]
       [1631742.144088]  btrfs_ioctl_send+0x1037/0x1270 [btrfs]
       [1631742.144645]  _btrfs_ioctl_send+0x80/0x110 [btrfs]
       [1631742.145161]  ? trace_sched_stick_numa+0xe0/0xe0
       [1631742.145685]  btrfs_ioctl+0x13fe/0x3120 [btrfs]
       [1631742.146179]  ? account_entity_enqueue+0xd3/0x100
       [1631742.146662]  ? reweight_entity+0x154/0x1a0
       [1631742.147135]  ? update_curr+0x20/0x2a0
       [1631742.147593]  ? check_preempt_wakeup+0x103/0x250
       [1631742.148053]  ? do_vfs_ioctl+0xa2/0x6f0
       [1631742.148510]  ? btrfs_ioctl_get_supported_features+0x30/0x30 [btrfs]
       [1631742.148942]  do_vfs_ioctl+0xa2/0x6f0
       [1631742.149361]  ? __fget+0x113/0x200
       [1631742.149767]  ksys_ioctl+0x70/0x80
       [1631742.150159]  __x64_sys_ioctl+0x16/0x20
       [1631742.150543]  do_syscall_64+0x60/0x1b0
       [1631742.150931]  entry_SYSCALL_64_after_hwframe+0x49/0xbe
       [1631742.151326] RIP: 0033:0x7f4cd9f5add7
       (...)
       [1631742.152509] RSP: 002b:00007ffe91017708 EFLAGS: 00000202 ORIG_RAX: 0000000000000010
       [1631742.152892] RAX: ffffffffffffffda RBX: 0000000000000105 RCX: 00007f4cd9f5add7
       [1631742.153268] RDX: 00007ffe91017790 RSI: 0000000040489426 RDI: 0000000000000007
       [1631742.153633] RBP: 0000000000000007 R08: 00007f4cd9e79700 R09: 00007f4cd9e79700
       [1631742.153999] R10: 00007f4cd9e799d0 R11: 0000000000000202 R12: 0000000000000003
       [1631742.154365] R13: 0000555dfae53020 R14: 0000000000000000 R15: 0000000000000001
       (...)
       [1631742.156696] ---[ end trace 5dac9f96dcc3fd6b ]---
      
      That BUG_ON happens because while send is using a node, that node is COWed
      by a concurrent deduplication, gets freed and gets reused as a leaf (because
      a transaction commit happened in between), so when it attempts to read a
      slot from the extent buffer, at ctree.c:read_node_slot(), the extent buffer
      contents were wiped out and it now matches a leaf (which can even belong to
      some other tree now), hitting the BUG_ON(level == 0).
      
      Fix this concurrency issue by not allowing send and deduplication to run
      in parallel if both operate on the same readonly trees, returning EAGAIN
      to user space and logging an exlicit warning in dmesg/syslog.
      
      [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=be6821f82c3cc36e026f5afd10249988852b35ea
      [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6f2f0b394b54e2b159ef969a0b5274e9bbf82ff2
      [3] https://lore.kernel.org/linux-btrfs/CAL3q7H7iqSEEyFaEtpRZw3cp613y+4k2Q8b4W7mweR3tZA05bQ@mail.gmail.com/
      
      
      
      CC: stable@vger.kernel.org # 4.4+
      Signed-off-by: default avatarFilipe Manana <fdmanana@suse.com>
      Reviewed-by: default avatarDavid Sterba <dsterba@suse.com>
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      62d54f3a
    • Filipe Manana's avatar
      Btrfs: send, flush dellaloc in order to avoid data loss · 9f89d5de
      Filipe Manana authored
      When we set a subvolume to read-only mode we do not flush dellaloc for any
      of its inodes (except if the filesystem is mounted with -o flushoncommit),
      since it does not affect correctness for any subsequent operations - except
      for a future send operation. The send operation will not be able to see the
      delalloc data since the respective file extent items, inode item updates,
      backreferences, etc, have not hit yet the subvolume and extent trees.
      
      Effectively this means data loss, since the send stream will not contain
      any data from existing delalloc. Another problem from this is that if the
      writeback starts and finishes while the send operation is in progress, we
      have the subvolume tree being being modified concurrently which can result
      in send failing unexpectedly with EIO or hitting runtime errors, assertion
      failures or hitting BUG_ONs, etc.
      
      Simple reproducer:
      
        $ mkfs.btrfs -f /dev/sdb
        $ mount /dev/sdb /mnt
      
        $ btrfs subvolume create /mnt/sv
        $ xfs_io -f -c "pwrite -S 0xea 0 108K" /mnt/sv/foo
      
        $ btrfs property set /mnt/sv ro true
        $ btrfs send -f /tmp/send.stream /mnt/sv
      
        $ od -t x1 -A d /mnt/sv/foo
        0000000 ea ea ea ea ea ea ea ea ea ea ea ea ea ea ea ea
        *
        0110592
      
        $ umount /mnt
        $ mkfs.btrfs -f /dev/sdc
        $ mount /dev/sdc /mnt
      
        $ btrfs receive -f /tmp/send.stream /mnt
        $ echo $?
        0
        $ od -t x1 -A d /mnt/sv/foo
        0000000
        # ---> empty file
      
      Since this a problem that affects send only, fix it in send by flushing
      dellaloc for all the roots used by the send operation before send starts
      to process the commit roots.
      
      This is a problem that affects send since it was introduced (commit
      31db9f7c ("Btrfs: introduce BTRFS_IOC_SEND for btrfs send/receive"))
      but backporting it to older kernels has some dependencies:
      
      - For kernels between 3.19 and 4.20, it depends on commit 3cd24c69
        ("btrfs: use tagged writepage to mitigate livelock of snapshot") because
        the function btrfs_start_delalloc_snapshot() does not exist before that
        commit. So one has to either pick that commit or replace the calls to
        btrfs_start_delalloc_snapshot() in this patch with calls to
        btrfs_start_delalloc_inodes().
      
      - For kernels older than 3.19 it also requires commit e5fa8f86
      
      
        ("Btrfs: ensure send always works on roots without orphans") because
        it depends on the function ensure_commit_roots_uptodate() which that
        commits introduced.
      
      - No dependencies for 5.0+ kernels.
      
      A test case for fstests follows soon.
      
      CC: stable@vger.kernel.org # 3.19+
      Signed-off-by: default avatarFilipe Manana <fdmanana@suse.com>
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      9f89d5de
    • Robbie Ko's avatar
      Btrfs: send, improve clone range · 040ee612
      Robbie Ko authored
      
      
      Improve clone_range in two scenarios.
      
      1. Remove the limit of inode size when find clone inodes We can do
         partial clone, so there is no need to limit the size of the candidate
         inode.  When clone a range, we clone the legal range only by bytenr,
         offset, len, inode size.
      
      2. In the scenarios of rewrite or clone_range, data_offset rarely
         matches exactly, so the chance of a clone is missed.
      
      e.g.
          1. Write a 1M file
              dd if=/dev/zero of=1M bs=1M count=1
      
          2. Clone 1M file
             cp --reflink 1M clone
      
          3. Rewrite 4k on the clone file
             dd if=/dev/zero of=clone bs=4k count=1 conv=notrunc
      
          The disk layout is as follows:
          item 16 key (257 EXTENT_DATA 0) itemoff 15353 itemsize 53
      	extent data disk byte 1103101952 nr 1048576
      	extent data offset 0 nr 1048576 ram 1048576
      	extent compression(none)
          ...
          item 22 key (258 EXTENT_DATA 0) itemoff 14959 itemsize 53
      	extent data disk byte 1104150528 nr 4096
      	extent data offset 0 nr 4096 ram 4096
      	extent compression(none)
          item 23 key (258 EXTENT_DATA 4096) itemoff 14906 itemsize 53
      	extent data disk byte 1103101952 nr 1048576
      	extent data offset 4096 nr 1044480 ram 1048576
      	extent compression(none)
      
      When send, inode 258 file offset 4096~1048576 (item 23) has a chance to
      clone_range, but because data_offset does not match inode 257 (item 16),
      it causes missed clone and can only transfer actual data.
      
      Improve the problem by judging whether the current data_offset has
      overlap with the file extent item, and if so, adjusting offset and
      extent_len so that we can clone correctly.
      Reviewed-by: default avatarFilipe Manana <fdmanana@suse.com>
      Signed-off-by: default avatarRobbie Ko <robbieko@synology.com>
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      040ee612
  10. 04 Jan, 2019 1 commit
    • Linus Torvalds's avatar
      Remove 'type' argument from access_ok() function · 96d4f267
      Linus Torvalds authored
      
      
      Nobody has actually used the type (VERIFY_READ vs VERIFY_WRITE) argument
      of the user address range verification function since we got rid of the
      old racy i386-only code to walk page tables by hand.
      
      It existed because the original 80386 would not honor the write protect
      bit when in kernel mode, so you had to do COW by hand before doing any
      user access.  But we haven't supported that in a long time, and these
      days the 'type' argument is a purely historical artifact.
      
      A discussion about extending 'user_access_begin()' to do the range
      checking resulted this patch, because there is no way we're going to
      move the old VERIFY_xyz interface to that model.  And it's best done at
      the end of the merge window when I've done most of my merges, so let's
      just get this done once and for all.
      
      This patch was mostly done with a sed-script, with manual fix-ups for
      the cases that weren't of the trivial 'access_ok(VERIFY_xyz' form.
      
      There were a couple of notable cases:
      
       - csky still had the old "verify_area()" name as an alias.
      
       - the iter_iov code had magical hardcoded knowledge of the actual
         values of VERIFY_{READ,WRITE} (not that they mattered, since nothing
         really used it)
      
       - microblaze used the type argument for a debug printout
      
      but other than those oddities this should be a total no-op patch.
      
      I tried to fix up all architectures, did fairly extensive grepping for
      access_ok() uses, and the changes are trivial, but I may have missed
      something.  Any missed conversion should be trivially fixable, though.
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      96d4f267
  11. 17 Dec, 2018 2 commits
  12. 21 Nov, 2018 1 commit
    • Robbie Ko's avatar
      Btrfs: send, fix infinite loop due to directory rename dependencies · a4390aee
      Robbie Ko authored
      
      
      When doing an incremental send, due to the need of delaying directory move
      (rename) operations we can end up in infinite loop at
      apply_children_dir_moves().
      
      An example scenario that triggers this problem is described below, where
      directory names correspond to the numbers of their respective inodes.
      
      Parent snapshot:
      
       .
       |--- 261/
             |--- 271/
                   |--- 266/
                         |--- 259/
                         |--- 260/
                         |     |--- 267
                         |
                         |--- 264/
                         |     |--- 258/
                         |           |--- 257/
                         |
                         |--- 265/
                         |--- 268/
                         |--- 269/
                         |     |--- 262/
                         |
                         |--- 270/
                         |--- 272/
                         |     |--- 263/
                         |     |--- 275/
                         |
                         |--- 274/
                               |--- 273/
      
      Send snapshot:
      
       .
       |-- 275/
            |-- 274/
                 |-- 273/
                      |-- 262/
                           |-- 269/
                                |-- 258/
                                     |-- 271/
                                          |-- 268/
                                               |-- 267/
                                                    |-- 270/
                                                         |-- 259/
                                                         |    |-- 265/
                                                         |
                                                         |-- 272/
                                                              |-- 257/
                                                                   |-- 260/
                                                                   |-- 264/
                                                                        |-- 263/
                                                                             |-- 261/
                                                                                  |-- 266/
      
      When processing inode 257 we delay its move (rename) operation because its
      new parent in the send snapshot, inode 272, was not yet processed. Then
      when processing inode 272, we delay the move operation for that inode
      because inode 274 is its ancestor in the send snapshot. Finally we delay
      the move operation for inode 274 when processing it because inode 275 is
      its new parent in the send snapshot and was not yet moved.
      
      When finishing processing inode 275, we start to do the move operations
      that were previously delayed (at apply_children_dir_moves()), resulting in
      the following iterations:
      
      1) We issue the move operation for inode 274;
      
      2) Because inode 262 depended on the move operation of inode 274 (it was
         delayed because 274 is its ancestor in the send snapshot), we issue the
         move operation for inode 262;
      
      3) We issue the move operation for inode 272, because it was delayed by
         inode 274 too (ancestor of 272 in the send snapshot);
      
      4) We issue the move operation for inode 269 (it was delayed by 262);
      
      5) We issue the move operation for inode 257 (it was delayed by 272);
      
      6) We issue the move operation for inode 260 (it was delayed by 272);
      
      7) We issue the move operation for inode 258 (it was delayed by 269);
      
      8) We issue the move operation for inode 264 (it was delayed by 257);
      
      9) We issue the move operation for inode 271 (it was delayed by 258);
      
      10) We issue the move operation for inode 263 (it was delayed by 264);
      
      11) We issue the move operation for inode 268 (it was delayed by 271);
      
      12) We verify if we can issue the move operation for inode 270 (it was
          delayed by 271). We detect a path loop in the current state, because
          inode 267 needs to be moved first before we can issue the move
          operation for inode 270. So we delay again the move operation for
          inode 270, this time we will attempt to do it after inode 267 is
          moved;
      
      13) We issue the move operation for inode 261 (it was delayed by 263);
      
      14) We verify if we can issue the move operation for inode 266 (it was
          delayed by 263). We detect a path loop in the current state, because
          inode 270 needs to be moved first before we can issue the move
          operation for inode 266. So we delay again the move operation for
          inode 266, this time we will attempt to do it after inode 270 is
          moved (its move operation was delayed in step 12);
      
      15) We issue the move operation for inode 267 (it was delayed by 268);
      
      16) We verify if we can issue the move operation for inode 266 (it was
          delayed by 270). We detect a path loop in the current state, because
          inode 270 needs to be moved first before we can issue the move
          operation for inode 266. So we delay again the move operation for
          inode 266, this time we will attempt to do it after inode 270 is
          moved (its move operation was delayed in step 12). So here we added
          again the same delayed move operation that we added in step 14;
      
      17) We attempt again to see if we can issue the move operation for inode
          266, and as in step 16, we realize we can not due to a path loop in
          the current state due to a dependency on inode 270. Again we delay
          inode's 266 rename to happen after inode's 270 move operation, adding
          the same dependency to the empty stack that we did in steps 14 and 16.
          The next iteration will pick the same move dependency on the stack
          (the only entry) and realize again there is still a path loop and then
          again the same dependency to the stack, over and over, resulting in
          an infinite loop.
      
      So fix this by preventing adding the same move dependency entries to the
      stack by removing each pending move record from the red black tree of
      pending moves. This way the next call to get_pending_dir_moves() will
      not return anything for the current parent inode.
      
      A test case for fstests, with this reproducer, follows soon.
      Signed-off-by: default avatarRobbie Ko <robbieko@synology.com>
      Reviewed-by: default avatarFilipe Manana <fdmanana@suse.com>
      [Wrote changelog with example and more clear explanation]
      Signed-off-by: default avatarFilipe Manana <fdmanana@suse.com>
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      a4390aee
  13. 15 Oct, 2018 2 commits
  14. 06 Aug, 2018 4 commits
    • Filipe Manana's avatar
      Btrfs: send, fix incorrect file layout after hole punching beyond eof · 22d3151c
      Filipe Manana authored
      When doing an incremental send, if we have a file in the parent snapshot
      that has prealloc extents beyond EOF and in the send snapshot it got a
      hole punch that partially covers the prealloc extents, the send stream,
      when replayed by a receiver, can result in a file that has a size bigger
      than it should and filled with zeroes past the correct EOF.
      
      For example:
      
        $ mkfs.btrfs -f /dev/sdb
        $ mount /dev/sdb /mnt
      
        $ xfs_io -f -c "falloc -k 0 4M" /mnt/foobar
        $ xfs_io -c "pwrite -S 0xea 0 1M" /mnt/foobar
      
        $ btrfs subvolume snapshot -r /mnt /mnt/snap1
        $ btrfs send -f /tmp/1.send /mnt/snap1
      
        $ xfs_io -c "fpunch 1M 2M" /mnt/foobar
      
        $ btrfs subvolume snapshot -r /mnt /mnt/snap2
        $ btrfs send -f /tmp/2.send -p /mnt/snap1 /mnt/snap2
      
        $ stat --format %s /mnt/snap2/foobar
        1048576
        $ md5sum /mnt/snap2/foobar
        d31659e82e87798acd4669a1e0a19d4f  /mnt/snap2/foobar
      
        $ umount /mnt
        $ mkfs.btrfs -f /dev/sdc
        $ mount /dev/sdc /mnt
      
        $ btrfs receive -f /mnt/1.snap /mnt
        $ btrfs receive -f /mnt/2.snap /mnt
      
        $ stat --format %s /mnt/snap2/foobar
        3145728
        # --> should be 1Mb and not 3Mb (which was the end offset of hole
        #     punch operation)
        $ md5sum /mnt/snap2/foobar
        117baf295297c2a995f92da725b0b651  /mnt/snap2/foobar
        # --> should be d31659e82e87798acd4669a1e0a19d4f as in the original fs
      
      This issue actually happens only since commit ffa7c429 ("Btrfs: send,
      do not issue unnecessary truncate operations"), but before that commit we
      were issuing a write operation full of zeroes (to "punch" a hole) which
      was extending the file size beyond the correct value and then immediately
      issue a truncate operation to the correct size and undoing the previous
      write operation. Since the send protocol does not support fallocate, for
      extent preallocation and hole punching, fix this by not even attempting
      to send a "hole" (regular write full of zeroes) if it starts at an offset
      greater then or equals to the file's size. This approach, besides being
      much more simple then making send issue the truncate operation, adds the
      benefit of avoiding the useless pair of write of zeroes and truncate
      operations, saving time and IO at the receiver and reducing the size of
      the send stream.
      
      A test case for fstests follows soon.
      
      Fixes: ffa7c429
      
       ("Btrfs: send, do not issue unnecessary truncate operations")
      CC: stable@vger.kernel.org # 4.17+
      Signed-off-by: default avatarFilipe Manana <fdmanana@suse.com>
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      22d3151c
    • Filipe Manana's avatar
      Btrfs: fix send failure when root has deleted files still open · 46b2f459
      Filipe Manana authored
      
      
      The more common use case of send involves creating a RO snapshot and then
      use it for a send operation. In this case it's not possible to have inodes
      in the snapshot that have a link count of zero (inode with an orphan item)
      since during snapshot creation we do the orphan cleanup. However, other
      less common use cases for send can end up seeing inodes with a link count
      of zero and in this case the send operation fails with a ENOENT error
      because any attempt to generate a path for the inode, with the purpose
      of creating it or updating it at the receiver, fails since there are no
      inode reference items. One use case it to use a regular subvolume for
      a send operation after turning it to RO mode or turning a RW snapshot
      into RO mode and then using it for a send operation. In both cases, if a
      file gets all its hard links deleted while there is an open file
      descriptor before turning the subvolume/snapshot into RO mode, the send
      operation will encounter an inode with a link count of zero and then
      fail with errno ENOENT.
      
      Example using a full send with a subvolume:
      
        $ mkfs.btrfs -f /dev/sdb
        $ mount /dev/sdb /mnt
      
        $ btrfs subvolume create /mnt/sv1
        $ touch /mnt/sv1/foo
        $ touch /mnt/sv1/bar
      
        # keep an open file descriptor on file bar
        $ exec 73</mnt/sv1/bar
        $ unlink /mnt/sv1/bar
      
        # Turn the subvolume to RO mode and use it for a full send, while
        # holding the open file descriptor.
        $ btrfs property set /mnt/sv1 ro true
      
        $ btrfs send -f /tmp/full.send /mnt/sv1
        At subvol /mnt/sv1
        ERROR: send ioctl failed with -2: No such file or directory
      
      Example using an incremental send with snapshots:
      
        $ mkfs.btrfs -f /dev/sdb
        $ mount /dev/sdb /mnt
      
        $ btrfs subvolume create /mnt/sv1
        $ touch /mnt/sv1/foo
        $ touch /mnt/sv1/bar
      
        $ btrfs subvolume snapshot -r /mnt/sv1 /mnt/snap1
      
        $ echo "hello world" >> /mnt/sv1/bar
      
        $ btrfs subvolume snapshot -r /mnt/sv1 /mnt/snap2
      
        # Turn the second snapshot to RW mode and delete file foo while
        # holding an open file descriptor on it.
        $ btrfs property set /mnt/snap2 ro false
        $ exec 73</mnt/snap2/foo
        $ unlink /mnt/snap2/foo
      
        # Set the second snapshot back to RO mode and do an incremental send.
        $ btrfs property set /mnt/snap2 ro true
      
        $ btrfs send -f /tmp/inc.send -p /mnt/snap1 /mnt/snap2
        At subvol /mnt/snap2
        ERROR: send ioctl failed with -2: No such file or directory
      
      So fix this by ignoring inodes with a link count of zero if we are either
      doing a full send or if they do not exist in the parent snapshot (they
      are new in the send snapshot), and unlink all paths found in the parent
      snapshot when doing an incremental send (and ignoring all other inode
      items, such as xattrs and extents).
      
      A test case for fstests follows soon.
      
      CC: stable@vger.kernel.org # 4.4+
      Reported-by: default avatarMartin Wilck <martin.wilck@suse.com>
      Signed-off-by: default avatarFilipe Manana <fdmanana@suse.com>
      Reviewed-by: default avatarDavid Sterba <dsterba@suse.com>
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      46b2f459
    • Filipe Manana's avatar
      Btrfs: remove unused key assignment when doing a full send · ca5d2ba1
      Filipe Manana authored
      At send.c:full_send_tree() we were setting the 'key' variable in the loop
      while never using it later. We were also using two btrfs_key variables
      to store the initial key for search and the key found in every iteration
      of the loop. So remove this useless key assignment and use the same
      btrfs_key variable to store the initial search key and the key found in
      each iteration. This was introduced in the initial send commit but was
      never used (commit 31db9f7c
      
       ("Btrfs: introduce BTRFS_IOC_SEND for
      btrfs send/receive").
      Signed-off-by: default avatarFilipe Manana <fdmanana@suse.com>
      Reviewed-by: default avatarDavid Sterba <dsterba@suse.com>
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      ca5d2ba1
    • Qu Wenruo's avatar
      btrfs: Get rid of the confusing btrfs_file_extent_inline_len · e41ca589
      Qu Wenruo authored
      
      
      We used to call btrfs_file_extent_inline_len() to get the uncompressed
      data size of an inlined extent.
      
      However this function is hiding evil, for compressed extent, it has no
      choice but to directly read out ram_bytes from btrfs_file_extent_item.
      While for uncompressed extent, it uses item size to calculate the real
      data size, and ignoring ram_bytes completely.
      
      In fact, for corrupted ram_bytes, due to above behavior kernel
      btrfs_print_leaf() can't even print correct ram_bytes to expose the bug.
      
      Since we have the tree-checker to verify all EXTENT_DATA, such mismatch
      can be detected pretty easily, thus we can trust ram_bytes without the
      evil btrfs_file_extent_inline_len().
      Signed-off-by: default avatarQu Wenruo <wqu@suse.com>
      Reviewed-by: default avatarDavid Sterba <dsterba@suse.com>
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      e41ca589
  15. 28 May, 2018 3 commits
  16. 02 May, 2018 1 commit
    • Filipe Manana's avatar
      Btrfs: send, fix missing truncate for inode with prealloc extent past eof · a6aa10c7
      Filipe Manana authored
      
      
      An incremental send operation can miss a truncate operation when an inode
      has an increased size in the send snapshot and a prealloc extent beyond
      its size.
      
      Consider the following scenario where a necessary truncate operation is
      missing in the incremental send stream:
      
      1) In the parent snapshot an inode has a size of 1282957 bytes and it has
         no prealloc extents beyond its size;
      
      2) In the the send snapshot it has a size of 5738496 bytes and has a new
         extent at offsets 1884160 (length of 106496 bytes) and a prealloc
         extent beyond eof at offset 6729728 (and a length of 339968 bytes);
      
      3) When processing the prealloc extent, at offset 6729728, we end up at
         send.c:send_write_or_clone() and set the @len variable to a value of
         18446744073708560384 because @offset plus the original @len value is
         larger then the inode's size (6729728 + 339968 > 5738496). We then
         call send_extent_data(), with that @offset and @len, which in turn
         calls send_write(), and then the later calls fill_read_buf(). Because
         the offset passed to fill_read_buf() is greater then inode's i_size,
         this function returns 0 immediately, which makes send_write() and
         send_extent_data() do nothing and return immediately as well. When
         we get back to send.c:send_write_or_clone() we adjust the value
         of sctx->cur_inode_next_write_offset to @offset plus @len, which
         corresponds to 6729728 + 18446744073708560384 = 5738496, which is
         precisely the the size of the inode in the send snapshot;
      
      4) Later when at send.c:finish_inode_if_needed() we determine that
         we don't need to issue a truncate operation because the value of
         sctx->cur_inode_next_write_offset corresponds to the inode's new
         size, 5738496 bytes. This is wrong because the last write operation
         that was issued started at offset 1884160 with a length of 106496
         bytes, so the correct value for sctx->cur_inode_next_write_offset
         should be 1990656 (1884160 + 106496), so that a truncate operation
         with a value of 5738496 bytes would have been sent to insert a
         trailing hole at the destination.
      
      So fix the issue by making send.c:send_write_or_clone() not attempt
      to send write or clone operations for extents that start beyond the
      inode's size, since such attempts do nothing but waste time by
      calling helper functions and allocating path structures, and send
      currently has no fallocate command in order to create prealloc extents
      at the destination (either beyond a file's eof or not).
      
      The issue was found running the test btrfs/007 from fstests using a seed
      value of 1524346151 for fsstress.
      Reported-by: default avatarGu, Jinxiang <gujx@cn.fujitsu.com>
      Fixes: ffa7c429
      
       ("Btrfs: send, do not issue unnecessary truncate operations")
      Signed-off-by: default avatarFilipe Manana <fdmanana@suse.com>
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      a6aa10c7
  17. 12 Apr, 2018 1 commit
  18. 26 Mar, 2018 4 commits
    • Liu Bo's avatar
      Btrfs: send: fix typo in TLV_PUT · 895a72be
      Liu Bo authored
      
      
      According to tlv_put()'s prototype, data and attrlen needs to be
      exchanged in the macro, but seems all callers are already aware of
      this misorder and are therefore not affected.
      Signed-off-by: default avatarLiu Bo <bo.li.liu@oracle.com>
      Reviewed-by: default avatarDavid Sterba <dsterba@suse.com>
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      895a72be
    • Filipe Manana's avatar
      Btrfs: send, do not issue unnecessary truncate operations · ffa7c429
      Filipe Manana authored
      
      
      When send finishes processing an inode representing a regular file, it
      always issues a truncate operation for that file, even if its size did
      not change or the last write sets the file size correctly. In the most
      common cases, the issued write operations set the file to correct size
      (either full or incremental sends) or the file size did not change (for
      incremental sends), so the only case where a truncate operation is needed
      is when a file size becomes smaller in the send snapshot when compared
      to the parent snapshot.
      
      By not issuing unnecessary truncate operations we reduce the stream size
      and save time in the receiver. Currently truncating a file to the same
      size triggers writeback of its last page (if it's dirty) and waits for it
      to complete (only if the file size is not aligned with the filesystem's
      sector size). This is being fixed by another patch and is independent of
      this change (that patch's title is "Btrfs: skip writeback of last page
      when truncating file to same size").
      
      The following script was used to measure time spent by a receiver without
      this change applied, with this change applied, and without this change and
      with the truncate fix applied (the fix to not make it start and wait for
      writeback to complete).
      
        $ cat test_send.sh
        #!/bin/bash
      
        SRC_DEV=/dev/sdc
        DST_DEV=/dev/sdd
        SRC_MNT=/mnt/sdc
        DST_MNT=/mnt/sdd
      
        mkfs.btrfs -f $SRC_DEV >/dev/null
        mkfs.btrfs -f $DST_DEV >/dev/null
        mount $SRC_DEV $SRC_MNT
        mount $DST_DEV $DST_MNT
      
        echo "Creating source filesystem"
        for ((t = 0; t < 10; t++)); do
            (
                for ((i = 1; i <= 20000; i++)); do
                    xfs_io -f -c "pwrite -S 0xab 0 5000" \
                        $SRC_MNT/file_$i > /dev/null
                done
            ) &
           worker_pids[$t]=$!
        done
        wait ${worker_pids[@]}
      
        echo "Creating and sending snapshot"
        btrfs subvolume snapshot -r $SRC_MNT $SRC_MNT/snap1 >/dev/null
        /usr/bin/time -f "send took %e seconds"    \
               btrfs send -f $SRC_MNT/send_file $SRC_MNT/snap1
        /usr/bin/time -f "receive took %e seconds" \
               btrfs receive -f $SRC_MNT/send_file $DST_MNT
      
        umount $SRC_MNT
        umount $DST_MNT
      
      The results, which are averages for 5 runs for each case, were the
      following:
      
      * Without this change
      
      average receive time was 26.49 seconds
      standard deviation of 2.53 seconds
      
      * Without this change and with the truncate fix
      
      average receive time was 12.51 seconds
      standard deviation of 0.32 seconds
      
      * With this change and without the truncate fix
      
      average receive time was 10.02 seconds
      standard deviation of 1.11 seconds
      Signed-off-by: default avatarFilipe Manana <fdmanana@suse.com>
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      ffa7c429
    • David Sterba's avatar
      btrfs: add more __cold annotations · e67c718b
      David Sterba authored
      
      
      The __cold functions are placed to a special section, as they're
      expected to be called rarely. This could help i-cache prefetches or help
      compiler to decide which branches are more/less likely to be taken
      without any other annotations needed.
      
      Though we can't add more __exit annotations, it's still possible to add
      __cold (that's also added with __exit). That way the following function
      categories are tagged:
      
      - printf wrappers, error messages
      - exit helpers
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      e67c718b
    • Nikolay Borisov's avatar
      btrfs: Remove custom crc32c init code · 9678c543
      Nikolay Borisov authored
      The custom crc32 init code was introduced in
      14a958e6 ("Btrfs: fix btrfs boot when compiled as built-in") to
      enable using btrfs as a built-in. However, later as pointed out by
      60efa5eb
      
       ("Btrfs: use late_initcall instead of module_init") this
      wasn't enough and finally btrfs was switched to late_initcall which
      comes after the generic crc32c implementation is initiliased. The
      latter commit superseeded the former. Now that we don't have to
      maintain our own code let's just remove it and switch to using the
      generic implementation.
      
      Despite touching a lot of files the patch is really simple. Here is the gist of
      the changes:
      
      1. Select LIBCRC32C rather than the low-level modules.
      2. s/btrfs_crc32c/crc32c/g
      3. replace hash.h with linux/crc32c.h
      4. Move the btrfs namehash funcs to ctree.h and change the tree accordingly.
      
      I've tested this with btrfs being both a module and a built-in and xfstest
      doesn't complain.
      
      Does seem to fix the longstanding problem of not automatically selectiong
      the crc32c module when btrfs is used. Possibly there is a workaround in
      dracut.
      
      The modinfo confirms that now all the module dependencies are there:
      
      before:
      depends:        zstd_compress,zstd_decompress,raid6_pq,xor,zlib_deflate
      
      after:
      depends:        libcrc32c,zstd_compress,zstd_decompress,raid6_pq,xor,zlib_deflate
      Signed-off-by: default avatarNikolay Borisov <nborisov@suse.com>
      Reviewed-by: default avatarDavid Sterba <dsterba@suse.com>
      [ add more info to changelog from mails ]
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      9678c543
  19. 01 Mar, 2018 1 commit
    • Filipe Manana's avatar
      Btrfs: send, fix issuing write op when processing hole in no data mode · d4dfc0f4
      Filipe Manana authored
      
      
      When doing an incremental send of a filesystem with the no-holes feature
      enabled, we end up issuing a write operation when using the no data mode
      send flag, instead of issuing an update extent operation. Fix this by
      issuing the update extent operation instead.
      
      Trivial reproducer:
      
        $ mkfs.btrfs -f -O no-holes /dev/sdc
        $ mkfs.btrfs -f /dev/sdd
        $ mount /dev/sdc /mnt/sdc
        $ mount /dev/sdd /mnt/sdd
      
        $ xfs_io -f -c "pwrite -S 0xab 0 32K" /mnt/sdc/foobar
        $ btrfs subvolume snapshot -r /mnt/sdc /mnt/sdc/snap1
      
        $ xfs_io -c "fpunch 8K 8K" /mnt/sdc/foobar
        $ btrfs subvolume snapshot -r /mnt/sdc /mnt/sdc/snap2
      
        $ btrfs send /mnt/sdc/snap1 | btrfs receive /mnt/sdd
        $ btrfs send --no-data -p /mnt/sdc/snap1 /mnt/sdc/snap2 \
             | btrfs receive -vv /mnt/sdd
      
      Before this change the output of the second receive command is:
      
        receiving snapshot snap2 uuid=f6922049-8c22-e544-9ff9-fc6755918447...
        utimes
        write foobar, offset 8192, len 8192
        utimes foobar
        BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=f6922049-8c22-e544-9ff9-...
      
      After this change it is:
      
        receiving snapshot snap2 uuid=564d36a3-ebc8-7343-aec9-bf6fda278e64...
        utimes
        update_extent foobar: offset=8192, len=8192
        utimes foobar
        BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=564d36a3-ebc8-7343-aec9-bf6fda278e64...
      Signed-off-by: default avatarFilipe Manana <fdmanana@suse.com>
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      d4dfc0f4
  20. 22 Jan, 2018 1 commit
  21. 28 Nov, 2017 1 commit
    • Filipe Manana's avatar
      Btrfs: incremental send, fix wrong unlink path after renaming file · ea37d599
      Filipe Manana authored
      
      
      Under some circumstances, an incremental send operation can issue wrong
      paths for unlink commands related to files that have multiple hard links
      and some (or all) of those links were renamed between the parent and send
      snapshots. Consider the following example:
      
      Parent snapshot
      
       .                                                      (ino 256)
       |---- a/                                               (ino 257)
       |     |---- b/                                         (ino 259)
       |     |     |---- c/                                   (ino 260)
       |     |     |---- f2                                   (ino 261)
       |     |
       |     |---- f2l1                                       (ino 261)
       |
       |---- d/                                               (ino 262)
             |---- f1l1_2                                     (ino 258)
             |---- f2l2                                       (ino 261)
             |---- f1_2                                       (ino 258)
      
      Send snapshot
      
       .                                                      (ino 256)
       |---- a/                                               (ino 257)
       |     |---- f2l1/                                      (ino 263)
       |             |---- b2/                                (ino 259)
       |                   |---- c/                           (ino 260)
       |                   |     |---- d3                     (ino 262)
       |                   |           |---- f1l1_2           (ino 258)
       |                   |           |---- f2l2_2           (ino 261)
       |                   |           |---- f1_2             (ino 258)
       |                   |
       |                   |---- f2                           (ino 261)
       |                   |---- f1l2                         (ino 258)
       |
       |---- d                                                (ino 261)
      
      When computing the incremental send stream the following steps happen:
      
      1) When processing inode 261, a rename operation is issued that renames
         inode 262, which currently as a path of "d", to an orphan name of
         "o262-7-0". This is done because in the send snapshot, inode 261 has
         of its hard links with a path of "d" as well.
      
      2) Two link operations are issued that create the new hard links for
         inode 261, whose names are "d" and "f2l2_2", at paths "/" and
         "o262-7-0/" respectively.
      
      3) Still while processing inode 261, unlink operations are issued to
         remove the old hard links of inode 261, with names "f2l1" and "f2l2",
         at paths "a/" and "d/". However path "d/" does not correspond anymore
         to the directory inode 262 but corresponds instead to a hard link of
         inode 261 (link command issued in the previous step). This makes the
         receiver fail with a ENOTDIR error when attempting the unlink
         operation.
      
      The problem happens because before sending the unlink operation, we failed
      to detect that inode 262 was one of ancestors for inode 261 in the parent
      snapshot, and therefore we didn't recompute the path for inode 262 before
      issuing the unlink operation for the link named "f2l2" of inode 262. The
      detection failed because the function "is_ancestor()" only follows the
      first hard link it finds for an inode instead of all of its hard links
      (as it was originally created for being used with directories only, for
      which only one hard link exists). So fix this by making "is_ancestor()"
      follow all hard links of the input inode.
      
      A test case for fstests follows soon.
      Signed-off-by: default avatarFilipe Manana <fdmanana@suse.com>
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      ea37d599
  22. 01 Nov, 2017 2 commits
    • Zygo Blaxell's avatar
      btrfs: add a flag to iterate_inodes_from_logical to find all extent refs for uncompressed extents · c995ab3c
      Zygo Blaxell authored
      
      
      The LOGICAL_INO ioctl provides a backward mapping from extent bytenr and
      offset (encoded as a single logical address) to a list of extent refs.
      LOGICAL_INO complements TREE_SEARCH, which provides the forward mapping
      (extent ref -> extent bytenr and offset, or logical address).  These are
      useful capabilities for programs that manipulate extents and extent
      references from userspace (e.g. dedup and defrag utilities).
      
      When the extents are uncompressed (and not encrypted and not other),
      check_extent_in_eb performs filtering of the extent refs to remove any
      extent refs which do not contain the same extent offset as the 'logical'
      parameter's extent offset.  This prevents LOGICAL_INO from returning
      references to more than a single block.
      
      To find the set of extent references to an uncompressed extent from [a, b),
      userspace has to run a loop like this pseudocode:
      
      	for (i = a; i < b; ++i)
      		extent_ref_set += LOGICAL_INO(i);
      
      At each iteration of the loop (up to 32768 iterations for a 128M extent),
      data we are interested in is collected in the kernel, then deleted by
      the filter in check_extent_in_eb.
      
      When the extents are compressed (or encrypted or other), the 'logical'
      parameter must be an extent bytenr (the 'a' parameter in the loop).
      No filtering by extent offset is done (or possible?) so the result is
      the complete set of extent refs for the entire extent.  This removes
      the need for the loop, since we get all the extent refs in one call.
      
      Add an 'ignore_offset' argument to iterate_inodes_from_logical,
      [...several levels of function call graph...], and check_extent_in_eb, so
      that we can disable the extent offset filtering for uncompressed extents.
      This flag can be set by an improved version of the LOGICAL_INO ioctl to
      get either behavior as desired.
      
      There is no functional change in this patch.  The new flag is always
      false.
      Signed-off-by: default avatarZygo Blaxell <ce3g8jdj@umail.furryterror.org>
      Reviewed-by: default avatarDavid Sterba <dsterba@suse.com>
      [ minor coding style fixes ]
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      c995ab3c
    • Nikolay Borisov's avatar
      btrfs: send: remove unused code · eb7b9d6a
      Nikolay Borisov authored
      This code was first introduced in 31db9f7c ("Btrfs: introduce
      BTRFS_IOC_SEND for btrfs send/receive") and it was not functional, then
      it got slightly refactored in e938c8ad
      
       ("Btrfs: code cleanups for
      send/receive"), alas it was still dead. So let's remove it for good!
      Signed-off-by: default avatarNikolay Borisov <nborisov@suse.com>
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
      eb7b9d6a
  23. 30 Oct, 2017 1 commit