linux/fs/xfs
Long Li e4c3b72a6e xfs: ensure submit buffers on LSN boundaries in error handlers
While performing the IO fault injection test, I caught the following data
corruption report:

 XFS (dm-0): Internal error ltbno + ltlen > bno at line 1957 of file fs/xfs/libxfs/xfs_alloc.c.  Caller xfs_free_ag_extent+0x79c/0x1130
 CPU: 3 PID: 33 Comm: kworker/3:0 Not tainted 6.5.0-rc7-next-20230825-00001-g7f8666926889 #214
 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20190727_073836-buildvm-ppc64le-16.ppc.fedoraproject.org-3.fc31 04/01/2014
 Workqueue: xfs-inodegc/dm-0 xfs_inodegc_worker
 Call Trace:
  <TASK>
  dump_stack_lvl+0x50/0x70
  xfs_corruption_error+0x134/0x150
  xfs_free_ag_extent+0x7d3/0x1130
  __xfs_free_extent+0x201/0x3c0
  xfs_trans_free_extent+0x29b/0xa10
  xfs_extent_free_finish_item+0x2a/0xb0
  xfs_defer_finish_noroll+0x8d1/0x1b40
  xfs_defer_finish+0x21/0x200
  xfs_itruncate_extents_flags+0x1cb/0x650
  xfs_free_eofblocks+0x18f/0x250
  xfs_inactive+0x485/0x570
  xfs_inodegc_worker+0x207/0x530
  process_scheduled_works+0x24a/0xe10
  worker_thread+0x5ac/0xc60
  kthread+0x2cd/0x3c0
  ret_from_fork+0x4a/0x80
  ret_from_fork_asm+0x11/0x20
  </TASK>
 XFS (dm-0): Corruption detected. Unmount and run xfs_repair

After analyzing the disk image, it was found that the corruption was
triggered by the fact that extent was recorded in both inode datafork
and AGF btree blocks. After a long time of reproduction and analysis,
we found that the reason of free sapce btree corruption was that the
AGF btree was not recovered correctly.

Consider the following situation, Checkpoint A and Checkpoint B are in
the same record and share the same start LSN1, buf items of same object
(AGF btree block) is included in both Checkpoint A and Checkpoint B. If
the buf item in Checkpoint A has been recovered and updates metadata LSN
permanently, then the buf item in Checkpoint B cannot be recovered,
because log recovery skips items with a metadata LSN >= the current LSN
of the recovery item. If there is still an inode item in Checkpoint B
that records the Extent X, the Extent X will be recorded in both inode
datafork and AGF btree block after Checkpoint B is recovered. Such
transaction can be seen when allocing enxtent for inode bmap, it record
both the addition of extent to the inode extent list and the removing
extent from the AGF.

  |------------Record (LSN1)------------------|---Record (LSN2)---|
  |-------Checkpoint A----------|----------Checkpoint B-----------|
  |     Buf Item(Extent X)      | Buf Item / Inode item(Extent X) |
  |     Extent X is freed       |     Extent X is allocated       |

After commit 12818d24db ("xfs: rework log recovery to submit buffers
on LSN boundaries") was introduced, we submit buffers on lsn boundaries
during log recovery. The above problem can be avoided under normal paths,
but it's not guaranteed under abnormal paths. Consider the following
process, if an error was encountered after recover buf item in Checkpoint
A and before recover buf item in Checkpoint B, buffers that have been
added to the buffer_list will still be submitted, this violates the
submits rule on lsn boundaries. So buf item in Checkpoint B cannot be
recovered on the next mount due to current lsn of transaction equal to
metadata lsn on disk. The detailed process of the problem is as follows.

First Mount:

  xlog_do_recovery_pass
    error = xlog_recover_process
      xlog_recover_process_data
        xlog_recover_process_ophdr
          xlog_recovery_process_trans
            ...
              /* recover buf item in Checkpoint A */
              xlog_recover_buf_commit_pass2
                xlog_recover_do_reg_buffer
                /* add buffer of agf btree block to buffer_list */
                xfs_buf_delwri_queue(bp, buffer_list)
            ...
            ==> Encounter read IO error and return
    /* submit buffers regardless of error */
    if (!list_empty(&buffer_list))
      xfs_buf_delwri_submit(&buffer_list);

    <buf items of agf btree block in Checkpoint A recovery success>

Second Mount:

  xlog_do_recovery_pass
    error = xlog_recover_process
      xlog_recover_process_data
        xlog_recover_process_ophdr
          xlog_recovery_process_trans
            ...
              /* recover buf item in Checkpoint B */
              xlog_recover_buf_commit_pass2
                /* buffer of agf btree block wouldn't added to
                   buffer_list due to lsn equal to current_lsn */
                if (XFS_LSN_CMP(lsn, current_lsn) >= 0)
                  goto out_release

    <buf items of agf btree block in Checkpoint B wouldn't recovery>

In order to make sure that submits buffers on lsn boundaries in the
abnormal paths, we need to check error status before submit buffers that
have been added from the last record processed. If error status exist,
buffers in the bufffer_list should not be writen to disk.

Canceling the buffers in the buffer_list directly isn't correct, unlike
any other place where write list was canceled, these buffers has been
initialized by xfs_buf_item_init() during recovery and held by buf item,
buf items will not be released in xfs_buf_delwri_cancel(), it's not easy
to solve.

If the filesystem has been shut down, then delwri list submission will
error out all buffers on the list via IO submission/completion and do
all the correct cleanup automatically. So shutting down the filesystem
could prevents buffers in the bufffer_list from being written to disk.

Fixes: 50d5c8d8e9 ("xfs: check LSN ordering for v5 superblocks during recovery")
Signed-off-by: Long Li <leo.lilong@huawei.com>
Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>
Signed-off-by: Chandan Babu R <chandanbabu@kernel.org>
2024-02-17 09:34:52 +05:30
..
libxfs xfs: use xfs_defer_alloc a bit more 2024-02-13 18:07:36 +05:30
scrub xfs: disable sparse inode chunk alignment check when there is no alignment 2024-02-17 09:18:53 +05:30
Kconfig xfs: fix again select in kconfig XFS_ONLINE_SCRUB_STATS 2023-11-13 09:11:41 +05:30
Makefile xfs: convert kmem_alloc() to kmalloc() 2024-02-13 18:07:34 +05:30
mrlock.h
xfs_acl.c xfs: convert kmem_free() for kvmalloc users to kvfree() 2024-02-13 18:07:34 +05:30
xfs_acl.h
xfs_aops.c fs: convert error_remove_page to error_remove_folio 2023-12-10 16:51:42 -08:00
xfs_aops.h
xfs_attr_inactive.c xfs: make inode unlinked bucket recovery work with quotacheck 2023-09-12 10:31:07 -07:00
xfs_attr_item.c xfs: place intent recovery under NOFS allocation context 2024-02-13 18:07:35 +05:30
xfs_attr_item.h
xfs_attr_list.c xfs: clean up remaining GFP_NOFS users 2024-02-13 18:07:35 +05:30
xfs_bio_io.c
xfs_bmap_item.c xfs: place intent recovery under NOFS allocation context 2024-02-13 18:07:35 +05:30
xfs_bmap_item.h
xfs_bmap_util.c xfs: use GFP_KERNEL in pure transaction contexts 2024-02-13 18:07:35 +05:30
xfs_bmap_util.h xfs: indicate if xfs_bmap_adjacent changed ap->blkno 2023-12-22 11:18:11 +05:30
xfs_buf_item_recover.c xfs: convert remaining kmem_free() to kfree() 2024-02-13 18:07:34 +05:30
xfs_buf_item.c xfs: convert remaining kmem_free() to kfree() 2024-02-13 18:07:34 +05:30
xfs_buf_item.h
xfs_buf.c xfs: clean up remaining GFP_NOFS users 2024-02-13 18:07:35 +05:30
xfs_buf.h xfs: force all buffers to be written during btree bulk load 2023-12-15 10:03:27 -08:00
xfs_dahash_test.c xfs: test the ascii case-insensitive hash 2023-04-11 19:05:05 -07:00
xfs_dahash_test.h xfs: test dir/attr hash when loading module 2023-03-19 09:55:49 -07:00
xfs_dir2_readdir.c xfs: make if_data a void pointer 2023-12-29 13:37:03 +05:30
xfs_discard.c xfs: use an empty transaction for fstrim 2024-02-13 18:07:34 +05:30
xfs_discard.h xfs: move log discard work to xfs_discard.c 2023-10-04 09:24:02 +11:00
xfs_dquot_item_recover.c xfs: dquot recovery does not validate the recovered dquot 2023-11-22 23:39:36 +05:30
xfs_dquot_item.c
xfs_dquot_item.h
xfs_dquot.c xfs: convert kmem_free() for kvmalloc users to kvfree() 2024-02-13 18:07:34 +05:30
xfs_dquot.h xfs: repair quotas 2023-12-15 10:03:45 -08:00
xfs_drain.c xfs: minimize overhead of drain wakeups by using jump labels 2023-04-11 18:59:59 -07:00
xfs_drain.h xfs: minimize overhead of drain wakeups by using jump labels 2023-04-11 18:59:59 -07:00
xfs_error.c xfs: convert remaining kmem_free() to kfree() 2024-02-13 18:07:34 +05:30
xfs_error.h
xfs_export.c xfs: fix reloading entire unlinked bucket lists 2023-09-24 18:12:13 -07:00
xfs_export.h
xfs_extent_busy.c xfs: convert remaining kmem_free() to kfree() 2024-02-13 18:07:34 +05:30
xfs_extent_busy.h xfs: repair free space btrees 2023-12-15 10:03:32 -08:00
xfs_extfree_item.c xfs: convert remaining kmem_free() to kfree() 2024-02-13 18:07:34 +05:30
xfs_extfree_item.h
xfs_file.c xfs: allow read IO and FICLONE to run concurrently 2023-10-23 12:02:26 +05:30
xfs_filestream.c xfs: convert remaining kmem_free() to kfree() 2024-02-13 18:07:34 +05:30
xfs_filestream.h xfs: pass perag to filestreams tracing 2023-02-13 09:14:56 +11:00
xfs_fsmap.c xfs: convert do_div calls to xfs_rtb_to_rtx helper calls 2023-10-17 16:25:55 -07:00
xfs_fsmap.h
xfs_fsops.c New code for 6.8: 2024-01-10 08:45:22 -08:00
xfs_fsops.h xfs: clean up xfs_fsops.h 2023-12-07 14:51:07 +05:30
xfs_globals.c xfs: add debug knobs to control btree bulk load slack factors 2023-12-15 10:03:28 -08:00
xfs_health.c xfs: set inode sick state flags when we zap either ondisk fork 2023-12-15 10:03:35 -08:00
xfs_icache.c xfs: use __GFP_NOLOCKDEP instead of GFP_NOFS 2024-02-13 18:07:34 +05:30
xfs_icache.h xfs: use per-mount cpumask to track nonempty percpu inodegc lists 2023-09-11 08:39:03 -07:00
xfs_icreate_item.c xfs: convert kmem_free() for kvmalloc users to kvfree() 2024-02-13 18:07:34 +05:30
xfs_icreate_item.h
xfs_inode_item_recover.c xfs: convert remaining kmem_free() to kfree() 2024-02-13 18:07:34 +05:30
xfs_inode_item.c xfs: convert kmem_free() for kvmalloc users to kvfree() 2024-02-13 18:07:34 +05:30
xfs_inode_item.h xfs: fix AGF vs inode cluster buffer deadlock 2023-06-05 04:08:27 +10:00
xfs_inode.c xfs: convert remaining kmem_free() to kfree() 2024-02-13 18:07:34 +05:30
xfs_inode.h xfs: set inode sick state flags when we zap either ondisk fork 2023-12-15 10:03:35 -08:00
xfs_ioctl32.c
xfs_ioctl32.h arch: Remove Itanium (IA-64) architecture 2023-09-11 08:13:17 +00:00
xfs_ioctl.c xfs: convert remaining kmem_free() to kfree() 2024-02-13 18:07:34 +05:30
xfs_ioctl.h
xfs_iomap.c xfs: don't allocate into the data fork for an unshare request 2023-05-02 09:14:51 +10:00
xfs_iomap.h
xfs_iops.c xfs: convert remaining kmem_free() to kfree() 2024-02-13 18:07:34 +05:30
xfs_iops.h
xfs_itable.c xfs: convert remaining kmem_free() to kfree() 2024-02-13 18:07:34 +05:30
xfs_itable.h
xfs_iunlink_item.c xfs: create traced helper to get extra perag references 2023-04-11 18:59:55 -07:00
xfs_iunlink_item.h
xfs_iwalk.c xfs: convert remaining kmem_free() to kfree() 2024-02-13 18:07:34 +05:30
xfs_iwalk.h
xfs_linux.h xfs: convert remaining kmem_free() to kfree() 2024-02-13 18:07:34 +05:30
xfs_log_cil.c xfs: place the CIL under nofs allocation context 2024-02-13 18:07:35 +05:30
xfs_log_priv.h xfs: use xfs_defer_pending objects to recover intent items 2023-12-06 18:45:14 -08:00
xfs_log_recover.c xfs: ensure submit buffers on LSN boundaries in error handlers 2024-02-17 09:34:52 +05:30
xfs_log.c xfs: use GFP_KERNEL in pure transaction contexts 2024-02-13 18:07:35 +05:30
xfs_log.h
xfs_message.c
xfs_message.h
xfs_mount.c xfs: convert remaining kmem_free() to kfree() 2024-02-13 18:07:34 +05:30
xfs_mount.h New code for 6.7: 2023-11-08 13:22:16 -08:00
xfs_mru_cache.c xfs: use GFP_KERNEL in pure transaction contexts 2024-02-13 18:07:35 +05:30
xfs_mru_cache.h
xfs_notify_failure.c mm, pmem, xfs: Introduce MF_MEM_PRE_REMOVE for unbind 2023-12-07 14:34:26 +05:30
xfs_pnfs.c
xfs_pnfs.h
xfs_pwork.c
xfs_pwork.h
xfs_qm_bhv.c
xfs_qm_syscalls.c
xfs_qm.c xfs: use __GFP_NOLOCKDEP instead of GFP_NOFS 2024-02-13 18:07:34 +05:30
xfs_qm.h xfs: dynamically allocate the xfs-qm shrinker 2023-10-04 10:32:26 -07:00
xfs_quota.h xfs: turn the xfs_trans_mod_dquot_byino stub into an inline function 2023-12-22 11:18:10 +05:30
xfs_quotaops.c
xfs_refcount_item.c xfs: place intent recovery under NOFS allocation context 2024-02-13 18:07:35 +05:30
xfs_refcount_item.h
xfs_reflink.c xfs: remove __xfs_free_extent_later 2023-12-06 18:45:18 -08:00
xfs_reflink.h
xfs_rmap_item.c xfs: place intent recovery under NOFS allocation context 2024-02-13 18:07:35 +05:30
xfs_rmap_item.h
xfs_rtalloc.c xfs: convert remaining kmem_free() to kfree() 2024-02-13 18:07:34 +05:30
xfs_rtalloc.h xfs: move xfs_bmap_rtalloc to xfs_rtalloc.c 2023-12-22 11:18:11 +05:30
xfs_stats.c
xfs_stats.h
xfs_super.c xfs: convert remaining kmem_free() to kfree() 2024-02-13 18:07:34 +05:30
xfs_super.h xfs: create scaffolding for creating debugfs entries 2023-08-10 07:48:07 -07:00
xfs_symlink.c xfs: make if_data a void pointer 2023-12-29 13:37:03 +05:30
xfs_symlink.h
xfs_sysctl.c fs: Remove the now superfluous sentinel elements from ctl_table array 2023-12-28 04:57:57 -08:00
xfs_sysctl.h xfs: add debug knobs to control btree bulk load slack factors 2023-12-15 10:03:28 -08:00
xfs_sysfs.c xfs: remove duplicate ifdefs 2024-02-17 09:32:32 +05:30
xfs_sysfs.h
xfs_trace.c
xfs_trace.h xfs: convert kmem_alloc() to kmalloc() 2024-02-13 18:07:34 +05:30
xfs_trans_ail.c xfs: convert remaining kmem_free() to kfree() 2024-02-13 18:07:34 +05:30
xfs_trans_buf.c
xfs_trans_dquot.c
xfs_trans_priv.h
xfs_trans.c xfs: repair inode fork block mapping data structures 2023-12-15 10:03:39 -08:00
xfs_trans.h xfs: repair inode fork block mapping data structures 2023-12-15 10:03:39 -08:00
xfs_xattr.c xfs: set inode sick state flags when we zap either ondisk fork 2023-12-15 10:03:35 -08:00
xfs_xattr.h xfs: move xfs_xattr_handlers to .rodata 2023-10-10 13:49:20 +02:00
xfs.h