When hns_roce_v2_destroy_qp() is called, the brief calling process of the
driver is as follows:
......
hns_roce_v2_destroy_qp
hns_roce_v2_qp_modify
hns_roce_cmd_mbox
hns_roce_qp_destroy
If hns_roce_cmd_mbox() detects that the hardware is being reset during the
execution of the hns_roce_cmd_mbox(), the driver will not be able to get
the return value from the hardware (the firmware cannot respond to the
driver's mailbox during the hardware reset phase).
The driver needs to wait for the hardware reset to complete before
continuing to execute hns_roce_qp_destroy(), otherwise it may happen that
the driver releases the resources but the hardware is still accessing. In
order to fix this problem, HNS RoCE needs to add a piece of code to wait
for the hardware reset to complete.
The original interface get_hw_reset_stat() is the instantaneous state of
the hardware reset, which cannot accurately reflect whether the hardware
reset is completed, so it needs to be replaced with the ae_dev_reset_cnt
interface.
The sign that the hardware reset is complete is that the return value of
the ae_dev_reset_cnt interface is greater than the original value
reset_cnt recorded by the driver.
Fixes: 6a04aed6af ("RDMA/hns: Fix the chip hanging caused by sending mailbox&CMQ during reset")
Link: https://lore.kernel.org/r/20211123142402.26936-1-liangwenpeng@huawei.com
Signed-off-by: Yangyang Li <liyangyang20@huawei.com>
Signed-off-by: Wenpeng Liang <liangwenpeng@huawei.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
is_reset is used to indicate whether the hardware starts to reset. When
hns_roce_hw_v2_reset_notify_down() is called, the hardware has not yet
started to reset. If is_reset is set at this time, all mailbox operations
of resource destroy actions will be intercepted by driver. When the driver
cleans up resources, but the hardware is still accessed, the following
errors will appear:
arm-smmu-v3 arm-smmu-v3.2.auto: event 0x10 received:
arm-smmu-v3 arm-smmu-v3.2.auto: 0x0000350100000010
arm-smmu-v3 arm-smmu-v3.2.auto: 0x000002088000003f
arm-smmu-v3 arm-smmu-v3.2.auto: 0x00000000a50e0800
arm-smmu-v3 arm-smmu-v3.2.auto: 0x0000000000000000
arm-smmu-v3 arm-smmu-v3.2.auto: event 0x10 received:
arm-smmu-v3 arm-smmu-v3.2.auto: 0x0000350100000010
arm-smmu-v3 arm-smmu-v3.2.auto: 0x000002088000043e
arm-smmu-v3 arm-smmu-v3.2.auto: 0x00000000a50a0800
arm-smmu-v3 arm-smmu-v3.2.auto: 0x0000000000000000
arm-smmu-v3 arm-smmu-v3.2.auto: event 0x10 received:
arm-smmu-v3 arm-smmu-v3.2.auto: 0x0000350100000010
arm-smmu-v3 arm-smmu-v3.2.auto: 0x0000020880000436
arm-smmu-v3 arm-smmu-v3.2.auto: 0x00000000a50a0880
arm-smmu-v3 arm-smmu-v3.2.auto: 0x0000000000000000
arm-smmu-v3 arm-smmu-v3.2.auto: event 0x10 received:
arm-smmu-v3 arm-smmu-v3.2.auto: 0x0000350100000010
arm-smmu-v3 arm-smmu-v3.2.auto: 0x000002088000043a
arm-smmu-v3 arm-smmu-v3.2.auto: 0x00000000a50e0840
hns3 0000:35:00.0: INT status: CMDQ(0x0) HW errors(0x0) other(0x0)
arm-smmu-v3 arm-smmu-v3.2.auto: 0x0000000000000000
hns3 0000:35:00.0: received unknown or unhandled event of vector0
arm-smmu-v3 arm-smmu-v3.2.auto: event 0x10 received:
arm-smmu-v3 arm-smmu-v3.2.auto: 0x0000350100000010
{34}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 7
is_reset will be set correctly in check_aedev_reset_status(), so the
setting in hns_roce_hw_v2_reset_notify_down() should be deleted.
Fixes: 726be12f5c ("RDMA/hns: Set reset flag when hw resetting")
Link: https://lore.kernel.org/r/20211123084809.37318-1-liangwenpeng@huawei.com
Signed-off-by: Yangyang Li <liyangyang20@huawei.com>
Signed-off-by: Wenpeng Liang <liangwenpeng@huawei.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
For the case of IB_MR_TYPE_DM the mr does doesn't have a umem, even though
it is a user MR. This causes function mlx5_free_priv_descs() to think that
it is a kernel MR, leading to wrongly accessing mr->descs that will get
wrong values in the union which leads to attempt to release resources that
were not allocated in the first place.
For example:
DMA-API: mlx5_core 0000:08:00.1: device driver tries to free DMA memory it has not allocated [device address=0x0000000000000000] [size=0 bytes]
WARNING: CPU: 8 PID: 1021 at kernel/dma/debug.c:961 check_unmap+0x54f/0x8b0
RIP: 0010:check_unmap+0x54f/0x8b0
Call Trace:
debug_dma_unmap_page+0x57/0x60
mlx5_free_priv_descs+0x57/0x70 [mlx5_ib]
mlx5_ib_dereg_mr+0x1fb/0x3d0 [mlx5_ib]
ib_dereg_mr_user+0x60/0x140 [ib_core]
uverbs_destroy_uobject+0x59/0x210 [ib_uverbs]
uobj_destroy+0x3f/0x80 [ib_uverbs]
ib_uverbs_cmd_verbs+0x435/0xd10 [ib_uverbs]
? uverbs_finalize_object+0x50/0x50 [ib_uverbs]
? lock_acquire+0xc4/0x2e0
? lock_acquired+0x12/0x380
? lock_acquire+0xc4/0x2e0
? lock_acquire+0xc4/0x2e0
? ib_uverbs_ioctl+0x7c/0x140 [ib_uverbs]
? lock_release+0x28a/0x400
ib_uverbs_ioctl+0xc0/0x140 [ib_uverbs]
? ib_uverbs_ioctl+0x7c/0x140 [ib_uverbs]
__x64_sys_ioctl+0x7f/0xb0
do_syscall_64+0x38/0x90
Fix it by reorganizing the dereg flow and mlx5_ib_mr structure:
- Move the ib_umem field into the user MRs structure in the union as it's
applicable only there.
- Function mlx5_ib_dereg_mr() will now call mlx5_free_priv_descs() only
in case there isn't udata, which indicates that this isn't a user MR.
Fixes: f18ec42231 ("RDMA/mlx5: Use a union inside mlx5_ib_mr")
Link: https://lore.kernel.org/r/66bb1dd253c1fd7ceaa9fc411061eefa457b86fb.1637581144.git.leonro@nvidia.com
Signed-off-by: Alaa Hleihel <alaa@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
When optional counter support was added the allocation of the memory
holding the counter descriptors was not cleared properly. This caused
WARN_ON()s in the IB/sysfs code to be hit.
This is because the uninitialized memory made some of the counters wrongly
look like optional counters. Use kzalloc.
While here change the sizeof() calls to use the pointer rather than the
name of the type.
WARNING: CPU: 0 PID: 32644 at drivers/infiniband/core/sysfs.c:1064 ib_setup_port_attrs+0x7e1/0x890 [ib_core]
CPU: 0 PID: 32644 Comm: kworker/0:2 Tainted: G S W 5.15.0+ #36
Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS SE5C610.86B.01.01.0018.C4.072020161249 07/20/2016
Workqueue: events work_for_cpu_fn
RIP: 0010:ib_setup_port_attrs+0x7e1/0x890 [ib_core]
RSP: 0018:ffffc90006ea3c40 EFLAGS: 00010202
RAX: 0000000000000068 RBX: ffff888106ad8000 RCX: 0000000000000138
RDX: ffff888126c84c00 RSI: ffff888103c41000 RDI: 0000000000000124
RBP: ffff88810f63a801 R08: ffff888126c8a000 R09: 0000000000000001
R10: ffffffffa09acf20 R11: 0000000000000065 R12: ffff88810f63a800
R13: ffff88810f63a800 R14: ffff88810f63a8e0 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffff888667a00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005590102cb078 CR3: 000000000240a003 CR4: 00000000001706f0
Call Trace:
ib_register_device.cold.44+0x23e/0x2d0 [ib_core]
rvt_register_device+0xfa/0x230 [rdmavt]
hfi1_register_ib_device+0x623/0x690 [hfi1]
init_one.cold.36+0x2d1/0x49b [hfi1]
local_pci_probe+0x45/0x80
work_for_cpu_fn+0x16/0x20
process_one_work+0x1b1/0x360
worker_thread+0x1d4/0x3a0
kthread+0x11a/0x140
ret_from_fork+0x22/0x30
Fixes: 5e2ddd1e59 ("RDMA/counter: Add optional counter support")
Link: https://lore.kernel.org/r/20211115200913.124104.47770.stgit@awfm-01.cornelisnetworks.com
Reviewed-by: Mike Marciniszyn <mike.marciniszyn@cornelisnetworks.com>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Pull rdma updates from Jason Gunthorpe:
"A typical collection of patches this cycle, mostly fixing with a few
new features:
- Fixes from static tools. clang warnings, dead code, unused
variable, coccinelle sweeps, etc
- Driver bug fixes and minor improvements in rxe, bnxt_re, hfi1,
mlx5, irdma, qedr
- rtrs ULP bug fixes an improvments
- Additional counters for bnxt_re
- Support verbs CQ notifications in EFA
- Continued reworking and fixing of rxe
- netlink control to enable/disable optional device counters
- rxe now can use AH objects for its UD path, fixing various bugs in
the process
- Add DMABUF support to EFA"
* tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma: (103 commits)
RDMA/core: Require the driver to set the IOVA correctly during rereg_mr
RDMA/bnxt_re: Remove unsupported bnxt_re_modify_ah callback
RDMA/irdma: optimize rx path by removing unnecessary copy
RDMA/qed: Use helper function to set GUIDs
RDMA/hns: Use the core code to manage the fixed mmap entries
IB/opa_vnic: Rebranding of OPA VNIC driver to Cornelis Networks
IB/qib: Rebranding of qib driver to Cornelis Networks
IB/hfi1: Rebranding of hfi1 driver to Cornelis Networks
RDMA/bnxt_re: Use helper function to set GUIDs
RDMA/bnxt_re: Fix kernel panic when trying to access bnxt_re_stat_descs
RDMA/qedr: Fix NULL deref for query_qp on the GSI QP
RDMA/hns: Modify the value of MAX_LP_MSG_LEN to meet hardware compatibility
RDMA/hns: Fix initial arm_st of CQ
RDMA/rxe: Make rxe_type_info static const
RDMA/rxe: Use 'bitmap_zalloc()' when applicable
RDMA/rxe: Save a few bytes from struct rxe_pool
RDMA/irdma: Remove the unused variable local_qp
RDMA/core: Fix missed initialization of rdma_hw_stats::lock
RDMA/efa: Add support for dmabuf memory regions
RDMA/umem: Allow pinned dmabuf umem usage
...
Patches held over for a possible rc8.
* for-rc:
RDMA/qedr: Fix NULL deref for query_qp on the GSI QP
RDMA/hns: Modify the value of MAX_LP_MSG_LEN to meet hardware compatibility
RDMA/hns: Fix initial arm_st of CQ
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
