forked from Minki/linux
9257b4a206
IOVA allocation has two problems that impede high-throughput I/O. First, it can do a linear search over the allocated IOVA ranges. Second, the rbtree spinlock that serializes IOVA allocations becomes contended. Address these problems by creating an API for caching allocated IOVA ranges, so that the IOVA allocator isn't accessed frequently. This patch adds a per-CPU cache, from which CPUs can alloc/free IOVAs without taking the rbtree spinlock. The per-CPU caches are backed by a global cache, to avoid invoking the (linear-time) IOVA allocator without needing to make the per-CPU cache size excessive. This design is based on magazines, as described in "Magazines and Vmem: Extending the Slab Allocator to Many CPUs and Arbitrary Resources" (currently available at https://www.usenix.org/legacy/event/usenix01/bonwick.html) Adding caching on top of the existing rbtree allocator maintains the property that IOVAs are densely packed in the IO virtual address space, which is important for keeping IOMMU page table usage low. To keep the cache size reasonable, we bound the IOVA space a CPU can cache by 32 MiB (we cache a bounded number of IOVA ranges, and only ranges of size <= 128 KiB). The shared global cache is bounded at 4 MiB of IOVA space. Signed-off-by: Omer Peleg <omer@cs.technion.ac.il> [mad@cs.technion.ac.il: rebased, cleaned up and reworded the commit message] Signed-off-by: Adam Morrison <mad@cs.technion.ac.il> Reviewed-by: Shaohua Li <shli@fb.com> Reviewed-by: Ben Serebrin <serebrin@google.com> [dwmw2: split out VT-d part into a separate patch] Signed-off-by: David Woodhouse <David.Woodhouse@intel.com> |
||
|---|---|---|
| .. | ||
| amd_iommu_init.c | ||
| amd_iommu_proto.h | ||
| amd_iommu_types.h | ||
| amd_iommu_v2.c | ||
| amd_iommu.c | ||
| arm-smmu-v3.c | ||
| arm-smmu.c | ||
| dma-iommu.c | ||
| dmar.c | ||
| exynos-iommu.c | ||
| fsl_pamu_domain.c | ||
| fsl_pamu_domain.h | ||
| fsl_pamu.c | ||
| fsl_pamu.h | ||
| intel_irq_remapping.c | ||
| intel-iommu.c | ||
| intel-svm.c | ||
| io-pgtable-arm-v7s.c | ||
| io-pgtable-arm.c | ||
| io-pgtable.c | ||
| io-pgtable.h | ||
| iommu-sysfs.c | ||
| iommu-traces.c | ||
| iommu.c | ||
| iova.c | ||
| ipmmu-vmsa.c | ||
| irq_remapping.c | ||
| irq_remapping.h | ||
| Kconfig | ||
| Makefile | ||
| msm_iommu_dev.c | ||
| msm_iommu_hw-8xxx.h | ||
| msm_iommu.c | ||
| msm_iommu.h | ||
| mtk_iommu.c | ||
| of_iommu.c | ||
| omap-iommu-debug.c | ||
| omap-iommu.c | ||
| omap-iommu.h | ||
| omap-iopgtable.h | ||
| rockchip-iommu.c | ||
| s390-iommu.c | ||
| tegra-gart.c | ||
| tegra-smmu.c | ||