[PATCH V3 0/8] IOMMU probe deferral support

[Sricharan R]

Initial post from Laurent Pinchart[1]. This is
series calls the dma ops configuration for the devices
at a generic place so that it works for all busses.
The dma_configure_ops for a device is now called during
the device_attach callback just before the probe of the
bus/driver is called. Similarly dma_deconfigure is called during
device/driver_detach path.


pci_bus_add_devices (platform/amba)(_device_create/driver_register)
| |
pci_bus_add_device (device_add/driver_register)
| |
device_attach device_initial_probe
| |
__device_attach_driver __device_attach_driver
|
driver_probe_device
|
really_probe
|
dma_configure

Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.

If the ACPI bus code follows the same, we can add acpi_dma_configure
at the same place as of_dma_configure.

This series is based on the recently merged Generic DT bindings for
PCI IOMMUs and ARM SMMU from Robin Murphy ***@arm.com [2]

This time tested this with platform and pci device for probe deferral
and reprobe on arm64 based platform. There is an issue on the cleanup
path for arm64 though, where there is WARN_ON if the dma_ops is reset while
device is attached to an domain in arch_teardown_dma_ops.
But with iommu_groups created from the iommu driver, the device is always
attached to a domain/default_domain. So so the WARN has to be removed/handled
probably.

Previous post of this series [3].

[V3]
* Removed the patch to split dma_masks/dma_ops configuration separately
based on review comments that both masks and ops are required only
during the device probe time.

* Reworked the series based on Generic DT bindings series [2].

* Added call to iommu's remove_device in the cleanup path for arm and arm64.

* Removed the notifier trick in arm64 to handle early device registration.

* Added reset of dma_ops in cleanup path for arm based on comments.

* Fixed the pci_iommu_configure path and tested with PCI device as well.

* Fixed a bug to return the correct iommu_ops from patch 7 [4] in last post.

* Fixed few other cosmetic comments.

[V2]
* Updated the Initial post to call dma_configure/deconfigure from generic code

* Added iommu add_device callback from of_iommu_configure path

[V1]
* Initial post

[1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
[2] http://www.spinics.net/lists/devicetree/msg142943.html
[3] https://www.mail-archive.com/***@lists.linux-foundation.org/msg13941.html
[4] https://www.mail-archive.com/***@lists.linux-foundation.org/msg13940.html



Laurent Pinchart (4):
arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()
of: dma: Move range size workaround to of_dma_get_range()
of: dma: Make of_dma_deconfigure() public
iommu: of: Handle IOMMU lookup failure with deferred probing or error

Sricharan R (4):
drivers: platform: Configure dma operations at probe time
arm: dma-mapping: Reset the device's dma_ops
arm/arm64: dma-mapping: Call iommu's remove_device callback during
device detach
arm64: dma-mapping: Remove the notifier trick to handle early setting
of dma_ops

arch/arm/mm/dma-mapping.c | 18 ++++++++
arch/arm64/mm/dma-mapping.c | 107 +++++---------------------------------------
drivers/base/dd.c | 10 +++++
drivers/base/dma-mapping.c | 11 +++++
drivers/iommu/of_iommu.c | 47 +++++++++++++++++--
drivers/of/address.c | 20 ++++++++-
drivers/of/device.c | 34 +++++++-------
drivers/of/platform.c | 9 ----
drivers/pci/probe.c | 5 +--
include/linux/dma-mapping.h | 3 ++
include/linux/of_device.h | 7 ++-
11 files changed, 138 insertions(+), 133 deletions(-)

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+***@ideasonboard.com>

The arch_setup_dma_ops() function is in charge of setting dma_ops with a
call to set_dma_ops(). set_dma_ops() is also called from

- highbank and mvebu bus notifiers
- dmabounce (to be replaced with swiotlb)
- arm_iommu_attach_device

(arm_iommu_attach_device is itself called from IOMMU and bus master
device drivers)

To allow the arch_setup_dma_ops() call to be moved from device add time
to device probe time we must ensure that dma_ops already setup by any of
the above callers will not be overridden.

Aftering replacing dmabounce with swiotlb, converting IOMMU drivers to
of_xlate and taking care of highbank and mvebu, the workaround should be
removed.

Signed-off-by: Laurent Pinchart <laurent.pinchart+***@ideasonboard.com>
---
arch/arm/mm/dma-mapping.c | 9 +++++++++
1 file changed, 9 insertions(+)

diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
index c6834c0..dde6514 100644
--- a/arch/arm/mm/dma-mapping.c
+++ b/arch/arm/mm/dma-mapping.c
@@ -2322,6 +2322,15 @@ void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
struct dma_map_ops *dma_ops;

dev->archdata.dma_coherent = coherent;
+
+ /*
+ * Don't override the dma_ops if they have already been set. Ideally
+ * this should be the only location where dma_ops are set, remove this
+ * check when all other callers of set_dma_ops will have disappeared.
+ */
+ if (dev->archdata.dma_ops)
+ return;
+
if (arm_setup_iommu_dma_ops(dev, dma_base, size, iommu))
dma_ops = arm_get_iommu_dma_map_ops(coherent);
else

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+***@ideasonboard.com>

Invalid dma-ranges values should be worked around when retrieving the
DMA range in of_dma_get_range(), not by all callers of the function.
This isn't much of a problem now that we have a single caller, but that
situation will change when moving DMA configuration to device probe
time.

Signed-off-by: Laurent Pinchart <laurent.pinchart+***@ideasonboard.com>
---
drivers/of/address.c | 20 ++++++++++++++++++--
drivers/of/device.c | 15 ---------------
2 files changed, 18 insertions(+), 17 deletions(-)

diff --git a/drivers/of/address.c b/drivers/of/address.c
index 02b2903..6aeb816 100644
--- a/drivers/of/address.c
+++ b/drivers/of/address.c
@@ -819,8 +819,8 @@ EXPORT_SYMBOL(of_io_request_and_map);
* CPU addr (phys_addr_t) : pna cells
* size : nsize cells
*
- * It returns -ENODEV if "dma-ranges" property was not found
- * for this device in DT.
+ * Return 0 on success, -ENODEV if the "dma-ranges" property was not found for
+ * this device in DT, or -EINVAL if the CPU address or size is invalid.
*/
int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
{
@@ -880,6 +880,22 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
*dma_addr = dmaaddr;

*size = of_read_number(ranges + naddr + pna, nsize);
+ /*
+ * DT nodes sometimes incorrectly set the size as a mask. Work around
+ * those incorrect DT by computing the size as mask + 1.
+ */
+ if (*size & 1) {
+ pr_warn("%s: size 0x%llx for dma-range in node(%s) set as mask\n",
+ __func__, *size, np->full_name);
+ *size = *size + 1;
+ }
+
+ if (!*size) {
+ pr_err("%s: invalid size zero for dma-range in node(%s)\n",
+ __func__, np->full_name);
+ ret = -EINVAL;
+ goto out;
+ }

pr_debug("dma_addr(%llx) cpu_addr(%llx) size(%llx)\n",
*dma_addr, *paddr, *size);
diff --git a/drivers/of/device.c b/drivers/of/device.c
index fd5cfad..d9898d9 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -110,21 +110,6 @@ void of_dma_configure(struct device *dev, struct device_node *np)
size = dev->coherent_dma_mask + 1;
} else {
offset = PFN_DOWN(paddr - dma_addr);
-
- /*
- * Add a work around to treat the size as mask + 1 in case
- * it is defined in DT as a mask.
- */
- if (size & 1) {
- dev_warn(dev, "Invalid size 0x%llx for dma-range\n",
- size);
- size = size + 1;
- }
-
- if (!size) {
- dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
- return;
- }
dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
}

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+***@ideasonboard.com>

As part of moving DMA initializing to probe time the
of_dma_deconfigure() function will need to be called from different
source files. Make it public and move it to drivers/of/device.c where
the of_dma_configure() function is.

Signed-off-by: Laurent Pinchart <laurent.pinchart+***@ideasonboard.com>
---
drivers/of/device.c | 12 ++++++++++++
drivers/of/platform.c | 5 -----
include/linux/of_device.h | 3 +++
3 files changed, 15 insertions(+), 5 deletions(-)

diff --git a/drivers/of/device.c b/drivers/of/device.c
index d9898d9..1c843e2 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -136,6 +136,18 @@ void of_dma_configure(struct device *dev, struct device_node *np)
}
EXPORT_SYMBOL_GPL(of_dma_configure);

+/**
+ * of_dma_deconfigure - Clean up DMA configuration
+ * @dev: Device for which to clean up DMA configuration
+ *
+ * Clean up all configuration performed by of_dma_configure_ops() and free all
+ * resources that have been allocated.
+ */
+void of_dma_deconfigure(struct device *dev)
+{
+ arch_teardown_dma_ops(dev);
+}
+
int of_device_register(struct platform_device *pdev)
{
device_initialize(&pdev->dev);
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index f39ccd5..9cb7090 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -153,11 +153,6 @@ struct platform_device *of_device_alloc(struct device_node *np,
}
EXPORT_SYMBOL(of_device_alloc);

-static void of_dma_deconfigure(struct device *dev)
-{
- arch_teardown_dma_ops(dev);
-}
-
/**
* of_platform_device_create_pdata - Alloc, initialize and register an of_device
* @np: pointer to node to create device for
diff --git a/include/linux/of_device.h b/include/linux/of_device.h
index cc7dd687..d20a31a 100644
--- a/include/linux/of_device.h
+++ b/include/linux/of_device.h
@@ -56,6 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
}

void of_dma_configure(struct device *dev, struct device_node *np);
+void of_dma_deconfigure(struct device *dev);
#else /* CONFIG_OF */

static inline int of_driver_match_device(struct device *dev,
@@ -100,6 +101,8 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
}
static inline void of_dma_configure(struct device *dev, struct device_node *np)
{}
+static inline void of_dma_deconfigure(struct device *dev)
+{}
#endif /* CONFIG_OF */

#endif /* _LINUX_OF_DEVICE_H */

[Sricharan R]

Configuring DMA ops at probe time will allow deferring device probe when
the IOMMU isn't available yet. The dma_configure for the device is now called
from the generic device_attach callback just before the bus/driver probe
is called. This way, configuring the dma ops for the device would be called
at the same place for all bus_types, hence the deferred probing mechanism
should work for all buses as well.

pci_bus_add_devices (platform/amba)(_device_create/driver_register)
| |
pci_bus_add_device (device_add/driver_register)
| |
device_attach device_initial_probe
| |
__device_attach_driver __device_attach_driver
|
driver_probe_device
|
really_probe
|
dma_configure

Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.

Signed-off-by: Sricharan R <***@codeaurora.org>
---
drivers/base/dd.c | 10 ++++++++++
drivers/base/dma-mapping.c | 11 +++++++++++
drivers/of/platform.c | 4 ----
drivers/pci/probe.c | 5 +----
include/linux/dma-mapping.h | 3 +++
5 files changed, 25 insertions(+), 8 deletions(-)

diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index 16688f5..cfebd48 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -19,6 +19,7 @@

#include <linux/device.h>
#include <linux/delay.h>
+#include <linux/dma-mapping.h>
#include <linux/module.h>
#include <linux/kthread.h>
#include <linux/wait.h>
@@ -353,6 +354,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
if (ret)
goto pinctrl_bind_failed;

+ ret = dma_configure(dev);
+ if (ret)
+ goto dma_failed;
+
if (driver_sysfs_add(dev)) {
printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
__func__, dev_name(dev));
@@ -395,6 +400,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
goto done;

probe_failed:
+ dma_deconfigure(dev);
+dma_failed:
if (dev->bus)
blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
@@ -780,6 +787,9 @@ static void __device_release_driver(struct device *dev)
dev->bus->remove(dev);
else if (drv->remove)
drv->remove(dev);
+
+ dma_deconfigure(dev);
+
devres_release_all(dev);
dev->driver = NULL;
dev_set_drvdata(dev, NULL);
diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index d799662..54e87f5 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -10,6 +10,7 @@
#include <linux/dma-mapping.h>
#include <linux/export.h>
#include <linux/gfp.h>
+#include <linux/of_device.h>
#include <linux/slab.h>
#include <linux/vmalloc.h>

@@ -166,6 +167,16 @@ void dmam_free_noncoherent(struct device *dev, size_t size, void *vaddr,
}
EXPORT_SYMBOL(dmam_free_noncoherent);

+int dma_configure(struct device *dev)
+{
+ return of_dma_configure(dev, dev->of_node);
+}
+
+void dma_deconfigure(struct device *dev)
+{
+ of_dma_deconfigure(dev);
+}
+
#ifdef CONFIG_HAVE_GENERIC_DMA_COHERENT

static void dmam_coherent_decl_release(struct device *dev, void *res)
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index 9cb7090..adbd77c 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -181,11 +181,9 @@ static struct platform_device *of_platform_device_create_pdata(

dev->dev.bus = &platform_bus_type;
dev->dev.platform_data = platform_data;
- of_dma_configure(&dev->dev, dev->dev.of_node);
of_msi_configure(&dev->dev, dev->dev.of_node);

if (of_device_add(dev) != 0) {
- of_dma_deconfigure(&dev->dev);
platform_device_put(dev);
goto err_clear_flag;
}
@@ -242,7 +240,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
dev_set_name(&dev->dev, "%s", bus_id);
else
of_device_make_bus_id(&dev->dev);
- of_dma_configure(&dev->dev, dev->dev.of_node);

/* Allow the HW Peripheral ID to be overridden */
prop = of_get_property(node, "arm,primecell-periphid", NULL);
@@ -536,7 +533,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
amba_device_unregister(to_amba_device(dev));
#endif

- of_dma_deconfigure(dev);
of_node_clear_flag(dev->of_node, OF_POPULATED);
of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
return 0;
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 93f280d..85c9553 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1724,10 +1724,7 @@ static void pci_dma_configure(struct pci_dev *dev)
{
struct device *bridge = pci_get_host_bridge_device(dev);

- if (IS_ENABLED(CONFIG_OF) &&
- bridge->parent && bridge->parent->of_node) {
- of_dma_configure(&dev->dev, bridge->parent->of_node);
- } else if (has_acpi_companion(bridge)) {
+ if (has_acpi_companion(bridge)) {
struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
enum dev_dma_attr attr = acpi_get_dma_attr(adev);

diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
index 66533e1..2766dbe 100644
--- a/include/linux/dma-mapping.h
+++ b/include/linux/dma-mapping.h
@@ -656,6 +656,9 @@ dma_mark_declared_memory_occupied(struct device *dev,
}
#endif /* CONFIG_HAVE_GENERIC_DMA_COHERENT */

+int dma_configure(struct device *dev);
+void dma_deconfigure(struct device *dev);
+
/*
* Managed DMA API
*/

[Robin Murphy]

+Lorenzo

Post by Sricharan R
Configuring DMA ops at probe time will allow deferring device probe when
the IOMMU isn't available yet. The dma_configure for the device is now called
from the generic device_attach callback just before the bus/driver probe
is called. This way, configuring the dma ops for the device would be called
at the same place for all bus_types, hence the deferred probing mechanism
should work for all buses as well.
pci_bus_add_devices (platform/amba)(_device_create/driver_register)
| |
pci_bus_add_device (device_add/driver_register)
| |
device_attach device_initial_probe
| |
__device_attach_driver __device_attach_driver
|
driver_probe_device
|
really_probe
|
dma_configure
Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.
---
drivers/base/dd.c | 10 ++++++++++
drivers/base/dma-mapping.c | 11 +++++++++++
drivers/of/platform.c | 4 ----
drivers/pci/probe.c | 5 +----
include/linux/dma-mapping.h | 3 +++
5 files changed, 25 insertions(+), 8 deletions(-)
diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index 16688f5..cfebd48 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -19,6 +19,7 @@
#include <linux/device.h>
#include <linux/delay.h>
+#include <linux/dma-mapping.h>
#include <linux/module.h>
#include <linux/kthread.h>
#include <linux/wait.h>
@@ -353,6 +354,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
if (ret)
goto pinctrl_bind_failed;
+ ret = dma_configure(dev);
+ if (ret)
+ goto dma_failed;
+
if (driver_sysfs_add(dev)) {
printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
__func__, dev_name(dev));
@@ -395,6 +400,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
goto done;
+ dma_deconfigure(dev);
if (dev->bus)
blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
@@ -780,6 +787,9 @@ static void __device_release_driver(struct device *dev)
dev->bus->remove(dev);
else if (drv->remove)
drv->remove(dev);
+
+ dma_deconfigure(dev);
+
devres_release_all(dev);
dev->driver = NULL;
dev_set_drvdata(dev, NULL);
diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index d799662..54e87f5 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -10,6 +10,7 @@
#include <linux/dma-mapping.h>
#include <linux/export.h>
#include <linux/gfp.h>
+#include <linux/of_device.h>
#include <linux/slab.h>
#include <linux/vmalloc.h>
@@ -166,6 +167,16 @@ void dmam_free_noncoherent(struct device *dev, size_t size, void *vaddr,
}
EXPORT_SYMBOL(dmam_free_noncoherent);
+int dma_configure(struct device *dev)
+{
+ return of_dma_configure(dev, dev->of_node);
+}
+
+void dma_deconfigure(struct device *dev)
+{
+ of_dma_deconfigure(dev);
+}
+
#ifdef CONFIG_HAVE_GENERIC_DMA_COHERENT
static void dmam_coherent_decl_release(struct device *dev, void *res)
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index 9cb7090..adbd77c 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -181,11 +181,9 @@ static struct platform_device *of_platform_device_create_pdata(
dev->dev.bus = &platform_bus_type;
dev->dev.platform_data = platform_data;
- of_dma_configure(&dev->dev, dev->dev.of_node);
of_msi_configure(&dev->dev, dev->dev.of_node);
if (of_device_add(dev) != 0) {
- of_dma_deconfigure(&dev->dev);
platform_device_put(dev);
goto err_clear_flag;
}
@@ -242,7 +240,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
dev_set_name(&dev->dev, "%s", bus_id);
else
of_device_make_bus_id(&dev->dev);
- of_dma_configure(&dev->dev, dev->dev.of_node);
/* Allow the HW Peripheral ID to be overridden */
prop = of_get_property(node, "arm,primecell-periphid", NULL);
@@ -536,7 +533,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
amba_device_unregister(to_amba_device(dev));
#endif
- of_dma_deconfigure(dev);
of_node_clear_flag(dev->of_node, OF_POPULATED);
of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
return 0;
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 93f280d..85c9553 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1724,10 +1724,7 @@ static void pci_dma_configure(struct pci_dev *dev)
{
struct device *bridge = pci_get_host_bridge_device(dev);
- if (IS_ENABLED(CONFIG_OF) &&
- bridge->parent && bridge->parent->of_node) {
- of_dma_configure(&dev->dev, bridge->parent->of_node);
- } else if (has_acpi_companion(bridge)) {
+ if (has_acpi_companion(bridge)) {
struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
enum dev_dma_attr attr = acpi_get_dma_attr(adev);

It seems a bit awkward leaving pci_dma_configure here, doing DMA
configuration at device add, after we've allegedly moved DMA
configuration to driver probe. Lorenzo, do you foresee any issues if
this probe-time of_dma_configure() path were to also multiplex
acpi_dma_configure() in future, such that everything would be back in
the same place eventually?

Conversely, is there actually any issue with leaving pci_dma_configure()
unchanged, and simply moving the call from pci_device_add() into
dma_configure()?

Robin.

Post by Sricharan R
diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
index 66533e1..2766dbe 100644
--- a/include/linux/dma-mapping.h
+++ b/include/linux/dma-mapping.h
@@ -656,6 +656,9 @@ dma_mark_declared_memory_occupied(struct device *dev,
}
#endif /* CONFIG_HAVE_GENERIC_DMA_COHERENT */
+int dma_configure(struct device *dev);
+void dma_deconfigure(struct device *dev);
+
/*
* Managed DMA API
*/

[Sricharan]

Hi Robin,

Post by Robin Murphy

Post by Sricharan R
Configuring DMA ops at probe time will allow deferring device probe when
the IOMMU isn't available yet. The dma_configure for the device is now called
from the generic device_attach callback just before the bus/driver probe
is called. This way, configuring the dma ops for the device would be called
at the same place for all bus_types, hence the deferred probing mechanism
should work for all buses as well.
pci_bus_add_devices (platform/amba)(_device_create/driver_register)
| |
pci_bus_add_device (device_add/driver_register)
| |
device_attach device_initial_probe
| |
__device_attach_driver __device_attach_driver
|
driver_probe_device
|
really_probe
|
dma_configure
Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.
---
drivers/base/dd.c | 10 ++++++++++
drivers/base/dma-mapping.c | 11 +++++++++++
drivers/of/platform.c | 4 ----
drivers/pci/probe.c | 5 +----
include/linux/dma-mapping.h | 3 +++
5 files changed, 25 insertions(+), 8 deletions(-)
diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index 16688f5..cfebd48 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -19,6 +19,7 @@
#include <linux/device.h>
#include <linux/delay.h>
+#include <linux/dma-mapping.h>
#include <linux/module.h>
#include <linux/kthread.h>
#include <linux/wait.h>
@@ -353,6 +354,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
if (ret)
goto pinctrl_bind_failed;
+ ret = dma_configure(dev);
+ if (ret)
+ goto dma_failed;
+
if (driver_sysfs_add(dev)) {
printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
__func__, dev_name(dev));
@@ -395,6 +400,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
goto done;
+ dma_deconfigure(dev);
if (dev->bus)
blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
@@ -780,6 +787,9 @@ static void __device_release_driver(struct device *dev)
dev->bus->remove(dev);
else if (drv->remove)
drv->remove(dev);
+
+ dma_deconfigure(dev);
+
devres_release_all(dev);
dev->driver = NULL;
dev_set_drvdata(dev, NULL);
diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index d799662..54e87f5 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -10,6 +10,7 @@
#include <linux/dma-mapping.h>
#include <linux/export.h>
#include <linux/gfp.h>
+#include <linux/of_device.h>
#include <linux/slab.h>
#include <linux/vmalloc.h>
@@ -166,6 +167,16 @@ void dmam_free_noncoherent(struct device *dev, size_t size, void *vaddr,
}
EXPORT_SYMBOL(dmam_free_noncoherent);
+int dma_configure(struct device *dev)
+{
+ return of_dma_configure(dev, dev->of_node);
+}
+
+void dma_deconfigure(struct device *dev)
+{
+ of_dma_deconfigure(dev);
+}
+
#ifdef CONFIG_HAVE_GENERIC_DMA_COHERENT
static void dmam_coherent_decl_release(struct device *dev, void *res)
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index 9cb7090..adbd77c 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -181,11 +181,9 @@ static struct platform_device *of_platform_device_create_pdata(
dev->dev.bus = &platform_bus_type;
dev->dev.platform_data = platform_data;
- of_dma_configure(&dev->dev, dev->dev.of_node);
of_msi_configure(&dev->dev, dev->dev.of_node);
if (of_device_add(dev) != 0) {
- of_dma_deconfigure(&dev->dev);
platform_device_put(dev);
goto err_clear_flag;
}
@@ -242,7 +240,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
dev_set_name(&dev->dev, "%s", bus_id);
else
of_device_make_bus_id(&dev->dev);
- of_dma_configure(&dev->dev, dev->dev.of_node);
/* Allow the HW Peripheral ID to be overridden */
prop = of_get_property(node, "arm,primecell-periphid", NULL);
@@ -536,7 +533,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
amba_device_unregister(to_amba_device(dev));
#endif
- of_dma_deconfigure(dev);
of_node_clear_flag(dev->of_node, OF_POPULATED);
of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
return 0;
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 93f280d..85c9553 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1724,10 +1724,7 @@ static void pci_dma_configure(struct pci_dev *dev)
{
struct device *bridge = pci_get_host_bridge_device(dev);
- if (IS_ENABLED(CONFIG_OF) &&
- bridge->parent && bridge->parent->of_node) {
- of_dma_configure(&dev->dev, bridge->parent->of_node);
- } else if (has_acpi_companion(bridge)) {
+ if (has_acpi_companion(bridge)) {
struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
enum dev_dma_attr attr = acpi_get_dma_attr(adev);

It seems a bit awkward leaving pci_dma_configure here, doing DMA
configuration at device add, after we've allegedly moved DMA
configuration to driver probe. Lorenzo, do you foresee any issues if
this probe-time of_dma_configure() path were to also multiplex
acpi_dma_configure() in future, such that everything would be back in
the same place eventually?
Conversely, is there actually any issue with leaving pci_dma_configure()
unchanged, and simply moving the call from pci_device_add() into
dma_configure()?

Ya i removed only the CONFIG_OF part out of this, and was hoping that
the acpi configure can also be called from the dma_configure during
probe later. I did not have an ACPI based platform though.
As you said, if that looks right to the ACPI world, it can be
moved out from here. I can try mergin series (after fixing the vfio errors
part) with the ACPI IO port and see how it behaves.

Regards,
Sricharan

[Lorenzo Pieralisi]

Post by Sricharan
Hi Robin,

Post by Robin Murphy

Post by Sricharan R
Configuring DMA ops at probe time will allow deferring device probe when
the IOMMU isn't available yet. The dma_configure for the device is now called
from the generic device_attach callback just before the bus/driver probe
is called. This way, configuring the dma ops for the device would be called
at the same place for all bus_types, hence the deferred probing mechanism
should work for all buses as well.
pci_bus_add_devices (platform/amba)(_device_create/driver_register)
| |
pci_bus_add_device (device_add/driver_register)
| |
device_attach device_initial_probe
| |
__device_attach_driver __device_attach_driver
|
driver_probe_device
|
really_probe
|
dma_configure
Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.
---
drivers/base/dd.c | 10 ++++++++++
drivers/base/dma-mapping.c | 11 +++++++++++
drivers/of/platform.c | 4 ----
drivers/pci/probe.c | 5 +----
include/linux/dma-mapping.h | 3 +++
5 files changed, 25 insertions(+), 8 deletions(-)
diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index 16688f5..cfebd48 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -19,6 +19,7 @@
#include <linux/device.h>
#include <linux/delay.h>
+#include <linux/dma-mapping.h>
#include <linux/module.h>
#include <linux/kthread.h>
#include <linux/wait.h>
@@ -353,6 +354,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
if (ret)
goto pinctrl_bind_failed;
+ ret = dma_configure(dev);
+ if (ret)
+ goto dma_failed;
+
if (driver_sysfs_add(dev)) {
printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
__func__, dev_name(dev));
@@ -395,6 +400,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
goto done;
+ dma_deconfigure(dev);
if (dev->bus)
blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
@@ -780,6 +787,9 @@ static void __device_release_driver(struct device *dev)
dev->bus->remove(dev);
else if (drv->remove)
drv->remove(dev);
+
+ dma_deconfigure(dev);
+
devres_release_all(dev);
dev->driver = NULL;
dev_set_drvdata(dev, NULL);
diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index d799662..54e87f5 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -10,6 +10,7 @@
#include <linux/dma-mapping.h>
#include <linux/export.h>
#include <linux/gfp.h>
+#include <linux/of_device.h>
#include <linux/slab.h>
#include <linux/vmalloc.h>
@@ -166,6 +167,16 @@ void dmam_free_noncoherent(struct device *dev, size_t size, void *vaddr,
}
EXPORT_SYMBOL(dmam_free_noncoherent);
+int dma_configure(struct device *dev)
+{
+ return of_dma_configure(dev, dev->of_node);
+}
+
+void dma_deconfigure(struct device *dev)
+{
+ of_dma_deconfigure(dev);
+}
+
#ifdef CONFIG_HAVE_GENERIC_DMA_COHERENT
static void dmam_coherent_decl_release(struct device *dev, void *res)
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index 9cb7090..adbd77c 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -181,11 +181,9 @@ static struct platform_device *of_platform_device_create_pdata(
dev->dev.bus = &platform_bus_type;
dev->dev.platform_data = platform_data;
- of_dma_configure(&dev->dev, dev->dev.of_node);
of_msi_configure(&dev->dev, dev->dev.of_node);
if (of_device_add(dev) != 0) {
- of_dma_deconfigure(&dev->dev);
platform_device_put(dev);
goto err_clear_flag;
}
@@ -242,7 +240,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
dev_set_name(&dev->dev, "%s", bus_id);
else
of_device_make_bus_id(&dev->dev);
- of_dma_configure(&dev->dev, dev->dev.of_node);
/* Allow the HW Peripheral ID to be overridden */
prop = of_get_property(node, "arm,primecell-periphid", NULL);
@@ -536,7 +533,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
amba_device_unregister(to_amba_device(dev));
#endif
- of_dma_deconfigure(dev);
of_node_clear_flag(dev->of_node, OF_POPULATED);
of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
return 0;
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 93f280d..85c9553 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1724,10 +1724,7 @@ static void pci_dma_configure(struct pci_dev *dev)
{
struct device *bridge = pci_get_host_bridge_device(dev);
- if (IS_ENABLED(CONFIG_OF) &&
- bridge->parent && bridge->parent->of_node) {
- of_dma_configure(&dev->dev, bridge->parent->of_node);
- } else if (has_acpi_companion(bridge)) {
+ if (has_acpi_companion(bridge)) {
struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
enum dev_dma_attr attr = acpi_get_dma_attr(adev);

It seems a bit awkward leaving pci_dma_configure here, doing DMA
configuration at device add, after we've allegedly moved DMA
configuration to driver probe. Lorenzo, do you foresee any issues if
this probe-time of_dma_configure() path were to also multiplex
acpi_dma_configure() in future, such that everything would be back in
the same place eventually?
Conversely, is there actually any issue with leaving pci_dma_configure()
unchanged, and simply moving the call from pci_device_add() into
dma_configure()?

Ya i removed only the CONFIG_OF part out of this, and was hoping that
the acpi configure can also be called from the dma_configure during
probe later. I did not have an ACPI based platform though. As you
said, if that looks right to the ACPI world, it can be moved out from
here. I can try mergin series (after fixing the vfio errors part) with
the ACPI IO port and see how it behaves.

I do not expect any issue from this change, as long as, as Robin said,
it is done consistently which means that I am not ok with this patch
as it stands (ie you should defer pci_dma_configure() in its entirety,
not just the DT bits, which obviously requires some ACPI testing too).

CC me in please in the next posting since this affects the ACPI probing
path too, we have to coordinate this series with my ACPI IORT SMMU
enablement patches:

https://lkml.org/lkml/2016/10/18/506

Thanks,
Lorenzo

[Sricharan]

Hi Lorenzo,

-----Original Message-----
Sent: Thursday, October 27, 2016 4:19 PM
Subject: Re: [PATCH V3 4/8] drivers: platform: Configure dma operations at probe time

Post by Sricharan
Hi Robin,

Post by Robin Murphy

Post by Sricharan R
Configuring DMA ops at probe time will allow deferring device probe when
the IOMMU isn't available yet. The dma_configure for the device is now called
from the generic device_attach callback just before the bus/driver probe
is called. This way, configuring the dma ops for the device would be called
at the same place for all bus_types, hence the deferred probing mechanism
should work for all buses as well.
pci_bus_add_devices (platform/amba)(_device_create/driver_register)
| |
pci_bus_add_device (device_add/driver_register)
| |
device_attach device_initial_probe
| |
__device_attach_driver __device_attach_driver
|
driver_probe_device
|
really_probe
|
dma_configure
Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.
---
drivers/base/dd.c | 10 ++++++++++
drivers/base/dma-mapping.c | 11 +++++++++++
drivers/of/platform.c | 4 ----
drivers/pci/probe.c | 5 +----
include/linux/dma-mapping.h | 3 +++
5 files changed, 25 insertions(+), 8 deletions(-)
diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index 16688f5..cfebd48 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -19,6 +19,7 @@
#include <linux/device.h>
#include <linux/delay.h>
+#include <linux/dma-mapping.h>
#include <linux/module.h>
#include <linux/kthread.h>
#include <linux/wait.h>
@@ -353,6 +354,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
if (ret)
goto pinctrl_bind_failed;
+ ret = dma_configure(dev);
+ if (ret)
+ goto dma_failed;
+
if (driver_sysfs_add(dev)) {
printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
__func__, dev_name(dev));
@@ -395,6 +400,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
goto done;
+ dma_deconfigure(dev);
if (dev->bus)
blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
@@ -780,6 +787,9 @@ static void __device_release_driver(struct device *dev)
dev->bus->remove(dev);
else if (drv->remove)
drv->remove(dev);
+
+ dma_deconfigure(dev);
+
devres_release_all(dev);
dev->driver = NULL;
dev_set_drvdata(dev, NULL);
diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index d799662..54e87f5 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -10,6 +10,7 @@
#include <linux/dma-mapping.h>
#include <linux/export.h>
#include <linux/gfp.h>
+#include <linux/of_device.h>
#include <linux/slab.h>
#include <linux/vmalloc.h>
@@ -166,6 +167,16 @@ void dmam_free_noncoherent(struct device *dev, size_t size, void *vaddr,
}
EXPORT_SYMBOL(dmam_free_noncoherent);
+int dma_configure(struct device *dev)
+{
+ return of_dma_configure(dev, dev->of_node);
+}
+
+void dma_deconfigure(struct device *dev)
+{
+ of_dma_deconfigure(dev);
+}
+
#ifdef CONFIG_HAVE_GENERIC_DMA_COHERENT
static void dmam_coherent_decl_release(struct device *dev, void *res)
diff --git a/drivers/of/platform.c b/drivers/of/platform.c
index 9cb7090..adbd77c 100644
--- a/drivers/of/platform.c
+++ b/drivers/of/platform.c
@@ -181,11 +181,9 @@ static struct platform_device *of_platform_device_create_pdata(
dev->dev.bus = &platform_bus_type;
dev->dev.platform_data = platform_data;
- of_dma_configure(&dev->dev, dev->dev.of_node);
of_msi_configure(&dev->dev, dev->dev.of_node);
if (of_device_add(dev) != 0) {
- of_dma_deconfigure(&dev->dev);
platform_device_put(dev);
goto err_clear_flag;
}
@@ -242,7 +240,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
dev_set_name(&dev->dev, "%s", bus_id);
else
of_device_make_bus_id(&dev->dev);
- of_dma_configure(&dev->dev, dev->dev.of_node);
/* Allow the HW Peripheral ID to be overridden */
prop = of_get_property(node, "arm,primecell-periphid", NULL);
@@ -536,7 +533,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
amba_device_unregister(to_amba_device(dev));
#endif
- of_dma_deconfigure(dev);
of_node_clear_flag(dev->of_node, OF_POPULATED);
of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
return 0;
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 93f280d..85c9553 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1724,10 +1724,7 @@ static void pci_dma_configure(struct pci_dev *dev)
{
struct device *bridge = pci_get_host_bridge_device(dev);
- if (IS_ENABLED(CONFIG_OF) &&
- bridge->parent && bridge->parent->of_node) {
- of_dma_configure(&dev->dev, bridge->parent->of_node);
- } else if (has_acpi_companion(bridge)) {
+ if (has_acpi_companion(bridge)) {
struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
enum dev_dma_attr attr = acpi_get_dma_attr(adev);

It seems a bit awkward leaving pci_dma_configure here, doing DMA
configuration at device add, after we've allegedly moved DMA
configuration to driver probe. Lorenzo, do you foresee any issues if
this probe-time of_dma_configure() path were to also multiplex
acpi_dma_configure() in future, such that everything would be back in
the same place eventually?
Conversely, is there actually any issue with leaving pci_dma_configure()
unchanged, and simply moving the call from pci_device_add() into
dma_configure()?

Ya i removed only the CONFIG_OF part out of this, and was hoping that
the acpi configure can also be called from the dma_configure during
probe later. I did not have an ACPI based platform though. As you
said, if that looks right to the ACPI world, it can be moved out from
here. I can try mergin series (after fixing the vfio errors part) with
the ACPI IO port and see how it behaves.

I do not expect any issue from this change, as long as, as Robin said,
it is done consistently which means that I am not ok with this patch
as it stands (ie you should defer pci_dma_configure() in its entirety,
not just the DT bits, which obviously requires some ACPI testing too).
CC me in please in the next posting since this affects the ACPI probing
path too, we have to coordinate this series with my ACPI IORT SMMU
https://lkml.org/lkml/2016/10/18/506

Ok thanks, i will have you in looped for the V4 i am posting
and we can have it tested on ACPI as well.

Regards,
Sricharan

[Sricharan R]

From: Laurent Pinchart <laurent.pinchart+***@ideasonboard.com>

Failures to look up an IOMMU when parsing the DT iommus property need to
be handled separately from the .of_xlate() failures to support deferred
probing.

The lack of a registered IOMMU can be caused by the lack of a driver for
the IOMMU, the IOMMU device probe not having been performed yet, having
been deferred, or having failed.

The first case occurs when the device tree describes the bus master and
IOMMU topology correctly but no device driver exists for the IOMMU yet
or the device driver has not been compiled in. Return NULL, the caller
will configure the device without an IOMMU.

The second and third cases are handled by deferring the probe of the bus
master device which will eventually get reprobed after the IOMMU.

The last case is currently handled by deferring the probe of the bus
master device as well. A mechanism to either configure the bus master
device without an IOMMU or to fail the bus master device probe depending
on whether the IOMMU is optional or mandatory would be a good
enhancement.

The current iommu framework handles pci and non-pci devices separately,
so taken care of both the paths in this patch. The iommu's add_device
callback is invoked after the master's configuration data is added in
xlate. This is needed because the iommu core calls add_device callback
during the BUS_ADD_DEVICE notifier, which is of no use now. Eventually
that call has to be removed.

Signed-off-by: Sricharan R <***@codeaurora.org>
---
drivers/iommu/of_iommu.c | 47 +++++++++++++++++++++++++++++++++++++++++++----
drivers/of/device.c | 7 ++++++-
include/linux/of_device.h | 6 ++++--
3 files changed, 53 insertions(+), 7 deletions(-)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 5b82862..1a5e28b 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -23,6 +23,7 @@
#include <linux/of.h>
#include <linux/of_iommu.h>
#include <linux/of_pci.h>
+#include <linux/pci.h>
#include <linux/slab.h>

static const struct of_device_id __iommu_of_table_sentinel
@@ -167,12 +168,29 @@ static const struct iommu_ops
return NULL;

ops = of_iommu_get_ops(iommu_spec.np);
+
+ if (!ops) {
+ const struct of_device_id *oid;
+
+ oid = of_match_node(&__iommu_of_table, iommu_spec.np);
+ ops = oid ? ERR_PTR(-EPROBE_DEFER) : NULL;
+ return ops;
+ }
+
if (!ops || !ops->of_xlate ||
iommu_fwspec_init(&pdev->dev, &iommu_spec.np->fwnode, ops) ||
ops->of_xlate(&pdev->dev, &iommu_spec))
ops = NULL;

+ if (ops && ops->add_device) {
+ ops = (ops->add_device(&pdev->dev) == 0) ? ops : NULL;
+
+ if (!ops)
+ dev_err(&pdev->dev, "Failed to setup iommu ops\n");
+ }
+
of_node_put(iommu_spec.np);
+
return ops;
}

@@ -183,9 +201,15 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
struct device_node *np;
const struct iommu_ops *ops = NULL;
int idx = 0;
+ struct device *bridge;
+
+ if (dev_is_pci(dev)) {
+ bridge = pci_get_host_bridge_device(to_pci_dev(dev));

- if (dev_is_pci(dev))
- return of_pci_iommu_configure(to_pci_dev(dev), master_np);
+ if (bridge && bridge->parent && bridge->parent->of_node)
+ return of_pci_iommu_configure(to_pci_dev(dev),
+ bridge->parent->of_node);
+ }

/*
* We don't currently walk up the tree looking for a parent IOMMU.
@@ -198,6 +222,14 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
np = iommu_spec.np;
ops = of_iommu_get_ops(np);

+ if (!ops) {
+ const struct of_device_id *oid;
+
+ oid = of_match_node(&__iommu_of_table, np);
+ ops = oid ? ERR_PTR(-EPROBE_DEFER) : NULL;
+ goto err_put_node;
+ }
+
if (!ops || !ops->of_xlate ||
iommu_fwspec_init(dev, &np->fwnode, ops) ||
ops->of_xlate(dev, &iommu_spec))
@@ -207,11 +239,18 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
idx++;
}

+ if (ops && ops->add_device) {
+ ops = (ops->add_device(dev) == 0) ? ops : NULL;
+
+ if (!ops)
+ dev_err(dev, "Failed to setup iommu_ops\n");
+ }
+
return ops;

err_put_node:
of_node_put(np);
- return NULL;
+ return ops;
}

static int __init of_iommu_init(void)
@@ -222,7 +261,7 @@ static int __init of_iommu_init(void)
for_each_matching_node_and_match(np, matches, &match) {
const of_iommu_init_fn init_fn = match->data;

- if (init_fn(np))
+ if (init_fn && init_fn(np))
pr_err("Failed to initialise IOMMU %s\n",
of_node_full_name(np));
}
diff --git a/drivers/of/device.c b/drivers/of/device.c
index 1c843e2..c8e74d7 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
* can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
* to fix up DMA configuration.
*/
-void of_dma_configure(struct device *dev, struct device_node *np)
+int of_dma_configure(struct device *dev, struct device_node *np)
{
u64 dma_addr, paddr, size;
int ret;
@@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
coherent ? " " : " not ");

iommu = of_iommu_configure(dev, np);
+ if (IS_ERR(iommu))
+ return PTR_ERR(iommu);
+
dev_dbg(dev, "device is%sbehind an iommu\n",
iommu ? " " : " not ");

arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
+
+ return 0;
}
EXPORT_SYMBOL_GPL(of_dma_configure);

diff --git a/include/linux/of_device.h b/include/linux/of_device.h
index d20a31a..2bdb872 100644
--- a/include/linux/of_device.h
+++ b/include/linux/of_device.h
@@ -55,7 +55,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
return of_node_get(cpu_dev->of_node);
}

-void of_dma_configure(struct device *dev, struct device_node *np);
+int of_dma_configure(struct device *dev, struct device_node *np);
void of_dma_deconfigure(struct device *dev);
#else /* CONFIG_OF */

@@ -100,7 +100,9 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
return NULL;
}
static inline void of_dma_configure(struct device *dev, struct device_node *np)
-{}
+{
+ return 0;
+}
static inline void of_dma_deconfigure(struct device *dev)
{}
#endif /* CONFIG_OF */

[Robin Murphy]

Post by Sricharan R
Failures to look up an IOMMU when parsing the DT iommus property need to
be handled separately from the .of_xlate() failures to support deferred
probing.
The lack of a registered IOMMU can be caused by the lack of a driver for
the IOMMU, the IOMMU device probe not having been performed yet, having
been deferred, or having failed.
The first case occurs when the device tree describes the bus master and
IOMMU topology correctly but no device driver exists for the IOMMU yet
or the device driver has not been compiled in. Return NULL, the caller
will configure the device without an IOMMU.
The second and third cases are handled by deferring the probe of the bus
master device which will eventually get reprobed after the IOMMU.
The last case is currently handled by deferring the probe of the bus
master device as well. A mechanism to either configure the bus master
device without an IOMMU or to fail the bus master device probe depending
on whether the IOMMU is optional or mandatory would be a good
enhancement.
The current iommu framework handles pci and non-pci devices separately,
so taken care of both the paths in this patch. The iommu's add_device
callback is invoked after the master's configuration data is added in
xlate. This is needed because the iommu core calls add_device callback
during the BUS_ADD_DEVICE notifier, which is of no use now. Eventually
that call has to be removed.

Laurent's signoff seems to have gone missing here.

Post by Sricharan R
---
drivers/iommu/of_iommu.c | 47 +++++++++++++++++++++++++++++++++++++++++++----
drivers/of/device.c | 7 ++++++-
include/linux/of_device.h | 6 ++++--
3 files changed, 53 insertions(+), 7 deletions(-)
diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 5b82862..1a5e28b 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -23,6 +23,7 @@
#include <linux/of.h>
#include <linux/of_iommu.h>
#include <linux/of_pci.h>
+#include <linux/pci.h>
#include <linux/slab.h>
static const struct of_device_id __iommu_of_table_sentinel
@@ -167,12 +168,29 @@ static const struct iommu_ops
return NULL;
ops = of_iommu_get_ops(iommu_spec.np);
+
+ if (!ops) {
+ const struct of_device_id *oid;
+
+ oid = of_match_node(&__iommu_of_table, iommu_spec.np);
+ ops = oid ? ERR_PTR(-EPROBE_DEFER) : NULL;

It would seem even simpler and more convenient to roll this into the end
of of_iommu_get_ops():

if (!ops && of_match_node(&__iommu_of_table, iommu_spec.np))
ops = ERR_PTR(-EPROBE_DEFER);

then just fix up the existing !ops checks to !IS_ERR(ops) appropriately.

Post by Sricharan R
+ return ops;
+ }
+
if (!ops || !ops->of_xlate ||
iommu_fwspec_init(&pdev->dev, &iommu_spec.np->fwnode, ops) ||
ops->of_xlate(&pdev->dev, &iommu_spec))
ops = NULL;
+ if (ops && ops->add_device) {
+ ops = (ops->add_device(&pdev->dev) == 0) ? ops : NULL;

This is a really obtuse way of writing

if (ops->add_device(...))
ops = NULL;

However, given that we're now returning an ERR_PTR, it would be worth
capturing the return value of add_device and propagating the error back
up - if the IOMMU driver has refused one of its masters for some reason,
it probably isn't safe to allow that device to do DMA either way, so we
ought to prevent it probing at all.

Post by Sricharan R
+
+ if (!ops)
+ dev_err(&pdev->dev, "Failed to setup iommu ops\n");

Given the above, I think this should be more along the lines of "Device
rejected by IOMMU: %d" with the actual error code as well. It's one of
those "if you ever see it, you're going to have to debug it" cases, so
the clearer the better.

Post by Sricharan R
+ }
+
of_node_put(iommu_spec.np);
+
return ops;
}
@@ -183,9 +201,15 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
struct device_node *np;
const struct iommu_ops *ops = NULL;
int idx = 0;
+ struct device *bridge;
+
+ if (dev_is_pci(dev)) {
+ bridge = pci_get_host_bridge_device(to_pci_dev(dev));
- if (dev_is_pci(dev))
- return of_pci_iommu_configure(to_pci_dev(dev), master_np);
+ if (bridge && bridge->parent && bridge->parent->of_node)
+ return of_pci_iommu_configure(to_pci_dev(dev),
+ bridge->parent->of_node);

else fall through to treating it as a platform device?

...that's not right. Anyway, this is PCI-specific stuff so should be in
the PCI-specific helper function.

Post by Sricharan R
+ }
/*
* We don't currently walk up the tree looking for a parent IOMMU.
@@ -198,6 +222,14 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
np = iommu_spec.np;
ops = of_iommu_get_ops(np);
+ if (!ops) {
+ const struct of_device_id *oid;
+
+ oid = of_match_node(&__iommu_of_table, np);
+ ops = oid ? ERR_PTR(-EPROBE_DEFER) : NULL;
+ goto err_put_node;
+ }

Same comment as above. Especially since moving it to of_iommu_get_ops()
would obviate the duplication.

Post by Sricharan R
+
if (!ops || !ops->of_xlate ||
iommu_fwspec_init(dev, &np->fwnode, ops) ||
ops->of_xlate(dev, &iommu_spec))
@@ -207,11 +239,18 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
idx++;
}
+ if (ops && ops->add_device) {
+ ops = (ops->add_device(dev) == 0) ? ops : NULL;
+
+ if (!ops)
+ dev_err(dev, "Failed to setup iommu_ops\n");
+ }
+

It would be nice to avoid duplicating this as well.

Robin.

Post by Sricharan R
return ops;
of_node_put(np);
- return NULL;
+ return ops;
}
static int __init of_iommu_init(void)
@@ -222,7 +261,7 @@ static int __init of_iommu_init(void)
for_each_matching_node_and_match(np, matches, &match) {
const of_iommu_init_fn init_fn = match->data;
- if (init_fn(np))
+ if (init_fn && init_fn(np))
pr_err("Failed to initialise IOMMU %s\n",
of_node_full_name(np));
}
diff --git a/drivers/of/device.c b/drivers/of/device.c
index 1c843e2..c8e74d7 100644
--- a/drivers/of/device.c
+++ b/drivers/of/device.c
@@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
* can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
* to fix up DMA configuration.
*/
-void of_dma_configure(struct device *dev, struct device_node *np)
+int of_dma_configure(struct device *dev, struct device_node *np)
{
u64 dma_addr, paddr, size;
int ret;
@@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
coherent ? " " : " not ");
iommu = of_iommu_configure(dev, np);
+ if (IS_ERR(iommu))
+ return PTR_ERR(iommu);
+
dev_dbg(dev, "device is%sbehind an iommu\n",
iommu ? " " : " not ");
arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
+
+ return 0;
}
EXPORT_SYMBOL_GPL(of_dma_configure);
diff --git a/include/linux/of_device.h b/include/linux/of_device.h
index d20a31a..2bdb872 100644
--- a/include/linux/of_device.h
+++ b/include/linux/of_device.h
@@ -55,7 +55,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
return of_node_get(cpu_dev->of_node);
}
-void of_dma_configure(struct device *dev, struct device_node *np);
+int of_dma_configure(struct device *dev, struct device_node *np);
void of_dma_deconfigure(struct device *dev);
#else /* CONFIG_OF */
@@ -100,7 +100,9 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
return NULL;
}
static inline void of_dma_configure(struct device *dev, struct device_node *np)
-{}
+{
+ return 0;
+}
static inline void of_dma_deconfigure(struct device *dev)
{}
#endif /* CONFIG_OF */

[Sricharan]

Hi Robin,

Post by Robin Murphy

Post by Sricharan R
Failures to look up an IOMMU when parsing the DT iommus property need to
be handled separately from the .of_xlate() failures to support deferred
probing.
The lack of a registered IOMMU can be caused by the lack of a driver for
the IOMMU, the IOMMU device probe not having been performed yet, having
been deferred, or having failed.
The first case occurs when the device tree describes the bus master and
IOMMU topology correctly but no device driver exists for the IOMMU yet
or the device driver has not been compiled in. Return NULL, the caller
will configure the device without an IOMMU.
The second and third cases are handled by deferring the probe of the bus
master device which will eventually get reprobed after the IOMMU.
The last case is currently handled by deferring the probe of the bus
master device as well. A mechanism to either configure the bus master
device without an IOMMU or to fail the bus master device probe depending
on whether the IOMMU is optional or mandatory would be a good
enhancement.
The current iommu framework handles pci and non-pci devices separately,
so taken care of both the paths in this patch. The iommu's add_device
callback is invoked after the master's configuration data is added in
xlate. This is needed because the iommu core calls add_device callback
during the BUS_ADD_DEVICE notifier, which is of no use now. Eventually
that call has to be removed.

Laurent's signoff seems to have gone missing here.

Ah, preserved his authorship, but missed this. Will add it back.

Post by Robin Murphy

Post by Sricharan R
---
drivers/iommu/of_iommu.c | 47 +++++++++++++++++++++++++++++++++++++++++++----
drivers/of/device.c | 7 ++++++-
include/linux/of_device.h | 6 ++++--
3 files changed, 53 insertions(+), 7 deletions(-)
diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 5b82862..1a5e28b 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c
@@ -23,6 +23,7 @@
#include <linux/of.h>
#include <linux/of_iommu.h>
#include <linux/of_pci.h>
+#include <linux/pci.h>
#include <linux/slab.h>
static const struct of_device_id __iommu_of_table_sentinel
@@ -167,12 +168,29 @@ static const struct iommu_ops
return NULL;
ops = of_iommu_get_ops(iommu_spec.np);
+
+ if (!ops) {
+ const struct of_device_id *oid;
+
+ oid = of_match_node(&__iommu_of_table, iommu_spec.np);
+ ops = oid ? ERR_PTR(-EPROBE_DEFER) : NULL;

It would seem even simpler and more convenient to roll this into the end

ok, understand. Will move this there.

Post by Robin Murphy
if (!ops && of_match_node(&__iommu_of_table, iommu_spec.np))
ops = ERR_PTR(-EPROBE_DEFER);
then just fix up the existing !ops checks to !IS_ERR(ops) appropriately.

ok.

Post by Robin Murphy

Post by Sricharan R
+ return ops;
+ }
+
if (!ops || !ops->of_xlate ||
iommu_fwspec_init(&pdev->dev, &iommu_spec.np->fwnode, ops) ||
ops->of_xlate(&pdev->dev, &iommu_spec))
ops = NULL;
+ if (ops && ops->add_device) {
+ ops = (ops->add_device(&pdev->dev) == 0) ? ops : NULL;

This is a really obtuse way of writing
if (ops->add_device(...))
ops = NULL;

ok, will change it this way.

Post by Robin Murphy
However, given that we're now returning an ERR_PTR, it would be worth
capturing the return value of add_device and propagating the error back
up - if the IOMMU driver has refused one of its masters for some reason,
it probably isn't safe to allow that device to do DMA either way, so we
ought to prevent it probing at all.

ok, will return the err value instead of NULL here.

Post by Robin Murphy

Post by Sricharan R
+
+ if (!ops)
+ dev_err(&pdev->dev, "Failed to setup iommu ops\n");

Given the above, I think this should be more along the lines of "Device
rejected by IOMMU: %d" with the actual error code as well. It's one of
those "if you ever see it, you're going to have to debug it" cases, so
the clearer the better.

ok, will reword the print.

Post by Robin Murphy

Post by Sricharan R
+ }
+
of_node_put(iommu_spec.np);
+
return ops;
}
@@ -183,9 +201,15 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
struct device_node *np;
const struct iommu_ops *ops = NULL;
int idx = 0;
+ struct device *bridge;
+
+ if (dev_is_pci(dev)) {
+ bridge = pci_get_host_bridge_device(to_pci_dev(dev));
- if (dev_is_pci(dev))
- return of_pci_iommu_configure(to_pci_dev(dev), master_np);
+ if (bridge && bridge->parent && bridge->parent->of_node)
+ return of_pci_iommu_configure(to_pci_dev(dev),
+ bridge->parent->of_node);

else fall through to treating it as a platform device?

ha, surely wrong. Will correct this and move it to the of_pci_iommu_configure helper.

Post by Robin Murphy
...that's not right. Anyway, this is PCI-specific stuff so should be in
the PCI-specific helper function.

Post by Sricharan R
+ }
/*
* We don't currently walk up the tree looking for a parent IOMMU.
@@ -198,6 +222,14 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
np = iommu_spec.np;
ops = of_iommu_get_ops(np);
+ if (!ops) {
+ const struct of_device_id *oid;
+
+ oid = of_match_node(&__iommu_of_table, np);
+ ops = oid ? ERR_PTR(-EPROBE_DEFER) : NULL;
+ goto err_put_node;
+ }

Same comment as above. Especially since moving it to of_iommu_get_ops()
would obviate the duplication.

ok.

Post by Robin Murphy

Post by Sricharan R
+
if (!ops || !ops->of_xlate ||
iommu_fwspec_init(dev, &np->fwnode, ops) ||
ops->of_xlate(dev, &iommu_spec))
@@ -207,11 +239,18 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
idx++;
}
+ if (ops && ops->add_device) {
+ ops = (ops->add_device(dev) == 0) ? ops : NULL;
+
+ if (!ops)
+ dev_err(dev, "Failed to setup iommu_ops\n");
+ }
+

It would be nice to avoid duplicating this as well.

ok, sure, will correct.

Regards,
Sricharan

[Sricharan R]

The dma_ops for the device is not getting set to NULL in
arch_tear_down_dma_ops and this causes an issue when the
device's probe gets deferred and retried. So reset the
dma_ops to NULL.

Signed-off-by: Sricharan R <***@codeaurora.org>
---
arch/arm/mm/dma-mapping.c | 1 +
1 file changed, 1 insertion(+)

diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
index dde6514..b9191f0 100644
--- a/arch/arm/mm/dma-mapping.c
+++ b/arch/arm/mm/dma-mapping.c
@@ -2295,6 +2295,7 @@ static void arm_teardown_iommu_dma_ops(struct device *dev)

__arm_iommu_detach_device(dev);
arm_iommu_release_mapping(mapping);
+ set_dma_ops(dev, NULL);
}

#else

[Robin Murphy]

Post by Sricharan R
The dma_ops for the device is not getting set to NULL in
arch_tear_down_dma_ops and this causes an issue when the
device's probe gets deferred and retried. So reset the
dma_ops to NULL.

Reviewed-by: Robin Murphy <***@arm.com>

This seems like it could stand independently from the rest of the series
- might be worth rewording the commit message to more general terms,
i.e. arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
thus clearing the ops set by the latter, and sending it to Russell as a
fix in its own right.

Robin.

Post by Sricharan R
---
arch/arm/mm/dma-mapping.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
index dde6514..b9191f0 100644
--- a/arch/arm/mm/dma-mapping.c
+++ b/arch/arm/mm/dma-mapping.c
@@ -2295,6 +2295,7 @@ static void arm_teardown_iommu_dma_ops(struct device *dev)
__arm_iommu_detach_device(dev);
arm_iommu_release_mapping(mapping);
+ set_dma_ops(dev, NULL);
}
#else

[Sricharan]

Hi Robin,

-----Original Message-----
Sent: Wednesday, October 26, 2016 8:37 PM
Subject: Re: [PATCH V3 6/8] arm: dma-mapping: Reset the device's dma_ops

Post by Sricharan R
The dma_ops for the device is not getting set to NULL in
arch_tear_down_dma_ops and this causes an issue when the
device's probe gets deferred and retried. So reset the
dma_ops to NULL.

Thanks.

This seems like it could stand independently from the rest of the series
- might be worth rewording the commit message to more general terms,
i.e. arch_teardown_dma_ops() being the inverse of arch_setup_dma_ops()
thus clearing the ops set by the latter, and sending it to Russell as a
fix in its own right.

Ok, will reword the commit log and push this separately.

Regards,
Sricharan

[Sricharan R]

dma_deconfigure calls arch_teardown_dma_ops in the device_detach path,
which is called when the device gets detached from the driver.
When the device was added, iommu's add_device callback was used to
add the device in to its iommu_group and setup the device to be ready
to use its iommu. Similarly, call remove_device callback to remove the
device from the group and reset any other device's iommu configurations.

Signed-off-by: Sricharan R <***@codeaurora.org>
---
arch/arm/mm/dma-mapping.c | 8 ++++++++
arch/arm64/mm/dma-mapping.c | 7 +++++++
2 files changed, 15 insertions(+)

diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
index b9191f0..cbe22de 100644
--- a/arch/arm/mm/dma-mapping.c
+++ b/arch/arm/mm/dma-mapping.c
@@ -2289,11 +2289,19 @@ static bool arm_setup_iommu_dma_ops(struct device *dev, u64 dma_base, u64 size,
static void arm_teardown_iommu_dma_ops(struct device *dev)
{
struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev);
+ const struct iommu_ops *ops;

if (!mapping)
return;

__arm_iommu_detach_device(dev);
+
+ if (dev->iommu_fwspec) {
+ ops = dev->iommu_fwspec->ops;
+ if (ops->remove_device)
+ ops->remove_device(dev);
+ }
+
arm_iommu_release_mapping(mapping);
set_dma_ops(dev, NULL);
}
diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
index 610d8e5..faf4b92 100644
--- a/arch/arm64/mm/dma-mapping.c
+++ b/arch/arm64/mm/dma-mapping.c
@@ -938,6 +938,13 @@ static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
void arch_teardown_dma_ops(struct device *dev)
{
struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
+ const struct iommu_ops *ops;
+
+ if (dev->iommu_fwspec) {
+ ops = dev->iommu_fwspec->ops;
+ if (ops->remove_device)
+ ops->remove_device(dev);
+ }

if (WARN_ON(domain))
iommu_detach_device(domain, dev);

[Robin Murphy]

Post by Sricharan R
dma_deconfigure calls arch_teardown_dma_ops in the device_detach path,
which is called when the device gets detached from the driver.
When the device was added, iommu's add_device callback was used to
add the device in to its iommu_group and setup the device to be ready
to use its iommu. Similarly, call remove_device callback to remove the
device from the group and reset any other device's iommu configurations.
---
arch/arm/mm/dma-mapping.c | 8 ++++++++
arch/arm64/mm/dma-mapping.c | 7 +++++++
2 files changed, 15 insertions(+)
diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
index b9191f0..cbe22de 100644
--- a/arch/arm/mm/dma-mapping.c
+++ b/arch/arm/mm/dma-mapping.c
@@ -2289,11 +2289,19 @@ static bool arm_setup_iommu_dma_ops(struct device *dev, u64 dma_base, u64 size,
static void arm_teardown_iommu_dma_ops(struct device *dev)
{
struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev);
+ const struct iommu_ops *ops;
if (!mapping)
return;
__arm_iommu_detach_device(dev);
+
+ if (dev->iommu_fwspec) {
+ ops = dev->iommu_fwspec->ops;
+ if (ops->remove_device)
+ ops->remove_device(dev);
+ }
+

Yuck. It's a little unfortunate that we have to do this at all, but I
see why. Still, it should be done in common code, not duplicated across
arch code, not least for symmetry with where the matching add_device
happened (although I think of_dma_deconfigure() would suffice, I'm not
sure we really need to add an of_iommu_deconfigure() just for this).

It's also broken for IOMMU drivers which rely on the
of_iommu_configure() mechanism but have not yet been converted to use
iommu_fwspec (Exynos, MSM, etc.)

Robin.

Post by Sricharan R
arm_iommu_release_mapping(mapping);
set_dma_ops(dev, NULL);
}
diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
index 610d8e5..faf4b92 100644
--- a/arch/arm64/mm/dma-mapping.c
+++ b/arch/arm64/mm/dma-mapping.c
@@ -938,6 +938,13 @@ static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
void arch_teardown_dma_ops(struct device *dev)
{
struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
+ const struct iommu_ops *ops;
+
+ if (dev->iommu_fwspec) {
+ ops = dev->iommu_fwspec->ops;
+ if (ops->remove_device)
+ ops->remove_device(dev);
+ }
if (WARN_ON(domain))
iommu_detach_device(domain, dev);

[Sricharan]

Hi Robin,

Post by Robin Murphy

Post by Sricharan R
dma_deconfigure calls arch_teardown_dma_ops in the device_detach path,
which is called when the device gets detached from the driver.
When the device was added, iommu's add_device callback was used to
add the device in to its iommu_group and setup the device to be ready
to use its iommu. Similarly, call remove_device callback to remove the
device from the group and reset any other device's iommu configurations.
---
arch/arm/mm/dma-mapping.c | 8 ++++++++
arch/arm64/mm/dma-mapping.c | 7 +++++++
2 files changed, 15 insertions(+)
diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
index b9191f0..cbe22de 100644
--- a/arch/arm/mm/dma-mapping.c
+++ b/arch/arm/mm/dma-mapping.c
@@ -2289,11 +2289,19 @@ static bool arm_setup_iommu_dma_ops(struct device *dev, u64 dma_base, u64 size,
static void arm_teardown_iommu_dma_ops(struct device *dev)
{
struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev);
+ const struct iommu_ops *ops;
if (!mapping)
return;
__arm_iommu_detach_device(dev);
+
+ if (dev->iommu_fwspec) {
+ ops = dev->iommu_fwspec->ops;
+ if (ops->remove_device)
+ ops->remove_device(dev);
+ }
+

Yuck. It's a little unfortunate that we have to do this at all, but I
see why. Still, it should be done in common code, not duplicated across
arch code, not least for symmetry with where the matching add_device
happened (although I think of_dma_deconfigure() would suffice, I'm not
sure we really need to add an of_iommu_deconfigure() just for this).

So one way is its already called via a the BUS_NOTIFY_REMOVED_DEVICE
notifier in iommu_bus_notifier. I put it here, one for symmetry and
another thinking that the remove_device callback should be done
before the dma_ops is set to NULL. If thats not required then the existing
iommu_bus_notifier itself can do the cleanup. The corresponding
add_device in that notifier is dummy now, i will add a patch to remove
that. If not the whole thing, we can add of_iommu_deconfigure as well.

Post by Robin Murphy
It's also broken for IOMMU drivers which rely on the
of_iommu_configure() mechanism but have not yet been converted to use
iommu_fwspec (Exynos, MSM, etc.)

I did this, because fwspec was the right way to get ops in the future,
but yes its getting broken for those which are not converted,
will have to use dev->bus->iommu_ops in those cases. Will address this
while changing the above.

Regards,
Sricharan

[Sricharan R]

With arch_setup_dma_ops now being called late during device's probe after the
device's iommu is probed, the notifier trick required to handle the early
setup of dma_ops before the iommu group gets created is not required.
So removing the notifier's here.

Signed-off-by: Sricharan R <***@codeaurora.org>
---
arch/arm64/mm/dma-mapping.c | 100 ++------------------------------------------
1 file changed, 3 insertions(+), 97 deletions(-)

diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
index faf4b92..eb593af 100644
--- a/arch/arm64/mm/dma-mapping.c
+++ b/arch/arm64/mm/dma-mapping.c
@@ -799,24 +799,6 @@ static struct dma_map_ops iommu_dma_ops = {
.mapping_error = iommu_dma_mapping_error,
};

-/*
- * TODO: Right now __iommu_setup_dma_ops() gets called too early to do
- * everything it needs to - the device is only partially created and the
- * IOMMU driver hasn't seen it yet, so it can't have a group. Thus we
- * need this delayed attachment dance. Once IOMMU probe ordering is sorted
- * to move the arch_setup_dma_ops() call later, all the notifier bits below
- * become unnecessary, and will go away.
- */
-struct iommu_dma_notifier_data {
- struct list_head list;
- struct device *dev;
- const struct iommu_ops *ops;
- u64 dma_base;
- u64 size;
-};
-static LIST_HEAD(iommu_dma_masters);
-static DEFINE_MUTEX(iommu_dma_notifier_lock);
-
static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
u64 dma_base, u64 size)
{
@@ -837,79 +819,9 @@ static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
return true;
}

-static void queue_iommu_attach(struct device *dev, const struct iommu_ops *ops,
- u64 dma_base, u64 size)
-{
- struct iommu_dma_notifier_data *iommudata;
-
- iommudata = kzalloc(sizeof(*iommudata), GFP_KERNEL);
- if (!iommudata)
- return;
-
- iommudata->dev = dev;
- iommudata->ops = ops;
- iommudata->dma_base = dma_base;
- iommudata->size = size;
-
- mutex_lock(&iommu_dma_notifier_lock);
- list_add(&iommudata->list, &iommu_dma_masters);
- mutex_unlock(&iommu_dma_notifier_lock);
-}
-
-static int __iommu_attach_notifier(struct notifier_block *nb,
- unsigned long action, void *data)
-{
- struct iommu_dma_notifier_data *master, *tmp;
-
- if (action != BUS_NOTIFY_BIND_DRIVER)
- return 0;
-
- mutex_lock(&iommu_dma_notifier_lock);
- list_for_each_entry_safe(master, tmp, &iommu_dma_masters, list) {
- if (data == master->dev && do_iommu_attach(master->dev,
- master->ops, master->dma_base, master->size)) {
- list_del(&master->list);
- kfree(master);
- break;
- }
- }
- mutex_unlock(&iommu_dma_notifier_lock);
- return 0;
-}
-
-static int __init register_iommu_dma_ops_notifier(struct bus_type *bus)
-{
- struct notifier_block *nb = kzalloc(sizeof(*nb), GFP_KERNEL);
- int ret;
-
- if (!nb)
- return -ENOMEM;
-
- nb->notifier_call = __iommu_attach_notifier;
-
- ret = bus_register_notifier(bus, nb);
- if (ret) {
- pr_warn("Failed to register DMA domain notifier; IOMMU DMA ops unavailable on bus '%s'\n",
- bus->name);
- kfree(nb);
- }
- return ret;
-}
-
static int __init __iommu_dma_init(void)
{
- int ret;
-
- ret = iommu_dma_init();
- if (!ret)
- ret = register_iommu_dma_ops_notifier(&platform_bus_type);
- if (!ret)
- ret = register_iommu_dma_ops_notifier(&amba_bustype);
-#ifdef CONFIG_PCI
- if (!ret)
- ret = register_iommu_dma_ops_notifier(&pci_bus_type);
-#endif
- return ret;
+ return iommu_dma_init();
}
arch_initcall(__iommu_dma_init);

@@ -920,18 +832,12 @@ static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,

if (!ops)
return;
- /*
- * TODO: As a concession to the future, we're ready to handle being
- * called both early and late (i.e. after bus_add_device). Once all
- * the platform bus code is reworked to call us late and the notifier
- * junk above goes away, move the body of do_iommu_attach here.
- */
+
group = iommu_group_get(dev);
+
if (group) {
do_iommu_attach(dev, ops, dma_base, size);
iommu_group_put(group);
- } else {
- queue_iommu_attach(dev, ops, dma_base, size);
}
}

[Sricharan]

Hi,

-----Original Message-----
Sent: Tuesday, October 04, 2016 10:34 PM
Subject: [PATCH V3 8/8] arm64: dma-mapping: Remove the notifier trick to handle early setting of dma_ops
With arch_setup_dma_ops now being called late during device's probe after the
device's iommu is probed, the notifier trick required to handle the early
setup of dma_ops before the iommu group gets created is not required.
So removing the notifier's here.
---
arch/arm64/mm/dma-mapping.c | 100 ++------------------------------------------
1 file changed, 3 insertions(+), 97 deletions(-)
diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
index faf4b92..eb593af 100644
--- a/arch/arm64/mm/dma-mapping.c
+++ b/arch/arm64/mm/dma-mapping.c
@@ -799,24 +799,6 @@ static struct dma_map_ops iommu_dma_ops = {
.mapping_error = iommu_dma_mapping_error,
};
-/*
- * TODO: Right now __iommu_setup_dma_ops() gets called too early to do
- * everything it needs to - the device is only partially created and the
- * IOMMU driver hasn't seen it yet, so it can't have a group. Thus we
- * need this delayed attachment dance. Once IOMMU probe ordering is sorted
- * to move the arch_setup_dma_ops() call later, all the notifier bits below
- * become unnecessary, and will go away.
- */
-struct iommu_dma_notifier_data {
- struct list_head list;
- struct device *dev;
- const struct iommu_ops *ops;
- u64 dma_base;
- u64 size;
-};
-static LIST_HEAD(iommu_dma_masters);
-static DEFINE_MUTEX(iommu_dma_notifier_lock);
-
static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
u64 dma_base, u64 size)
{
@@ -837,79 +819,9 @@ static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
return true;
}
-static void queue_iommu_attach(struct device *dev, const struct iommu_ops *ops,
- u64 dma_base, u64 size)
-{
- struct iommu_dma_notifier_data *iommudata;
-
- iommudata = kzalloc(sizeof(*iommudata), GFP_KERNEL);
- if (!iommudata)
- return;
-
- iommudata->dev = dev;
- iommudata->ops = ops;
- iommudata->dma_base = dma_base;
- iommudata->size = size;
-
- mutex_lock(&iommu_dma_notifier_lock);
- list_add(&iommudata->list, &iommu_dma_masters);
- mutex_unlock(&iommu_dma_notifier_lock);
-}
-
-static int __iommu_attach_notifier(struct notifier_block *nb,
- unsigned long action, void *data)
-{
- struct iommu_dma_notifier_data *master, *tmp;
-
- if (action != BUS_NOTIFY_BIND_DRIVER)
- return 0;
-
- mutex_lock(&iommu_dma_notifier_lock);
- list_for_each_entry_safe(master, tmp, &iommu_dma_masters, list) {
- if (data == master->dev && do_iommu_attach(master->dev,
- master->ops, master->dma_base, master->size)) {
- list_del(&master->list);
- kfree(master);
- break;
- }
- }
- mutex_unlock(&iommu_dma_notifier_lock);
- return 0;
-}
-
-static int __init register_iommu_dma_ops_notifier(struct bus_type *bus)
-{
- struct notifier_block *nb = kzalloc(sizeof(*nb), GFP_KERNEL);
- int ret;
-
- if (!nb)
- return -ENOMEM;
-
- nb->notifier_call = __iommu_attach_notifier;
-
- ret = bus_register_notifier(bus, nb);
- if (ret) {
- pr_warn("Failed to register DMA domain notifier; IOMMU DMA ops unavailable on bus '%s'\n",
- bus->name);
- kfree(nb);
- }
- return ret;
-}
-
static int __init __iommu_dma_init(void)
{
- int ret;
-
- ret = iommu_dma_init();
- if (!ret)
- ret = register_iommu_dma_ops_notifier(&platform_bus_type);
- if (!ret)
- ret = register_iommu_dma_ops_notifier(&amba_bustype);
-#ifdef CONFIG_PCI
- if (!ret)
- ret = register_iommu_dma_ops_notifier(&pci_bus_type);
-#endif
- return ret;
+ return iommu_dma_init();
}
arch_initcall(__iommu_dma_init);
@@ -920,18 +832,12 @@ static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
if (!ops)
return;
- /*
- * TODO: As a concession to the future, we're ready to handle being
- * called both early and late (i.e. after bus_add_device). Once all
- * the platform bus code is reworked to call us late and the notifier
- * junk above goes away, move the body of do_iommu_attach here.
- */
+
group = iommu_group_get(dev);
+
if (group) {
do_iommu_attach(dev, ops, dma_base, size);
iommu_group_put(group);
- } else {
- queue_iommu_attach(dev, ops, dma_base, size);
}
}

I should have has this as well for being removed,

From: Sricharan R <***@codeaurora.org>
Date: Fri, 7 Oct 2016 19:20:21 +0530
Subject: [PATCH] iommu:/arm-smmu: Avoid early iommu device registration

of_platform_device_create was called early in the init
to have the smmu probed before the master. But now with
the probe deferral, this is not needed.

Signed-off-by: Sricharan R <***@codeaurora.org>
---
drivers/iommu/arm-smmu.c | 3 ---
1 file changed, 3 deletions(-)

diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index c841eb7..083489e4 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -2066,9 +2066,6 @@ static int __init arm_smmu_of_init(struct device_node *np)
if (ret)
return ret;

- if (!of_platform_device_create(np, NULL, platform_bus_type.dev_root))
- return -ENODEV;
-
return 0;
}
IOMMU_OF_DECLARE(arm_smmuv1, "arm,smmu-v1", arm_smmu_of_init);

[Robin Murphy]

Post by Sricharan R
With arch_setup_dma_ops now being called late during device's probe after the
device's iommu is probed, the notifier trick required to handle the early
setup of dma_ops before the iommu group gets created is not required.
So removing the notifier's here.

Hooray!

Post by Sricharan R
---
arch/arm64/mm/dma-mapping.c | 100 ++------------------------------------------
1 file changed, 3 insertions(+), 97 deletions(-)
diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
index faf4b92..eb593af 100644
--- a/arch/arm64/mm/dma-mapping.c
+++ b/arch/arm64/mm/dma-mapping.c
@@ -799,24 +799,6 @@ static struct dma_map_ops iommu_dma_ops = {
.mapping_error = iommu_dma_mapping_error,
};
-/*
- * TODO: Right now __iommu_setup_dma_ops() gets called too early to do
- * everything it needs to - the device is only partially created and the
- * IOMMU driver hasn't seen it yet, so it can't have a group. Thus we
- * need this delayed attachment dance. Once IOMMU probe ordering is sorted
- * to move the arch_setup_dma_ops() call later, all the notifier bits below
- * become unnecessary, and will go away.
- */
-struct iommu_dma_notifier_data {
- struct list_head list;
- struct device *dev;
- const struct iommu_ops *ops;
- u64 dma_base;
- u64 size;
-};
-static LIST_HEAD(iommu_dma_masters);
-static DEFINE_MUTEX(iommu_dma_notifier_lock);
-
static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
u64 dma_base, u64 size)

This should go as well...

Post by Sricharan R
{
@@ -837,79 +819,9 @@ static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
return true;
}
-static void queue_iommu_attach(struct device *dev, const struct iommu_ops *ops,
- u64 dma_base, u64 size)
-{
- struct iommu_dma_notifier_data *iommudata;
-
- iommudata = kzalloc(sizeof(*iommudata), GFP_KERNEL);
- if (!iommudata)
- return;
-
- iommudata->dev = dev;
- iommudata->ops = ops;
- iommudata->dma_base = dma_base;
- iommudata->size = size;
-
- mutex_lock(&iommu_dma_notifier_lock);
- list_add(&iommudata->list, &iommu_dma_masters);
- mutex_unlock(&iommu_dma_notifier_lock);
-}
-
-static int __iommu_attach_notifier(struct notifier_block *nb,
- unsigned long action, void *data)
-{
- struct iommu_dma_notifier_data *master, *tmp;
-
- if (action != BUS_NOTIFY_BIND_DRIVER)
- return 0;
-
- mutex_lock(&iommu_dma_notifier_lock);
- list_for_each_entry_safe(master, tmp, &iommu_dma_masters, list) {
- if (data == master->dev && do_iommu_attach(master->dev,
- master->ops, master->dma_base, master->size)) {
- list_del(&master->list);
- kfree(master);
- break;
- }
- }
- mutex_unlock(&iommu_dma_notifier_lock);
- return 0;
-}
-
-static int __init register_iommu_dma_ops_notifier(struct bus_type *bus)
-{
- struct notifier_block *nb = kzalloc(sizeof(*nb), GFP_KERNEL);
- int ret;
-
- if (!nb)
- return -ENOMEM;
-
- nb->notifier_call = __iommu_attach_notifier;
-
- ret = bus_register_notifier(bus, nb);
- if (ret) {
- pr_warn("Failed to register DMA domain notifier; IOMMU DMA ops unavailable on bus '%s'\n",
- bus->name);
- kfree(nb);
- }
- return ret;
-}
-
static int __init __iommu_dma_init(void)
{
- int ret;
-
- ret = iommu_dma_init();
- if (!ret)
- ret = register_iommu_dma_ops_notifier(&platform_bus_type);
- if (!ret)
- ret = register_iommu_dma_ops_notifier(&amba_bustype);
-#ifdef CONFIG_PCI
- if (!ret)
- ret = register_iommu_dma_ops_notifier(&pci_bus_type);
-#endif
- return ret;
+ return iommu_dma_init();
}
arch_initcall(__iommu_dma_init);
@@ -920,18 +832,12 @@ static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
if (!ops)
return;
- /*
- * TODO: As a concession to the future, we're ready to handle being
- * called both early and late (i.e. after bus_add_device). Once all
- * the platform bus code is reworked to call us late and the notifier
- * junk above goes away, move the body of do_iommu_attach here.

...per this commment. It has no need to be a separate function once this
is the only call site (plus it has a misleadingly inaccurate name anyway).

Post by Sricharan R
- */
+
group = iommu_group_get(dev);
+
if (group) {
do_iommu_attach(dev, ops, dma_base, size);
iommu_group_put(group);
- } else {
- queue_iommu_attach(dev, ops, dma_base, size);
}
}

I overlooked it in the last patch as the relevant context was in the
cover letter, so I'll just add the comment here; the domain check and
WARN_ON() in arch_teardown_dma_ops() is mostly a left-over from the
'fake default domain' bodge that was already gone from the final version
of the arm64 dma_ops series. It's based on the assumption that
arch_teardown_dma_ops() is called after the device is removed from the
bus - I'm not convinced that assumption was ever actually true, it now
doesn't necessarily do anything anyway (since detach_device is
deprecated). I'll spin a cleanup patch for that myself...

Robin.

[Sricharan]

Hi Robin,

Post by Robin Murphy

Post by Sricharan R
With arch_setup_dma_ops now being called late during device's probe after the
device's iommu is probed, the notifier trick required to handle the early
setup of dma_ops before the iommu group gets created is not required.
So removing the notifier's here.

Hooray!

Post by Sricharan R
---
arch/arm64/mm/dma-mapping.c | 100 ++------------------------------------------
1 file changed, 3 insertions(+), 97 deletions(-)
diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
index faf4b92..eb593af 100644
--- a/arch/arm64/mm/dma-mapping.c
+++ b/arch/arm64/mm/dma-mapping.c
@@ -799,24 +799,6 @@ static struct dma_map_ops iommu_dma_ops = {
.mapping_error = iommu_dma_mapping_error,
};
-/*
- * TODO: Right now __iommu_setup_dma_ops() gets called too early to do
- * everything it needs to - the device is only partially created and the
- * IOMMU driver hasn't seen it yet, so it can't have a group. Thus we
- * need this delayed attachment dance. Once IOMMU probe ordering is sorted
- * to move the arch_setup_dma_ops() call later, all the notifier bits below
- * become unnecessary, and will go away.
- */
-struct iommu_dma_notifier_data {
- struct list_head list;
- struct device *dev;
- const struct iommu_ops *ops;
- u64 dma_base;
- u64 size;
-};
-static LIST_HEAD(iommu_dma_masters);
-static DEFINE_MUTEX(iommu_dma_notifier_lock);
-
static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
u64 dma_base, u64 size)

This should go as well...

ok.

Post by Robin Murphy

Post by Sricharan R
{
@@ -837,79 +819,9 @@ static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
return true;
}
-static void queue_iommu_attach(struct device *dev, const struct iommu_ops *ops,
- u64 dma_base, u64 size)
-{
- struct iommu_dma_notifier_data *iommudata;
-
- iommudata = kzalloc(sizeof(*iommudata), GFP_KERNEL);
- if (!iommudata)
- return;
-
- iommudata->dev = dev;
- iommudata->ops = ops;
- iommudata->dma_base = dma_base;
- iommudata->size = size;
-
- mutex_lock(&iommu_dma_notifier_lock);
- list_add(&iommudata->list, &iommu_dma_masters);
- mutex_unlock(&iommu_dma_notifier_lock);
-}
-
-static int __iommu_attach_notifier(struct notifier_block *nb,
- unsigned long action, void *data)
-{
- struct iommu_dma_notifier_data *master, *tmp;
-
- if (action != BUS_NOTIFY_BIND_DRIVER)
- return 0;
-
- mutex_lock(&iommu_dma_notifier_lock);
- list_for_each_entry_safe(master, tmp, &iommu_dma_masters, list) {
- if (data == master->dev && do_iommu_attach(master->dev,
- master->ops, master->dma_base, master->size)) {
- list_del(&master->list);
- kfree(master);
- break;
- }
- }
- mutex_unlock(&iommu_dma_notifier_lock);
- return 0;
-}
-
-static int __init register_iommu_dma_ops_notifier(struct bus_type *bus)
-{
- struct notifier_block *nb = kzalloc(sizeof(*nb), GFP_KERNEL);
- int ret;
-
- if (!nb)
- return -ENOMEM;
-
- nb->notifier_call = __iommu_attach_notifier;
-
- ret = bus_register_notifier(bus, nb);
- if (ret) {
- pr_warn("Failed to register DMA domain notifier; IOMMU DMA ops unavailable on bus '%s'\n",
- bus->name);
- kfree(nb);
- }
- return ret;
-}
-
static int __init __iommu_dma_init(void)
{
- int ret;
-
- ret = iommu_dma_init();
- if (!ret)
- ret = register_iommu_dma_ops_notifier(&platform_bus_type);
- if (!ret)
- ret = register_iommu_dma_ops_notifier(&amba_bustype);
-#ifdef CONFIG_PCI
- if (!ret)
- ret = register_iommu_dma_ops_notifier(&pci_bus_type);
-#endif
- return ret;
+ return iommu_dma_init();
}
arch_initcall(__iommu_dma_init);
@@ -920,18 +832,12 @@ static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
if (!ops)
return;
- /*
- * TODO: As a concession to the future, we're ready to handle being
- * called both early and late (i.e. after bus_add_device). Once all
- * the platform bus code is reworked to call us late and the notifier
- * junk above goes away, move the body of do_iommu_attach here.

...per this commment. It has no need to be a separate function once this
is the only call site (plus it has a misleadingly inaccurate name anyway).

ya, missed to remove that function as well. Will do it in the next.

Regards,
Sricharan

[Marek Szyprowski]

Hi Sricharan,

Post by Sricharan R
Initial post from Laurent Pinchart[1]. This is
series calls the dma ops configuration for the devices
at a generic place so that it works for all busses.
The dma_configure_ops for a device is now called during
the device_attach callback just before the probe of the
bus/driver is called. Similarly dma_deconfigure is called during
device/driver_detach path.
pci_bus_add_devices (platform/amba)(_device_create/driver_register)
| |
pci_bus_add_device (device_add/driver_register)
| |
device_attach device_initial_probe
| |
__device_attach_driver __device_attach_driver
|
driver_probe_device
|
really_probe
|
dma_configure
Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.
If the ACPI bus code follows the same, we can add acpi_dma_configure
at the same place as of_dma_configure.
This series is based on the recently merged Generic DT bindings for
This time tested this with platform and pci device for probe deferral
and reprobe on arm64 based platform. There is an issue on the cleanup
path for arm64 though, where there is WARN_ON if the dma_ops is reset while
device is attached to an domain in arch_teardown_dma_ops.
But with iommu_groups created from the iommu driver, the device is always
attached to a domain/default_domain. So so the WARN has to be removed/handled
probably.

Thanks for continuing work on this feature! Your can add my:

Tested-by: Marek Szyprowski <***@samsung.com>

It works fine with Exynos SYSMMU driver, although a patch is needed to fix
infinite loop due to list corruption (same element is added twice if master
device fails with deferred probe):

From: Marek Szyprowski <***@samsung.com>
Date: Mon, 10 Oct 2016 14:22:42 +0200
Subject: [PATCH] iommu/exynos: ensure that sysmmu is added only once to its
master

Since adding IOMMU deferred probing support, of_xlate() callback might
be called more than once for given master device (for example it happens
when masters device driver fails with EPROBE_DEFER), so ensure that
SYSMMU controller is added to its master device (owner) only once.

Signed-off-by: Marek Szyprowski <***@samsung.com>
---
drivers/iommu/exynos-iommu.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c
index 30808e91b775..1525a86eb829 100644
--- a/drivers/iommu/exynos-iommu.c
+++ b/drivers/iommu/exynos-iommu.c
@@ -1253,7 +1253,7 @@ static int exynos_iommu_of_xlate(struct device *dev,
{
struct exynos_iommu_owner *owner = dev->archdata.iommu;
struct platform_device *sysmmu = of_find_device_by_node(spec->np);
- struct sysmmu_drvdata *data;
+ struct sysmmu_drvdata *data, *entry;

if (!sysmmu)
return -ENODEV;
@@ -1271,6 +1271,10 @@ static int exynos_iommu_of_xlate(struct device *dev,
dev->archdata.iommu = owner;
}

+ list_for_each_entry(entry, &owner->controllers, owner_node)
+ if (entry == data)
+ return 0;
+
list_add_tail(&data->owner_node, &owner->controllers);
return 0;
}
--
1.9.1

Post by Sricharan R
Previous post of this series [3].
[V3]
* Removed the patch to split dma_masks/dma_ops configuration separately
based on review comments that both masks and ops are required only
during the device probe time.
* Reworked the series based on Generic DT bindings series [2].
* Added call to iommu's remove_device in the cleanup path for arm and arm64.
* Removed the notifier trick in arm64 to handle early device registration.
* Added reset of dma_ops in cleanup path for arm based on comments.
* Fixed the pci_iommu_configure path and tested with PCI device as well.
* Fixed a bug to return the correct iommu_ops from patch 7 [4] in last post.
* Fixed few other cosmetic comments.
[V2]
* Updated the Initial post to call dma_configure/deconfigure from generic code
* Added iommu add_device callback from of_iommu_configure path
[V1]
* Initial post
[1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
[2] http://www.spinics.net/lists/devicetree/msg142943.html
arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()
of: dma: Move range size workaround to of_dma_get_range()
of: dma: Make of_dma_deconfigure() public
iommu: of: Handle IOMMU lookup failure with deferred probing or error
drivers: platform: Configure dma operations at probe time
arm: dma-mapping: Reset the device's dma_ops
arm/arm64: dma-mapping: Call iommu's remove_device callback during
device detach
arm64: dma-mapping: Remove the notifier trick to handle early setting
of dma_ops
arch/arm/mm/dma-mapping.c | 18 ++++++++
arch/arm64/mm/dma-mapping.c | 107 +++++---------------------------------------
drivers/base/dd.c | 10 +++++
drivers/base/dma-mapping.c | 11 +++++
drivers/iommu/of_iommu.c | 47 +++++++++++++++++--
drivers/of/address.c | 20 ++++++++-
drivers/of/device.c | 34 +++++++-------
drivers/of/platform.c | 9 ----
drivers/pci/probe.c | 5 +--
include/linux/dma-mapping.h | 3 ++
include/linux/of_device.h | 7 ++-
11 files changed, 138 insertions(+), 133 deletions(-)

Best regards

[Sricharan]

Hi Marek,

Post by Marek Szyprowski
Hi Sricharan,

Post by Sricharan R
Initial post from Laurent Pinchart[1]. This is
series calls the dma ops configuration for the devices
at a generic place so that it works for all busses.
The dma_configure_ops for a device is now called during
the device_attach callback just before the probe of the
bus/driver is called. Similarly dma_deconfigure is called during
device/driver_detach path.
pci_bus_add_devices (platform/amba)(_device_create/driver_register)
| |
pci_bus_add_device (device_add/driver_register)
| |
device_attach device_initial_probe
| |
__device_attach_driver __device_attach_driver
|
driver_probe_device
|
really_probe
|
dma_configure
Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.
If the ACPI bus code follows the same, we can add acpi_dma_configure
at the same place as of_dma_configure.
This series is based on the recently merged Generic DT bindings for
This time tested this with platform and pci device for probe deferral
and reprobe on arm64 based platform. There is an issue on the cleanup
path for arm64 though, where there is WARN_ON if the dma_ops is reset while
device is attached to an domain in arch_teardown_dma_ops.
But with iommu_groups created from the iommu driver, the device is always
attached to a domain/default_domain. So so the WARN has to be removed/handled
probably.

Thanks for testing this. So the for the below fix, the remove_device callback
gets called on the dma_ops cleanup path, so would it be easy to remove the
data for the device there ?

Regards,
Sricharan

Post by Marek Szyprowski
It works fine with Exynos SYSMMU driver, although a patch is needed to fix
infinite loop due to list corruption (same element is added twice if master
Date: Mon, 10 Oct 2016 14:22:42 +0200
Subject: [PATCH] iommu/exynos: ensure that sysmmu is added only once to its
master
Since adding IOMMU deferred probing support, of_xlate() callback might
be called more than once for given master device (for example it happens
when masters device driver fails with EPROBE_DEFER), so ensure that
SYSMMU controller is added to its master device (owner) only once.
---
drivers/iommu/exynos-iommu.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c
index 30808e91b775..1525a86eb829 100644
--- a/drivers/iommu/exynos-iommu.c
+++ b/drivers/iommu/exynos-iommu.c
@@ -1253,7 +1253,7 @@ static int exynos_iommu_of_xlate(struct device *dev,
{
struct exynos_iommu_owner *owner = dev->archdata.iommu;
struct platform_device *sysmmu = of_find_device_by_node(spec->np);
- struct sysmmu_drvdata *data;
+ struct sysmmu_drvdata *data, *entry;
if (!sysmmu)
return -ENODEV;
@@ -1271,6 +1271,10 @@ static int exynos_iommu_of_xlate(struct device *dev,
dev->archdata.iommu = owner;
}
+ list_for_each_entry(entry, &owner->controllers, owner_node)
+ if (entry == data)
+ return 0;
+
list_add_tail(&data->owner_node, &owner->controllers);
return 0;
}
--
1.9.1

[Marek Szyprowski]

Hi Sricharan,

Post by Sricharan

Post by Sricharan R
Initial post from Laurent Pinchart[1]. This is
series calls the dma ops configuration for the devices
at a generic place so that it works for all busses.
The dma_configure_ops for a device is now called during
the device_attach callback just before the probe of the
bus/driver is called. Similarly dma_deconfigure is called during
device/driver_detach path.
pci_bus_add_devices (platform/amba)(_device_create/driver_register)
| |
pci_bus_add_device (device_add/driver_register)
| |
device_attach device_initial_probe
| |
__device_attach_driver __device_attach_driver
|
driver_probe_device
|
really_probe
|
dma_configure
Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.
If the ACPI bus code follows the same, we can add acpi_dma_configure
at the same place as of_dma_configure.
This series is based on the recently merged Generic DT bindings for
This time tested this with platform and pci device for probe deferral
and reprobe on arm64 based platform. There is an issue on the cleanup
path for arm64 though, where there is WARN_ON if the dma_ops is reset while
device is attached to an domain in arch_teardown_dma_ops.
But with iommu_groups created from the iommu driver, the device is always
attached to a domain/default_domain. So so the WARN has to be removed/handled
probably.

Thanks for testing this. So the for the below fix, the remove_device callback
gets called on the dma_ops cleanup path, so would it be easy to remove the
data for the device there ?

I assumed that IOMMU driver cannot be removed reliably, so all
structures that it
creates are permanent. I didn't use device_add()/device_remove()
callbacks, because
in current implementation device_add() is called too late (after
dma-mapping glue
triggers device_attach_iommu()).

Maybe once your patchset is merged, I will move creation and management
of the all
IOMMU related structures to device_add/remove callbacks.

Best regards

[Sricharan]

Hi Marek,

Post by Marek Szyprowski

Post by Sricharan

Post by Sricharan R
Initial post from Laurent Pinchart[1]. This is
series calls the dma ops configuration for the devices
at a generic place so that it works for all busses.
The dma_configure_ops for a device is now called during
the device_attach callback just before the probe of the
bus/driver is called. Similarly dma_deconfigure is called during
device/driver_detach path.
pci_bus_add_devices (platform/amba)(_device_create/driver_register)
| |
pci_bus_add_device (device_add/driver_register)
| |
device_attach device_initial_probe
| |
__device_attach_driver __device_attach_driver
|
driver_probe_device
|
really_probe
|
dma_configure
Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.
If the ACPI bus code follows the same, we can add acpi_dma_configure
at the same place as of_dma_configure.
This series is based on the recently merged Generic DT bindings for
This time tested this with platform and pci device for probe deferral
and reprobe on arm64 based platform. There is an issue on the cleanup
path for arm64 though, where there is WARN_ON if the dma_ops is reset while
device is attached to an domain in arch_teardown_dma_ops.
But with iommu_groups created from the iommu driver, the device is always
attached to a domain/default_domain. So so the WARN has to be removed/handled
probably.

Thanks for testing this. So the for the below fix, the remove_device callback
gets called on the dma_ops cleanup path, so would it be easy to remove the
data for the device there ?

I assumed that IOMMU driver cannot be removed reliably, so all
structures that it
creates are permanent. I didn't use device_add()/device_remove()
callbacks, because
in current implementation device_add() is called too late (after
dma-mapping glue
triggers device_attach_iommu()).
Maybe once your patchset is merged, I will move creation and management
of the all
IOMMU related structures to device_add/remove callbacks.

ok understand it.

Regards,
Sricharan

[Sricharan]

-----Original Message-----
Sent: Wednesday, October 12, 2016 11:55 AM
Subject: RE: [PATCH V3 0/8] IOMMU probe deferral support
Hi Marek,

Post by Marek Szyprowski
Hi Sricharan,

Post by Sricharan R
Initial post from Laurent Pinchart[1]. This is
series calls the dma ops configuration for the devices
at a generic place so that it works for all busses.
The dma_configure_ops for a device is now called during
the device_attach callback just before the probe of the
bus/driver is called. Similarly dma_deconfigure is called during
device/driver_detach path.
pci_bus_add_devices (platform/amba)(_device_create/driver_register)
| |
pci_bus_add_device (device_add/driver_register)
| |
device_attach device_initial_probe
| |
__device_attach_driver __device_attach_driver
|
driver_probe_device
|
really_probe
|
dma_configure
Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.
If the ACPI bus code follows the same, we can add acpi_dma_configure
at the same place as of_dma_configure.
This series is based on the recently merged Generic DT bindings for
This time tested this with platform and pci device for probe deferral
and reprobe on arm64 based platform. There is an issue on the cleanup
path for arm64 though, where there is WARN_ON if the dma_ops is reset while
device is attached to an domain in arch_teardown_dma_ops.
But with iommu_groups created from the iommu driver, the device is always
attached to a domain/default_domain. So so the WARN has to be removed/handled
probably.

Hi Will,Robin,Joerg,

I have tested the probe deferral for platform/pcie bus devices based on latest Generic bindings
series merged [1], for pci/arm-smmu.
It will good to know from you on whats the right way to take this forward ?

Regards,
Sricharan

[Sricharan]

Resending, missed out on the link last time.

-----Original Message-----
Sent: Monday, October 10, 2016 6:07 PM
Subject: Re: [PATCH V3 0/8] IOMMU probe deferral support
Hi Sricharan,

Post by Sricharan R
Initial post from Laurent Pinchart[1]. This is
series calls the dma ops configuration for the devices
at a generic place so that it works for all busses.
The dma_configure_ops for a device is now called during
the device_attach callback just before the probe of the
bus/driver is called. Similarly dma_deconfigure is called during
device/driver_detach path.
pci_bus_add_devices (platform/amba)(_device_create/driver_register)
| |
pci_bus_add_device (device_add/driver_register)
| |
device_attach device_initial_probe
| |
__device_attach_driver __device_attach_driver
|
driver_probe_device
|
really_probe
|
dma_configure
Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.
If the ACPI bus code follows the same, we can add acpi_dma_configure
at the same place as of_dma_configure.
This series is based on the recently merged Generic DT bindings for
This time tested this with platform and pci device for probe deferral
and reprobe on arm64 based platform. There is an issue on the cleanup
path for arm64 though, where there is WARN_ON if the dma_ops is reset while
device is attached to an domain in arch_teardown_dma_ops.
But with iommu_groups created from the iommu driver, the device is always
attached to a domain/default_domain. So so the WARN has to be removed/handled
probably.

Hi Will,Robin,Joerg,

I have tested the probe deferral for platform/pcie bus devices based on latest Generic DT bindings
series merged [1], for pci/arm-smmu.
It will be good to know from you on whats the right way to take this forward ?

[1] http://www.spinics.net/lists/devicetree/msg142943.html

Regards,
Sricharan

[Archit Taneja]

Post by Sricharan R
Initial post from Laurent Pinchart[1]. This is
series calls the dma ops configuration for the devices
at a generic place so that it works for all busses.
The dma_configure_ops for a device is now called during
the device_attach callback just before the probe of the
bus/driver is called. Similarly dma_deconfigure is called during
device/driver_detach path.
pci_bus_add_devices (platform/amba)(_device_create/driver_register)
| |
pci_bus_add_device (device_add/driver_register)
| |
device_attach device_initial_probe
| |
__device_attach_driver __device_attach_driver
|
driver_probe_device
|
really_probe
|
dma_configure
Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.
If the ACPI bus code follows the same, we can add acpi_dma_configure
at the same place as of_dma_configure.
This series is based on the recently merged Generic DT bindings for
This time tested this with platform and pci device for probe deferral
and reprobe on arm64 based platform. There is an issue on the cleanup
path for arm64 though, where there is WARN_ON if the dma_ops is reset while
device is attached to an domain in arch_teardown_dma_ops.
But with iommu_groups created from the iommu driver, the device is always
attached to a domain/default_domain. So so the WARN has to be removed/handled
probably.
Previous post of this series [3].
[V3]
* Removed the patch to split dma_masks/dma_ops configuration separately
based on review comments that both masks and ops are required only
during the device probe time.
* Reworked the series based on Generic DT bindings series [2].
* Added call to iommu's remove_device in the cleanup path for arm and arm64.
* Removed the notifier trick in arm64 to handle early device registration.
* Added reset of dma_ops in cleanup path for arm based on comments.
* Fixed the pci_iommu_configure path and tested with PCI device as well.
* Fixed a bug to return the correct iommu_ops from patch 7 [4] in last post.
* Fixed few other cosmetic comments.
[V2]
* Updated the Initial post to call dma_configure/deconfigure from generic code
* Added iommu add_device callback from of_iommu_configure path
[V1]
* Initial post
[1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
[2] http://www.spinics.net/lists/devicetree/msg142943.html
arm: dma-mapping: Don't override dma_ops in arch_setup_dma_ops()
of: dma: Move range size workaround to of_dma_get_range()
of: dma: Make of_dma_deconfigure() public
iommu: of: Handle IOMMU lookup failure with deferred probing or error
drivers: platform: Configure dma operations at probe time
arm: dma-mapping: Reset the device's dma_ops
arm/arm64: dma-mapping: Call iommu's remove_device callback during
device detach
arm64: dma-mapping: Remove the notifier trick to handle early setting
of dma_ops
arch/arm/mm/dma-mapping.c | 18 ++++++++
arch/arm64/mm/dma-mapping.c | 107 +++++---------------------------------------
drivers/base/dd.c | 10 +++++
drivers/base/dma-mapping.c | 11 +++++
drivers/iommu/of_iommu.c | 47 +++++++++++++++++--
drivers/of/address.c | 20 ++++++++-
drivers/of/device.c | 34 +++++++-------
drivers/of/platform.c | 9 ----
drivers/pci/probe.c | 5 +--
include/linux/dma-mapping.h | 3 ++
include/linux/of_device.h | 7 ++-
11 files changed, 138 insertions(+), 133 deletions(-)

[Robin Murphy]

Hi Sricharan,

Post by Sricharan R
Initial post from Laurent Pinchart[1]. This is
series calls the dma ops configuration for the devices
at a generic place so that it works for all busses.
The dma_configure_ops for a device is now called during
the device_attach callback just before the probe of the
bus/driver is called. Similarly dma_deconfigure is called during
device/driver_detach path.
pci_bus_add_devices (platform/amba)(_device_create/driver_register)
| |
pci_bus_add_device (device_add/driver_register)
| |
device_attach device_initial_probe
| |
__device_attach_driver __device_attach_driver
|
driver_probe_device
|
really_probe
|
dma_configure
Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.
If the ACPI bus code follows the same, we can add acpi_dma_configure
at the same place as of_dma_configure.
This series is based on the recently merged Generic DT bindings for
This time tested this with platform and pci device for probe deferral
and reprobe on arm64 based platform. There is an issue on the cleanup
path for arm64 though, where there is WARN_ON if the dma_ops is reset while
device is attached to an domain in arch_teardown_dma_ops.
But with iommu_groups created from the iommu driver, the device is always
attached to a domain/default_domain. So so the WARN has to be removed/handled
probably.

I've finally got the chance to take a proper look at this series (sorry
for the delay). First up, with these patches on top of 4.9-rc2, my
little script for unbinding some PCI devices and rebinding them to VFIO
now goes horribly, horribly wrong.

Robin.

[ 39.901592] iommu: Removing device 0000:08:00.0 from group 0
[ 39.907383] ------------[ cut here ]------------
[ 39.911969] WARNING: CPU: 0 PID: 174 at
arch/arm64/mm/dma-mapping.c:856 arch_teardown_dma_ops+0x48/0x68
[ 39.921266] Modules linked in:
[ 39.924290]
[ 39.925766] CPU: 0 PID: 174 Comm: vfio Not tainted 4.9.0-rc2+ #1249
[ 39.931967] Hardware name: ARM Juno development board (r1) (DT)
[ 39.937826] task: ffffffc975ee9900 task.stack: ffffffc974d60000
[ 39.943687] PC is at arch_teardown_dma_ops+0x48/0x68
[ 39.948603] LR is at arch_teardown_dma_ops+0x34/0x68
[ 39.953516] pc : [<ffffff80080948f8>] lr : [<ffffff80080948e4>]
pstate: 60000145
[ 39.960834] sp : ffffffc974d63ca0
[ 39.964112] x29: ffffffc974d63ca0 x28: ffffffc974d60000
[ 39.969377] x27: ffffff80088a2000 x26: 0000000000000040
[ 39.974642] x25: 0000000000000123 x24: ffffffc976a48918
[ 39.979907] x23: ffffffc974d63eb8 x22: ffffff8008db7550
[ 39.985171] x21: 000000000000000d x20: ffffffc9763e9b50
[ 39.990435] x19: ffffffc9763f20a0 x18: 0000000000000010
[ 39.995699] x17: 0000007f99c18018 x16: ffffff80080c0580
[ 40.000964] x15: ffffff8008bb7000 x14: 000137c100013798
[ 40.006228] x13: ffffffffff000000 x12: ffffffffffffffff
[ 40.011492] x11: 0000000000000018 x10: 000000000000000d
[ 40.016757] x9 : 0000000040000000 x8 : 0000000000210d00
[ 40.022021] x7 : 00000049771bc000 x6 : 00000000001f17ed
[ 40.027286] x5 : ffffff80084c4208 x4 : 0000000000000080
[ 40.032551] x3 : ffffffc975ea9800 x2 : ffffffbf25d7aa50
[ 40.037815] x1 : 0000000000000000 x0 : 0000000000000080
[ 40.043078]
[ 40.044549] ---[ end trace 35c1e743d6e6c035 ]---
[ 40.049117] Call trace:
[ 40.051537] Exception stack(0xffffffc974d63ad0 to 0xffffffc974d63c00)
[ 40.057914] 3ac0: ffffffc9763f20a0
0000008000000000
[ 40.065668] 3ae0: ffffffc974d63ca0 ffffff80080948f8 ffffffbf25d7aa40
ffffffc975ea9800
[ 40.073421] 3b00: ffffffc974d60000 000000000002fc80 ffffffc976801e00
ffffffc974d60000
[ 40.081175] 3b20: ffffff80084c4208 0000000000000040 ffffff80088a2000
ffffffc974d60000
[ 40.088928] 3b40: ffffff80084caf78 ffffffc974d60000 ffffffc974d60000
ffffffc974d60000
[ 40.096682] 3b60: ffffffc975ea9800 ffffffc974d60000 0000000000000080
0000000000000000
[ 40.104435] 3b80: ffffffbf25d7aa50 ffffffc975ea9800 0000000000000080
ffffff80084c4208
[ 40.112188] 3ba0: 00000000001f17ed 00000049771bc000 0000000000210d00
0000000040000000
[ 40.119941] 3bc0: 000000000000000d 0000000000000018 ffffffffffffffff
ffffffffff000000
[ 40.127695] 3be0: 000137c100013798 ffffff8008bb7000 ffffff80080c0580
0000007f99c18018
[ 40.135450] [<ffffff80080948f8>] arch_teardown_dma_ops+0x48/0x68
[ 40.141400] [<ffffff8008764a14>] of_dma_deconfigure+0xc/0x18
[ 40.147005] [<ffffff8008552804>] dma_deconfigure+0xc/0x18
[ 40.152353] [<ffffff800853ba10>] __device_release_driver+0x88/0x120
[ 40.158560] [<ffffff800853bacc>] device_release_driver+0x24/0x38
[ 40.164507] [<ffffff800853a868>] unbind_store+0xe8/0x110
[ 40.169767] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
[ 40.175113] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
[ 40.180458] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
[ 40.186063] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
[ 40.191237] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
[ 40.196239] [<ffffff80081c7274>] SyS_write+0x44/0xa0
[ 40.201155] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
[ 40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
[ 40.212382] vfio-pci: probe of 0000:08:00.0 failed with error -22
[ 40.228075] ------------[ cut here ]------------
[ 40.235263] WARNING: CPU: 1 PID: 174 at ./include/linux/kref.h:46
kobject_get+0x64/0x88
[ 40.243181] Modules linked in:
[ 40.246201]
[ 40.247673] CPU: 1 PID: 174 Comm: vfio Tainted: G W
4.9.0-rc2+ #1249
[ 40.255076] Hardware name: ARM Juno development board (r1) (DT)
[ 40.260932] task: ffffffc975ee9900 task.stack: ffffffc974d60000
[ 40.266787] PC is at kobject_get+0x64/0x88
[ 40.270840] LR is at iommu_bus_notifier+0x40/0x110
[ 40.275577] pc : [<ffffff800834d20c>] lr : [<ffffff80084c3fd0>]
pstate: 80000145
[ 40.282894] sp : ffffffc974d63c00
[ 40.286169] x29: ffffffc974d63c00 x28: ffffffc974d60000
[ 40.291431] x27: ffffff80088a2000 x26: 0000000000000040
[ 40.296692] x25: 0000000000000123 x24: ffffffc974c8f418
[ 40.301953] x23: 0000000000000006 x22: ffffffc9763f10a0
[ 40.307214] x21: ffffffc9763e9a00 x20: ffffffc9763f10a0
[ 40.312474] x19: ffffffc9763ebc80 x18: 0000007fd65069e0
[ 40.317734] x17: 0000007f8d0ae3c0 x16: ffffff80081c7230
[ 40.322995] x15: 0000007f8d136588 x14: ffffffffffffffff
[ 40.328255] x13: 0000000000000004 x12: 0000000000000030
[ 40.333515] x11: 0000000000000030 x10: 0101010101010101
[ 40.338775] x9 : feff716475687163 x8 : 7f7f7f7f7f7f7f7f
[ 40.344035] x7 : feff716475687163 x6 : ffffffc976abf400
[ 40.349295] x5 : ffffffc976abf400 x4 : 0000000000000000
[ 40.354555] x3 : ffffff80084c3f90 x2 : ffffffc9763ebcb8
[ 40.359814] x1 : 0000000000000001 x0 : ffffff8008d4f000
[ 40.365074]
[ 40.366542] ---[ end trace 35c1e743d6e6c036 ]---
[ 40.371107] Call trace:
[ 40.373523] Exception stack(0xffffffc974d63a30 to 0xffffffc974d63b60)
[ 40.379895] 3a20: ffffffc9763ebc80
0000008000000000
[ 40.387643] 3a40: ffffffc974d63c00 ffffff800834d20c ffffffc976812400
ffffff8008237d94
[ 40.395391] 3a60: ffffffbf25d78940 ffffffc974d60000 ffffffc975e259d8
0000000000005b81
[ 40.403139] 3a80: ffffffc974d60000 ffffff8008d4b31f ffffff8008b0f000
ffffffc976811c80
[ 40.410887] 3aa0: ffffffc974d60000 ffffffc974d60000 ffffffc974d60000
ffffff8008237000
[ 40.418634] 3ac0: ffffffc975e259d8 ffffff8008b1b9a8 ffffff8008d4f000
0000000000000001
[ 40.426382] 3ae0: ffffffc9763ebcb8 ffffff80084c3f90 0000000000000000
ffffffc976abf400
[ 40.434130] 3b00: ffffffc976abf400 feff716475687163 7f7f7f7f7f7f7f7f
feff716475687163
[ 40.441877] 3b20: 0101010101010101 0000000000000030 0000000000000030
0000000000000004
[ 40.449625] 3b40: ffffffffffffffff 0000007f8d136588 ffffff80081c7230
0000007f8d0ae3c0
[ 40.457372] [<ffffff800834d20c>] kobject_get+0x64/0x88
[ 40.462455] [<ffffff80084c3fd0>] iommu_bus_notifier+0x40/0x110
[ 40.468227] [<ffffff80080da288>] notifier_call_chain+0x50/0x90
[ 40.473997] [<ffffff80080da694>] __blocking_notifier_call_chain+0x4c/0x90
[ 40.480713] [<ffffff80080da6ec>] blocking_notifier_call_chain+0x14/0x20
[ 40.487259] [<ffffff800853b9e4>] __device_release_driver+0x5c/0x120
[ 40.493460] [<ffffff800853bacc>] device_release_driver+0x24/0x38
[ 40.499402] [<ffffff800853a868>] unbind_store+0xe8/0x110
[ 40.504656] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
[ 40.509997] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
[ 40.515337] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
[ 40.520936] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
[ 40.526104] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
[ 40.531100] [<ffffff80081c7274>] SyS_write+0x44/0xa0
[ 40.536011] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
[ 40.541324] ata1.00: disabled
[ 40.544878] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[ 40.550062] sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result:
hostbyte=0x04 driverbyte=0x00
[ 40.558871] sd 0:0:0:0: [sda] Stopping disk
[ 40.563037] sd 0:0:0:0: [sda] Start/Stop Unit failed: Result:
hostbyte=0x04 driverbyte=0x00
[ 40.586990] Unable to handle kernel paging request at virtual address
0002003e
[ 40.594702] pgd = ffffffc974c80000
[ 40.598165] [0002003e] *pgd=00000009f5102003[ 40.602241] ,
*pud=00000009f5102003
, *pmd=0000000000000000[ 40.607694]
[ 40.609171] Internal error: Oops: 96000006 [#1] PREEMPT SMP
[ 40.614684] Modules linked in:
[ 40.617712] CPU: 3 PID: 174 Comm: vfio Tainted: G W
4.9.0-rc2+ #1249
[ 40.625118] Hardware name: ARM Juno development board (r1) (DT)
[ 40.630977] task: ffffffc975ee9900 task.stack: ffffffc974d60000
[ 40.636841] PC is at kobject_get+0x14/0x88
[ 40.640897] LR is at iommu_get_domain_for_dev+0x1c/0x48
[ 40.646068] pc : [<ffffff800834d1bc>] lr : [<ffffff80084c3dec>]
pstate: 60000145
[ 40.653387] sp : ffffffc974d63c60
[ 40.656664] x29: ffffffc974d63c60 x28: ffffffc974d60000
[ 40.661928] x27: ffffff80088a2000 x26: 0000000000000040
[ 40.667193] x25: 0000000000000123 x24: ffffffc974c8f418
[ 40.672457] x23: ffffffc974d63eb8 x22: ffffff8008dab568
[ 40.677720] x21: 000000000000000d x20: ffffff8008dab568
[ 40.682984] x19: 0000000000020002 x18: 0000000000000000
[ 40.688246] x17: 0000000000000007 x16: 0000000000000001
[ 40.693509] x15: ffffffc974cd091c x14: ffffffffffffffff
[ 40.698773] x13: ffffffc974cd01cd x12: 0000000000000030
[ 40.704036] x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f
[ 40.709300] x9 : 0000000040000000 x8 : 0000000000210d00
[ 40.714563] x7 : ffffffc975f95018 x6 : 0000000000000000
[ 40.719826] x5 : 0000000000000000 x4 : ffffffc9763f1210
[ 40.725088] x3 : 0000000000000000 x2 : ffffffc9763f10e8
[ 40.730351] x1 : 0000000000000000 x0 : 0000000000020002
[ 40.735613]
[ 40.737085] Process vfio (pid: 174, stack limit = 0xffffffc974d60020)
[ 40.743460] Stack: (0xffffffc974d63c60 to 0xffffffc974d64000)
[ 40.749150] 3c60: ffffffc974d63c80 ffffff80084c3dec ffffffc9763e9a00
ffffff80085377a4
[ 40.756904] 3c80: ffffffc974d63ca0 ffffff80080948c4 ffffffc9763f10a0
ffffff8008dab568
[ 40.764658] 3ca0: ffffffc974d63cc0 ffffff8008764a14 ffffffc9763f10a0
ffffff8008dab568
[ 40.772411] 3cc0: ffffffc974d63cd0 ffffff8008552804 ffffffc974d63ce0
ffffff800853ba10
[ 40.780165] 3ce0: ffffffc974d63d00 ffffff800853bacc ffffffc9763f1100
ffffffc9763f10a0
[ 40.787918] 3d00: ffffffc974d63d20 ffffff800853a868 ffffff8008d68f18
ffffffc9763f10a0
[ 40.795672] 3d20: ffffffc974d63d50 ffffff8008539c70 000000000000000d
ffffffc974c8f400
[ 40.803425] 3d40: ffffffc9757d5880 0000000000000000 ffffffc974d63d60
ffffff800823ab18
[ 40.811178] 3d60: ffffffc974d63d70 ffffff8008239ea8 ffffffc974d63dc0
ffffff80081c507c
[ 40.818931] 3d80: 000000000000000d 0000000000000000 ffffffc974c8f100
ffffffc974d63eb8
[ 40.826684] 3da0: 000000001285f6a0 0000000000000015 0000000000000123
ffffff80080bf6ac
[ 40.834437] 3dc0: ffffffc974d63e40 ffffff80081c5e80 000000000000000d
0000000000000000
[ 40.842190] 3de0: ffffffc974d63e30 ffffff80080c087c ffffffc974d63e20
ffffff80081c5c0c
[ 40.849943] 3e00: ffffffc974c8f100 0000000000000001 ffffffc974c8f100
ffffffc974d63eb8
[ 40.857696] 3e20: ffffffc974d63e40 ffffff80081c5f48 000000000000000d
ffffffc974c8f100
[ 40.865450] 3e40: ffffffc974d63e80 ffffff80081c7274 ffffffc974c8f100
ffffffc974c8f100
[ 40.873203] 3e60: 000000001285f6a0 000000000000000d 0000000060000000
0000000000000000
[ 40.880956] 3e80: 0000000000000000 ffffff8008082ef0 0000000000000000
0000000000000001
[ 40.888709] 3ea0: ffffffffffffffff 0000007f8d0ae3dc 0000000000000000
0000000000000000
[ 40.896461] 3ec0: 0000000000000001 000000001285f6a0 000000000000000d
0000000000000000
[ 40.904215] 3ee0: ae2e2e2e3f464b49 0000000000000000 000000001285f6b0
39322f392f2f2f2f
[ 40.911968] 3f00: 0000000000000040 fefefeff2f2d2f2f 7f7f7f7f7f7f7f7f
0101010101010101
[ 40.919721] 3f20: 0000000000000002 0000000000000004 ffffffffffffffff
0000007f8d136588
[ 40.927474] 3f40: 0000000000000000 0000007f8d0ae3c0 0000007fd65069e0
00000000004ee000
[ 40.935226] 3f60: 0000000000000001 000000001285f6a0 000000000000000d
0000000000000001
[ 40.942980] 3f80: 0000000000000020 000000001285eed8 00000000004ba158
0000000000000000
[ 40.950732] 3fa0: 0000000000000000 0000007fd6507f30 000000000040e74c
0000007fd6507130
[ 40.958485] 3fc0: 0000007f8d0ae3dc 0000000060000000 0000000000000001
0000000000000040
[ 40.966238] 3fe0: 0000000000000000 0000000000000000 0000002000103a00
4000000010000000
[ 40.973986] Call trace:
[ 40.976405] Exception stack(0xffffffc974d63a90 to 0xffffffc974d63bc0)
[ 40.982780] 3a80: 0000000000020002
0000008000000000
[ 40.990533] 3aa0: ffffffc974d63c60 ffffff800834d1bc ffffffc974d63ae0
ffffff80085377a4
[ 40.998287] 3ac0: ffffffc974d63b10 ffffff8008537424 ffffffc975e3ac28
ffffffc975e3ac38
[ 41.006041] 3ae0: ffffffc974d63b30 ffffff80081737cc ffffffc975e3ac38
ffffff8008da62c0
[ 41.013794] 3b00: ffffffc975e98100 ffffff80085401b0 0000000000000001
ffffff8008540a08
[ 41.021547] 3b20: 00000000000036b8 0000000000000040 0000000000020002
0000000000000000
[ 41.029300] 3b40: ffffffc9763f10e8 0000000000000000 ffffffc9763f1210
0000000000000000
[ 41.037053] 3b60: 0000000000000000 ffffffc975f95018 0000000000210d00
0000000040000000
[ 41.044805] 3b80: 7f7f7f7f7f7f7f7f 0101010101010101 0000000000000030
ffffffc974cd01cd
[ 41.052558] 3ba0: ffffffffffffffff ffffffc974cd091c 0000000000000001
0000000000000007
[ 41.060311] [<ffffff800834d1bc>] kobject_get+0x14/0x88
[ 41.065398] [<ffffff80084c3dec>] iommu_get_domain_for_dev+0x1c/0x48
[ 41.071607] [<ffffff80080948c4>] arch_teardown_dma_ops+0x14/0x68
[ 41.077556] [<ffffff8008764a14>] of_dma_deconfigure+0xc/0x18
[ 41.083161] [<ffffff8008552804>] dma_deconfigure+0xc/0x18
[ 41.088509] [<ffffff800853ba10>] __device_release_driver+0x88/0x120
[ 41.094715] [<ffffff800853bacc>] device_release_driver+0x24/0x38
[ 41.100663] [<ffffff800853a868>] unbind_store+0xe8/0x110
[ 41.105922] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
[ 41.111268] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
[ 41.116612] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
[ 41.122216] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
[ 41.127390] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
[ 41.132391] [<ffffff80081c7274>] SyS_write+0x44/0xa0
[ 41.137307] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
[ 41.142567] Code: 910003fd f9000bf3 aa0003f3 b4000180 (3940f000)
[ 41.148667] ---[ end trace 35c1e743d6e6c037 ]---
Segmentation fault
/ #

[Sricharan]

Hi Robin,

Post by Robin Murphy

Post by Sricharan R
Initial post from Laurent Pinchart[1]. This is
series calls the dma ops configuration for the devices
at a generic place so that it works for all busses.
The dma_configure_ops for a device is now called during
the device_attach callback just before the probe of the
bus/driver is called. Similarly dma_deconfigure is called during
device/driver_detach path.
pci_bus_add_devices (platform/amba)(_device_create/driver_register)
| |
pci_bus_add_device (device_add/driver_register)
| |
device_attach device_initial_probe
| |
__device_attach_driver __device_attach_driver
|
driver_probe_device
|
really_probe
|
dma_configure
Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.
If the ACPI bus code follows the same, we can add acpi_dma_configure
at the same place as of_dma_configure.
This series is based on the recently merged Generic DT bindings for
This time tested this with platform and pci device for probe deferral
and reprobe on arm64 based platform. There is an issue on the cleanup
path for arm64 though, where there is WARN_ON if the dma_ops is reset while
device is attached to an domain in arch_teardown_dma_ops.
But with iommu_groups created from the iommu driver, the device is always
attached to a domain/default_domain. So so the WARN has to be removed/handled
probably.

I've finally got the chance to take a proper look at this series (sorry
for the delay). First up, with these patches on top of 4.9-rc2, my
little script for unbinding some PCI devices and rebinding them to VFIO
now goes horribly, horribly wrong.
Robin.

Thanks for looking in to this.
I was trying to reproduce the below with a command like this in my setup.

echo 0002\:00\:00.0 > /sys/bus/pci/devices/0002\:00\:000.0/driver/unbind0.0/driver/unbind
echo 0x17cb 0x0104 > /sys/bus/pci/drivers/vfio-pci/new_id

But for me the unbind and reconfiguring/adding the iommus_ops to vfio-pci did
succeed, although the WARN_ON in arch_teardown_dma_ops was there, that
could be suppresed. The vfio_pci_probe was not going through because
the pci hdr_type was not PCI_HEADER_TYPE_NORMAL.
But anyways iommu unbind/rebind seemed to be going through.

If i can get what your script is doing, i can try that and see what happens.

Regards,
Sricharan

Post by Robin Murphy
[ 39.901592] iommu: Removing device 0000:08:00.0 from group 0
[ 39.907383] ------------[ cut here ]------------
[ 39.911969] WARNING: CPU: 0 PID: 174 at
arch/arm64/mm/dma-mapping.c:856 arch_teardown_dma_ops+0x48/0x68
[ 39.924290]
[ 39.925766] CPU: 0 PID: 174 Comm: vfio Not tainted 4.9.0-rc2+ #1249
[ 39.931967] Hardware name: ARM Juno development board (r1) (DT)
[ 39.937826] task: ffffffc975ee9900 task.stack: ffffffc974d60000
[ 39.943687] PC is at arch_teardown_dma_ops+0x48/0x68
[ 39.948603] LR is at arch_teardown_dma_ops+0x34/0x68
[ 39.953516] pc : [<ffffff80080948f8>] lr : [<ffffff80080948e4>]
pstate: 60000145
[ 39.960834] sp : ffffffc974d63ca0
[ 39.964112] x29: ffffffc974d63ca0 x28: ffffffc974d60000
[ 39.969377] x27: ffffff80088a2000 x26: 0000000000000040
[ 39.974642] x25: 0000000000000123 x24: ffffffc976a48918
[ 39.979907] x23: ffffffc974d63eb8 x22: ffffff8008db7550
[ 39.985171] x21: 000000000000000d x20: ffffffc9763e9b50
[ 39.990435] x19: ffffffc9763f20a0 x18: 0000000000000010
[ 39.995699] x17: 0000007f99c18018 x16: ffffff80080c0580
[ 40.000964] x15: ffffff8008bb7000 x14: 000137c100013798
[ 40.006228] x13: ffffffffff000000 x12: ffffffffffffffff
[ 40.011492] x11: 0000000000000018 x10: 000000000000000d
[ 40.016757] x9 : 0000000040000000 x8 : 0000000000210d00
[ 40.022021] x7 : 00000049771bc000 x6 : 00000000001f17ed
[ 40.027286] x5 : ffffff80084c4208 x4 : 0000000000000080
[ 40.032551] x3 : ffffffc975ea9800 x2 : ffffffbf25d7aa50
[ 40.037815] x1 : 0000000000000000 x0 : 0000000000000080
[ 40.043078]
[ 40.044549] ---[ end trace 35c1e743d6e6c035 ]---
[ 40.051537] Exception stack(0xffffffc974d63ad0 to 0xffffffc974d63c00)
[ 40.057914] 3ac0: ffffffc9763f20a0
0000008000000000
[ 40.065668] 3ae0: ffffffc974d63ca0 ffffff80080948f8 ffffffbf25d7aa40
ffffffc975ea9800
[ 40.073421] 3b00: ffffffc974d60000 000000000002fc80 ffffffc976801e00
ffffffc974d60000
[ 40.081175] 3b20: ffffff80084c4208 0000000000000040 ffffff80088a2000
ffffffc974d60000
[ 40.088928] 3b40: ffffff80084caf78 ffffffc974d60000 ffffffc974d60000
ffffffc974d60000
[ 40.096682] 3b60: ffffffc975ea9800 ffffffc974d60000 0000000000000080
0000000000000000
[ 40.104435] 3b80: ffffffbf25d7aa50 ffffffc975ea9800 0000000000000080
ffffff80084c4208
[ 40.112188] 3ba0: 00000000001f17ed 00000049771bc000 0000000000210d00
0000000040000000
[ 40.119941] 3bc0: 000000000000000d 0000000000000018 ffffffffffffffff
ffffffffff000000
[ 40.127695] 3be0: 000137c100013798 ffffff8008bb7000 ffffff80080c0580
0000007f99c18018
[ 40.135450] [<ffffff80080948f8>] arch_teardown_dma_ops+0x48/0x68
[ 40.141400] [<ffffff8008764a14>] of_dma_deconfigure+0xc/0x18
[ 40.147005] [<ffffff8008552804>] dma_deconfigure+0xc/0x18
[ 40.152353] [<ffffff800853ba10>] __device_release_driver+0x88/0x120
[ 40.158560] [<ffffff800853bacc>] device_release_driver+0x24/0x38
[ 40.164507] [<ffffff800853a868>] unbind_store+0xe8/0x110
[ 40.169767] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
[ 40.175113] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
[ 40.180458] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
[ 40.186063] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
[ 40.191237] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
[ 40.196239] [<ffffff80081c7274>] SyS_write+0x44/0xa0
[ 40.201155] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
[ 40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
[ 40.212382] vfio-pci: probe of 0000:08:00.0 failed with error -22
[ 40.228075] ------------[ cut here ]------------
[ 40.235263] WARNING: CPU: 1 PID: 174 at ./include/linux/kref.h:46
kobject_get+0x64/0x88
[ 40.246201]
[ 40.247673] CPU: 1 PID: 174 Comm: vfio Tainted: G W
4.9.0-rc2+ #1249
[ 40.255076] Hardware name: ARM Juno development board (r1) (DT)
[ 40.260932] task: ffffffc975ee9900 task.stack: ffffffc974d60000
[ 40.266787] PC is at kobject_get+0x64/0x88
[ 40.270840] LR is at iommu_bus_notifier+0x40/0x110
[ 40.275577] pc : [<ffffff800834d20c>] lr : [<ffffff80084c3fd0>]
pstate: 80000145
[ 40.282894] sp : ffffffc974d63c00
[ 40.286169] x29: ffffffc974d63c00 x28: ffffffc974d60000
[ 40.291431] x27: ffffff80088a2000 x26: 0000000000000040
[ 40.296692] x25: 0000000000000123 x24: ffffffc974c8f418
[ 40.301953] x23: 0000000000000006 x22: ffffffc9763f10a0
[ 40.307214] x21: ffffffc9763e9a00 x20: ffffffc9763f10a0
[ 40.312474] x19: ffffffc9763ebc80 x18: 0000007fd65069e0
[ 40.317734] x17: 0000007f8d0ae3c0 x16: ffffff80081c7230
[ 40.322995] x15: 0000007f8d136588 x14: ffffffffffffffff
[ 40.328255] x13: 0000000000000004 x12: 0000000000000030
[ 40.333515] x11: 0000000000000030 x10: 0101010101010101
[ 40.338775] x9 : feff716475687163 x8 : 7f7f7f7f7f7f7f7f
[ 40.344035] x7 : feff716475687163 x6 : ffffffc976abf400
[ 40.349295] x5 : ffffffc976abf400 x4 : 0000000000000000
[ 40.354555] x3 : ffffff80084c3f90 x2 : ffffffc9763ebcb8
[ 40.359814] x1 : 0000000000000001 x0 : ffffff8008d4f000
[ 40.365074]
[ 40.366542] ---[ end trace 35c1e743d6e6c036 ]---
[ 40.373523] Exception stack(0xffffffc974d63a30 to 0xffffffc974d63b60)
[ 40.379895] 3a20: ffffffc9763ebc80
0000008000000000
[ 40.387643] 3a40: ffffffc974d63c00 ffffff800834d20c ffffffc976812400
ffffff8008237d94
[ 40.395391] 3a60: ffffffbf25d78940 ffffffc974d60000 ffffffc975e259d8
0000000000005b81
[ 40.403139] 3a80: ffffffc974d60000 ffffff8008d4b31f ffffff8008b0f000
ffffffc976811c80
[ 40.410887] 3aa0: ffffffc974d60000 ffffffc974d60000 ffffffc974d60000
ffffff8008237000
[ 40.418634] 3ac0: ffffffc975e259d8 ffffff8008b1b9a8 ffffff8008d4f000
0000000000000001
[ 40.426382] 3ae0: ffffffc9763ebcb8 ffffff80084c3f90 0000000000000000
ffffffc976abf400
[ 40.434130] 3b00: ffffffc976abf400 feff716475687163 7f7f7f7f7f7f7f7f
feff716475687163
[ 40.441877] 3b20: 0101010101010101 0000000000000030 0000000000000030
0000000000000004
[ 40.449625] 3b40: ffffffffffffffff 0000007f8d136588 ffffff80081c7230
0000007f8d0ae3c0
[ 40.457372] [<ffffff800834d20c>] kobject_get+0x64/0x88
[ 40.462455] [<ffffff80084c3fd0>] iommu_bus_notifier+0x40/0x110
[ 40.468227] [<ffffff80080da288>] notifier_call_chain+0x50/0x90
[ 40.473997] [<ffffff80080da694>] __blocking_notifier_call_chain+0x4c/0x90
[ 40.480713] [<ffffff80080da6ec>] blocking_notifier_call_chain+0x14/0x20
[ 40.487259] [<ffffff800853b9e4>] __device_release_driver+0x5c/0x120
[ 40.493460] [<ffffff800853bacc>] device_release_driver+0x24/0x38
[ 40.499402] [<ffffff800853a868>] unbind_store+0xe8/0x110
[ 40.504656] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
[ 40.509997] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
[ 40.515337] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
[ 40.520936] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
[ 40.526104] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
[ 40.531100] [<ffffff80081c7274>] SyS_write+0x44/0xa0
[ 40.536011] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
[ 40.541324] ata1.00: disabled
[ 40.544878] sd 0:0:0:0: [sda] Synchronizing SCSI cache
hostbyte=0x04 driverbyte=0x00
[ 40.558871] sd 0:0:0:0: [sda] Stopping disk
hostbyte=0x04 driverbyte=0x00
[ 40.586990] Unable to handle kernel paging request at virtual address
0002003e
[ 40.594702] pgd = ffffffc974c80000
[ 40.598165] [0002003e] *pgd=00000009f5102003[ 40.602241] ,
*pud=00000009f5102003
, *pmd=0000000000000000[ 40.607694]
[ 40.609171] Internal error: Oops: 96000006 [#1] PREEMPT SMP
[ 40.617712] CPU: 3 PID: 174 Comm: vfio Tainted: G W
4.9.0-rc2+ #1249
[ 40.625118] Hardware name: ARM Juno development board (r1) (DT)
[ 40.630977] task: ffffffc975ee9900 task.stack: ffffffc974d60000
[ 40.636841] PC is at kobject_get+0x14/0x88
[ 40.640897] LR is at iommu_get_domain_for_dev+0x1c/0x48
[ 40.646068] pc : [<ffffff800834d1bc>] lr : [<ffffff80084c3dec>]
pstate: 60000145
[ 40.653387] sp : ffffffc974d63c60
[ 40.656664] x29: ffffffc974d63c60 x28: ffffffc974d60000
[ 40.661928] x27: ffffff80088a2000 x26: 0000000000000040
[ 40.667193] x25: 0000000000000123 x24: ffffffc974c8f418
[ 40.672457] x23: ffffffc974d63eb8 x22: ffffff8008dab568
[ 40.677720] x21: 000000000000000d x20: ffffff8008dab568
[ 40.682984] x19: 0000000000020002 x18: 0000000000000000
[ 40.688246] x17: 0000000000000007 x16: 0000000000000001
[ 40.693509] x15: ffffffc974cd091c x14: ffffffffffffffff
[ 40.698773] x13: ffffffc974cd01cd x12: 0000000000000030
[ 40.704036] x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f
[ 40.709300] x9 : 0000000040000000 x8 : 0000000000210d00
[ 40.714563] x7 : ffffffc975f95018 x6 : 0000000000000000
[ 40.719826] x5 : 0000000000000000 x4 : ffffffc9763f1210
[ 40.725088] x3 : 0000000000000000 x2 : ffffffc9763f10e8
[ 40.730351] x1 : 0000000000000000 x0 : 0000000000020002
[ 40.735613]
[ 40.737085] Process vfio (pid: 174, stack limit = 0xffffffc974d60020)
[ 40.743460] Stack: (0xffffffc974d63c60 to 0xffffffc974d64000)
[ 40.749150] 3c60: ffffffc974d63c80 ffffff80084c3dec ffffffc9763e9a00
ffffff80085377a4
[ 40.756904] 3c80: ffffffc974d63ca0 ffffff80080948c4 ffffffc9763f10a0
ffffff8008dab568
[ 40.764658] 3ca0: ffffffc974d63cc0 ffffff8008764a14 ffffffc9763f10a0
ffffff8008dab568
[ 40.772411] 3cc0: ffffffc974d63cd0 ffffff8008552804 ffffffc974d63ce0
ffffff800853ba10
[ 40.780165] 3ce0: ffffffc974d63d00 ffffff800853bacc ffffffc9763f1100
ffffffc9763f10a0
[ 40.787918] 3d00: ffffffc974d63d20 ffffff800853a868 ffffff8008d68f18
ffffffc9763f10a0
[ 40.795672] 3d20: ffffffc974d63d50 ffffff8008539c70 000000000000000d
ffffffc974c8f400
[ 40.803425] 3d40: ffffffc9757d5880 0000000000000000 ffffffc974d63d60
ffffff800823ab18
[ 40.811178] 3d60: ffffffc974d63d70 ffffff8008239ea8 ffffffc974d63dc0
ffffff80081c507c
[ 40.818931] 3d80: 000000000000000d 0000000000000000 ffffffc974c8f100
ffffffc974d63eb8
[ 40.826684] 3da0: 000000001285f6a0 0000000000000015 0000000000000123
ffffff80080bf6ac
[ 40.834437] 3dc0: ffffffc974d63e40 ffffff80081c5e80 000000000000000d
0000000000000000
[ 40.842190] 3de0: ffffffc974d63e30 ffffff80080c087c ffffffc974d63e20
ffffff80081c5c0c
[ 40.849943] 3e00: ffffffc974c8f100 0000000000000001 ffffffc974c8f100
ffffffc974d63eb8
[ 40.857696] 3e20: ffffffc974d63e40 ffffff80081c5f48 000000000000000d
ffffffc974c8f100
[ 40.865450] 3e40: ffffffc974d63e80 ffffff80081c7274 ffffffc974c8f100
ffffffc974c8f100
[ 40.873203] 3e60: 000000001285f6a0 000000000000000d 0000000060000000
0000000000000000
[ 40.880956] 3e80: 0000000000000000 ffffff8008082ef0 0000000000000000
0000000000000001
[ 40.888709] 3ea0: ffffffffffffffff 0000007f8d0ae3dc 0000000000000000
0000000000000000
[ 40.896461] 3ec0: 0000000000000001 000000001285f6a0 000000000000000d
0000000000000000
[ 40.904215] 3ee0: ae2e2e2e3f464b49 0000000000000000 000000001285f6b0
39322f392f2f2f2f
[ 40.911968] 3f00: 0000000000000040 fefefeff2f2d2f2f 7f7f7f7f7f7f7f7f
0101010101010101
[ 40.919721] 3f20: 0000000000000002 0000000000000004 ffffffffffffffff
0000007f8d136588
[ 40.927474] 3f40: 0000000000000000 0000007f8d0ae3c0 0000007fd65069e0
00000000004ee000
[ 40.935226] 3f60: 0000000000000001 000000001285f6a0 000000000000000d
0000000000000001
[ 40.942980] 3f80: 0000000000000020 000000001285eed8 00000000004ba158
0000000000000000
[ 40.950732] 3fa0: 0000000000000000 0000007fd6507f30 000000000040e74c
0000007fd6507130
[ 40.958485] 3fc0: 0000007f8d0ae3dc 0000000060000000 0000000000000001
0000000000000040
[ 40.966238] 3fe0: 0000000000000000 0000000000000000 0000002000103a00
4000000010000000
[ 40.976405] Exception stack(0xffffffc974d63a90 to 0xffffffc974d63bc0)
[ 40.982780] 3a80: 0000000000020002
0000008000000000
[ 40.990533] 3aa0: ffffffc974d63c60 ffffff800834d1bc ffffffc974d63ae0
ffffff80085377a4
[ 40.998287] 3ac0: ffffffc974d63b10 ffffff8008537424 ffffffc975e3ac28
ffffffc975e3ac38
[ 41.006041] 3ae0: ffffffc974d63b30 ffffff80081737cc ffffffc975e3ac38
ffffff8008da62c0
[ 41.013794] 3b00: ffffffc975e98100 ffffff80085401b0 0000000000000001
ffffff8008540a08
[ 41.021547] 3b20: 00000000000036b8 0000000000000040 0000000000020002
0000000000000000
[ 41.029300] 3b40: ffffffc9763f10e8 0000000000000000 ffffffc9763f1210
0000000000000000
[ 41.037053] 3b60: 0000000000000000 ffffffc975f95018 0000000000210d00
0000000040000000
[ 41.044805] 3b80: 7f7f7f7f7f7f7f7f 0101010101010101 0000000000000030
ffffffc974cd01cd
[ 41.052558] 3ba0: ffffffffffffffff ffffffc974cd091c 0000000000000001
0000000000000007
[ 41.060311] [<ffffff800834d1bc>] kobject_get+0x14/0x88
[ 41.065398] [<ffffff80084c3dec>] iommu_get_domain_for_dev+0x1c/0x48
[ 41.071607] [<ffffff80080948c4>] arch_teardown_dma_ops+0x14/0x68
[ 41.077556] [<ffffff8008764a14>] of_dma_deconfigure+0xc/0x18
[ 41.083161] [<ffffff8008552804>] dma_deconfigure+0xc/0x18
[ 41.088509] [<ffffff800853ba10>] __device_release_driver+0x88/0x120
[ 41.094715] [<ffffff800853bacc>] device_release_driver+0x24/0x38
[ 41.100663] [<ffffff800853a868>] unbind_store+0xe8/0x110
[ 41.105922] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
[ 41.111268] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
[ 41.116612] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
[ 41.122216] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
[ 41.127390] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
[ 41.132391] [<ffffff80081c7274>] SyS_write+0x44/0xa0
[ 41.137307] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
[ 41.142567] Code: 910003fd f9000bf3 aa0003f3 b4000180 (3940f000)
[ 41.148667] ---[ end trace 35c1e743d6e6c037 ]---
Segmentation fault
/ #
_______________________________________________
linux-arm-kernel mailing list
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[Robin Murphy]

Post by Sricharan
Hi Robin,

Post by Robin Murphy

Post by Sricharan R
Initial post from Laurent Pinchart[1]. This is
series calls the dma ops configuration for the devices
at a generic place so that it works for all busses.
The dma_configure_ops for a device is now called during
the device_attach callback just before the probe of the
bus/driver is called. Similarly dma_deconfigure is called during
device/driver_detach path.
pci_bus_add_devices (platform/amba)(_device_create/driver_register)
| |
pci_bus_add_device (device_add/driver_register)
| |
device_attach device_initial_probe
| |
__device_attach_driver __device_attach_driver
|
driver_probe_device
|
really_probe
|
dma_configure
Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.
If the ACPI bus code follows the same, we can add acpi_dma_configure
at the same place as of_dma_configure.
This series is based on the recently merged Generic DT bindings for
This time tested this with platform and pci device for probe deferral
and reprobe on arm64 based platform. There is an issue on the cleanup
path for arm64 though, where there is WARN_ON if the dma_ops is reset while
device is attached to an domain in arch_teardown_dma_ops.
But with iommu_groups created from the iommu driver, the device is always
attached to a domain/default_domain. So so the WARN has to be removed/handled
probably.

I've finally got the chance to take a proper look at this series (sorry
for the delay). First up, with these patches on top of 4.9-rc2, my
little script for unbinding some PCI devices and rebinding them to VFIO
now goes horribly, horribly wrong.
Robin.

Thanks for looking in to this.
I was trying to reproduce the below with a command like this in my setup.
echo 0002\:00\:00.0 > /sys/bus/pci/devices/0002\:00\:000.0/driver/unbind0.0/driver/unbind
echo 0x17cb 0x0104 > /sys/bus/pci/drivers/vfio-pci/new_id
But for me the unbind and reconfiguring/adding the iommus_ops to vfio-pci did
succeed, although the WARN_ON in arch_teardown_dma_ops was there, that

Oh, yes, I hacked that out already to cut the noise down.

Post by Sricharan
could be suppresed. The vfio_pci_probe was not going through because
the pci hdr_type was not PCI_HEADER_TYPE_NORMAL.
But anyways iommu unbind/rebind seemed to be going through.
If i can get what your script is doing, i can try that and see what happens.

---8<---
#!/bin/sh

#Juno Sky2, SATA
DEVICES='0000:08:00.0 0000:03:00.0'

for DEV in $DEVICES
do
BUSDEV=/sys/bus/pci/devices/$DEV
GROUP=$(basename $(readlink $BUSDEV/iommu_group))
DRV=$(readlink -f $BUSDEV/driver)
read VID < $BUSDEV/vendor
read DID < $BUSDEV/device

echo $DEV > $BUSDEV/driver/unbind
echo $VID $DID > /sys/bus/pci/drivers/vfio-pci/new_id
done
# it would then goes on to launch kvmtool and rebind the original
# drivers afterwards, but that doesn't matter here
--->8---

The segfault doesn't always happen, but the kref warnings and the
vfio-pci driver failing to probe certainly do.

Post by Sricharan
Regards,
Sricharan

Post by Robin Murphy
[ 39.901592] iommu: Removing device 0000:08:00.0 from group 0

Yikes, on second look, that definitely shouldn't be happening.
Everything below is probably the resulting fallout.

Robin.

Post by Sricharan

Post by Robin Murphy
[ 39.907383] ------------[ cut here ]------------
[ 39.911969] WARNING: CPU: 0 PID: 174 at
arch/arm64/mm/dma-mapping.c:856 arch_teardown_dma_ops+0x48/0x68
[ 39.924290]
[ 39.925766] CPU: 0 PID: 174 Comm: vfio Not tainted 4.9.0-rc2+ #1249
[ 39.931967] Hardware name: ARM Juno development board (r1) (DT)
[ 39.937826] task: ffffffc975ee9900 task.stack: ffffffc974d60000
[ 39.943687] PC is at arch_teardown_dma_ops+0x48/0x68
[ 39.948603] LR is at arch_teardown_dma_ops+0x34/0x68
[ 39.953516] pc : [<ffffff80080948f8>] lr : [<ffffff80080948e4>]
pstate: 60000145
[ 39.960834] sp : ffffffc974d63ca0
[ 39.964112] x29: ffffffc974d63ca0 x28: ffffffc974d60000
[ 39.969377] x27: ffffff80088a2000 x26: 0000000000000040
[ 39.974642] x25: 0000000000000123 x24: ffffffc976a48918
[ 39.979907] x23: ffffffc974d63eb8 x22: ffffff8008db7550
[ 39.985171] x21: 000000000000000d x20: ffffffc9763e9b50
[ 39.990435] x19: ffffffc9763f20a0 x18: 0000000000000010
[ 39.995699] x17: 0000007f99c18018 x16: ffffff80080c0580
[ 40.000964] x15: ffffff8008bb7000 x14: 000137c100013798
[ 40.006228] x13: ffffffffff000000 x12: ffffffffffffffff
[ 40.011492] x11: 0000000000000018 x10: 000000000000000d
[ 40.016757] x9 : 0000000040000000 x8 : 0000000000210d00
[ 40.022021] x7 : 00000049771bc000 x6 : 00000000001f17ed
[ 40.027286] x5 : ffffff80084c4208 x4 : 0000000000000080
[ 40.032551] x3 : ffffffc975ea9800 x2 : ffffffbf25d7aa50
[ 40.037815] x1 : 0000000000000000 x0 : 0000000000000080
[ 40.043078]
[ 40.044549] ---[ end trace 35c1e743d6e6c035 ]---
[ 40.051537] Exception stack(0xffffffc974d63ad0 to 0xffffffc974d63c00)
[ 40.057914] 3ac0: ffffffc9763f20a0
0000008000000000
[ 40.065668] 3ae0: ffffffc974d63ca0 ffffff80080948f8 ffffffbf25d7aa40
ffffffc975ea9800
[ 40.073421] 3b00: ffffffc974d60000 000000000002fc80 ffffffc976801e00
ffffffc974d60000
[ 40.081175] 3b20: ffffff80084c4208 0000000000000040 ffffff80088a2000
ffffffc974d60000
[ 40.088928] 3b40: ffffff80084caf78 ffffffc974d60000 ffffffc974d60000
ffffffc974d60000
[ 40.096682] 3b60: ffffffc975ea9800 ffffffc974d60000 0000000000000080
0000000000000000
[ 40.104435] 3b80: ffffffbf25d7aa50 ffffffc975ea9800 0000000000000080
ffffff80084c4208
[ 40.112188] 3ba0: 00000000001f17ed 00000049771bc000 0000000000210d00
0000000040000000
[ 40.119941] 3bc0: 000000000000000d 0000000000000018 ffffffffffffffff
ffffffffff000000
[ 40.127695] 3be0: 000137c100013798 ffffff8008bb7000 ffffff80080c0580
0000007f99c18018
[ 40.135450] [<ffffff80080948f8>] arch_teardown_dma_ops+0x48/0x68
[ 40.141400] [<ffffff8008764a14>] of_dma_deconfigure+0xc/0x18
[ 40.147005] [<ffffff8008552804>] dma_deconfigure+0xc/0x18
[ 40.152353] [<ffffff800853ba10>] __device_release_driver+0x88/0x120
[ 40.158560] [<ffffff800853bacc>] device_release_driver+0x24/0x38
[ 40.164507] [<ffffff800853a868>] unbind_store+0xe8/0x110
[ 40.169767] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
[ 40.175113] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
[ 40.180458] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
[ 40.186063] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
[ 40.191237] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
[ 40.196239] [<ffffff80081c7274>] SyS_write+0x44/0xa0
[ 40.201155] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
[ 40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
[ 40.212382] vfio-pci: probe of 0000:08:00.0 failed with error -22
[ 40.228075] ------------[ cut here ]------------
[ 40.235263] WARNING: CPU: 1 PID: 174 at ./include/linux/kref.h:46
kobject_get+0x64/0x88
[ 40.246201]
[ 40.247673] CPU: 1 PID: 174 Comm: vfio Tainted: G W
4.9.0-rc2+ #1249
[ 40.255076] Hardware name: ARM Juno development board (r1) (DT)
[ 40.260932] task: ffffffc975ee9900 task.stack: ffffffc974d60000
[ 40.266787] PC is at kobject_get+0x64/0x88
[ 40.270840] LR is at iommu_bus_notifier+0x40/0x110
[ 40.275577] pc : [<ffffff800834d20c>] lr : [<ffffff80084c3fd0>]
pstate: 80000145
[ 40.282894] sp : ffffffc974d63c00
[ 40.286169] x29: ffffffc974d63c00 x28: ffffffc974d60000
[ 40.291431] x27: ffffff80088a2000 x26: 0000000000000040
[ 40.296692] x25: 0000000000000123 x24: ffffffc974c8f418
[ 40.301953] x23: 0000000000000006 x22: ffffffc9763f10a0
[ 40.307214] x21: ffffffc9763e9a00 x20: ffffffc9763f10a0
[ 40.312474] x19: ffffffc9763ebc80 x18: 0000007fd65069e0
[ 40.317734] x17: 0000007f8d0ae3c0 x16: ffffff80081c7230
[ 40.322995] x15: 0000007f8d136588 x14: ffffffffffffffff
[ 40.328255] x13: 0000000000000004 x12: 0000000000000030
[ 40.333515] x11: 0000000000000030 x10: 0101010101010101
[ 40.338775] x9 : feff716475687163 x8 : 7f7f7f7f7f7f7f7f
[ 40.344035] x7 : feff716475687163 x6 : ffffffc976abf400
[ 40.349295] x5 : ffffffc976abf400 x4 : 0000000000000000
[ 40.354555] x3 : ffffff80084c3f90 x2 : ffffffc9763ebcb8
[ 40.359814] x1 : 0000000000000001 x0 : ffffff8008d4f000
[ 40.365074]
[ 40.366542] ---[ end trace 35c1e743d6e6c036 ]---
[ 40.373523] Exception stack(0xffffffc974d63a30 to 0xffffffc974d63b60)
[ 40.379895] 3a20: ffffffc9763ebc80
0000008000000000
[ 40.387643] 3a40: ffffffc974d63c00 ffffff800834d20c ffffffc976812400
ffffff8008237d94
[ 40.395391] 3a60: ffffffbf25d78940 ffffffc974d60000 ffffffc975e259d8
0000000000005b81
[ 40.403139] 3a80: ffffffc974d60000 ffffff8008d4b31f ffffff8008b0f000
ffffffc976811c80
[ 40.410887] 3aa0: ffffffc974d60000 ffffffc974d60000 ffffffc974d60000
ffffff8008237000
[ 40.418634] 3ac0: ffffffc975e259d8 ffffff8008b1b9a8 ffffff8008d4f000
0000000000000001
[ 40.426382] 3ae0: ffffffc9763ebcb8 ffffff80084c3f90 0000000000000000
ffffffc976abf400
[ 40.434130] 3b00: ffffffc976abf400 feff716475687163 7f7f7f7f7f7f7f7f
feff716475687163
[ 40.441877] 3b20: 0101010101010101 0000000000000030 0000000000000030
0000000000000004
[ 40.449625] 3b40: ffffffffffffffff 0000007f8d136588 ffffff80081c7230
0000007f8d0ae3c0
[ 40.457372] [<ffffff800834d20c>] kobject_get+0x64/0x88
[ 40.462455] [<ffffff80084c3fd0>] iommu_bus_notifier+0x40/0x110
[ 40.468227] [<ffffff80080da288>] notifier_call_chain+0x50/0x90
[ 40.473997] [<ffffff80080da694>] __blocking_notifier_call_chain+0x4c/0x90
[ 40.480713] [<ffffff80080da6ec>] blocking_notifier_call_chain+0x14/0x20
[ 40.487259] [<ffffff800853b9e4>] __device_release_driver+0x5c/0x120
[ 40.493460] [<ffffff800853bacc>] device_release_driver+0x24/0x38
[ 40.499402] [<ffffff800853a868>] unbind_store+0xe8/0x110
[ 40.504656] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
[ 40.509997] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
[ 40.515337] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
[ 40.520936] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
[ 40.526104] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
[ 40.531100] [<ffffff80081c7274>] SyS_write+0x44/0xa0
[ 40.536011] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
[ 40.541324] ata1.00: disabled
[ 40.544878] sd 0:0:0:0: [sda] Synchronizing SCSI cache
hostbyte=0x04 driverbyte=0x00
[ 40.558871] sd 0:0:0:0: [sda] Stopping disk
hostbyte=0x04 driverbyte=0x00
[ 40.586990] Unable to handle kernel paging request at virtual address
0002003e
[ 40.594702] pgd = ffffffc974c80000
[ 40.598165] [0002003e] *pgd=00000009f5102003[ 40.602241] ,
*pud=00000009f5102003
, *pmd=0000000000000000[ 40.607694]
[ 40.609171] Internal error: Oops: 96000006 [#1] PREEMPT SMP
[ 40.617712] CPU: 3 PID: 174 Comm: vfio Tainted: G W
4.9.0-rc2+ #1249
[ 40.625118] Hardware name: ARM Juno development board (r1) (DT)
[ 40.630977] task: ffffffc975ee9900 task.stack: ffffffc974d60000
[ 40.636841] PC is at kobject_get+0x14/0x88
[ 40.640897] LR is at iommu_get_domain_for_dev+0x1c/0x48
[ 40.646068] pc : [<ffffff800834d1bc>] lr : [<ffffff80084c3dec>]
pstate: 60000145
[ 40.653387] sp : ffffffc974d63c60
[ 40.656664] x29: ffffffc974d63c60 x28: ffffffc974d60000
[ 40.661928] x27: ffffff80088a2000 x26: 0000000000000040
[ 40.667193] x25: 0000000000000123 x24: ffffffc974c8f418
[ 40.672457] x23: ffffffc974d63eb8 x22: ffffff8008dab568
[ 40.677720] x21: 000000000000000d x20: ffffff8008dab568
[ 40.682984] x19: 0000000000020002 x18: 0000000000000000
[ 40.688246] x17: 0000000000000007 x16: 0000000000000001
[ 40.693509] x15: ffffffc974cd091c x14: ffffffffffffffff
[ 40.698773] x13: ffffffc974cd01cd x12: 0000000000000030
[ 40.704036] x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f
[ 40.709300] x9 : 0000000040000000 x8 : 0000000000210d00
[ 40.714563] x7 : ffffffc975f95018 x6 : 0000000000000000
[ 40.719826] x5 : 0000000000000000 x4 : ffffffc9763f1210
[ 40.725088] x3 : 0000000000000000 x2 : ffffffc9763f10e8
[ 40.730351] x1 : 0000000000000000 x0 : 0000000000020002
[ 40.735613]
[ 40.737085] Process vfio (pid: 174, stack limit = 0xffffffc974d60020)
[ 40.743460] Stack: (0xffffffc974d63c60 to 0xffffffc974d64000)
[ 40.749150] 3c60: ffffffc974d63c80 ffffff80084c3dec ffffffc9763e9a00
ffffff80085377a4
[ 40.756904] 3c80: ffffffc974d63ca0 ffffff80080948c4 ffffffc9763f10a0
ffffff8008dab568
[ 40.764658] 3ca0: ffffffc974d63cc0 ffffff8008764a14 ffffffc9763f10a0
ffffff8008dab568
[ 40.772411] 3cc0: ffffffc974d63cd0 ffffff8008552804 ffffffc974d63ce0
ffffff800853ba10
[ 40.780165] 3ce0: ffffffc974d63d00 ffffff800853bacc ffffffc9763f1100
ffffffc9763f10a0
[ 40.787918] 3d00: ffffffc974d63d20 ffffff800853a868 ffffff8008d68f18
ffffffc9763f10a0
[ 40.795672] 3d20: ffffffc974d63d50 ffffff8008539c70 000000000000000d
ffffffc974c8f400
[ 40.803425] 3d40: ffffffc9757d5880 0000000000000000 ffffffc974d63d60
ffffff800823ab18
[ 40.811178] 3d60: ffffffc974d63d70 ffffff8008239ea8 ffffffc974d63dc0
ffffff80081c507c
[ 40.818931] 3d80: 000000000000000d 0000000000000000 ffffffc974c8f100
ffffffc974d63eb8
[ 40.826684] 3da0: 000000001285f6a0 0000000000000015 0000000000000123
ffffff80080bf6ac
[ 40.834437] 3dc0: ffffffc974d63e40 ffffff80081c5e80 000000000000000d
0000000000000000
[ 40.842190] 3de0: ffffffc974d63e30 ffffff80080c087c ffffffc974d63e20
ffffff80081c5c0c
[ 40.849943] 3e00: ffffffc974c8f100 0000000000000001 ffffffc974c8f100
ffffffc974d63eb8
[ 40.857696] 3e20: ffffffc974d63e40 ffffff80081c5f48 000000000000000d
ffffffc974c8f100
[ 40.865450] 3e40: ffffffc974d63e80 ffffff80081c7274 ffffffc974c8f100
ffffffc974c8f100
[ 40.873203] 3e60: 000000001285f6a0 000000000000000d 0000000060000000
0000000000000000
[ 40.880956] 3e80: 0000000000000000 ffffff8008082ef0 0000000000000000
0000000000000001
[ 40.888709] 3ea0: ffffffffffffffff 0000007f8d0ae3dc 0000000000000000
0000000000000000
[ 40.896461] 3ec0: 0000000000000001 000000001285f6a0 000000000000000d
0000000000000000
[ 40.904215] 3ee0: ae2e2e2e3f464b49 0000000000000000 000000001285f6b0
39322f392f2f2f2f
[ 40.911968] 3f00: 0000000000000040 fefefeff2f2d2f2f 7f7f7f7f7f7f7f7f
0101010101010101
[ 40.919721] 3f20: 0000000000000002 0000000000000004 ffffffffffffffff
0000007f8d136588
[ 40.927474] 3f40: 0000000000000000 0000007f8d0ae3c0 0000007fd65069e0
00000000004ee000
[ 40.935226] 3f60: 0000000000000001 000000001285f6a0 000000000000000d
0000000000000001
[ 40.942980] 3f80: 0000000000000020 000000001285eed8 00000000004ba158
0000000000000000
[ 40.950732] 3fa0: 0000000000000000 0000007fd6507f30 000000000040e74c
0000007fd6507130
[ 40.958485] 3fc0: 0000007f8d0ae3dc 0000000060000000 0000000000000001
0000000000000040
[ 40.966238] 3fe0: 0000000000000000 0000000000000000 0000002000103a00
4000000010000000
[ 40.976405] Exception stack(0xffffffc974d63a90 to 0xffffffc974d63bc0)
[ 40.982780] 3a80: 0000000000020002
0000008000000000
[ 40.990533] 3aa0: ffffffc974d63c60 ffffff800834d1bc ffffffc974d63ae0
ffffff80085377a4
[ 40.998287] 3ac0: ffffffc974d63b10 ffffff8008537424 ffffffc975e3ac28
ffffffc975e3ac38
[ 41.006041] 3ae0: ffffffc974d63b30 ffffff80081737cc ffffffc975e3ac38
ffffff8008da62c0
[ 41.013794] 3b00: ffffffc975e98100 ffffff80085401b0 0000000000000001
ffffff8008540a08
[ 41.021547] 3b20: 00000000000036b8 0000000000000040 0000000000020002
0000000000000000
[ 41.029300] 3b40: ffffffc9763f10e8 0000000000000000 ffffffc9763f1210
0000000000000000
[ 41.037053] 3b60: 0000000000000000 ffffffc975f95018 0000000000210d00
0000000040000000
[ 41.044805] 3b80: 7f7f7f7f7f7f7f7f 0101010101010101 0000000000000030
ffffffc974cd01cd
[ 41.052558] 3ba0: ffffffffffffffff ffffffc974cd091c 0000000000000001
0000000000000007
[ 41.060311] [<ffffff800834d1bc>] kobject_get+0x14/0x88
[ 41.065398] [<ffffff80084c3dec>] iommu_get_domain_for_dev+0x1c/0x48
[ 41.071607] [<ffffff80080948c4>] arch_teardown_dma_ops+0x14/0x68
[ 41.077556] [<ffffff8008764a14>] of_dma_deconfigure+0xc/0x18
[ 41.083161] [<ffffff8008552804>] dma_deconfigure+0xc/0x18
[ 41.088509] [<ffffff800853ba10>] __device_release_driver+0x88/0x120
[ 41.094715] [<ffffff800853bacc>] device_release_driver+0x24/0x38
[ 41.100663] [<ffffff800853a868>] unbind_store+0xe8/0x110
[ 41.105922] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
[ 41.111268] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
[ 41.116612] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
[ 41.122216] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
[ 41.127390] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
[ 41.132391] [<ffffff80081c7274>] SyS_write+0x44/0xa0
[ 41.137307] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
[ 41.142567] Code: 910003fd f9000bf3 aa0003f3 b4000180 (3940f000)
[ 41.148667] ---[ end trace 35c1e743d6e6c037 ]---
Segmentation fault
/ #
_______________________________________________
linux-arm-kernel mailing list
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[Sricharan]

Hi Robin,

Post by Robin Murphy

Post by Robin Murphy
[ 39.901592] iommu: Removing device 0000:08:00.0 from group 0

Yikes, on second look, that definitely shouldn't be happening.
Everything below is probably the resulting fallout.

[ 40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops

I think the above print which says "failed to setup iommu_ops"
because the call ops->add_device failed in of_pci_iommu_configure
is the reason for the failure, in my case i simply do not get this even with
your scripts. ops->add_device succeeds in the rebind as well. So still
checking what could be happening in your case.


Regards,
Sricharan

Post by Robin Murphy
Robin.

Post by Robin Murphy
[ 39.907383] ------------[ cut here ]------------
[ 39.911969] WARNING: CPU: 0 PID: 174 at
arch/arm64/mm/dma-mapping.c:856 arch_teardown_dma_ops+0x48/0x68
[ 39.924290]
[ 39.925766] CPU: 0 PID: 174 Comm: vfio Not tainted 4.9.0-rc2+ #1249
[ 39.931967] Hardware name: ARM Juno development board (r1) (DT)
[ 39.937826] task: ffffffc975ee9900 task.stack: ffffffc974d60000
[ 39.943687] PC is at arch_teardown_dma_ops+0x48/0x68
[ 39.948603] LR is at arch_teardown_dma_ops+0x34/0x68
[ 39.953516] pc : [<ffffff80080948f8>] lr : [<ffffff80080948e4>]
pstate: 60000145
[ 39.960834] sp : ffffffc974d63ca0
[ 39.964112] x29: ffffffc974d63ca0 x28: ffffffc974d60000
[ 39.969377] x27: ffffff80088a2000 x26: 0000000000000040
[ 39.974642] x25: 0000000000000123 x24: ffffffc976a48918
[ 39.979907] x23: ffffffc974d63eb8 x22: ffffff8008db7550
[ 39.985171] x21: 000000000000000d x20: ffffffc9763e9b50
[ 39.990435] x19: ffffffc9763f20a0 x18: 0000000000000010
[ 39.995699] x17: 0000007f99c18018 x16: ffffff80080c0580
[ 40.000964] x15: ffffff8008bb7000 x14: 000137c100013798
[ 40.006228] x13: ffffffffff000000 x12: ffffffffffffffff
[ 40.011492] x11: 0000000000000018 x10: 000000000000000d
[ 40.016757] x9 : 0000000040000000 x8 : 0000000000210d00
[ 40.022021] x7 : 00000049771bc000 x6 : 00000000001f17ed
[ 40.027286] x5 : ffffff80084c4208 x4 : 0000000000000080
[ 40.032551] x3 : ffffffc975ea9800 x2 : ffffffbf25d7aa50
[ 40.037815] x1 : 0000000000000000 x0 : 0000000000000080
[ 40.043078]
[ 40.044549] ---[ end trace 35c1e743d6e6c035 ]---
[ 40.051537] Exception stack(0xffffffc974d63ad0 to 0xffffffc974d63c00)
[ 40.057914] 3ac0: ffffffc9763f20a0
0000008000000000
[ 40.065668] 3ae0: ffffffc974d63ca0 ffffff80080948f8 ffffffbf25d7aa40
ffffffc975ea9800
[ 40.073421] 3b00: ffffffc974d60000 000000000002fc80 ffffffc976801e00
ffffffc974d60000
[ 40.081175] 3b20: ffffff80084c4208 0000000000000040 ffffff80088a2000
ffffffc974d60000
[ 40.088928] 3b40: ffffff80084caf78 ffffffc974d60000 ffffffc974d60000
ffffffc974d60000
[ 40.096682] 3b60: ffffffc975ea9800 ffffffc974d60000 0000000000000080
0000000000000000
[ 40.104435] 3b80: ffffffbf25d7aa50 ffffffc975ea9800 0000000000000080
ffffff80084c4208
[ 40.112188] 3ba0: 00000000001f17ed 00000049771bc000 0000000000210d00
0000000040000000
[ 40.119941] 3bc0: 000000000000000d 0000000000000018 ffffffffffffffff
ffffffffff000000
[ 40.127695] 3be0: 000137c100013798 ffffff8008bb7000 ffffff80080c0580
0000007f99c18018
[ 40.135450] [<ffffff80080948f8>] arch_teardown_dma_ops+0x48/0x68
[ 40.141400] [<ffffff8008764a14>] of_dma_deconfigure+0xc/0x18
[ 40.147005] [<ffffff8008552804>] dma_deconfigure+0xc/0x18
[ 40.152353] [<ffffff800853ba10>] __device_release_driver+0x88/0x120
[ 40.158560] [<ffffff800853bacc>] device_release_driver+0x24/0x38
[ 40.164507] [<ffffff800853a868>] unbind_store+0xe8/0x110
[ 40.169767] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
[ 40.175113] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
[ 40.180458] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
[ 40.186063] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
[ 40.191237] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
[ 40.196239] [<ffffff80081c7274>] SyS_write+0x44/0xa0
[ 40.201155] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
[ 40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
[ 40.212382] vfio-pci: probe of 0000:08:00.0 failed with error -22
[ 40.228075] ------------[ cut here ]------------
[ 40.235263] WARNING: CPU: 1 PID: 174 at ./include/linux/kref.h:46
kobject_get+0x64/0x88
[ 40.246201]
[ 40.247673] CPU: 1 PID: 174 Comm: vfio Tainted: G W
4.9.0-rc2+ #1249
[ 40.255076] Hardware name: ARM Juno development board (r1) (DT)
[ 40.260932] task: ffffffc975ee9900 task.stack: ffffffc974d60000
[ 40.266787] PC is at kobject_get+0x64/0x88
[ 40.270840] LR is at iommu_bus_notifier+0x40/0x110
[ 40.275577] pc : [<ffffff800834d20c>] lr : [<ffffff80084c3fd0>]
pstate: 80000145
[ 40.282894] sp : ffffffc974d63c00
[ 40.286169] x29: ffffffc974d63c00 x28: ffffffc974d60000
[ 40.291431] x27: ffffff80088a2000 x26: 0000000000000040
[ 40.296692] x25: 0000000000000123 x24: ffffffc974c8f418
[ 40.301953] x23: 0000000000000006 x22: ffffffc9763f10a0
[ 40.307214] x21: ffffffc9763e9a00 x20: ffffffc9763f10a0
[ 40.312474] x19: ffffffc9763ebc80 x18: 0000007fd65069e0
[ 40.317734] x17: 0000007f8d0ae3c0 x16: ffffff80081c7230
[ 40.322995] x15: 0000007f8d136588 x14: ffffffffffffffff
[ 40.328255] x13: 0000000000000004 x12: 0000000000000030
[ 40.333515] x11: 0000000000000030 x10: 0101010101010101
[ 40.338775] x9 : feff716475687163 x8 : 7f7f7f7f7f7f7f7f
[ 40.344035] x7 : feff716475687163 x6 : ffffffc976abf400
[ 40.349295] x5 : ffffffc976abf400 x4 : 0000000000000000
[ 40.354555] x3 : ffffff80084c3f90 x2 : ffffffc9763ebcb8
[ 40.359814] x1 : 0000000000000001 x0 : ffffff8008d4f000
[ 40.365074]
[ 40.366542] ---[ end trace 35c1e743d6e6c036 ]---
[ 40.373523] Exception stack(0xffffffc974d63a30 to 0xffffffc974d63b60)
[ 40.379895] 3a20: ffffffc9763ebc80
0000008000000000
[ 40.387643] 3a40: ffffffc974d63c00 ffffff800834d20c ffffffc976812400
ffffff8008237d94
[ 40.395391] 3a60: ffffffbf25d78940 ffffffc974d60000 ffffffc975e259d8
0000000000005b81
[ 40.403139] 3a80: ffffffc974d60000 ffffff8008d4b31f ffffff8008b0f000
ffffffc976811c80
[ 40.410887] 3aa0: ffffffc974d60000 ffffffc974d60000 ffffffc974d60000
ffffff8008237000
[ 40.418634] 3ac0: ffffffc975e259d8 ffffff8008b1b9a8 ffffff8008d4f000
0000000000000001
[ 40.426382] 3ae0: ffffffc9763ebcb8 ffffff80084c3f90 0000000000000000
ffffffc976abf400
[ 40.434130] 3b00: ffffffc976abf400 feff716475687163 7f7f7f7f7f7f7f7f
feff716475687163
[ 40.441877] 3b20: 0101010101010101 0000000000000030 0000000000000030
0000000000000004
[ 40.449625] 3b40: ffffffffffffffff 0000007f8d136588 ffffff80081c7230
0000007f8d0ae3c0
[ 40.457372] [<ffffff800834d20c>] kobject_get+0x64/0x88
[ 40.462455] [<ffffff80084c3fd0>] iommu_bus_notifier+0x40/0x110
[ 40.468227] [<ffffff80080da288>] notifier_call_chain+0x50/0x90
[ 40.473997] [<ffffff80080da694>] __blocking_notifier_call_chain+0x4c/0x90
[ 40.480713] [<ffffff80080da6ec>] blocking_notifier_call_chain+0x14/0x20
[ 40.487259] [<ffffff800853b9e4>] __device_release_driver+0x5c/0x120
[ 40.493460] [<ffffff800853bacc>] device_release_driver+0x24/0x38
[ 40.499402] [<ffffff800853a868>] unbind_store+0xe8/0x110
[ 40.504656] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
[ 40.509997] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
[ 40.515337] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
[ 40.520936] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
[ 40.526104] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
[ 40.531100] [<ffffff80081c7274>] SyS_write+0x44/0xa0
[ 40.536011] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
[ 40.541324] ata1.00: disabled
[ 40.544878] sd 0:0:0:0: [sda] Synchronizing SCSI cache
hostbyte=0x04 driverbyte=0x00
[ 40.558871] sd 0:0:0:0: [sda] Stopping disk
hostbyte=0x04 driverbyte=0x00
[ 40.586990] Unable to handle kernel paging request at virtual address
0002003e
[ 40.594702] pgd = ffffffc974c80000
[ 40.598165] [0002003e] *pgd=00000009f5102003[ 40.602241] ,
*pud=00000009f5102003
, *pmd=0000000000000000[ 40.607694]
[ 40.609171] Internal error: Oops: 96000006 [#1] PREEMPT SMP
[ 40.617712] CPU: 3 PID: 174 Comm: vfio Tainted: G W
4.9.0-rc2+ #1249
[ 40.625118] Hardware name: ARM Juno development board (r1) (DT)
[ 40.630977] task: ffffffc975ee9900 task.stack: ffffffc974d60000
[ 40.636841] PC is at kobject_get+0x14/0x88
[ 40.640897] LR is at iommu_get_domain_for_dev+0x1c/0x48
[ 40.646068] pc : [<ffffff800834d1bc>] lr : [<ffffff80084c3dec>]
pstate: 60000145
[ 40.653387] sp : ffffffc974d63c60
[ 40.656664] x29: ffffffc974d63c60 x28: ffffffc974d60000
[ 40.661928] x27: ffffff80088a2000 x26: 0000000000000040
[ 40.667193] x25: 0000000000000123 x24: ffffffc974c8f418
[ 40.672457] x23: ffffffc974d63eb8 x22: ffffff8008dab568
[ 40.677720] x21: 000000000000000d x20: ffffff8008dab568
[ 40.682984] x19: 0000000000020002 x18: 0000000000000000
[ 40.688246] x17: 0000000000000007 x16: 0000000000000001
[ 40.693509] x15: ffffffc974cd091c x14: ffffffffffffffff
[ 40.698773] x13: ffffffc974cd01cd x12: 0000000000000030
[ 40.704036] x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f
[ 40.709300] x9 : 0000000040000000 x8 : 0000000000210d00
[ 40.714563] x7 : ffffffc975f95018 x6 : 0000000000000000
[ 40.719826] x5 : 0000000000000000 x4 : ffffffc9763f1210
[ 40.725088] x3 : 0000000000000000 x2 : ffffffc9763f10e8
[ 40.730351] x1 : 0000000000000000 x0 : 0000000000020002
[ 40.735613]
[ 40.737085] Process vfio (pid: 174, stack limit = 0xffffffc974d60020)
[ 40.743460] Stack: (0xffffffc974d63c60 to 0xffffffc974d64000)
[ 40.749150] 3c60: ffffffc974d63c80 ffffff80084c3dec ffffffc9763e9a00
ffffff80085377a4
[ 40.756904] 3c80: ffffffc974d63ca0 ffffff80080948c4 ffffffc9763f10a0
ffffff8008dab568
[ 40.764658] 3ca0: ffffffc974d63cc0 ffffff8008764a14 ffffffc9763f10a0
ffffff8008dab568
[ 40.772411] 3cc0: ffffffc974d63cd0 ffffff8008552804 ffffffc974d63ce0
ffffff800853ba10
[ 40.780165] 3ce0: ffffffc974d63d00 ffffff800853bacc ffffffc9763f1100
ffffffc9763f10a0
[ 40.787918] 3d00: ffffffc974d63d20 ffffff800853a868 ffffff8008d68f18
ffffffc9763f10a0
[ 40.795672] 3d20: ffffffc974d63d50 ffffff8008539c70 000000000000000d
ffffffc974c8f400
[ 40.803425] 3d40: ffffffc9757d5880 0000000000000000 ffffffc974d63d60
ffffff800823ab18
[ 40.811178] 3d60: ffffffc974d63d70 ffffff8008239ea8 ffffffc974d63dc0
ffffff80081c507c
[ 40.818931] 3d80: 000000000000000d 0000000000000000 ffffffc974c8f100
ffffffc974d63eb8
[ 40.826684] 3da0: 000000001285f6a0 0000000000000015 0000000000000123
ffffff80080bf6ac
[ 40.834437] 3dc0: ffffffc974d63e40 ffffff80081c5e80 000000000000000d
0000000000000000
[ 40.842190] 3de0: ffffffc974d63e30 ffffff80080c087c ffffffc974d63e20
ffffff80081c5c0c
[ 40.849943] 3e00: ffffffc974c8f100 0000000000000001 ffffffc974c8f100
ffffffc974d63eb8
[ 40.857696] 3e20: ffffffc974d63e40 ffffff80081c5f48 000000000000000d
ffffffc974c8f100
[ 40.865450] 3e40: ffffffc974d63e80 ffffff80081c7274 ffffffc974c8f100
ffffffc974c8f100
[ 40.873203] 3e60: 000000001285f6a0 000000000000000d 0000000060000000
0000000000000000
[ 40.880956] 3e80: 0000000000000000 ffffff8008082ef0 0000000000000000
0000000000000001
[ 40.888709] 3ea0: ffffffffffffffff 0000007f8d0ae3dc 0000000000000000
0000000000000000
[ 40.896461] 3ec0: 0000000000000001 000000001285f6a0 000000000000000d
0000000000000000
[ 40.904215] 3ee0: ae2e2e2e3f464b49 0000000000000000 000000001285f6b0
39322f392f2f2f2f
[ 40.911968] 3f00: 0000000000000040 fefefeff2f2d2f2f 7f7f7f7f7f7f7f7f
0101010101010101
[ 40.919721] 3f20: 0000000000000002 0000000000000004 ffffffffffffffff
0000007f8d136588
[ 40.927474] 3f40: 0000000000000000 0000007f8d0ae3c0 0000007fd65069e0
00000000004ee000
[ 40.935226] 3f60: 0000000000000001 000000001285f6a0 000000000000000d
0000000000000001
[ 40.942980] 3f80: 0000000000000020 000000001285eed8 00000000004ba158
0000000000000000
[ 40.950732] 3fa0: 0000000000000000 0000007fd6507f30 000000000040e74c
0000007fd6507130
[ 40.958485] 3fc0: 0000007f8d0ae3dc 0000000060000000 0000000000000001
0000000000000040
[ 40.966238] 3fe0: 0000000000000000 0000000000000000 0000002000103a00
4000000010000000
[ 40.976405] Exception stack(0xffffffc974d63a90 to 0xffffffc974d63bc0)
[ 40.982780] 3a80: 0000000000020002
0000008000000000
[ 40.990533] 3aa0: ffffffc974d63c60 ffffff800834d1bc ffffffc974d63ae0
ffffff80085377a4
[ 40.998287] 3ac0: ffffffc974d63b10 ffffff8008537424 ffffffc975e3ac28
ffffffc975e3ac38
[ 41.006041] 3ae0: ffffffc974d63b30 ffffff80081737cc ffffffc975e3ac38
ffffff8008da62c0
[ 41.013794] 3b00: ffffffc975e98100 ffffff80085401b0 0000000000000001
ffffff8008540a08
[ 41.021547] 3b20: 00000000000036b8 0000000000000040 0000000000020002
0000000000000000
[ 41.029300] 3b40: ffffffc9763f10e8 0000000000000000 ffffffc9763f1210
0000000000000000
[ 41.037053] 3b60: 0000000000000000 ffffffc975f95018 0000000000210d00
0000000040000000
[ 41.044805] 3b80: 7f7f7f7f7f7f7f7f 0101010101010101 0000000000000030
ffffffc974cd01cd
[ 41.052558] 3ba0: ffffffffffffffff ffffffc974cd091c 0000000000000001
0000000000000007
[ 41.060311] [<ffffff800834d1bc>] kobject_get+0x14/0x88
[ 41.065398] [<ffffff80084c3dec>] iommu_get_domain_for_dev+0x1c/0x48
[ 41.071607] [<ffffff80080948c4>] arch_teardown_dma_ops+0x14/0x68
[ 41.077556] [<ffffff8008764a14>] of_dma_deconfigure+0xc/0x18
[ 41.083161] [<ffffff8008552804>] dma_deconfigure+0xc/0x18
[ 41.088509] [<ffffff800853ba10>] __device_release_driver+0x88/0x120
[ 41.094715] [<ffffff800853bacc>] device_release_driver+0x24/0x38
[ 41.100663] [<ffffff800853a868>] unbind_store+0xe8/0x110
[ 41.105922] [<ffffff8008539c70>] drv_attr_store+0x20/0x30
[ 41.111268] [<ffffff800823ab18>] sysfs_kf_write+0x48/0x58
[ 41.116612] [<ffffff8008239ea8>] kernfs_fop_write+0xb0/0x1d8
[ 41.122216] [<ffffff80081c507c>] __vfs_write+0x1c/0x100
[ 41.127390] [<ffffff80081c5e80>] vfs_write+0xa0/0x1b8
[ 41.132391] [<ffffff80081c7274>] SyS_write+0x44/0xa0
[ 41.137307] [<ffffff8008082ef0>] el0_svc_naked+0x24/0x28
[ 41.142567] Code: 910003fd f9000bf3 aa0003f3 b4000180 (3940f000)
[ 41.148667] ---[ end trace 35c1e743d6e6c037 ]---
Segmentation fault
/ #
_______________________________________________
linux-arm-kernel mailing list
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

_______________________________________________
linux-arm-kernel mailing list
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[Sricharan]

Hi Robin,

Post by Robin Murphy

Post by Robin Murphy

Post by Robin Murphy
[ 39.901592] iommu: Removing device 0000:08:00.0 from group 0

Yikes, on second look, that definitely shouldn't be happening.
Everything below is probably the resulting fallout.

[ 40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
I think the above print which says "failed to setup iommu_ops"
because the call ops->add_device failed in of_pci_iommu_configure
is the reason for the failure, in my case i simply do not get this even with
your scripts. ops->add_device succeeds in the rebind as well. So still
checking what could be happening in your case.

I was looking at your code base from [1].The ops->add_device
callback from of_pci_iommu_configure on the rebind is the
one which is causing the failure. But not able to spot out
from code which point is causing the failure. It would be very helpful
if i can know which is the return value from the add_device callback
or point inside add_device callback which fails in your setup.


[1] git://linux-arm.org/linux-rm iommu/misc

Post by Robin Murphy
Regards,
Sricharan

[Sricharan]

Hi Robin,

Post by Sricharan

Post by Robin Murphy

Post by Robin Murphy
Yikes, on second look, that definitely shouldn't be happening.
Everything below is probably the resulting fallout.

[ 40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
I think the above print which says "failed to setup iommu_ops"
because the call ops->add_device failed in of_pci_iommu_configure
is the reason for the failure, in my case i simply do not get this even with
your scripts. ops->add_device succeeds in the rebind as well. So still
checking what could be happening in your case.

I was looking at your code base from [1].The ops->add_device
callback from of_pci_iommu_configure on the rebind is the
one which is causing the failure. But not able to spot out
from code which point is causing the failure. It would be very helpful
if i can know which is the return value from the add_device callback
or point inside add_device callback which fails in your setup.
[1] git://linux-arm.org/linux-rm iommu/misc

With little more try, i saw an issue where i had an failure
similar to what you reported. The issue happens when multiple
devices fall in to same group due to matching sids. I ended up
doing a fix like below and it would be nice to verify if it is the same
that we are seeing in your setup and if the fix makes a difference ?

From: Sricharan R <***@codeaurora.org>
Date: Fri, 4 Nov 2016 20:28:49 +0530
Subject: [PATCH] iommu/arm-smmu: Fix group's reference counting

iommu_group_get_for_dev which gets called in the add_device
callback, increases the reference count of the iommu_group,
so we do an iommu_group_put after that. iommu_group_get_for_dev
inturn calls device_group callback and in the case of arm-smmu
we call generic_device_group/pci_device_group which takes
care of increasing the group's reference. But when we return
an already existing group(when multiple devices have same group)
the reference is not incremented, resulting in issues when the
remove_device callback for the devices is invoked.
Fixing the same here.

Signed-off-by: Sricharan R <***@codeaurora.org>
---
drivers/iommu/arm-smmu.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index 71ce4b6..a1d0b3c 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -1516,8 +1516,10 @@ static struct iommu_group *arm_smmu_device_group(struct device *dev)
group = smmu->s2crs[idx].group;
}

- if (group)
+ if (group) {
+ iommu_group_get_by_id(iommu_group_id(group));
return group;
+ }

if (dev_is_pci(dev))
group = pci_device_group(dev);

[Will Deacon]

Post by Sricharan

Post by Sricharan

Post by Robin Murphy

Post by Robin Murphy
Yikes, on second look, that definitely shouldn't be happening.
Everything below is probably the resulting fallout.

[ 40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
I think the above print which says "failed to setup iommu_ops"
because the call ops->add_device failed in of_pci_iommu_configure
is the reason for the failure, in my case i simply do not get this even with
your scripts. ops->add_device succeeds in the rebind as well. So still
checking what could be happening in your case.

I was looking at your code base from [1].The ops->add_device
callback from of_pci_iommu_configure on the rebind is the
one which is causing the failure. But not able to spot out
from code which point is causing the failure. It would be very helpful
if i can know which is the return value from the add_device callback
or point inside add_device callback which fails in your setup.
[1] git://linux-arm.org/linux-rm iommu/misc

So this also applies to mainline.

Post by Sricharan
With little more try, i saw an issue where i had an failure
similar to what you reported. The issue happens when multiple
devices fall in to same group due to matching sids. I ended up
doing a fix like below and it would be nice to verify if it is the same
that we are seeing in your setup and if the fix makes a difference ?
Date: Fri, 4 Nov 2016 20:28:49 +0530
Subject: [PATCH] iommu/arm-smmu: Fix group's reference counting
iommu_group_get_for_dev which gets called in the add_device
callback, increases the reference count of the iommu_group,
so we do an iommu_group_put after that. iommu_group_get_for_dev
inturn calls device_group callback and in the case of arm-smmu
we call generic_device_group/pci_device_group which takes
care of increasing the group's reference. But when we return
an already existing group(when multiple devices have same group)
the reference is not incremented, resulting in issues when the
remove_device callback for the devices is invoked.
Fixing the same here.
---
drivers/iommu/arm-smmu.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index 71ce4b6..a1d0b3c 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -1516,8 +1516,10 @@ static struct iommu_group *arm_smmu_device_group(struct device *dev)
group = smmu->s2crs[idx].group;
}
- if (group)
+ if (group) {
+ iommu_group_get_by_id(iommu_group_id(group));
return group;
+ }

This is foul :(

I think we should either add a function for incrementing the refcount on a
group, or we should get a handle on the existing aliasing device and get
the group from that. As written, this patch is far too subtle.

Joerg -- do you have any preference?

Will

[Robin Murphy]

Hi Sricharan,

Post by Sricharan
Hi Robin,

Post by Sricharan

Post by Robin Murphy

Post by Robin Murphy
Yikes, on second look, that definitely shouldn't be happening.
Everything below is probably the resulting fallout.

[ 40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
I think the above print which says "failed to setup iommu_ops"
because the call ops->add_device failed in of_pci_iommu_configure
is the reason for the failure, in my case i simply do not get this even with
your scripts. ops->add_device succeeds in the rebind as well. So still
checking what could be happening in your case.

I was looking at your code base from [1].The ops->add_device
callback from of_pci_iommu_configure on the rebind is the
one which is causing the failure. But not able to spot out
from code which point is causing the failure. It would be very helpful
if i can know which is the return value from the add_device callback
or point inside add_device callback which fails in your setup.
[1] git://linux-arm.org/linux-rm iommu/misc

With little more try, i saw an issue where i had an failure
similar to what you reported. The issue happens when multiple
devices fall in to same group due to matching sids. I ended up
doing a fix like below and it would be nice to verify if it is the same
that we are seeing in your setup and if the fix makes a difference ?
Date: Fri, 4 Nov 2016 20:28:49 +0530
Subject: [PATCH] iommu/arm-smmu: Fix group's reference counting
iommu_group_get_for_dev which gets called in the add_device
callback, increases the reference count of the iommu_group,
so we do an iommu_group_put after that. iommu_group_get_for_dev
inturn calls device_group callback and in the case of arm-smmu
we call generic_device_group/pci_device_group which takes
care of increasing the group's reference. But when we return
an already existing group(when multiple devices have same group)
the reference is not incremented, resulting in issues when the
remove_device callback for the devices is invoked.
Fixing the same here.

Bah, yes, this does look like my fault - after flip-flopping between
about 3 different ways to keep refcounts for the S2CR entries, none of
which would quite work, I ripped it all out but apparently still got
things wrong, oh well. Thanks for figuring it out.

On the probe-deferral angle, whilst it's useful to have uncovered this
bug, I don't think we should actually be calling remove_device() from
DMA teardown. I think it's preferable from a user perspective if group
numbering remains stable, rather than changing depending on the order in
which they unbind/rebind VFIO drivers. I'm really keen to try and get
this in shape for 4.10, so I've taken the liberty of hacking up my own
branch (iommu/defer) based on v3 - would you mind taking a look at the
two "iommu/of:" commits to see what you think? (Ignore the PCI changes
to your later patches - that was an experiment which didn't really work out)

Post by Sricharan
---
drivers/iommu/arm-smmu.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index 71ce4b6..a1d0b3c 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -1516,8 +1516,10 @@ static struct iommu_group *arm_smmu_device_group(struct device *dev)
group = smmu->s2crs[idx].group;
}
- if (group)
+ if (group) {
+ iommu_group_get_by_id(iommu_group_id(group));
return group;

This might as well just be inline, i.e.:

return iommu_group_get_by_id(iommu_group_id(group));

It's a shame we have to go all round the houses when we have the group
right there, but this is probably the most expedient fix. I guess we can
extend the API with some sort of iommu_group_get(group) overload in
future if we really want to.

Robin.

Post by Sricharan
+ }
if (dev_is_pci(dev))
group = pci_device_group(dev);

[Sricharan]

Hi Robin,

Post by Robin Murphy

Post by Sricharan
Hi Robin,

Post by Sricharan

Post by Robin Murphy

Post by Robin Murphy
Yikes, on second look, that definitely shouldn't be happening.
Everything below is probably the resulting fallout.

[ 40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
I think the above print which says "failed to setup iommu_ops"
because the call ops->add_device failed in of_pci_iommu_configure
is the reason for the failure, in my case i simply do not get this even with
your scripts. ops->add_device succeeds in the rebind as well. So still
checking what could be happening in your case.

I was looking at your code base from [1].The ops->add_device
callback from of_pci_iommu_configure on the rebind is the
one which is causing the failure. But not able to spot out
from code which point is causing the failure. It would be very helpful
if i can know which is the return value from the add_device callback
or point inside add_device callback which fails in your setup.
[1] git://linux-arm.org/linux-rm iommu/misc

With little more try, i saw an issue where i had an failure
similar to what you reported. The issue happens when multiple
devices fall in to same group due to matching sids. I ended up
doing a fix like below and it would be nice to verify if it is the same
that we are seeing in your setup and if the fix makes a difference ?
Date: Fri, 4 Nov 2016 20:28:49 +0530
Subject: [PATCH] iommu/arm-smmu: Fix group's reference counting
iommu_group_get_for_dev which gets called in the add_device
callback, increases the reference count of the iommu_group,
so we do an iommu_group_put after that. iommu_group_get_for_dev
inturn calls device_group callback and in the case of arm-smmu
we call generic_device_group/pci_device_group which takes
care of increasing the group's reference. But when we return
an already existing group(when multiple devices have same group)
the reference is not incremented, resulting in issues when the
remove_device callback for the devices is invoked.
Fixing the same here.

Bah, yes, this does look like my fault - after flip-flopping between
about 3 different ways to keep refcounts for the S2CR entries, none of
which would quite work, I ripped it all out but apparently still got
things wrong, oh well. Thanks for figuring it out.
On the probe-deferral angle, whilst it's useful to have uncovered this
bug, I don't think we should actually be calling remove_device() from
DMA teardown. I think it's preferable from a user perspective if group
numbering remains stable, rather than changing depending on the order in
which they unbind/rebind VFIO drivers. I'm really keen to try and get
this in shape for 4.10, so I've taken the liberty of hacking up my own
branch (iommu/defer) based on v3 - would you mind taking a look at the
two "iommu/of:" commits to see what you think? (Ignore the PCI changes
to your later patches - that was an experiment which didn't really work out)

Ok, will take a look at this now and respond more on this.

Post by Robin Murphy

Post by Sricharan
---
drivers/iommu/arm-smmu.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index 71ce4b6..a1d0b3c 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -1516,8 +1516,10 @@ static struct iommu_group *arm_smmu_device_group(struct device *dev)
group = smmu->s2crs[idx].group;
}
- if (group)
+ if (group) {
+ iommu_group_get_by_id(iommu_group_id(group));
return group;

return iommu_group_get_by_id(iommu_group_id(group));
It's a shame we have to go all round the houses when we have the group
right there, but this is probably the most expedient fix. I guess we can
extend the API with some sort of iommu_group_get(group) overload in
future if we really want to.

ok, i can send this fix separately then. Otherwise, Will was suggesting on the
other thread that there should probably be a separate API to increment
the group refcount or get the group from the existing aliasing device.
As per me adding the api, looks like another option or post the above ?

Regards,
Sricharan

[Will Deacon]

Post by Sricharan

Post by Robin Murphy

Post by Sricharan
diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index 71ce4b6..a1d0b3c 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -1516,8 +1516,10 @@ static struct iommu_group *arm_smmu_device_group(struct device *dev)
group = smmu->s2crs[idx].group;
}
- if (group)
+ if (group) {
+ iommu_group_get_by_id(iommu_group_id(group));
return group;

return iommu_group_get_by_id(iommu_group_id(group));
It's a shame we have to go all round the houses when we have the group
right there, but this is probably the most expedient fix. I guess we can
extend the API with some sort of iommu_group_get(group) overload in
future if we really want to.

ok, i can send this fix separately then. Otherwise, Will was suggesting on the
other thread that there should probably be a separate API to increment
the group refcount or get the group from the existing aliasing device.
As per me adding the api, looks like another option or post the above ?

I think adding a new function to the API is the way to go -- having code
like what you propose above littered around the drivers is hard to read and
pretty error-prone, since it looks like a NOP to people who aren't already
thinking about ref counts.

Will

[Sricharan]

Hi Robin,

Post by Sricharan
Hi Robin,

Post by Robin Murphy

Post by Sricharan
Hi Robin,

Post by Sricharan

Post by Robin Murphy

Post by Robin Murphy
Yikes, on second look, that definitely shouldn't be happening.
Everything below is probably the resulting fallout.

[ 40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
I think the above print which says "failed to setup iommu_ops"
because the call ops->add_device failed in of_pci_iommu_configure
is the reason for the failure, in my case i simply do not get this even with
your scripts. ops->add_device succeeds in the rebind as well. So still
checking what could be happening in your case.

I was looking at your code base from [1].The ops->add_device
callback from of_pci_iommu_configure on the rebind is the
one which is causing the failure. But not able to spot out
from code which point is causing the failure. It would be very helpful
if i can know which is the return value from the add_device callback
or point inside add_device callback which fails in your setup.
[1] git://linux-arm.org/linux-rm iommu/misc

With little more try, i saw an issue where i had an failure
similar to what you reported. The issue happens when multiple
devices fall in to same group due to matching sids. I ended up
doing a fix like below and it would be nice to verify if it is the same
that we are seeing in your setup and if the fix makes a difference ?
Date: Fri, 4 Nov 2016 20:28:49 +0530
Subject: [PATCH] iommu/arm-smmu: Fix group's reference counting
iommu_group_get_for_dev which gets called in the add_device
callback, increases the reference count of the iommu_group,
so we do an iommu_group_put after that. iommu_group_get_for_dev
inturn calls device_group callback and in the case of arm-smmu
we call generic_device_group/pci_device_group which takes
care of increasing the group's reference. But when we return
an already existing group(when multiple devices have same group)
the reference is not incremented, resulting in issues when the
remove_device callback for the devices is invoked.
Fixing the same here.

Bah, yes, this does look like my fault - after flip-flopping between
about 3 different ways to keep refcounts for the S2CR entries, none of
which would quite work, I ripped it all out but apparently still got
things wrong, oh well. Thanks for figuring it out.
On the probe-deferral angle, whilst it's useful to have uncovered this
bug, I don't think we should actually be calling remove_device() from
DMA teardown. I think it's preferable from a user perspective if group
numbering remains stable, rather than changing depending on the order in
which they unbind/rebind VFIO drivers. I'm really keen to try and get
this in shape for 4.10, so I've taken the liberty of hacking up my own
branch (iommu/defer) based on v3 - would you mind taking a look at the
two "iommu/of:" commits to see what you think? (Ignore the PCI changes
to your later patches - that was an experiment which didn't really work out)

Ok, will take a look at this now and respond more on this.

Sorry for the delayed response on this. I was OOO for the last few days.
So i tested this branch and it worked fine. I tested it with a pci device
for both normal and deferred probe cases. The of/iommu patches
are the cleanup/preparation patches and it looks fine. One thing is without
calling the remove_device callback, the resources like (smes for exmaple)
and the group association of the device all remain allocated. That does not
feel correct, given that the associated device does not exist. So to
understand that, what happens with VFIO in this case which makes the
group renumbering/rebinding a problem ?

Regards,
Sricharan

[Sricharan]

Hi Robin,

Post by Sricharan
Hi Robin,

Post by Sricharan
Hi Robin,

Post by Robin Murphy

Post by Sricharan
Hi Robin,

Post by Sricharan

Post by Robin Murphy

Post by Robin Murphy
Yikes, on second look, that definitely shouldn't be happening.
Everything below is probably the resulting fallout.

[ 40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
I think the above print which says "failed to setup iommu_ops"
because the call ops->add_device failed in of_pci_iommu_configure
is the reason for the failure, in my case i simply do not get this even with
your scripts. ops->add_device succeeds in the rebind as well. So still
checking what could be happening in your case.

I was looking at your code base from [1].The ops->add_device
callback from of_pci_iommu_configure on the rebind is the
one which is causing the failure. But not able to spot out
from code which point is causing the failure. It would be very helpful
if i can know which is the return value from the add_device callback
or point inside add_device callback which fails in your setup.
[1] git://linux-arm.org/linux-rm iommu/misc

With little more try, i saw an issue where i had an failure
similar to what you reported. The issue happens when multiple
devices fall in to same group due to matching sids. I ended up
doing a fix like below and it would be nice to verify if it is the same
that we are seeing in your setup and if the fix makes a difference ?
Date: Fri, 4 Nov 2016 20:28:49 +0530
Subject: [PATCH] iommu/arm-smmu: Fix group's reference counting
iommu_group_get_for_dev which gets called in the add_device
callback, increases the reference count of the iommu_group,
so we do an iommu_group_put after that. iommu_group_get_for_dev
inturn calls device_group callback and in the case of arm-smmu
we call generic_device_group/pci_device_group which takes
care of increasing the group's reference. But when we return
an already existing group(when multiple devices have same group)
the reference is not incremented, resulting in issues when the
remove_device callback for the devices is invoked.
Fixing the same here.

Bah, yes, this does look like my fault - after flip-flopping between
about 3 different ways to keep refcounts for the S2CR entries, none of
which would quite work, I ripped it all out but apparently still got
things wrong, oh well. Thanks for figuring it out.
On the probe-deferral angle, whilst it's useful to have uncovered this
bug, I don't think we should actually be calling remove_device() from
DMA teardown. I think it's preferable from a user perspective if group
numbering remains stable, rather than changing depending on the order in
which they unbind/rebind VFIO drivers. I'm really keen to try and get
this in shape for 4.10, so I've taken the liberty of hacking up my own
branch (iommu/defer) based on v3 - would you mind taking a look at the
two "iommu/of:" commits to see what you think? (Ignore the PCI changes
to your later patches - that was an experiment which didn't really work out)

Ok, will take a look at this now and respond more on this.

Sorry for the delayed response on this. I was OOO for the last few days.
So i tested this branch and it worked fine. I tested it with a pci device
for both normal and deferred probe cases. The of/iommu patches
are the cleanup/preparation patches and it looks fine. One thing is without
calling the remove_device callback, the resources like (smes for exmaple)
and the group association of the device all remain allocated. That does not
feel correct, given that the associated device does not exist. So to
understand that, what happens with VFIO in this case which makes the
group renumbering/rebinding a problem ?

Would it be ok if i post a V4 based on your branch above ?

Regards,
Sricharan

[Robin Murphy]

Post by Sricharan
Hi Robin,

Post by Sricharan
Hi Robin,

Post by Sricharan
Hi Robin,

Post by Robin Murphy

Post by Sricharan
Hi Robin,

Post by Sricharan

Post by Robin Murphy

Post by Robin Murphy
Yikes, on second look, that definitely shouldn't be happening.
Everything below is probably the resulting fallout.

[ 40.206703] vfio-pci 0000:08:00.0: Failed to setup iommu ops
I think the above print which says "failed to setup iommu_ops"
because the call ops->add_device failed in of_pci_iommu_configure
is the reason for the failure, in my case i simply do not get this even with
your scripts. ops->add_device succeeds in the rebind as well. So still
checking what could be happening in your case.

I was looking at your code base from [1].The ops->add_device
callback from of_pci_iommu_configure on the rebind is the
one which is causing the failure. But not able to spot out
from code which point is causing the failure. It would be very helpful
if i can know which is the return value from the add_device callback
or point inside add_device callback which fails in your setup.
[1] git://linux-arm.org/linux-rm iommu/misc

With little more try, i saw an issue where i had an failure
similar to what you reported. The issue happens when multiple
devices fall in to same group due to matching sids. I ended up
doing a fix like below and it would be nice to verify if it is the same
that we are seeing in your setup and if the fix makes a difference ?
Date: Fri, 4 Nov 2016 20:28:49 +0530
Subject: [PATCH] iommu/arm-smmu: Fix group's reference counting
iommu_group_get_for_dev which gets called in the add_device
callback, increases the reference count of the iommu_group,
so we do an iommu_group_put after that. iommu_group_get_for_dev
inturn calls device_group callback and in the case of arm-smmu
we call generic_device_group/pci_device_group which takes
care of increasing the group's reference. But when we return
an already existing group(when multiple devices have same group)
the reference is not incremented, resulting in issues when the
remove_device callback for the devices is invoked.
Fixing the same here.

Bah, yes, this does look like my fault - after flip-flopping between
about 3 different ways to keep refcounts for the S2CR entries, none of
which would quite work, I ripped it all out but apparently still got
things wrong, oh well. Thanks for figuring it out.
On the probe-deferral angle, whilst it's useful to have uncovered this
bug, I don't think we should actually be calling remove_device() from
DMA teardown. I think it's preferable from a user perspective if group
numbering remains stable, rather than changing depending on the order in
which they unbind/rebind VFIO drivers. I'm really keen to try and get
this in shape for 4.10, so I've taken the liberty of hacking up my own
branch (iommu/defer) based on v3 - would you mind taking a look at the
two "iommu/of:" commits to see what you think? (Ignore the PCI changes
to your later patches - that was an experiment which didn't really work out)

Ok, will take a look at this now and respond more on this.

Sorry for the delayed response on this. I was OOO for the last few days.
So i tested this branch and it worked fine. I tested it with a pci device
for both normal and deferred probe cases. The of/iommu patches
are the cleanup/preparation patches and it looks fine. One thing is without
calling the remove_device callback, the resources like (smes for exmaple)
and the group association of the device all remain allocated. That does not
feel correct, given that the associated device does not exist. So to
understand that, what happens with VFIO in this case which makes the
group renumbering/rebinding a problem ?

Would it be ok if i post a V4 based on your branch above ?

Sure, as long as none of the hacks slip through :) - I've just pushed
out a mild rework based on Lorenzo's v9, which I hope shouldn't break
anything for you.

Having thought a bit more about the add/remove thing, I'm inclined to
agree that the group numbering itself may not be that big an issue in
practice - sure, it could break my little script, but it looks like QEMU
and such work with the device ID rather than the group number directly,
so might not even notice. However, the fact remains that the callbacks
are intended to handle a device being added to/removed from its bus, and
will continue to do so on other platforms, so I don't like the idea of
introducing needlessly different behaviour. If you unbind a driver, the
stream IDs and everything don't stop existing at the hardware level; the
struct device to which the in-kernel data belongs still exists and
doesn't stop being associated with its bus. There's no good reason for
freeing SMEs that we'll only reallocate again (inadequately-specced
hardware with not enough SMRs/contexts is not a *good* reason), and
there are also some strong arguments against letting any stream IDs the
kernel knows about go back to bypass after a driver has been bound - by
keeping groups around as expected that's something we can implement
quite easily without having to completely lock down bypass for stream
IDs the kernel *doesn't* know about.

Robin.

Post by Sricharan
Regards,
Sricharan

[Sricharan]

Hi Robin,

<snip..>

Post by Robin Murphy

Post by Sricharan

Post by Sricharan

Post by Sricharan

Post by Robin Murphy

Post by Sricharan
iommu_group_get_for_dev which gets called in the add_device
callback, increases the reference count of the iommu_group,
so we do an iommu_group_put after that. iommu_group_get_for_dev
inturn calls device_group callback and in the case of arm-smmu
we call generic_device_group/pci_device_group which takes
care of increasing the group's reference. But when we return
an already existing group(when multiple devices have same group)
the reference is not incremented, resulting in issues when the
remove_device callback for the devices is invoked.
Fixing the same here.

Bah, yes, this does look like my fault - after flip-flopping between
about 3 different ways to keep refcounts for the S2CR entries, none of
which would quite work, I ripped it all out but apparently still got
things wrong, oh well. Thanks for figuring it out.
On the probe-deferral angle, whilst it's useful to have uncovered this
bug, I don't think we should actually be calling remove_device() from
DMA teardown. I think it's preferable from a user perspective if group
numbering remains stable, rather than changing depending on the order in
which they unbind/rebind VFIO drivers. I'm really keen to try and get
this in shape for 4.10, so I've taken the liberty of hacking up my own
branch (iommu/defer) based on v3 - would you mind taking a look at the
two "iommu/of:" commits to see what you think? (Ignore the PCI changes
to your later patches - that was an experiment which didn't really work out)

Ok, will take a look at this now and respond more on this.

Sorry for the delayed response on this. I was OOO for the last few days.
So i tested this branch and it worked fine. I tested it with a pci device
for both normal and deferred probe cases. The of/iommu patches
are the cleanup/preparation patches and it looks fine. One thing is without
calling the remove_device callback, the resources like (smes for exmaple)
and the group association of the device all remain allocated. That does not
feel correct, given that the associated device does not exist. So to
understand that, what happens with VFIO in this case which makes the
group renumbering/rebinding a problem ?

Would it be ok if i post a V4 based on your branch above ?

Sure, as long as none of the hacks slip through :) - I've just pushed
out a mild rework based on Lorenzo's v9, which I hope shouldn't break
anything for you.

Ok sure, i will test and just the post out the stuff from your branch then
mostly by tomorrow.

Post by Robin Murphy
Having thought a bit more about the add/remove thing, I'm inclined to
agree that the group numbering itself may not be that big an issue in
practice - sure, it could break my little script, but it looks like QEMU
and such work with the device ID rather than the group number directly,
so might not even notice. However, the fact remains that the callbacks
are intended to handle a device being added to/removed from its bus, and
will continue to do so on other platforms, so I don't like the idea of
introducing needlessly different behaviour. If you unbind a driver, the
stream IDs and everything don't stop existing at the hardware level; the
struct device to which the in-kernel data belongs still exists and
doesn't stop being associated with its bus. There's no good reason for
freeing SMEs that we'll only reallocate again (inadequately-specced
hardware with not enough SMRs/contexts is not a *good* reason), and

ok, so SMRs/contexts was the reason i was adding the remove_dev
callback, but if thats not good enough then there was no other
intention.

Post by Robin Murphy
there are also some strong arguments against letting any stream IDs the
kernel knows about go back to bypass after a driver has been bound - by

ok, but not sure why is this so ?

Post by Robin Murphy
keeping groups around as expected that's something we can implement
quite easily without having to completely lock down bypass for stream
IDs the kernel *doesn't* know about.

So do you mean in this case to keep the unbound device's group/context bank
to bypass rather than resetting the streamids ?

Regards,
Sricharan

[Robin Murphy]

Post by Sricharan
Hi Robin,
<snip..>

Post by Robin Murphy

Post by Sricharan

Post by Sricharan

Post by Sricharan

Post by Robin Murphy

Post by Sricharan
iommu_group_get_for_dev which gets called in the add_device
callback, increases the reference count of the iommu_group,
so we do an iommu_group_put after that. iommu_group_get_for_dev
inturn calls device_group callback and in the case of arm-smmu
we call generic_device_group/pci_device_group which takes
care of increasing the group's reference. But when we return
an already existing group(when multiple devices have same group)
the reference is not incremented, resulting in issues when the
remove_device callback for the devices is invoked.
Fixing the same here.

Bah, yes, this does look like my fault - after flip-flopping between
about 3 different ways to keep refcounts for the S2CR entries, none of
which would quite work, I ripped it all out but apparently still got
things wrong, oh well. Thanks for figuring it out.
On the probe-deferral angle, whilst it's useful to have uncovered this
bug, I don't think we should actually be calling remove_device() from
DMA teardown. I think it's preferable from a user perspective if group
numbering remains stable, rather than changing depending on the order in
which they unbind/rebind VFIO drivers. I'm really keen to try and get
this in shape for 4.10, so I've taken the liberty of hacking up my own
branch (iommu/defer) based on v3 - would you mind taking a look at the
two "iommu/of:" commits to see what you think? (Ignore the PCI changes
to your later patches - that was an experiment which didn't really work out)

Ok, will take a look at this now and respond more on this.

Sorry for the delayed response on this. I was OOO for the last few days.
So i tested this branch and it worked fine. I tested it with a pci device
for both normal and deferred probe cases. The of/iommu patches
are the cleanup/preparation patches and it looks fine. One thing is without
calling the remove_device callback, the resources like (smes for exmaple)
and the group association of the device all remain allocated. That does not
feel correct, given that the associated device does not exist. So to
understand that, what happens with VFIO in this case which makes the
group renumbering/rebinding a problem ?

Would it be ok if i post a V4 based on your branch above ?

Sure, as long as none of the hacks slip through :) - I've just pushed
out a mild rework based on Lorenzo's v9, which I hope shouldn't break
anything for you.

Ok sure, i will test and just the post out the stuff from your branch then
mostly by tomorrow.

Cool. We're rather hoping that the ACPI stuff is good to go for 4.10
now, so it's probably worth pulling the rest of that in (beyond the one
patch I picked) to make sure the of_dma_configure/acpi_dma_configure
paths don't inadvertently diverge.

Post by Sricharan

Post by Robin Murphy
Having thought a bit more about the add/remove thing, I'm inclined to
agree that the group numbering itself may not be that big an issue in
practice - sure, it could break my little script, but it looks like QEMU
and such work with the device ID rather than the group number directly,
so might not even notice. However, the fact remains that the callbacks
are intended to handle a device being added to/removed from its bus, and
will continue to do so on other platforms, so I don't like the idea of
introducing needlessly different behaviour. If you unbind a driver, the
stream IDs and everything don't stop existing at the hardware level; the
struct device to which the in-kernel data belongs still exists and
doesn't stop being associated with its bus. There's no good reason for
freeing SMEs that we'll only reallocate again (inadequately-specced
hardware with not enough SMRs/contexts is not a *good* reason), and

ok, so SMRs/contexts was the reason i was adding the remove_dev
callback, but if thats not good enough then there was no other
intention.

Post by Robin Murphy
there are also some strong arguments against letting any stream IDs the
kernel knows about go back to bypass after a driver has been bound - by

ok, but not sure why is this so ?

Any device the kernel is in control of, having bound a driver to it,
definitely should not be doing DMA after that driver is unbound...

Post by Sricharan

Post by Robin Murphy
keeping groups around as expected that's something we can implement
quite easily without having to completely lock down bypass for stream
IDs the kernel *doesn't* know about.

So do you mean in this case to keep the unbound device's group/context bank
to bypass rather than resetting the streamids ?

...which we can easily enforce by keeping the device attached to its
default domain, in which nothing should be mapped by that point (we
could even have a group notifier switch its S2CRs to faulting entries
for extreme paranoia). Freeing the SMRs means those stream IDs would
instead fall back to the default "unmatched" behaviour, which in general
is going to be bypass, and thus allow DMA attacks.

It's harder to disable unmatched bypass in general, because we may have
devices which physically master through the SMMU but want low latency
more than they want translation (at the moment the best we can do is
leave the kernel unaware of those stream IDs), or there could be unknown
devices under control of the firmware or other agents which we would
disrupt by hitting a system-wide switch.

Robin.

Post by Sricharan
Regards,
Sricharan

[Sricharan]

Hi Robin,

Post by Robin Murphy

Post by Sricharan
<snip..>

Post by Robin Murphy

Post by Sricharan

Post by Sricharan

Post by Sricharan

Post by Robin Murphy

Post by Sricharan
iommu_group_get_for_dev which gets called in the add_device
callback, increases the reference count of the iommu_group,
so we do an iommu_group_put after that. iommu_group_get_for_dev
inturn calls device_group callback and in the case of arm-smmu
we call generic_device_group/pci_device_group which takes
care of increasing the group's reference. But when we return
an already existing group(when multiple devices have same group)
the reference is not incremented, resulting in issues when the
remove_device callback for the devices is invoked.
Fixing the same here.

Bah, yes, this does look like my fault - after flip-flopping between
about 3 different ways to keep refcounts for the S2CR entries, none of
which would quite work, I ripped it all out but apparently still got
things wrong, oh well. Thanks for figuring it out.
On the probe-deferral angle, whilst it's useful to have uncovered this
bug, I don't think we should actually be calling remove_device() from
DMA teardown. I think it's preferable from a user perspective if group
numbering remains stable, rather than changing depending on the order in
which they unbind/rebind VFIO drivers. I'm really keen to try and get
this in shape for 4.10, so I've taken the liberty of hacking up my own
branch (iommu/defer) based on v3 - would you mind taking a look at the
two "iommu/of:" commits to see what you think? (Ignore the PCI changes
to your later patches - that was an experiment which didn't really work out)

Ok, will take a look at this now and respond more on this.

Sorry for the delayed response on this. I was OOO for the last few days.
So i tested this branch and it worked fine. I tested it with a pci device
for both normal and deferred probe cases. The of/iommu patches
are the cleanup/preparation patches and it looks fine. One thing is without
calling the remove_device callback, the resources like (smes for exmaple)
and the group association of the device all remain allocated. That does not
feel correct, given that the associated device does not exist. So to
understand that, what happens with VFIO in this case which makes the
group renumbering/rebinding a problem ?

Would it be ok if i post a V4 based on your branch above ?

Sure, as long as none of the hacks slip through :) - I've just pushed
out a mild rework based on Lorenzo's v9, which I hope shouldn't break
anything for you.

Ok sure, i will test and just the post out the stuff from your branch then
mostly by tomorrow.

Cool. We're rather hoping that the ACPI stuff is good to go for 4.10
now, so it's probably worth pulling the rest of that in (beyond the one
patch I picked) to make sure the of_dma_configure/acpi_dma_configure
paths don't inadvertently diverge.

I rebased and was testing your branch with Lorenzo's series. One thing
i am still trying to get right is the acpi_dma_configure call. With your
series dma_configure calls pci_dma/of_dma configure, so i am just adding
acpi_dma_configure call there for non-pci ACPI devices as well. I see that
acpi_dma_configure right now is called from acpi_bind_one and
iort_add_smmu_platform_device, both go through the really_probe function
path, so moving acpi_dma_configure from the above the two functions
to dma_configure. I remember we discussed this on another thread, so
hopefully it is correct. I do not have an platform to test the ACPI though.
I will take some testing help on V4 for this.

Post by Robin Murphy

Post by Sricharan

Post by Robin Murphy
Having thought a bit more about the add/remove thing, I'm inclined to
agree that the group numbering itself may not be that big an issue in
practice - sure, it could break my little script, but it looks like QEMU
and such work with the device ID rather than the group number directly,
so might not even notice. However, the fact remains that the callbacks
are intended to handle a device being added to/removed from its bus, and
will continue to do so on other platforms, so I don't like the idea of
introducing needlessly different behaviour. If you unbind a driver, the
stream IDs and everything don't stop existing at the hardware level; the
struct device to which the in-kernel data belongs still exists and
doesn't stop being associated with its bus. There's no good reason for
freeing SMEs that we'll only reallocate again (inadequately-specced
hardware with not enough SMRs/contexts is not a *good* reason), and

ok, so SMRs/contexts was the reason i was adding the remove_dev
callback, but if thats not good enough then there was no other
intention.

Post by Robin Murphy
there are also some strong arguments against letting any stream IDs the
kernel knows about go back to bypass after a driver has been bound - by

ok, but not sure why is this so ?

Any device the kernel is in control of, having bound a driver to it,
definitely should not be doing DMA after that driver is unbound...

ok.

Post by Robin Murphy

Post by Sricharan

Post by Robin Murphy
keeping groups around as expected that's something we can implement
quite easily without having to completely lock down bypass for stream
IDs the kernel *doesn't* know about.

So do you mean in this case to keep the unbound device's group/context bank
to bypass rather than resetting the streamids ?

...which we can easily enforce by keeping the device attached to its
default domain, in which nothing should be mapped by that point (we
could even have a group notifier switch its S2CRs to faulting entries
for extreme paranoia). Freeing the SMRs means those stream IDs would
instead fall back to the default "unmatched" behaviour, which in general
is going to be bypass, and thus allow DMA attacks.
It's harder to disable unmatched bypass in general, because we may have
devices which physically master through the SMMU but want low latency
more than they want translation (at the moment the best we can do is
leave the kernel unaware of those stream IDs), or there could be unknown
devices under control of the firmware or other agents which we would
disrupt by hitting a system-wide switch.

ok thanks, understand the point now. Agree that putting the previously bound
devices to fault is right than putting it to bypass. So the notifier to switch the
S2CRs to faulting entries can be added as a separate patch on top of this.

Regards,
Sricharan

[Lorenzo Pieralisi]

On Mon, Nov 28, 2016 at 11:12:08PM +0530, Sricharan wrote:

[...]

Post by Sricharan

Post by Robin Murphy
Cool. We're rather hoping that the ACPI stuff is good to go for 4.10
now, so it's probably worth pulling the rest of that in (beyond the one
patch I picked) to make sure the of_dma_configure/acpi_dma_configure
paths don't inadvertently diverge.

I rebased and was testing your branch with Lorenzo's series. One thing
i am still trying to get right is the acpi_dma_configure call. With your
series dma_configure calls pci_dma/of_dma configure, so i am just adding
acpi_dma_configure call there for non-pci ACPI devices as well. I see that
acpi_dma_configure right now is called from acpi_bind_one and
iort_add_smmu_platform_device, both go through the really_probe function
path, so moving acpi_dma_configure from the above the two functions
to dma_configure. I remember we discussed this on another thread, so
hopefully it is correct. I do not have an platform to test the ACPI though.
I will take some testing help on V4 for this.

I am happy to test it for you please just send me a pointer at your v4
code.

Thank you !
Lorenzo

[Sricharan]

Hi Lorenzo,

-----Original Message-----
Sent: Monday, November 28, 2016 11:44 PM
Subject: Re: [PATCH V3 0/8] IOMMU probe deferral support
[...]

Post by Sricharan

Post by Robin Murphy
Cool. We're rather hoping that the ACPI stuff is good to go for 4.10
now, so it's probably worth pulling the rest of that in (beyond the one
patch I picked) to make sure the of_dma_configure/acpi_dma_configure
paths don't inadvertently diverge.

I rebased and was testing your branch with Lorenzo's series. One thing
i am still trying to get right is the acpi_dma_configure call. With your
series dma_configure calls pci_dma/of_dma configure, so i am just adding
acpi_dma_configure call there for non-pci ACPI devices as well. I see that
acpi_dma_configure right now is called from acpi_bind_one and
iort_add_smmu_platform_device, both go through the really_probe function
path, so moving acpi_dma_configure from the above the two functions
to dma_configure. I remember we discussed this on another thread, so
hopefully it is correct. I do not have an platform to test the ACPI though.
I will take some testing help on V4 for this.

I am happy to test it for you please just send me a pointer at your v4
code.

I posted the v4 and CCed you there. So i am little skeptical about the acpi
changes that i have posted. I was checking for a function equivalent
in acpi as of_match_node in DT, to figure out if the iommu_spec.np that
the master device is pointing to is there in the iommu_of_table and based
on that we can decide if to defer the probe. I was seeing iort_scan_node
was its equivalent. But if that is not correct, then last patch has to be reworked.
Anyways will be good to know your feedback on this.

Regards,
Sricharan

[Lorenzo Pieralisi]

Hi Sricharan,

Post by Sricharan
Hi Lorenzo,

-----Original Message-----
Sent: Monday, November 28, 2016 11:44 PM
Subject: Re: [PATCH V3 0/8] IOMMU probe deferral support
[...]

Post by Sricharan

Post by Robin Murphy
Cool. We're rather hoping that the ACPI stuff is good to go for 4.10
now, so it's probably worth pulling the rest of that in (beyond the one
patch I picked) to make sure the of_dma_configure/acpi_dma_configure
paths don't inadvertently diverge.

I rebased and was testing your branch with Lorenzo's series. One thing
i am still trying to get right is the acpi_dma_configure call. With your
series dma_configure calls pci_dma/of_dma configure, so i am just adding
acpi_dma_configure call there for non-pci ACPI devices as well. I see that
acpi_dma_configure right now is called from acpi_bind_one and
iort_add_smmu_platform_device, both go through the really_probe function
path, so moving acpi_dma_configure from the above the two functions
to dma_configure. I remember we discussed this on another thread, so
hopefully it is correct. I do not have an platform to test the ACPI though.
I will take some testing help on V4 for this.

I am happy to test it for you please just send me a pointer at your v4
code.

I posted the v4 and CCed you there. So i am little skeptical about the acpi
changes that i have posted. I was checking for a function equivalent
in acpi as of_match_node in DT, to figure out if the iommu_spec.np that
the master device is pointing to is there in the iommu_of_table and based
on that we can decide if to defer the probe. I was seeing iort_scan_node
was its equivalent. But if that is not correct, then last patch has to be reworked.
Anyways will be good to know your feedback on this.

Sure I will test it asap, thanks for putting it together.

Thanks,
Lorenzo