Master-aware devices and sideband ID data

Is there a specific case you've run across where conveying this additional
master info would be needed, or are you just thinking through how to fully
describe hardware?

Are there really cases out there were there is a hardwired hardware
relationship between RID and SID?

Post by Mark Rutland
or in the absence of any IOMMU(s).
* PCIe Requester IDs (RIDs) (bus:device.function AKA BDF)
* SMMU Stream IDs (SIDs)
* GICv3 ITS Device IDs (DIDs)
In the case of a system with PCIe, SMMU and GICv3, the master IDs are
+-------------+
| PCIe master |
+-------------+
||
|| Requester ID (RID)
|| Probeable from RC.
\/
+-------------------+
| PCIe Root Complex |
+-------------------+
||
|| SMMU Stream ID (SID)
|| Derived from RID, static non-probeable mapping.

The FSL LS2085A SoC has an actual RID->SID mapping table in the PCI
controller, but it is not static in the sense of fixed in hardware or
firmware. It's
a programmable mapping table, and we envision Linux programming this table.

Post by Mark Rutland
\/
+--------------+
| SMMU (IOMMU) |
+--------------+
||
|| ITS Device ID (DID)
|| Derived from SID, static non-probeable mapping.
\/

Is this even architecturally possible on ARM? Can the SMMU transform stream
IDs into some different number?

Post by Mark Rutland
+----------------------------+
| GICv3 ITS (MSI controller) |
+----------------------------+
In simpler cases, you might have a simpler set of master ID
+-----------------+
| Platform device |
+-----------------+
||
|| ITS Device ID (DID)
|| Hard-wired on the bus.
\/
+----------------------------+
| GICv3 ITS (MSI controller) |
+----------------------------+
Ignoring current bindings for the moment, I can see how we can describe
this with a generic master id-binding, with master-id-map along the
<child-id-base child-id-length parent parent-id-base>
For the PCIe example, this would look something like the following (with
...
/* Requester ID of PCIe endpoints, implicit at runtime */
master-id-cells = <1>;
master-id-map = <0 0x10000 &SMMU 0>;
}
...
/* SID, derived from RID */
master-id-cells = <1>;
/*
* For some reason the high bit of the ID was negated.
*/
master-id-map = <0x0000 0x8000 &ITS 0x0 0x8000>,
<0x8000 0x8000 &ITS 0x0 0x0000>;
};
...
/* DID, derived from SID */
master-id-cells = <2>;
/*
* Master traffic not propagated beyond this point, so no
* master-id-ranges
*/
};
master-id-cells = <1>;
master-ids = <0xf>, <0xb>;
master-id-map = <0 0xf &ITS 0 0>;
};
...
/* DID */
master-id-cells = <2>;
};
However, this conflicts/overlaps with existing bindings (at least iommus
and msi-parent), and I'm not sure how to reconcile them. Am I missing a
neat way of describing this that works with the existing bindings?
It's also not clear to me if it's sufficient to have a generic
"master-ids" property (with the relevant type being implicit from
context), or if each type of ID needs each type of ID (RID, SID, DID,
etc) needs its own.
Which other devices out there which require side-band master
information, and what do they require?

For us, at least, the master-id-map cannot be a static property.
Our PCI controller's DID->SID mapping table is programmable
and we won't know how to program it until Linux is running. A PCI
SRIOV card may have virtual functions that appear as hot-plugged
devices and there is no way that boot firmware would have the knowledge
to have set that table up ahead of time.

What we do need is some way to have boot firmware convey to
the OS what SIDs are available to be used in the DID->SID
table. We have a very limited number of SIDs available and boot
firmware may need to convey to Linux that for example-- PCI controller #2
has only 8 SIDs available for DID->SID mappings.

We had invented a property called "available-stream-ids" for some
proof of concept work, but perhaps the master-id-map property could
be used to convey available stream IDs by using a child-id-length
of 0. So to convey that stream IDs 5,6,7,8 are available:

PCI: ***@af000000 {
...
/* Requester ID of PCIe endpoints, implicit at runtime */
master-id-cells = <1>;

/* RIDS idmapped to SIDS @ SMMU */
master-id-map = <0 0 &SMMU 5
0 0 &SMMU 6
0 0 &SMMU 7
0 0 &SMMU 8>;
};

I'm not sure the above would conflict with the existing IOMMU
binding, but would seem to be an extension of it.

If you have a simple master with 2 stream IDS do this: ...
iommus = <&{/smmu} 23>, <&{/smmu} 24>;

If you have a static DID->SID mapping to convey, use the map as
you proposed:
master-id-cells = <1>;
master-id-map = <0 0x10000 &SMMU 0>;;

If your DID->SID is not static, but you need to convey the SIDs that are
available to this bus:
master-id-cells = <1>;
master-id-map = <0 0 &SMMU 23
0 0 &SMMU 24>;

I am less sure about how to deal with expressing the relationship of masters to
the GIC ITS, but is there really a problem there?

Thanks,
Stuart

Will Deacon

2015-05-08 15:49:03 UTC

Hi Stuart,

Is there a specific case you've run across where conveying this additional
master info would be needed, or are you just thinking through how to fully
describe hardware?

It's a combination of a few things:

1. We are aware of hardware that has ID transformations
2. ACPI allows this to be described in IORT
3. We need to be general enough to encompass current and future designs
in the device-tree binding.

Post by Stuart Yoder
Are there really cases out there were there is a hardwired hardware
relationship between RID and SID?

Yes, I believe it's actually the common case (at least, from Linux's
perspective).

Ok, so I assume you're not planning to use ACPI with this system?

Also, we can't just program this table willy-nilly, as I'm sure you're
aware. For example, updating a the SMRs in a live SMMU is a big no-no,
so although Linux may initialise the table, it can only be safely changed
at device init/teardown time, surely?

Post by Mark Rutland
\/
+--------------+
| SMMU (IOMMU) |
+--------------+
||
|| ITS Device ID (DID)
|| Derived from SID, static non-probeable mapping.
\/

Is this even architecturally possible on ARM? Can the SMMU transform stream
IDs into some different number?

From the OS perspective, this is what it looks like. What we'd actually
have in hardware is multiple SMMUs sharing an ITS, so there would need
to be something prepending an SMMU ID onto the StreamID to construct a
device ID, otherwise you'd have devices inexplicably sharing MSIs.

Post by Mark Rutland
However, this conflicts/overlaps with existing bindings (at least iommus
and msi-parent), and I'm not sure how to reconcile them. Am I missing a
neat way of describing this that works with the existing bindings?
It's also not clear to me if it's sufficient to have a generic
"master-ids" property (with the relevant type being implicit from
context), or if each type of ID needs each type of ID (RID, SID, DID,
etc) needs its own.
Which other devices out there which require side-band master
information, and what do they require?

For us, at least, the master-id-map cannot be a static property.
Our PCI controller's DID->SID mapping table is programmable
and we won't know how to program it until Linux is running. A PCI
SRIOV card may have virtual functions that appear as hot-plugged
devices and there is no way that boot firmware would have the knowledge
to have set that table up ahead of time.
What we do need is some way to have boot firmware convey to
the OS what SIDs are available to be used in the DID->SID
table. We have a very limited number of SIDs available and boot
firmware may need to convey to Linux that for example-- PCI controller #2
has only 8 SIDs available for DID->SID mappings.
We had invented a property called "available-stream-ids" for some
proof of concept work, but perhaps the master-id-map property could
be used to convey available stream IDs by using a child-id-length
...
/* Requester ID of PCIe endpoints, implicit at runtime */
master-id-cells = <1>;
master-id-map = <0 0 &SMMU 5
0 0 &SMMU 6
0 0 &SMMU 7
0 0 &SMMU 8>;
};
I'm not sure the above would conflict with the existing IOMMU
binding, but would seem to be an extension of it.
If you have a simple master with 2 stream IDS do this: ...
iommus = <&{/smmu} 23>, <&{/smmu} 24>;
If you have a static DID->SID mapping to convey, use the map as
master-id-cells = <1>;
master-id-map = <0 0x10000 &SMMU 0>;;
If your DID->SID is not static, but you need to convey the SIDs that are
master-id-cells = <1>;
master-id-map = <0 0 &SMMU 23
0 0 &SMMU 24>;

We need to be careful with terminology here. Can we use RID to mean the PCI
requester ID and DID to mean the GIC ITS device ID?

Post by Stuart Yoder
I am less sure about how to deal with expressing the relationship of masters to
the GIC ITS, but is there really a problem there?

Well we need to know the DID at the ITS for an arbitrary device, which is
constructed by performing the RID->SID and SID->DID transformations.

Mark probably has more feedback on the binding side of things.

Will

Stuart Yoder

2015-05-08 19:30:00 UTC

Post by Will Deacon
Hi Stuart,

Is there a specific case you've run across where conveying this additional
master info would be needed, or are you just thinking through how to fully
describe hardware?

1. We are aware of hardware that has ID transformations
2. ACPI allows this to be described in IORT
3. We need to be general enough to encompass current and future designs
in the device-tree binding.

Post by Stuart Yoder
Are there really cases out there were there is a hardwired hardware
relationship between RID and SID?

Yes, I believe it's actually the common case (at least, from Linux's
perspective).

Ok, so I assume you're not planning to use ACPI with this system?

Not initially, but perhaps in the future. I guess I don't know the implications
to ACPI yet.

Post by Will Deacon
Also, we can't just program this table willy-nilly, as I'm sure you're
aware. For example, updating a the SMRs in a live SMMU is a big no-no,
so although Linux may initialise the table, it can only be safely changed
at device init/teardown time, surely?

Correct. That would not be updated when a device is live.

Post by Mark Rutland
\/
+--------------+
| SMMU (IOMMU) |
+--------------+
||
|| ITS Device ID (DID)
|| Derived from SID, static non-probeable mapping.
\/

Is this even architecturally possible on ARM? Can the SMMU transform stream
IDs into some different number?

For us, at least, the master-id-map cannot be a static property.
Our PCI controller's DID->SID mapping table is programmable
and we won't know how to program it until Linux is running. A PCI
SRIOV card may have virtual functions that appear as hot-plugged
devices and there is no way that boot firmware would have the knowledge
to have set that table up ahead of time.
What we do need is some way to have boot firmware convey to
the OS what SIDs are available to be used in the DID->SID
table. We have a very limited number of SIDs available and boot
firmware may need to convey to Linux that for example-- PCI controller #2
has only 8 SIDs available for DID->SID mappings.
We had invented a property called "available-stream-ids" for some
proof of concept work, but perhaps the master-id-map property could
be used to convey available stream IDs by using a child-id-length
...
/* Requester ID of PCIe endpoints, implicit at runtime */
master-id-cells = <1>;
master-id-map = <0 0 &SMMU 5
0 0 &SMMU 6
0 0 &SMMU 7
0 0 &SMMU 8>;
};
I'm not sure the above would conflict with the existing IOMMU
binding, but would seem to be an extension of it.
If you have a simple master with 2 stream IDS do this: ...
iommus = <&{/smmu} 23>, <&{/smmu} 24>;
If you have a static DID->SID mapping to convey, use the map as
master-id-cells = <1>;
master-id-map = <0 0x10000 &SMMU 0>;;
If your DID->SID is not static, but you need to convey the SIDs that are
master-id-cells = <1>;
master-id-map = <0 0 &SMMU 23
0 0 &SMMU 24>;

We need to be careful with terminology here. Can we use RID to mean the PCI
requester ID and DID to mean the GIC ITS device ID?

Sorry, that was a typo on my part...where I said DID I meant RID.

Thanks,
Stuart

Will Deacon

2015-05-11 09:52:36 UTC

Post by Stuart Yoder
The FSL LS2085A SoC has an actual RID->SID mapping table in the PCI
controller, but it is not static in the sense of fixed in hardware or
firmware. It's
a programmable mapping table, and we envision Linux programming this table.

Ok, so I assume you're not planning to use ACPI with this system?

Not initially, but perhaps in the future. I guess I don't know the implications
to ACPI yet.

Well, I suggest you take a look at the IORT spec[1] asap, because I don't think
it can describe your system if you want to go in the direction of dynamic
mappings.

Correct. That would not be updated when a device is live.

Ok, so the best way for Linux is probably to abstract this in the bus code
and have that allocate the IDs when a device is `hotplugged' on. This would
also fit well with consolidating the group creation for platform and PCI
devices.

Will

[1] http://infocenter.arm.com/help/topic/com.arm.doc.den0049a/DEN0049A_IO_Remapping_Table.pdf

Chalamarla, Tirumalesh

2015-05-26 22:20:59 UTC

This is some thing we also like to see in ITS and SMMU drivers.

Post by Mark Rutland
Hi all,
For some devices, identification of particular masters is critical to
their operation (e.g. IOMMUs, MSI controllers). The identity of masters
is determined from sideband signals on the bus, and it may or may not be
possible to probe and/or configure this information. Worse still, these
information may be rewritten by intermediate buses, so the
information for a given master may differ at different points in the bus
hierarchy.
We currently describe this master information for devices attached to
IOMMUs, with a master ID being encoded in the iommu-cells. However this
only covers the ID between the master and its IOMMU(s), and we don't
currently have a mechanism to describe the master IDs as they are seen
by devices beyond the IOMMU(s), or in the absence of any IOMMU(s).
* PCIe Requester IDs (RIDs) (bus:device.function AKA BDF)
* SMMU Stream IDs (SIDs)
* GICv3 ITS Device IDs (DIDs)
In the case of a system with PCIe, SMMU and GICv3, the master IDs are
+-------------+
| PCIe master |
+-------------+
||
|| Requester ID (RID)
|| Probeable from RC.
\/
+-------------------+
| PCIe Root Complex |
+-------------------+
||
|| SMMU Stream ID (SID)
|| Derived from RID, static non-probeable mapping.
\/
+--------------+
| SMMU (IOMMU) |
+--------------+
||
|| ITS Device ID (DID)
|| Derived from SID, static non-probeable mapping.
\/
+----------------------------+
| GICv3 ITS (MSI controller) |
+----------------------------+
In simpler cases, you might have a simpler set of master ID
+-----------------+
| Platform device |
+-----------------+
||
|| ITS Device ID (DID)
|| Hard-wired on the bus.
\/
+----------------------------+
| GICv3 ITS (MSI controller) |
+----------------------------+
Ignoring current bindings for the moment, I can see how we can describe
this with a generic master id-binding, with master-id-map along the
<child-id-base child-id-length parent parent-id-base>
For the PCIe example, this would look something like the following (with
...
/* Requester ID of PCIe endpoints, implicit at runtime */
master-id-cells = <1>;
master-id-map = <0 0x10000 &SMMU 0>;
}
...
/* SID, derived from RID */
master-id-cells = <1>;
/*
* For some reason the high bit of the ID was negated.
*/
master-id-map = <0x0000 0x8000 &ITS 0x0 0x8000>,
<0x8000 0x8000 &ITS 0x0 0x0000>;
};
...
/* DID, derived from SID */
master-id-cells = <2>;
/*
* Master traffic not propagated beyond this point, so no
* master-id-ranges
*/
};

I think it is nice to have max IDs supported by masters. so that drivers can check and enforce.

Post by Mark Rutland
master-id-cells = <1>;
master-ids = <0xf>, <0xb>;
master-id-map = <0 0xf &ITS 0 0>;
};
...
/* DID */
master-id-cells = <2>;
};

Is this is not depending heavily on discover order, how do drivers know which device to get which ID. is it implicitly assumed in discovery order?
what happens to hot pluggable devices.

Thanks,
Tirumalesh.

Mark Rutland

2015-05-27 17:39:53 UTC

Post by Chalamarla, Tirumalesh
This is some thing we also like to see in ITS and SMMU drivers.

I think it is nice to have max IDs supported by masters. so that drivers can check and enforce.

The set of IDs that we expect to transform (i.e. those masters use)
would be implicit in the first master-id-map from a master. In the PCI
example above, no master is expected to generate an ID outside of the
range 0-0x10000 inclusive (and that's all the SMMU would see).

For devices which are not hotpluggable, the nodes for those devices
would describe the specific set of IDs they use.

Generally, between a master and a slave there might not be one
contiguous set of valid IDs as these may be rewritten along the way (as
happens at the SMMU between the PCI root complex and the ITS in the
example above).

Which drivers do you think need this information? What exactly are they
trying to check and enforce?

Post by Mark Rutland
master-id-cells = <1>;
master-ids = <0xf>, <0xb>;
master-id-map = <0 0xf &ITS 0 0>;
};
...
/* DID */
master-id-cells = <2>;
};

Is this is not depending heavily on discover order, how do drivers
know which device to get which ID. is it implicitly assumed in
discovery order?

I'm not sure I follow the question. Could you elaborate?

Post by Chalamarla, Tirumalesh
what happens to hot pluggable devices.

It would be necessary to be able to discover the ID assigned to the
device by the hotpluggable bus. For example, this could depend on which
slot the device is plugged into.

If you can't discover the ID associated with a hotpluggable device from
the bus it is plugged into I can't see how hotplug could work.

From that point on I would expect the ID transformations to be static as
in the example.

Thanks,
Mark.

Chalamarla, Tirumalesh

2015-05-29 17:46:07 UTC

Post by Chalamarla, Tirumalesh
This is some thing we also like to see in ITS and SMMU drivers.

I think it is nice to have max IDs supported by masters. so that drivers can check and enforce.

The set of IDs that we expect to transform (i.e. those masters use)
would be implicit in the first master-id-map from a master. In the PCI
example above, no master is expected to generate an ID outside of the
range 0-0x10000 inclusive (and that's all the SMMU would see).
For devices which are not hotpluggable, the nodes for those devices
would describe the specific set of IDs they use.
Generally, between a master and a slave there might not be one
contiguous set of valid IDs as these may be rewritten along the way (as
happens at the SMMU between the PCI root complex and the ITS in the
example above).
Which drivers do you think need this information? What exactly are they
trying to check and enforce?

I think, i miss understood by the names, so it is not base it is some thing like

x => f(x) => y (and f(x) is bitwise | )

ok now i see it.

Post by Mark Rutland
master-id-cells = <1>;
master-ids = <0xf>, <0xb>;
master-id-map = <0 0xf &ITS 0 0>;
};
...
/* DID */
master-id-cells = <2>;
};

Is this is not depending heavily on discover order, how do drivers
know which device to get which ID. is it implicitly assumed in
discovery order?

I'm not sure I follow the question. Could you elaborate?

Post by Chalamarla, Tirumalesh
what happens to hot pluggable devices.

It would be necessary to be able to discover the ID assigned to the
device by the hotpluggable bus. For example, this could depend on which
slot the device is plugged into.
If you can't discover the ID associated with a hotpluggable device from
the bus it is plugged into I can't see how hotplug could work.
From that point on I would expect the ID transformations to be static as
in the example.

what if i have an SMMU with two ITS masters and i want to decide which one to use based on runtime like which CPU i am serving.

i am thinking in terms go NUMA based systems, where my pci device is on node 0, if the interrupt is targeted to CPU on node 1, i would like to use ITS1, otherwise
ITS0.

is it possible to specify some thing like that or do you think its is best to leave to ITS driver.

Thanks,
Tirumalesh.

Post by Mark Rutland
Thanks,
Mark.

Mark Rutland

2015-06-01 10:22:08 UTC

Post by Chalamarla, Tirumalesh
This is some thing we also like to see in ITS and SMMU drivers.

I think it is nice to have max IDs supported by masters. so that drivers can check and enforce.

The set of IDs that we expect to transform (i.e. those masters use)
would be implicit in the first master-id-map from a master. In the PCI
example above, no master is expected to generate an ID outside of the
range 0-0x10000 inclusive (and that's all the SMMU would see).
For devices which are not hotpluggable, the nodes for those devices
would describe the specific set of IDs they use.
Generally, between a master and a slave there might not be one
contiguous set of valid IDs as these may be rewritten along the way (as
happens at the SMMU between the PCI root complex and the ITS in the
example above).
Which drivers do you think need this information? What exactly are they
trying to check and enforce?

I think, i miss understood by the names, so it is not base it is some thing like
x => f(x) => y (and f(x) is bitwise | )
ok now i see it.

It's subtraction and addition rather than bitwise ORing.

Consider a SID 0x8040 entering the SMMU. This falls in the range
described by the SMMU's second master-id-map entry (I've added brackets
to separate each entry:

<(0x8000) (0x8000) (&ITS) (0x0 0x0000)>

As 0x8040 falls in the range described by the first two elements, we kow
we can translate the ID to calculate the ID at the ITS. To do so, we
subtract the child-id-base (0x8000), and add the parent-id-base (0x0
0x0000). That give us 0x0 0x0040 as our ITS device ID.

Post by Mark Rutland
master-id-cells = <1>;
master-ids = <0xf>, <0xb>;
master-id-map = <0 0xf &ITS 0 0>;
};
...
/* DID */
master-id-cells = <2>;
};

Is this is not depending heavily on discover order, how do drivers
know which device to get which ID. is it implicitly assumed in
discovery order?

I'm not sure I follow the question. Could you elaborate?

Post by Chalamarla, Tirumalesh
what happens to hot pluggable devices.

It would be necessary to be able to discover the ID assigned to the
device by the hotpluggable bus. For example, this could depend on which
slot the device is plugged into.
If you can't discover the ID associated with a hotpluggable device from
the bus it is plugged into I can't see how hotplug could work.
From that point on I would expect the ID transformations to be static as
in the example.

what if i have an SMMU with two ITS masters and i want to decide which
one to use based on runtime like which CPU i am serving.
i am thinking in terms go NUMA based systems, where my pci device is
on node 0, if the interrupt is targeted to CPU on node 1, i would like
to use ITS1, otherwise
ITS0.

The binding can express that both these translation paths exist. For
example, on the SMMU node you could have:

master-id-map = <0x0000 0x8000 &ITS_1 0x0 0x0000>,
<0x0000 0x8000 &ITS_2 0x0 0x0000>;

Which would mean that master IDs are idmapped to both ITS instances.

Post by Chalamarla, Tirumalesh
is it possible to specify some thing like that or do you think its is
best to leave to ITS driver.

It's possible to specify that the paths exist. I expect that software
would select which to use at runtime.

Thanks,
Mark.

Chalamarla, Tirumalesh

2015-06-04 22:19:30 UTC

Post by Chalamarla, Tirumalesh
This is some thing we also like to see in ITS and SMMU drivers.

I think it is nice to have max IDs supported by masters. so that drivers can check and enforce.

The set of IDs that we expect to transform (i.e. those masters use)
would be implicit in the first master-id-map from a master. In the PCI
example above, no master is expected to generate an ID outside of the
range 0-0x10000 inclusive (and that's all the SMMU would see).
For devices which are not hotpluggable, the nodes for those devices
would describe the specific set of IDs they use.
Generally, between a master and a slave there might not be one
contiguous set of valid IDs as these may be rewritten along the way (as
happens at the SMMU between the PCI root complex and the ITS in the
example above).
Which drivers do you think need this information? What exactly are they
trying to check and enforce?

I think, i miss understood by the names, so it is not base it is some thing like
x => f(x) => y (and f(x) is bitwise | )
ok now i see it.

It's subtraction and addition rather than bitwise ORing.
Consider a SID 0x8040 entering the SMMU. This falls in the range
described by the SMMU's second master-id-map entry (I've added brackets
<(0x8000) (0x8000) (&ITS) (0x0 0x0000)>
As 0x8040 falls in the range described by the first two elements, we kow
we can translate the ID to calculate the ID at the ITS. To do so, we
subtract the child-id-base (0x8000), and add the parent-id-base (0x0
0x0000). That give us 0x0 0x0040 as our ITS device ID.

fine.

Post by Mark Rutland
master-id-cells = <1>;
master-ids = <0xf>, <0xb>;
master-id-map = <0 0xf &ITS 0 0>;
};
...
/* DID */
master-id-cells = <2>;
};

Is this is not depending heavily on discover order, how do drivers
know which device to get which ID. is it implicitly assumed in
discovery order?

I'm not sure I follow the question. Could you elaborate?

Post by Chalamarla, Tirumalesh
what happens to hot pluggable devices.

It would be necessary to be able to discover the ID assigned to the
device by the hotpluggable bus. For example, this could depend on which
slot the device is plugged into.
If you can't discover the ID associated with a hotpluggable device from
the bus it is plugged into I can't see how hotplug could work.
From that point on I would expect the ID transformations to be static as
in the example.

what if i have an SMMU with two ITS masters and i want to decide which
one to use based on runtime like which CPU i am serving.
i am thinking in terms go NUMA based systems, where my pci device is
on node 0, if the interrupt is targeted to CPU on node 1, i would like
to use ITS1, otherwise
ITS0.

The binding can express that both these translation paths exist. For
master-id-map = <0x0000 0x8000 &ITS_1 0x0 0x0000>,
<0x0000 0x8000 &ITS_2 0x0 0x0000>;
Which would mean that master IDs are idmapped to both ITS instances.

Post by Chalamarla, Tirumalesh
is it possible to specify some thing like that or do you think its is
best to leave to ITS driver.

It's possible to specify that the paths exist. I expect that software
would select which to use at runtime.

My worry is how to define any priorities/preferences between masters.
in general the proposal looks reasonable.

Thanks,
Tirumalesh.

Post by Mark Rutland
Thanks,
Mark.

Will Deacon

2015-06-05 09:05:34 UTC

Post by Mark Rutland
It's possible to specify that the paths exist. I expect that software
would select which to use at runtime.

My worry is how to define any priorities/preferences between masters.
in general the proposal looks reasonable.

I agree that the proposal looks reasonable (in terms of the ability to
describe the sort of topologies that we will face) but I still don't
understand what I need to do in e.g. my IOMMU driver to support this
binding whilst continuing to support the existing iommus binding, which
is relied upon to configure dma-mapping.

Mark: how do you see this co-existing/merging with the current bindings?
I don't think it's practical to throw away what we have and move over to
something totally different all in one go, but there clearly *is* benefit
in your proposal over the existing scheme.

Will

Mark Rutland

2015-06-09 10:17:54 UTC

Post by Mark Rutland
It's possible to specify that the paths exist. I expect that software
would select which to use at runtime.

My worry is how to define any priorities/preferences between masters.
in general the proposal looks reasonable.

As I mentioned in my initial mail, it's not clear to me how this can be
reconciled with the current bindings. Everything I've been able to come
up with so far at best ends up describing the same thing repeatedly.

I'll see what I can come up with. Any sugestions are welcome!

Post by Will Deacon
I don't think it's practical to throw away what we have and move over to
something totally different all in one go, but there clearly *is* benefit
in your proposal over the existing scheme.

I can see that's probably not practical. :(

Do we know what we're going to do w.r.t. IORT? That's going to require
the kernel to be able to handle a similar description to this proposal.

Mark.

Will Deacon

2015-06-10 08:11:21 UTC

Post by Mark Rutland
It's possible to specify that the paths exist. I expect that software
would select which to use at runtime.

My worry is how to define any priorities/preferences between masters.
in general the proposal looks reasonable.

As I mentioned in my initial mail, it's not clear to me how this can be
reconciled with the current bindings. Everything I've been able to come
up with so far at best ends up describing the same thing repeatedly.
I'll see what I can come up with. Any sugestions are welcome!

Well, I'd start with the current iommu binding and see if you can figure
out which properties need adding to it in order to describe the sort of
topologies you can describe with your original proposal.

I can see that's probably not practical. :(
Do we know what we're going to do w.r.t. IORT? That's going to require
the kernel to be able to handle a similar description to this proposal.

IORT is part of ACPI, so it doesn't care about the existing DT binding
and doesn't need to co-exist at runtime.

Will

Chalamarla, Tirumalesh

2015-07-02 20:26:55 UTC

any update on this?
On Jun 10, 2015, at 1:11 AM, Will Deacon <***@arm.com<mailto:***@arm.com>> wrote:

On Tue, Jun 09, 2015 at 11:17:54AM +0100, Mark Rutland wrote:
On Fri, Jun 05, 2015 at 10:05:34AM +0100, Will Deacon wrote:
On Thu, Jun 04, 2015 at 11:19:30PM +0100, Chalamarla, Tirumalesh wrote:
On Jun 1, 2015, at 3:22 AM, Mark Rutland <***@arm.com<mailto:***@arm.com>> wrote:
It's possible to specify that the paths exist. I expect that software
would select which to use at runtime.

My worry is how to define any priorities/preferences between masters.
in general the proposal looks reasonable.

I agree that the proposal looks reasonable (in terms of the ability to
describe the sort of topologies that we will face) but I still don't
understand what I need to do in e.g. my IOMMU driver to support this
binding whilst continuing to support the existing iommus binding, which
is relied upon to configure dma-mapping.

Mark: how do you see this co-existing/merging with the current bindings?

As I mentioned in my initial mail, it's not clear to me how this can be
reconciled with the current bindings. Everything I've been able to come
up with so far at best ends up describing the same thing repeatedly.

I'll see what I can come up with. Any sugestions are welcome!

Well, I'd start with the current iommu binding and see if you can figure
out which properties need adding to it in order to describe the sort of
topologies you can describe with your original proposal.

I don't think it's practical to throw away what we have and move over to
something totally different all in one go, but there clearly *is* benefit
in your proposal over the existing scheme.

I can see that's probably not practical. :(

Do we know what we're going to do w.r.t. IORT? That's going to require
the kernel to be able to handle a similar description to this proposal.

IORT is part of ACPI, so it doesn't care about the existing DT binding
and doesn't need to co-exist at runtime.

Will

Mark Rutland

2015-07-08 13:30:50 UTC

Post by Mark Rutland
It's possible to specify that the paths exist. I expect that software
would select which to use at runtime.

My worry is how to define any priorities/preferences between masters.
in general the proposal looks reasonable.

As I mentioned in my initial mail, it's not clear to me how this can be
reconciled with the current bindings. Everything I've been able to come
up with so far at best ends up describing the same thing repeatedly.
I'll see what I can come up with. Any sugestions are welcome!

I can't see a way of keeping the ID transformations explicit with the
existing bindings, but I think we can simply fold these down into
properties in the master nodes, given we expect each ID to be derived
from some initial master ID anyway.

So, to cater for the ITS we would need to pass master IDs along with the
MSI parent information, which we could do by extending msi-parent or by
introducing a new msis property which behaves similarly to the iommus
property, describing the MSI controllers the device can address (via any
IOMMUs), along with any controller-specific identification data.

Which means we'd have DT fragments like the following for an arbitrary
platform device:

its0: its {
...
msi-controller;
#msi-cells = <1>; // DeviceId
};

its1: its {
...
msi-controller;
#msi-cells = <1>; // DeviceId
};

smmu: smmu {
...
iommu-cells = <1>; // StreamId
};

device {
...
iommus = <&its 0>;
/* Can use either ITS, but has a different ID at each */
msis = <&its0 0x0>, <&its1 0x400>;
};

That doesn't allow you to describe a device with multiple mater ports
where each master port might want to generate MSIs, but I'm not sure if
that's a real case.

For PCIe root complexes, we'd need to describe the BDF -> iommu-cells
and BDF -> msi-cells translations separately with new properties on the
node for the root complex itself.

Is there anything obviously broken with the above approach?

Thanks,
Mark.

Will Deacon

2015-07-08 16:02:28 UTC

Post by Will Deacon
Mark: how do you see this co-existing/merging with the current bindings?

As I mentioned in my initial mail, it's not clear to me how this can be
reconciled with the current bindings. Everything I've been able to come
up with so far at best ends up describing the same thing repeatedly.
I'll see what I can come up with. Any sugestions are welcome!

I can't see a way of keeping the ID transformations explicit with the
existing bindings, but I think we can simply fold these down into
properties in the master nodes, given we expect each ID to be derived
from some initial master ID anyway.
So, to cater for the ITS we would need to pass master IDs along with the
MSI parent information, which we could do by extending msi-parent or by
introducing a new msis property which behaves similarly to the iommus
property, describing the MSI controllers the device can address (via any
IOMMUs), along with any controller-specific identification data.
Which means we'd have DT fragments like the following for an arbitrary
its0: its {
...
msi-controller;
#msi-cells = <1>; // DeviceId
};
its1: its {
...
msi-controller;
#msi-cells = <1>; // DeviceId
};
smmu: smmu {
...
iommu-cells = <1>; // StreamId
};
device {
...
iommus = <&its 0>;
/* Can use either ITS, but has a different ID at each */
msis = <&its0 0x0>, <&its1 0x400>;
};
That doesn't allow you to describe a device with multiple mater ports
where each master port might want to generate MSIs, but I'm not sure if
that's a real case.

In this case, I think we'd need something extra to define precisely how
those master ports relate to the rest of the system anyway. That would
likely be a device-specific property, I reckon.

Post by Mark Rutland
For PCIe root complexes, we'd need to describe the BDF -> iommu-cells
and BDF -> msi-cells translations separately with new properties on the
node for the root complex itself.
Is there anything obviously broken with the above approach?

Works for me. Can you write this up as a binding extension to msi-parent,
please?

Will

Mark Rutland

2015-07-16 13:34:41 UTC

Hi Will,

The below is an attempt at an MSI binding, derived from my original
example. It extends msi-parent inoto a phandle+(optional args) style
property.

I haven't yet managed to come up with a sane way of describing
the Bus-ID/BDF -> {iommu,msi}-cells translation, but this should
hopefully cover the platform device case.

For the Bus-ID translation case I'm not sure it's sane to attempt to use
msi-parent, given that the transformation description will necessarily
have to describe the parents anyway.

Thanks,
Mark.

---->8----
From 429dca4bba98732c492e95bdf395aa2ccc634e69 Mon Sep 17 00:00:00 2001
From: Mark Rutland <***@arm.com>
Date: Thu, 9 Jul 2015 17:53:00 +0100
Subject: [PATCH] Documentation: dt: add generic MSI bindings

Currently msi-parent is in use in a couple of drviers despite being
fairly underspecified. This patch adds a generic binding for MSIs
(including the existing msi-parent property) enabling the description of
platform devices capable of using MSIs.

This binding does not yet cover the general case. Currently the binding
does not cover the relationship between bus IDs (e.g. PCIe BDF) and
sideband data.

Signed-off-by: Mark Rutland <***@arm.com>
---
.../bindings/interrupt-controller/msi.txt | 135 +++++++++++++++++++++
1 file changed, 135 insertions(+)
create mode 100644 Documentation/devicetree/bindings/interrupt-controller/msi.txt

diff --git a/Documentation/devicetree/bindings/interrupt-controller/msi.txt b/Documentation/devicetree/bindings/interrupt-controller/msi.txt
new file mode 100644
index 0000000..c60c034
--- /dev/null
+++ b/Documentation/devicetree/bindings/interrupt-controller/msi.txt
@@ -0,0 +1,135 @@
+This document describes the generic device tree binding for MSI controllers and
+their master(s).
+
+Message Signaled Interrupts (MSIs) are a class of interrupts generated by a
+write to an MMIO address.
+
+MSIs were originally specified by PCI (and are used with PCIe), but may also be
+used with other busses, and hence a mechanism is required to relate devices on
+those busses to the MSI controllers which they are capable of using,
+potentially including additional information.
+
+MSIs are distinguished by some combination of:
+
+- The doorbell (the MMIO address written to).
+
+ Devices may be configured by software to write to arbitrary doorbells which
+ they can address. An MSI controller may feature a number of doorbells.
+
+- The payload (the value written to the doorbell).
+
+ Devices may be configured to write an arbitrary payload chosen by software.
+ MSI controllers may have restrictions on permitted payloads.
+
+- Sideband information accompanying the write.
+
+ Typically this is neither configurable nor probeable, and depends on the path
+ taken through the memory system (i.e. it is a property of the combination of
+ MSI controller and device rather than a property of either in isolation).
+
+
+MSI controllers:
+================
+
+An MSI controller signals interrupts to a CPU when a write is made to an MMIO
+address by some master. An MSI controller may feature a number of doorbells.
+
+Required properties:
+--------------------
+
+- msi-controller: Identifies the node as an MSI controller.
+
+Optional properties:
+--------------------
+
+- #msi-cells: The number of cells in an msi-specifier, required if not zero.
+
+ Typically this will encode information related to sideband data, and will
+ not encode doorbells or payloads as these can be configured dynamically.
+
+ The meaning of the msi-specifier is defined by the device tree binding of
+ the specific MSI controller.
+
+
+MSI clients
+===========
+
+MSI clients are devices which generate MSIs. For each MSI they wish to
+generate, the doorbell and payload may be configured, though sideband
+information may not be configurable.
+
+Required properties:
+--------------------
+
+- msi-parent: A list of phandle + msi-specifier pairs, one for each MSI
+ controller which the device is capable of using.
+
+ This property is unordered, and MSIs may be allocated from any combination of
+ MSI controllers listed in the msi-parent property.
+
+ If a device has restrictions on the allocation of MSIs, these restrictions
+ must be described with additional properties.
+
+ When #msi-cells is non-zero, busses with an msi-parent will require
+ additional properties to describe the relationship between devices on the bus
+ and the set of MSIs they can potentially generate.
+
+
+Example
+=======
+
+/ {
+ #address-cells = <1>;
+ #size-cells = <1>;
+
+ msi_a: msi-***@a {
+ reg = <0xa 0xf00>;
+ compatible = "vendor-a,some-controller";
+ msi-controller;
+ /* No sideband data, so #msi-cells omitted */
+ };
+
+ msi_b: msi-***@b {
+ reg = <0xb 0xf00>;
+ compatible = "vendor-b,another-controller";
+ msi-controller;
+ /* Each device has some unique ID */
+ #msi-cells = <1>;
+ };
+
+ msi_c: msi-***@c {
+ reg = <0xb 0xf00>;
+ compatible = "vendor-b,another-controller";
+ msi-controller;
+ /* Each device has some unique ID */
+ #msi-cells = <1>;
+ };
+
+ ***@0 {
+ reg = <0x0 0xf00>;
+ compatible = "vendor-c,some-device";
+
+ /* Can only generate MSIs to msi_a */
+ msi-parent = <&msi_a>;
+ };
+
+ ***@1 {
+ reg = <0x1 0xf00>;
+ compatible = "vendor-c,some-device";
+
+ /*
+ * Can generate MSIs to either A or B.
+ */
+ msi-parent = <&msi_a>, <&msi_b 0x17>;
+ };
+
+ ***@2 {
+ reg = <0x2 0xf00>;
+ compatible = "vendor-c,some-device";
+ /*
+ * Has different IDs at each MSI controller.
+ * Can generate MSIs to all of the MSI controllers.
+ */
+ msi-parent = <&msi_a>, <&msi_b 0x17>, <&msi_c 0x53>;
+ };
+};

--
1.9.1

Will Deacon

2015-07-17 10:36:09 UTC