3.2. RDMA Path

Several general objects are provides for storing paths. In RDMA a path is the connection between two end ports. The path representation used in this library holds the full connection description, including all the necessary header fields. These fields will be used as necessary, depending on the context that the path is used in.


For efficiency the path caches certain expensive information. In the current implementation the cache is imperfect and does not clear automatically. For this reason users should treat path records as write once. This is also good practice since a path may be held onto for a time when working with the parallel MAD scheduler or other cases.

3.2.1. IB Paths

In the IBA a path is unidirectional, but it may have the special property of being reversible. For reversible paths the return path is computed by calling rdma.path.IBPath.reverse(). Further, the path may be a ‘forward’ or ‘reverse’ path relative to the associated rdma.devices.EndPort - the exact direction will depend on context - send paths are ‘forward’ while receive paths are ‘reverse’.

The attributes for a IBPath as associated with packet headers are:

Path Attribute Header Field
has_grh rdma.IBA.HdrLRH.LNH [0]
traffic_class rdma.IBA.HdrGRH.TClass
flow_label rdma.IBA.HdrGRH.flowLabel
hop_limit rdma.IBA.HdrGRH.hopLmt
pkey rdma.IBA.HdrBTH.PKey
dqpn rdma.IBA.HdrBTH.destQP
sqpsn rdma.IBA.HdrBTH.PSN
qkey rdma.IBA.HdrDETH.QKey
sqpn rdma.IBA.HdrDETH.srcQP
min_rnr_timer rdma.IBA.HdrAETH.syndrome

The rule for this class is that it represents the packet headers in the context it is used/created. Eg when sending a packet it represents the packet header to be sent. When receiving a packet it represents the packet headers as received.

There are additional fields for use by verbs, review Verbs and rdma.path.IBPath.

To make the use of paths simpler in some common cases there are get/set properties that access end port GID indexes, PKey indexes, and LID bits. When used as a setattr the property will update the full version. When used as a getattr the property will use the end port to compute the index. The properties are able to cache the results of these searches. For example:

# Use pkey index 0

# Use the default PKEY

Do the same things.


The usage of asymmetric paths hasn’t been implemented yet. My feeling for this is that the path itself should be a reversible path and have additional attributes for the asymmetric paths. Depending on context the asymmetric attributes could then be used.

Directed route paths are supported through the rdma.path.IBDRPath subclass that includes additional information for use in the rdma.IBA.SMPFormatDirected MAD.

3.2.2. rdma.path module

class rdma.path.IBDRPath(end_port, **kwargs)

Bases: rdma.path.IBPath

Describe a directed route path in an IB network using a VL15 QP0 packet, a LRH and rdma.IBA.SMPFormatDirected MADs.

end_port is the rdma.devices.EndPort this path is associated with. kwargs is applied to set attributes of the instance during initialization.

By default this class construct a DR path to the local port.


raises AttributeError, GID addressing is not possible for DR paths.


Holds rdma.IBA.SMPFormatDirected.drDLID


Holds rdma.IBA.SMPFormatDirected.drPath. len(drPath) is used to set hopCount.


Holds rdma.IBA.SMPFormatDirected.drSLID. Should be the same as SLID.


Returns False, GID addressing is not possible for DR paths.

class rdma.path.IBPath(end_port, **kwargs)

Bases: rdma.path.Path

Describe a path in an IB network with a LRH and GRH and BTH

Our convention is that a path describes the LRH/GRH/BTH header fields on the wire. So a packet being sent from here should have a path with SGID == self and the path for a received packet should have DGID == self. Ditto for [ds]qpn and [DS]LID. Paths for packets received will thus have reversed semantics, SGID == remote.

This includes all relevant addressing information as well as any method specific information (ie a UMAD agent_id) required to use it with the method in question. Since this includes the BTH information it must also include the DQPN if that is required (eg for unconnected communication)

end_port is the rdma.devices.EndPort this path is associated with. kwargs is applied to set attributes of the instance during initialization.


Holds rdma.IBA.HdrGRH.DGID


Holds rdma.IBA.HdrLRH.DLID


Cache and return the LMC portion of the DLID. Assignment updates the DLID using the value from rdma.devices.EndPort.lid.


Maximum MTU to use on this path


Holds rdma.IBA.HdrGRH.SGID


Cache and return the index of the SGID for the associated EndPort. Assignment updates the SGID value.


Holds rdma.IBA.HdrLRH.SL


Holds rdma.IBA.HdrLRH.SLID


Cache and return the LMC portion of the SLID. Assignment updates the SLID using the value from rdma.devices.EndPort.lid.


Destination ack response time. Used to compute qp_timeout


Holds rdma.IBA.HdrBTH.destQP


Destination queue PSN, holds the initial value for the remote sqpsn.


Destination issuable RD atomic


Holds rdma.IBA.HdrGRH.flowLabel


True if a rdma.IBA.HdrGRH is present


Holds rdma.IBA.HdrGRH.hopLmt


The timeout to use for MADs on this path. Expressed as float seconds.


Minimum value for rdma.IBA.HdrAETH.syndrome for RNR NAK. (FIXME: Where does this come from??)


The packet lifetime value for this path. The lifetime value for the path is the expected uni-directional transit time. If a value has not been provided then the port’s rdma.devices.EndPort.subnet_timeout is used. To convert to seconds use 4.096 uS * 2**(packet_life_time)


Holds rdma.IBA.HdrBTH.PKey


Cache and return the index of the PKey for the associated EndPort. Assignment updates the pkey value.


Holds rdma.IBA.HdrDETH.QKey


The timeout to use for RC/RD connections. This is 2 * packet_life_time + target_ack_delay. Expressed as float seconds.


Maximum injection rate for this path


Used to compute mad_timeout


Reverse this path in-place according to IBA 13.5.4. If for_reply is True then the reversed path will be usable as a MAD reply, otherwise it is simply reversed (the only difference is that reply paths have the hop_limit set to 255). Returns self


Source RC ack response time. Value is from rdma.ibverbs.Context.query_device().


Holds rdma.IBA.HdrDETH.srcQP


Source queue PSN, holds the initial value for rdma.IBA.HdrBTH.PSN


Source issuable RD atomic


Holds rdma.IBA.HdrGRH.TClass


Method specific - override the agent_id for rdma.umad.UMAD

class rdma.path.LazyIBPath(end_port, **kwargs)

Bases: rdma.path.IBPath

Similar to rdma.path.IBPath but the unpack of the actual data deferred until necessary since most of the time we do not care.

end_port is the rdma.devices.EndPort this path is associated with. kwargs is applied to set attributes of the instance during initialization.

class rdma.path.Path(end_port, **kwargs)

Bases: object

Describe an RDMA path. This also serves to cache the path for cases that need it in an AH or other format. Paths should be considered final, once construction is finished their content must never change. This is to prevent cached data in the path from becoming stale.

The repr() format will produce a string that is valid Python and is also compatible with from_spec_string().

end_port is the rdma.devices.EndPort this path is associated with. kwargs is applied to set attributes of the instance during initialization.


Return True if the path is complete and usable.


Return a new path object that is a copy of this one. This takes care of the internal caching mechanism. kwargs is applied to set attributes of the instance after copying.


Release any cached information.


End Port this path is associated with


Number of times a MAD will be resent, or the value of retry_cnt for a RC QP.

exception rdma.path.SAPathNotFoundError(fmt, err=None)

Bases: rdma.MADClassError

Thrown when a path record query fails with a no records error from the SM.

rdma.path.fill_path(qp, path, max_rd_atomic=255)

Fill in fields in path assuming path will be used with a QP. The filled fields are used to establish the QP parameters.

If max_rd_atomic is set then that at most that many responder resources for RDMA READ and ATOMICs will be provisioned. Since HCAs have limited responder resources this value should always be limited if possible. If RDMA READ and ATOMICs will not be used then it should be set to 0. Otherwise at least set to the sendq depth.


Construct a Path (or derived) instance from it’s repr string.

This parser is safe to use with untrusted data.

Raises ValueError:
 If the string can not be parsed.
rdma.path.from_string(s, default_end_port=None, require_dev=None, require_ep=None)

Convert the string s into an instance of Path or derived.

Supported formats for s are:
Format Example Creates
Port GID fe80::2:c903:0:1491 IBPath.DGID = s
Scope’d GID fe80::2:c903:0:1491%mlx4_0/1 IBPath.DGID = s
Port GUID 0002:c903:0000:1491 IBPath.DGID = fe80:: + s
LID 12 IBPath.DLID = 12
Hex LID 0xc IBPath.DLID = 12
DR Path 0,1, IBDRPath.drPath = ‘\0\1’
Path Spec IBPath(DLID=2,SL=2) IBPath.{DLID=2,SL=2}

If the format unambiguously specifies an end port, eg due to a provided scope or by specifying the subnet prefix then the result will have end_port set appropriately. Otherwise end_port is set to default_end_port.

require_dev and require_ep will restrict the lookup to returning a path for those conditions. If a scoped address is given that doesn’t match then ValueError is raised. These options should be used when a path is being parsed for use with an existing bound resource (eg a rdma.ibverbs.Context or rdma.ibverbs.)

FUTURE: This may return paths other than IB for other technologies.

Raises ValueError:
 If the string can not be parsed.
rdma.path.get_mad_path(mad, ep_addr, **kwargs)

Query the SA and return a path for ep_addr.

This is a simplified query function to return MAD paths from the end port associated with mad to the destination ep_addr. If ep_addr is a string then from_string() is called automatically, otherwise rdma.IBA.conv_ep_addr() is used. Thus this will accept a destination address string, an integer (DLID), GID (DGID) and GUID (DGID).

This returns a single reversible path.

If mad is an async instance then this routine returns a coroutine that will do the resolution, otherwise the new path is returned.

rdma.path.resolve_path(mad, path, reversible=False, properties=None)

Resolve path to a full path for use with a QP. path must have at least a DGID or DLID set.

properties is a dictionary of additional PR fields to set in the query.

If mad is an async instance then this routine returns a coroutine that will do the resolution, otherwise the new path is returned.

FUTURE: This routine may populate path with up to 3 full path records, one for GMPs, one for the forward direction and one for the return direction. If the path is being used for UD then it should probably set the reversible argument to True.


Table Of Contents

Previous topic

3.1. rdma.devices module

Next topic

3.3. rdma.tools module

This Page