| =============== | |
| RDMA Controller | |
| =============== | |
| .. Contents | |
| 1. Overview | |
| 1-1. What is RDMA controller? | |
| 1-2. Why RDMA controller needed? | |
| 1-3. How is RDMA controller implemented? | |
| 2. Usage Examples | |
| 3. RDMA Interface Files | |
| 1. Overview | |
| =========== | |
| 1-1. What is RDMA controller? | |
| ----------------------------- | |
| RDMA controller allows user to limit RDMA/IB specific resources that a given | |
| set of processes can use. These processes are grouped using RDMA controller. | |
| RDMA controller defines two resources which can be limited for processes of a | |
| cgroup. | |
| 1-2. Why RDMA controller needed? | |
| -------------------------------- | |
| Currently user space applications can easily take away all the rdma verb | |
| specific resources such as AH, CQ, QP, MR etc. Due to which other applications | |
| in other cgroup or kernel space ULPs may not even get chance to allocate any | |
| rdma resources. This can lead to service unavailability. | |
| Therefore RDMA controller is needed through which resource consumption | |
| of processes can be limited. Through this controller different rdma | |
| resources can be accounted. | |
| 1-3. How is RDMA controller implemented? | |
| ---------------------------------------- | |
| RDMA cgroup allows limit configuration of resources. Rdma cgroup maintains | |
| resource accounting per cgroup, per device using resource pool structure. | |
| Each such resource pool is limited up to 64 resources in given resource pool | |
| by rdma cgroup, which can be extended later if required. | |
| This resource pool object is linked to the cgroup css. Typically there | |
| are 0 to 4 resource pool instances per cgroup, per device in most use cases. | |
| But nothing limits to have it more. At present hundreds of RDMA devices per | |
| single cgroup may not be handled optimally, however there is no | |
| known use case or requirement for such configuration either. | |
| Since RDMA resources can be allocated from any process and can be freed by any | |
| of the child processes which shares the address space, rdma resources are | |
| always owned by the creator cgroup css. This allows process migration from one | |
| to other cgroup without major complexity of transferring resource ownership; | |
| because such ownership is not really present due to shared nature of | |
| rdma resources. Linking resources around css also ensures that cgroups can be | |
| deleted after processes migrated. This allow progress migration as well with | |
| active resources, even though that is not a primary use case. | |
| Whenever RDMA resource charging occurs, owner rdma cgroup is returned to | |
| the caller. Same rdma cgroup should be passed while uncharging the resource. | |
| This also allows process migrated with active RDMA resource to charge | |
| to new owner cgroup for new resource. It also allows to uncharge resource of | |
| a process from previously charged cgroup which is migrated to new cgroup, | |
| even though that is not a primary use case. | |
| Resource pool object is created in following situations. | |
| (a) User sets the limit and no previous resource pool exist for the device | |
| of interest for the cgroup. | |
| (b) No resource limits were configured, but IB/RDMA stack tries to | |
| charge the resource. So that it correctly uncharge them when applications are | |
| running without limits and later on when limits are enforced during uncharging, | |
| otherwise usage count will drop to negative. | |
| Resource pool is destroyed if all the resource limits are set to max and | |
| it is the last resource getting deallocated. | |
| User should set all the limit to max value if it intents to remove/unconfigure | |
| the resource pool for a particular device. | |
| IB stack honors limits enforced by the rdma controller. When application | |
| query about maximum resource limits of IB device, it returns minimum of | |
| what is configured by user for a given cgroup and what is supported by | |
| IB device. | |
| Following resources can be accounted by rdma controller. | |
| ========== ============================= | |
| hca_handle Maximum number of HCA Handles | |
| hca_object Maximum number of HCA Objects | |
| ========== ============================= | |
| RDMA devices from all network namespaces are listed. Each line starts with | |
| the device name. If more than one device has the same name, ``index=N`` | |
| follows the name, where ``N`` is the system-wide RDMA device index, unique | |
| among registered devices. When configuring a limit, the index is optional | |
| for a globally unique name and required for a duplicate name. A write without | |
| the required index fails with ``-ENOTUNIQ``. | |
| 2. Usage Examples | |
| ================= | |
| (a) Configure resource limit:: | |
| echo mlx4_0 hca_handle=2 hca_object=2000 > /sys/fs/cgroup/rdma/1/rdma.max | |
| echo ocrdma1 hca_handle=3 > /sys/fs/cgroup/rdma/2/rdma.max | |
| echo "rxe0 index=5 hca_handle=2" > /sys/fs/cgroup/rdma/3/rdma.max | |
| (b) Query resource limit:: | |
| cat /sys/fs/cgroup/rdma/2/rdma.max | |
| #Output: | |
| mlx4_0 hca_handle=2 hca_object=2000 | |
| ocrdma1 hca_handle=3 hca_object=max | |
| (c) Query current usage:: | |
| cat /sys/fs/cgroup/rdma/2/rdma.current | |
| #Output: | |
| mlx4_0 hca_handle=1 hca_object=20 | |
| ocrdma1 hca_handle=1 hca_object=23 | |
| (d) Delete resource limit:: | |
| echo mlx4_0 hca_handle=max hca_object=max > /sys/fs/cgroup/rdma/1/rdma.max | |
| 3. RDMA Interface Files | |
| ======================== | |
| The following interface files are available in each non-root RDMA cgroup. | |
| rdma.max | |
| A read-write file which describes the configured resource limit | |
| for an RDMA/IB device. See the Usage Examples above. | |
| rdma.current | |
| A read-only file which describes the current resource usage. | |
| rdma.peak | |
| A read-only nested-keyed file which shows the historical high | |
| watermark of resource usage per device since the cgroup was created. | |
| An example for mlx4 and ocrdma device follows:: | |
| mlx4_0 hca_handle=1 hca_object=20 | |
| ocrdma1 hca_handle=0 hca_object=23 | |
| rdma.events | |
| A read-only nested-keyed file which exists on non-root cgroups | |
| and contains the following keys: | |
| max | |
| The number of times a process in this cgroup or its | |
| descendants attempted an RDMA resource allocation that | |
| was rejected because a rdma.max limit in the subtree | |
| was reached. This is a hierarchical counter propagated | |
| upward to all ancestor cgroups. A value change in this | |
| file generates a file modified event. | |
| alloc_fail | |
| The number of RDMA resource allocation attempts that | |
| originated in this cgroup or its descendants and failed | |
| due to a rdma.max limit being reached. This is a | |
| hierarchical counter propagated upward. | |
| An example for mlx4 device follows:: | |
| mlx4_0 hca_handle.max=5 hca_handle.alloc_fail=3 hca_object.max=0 hca_object.alloc_fail=0 | |
| rdma.events.local | |
| Similar to rdma.events but the fields are local to the cgroup, | |
| i.e. not hierarchical. The file modified event generated on this | |
| file reflects only the local events. | |
| The following nested keys are defined. | |
| max | |
| The number of times a process in this cgroup or its | |
| descendants attempted an RDMA resource allocation that | |
| was rejected because this cgroup's own rdma.max limit | |
| was reached. | |
| alloc_fail | |
| The number of RDMA resource allocation attempts | |
| originating from this cgroup that failed due to this | |
| cgroup's or an ancestor's rdma.max limit. | |
| An example for mlx4 device follows:: | |
| mlx4_0 hca_handle.max=5 hca_handle.alloc_fail=0 hca_object.max=0 hca_object.alloc_fail=0 |