| ============== | |
| UALink Support | |
| ============== | |
| Overview | |
| ======== | |
| Connected GPUs in a pod can directly access the remove memory on another GPU | |
| over UALink. Unlike RMDA, there is no copy involved; it is direct loads/stores | |
| over the fabric. Shared memory can only be accessed by a remote GPU if the | |
| memory was exported and the importer has been authorized. For the memory to be | |
| shared, it must be part of a unified physical address space shared between | |
| nodes. This address space is called NPA (Nework Physical Address) space. This | |
| address space is partitioned between the GPUs so that each GPU has its own | |
| segment of the address space in which to export its memory. Each GPU maintains | |
| a dedicated set of page tables for their NPA space similar to GPUVM. Note that | |
| this mechanism only allows for GPU access to remote memory. The remote memory | |
| is not CPU accessible. | |
| Exported memory is pinned. This is similar to how dma-bufs in VRAM are pinned | |
| for P2P access. Remote TLB shootdowns from the exporter are used when the | |
| exported memory is freed in order to remove access by remote GPUs. | |
| To access remote memory, the driver can map NPA addresses into its per process | |
| GPUVM page tables just like local memory. Applications use opaque handles to | |
| represent remote memory. GPUs in a pod communicate with eachother directly to | |
| exchange NPA addresses between importers and exporters. If a node goes offline | |
| or is reset, their peers will clean up any remaining refrences that are lost | |
| when that happens. NPA addresses are exchanged directly between the kernel | |
| drivers using the scale up fabric. NPA addresses are never exposed to user | |
| space. | |
| On the importer, the NPA space is like another physical address space. NPA | |
| addresses can be used as physical addresses for GPUVM to provide GPU virtual | |
| addresses to the memory for processes using the GPU. | |
| On the exporter, the NPA space provides a way to expose discontiguous local | |
| memory as a contiguous address range for remote GPUs. This allows the exporter | |
| to locally manage the pages mapped into the NPA space. | |
| Remote NPAs are managed like another device specific TTM pool similar to | |
| doorbells or VRAM, however they cannot be CPU mapped. | |
| User Interface | |
| ============== | |
| Two IOCTLs are provided to export and import remote memory. | |
| Export Memory | |
| ------------- | |
| To export memory, a UALINK handle must be created for an allocation that can be | |
| shared with another node in the pod. To do this the exporter calls the GEM | |
| UALink IOCTL with the GEM handle to the buffer it wants to export. The IOCTL | |
| returns a unique 128 bit handle which can be shared with the remote host. | |
| Calling export on the same GEM handle always returns the same UALink handle. | |
| The UALink handle is destroyed when the GEM object reference count reaches 0. | |
| Import Memory | |
| ------------- | |
| To import remote memory, the UALink handle from the remote node must be | |
| converted from a UALink handle to a local GEM object which represents the local | |
| reference to the NPA space on the importer. If the memory has already been | |
| imported, it just returns a new reference to the existing GEM object. If not, | |
| the importer queries the exporter to get the NPA address. Once it has that, the | |
| importer can create the GEM to represent the NPA space used by the allocation. | |
| The GEM object is then exported to the caller as a dma-buf. The dma-buf is | |
| leveraged for dynamic attachment which provides the ability to revoke access | |
| when necessary. | |
| Device to Device Communications | |
| =============================== | |
| Devices communicate via a protocol implemented in firmware. Mesages sent to a | |
| remote node generate an interrupt on that node for servicing. |