| .. include:: <isonum.txt> | |
| ===================== | |
| VFIO Mediated devices | |
| ===================== | |
| :Copyright: |copy| 2016, NVIDIA CORPORATION. All rights reserved. | |
| :Author: Neo Jia <cjia@nvidia.com> | |
| :Author: Kirti Wankhede <kwankhede@nvidia.com> | |
| This program is free software; you can redistribute it and/or modify | |
| it under the terms of the GNU General Public License version 2 as | |
| published by the Free Software Foundation. | |
| Virtual Function I/O (VFIO) Mediated devices[1] | |
| =============================================== | |
| The number of use cases for virtualizing DMA devices that do not have built-in | |
| SR_IOV capability is increasing. Previously, to virtualize such devices, | |
| developers had to create their own management interfaces and APIs, and then | |
| integrate them with user space software. To simplify integration with user space | |
| software, we have identified common requirements and a unified management | |
| interface for such devices. | |
| The VFIO driver framework provides unified APIs for direct device access. It is | |
| an IOMMU/device-agnostic framework for exposing direct device access to user | |
| space in a secure, IOMMU-protected environment. This framework is used for | |
| multiple devices, such as GPUs, network adapters, and compute accelerators. With | |
| direct device access, virtual machines or user space applications have direct | |
| access to the physical device. This framework is reused for mediated devices. | |
| The mediated core driver provides a common interface for mediated device | |
| management that can be used by drivers of different devices. This module | |
| provides a generic interface to perform these operations: | |
| * Create and destroy a mediated device | |
| * Add a mediated device to and remove it from a mediated bus driver | |
| * Add a mediated device to and remove it from an IOMMU group | |
| The mediated core driver also provides an interface to register a bus driver. | |
| For example, the mediated VFIO mdev driver is designed for mediated devices and | |
| supports VFIO APIs. The mediated bus driver adds a mediated device to and | |
| removes it from a VFIO group. | |
| The following high-level block diagram shows the main components and interfaces | |
| in the VFIO mediated driver framework. The diagram shows NVIDIA, Intel, and IBM | |
| devices as examples, as these devices are the first devices to use this module:: | |
| +---------------+ | |
| | | | |
| | +-----------+ | mdev_register_driver() +--------------+ | |
| | | | +<------------------------+ | | |
| | | mdev | | | | | |
| | | bus | +------------------------>+ vfio_mdev.ko |<-> VFIO user | |
| | | driver | | probe()/remove() | | APIs | |
| | | | | +--------------+ | |
| | +-----------+ | | |
| | | | |
| | MDEV CORE | | |
| | MODULE | | |
| | mdev.ko | | |
| | +-----------+ | mdev_register_device() +--------------+ | |
| | | | +<------------------------+ | | |
| | | | | | nvidia.ko |<-> physical | |
| | | | +------------------------>+ | device | |
| | | | | callbacks +--------------+ | |
| | | Physical | | | |
| | | device | | mdev_register_device() +--------------+ | |
| | | interface | |<------------------------+ | | |
| | | | | | i915.ko |<-> physical | |
| | | | +------------------------>+ | device | |
| | | | | callbacks +--------------+ | |
| | | | | | |
| | | | | mdev_register_device() +--------------+ | |
| | | | +<------------------------+ | | |
| | | | | | ccw_device.ko|<-> physical | |
| | | | +------------------------>+ | device | |
| | | | | callbacks +--------------+ | |
| | +-----------+ | | |
| +---------------+ | |
| Registration Interfaces | |
| ======================= | |
| The mediated core driver provides the following types of registration | |
| interfaces: | |
| * Registration interface for a mediated bus driver | |
| * Physical device driver interface | |
| Registration Interface for a Mediated Bus Driver | |
| ------------------------------------------------ | |
| The registration interface for a mediated bus driver provides the following | |
| structure to represent a mediated device's driver:: | |
| /* | |
| * struct mdev_driver [2] - Mediated device's driver | |
| * @name: driver name | |
| * @probe: called when new device created | |
| * @remove: called when device removed | |
| * @driver: device driver structure | |
| */ | |
| struct mdev_driver { | |
| const char *name; | |
| int (*probe) (struct device *dev); | |
| void (*remove) (struct device *dev); | |
| struct device_driver driver; | |
| }; | |
| A mediated bus driver for mdev should use this structure in the function calls | |
| to register and unregister itself with the core driver: | |
| * Register:: | |
| extern int mdev_register_driver(struct mdev_driver *drv, | |
| struct module *owner); | |
| * Unregister:: | |
| extern void mdev_unregister_driver(struct mdev_driver *drv); | |
| The mediated bus driver is responsible for adding mediated devices to the VFIO | |
| group when devices are bound to the driver and removing mediated devices from | |
| the VFIO when devices are unbound from the driver. | |
| Physical Device Driver Interface | |
| -------------------------------- | |
| The physical device driver interface provides the mdev_parent_ops[3] structure | |
| to define the APIs to manage work in the mediated core driver that is related | |
| to the physical device. | |
| The structures in the mdev_parent_ops structure are as follows: | |
| * dev_attr_groups: attributes of the parent device | |
| * mdev_attr_groups: attributes of the mediated device | |
| * supported_config: attributes to define supported configurations | |
| The functions in the mdev_parent_ops structure are as follows: | |
| * create: allocate basic resources in a driver for a mediated device | |
| * remove: free resources in a driver when a mediated device is destroyed | |
| (Note that mdev-core provides no implicit serialization of create/remove | |
| callbacks per mdev parent device, per mdev type, or any other categorization. | |
| Vendor drivers are expected to be fully asynchronous in this respect or | |
| provide their own internal resource protection.) | |
| The callbacks in the mdev_parent_ops structure are as follows: | |
| * open: open callback of mediated device | |
| * close: close callback of mediated device | |
| * ioctl: ioctl callback of mediated device | |
| * read : read emulation callback | |
| * write: write emulation callback | |
| * mmap: mmap emulation callback | |
| A driver should use the mdev_parent_ops structure in the function call to | |
| register itself with the mdev core driver:: | |
| extern int mdev_register_device(struct device *dev, | |
| const struct mdev_parent_ops *ops); | |
| However, the mdev_parent_ops structure is not required in the function call | |
| that a driver should use to unregister itself with the mdev core driver:: | |
| extern void mdev_unregister_device(struct device *dev); | |
| Mediated Device Management Interface Through sysfs | |
| ================================================== | |
| The management interface through sysfs enables user space software, such as | |
| libvirt, to query and configure mediated devices in a hardware-agnostic fashion. | |
| This management interface provides flexibility to the underlying physical | |
| device's driver to support features such as: | |
| * Mediated device hot plug | |
| * Multiple mediated devices in a single virtual machine | |
| * Multiple mediated devices from different physical devices | |
| Links in the mdev_bus Class Directory | |
| ------------------------------------- | |
| The /sys/class/mdev_bus/ directory contains links to devices that are registered | |
| with the mdev core driver. | |
| Directories and files under the sysfs for Each Physical Device | |
| -------------------------------------------------------------- | |
| :: | |
| |- [parent physical device] | |
| |--- Vendor-specific-attributes [optional] | |
| |--- [mdev_supported_types] | |
| | |--- [<type-id>] | |
| | | |--- create | |
| | | |--- name | |
| | | |--- available_instances | |
| | | |--- device_api | |
| | | |--- description | |
| | | |--- [devices] | |
| | |--- [<type-id>] | |
| | | |--- create | |
| | | |--- name | |
| | | |--- available_instances | |
| | | |--- device_api | |
| | | |--- description | |
| | | |--- [devices] | |
| | |--- [<type-id>] | |
| | |--- create | |
| | |--- name | |
| | |--- available_instances | |
| | |--- device_api | |
| | |--- description | |
| | |--- [devices] | |
| * [mdev_supported_types] | |
| The list of currently supported mediated device types and their details. | |
| [<type-id>], device_api, and available_instances are mandatory attributes | |
| that should be provided by vendor driver. | |
| * [<type-id>] | |
| The [<type-id>] name is created by adding the device driver string as a prefix | |
| to the string provided by the vendor driver. This format of this name is as | |
| follows:: | |
| sprintf(buf, "%s-%s", dev_driver_string(parent->dev), group->name); | |
| (or using mdev_parent_dev(mdev) to arrive at the parent device outside | |
| of the core mdev code) | |
| * device_api | |
| This attribute should show which device API is being created, for example, | |
| "vfio-pci" for a PCI device. | |
| * available_instances | |
| This attribute should show the number of devices of type <type-id> that can be | |
| created. | |
| * [device] | |
| This directory contains links to the devices of type <type-id> that have been | |
| created. | |
| * name | |
| This attribute should show human readable name. This is optional attribute. | |
| * description | |
| This attribute should show brief features/description of the type. This is | |
| optional attribute. | |
| Directories and Files Under the sysfs for Each mdev Device | |
| ---------------------------------------------------------- | |
| :: | |
| |- [parent phy device] | |
| |--- [$MDEV_UUID] | |
| |--- remove | |
| |--- mdev_type {link to its type} | |
| |--- vendor-specific-attributes [optional] | |
| * remove (write only) | |
| Writing '1' to the 'remove' file destroys the mdev device. The vendor driver can | |
| fail the remove() callback if that device is active and the vendor driver | |
| doesn't support hot unplug. | |
| Example:: | |
| # echo 1 > /sys/bus/mdev/devices/$mdev_UUID/remove | |
| Mediated device Hot plug | |
| ------------------------ | |
| Mediated devices can be created and assigned at runtime. The procedure to hot | |
| plug a mediated device is the same as the procedure to hot plug a PCI device. | |
| Translation APIs for Mediated Devices | |
| ===================================== | |
| The following APIs are provided for translating user pfn to host pfn in a VFIO | |
| driver:: | |
| extern int vfio_pin_pages(struct device *dev, unsigned long *user_pfn, | |
| int npage, int prot, unsigned long *phys_pfn); | |
| extern int vfio_unpin_pages(struct device *dev, unsigned long *user_pfn, | |
| int npage); | |
| These functions call back into the back-end IOMMU module by using the pin_pages | |
| and unpin_pages callbacks of the struct vfio_iommu_driver_ops[4]. Currently | |
| these callbacks are supported in the TYPE1 IOMMU module. To enable them for | |
| other IOMMU backend modules, such as PPC64 sPAPR module, they need to provide | |
| these two callback functions. | |
| Using the Sample Code | |
| ===================== | |
| mtty.c in samples/vfio-mdev/ directory is a sample driver program to | |
| demonstrate how to use the mediated device framework. | |
| The sample driver creates an mdev device that simulates a serial port over a PCI | |
| card. | |
| 1. Build and load the mtty.ko module. | |
| This step creates a dummy device, /sys/devices/virtual/mtty/mtty/ | |
| Files in this device directory in sysfs are similar to the following:: | |
| # tree /sys/devices/virtual/mtty/mtty/ | |
| /sys/devices/virtual/mtty/mtty/ | |
| |-- mdev_supported_types | |
| | |-- mtty-1 | |
| | | |-- available_instances | |
| | | |-- create | |
| | | |-- device_api | |
| | | |-- devices | |
| | | `-- name | |
| | `-- mtty-2 | |
| | |-- available_instances | |
| | |-- create | |
| | |-- device_api | |
| | |-- devices | |
| | `-- name | |
| |-- mtty_dev | |
| | `-- sample_mtty_dev | |
| |-- power | |
| | |-- autosuspend_delay_ms | |
| | |-- control | |
| | |-- runtime_active_time | |
| | |-- runtime_status | |
| | `-- runtime_suspended_time | |
| |-- subsystem -> ../../../../class/mtty | |
| `-- uevent | |
| 2. Create a mediated device by using the dummy device that you created in the | |
| previous step:: | |
| # echo "83b8f4f2-509f-382f-3c1e-e6bfe0fa1001" > \ | |
| /sys/devices/virtual/mtty/mtty/mdev_supported_types/mtty-2/create | |
| 3. Add parameters to qemu-kvm:: | |
| -device vfio-pci,\ | |
| sysfsdev=/sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001 | |
| 4. Boot the VM. | |
| In the Linux guest VM, with no hardware on the host, the device appears | |
| as follows:: | |
| # lspci -s 00:05.0 -xxvv | |
| 00:05.0 Serial controller: Device 4348:3253 (rev 10) (prog-if 02 [16550]) | |
| Subsystem: Device 4348:3253 | |
| Physical Slot: 5 | |
| Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- | |
| Stepping- SERR- FastB2B- DisINTx- | |
| Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- | |
| <TAbort- <MAbort- >SERR- <PERR- INTx- | |
| Interrupt: pin A routed to IRQ 10 | |
| Region 0: I/O ports at c150 [size=8] | |
| Region 1: I/O ports at c158 [size=8] | |
| Kernel driver in use: serial | |
| 00: 48 43 53 32 01 00 00 02 10 02 00 07 00 00 00 00 | |
| 10: 51 c1 00 00 59 c1 00 00 00 00 00 00 00 00 00 00 | |
| 20: 00 00 00 00 00 00 00 00 00 00 00 00 48 43 53 32 | |
| 30: 00 00 00 00 00 00 00 00 00 00 00 00 0a 01 00 00 | |
| In the Linux guest VM, dmesg output for the device is as follows: | |
| serial 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10 | |
| 0000:00:05.0: ttyS1 at I/O 0xc150 (irq = 10) is a 16550A | |
| 0000:00:05.0: ttyS2 at I/O 0xc158 (irq = 10) is a 16550A | |
| 5. In the Linux guest VM, check the serial ports:: | |
| # setserial -g /dev/ttyS* | |
| /dev/ttyS0, UART: 16550A, Port: 0x03f8, IRQ: 4 | |
| /dev/ttyS1, UART: 16550A, Port: 0xc150, IRQ: 10 | |
| /dev/ttyS2, UART: 16550A, Port: 0xc158, IRQ: 10 | |
| 6. Using minicom or any terminal emulation program, open port /dev/ttyS1 or | |
| /dev/ttyS2 with hardware flow control disabled. | |
| 7. Type data on the minicom terminal or send data to the terminal emulation | |
| program and read the data. | |
| Data is loop backed from hosts mtty driver. | |
| 8. Destroy the mediated device that you created:: | |
| # echo 1 > /sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001/remove | |
| References | |
| ========== | |
| 1. See Documentation/vfio.txt for more information on VFIO. | |
| 2. struct mdev_driver in include/linux/mdev.h | |
| 3. struct mdev_parent_ops in include/linux/mdev.h | |
| 4. struct vfio_iommu_driver_ops in include/linux/vfio.h |