| ===================================================================== | |
| NVIDIA Tegra410 SoC Uncore Performance Monitoring Unit (PMU) | |
| ===================================================================== | |
| The NVIDIA Tegra410 SoC includes various system PMUs to measure key performance | |
| metrics like memory bandwidth, latency, and utilization: | |
| * Unified Coherence Fabric (UCF) | |
| * PCIE | |
| * PCIE-TGT | |
| * CPU Memory (CMEM) Latency | |
| * NVLink-C2C | |
| * NV-CLink | |
| * NV-DLink | |
| PMU Driver | |
| ---------- | |
| The PMU driver describes the available events and configuration of each PMU in | |
| sysfs. Please see the sections below to get the sysfs path of each PMU. Like | |
| other uncore PMU drivers, the driver provides "cpumask" sysfs attribute to show | |
| the CPU id used to handle the PMU event. There is also "associated_cpus" | |
| sysfs attribute, which contains a list of CPUs associated with the PMU instance. | |
| UCF PMU | |
| ------- | |
| The Unified Coherence Fabric (UCF) in the NVIDIA Tegra410 SoC serves as a | |
| distributed cache, last level for CPU Memory and CXL Memory, and cache coherent | |
| interconnect that supports hardware coherence across multiple coherently caching | |
| agents, including: | |
| * CPU clusters | |
| * GPU | |
| * PCIe Ordering Controller Unit (OCU) | |
| * Other IO-coherent requesters | |
| The events and configuration options of this PMU device are described in sysfs, | |
| see /sys/bus/event_source/devices/nvidia_ucf_pmu_<socket-id>. | |
| Some of the events available in this PMU can be used to measure bandwidth and | |
| utilization: | |
| * slc_access_rd: count the number of read requests to SLC. | |
| * slc_access_wr: count the number of write requests to SLC. | |
| * slc_bytes_rd: count the number of bytes transferred by slc_access_rd. | |
| * slc_bytes_wr: count the number of bytes transferred by slc_access_wr. | |
| * mem_access_rd: count the number of read requests to local or remote memory. | |
| * mem_access_wr: count the number of write requests to local or remote memory. | |
| * mem_bytes_rd: count the number of bytes transferred by mem_access_rd. | |
| * mem_bytes_wr: count the number of bytes transferred by mem_access_wr. | |
| * cycles: counts the UCF cycles. | |
| The average bandwidth is calculated as:: | |
| AVG_SLC_READ_BANDWIDTH_IN_GBPS = SLC_BYTES_RD / ELAPSED_TIME_IN_NS | |
| AVG_SLC_WRITE_BANDWIDTH_IN_GBPS = SLC_BYTES_WR / ELAPSED_TIME_IN_NS | |
| AVG_MEM_READ_BANDWIDTH_IN_GBPS = MEM_BYTES_RD / ELAPSED_TIME_IN_NS | |
| AVG_MEM_WRITE_BANDWIDTH_IN_GBPS = MEM_BYTES_WR / ELAPSED_TIME_IN_NS | |
| The average request rate is calculated as:: | |
| AVG_SLC_READ_REQUEST_RATE = SLC_ACCESS_RD / CYCLES | |
| AVG_SLC_WRITE_REQUEST_RATE = SLC_ACCESS_WR / CYCLES | |
| AVG_MEM_READ_REQUEST_RATE = MEM_ACCESS_RD / CYCLES | |
| AVG_MEM_WRITE_REQUEST_RATE = MEM_ACCESS_WR / CYCLES | |
| More details about what other events are available can be found in Tegra410 SoC | |
| technical reference manual. | |
| The events can be filtered based on source or destination. The source filter | |
| indicates the traffic initiator to the SLC, e.g local CPU, non-CPU device, or | |
| remote socket. The destination filter specifies the destination memory type, | |
| e.g. local system memory (CMEM), local GPU memory (GMEM), or remote memory. The | |
| local/remote classification of the destination filter is based on the home | |
| socket of the address, not where the data actually resides. The available | |
| filters are described in | |
| /sys/bus/event_source/devices/nvidia_ucf_pmu_<socket-id>/format/. | |
| The list of UCF PMU event filters: | |
| * Source filter: | |
| * src_loc_cpu: if set, count events from local CPU | |
| * src_loc_noncpu: if set, count events from local non-CPU device | |
| * src_rem: if set, count events from CPU, GPU, PCIE devices of remote socket | |
| * Destination filter: | |
| * dst_loc_cmem: if set, count events to local system memory (CMEM) address | |
| * dst_loc_gmem: if set, count events to local GPU memory (GMEM) address | |
| * dst_loc_other: if set, count events to local CXL memory address | |
| * dst_rem: if set, count events to CPU, GPU, and CXL memory address of remote socket | |
| If the source is not specified, the PMU will count events from all sources. If | |
| the destination is not specified, the PMU will count events to all destinations. | |
| Example usage: | |
| * Count event id 0x0 in socket 0 from all sources and to all destinations:: | |
| perf stat -a -e nvidia_ucf_pmu_0/event=0x0/ | |
| * Count event id 0x0 in socket 0 with source filter = local CPU and destination | |
| filter = local system memory (CMEM):: | |
| perf stat -a -e nvidia_ucf_pmu_0/event=0x0,src_loc_cpu=0x1,dst_loc_cmem=0x1/ | |
| * Count event id 0x0 in socket 1 with source filter = local non-CPU device and | |
| destination filter = remote memory:: | |
| perf stat -a -e nvidia_ucf_pmu_1/event=0x0,src_loc_noncpu=0x1,dst_rem=0x1/ | |
| PCIE PMU | |
| -------- | |
| This PMU is located in the SOC fabric connecting the PCIE root complex (RC) and | |
| the memory subsystem. It monitors all read/write traffic from the root port(s) | |
| or a particular BDF in a PCIE RC to local or remote memory. There is one PMU per | |
| PCIE RC in the SoC. Each RC can have up to 16 lanes that can be bifurcated into | |
| up to 8 root ports. The traffic from each root port can be filtered using RP or | |
| BDF filter. For example, specifying "src_rp_mask=0xFF" means the PMU counter will | |
| capture traffic from all RPs. Please see below for more details. | |
| The events and configuration options of this PMU device are described in sysfs, | |
| see /sys/bus/event_source/devices/nvidia_pcie_pmu_<socket-id>_rc_<pcie-rc-id>. | |
| The events in this PMU can be used to measure bandwidth, utilization, and | |
| latency: | |
| * rd_req: count the number of read requests by PCIE device. | |
| * wr_req: count the number of write requests by PCIE device. | |
| * rd_bytes: count the number of bytes transferred by rd_req. | |
| * wr_bytes: count the number of bytes transferred by wr_req. | |
| * rd_cum_outs: count outstanding rd_req each cycle. | |
| * cycles: count the clock cycles of SOC fabric connected to the PCIE interface. | |
| The average bandwidth is calculated as:: | |
| AVG_RD_BANDWIDTH_IN_GBPS = RD_BYTES / ELAPSED_TIME_IN_NS | |
| AVG_WR_BANDWIDTH_IN_GBPS = WR_BYTES / ELAPSED_TIME_IN_NS | |
| The average request rate is calculated as:: | |
| AVG_RD_REQUEST_RATE = RD_REQ / CYCLES | |
| AVG_WR_REQUEST_RATE = WR_REQ / CYCLES | |
| The average latency is calculated as:: | |
| FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS | |
| AVG_LATENCY_IN_CYCLES = RD_CUM_OUTS / RD_REQ | |
| AVERAGE_LATENCY_IN_NS = AVG_LATENCY_IN_CYCLES / FREQ_IN_GHZ | |
| The PMU events can be filtered based on the traffic source and destination. | |
| The source filter indicates the PCIE devices that will be monitored. The | |
| destination filter specifies the destination memory type, e.g. local system | |
| memory (CMEM), local GPU memory (GMEM), or remote memory. The local/remote | |
| classification of the destination filter is based on the home socket of the | |
| address, not where the data actually resides. These filters can be found in | |
| /sys/bus/event_source/devices/nvidia_pcie_pmu_<socket-id>_rc_<pcie-rc-id>/format/. | |
| The list of event filters: | |
| * Source filter: | |
| * src_rp_mask: bitmask of root ports that will be monitored. Each bit in this | |
| bitmask represents the RP index in the RC. If the bit is set, all devices under | |
| the associated RP will be monitored. E.g "src_rp_mask=0xF" will monitor | |
| devices in root port 0 to 3. | |
| * src_bdf: the BDF that will be monitored. This is a 16-bit value that | |
| follows formula: (bus << 8) + (device << 3) + (function). For example, the | |
| value of BDF 27:01.1 is 0x2781. | |
| * src_bdf_en: enable the BDF filter. If this is set, the BDF filter value in | |
| "src_bdf" is used to filter the traffic. | |
| Note that Root-Port and BDF filters are mutually exclusive and the PMU in | |
| each RC can only have one BDF filter for the whole counters. If BDF filter | |
| is enabled, the BDF filter value will be applied to all events. | |
| * Destination filter: | |
| * dst_loc_cmem: if set, count events to local system memory (CMEM) address | |
| * dst_loc_gmem: if set, count events to local GPU memory (GMEM) address | |
| * dst_loc_pcie_p2p: if set, count events to local PCIE peer address | |
| * dst_loc_pcie_cxl: if set, count events to local CXL memory address | |
| * dst_rem: if set, count events to remote memory address | |
| If the source filter is not specified, the PMU will count events from all root | |
| ports. If the destination filter is not specified, the PMU will count events | |
| to all destinations. | |
| Example usage: | |
| * Count event id 0x0 from root port 0 of PCIE RC-0 on socket 0 targeting all | |
| destinations:: | |
| perf stat -a -e nvidia_pcie_pmu_0_rc_0/event=0x0,src_rp_mask=0x1/ | |
| * Count event id 0x1 from root port 0 and 1 of PCIE RC-1 on socket 0 and | |
| targeting just local CMEM of socket 0:: | |
| perf stat -a -e nvidia_pcie_pmu_0_rc_1/event=0x1,src_rp_mask=0x3,dst_loc_cmem=0x1/ | |
| * Count event id 0x2 from root port 0 of PCIE RC-2 on socket 1 targeting all | |
| destinations:: | |
| perf stat -a -e nvidia_pcie_pmu_1_rc_2/event=0x2,src_rp_mask=0x1/ | |
| * Count event id 0x3 from root port 0 and 1 of PCIE RC-3 on socket 1 and | |
| targeting just local CMEM of socket 1:: | |
| perf stat -a -e nvidia_pcie_pmu_1_rc_3/event=0x3,src_rp_mask=0x3,dst_loc_cmem=0x1/ | |
| * Count event id 0x4 from BDF 01:01.0 of PCIE RC-4 on socket 0 targeting all | |
| destinations:: | |
| perf stat -a -e nvidia_pcie_pmu_0_rc_4/event=0x4,src_bdf=0x0180,src_bdf_en=0x1/ | |
| .. _NVIDIA_T410_PCIE_PMU_RC_Mapping_Section: | |
| Mapping the RC# to lspci segment number | |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
| Mapping the RC# to lspci segment number can be non-trivial; hence a new NVIDIA | |
| Designated Vendor Specific Capability (DVSEC) register is added into the PCIE config space | |
| for each RP. This DVSEC has vendor id "10de" and DVSEC id of "0x4". The DVSEC register | |
| contains the following information to map PCIE devices under the RP back to its RC# : | |
| - Bus# (byte 0xc) : bus number as reported by the lspci output | |
| - Segment# (byte 0xd) : segment number as reported by the lspci output | |
| - RP# (byte 0xe) : port number as reported by LnkCap attribute from lspci for a device with Root Port capability | |
| - RC# (byte 0xf): root complex number associated with the RP | |
| - Socket# (byte 0x10): socket number associated with the RP | |
| Example script for mapping lspci BDF to RC# and socket#:: | |
| #!/bin/bash | |
| while read bdf rest; do | |
| dvsec4_reg=$(lspci -vv -s $bdf | awk ' | |
| /Designated Vendor-Specific: Vendor=10de ID=0004/ { | |
| match($0, /\[([0-9a-fA-F]+)/, arr); | |
| print "0x" arr[1]; | |
| exit | |
| } | |
| ') | |
| if [ -n "$dvsec4_reg" ]; then | |
| bus=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0xc))).b) | |
| segment=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0xd))).b) | |
| rp=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0xe))).b) | |
| rc=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0xf))).b) | |
| socket=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0x10))).b) | |
| echo "$bdf: Bus=$bus, Segment=$segment, RP=$rp, RC=$rc, Socket=$socket" | |
| fi | |
| done < <(lspci -d 10de:) | |
| Example output:: | |
| 0001:00:00.0: Bus=00, Segment=01, RP=00, RC=00, Socket=00 | |
| 0002:80:00.0: Bus=80, Segment=02, RP=01, RC=01, Socket=00 | |
| 0002:a0:00.0: Bus=a0, Segment=02, RP=02, RC=01, Socket=00 | |
| 0002:c0:00.0: Bus=c0, Segment=02, RP=03, RC=01, Socket=00 | |
| 0002:e0:00.0: Bus=e0, Segment=02, RP=04, RC=01, Socket=00 | |
| 0003:00:00.0: Bus=00, Segment=03, RP=00, RC=02, Socket=00 | |
| 0004:00:00.0: Bus=00, Segment=04, RP=00, RC=03, Socket=00 | |
| 0005:00:00.0: Bus=00, Segment=05, RP=00, RC=04, Socket=00 | |
| 0005:40:00.0: Bus=40, Segment=05, RP=01, RC=04, Socket=00 | |
| 0005:c0:00.0: Bus=c0, Segment=05, RP=02, RC=04, Socket=00 | |
| 0006:00:00.0: Bus=00, Segment=06, RP=00, RC=05, Socket=00 | |
| 0009:00:00.0: Bus=00, Segment=09, RP=00, RC=00, Socket=01 | |
| 000a:80:00.0: Bus=80, Segment=0a, RP=01, RC=01, Socket=01 | |
| 000a:a0:00.0: Bus=a0, Segment=0a, RP=02, RC=01, Socket=01 | |
| 000a:e0:00.0: Bus=e0, Segment=0a, RP=03, RC=01, Socket=01 | |
| 000b:00:00.0: Bus=00, Segment=0b, RP=00, RC=02, Socket=01 | |
| 000c:00:00.0: Bus=00, Segment=0c, RP=00, RC=03, Socket=01 | |
| 000d:00:00.0: Bus=00, Segment=0d, RP=00, RC=04, Socket=01 | |
| 000d:40:00.0: Bus=40, Segment=0d, RP=01, RC=04, Socket=01 | |
| 000d:c0:00.0: Bus=c0, Segment=0d, RP=02, RC=04, Socket=01 | |
| 000e:00:00.0: Bus=00, Segment=0e, RP=00, RC=05, Socket=01 | |
| PCIE-TGT PMU | |
| ------------ | |
| This PMU is located in the SOC fabric connecting the PCIE root complex (RC) and | |
| the memory subsystem. It monitors traffic targeting PCIE BAR and CXL HDM ranges. | |
| There is one PCIE-TGT PMU per PCIE RC in the SoC. Each RC in Tegra410 SoC can | |
| have up to 16 lanes that can be bifurcated into up to 8 root ports (RP). The PMU | |
| provides RP filter to count PCIE BAR traffic to each RP and address filter to | |
| count access to PCIE BAR or CXL HDM ranges. The details of the filters are | |
| described in the following sections. | |
| Mapping the RC# to lspci segment number is similar to the PCIE PMU. Please see | |
| :ref:`NVIDIA_T410_PCIE_PMU_RC_Mapping_Section` for more info. | |
| The events and configuration options of this PMU device are available in sysfs, | |
| see /sys/bus/event_source/devices/nvidia_pcie_tgt_pmu_<socket-id>_rc_<pcie-rc-id>. | |
| The events in this PMU can be used to measure bandwidth and utilization: | |
| * rd_req: count the number of read requests to PCIE. | |
| * wr_req: count the number of write requests to PCIE. | |
| * rd_bytes: count the number of bytes transferred by rd_req. | |
| * wr_bytes: count the number of bytes transferred by wr_req. | |
| * cycles: count the clock cycles of SOC fabric connected to the PCIE interface. | |
| The average bandwidth is calculated as:: | |
| AVG_RD_BANDWIDTH_IN_GBPS = RD_BYTES / ELAPSED_TIME_IN_NS | |
| AVG_WR_BANDWIDTH_IN_GBPS = WR_BYTES / ELAPSED_TIME_IN_NS | |
| The average request rate is calculated as:: | |
| AVG_RD_REQUEST_RATE = RD_REQ / CYCLES | |
| AVG_WR_REQUEST_RATE = WR_REQ / CYCLES | |
| The PMU events can be filtered based on the destination root port or target | |
| address range. Filtering based on RP is only available for PCIE BAR traffic. | |
| Address filter works for both PCIE BAR and CXL HDM ranges. These filters can be | |
| found in sysfs, see | |
| /sys/bus/event_source/devices/nvidia_pcie_tgt_pmu_<socket-id>_rc_<pcie-rc-id>/format/. | |
| Destination filter settings: | |
| * dst_rp_mask: bitmask to select the root port(s) to monitor. E.g. "dst_rp_mask=0xFF" | |
| corresponds to all root ports (from 0 to 7) in the PCIE RC. Note that this filter is | |
| only available for PCIE BAR traffic. | |
| * dst_addr_base: BAR or CXL HDM filter base address. | |
| * dst_addr_mask: BAR or CXL HDM filter address mask. | |
| * dst_addr_en: enable BAR or CXL HDM address range filter. If this is set, the | |
| address range specified by "dst_addr_base" and "dst_addr_mask" will be used to filter | |
| the PCIE BAR and CXL HDM traffic address. The PMU uses the following comparison | |
| to determine if the traffic destination address falls within the filter range:: | |
| (txn's addr & dst_addr_mask) == (dst_addr_base & dst_addr_mask) | |
| If the comparison succeeds, then the event will be counted. | |
| If the destination filter is not specified, the RP filter will be configured by default | |
| to count PCIE BAR traffic to all root ports. | |
| Example usage: | |
| * Count event id 0x0 to root port 0 and 1 of PCIE RC-0 on socket 0:: | |
| perf stat -a -e nvidia_pcie_tgt_pmu_0_rc_0/event=0x0,dst_rp_mask=0x3/ | |
| * Count event id 0x1 for accesses to PCIE BAR or CXL HDM address range | |
| 0x10000 to 0x100FF on socket 0's PCIE RC-1:: | |
| perf stat -a -e nvidia_pcie_tgt_pmu_0_rc_1/event=0x1,dst_addr_base=0x10000,dst_addr_mask=0xFFF00,dst_addr_en=0x1/ | |
| CPU Memory (CMEM) Latency PMU | |
| ----------------------------- | |
| This PMU monitors latency events of memory read requests from the edge of the | |
| Unified Coherence Fabric (UCF) to local CPU DRAM: | |
| * RD_REQ counters: count read requests (32B per request). | |
| * RD_CUM_OUTS counters: accumulated outstanding request counter, which track | |
| how many cycles the read requests are in flight. | |
| * CYCLES counter: counts the number of elapsed cycles. | |
| The average latency is calculated as:: | |
| FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS | |
| AVG_LATENCY_IN_CYCLES = RD_CUM_OUTS / RD_REQ | |
| AVERAGE_LATENCY_IN_NS = AVG_LATENCY_IN_CYCLES / FREQ_IN_GHZ | |
| The events and configuration options of this PMU device are described in sysfs, | |
| see /sys/bus/event_source/devices/nvidia_cmem_latency_pmu_<socket-id>. | |
| Example usage:: | |
| perf stat -a -e '{nvidia_cmem_latency_pmu_0/rd_req/,nvidia_cmem_latency_pmu_0/rd_cum_outs/,nvidia_cmem_latency_pmu_0/cycles/}' | |
| NVLink-C2C PMU | |
| -------------- | |
| This PMU monitors latency events of memory read/write requests that pass through | |
| the NVIDIA Chip-to-Chip (C2C) interface. Bandwidth events are not available | |
| in this PMU, unlike the C2C PMU in Grace (Tegra241 SoC). | |
| The events and configuration options of this PMU device are available in sysfs, | |
| see /sys/bus/event_source/devices/nvidia_nvlink_c2c_pmu_<socket-id>. | |
| The list of events: | |
| * IN_RD_CUM_OUTS: accumulated outstanding request (in cycles) of incoming read requests. | |
| * IN_RD_REQ: the number of incoming read requests. | |
| * IN_WR_CUM_OUTS: accumulated outstanding request (in cycles) of incoming write requests. | |
| * IN_WR_REQ: the number of incoming write requests. | |
| * OUT_RD_CUM_OUTS: accumulated outstanding request (in cycles) of outgoing read requests. | |
| * OUT_RD_REQ: the number of outgoing read requests. | |
| * OUT_WR_CUM_OUTS: accumulated outstanding request (in cycles) of outgoing write requests. | |
| * OUT_WR_REQ: the number of outgoing write requests. | |
| * CYCLES: NVLink-C2C interface cycle counts. | |
| The incoming events count the reads/writes from remote device to the SoC. | |
| The outgoing events count the reads/writes from the SoC to remote device. | |
| The sysfs /sys/bus/event_source/devices/nvidia_nvlink_c2c_pmu_<socket-id>/peer | |
| contains the information about the connected device. | |
| When the C2C interface is connected to GPU(s), the user can use the | |
| "gpu_mask" parameter to filter traffic to/from specific GPU(s). Each bit represents the GPU | |
| index, e.g. "gpu_mask=0x1" corresponds to GPU 0 and "gpu_mask=0x3" is for GPU 0 and 1. | |
| The PMU will monitor all GPUs by default if not specified. | |
| When connected to another SoC, only the read events are available. | |
| The events can be used to calculate the average latency of the read/write requests:: | |
| C2C_FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS | |
| IN_RD_AVG_LATENCY_IN_CYCLES = IN_RD_CUM_OUTS / IN_RD_REQ | |
| IN_RD_AVG_LATENCY_IN_NS = IN_RD_AVG_LATENCY_IN_CYCLES / C2C_FREQ_IN_GHZ | |
| IN_WR_AVG_LATENCY_IN_CYCLES = IN_WR_CUM_OUTS / IN_WR_REQ | |
| IN_WR_AVG_LATENCY_IN_NS = IN_WR_AVG_LATENCY_IN_CYCLES / C2C_FREQ_IN_GHZ | |
| OUT_RD_AVG_LATENCY_IN_CYCLES = OUT_RD_CUM_OUTS / OUT_RD_REQ | |
| OUT_RD_AVG_LATENCY_IN_NS = OUT_RD_AVG_LATENCY_IN_CYCLES / C2C_FREQ_IN_GHZ | |
| OUT_WR_AVG_LATENCY_IN_CYCLES = OUT_WR_CUM_OUTS / OUT_WR_REQ | |
| OUT_WR_AVG_LATENCY_IN_NS = OUT_WR_AVG_LATENCY_IN_CYCLES / C2C_FREQ_IN_GHZ | |
| Example usage: | |
| * Count incoming traffic from all GPUs connected via NVLink-C2C:: | |
| perf stat -a -e nvidia_nvlink_c2c_pmu_0/in_rd_req/ | |
| * Count incoming traffic from GPU 0 connected via NVLink-C2C:: | |
| perf stat -a -e nvidia_nvlink_c2c_pmu_0/in_rd_cum_outs,gpu_mask=0x1/ | |
| * Count incoming traffic from GPU 1 connected via NVLink-C2C:: | |
| perf stat -a -e nvidia_nvlink_c2c_pmu_0/in_rd_cum_outs,gpu_mask=0x2/ | |
| * Count outgoing traffic to all GPUs connected via NVLink-C2C:: | |
| perf stat -a -e nvidia_nvlink_c2c_pmu_0/out_rd_req/ | |
| * Count outgoing traffic to GPU 0 connected via NVLink-C2C:: | |
| perf stat -a -e nvidia_nvlink_c2c_pmu_0/out_rd_cum_outs,gpu_mask=0x1/ | |
| * Count outgoing traffic to GPU 1 connected via NVLink-C2C:: | |
| perf stat -a -e nvidia_nvlink_c2c_pmu_0/out_rd_cum_outs,gpu_mask=0x2/ | |
| NV-CLink PMU | |
| ------------ | |
| This PMU monitors latency events of memory read requests that pass through | |
| the NV-CLINK interface. Bandwidth events are not available in this PMU. | |
| In Tegra410 SoC, the NV-CLink interface is used to connect to another Tegra410 | |
| SoC and this PMU only counts read traffic. | |
| The events and configuration options of this PMU device are available in sysfs, | |
| see /sys/bus/event_source/devices/nvidia_nvclink_pmu_<socket-id>. | |
| The list of events: | |
| * IN_RD_CUM_OUTS: accumulated outstanding request (in cycles) of incoming read requests. | |
| * IN_RD_REQ: the number of incoming read requests. | |
| * OUT_RD_CUM_OUTS: accumulated outstanding request (in cycles) of outgoing read requests. | |
| * OUT_RD_REQ: the number of outgoing read requests. | |
| * CYCLES: NV-CLINK interface cycle counts. | |
| The incoming events count the reads from remote device to the SoC. | |
| The outgoing events count the reads from the SoC to remote device. | |
| The events can be used to calculate the average latency of the read requests:: | |
| CLINK_FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS | |
| IN_RD_AVG_LATENCY_IN_CYCLES = IN_RD_CUM_OUTS / IN_RD_REQ | |
| IN_RD_AVG_LATENCY_IN_NS = IN_RD_AVG_LATENCY_IN_CYCLES / CLINK_FREQ_IN_GHZ | |
| OUT_RD_AVG_LATENCY_IN_CYCLES = OUT_RD_CUM_OUTS / OUT_RD_REQ | |
| OUT_RD_AVG_LATENCY_IN_NS = OUT_RD_AVG_LATENCY_IN_CYCLES / CLINK_FREQ_IN_GHZ | |
| Example usage: | |
| * Count incoming read traffic from remote SoC connected via NV-CLINK:: | |
| perf stat -a -e nvidia_nvclink_pmu_0/in_rd_req/ | |
| * Count outgoing read traffic to remote SoC connected via NV-CLINK:: | |
| perf stat -a -e nvidia_nvclink_pmu_0/out_rd_req/ | |
| NV-DLink PMU | |
| ------------ | |
| This PMU monitors latency events of memory read requests that pass through | |
| the NV-DLINK interface. Bandwidth events are not available in this PMU. | |
| In Tegra410 SoC, this PMU only counts CXL memory read traffic. | |
| The events and configuration options of this PMU device are available in sysfs, | |
| see /sys/bus/event_source/devices/nvidia_nvdlink_pmu_<socket-id>. | |
| The list of events: | |
| * IN_RD_CUM_OUTS: accumulated outstanding read requests (in cycles) to CXL memory. | |
| * IN_RD_REQ: the number of read requests to CXL memory. | |
| * CYCLES: NV-DLINK interface cycle counts. | |
| The events can be used to calculate the average latency of the read requests:: | |
| DLINK_FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS | |
| IN_RD_AVG_LATENCY_IN_CYCLES = IN_RD_CUM_OUTS / IN_RD_REQ | |
| IN_RD_AVG_LATENCY_IN_NS = IN_RD_AVG_LATENCY_IN_CYCLES / DLINK_FREQ_IN_GHZ | |
| Example usage: | |
| * Count read events to CXL memory:: | |
| perf stat -a -e '{nvidia_nvdlink_pmu_0/in_rd_req/,nvidia_nvdlink_pmu_0/in_rd_cum_outs/}' |