#Gpu memory monitor how to

See Use GPUs for compute-intensive workloads on Azure Kubernetes Service (AKS) to learn how to deploy an AKS cluster that includes GPU-enabled nodes. See Workbooks in Container insights for a description of the workbooks available for Container insights. However, for various reasons, the GPU-Z Memory Used counter may be below the amount of available dedicated video memory but the. GPU performance chartsĬontainer insights includes pre-configured charts for the metrics listed earlier in the table as a GPU workbook for every cluster. Monitoring the amount of per-GPU used video memory with a tool such as GPU-Z, reducing the texture quality (to lower the video memory footprint), or testing with a GPU with more video memory can give you hints. Once the upstream changes reach GA, this fix will not longer work, make plans to migrate to using your GPU vendor specific metrics exporter by December 31, 2022. For Arc enabled Kubernetes, enable feature gate DisableAcceleratorUsageMetrics=false in Kubelet configuration of the node and restart the Kubelet. * Based on Kubernetes upstream changes, these metrics are no longer collected out of the box, as a temporary hotfix, for AKS, upgrade your GPU Node pool to the latest version or *-2022.06.08 or higher. Number of GPUs in a node that can be used by Kubernetes. It is not possible to request or limit a fraction of a GPU.Īmount of GPU Memory in bytes available to use for a specific container.Īmount of GPU Memory in bytes used by a specific container.Ĭ/clusterId, /clusterName, gpuVendor It is not possible to request or limit a fraction of a GPU.Įach container can request one or more GPUs.

Duty cycle is a number between 1 and 100.Ĭ/clusterId, /clusterName, containerNameĮach container can specify limits as one or more GPUs. Percentage of time over the past sample period (60 seconds) during which GPU was busy/actively processing for a container. Metric nameĬ/clusterId, /clusterName, containerName, gpuId, gpuModel, gpuVendor

#Gpu memory monitor driver

Container insights collect GPU metrics through GPU driver pods running in the node. After provisioning cluster with GPU nodes, ensure that GPU driver is installed as required by AKS to run GPU workloads.