Welcome to today's presentation about integrating Nexus dashboard 4.2 with slurm. Our focus is to understand how to further improve job correlation. We will begin by examining the current landscape in which these tools operate and we.00:05We will lay the groundwork for deeper integration. Let's begin by outlining our agenda for today.00:201st, we'll outline the operational challenges in AI clusters. Next, we will examine the reasons behind choosing the Nexus dashboard. After that, we will describe how Slurm integrates with the Nexus dashboard. We will also explore DCGM Exporter and fleet collector before explaining how all of these components work together.00:26Ultimately, we will wrap up with a concise recap of the main points we have discussed.00:47In this section, we will examine the practical challenges that arise within AI and ML clusters. Understanding these challenges is essential for boosting performance and improving the overall efficiency of the system. Now, we will outline the areas where problems typically arise and show how AI ML clusters can.00:52Continue to face several operating challenges.01:121st, scaling GPU clusters from hundreds to thousands presents a major challenge indeed. Second, idle GPUs increase wasted capital and delay time to value, underscoring resource efficiency problems. Next, limited operational visibility arises from the lack of integration between job scheduling.01:17GPU health monitoring and network behavior across distributed networks. Finally, troubleshooting requires manual cross tool correlation across several utilities, a process that is both time consuming and inefficient in practice. Our customers commonly report three major issues indeed. 1st, they encounter G.01:36Few performance problems where it is unclear whether the network, a faulty optic or hardware failure is to blame.01:57Second, there is a lack of integration between separate dashboards for network operations and GPUs leading to siloed information. This siloing hinders the ability to diagnose issues efficiently. Lastly, delayed identification of the root cause often results in job failures causing frustration among data scientists.02:03Now let's discuss the Nexus dashboard and why it's a compelling choice for AI cluster management. The Nexus dashboard delivers unified visibility and precise control helping organizations overcome the operational hurdles we noted. Its ability to integrate a range of functions makes it an invaluable asset.02:23One failed GPU can cut your AI cluster performance by up to 40 %. Using Nexus dashboard 4.2, you can link infrastructure anomalies to affected AI jobs in real time. Nexus dashboard 4.2 offers advanced robust monitoring capabilities for AI tasks with a few key features.02:441st, it provides end to end visibility enabling comprehensive monitoring across the entire stack. Secondly, its topology aware visualization allows for the correlation of network elements with hardware components crucial for understanding infrastructure connections. 3rd, real time metrics tracking for throughput.03:05Latency and GPU utilization helps extract actionable insights. Lastly, its proactive troubleshooting detects anomalies against historical baselines, expediting root cause analyses with automated recommendations. To summarize, indeed, with Nexus dashboard 4.2.03:24We have faster insights leading to faster resolution. ND, it provides a unified view. Nexus dashboard aggregates data across the key domains in our system. In the area of job scheduling slurm acts as the data source, delivering critical visibility into job status.03:42Titioning runtime, user data, and GPU mapping. For GPU health, we rely on nvidia's DCGM exporter for critical real time metrics such as usage, RAM, temperature, and power. On the network side, Nexus switches are used delivering insights into optics, congestion, drop data packets.04:02RC errors and other anomalies.04:22This integration provides full visibility across all components enabling better decision making and improved operational efficiency. Now we focus on the essential solution architecture components of our system. At the core sits the Nexus dashboard 4.2, offering unified analytics.04:25X control and visualization. Slurm serves as the job scheduling source of truth ensuring accurate and reliable job management. Nvidia's DCGM exporter provides essential GPU health metrics through its metrics endpoint.04:45Additionally, the GPU Nick topology fleet collector maps out the physical topology between GPUs and NICs, improving compatibility and performance insights. The link layer discovery protocol or LLDP enables endpoint and topology discovery.05:00While MTLS provides optional secure telemetry collection to safeguard data transmission and integrity, what key outcomes do we seek from this integrated approach? We aim to speed up incident resolution by quickly identifying the root causes of job degradations, lowering the average time to resolve issues.05:18Is across platforms.05:38Improving GPU performance remains a priority, focusing on detecting and solving bottlenecks early. Network optimization reveals link level inefficiencies and aligns them with loads to prevent issues. The core objective is to maintain a single cohesive, unified operational workflow that consistently.05:40Provides standardized processes across AI routed and VXLAN EVPN fabrics.06:00Finally, we ensure auditability and accountability by maintaining comprehensive logs for every job in fabric, thereby enhancing transparency and traceability. We will now examine slurm integration in the Nexus dashboard.06:05This integration is essential for improving our job scheduling capabilities. Slurm is a core component delivering seamless AI alignment with HPC workflows across scales. This integration ensures job states, scheduling data, and resource allocations are managed efficiently, thereby boosting overall system.06:20Performance and reliability. So what is slurm? And why is it central to our daily operations?06:40Slurm stands for simple Linux Utility for resource management, and it has become the de facto widely adopted open source norm for AI and high performance computing clusters. It provides essential context about jobs, including job identifiers, GPUs or servers in use, the responsible user, job prioritization.06:46And duration. Next context is crucial for managing resources in an efficient and effective manner. Additionally, the integration of slurm enables full end to end correlation between network elements and their workloads, delivering higher visibility and more consistent operational performance.07:06Moreover, analytics are enhanced through slurm integrations. The AI resources overview component furnishes insights into AI clusters, scalable units, servers, live jobs, and GPU usage. A performance analytics dashboard provides a platform for job performance correlation boosting resource management and efficien.07:24Network performance metrics show network health by monitoring optics, congestion, errors, GPU metrics focus on GPU health, use, memory, temperature power as well. Predictive monitoring is also utilized for proactive anomaly detection, further enhancing system reliability and performance.07:45What are the steps to take to add slurm integration to the Nexus dashboard? Begin by navigating to the admin section, selecting integrations, and then add integration. Choose slurm as the integration type and proceed to enter authentication details, which includes the host name or IP address and necessary.08:05Credentials like username and password.08:24After reviewing the provided information and ensuring everything is correct, save your configuration. The integration will move from an initialized state to active, fully integrating slurm into your Nexus dashboard setup, ready for use. In order to view job details for AI fabrics, you must 1st integrate slum.08:27An open source job scheduler used to manage resources on high performance computing clusters. The data center GPU manager, more commonly known as the DCGM exporter, plays a pivotal role in enabling the detailed collection of GPU metrics, a topic we will examine in depth across.08:46That's the slides that follow. Understanding these integrations is key to optimizing AI workloads in complex systems. To collect GPU statistics, the DCGM and video data center GPU manager exporter should run on the host. DCGM exporter is a tool based on the Go API.09:06Is to Nvidia DCGM that enables users to collect GPU metrics and to understand workload behavior or to monitor GPUs within clusters.09:26The DCGM exporter exposes GPU metrics at an HTTP endpoint intended for monitoring solutions such as Prometheus. The DCP metrics included in the CSV file provide a list of counters to collect. This file is required right at the start of the DCGM exporter process.09:37We now focus on the fleet collector fully integrated into the nexus dashboard. This component is essential for preserving the operational reliability of our AI clusters by collecting and examining vital data.09:56Next we will study its functions and importance now. The GPU Nissy topology fleet collector is a tool that gathers snapshot data from AI servers to improve visibility into AI infrastructure. On Nexus dashboard, data maps the links between GPUs and NICs on a server, helping closely correlate.10:11GPU and network issues.10:31The GPU NIC topology fleet collector collects GPU information, model UUID, PCI addresses, network interface cards, NICs with specifications, GPU Nick topology distances, network interface details, IPs, max, drivers, DCGM.10:33Monitoring deployment status.10:53So what happens when you trigger an AI job in slurm? We can watch Nexus Dashboard 4.2 auto create a job specific topology showing exactly which GPUs and switches are involved in that job. Nexus Dashboard 4.2 closes the communication gap between data scientists and network engineers.10:55Enabling joint troubleshooting with shared task level network visibility, so stakeholders stay aligned and issues are resolved quickly across teams today.11:15This is how the freshly released nexus dashboard 4.2 appears. We feature an AI resources card that displays the count of AI clusters we run and what each cluster contains, backend, front end management, and storage, all governed and monitored through Nexus dashboard.11:25There are two distinct levels of AI job visibility available within ND. At the fabric level, users manage jobs within a specific AI fabric. The system level provides an overview across all AI fabrics. Additionally, we offer a job level deep dive feature allowing.11:42In detailed exploration of each jobs elements, including its overview, topology, optics, and GPU status. This highly granular view enables advanced analytics, anomaly detection, and activity tracking.12:00Thereby enabling precise and effective management of AI workloads today. We have introduced an AI job dashboard that displays all AI jobs in progress and shows how these jobs relate to the fabric that underpins the back end. It shows AI job details, start time, who started it, finish time, and whether there is a problem.12:15In the network or in the AI infra that affects job performance. So click on the jobs to learn more about it.12:35In this example, we can observe that out of twelve total live jobs, five are running while four are suspended currently. On the dashboard, five jobs that are now facing anomalies categorized into critical, major, and minor issues. This visualized data supports clear detection and resolution of these job related anomalies.12:41Enhancing overall job performance and reliability. We present the anomalies that affect the job.13:02They may involve the GPU, the optics or issues with network connectivity such as congestion. The analytics tab provides a more detailed view of drops, CRC errors, and power and temperature issues summarized. If any value crosses a threshold, the graph signal turns red and flags an anomaly.13:08We also have a GPU card to get detailed GPU information. If any GPU anomalies affect the AI job, they will also be highlighted. There's an optics tab with detailed information for each interface. We have a topology view to thoroughly inspect the connectivity in detail and to see where AI job.13:28Jobs run, e.g., on which GPUs or servers and how these connect to my leaf switches. The topology view will also show any anomaly on any of these components where a job runs as seen in the topology.13:48The anomalies are clearly summarized on the right hand side. By clicking on the anomalies, I can see what triggered each one, what the impact is, and how to fix it. Therefore, we can reach the root cause faster and resolve the problems. E.g., if max temp threshold is exceeded on one of the leaf interfaces.14:02It consequently causes a CRC error and a drop, which affects the AI job. I can clearly see it as an anomaly. We've now thoroughly explored how the Nexus dashboard's integrated tools enhance monitoring and management of AI workloads. Briefly summarizing the benefits of adopting slurm integration.14:20We clearly focus on key advantages such as improving time to value by delivering operational insights swiftly after integration.14:40The approach enhances GPU return on investment by rapidly driving up utilization at AI scale. It emphasizes risk reduction by proactively detecting issues well before SLAs are breached and supports future readiness for AI fabric expansion. Adoption is underpinned by open standards ensuring that.14:49There's NO vendor lock in and culminates in a single pane of glass for efficient smooth coordination between network and machine learning operations. Integrate slurm into your nexus dashboard today.15:09
