Real-world unified monitoring for Data Center and Campus Networks with Splunk

Good morning, good afternoon, everyone, and welcome to today's APJC experts inside series suches. So my name is John Sweet. I'm part of.00:09For Cisco global customer success team and join me today is also from our global CSS team. We are really glad you could make time for this session.00:20And today we are going to talking about something that's extremely relevant for anyone's operating modern IT environments that is real world unified monitoring for data center and the campus networks powered by Splunk.00:31So I should say this is not a theoretical conversation, so everything we are going to show you today is based on the actual customer deployments and the real operational scenarios that way we had worked through together with our customers in APJC regions.00:48So let's get started. So here's our agenda for today's sessions. We will covering the four sections. Firstly, we will do a briefly Splunk overview to make sure everyone here has the same baseline understanding of the Splunk.01:07And 2nd, we will go deep into the Cisco and the Splunk integration for netops. That is the core session core part for the today's session. So we will look at the architecture and the integration patterns and more importantly the real customers case studies.01:26And the 3rd one, we will touch on the LLM integration for AI empowered offs as well. And the lastly, of course we will open it up for the Q and A We will keep time for your questions at the end. So please feel free to drop questions in the chat as we go. We will collect them and address them during.01:45The Q and A period.02:05So let's start from the Splunk. So what is Splunk? Simple answer is that Splunk is a data platform and it is the design to ingest, store, and analysis and visualize the data from the virtually.02:10Sources. So what makes Splunk unique compared to the traditional monitoring tools is that it doesn't require any predefined schema for the data. So what does mean? You don't have to tell the Splunk the data looks like before you send it.02:30This black will figure that out when search time. So let's let's say in a typical enterprise environments, you may have the hundreds, maybe thousands of different data sources, like from network devices.02:49Servers or applications. Actually, all of them these products produce the logs and the metrics in a different formats. So our traditional tools maybe often struggle with this part. They require you normalize the data before ingestion.03:06Which is a quite slow parts of the your operation things in the previously, but for for the Splunk, it flips that model, so you ingest 1st and you define the structure later when you need it. So this makes extremely flexible for handling the cancel flag comp.03:25Flex or mixed environments that our customers operate every day.03:45So this shows just how broad the Splunk's data collection capability is. So you can see the source of ranging from the network devices, the, and the syslog over the TCP UDP.03:54And also to the, you know, Windows host active directory or scripted API inputs all the way through the legacy main mainframes can be sending the data to the Splunk. So a few highlights worth calling out specifically for network operation is that network device.04:10This is actually feeds in via you know the syslog or streaming tele telemetry.04:30And maybe from the servers and hosts using the universal forwarder and any anything with an API, maybe cloudflow platform or maybe from the orchestration tools. Custom scripts can push data into Splunk very easily.04:36Yeah, this is the highlights or advantages from the Splunk platform. So in that means if your infrastructure produce data, then Splunk can consume it regardless of the source or format or the environment is implemented.04:54So the next, let's talk about the who uses, you know, Splunk and what they use it for. Firstly, Splunk is often sold as the primarily like a security tool that means threats detection or compliance parts and that is absolutely true.05:17But we see the increasingly we see a lot of Splunk being adopted by the IT ops or NetOps teams as their primary operational intelligent platform as well.05:37So you can see in the slides there have the four major personnels that Splunk serves. The 1st one is security ops. That means the security monitoring detection for that team.05:50And for the IT ops, maybe AI ops quite hard topics recently, right? And business service monitoring and the real call analysis in the IT ops parts. And for the netops, maybe net network performance monitoring, interface analysis and anomaly detection will be quite useful.06:05For that part for using the splunk. And finally, we having the as well for application monitoring or end user experience tracking in that part. So what important here is the concept that's the unified data platform.06:25So all of these teams are joining from the same underlaying data source. That means when a network incident impacts that application, when a security event correlated with a specific traffic flow.06:44All those connections can be made because the data is in one place from Splunk. So this is what we mean when we're talking about breaking down silos in IT operations then.06:59Splunk bridge, the IT operations and the bridge of business outcomes as well. So instead of just, you know.07:20Checking the interface plant in our network terminology or CPU spike in in isolation in the server side. Teams can correlate those metrics with an actual business impact, which that mean which services are affected and how many users.07:28For how long for the incidents. So in that case the shift in conversation is significant rather than in network saying we had an interface flap or BGP neighbor down, Instead you can say the banking service experienced 15 latency increased for.07:475 min and it's affecting 1000 users in the conversation.08:07So that's kind of the languages has the NO resonates the vision of the business and Splunk makes it possible. So with that side, standard Splunk deployments have gaps in the context.08:12And three we heard consistently.08:32We summarize them here. The 1st one is limited end user use cases visibility. That means the teams want to know whether a specific user or from the raw side maybe VIP is having a good experience, not just whether an interface is up or.08:36Is down. And second, we set summarized the limited net of coverage out of box. That means the scenarios like just like ACI fabric analytics or AI network performance required additional integration work.08:55And 3rd one, we should say the Splunk machine learning capabilities are powerful, but they still required initial deployments and validation against your specific environments. So these three are exactly the gaps that our Cisco and Splunk integration.09:13And work is designed to be closed.09:33So after introduce the brief of in Splunk platform, let's now move into the core section of the our today's session, the Splunk and Cisco integration for NetOps use cases.09:40Actually to understand why this integration matters, let's 1st look at these challenges. So, more than enterprise IT environments are noticed they spend multiple domains like this topology.10:01So the problem is that which of these domains has historically been managed in isolation? So different teams, different tools, different dashboards, and in between each domain.10:17There it is what we call is gray area, a boundary zone where incidents often originate but are very difficult to diagnose. So because NO single team owns the visibility across that boundary. So this is common problems when operating the modernized IT.10:34The, environments. So this is the fundamental challenge of modernized IT ops and it's why the customer keep telling us we know something is wrong, but we can't figure out where because our tool don't talk to each other at all.10:54So, on the left we have several Cisco platforms like a catalyst center, SD WAN, ACI, inter site or Nexus dashboard. I should say each platform excellent within its each owned domain, but not designed to give a unified.11:17Cross domain view. So on the right hand side, we see that customers actually needed a unified dashboard, anomaly detection, and the daily health checks, data sharing to the other team and also virtualizations monitoring.11:38All in one places for the platform. So the gap between the two is exactly the problem. So with NO single tool spanning both slides, the customer were bridging it manually.11:55So we may familiar with the using the open source pipeline to meet the gap or maybe using the custom API scripts to running the scripts, gather the data and pushing to the another platform. And maybe we're also having the some integration platform CMP to gather the.12:11More information from the platform to the other other systems.12:31For the insights. But for Splunk, it collapsed that middle layer entirely, so contact connect your Cisco platform to the Splunk and those operations requirements on the right hand side fulfill the directory.12:38So NO custom glue code needed. So then why the Splunk specifically for NetOps, they will I will summarize these three reasons. The firstly, the Splunk is well known and trusted platform.12:56Even not in network side, most enterprise customers already have Splunk running for security parts. Maybe extending it to the means NO new tool to learn. It's building on something that the team already owns and the trusts.13:16And second part flexibility to meet operational requirements. So that in that case traditional NMS tool are rigid, that means the fix the data model, fix the views, but Splunk lets you ask any questions of your data in real time and build exactly the dash.13:35For your team needs.13:55Without waiting for a vendors roadmap for the release. And 3rd one wide ecosystem and app support. That means there are official Cisco apps or other vendors app for the Splunk that provide prebuilt passers, dashboards, and alert templates.13:56So then you are not starting from the zero or from scratch.14:16So in that case, our integration work with customer in the field generally follow the three steps. So we always start by asking the customer or operation teams, why do you actually need it to see or what do you actually want to have in the insight?14:26So the answers typically fall into three patterns. We hear repeatedly across customers. Some customers they need a single dashboard, they can customize, not five different vendors consoles.14:44And some customer also feedback they needed the visibility into virtualized infrastructure and to want to share the data with other teams. And lastly, sometimes some customers they need the monitored AI infrastructure.15:00Network performance from one place. So this kind of requirements sound straightforward, but they defined everything that follows what data to collect, which from which platforms and how to stream.15:19Structure the dashboard. So getting this step right is the what's making the rest of the project succeeded as to say. So then once the requirement has been defined from the customer.15:35The next step is getting the data into the Splunk. So each Cisco platform exposes data differently, maybe Catalyst Center SD WAN, Nexus dashboard and inter site or have their own APIs or other approach for the.15:53Data sending. For network devices at the edge, we also using the netconf or API calls or maybe need some SSH access depending on the device capabilities. So all of this ingested as a we call it ME.16:10RT?16:29Metrics, events, logs, and the traces in the Splunk terminology. So the critical point, get the tele telemetry architecture right from the start, what you collect, how often, and how you process it determines the quality of everything build on top of it, top of.16:31Date.16:51And moving next with the data flowing has been collected or from the platform, the final step is building the dashboards the operation teams actually needed. So we consistently see three domain spaces, the categories for requirements. The.16:551st one is from the enterprise network. That means the campus wireless, and the user experience with catalyst center.17:15And from the data center network, it's more coming from the flow visibility or fault correlation and interface monitoring from next dashboard or ACI, and AI parts, maybe GPU utilizations and network performance and access log.17:23Is needed for this dashboard. So each domain gets its own dedicated dashboards maybe for this team's operation questions. In the next we will walk through real example of how of all these three in the case studies and follow.17:43So, we summarized the three case studies which are in real customers engagement from three domain. So the 1st one is coming from the enterprise networking. So in this customer, they were already optimizing their catalyst center and deployments.18:07And they decided to expand their Splunk use cases.18:27So their specific ask was around wireless quality visibility and another important one was way IP user experience monitoring. So they wanted to track in real time whether the critical users were getting the network experience they expected.18:31So in this case, we'll build the customer requested dashboard with the minimum additional data collection effort for this customer. And second customer coming from the network data center networking. So they are.18:50They deployed ACI and Nexus dashboard during a period of business expansion. And as the environments grow, they felt like the fault analysis become increasingly difficult for the operation team. So they pilot a Nexus dashboard upgrade with the.19:07And after expanding with an ACI based DR center, we delivered requirements based dashboard that gave them the flow based visibility and across system correlation they had me missing before.19:27And 3rd one is quite interesting one. This is financial service customer dealing with operational issues that span backbone and data center boundaries. They need to integrated monitoring across both Cisco and also the 3rd party environments.19:45So we worked through wisdom to align on and also pilot Splunk in production and are now progressing toward integrations. So let's go into details of for each cases.20:01So the catalyst center, was already deployed and managing the customers campus environments. The data was there. So client health or access point performance or application experience was what was missing was a way to surface the data in splunk.20:22Alongside everything else the team monitors.20:42So the goal for this case was straightforward. Put it, pull it in and build the views they actually wanted. So the 1st dashboard we created for this customer was overall of network health summary. It's like a top level campus health across sites.20:45Connected devices, active incidents, and network quality scores or from catalyst centers API in the same Splunk interface the team already uses.21:05And the next we had a wireless network performance dashboard is the indicates the per SSID or per AP metrics association with roaming events or clients comes software in real time. So data that previously in the catalyst center assurance.21:20Use is now continuously monitored, can be monitored and alertable from the Splunk.21:40And another dashboards, user experience monitoring. In that case, instead of just the infrastructure health, we are now tracking individual user experience. That means latency and jitter or packet loss per clients and correlated with their specific AP and SSID.21:52So the team can immediately tell whether a complaint client issues or an AP issues or when issues from the Splunk dashboard. And the feature this customer cared about most was a VIP user monitor.22:11Green is kind of like a predefined list of high priority users. That means executive levels or VPs or critical roles is tracked in the real time.22:31So if a way IP connection quality jobs below the threshold, an alert fires before they picked out the phone to the IT or network teams. So this capability simplified didn't exist before the Splunk integration.22:43And the next network device inventory management, it means the all devices from the catalyst center, switches, router APs put into the Splunk which model. So software versions and also it's a side.23:04And the status. So a living inventory always in sync with the operational data. So replacing the manual spreadsheets or separate the cmdbs for managing or checking this assessment of the campus network environments.23:19So all of this built with a minimal additional data collection. So already had it. So our Splunk just consumed the data.23:36The next we're coming to the data center network case study. So this customer migrates to the ACI during the business expansion. So ACI delivered the scale they needed, but actually as the environments grow diagnosing outages become increasingly different difficult.23:51So three pain points feedback gathered from the customer. One is growing east to west traffic with NO flow level visibility, and second one is limited post incident analytics from the native ACI tool, and 3rd one was NO consolidated view across force.24:11Flows or interfaces from ACL fabrics.24:31So they discovered the Splunk and Nexus dashboard insights, integration at the one of our events in 2024 call events. Then we run a POC with customer and they adopted for this use cases.24:36And after completing an ACI based DR center expansion, we delivered the full dashboard set and the next step, they will be upgrading the next dashboard to leverage the tech traffic analytics for even deeper flow level insights.24:53So similarly, let me walk you through what we built for this customer from the Splunk levels. So 1st one is flow monitoring. This is the flow visibility dashboard, and this is where the customer highest priority requirements.25:13So what we are seeing here is netflow or streaming telemetry data from the ACI fabrics. We passed it and visualized in the Splunk dashboard. So the team can now see top talkers by EPG or traffic volume trains over time and specific flow records for.25:32Any workload to workload conversation.25:52So when an incident happens, they can immediately answer which workloads were communicating on which paths and what were the volumes of the data in seconds, not hours. So this single dashboard replaced what previously required a dedicated.25:55No analysis two.26:14And second, fabric consolidated force. This panel consolidates all the ASA force and also the next dashboard anomalies into a single prioritized view of. So the force are categorized by severity and mapped to the affected objects, tenants EPG or physical interface.26:20Says.26:40And the key differences from the native ACI or NDI view at this here is the ports and anomalies are correlated with a timeline from with the other data source. So the team can see not just the what filed but the what else happening in the environments at.26:41The moment.27:01So which is essential for the root cause analysis. And this is the interface health dashboard covering all physical ports across the fabrics, interface utilization or arrow counters from the interface CRC errors.27:02Or input or output discards and the flat history are all checked for continuously.27:22So in that case, our netops teams can set stress forward and alert on any of these metrics without waiting for Cisco to add that capa capacity to the ACI or NDI native tools. So they are, they can also trend interface health over weeks or months. So we.27:28Which is invaluable for the capacity planning and identifying the degrading hardware before it causes an outage in the.27:48Fabric side. And for the endpoint monitoring dashboard, 1st it shows the overall number of the endpoints that are connected to the ACI fabrics and the top shows the endpoints, maybe a sudden job or spike is an immediate.27:57Signal that something has been changed in your fabrics.28:17And on the right side, the color coded status tiles gives an at a glass health view of each of endpoints by analyzing the endpoint ND endpoint health score. So the green is normal, but if you see the orange or red, that means there's an issue for that endpoint.28:20And also at the bottom the endpoint's connection details shows exactly which workload is attached, leaf or port EPG, IP or Mac, so the team can quickly confirm whether an endpoint is where it's supposed to be.28:42So moving next for the 3rd case study, is a financial service customer and unlike the previous tool is a multi domain or I should say the multi technology challenges. So the customer runs a critical banking networks and when an outage happened, it made a new.29:03In the.29:24In the TV as well, the banking app went down with around 2000 users in a queue before it's recovered. So quick monitoring coverage at the boundary between their SDM fabric and the backbone is caused this problem and also NO integrated dashboard across domain.29:25In the operation center is another reason for this one?29:45And incidents escalating faster than the team could diagnose, these three points caused the problem, I should say. So our approach was faced. Firstly, we established the AI based operational roadmap, then we analyzed the NetOps requirements across all platforms. Then.29:48We built Splunk based monitoring domains by domain. So a full digital financial network pilot is now in planning and as of this month, we are in active discussion on LLM and Splunk integration which will cover next. So.30:08Let me show you the dashboard for this part with the Splunk dashboard. So firstly, this is top level of unified dashboard for this customer, the single view that covers all network domains simultaneously.30:27So on one screen, the network teams can see the health status of backbone, the ACI data center fabric and digital finance segment together. And data from each domain is ingested into Splunk and normalized onto a common timeline and severity model.30:45So what this mean when an incident starts, the team NO longer has to log into the three separate tools and manually created what they see. They look at dashboard 1st and identify which domain is showing the issues.31:04And drill down from there, the domain specific panels respect the team's boundaries, and the backbone team and DC teams each see their own domain clearly. But the integrated view about enables cross domain correlation that wasn't possible before.31:19So the next this dashboard gives you the consolidated health summary across all nodes in data center. So every switches and router in the fabric on the one screen. So each nodes you can see the key operation metrics side by side.31:43Interface status or in the right side CPU and memory utilization, routing protocol agency state and any active force. So the 4th points is not to drill into any single devices here is to get an immediate answer to the question.32:00Across my nodes right now, is there anything abnormal? So something stands out a node with high CPU or routing agency that's down, a cluster of interfaces error, the netop seems spot as here and don't move to the next level.32:17So the next level is device level dashboard focus on the single device and showing everything that happening on it. So you are looking at the per interfaces details, utilization, operational states, and the historical trains.32:38And this is where the team comes once if an issue point out to a specific note. So from here you can confirm exactly which info face is degrading, whether it's a sustained trained or sudden spike or correlated if it's any relevant syslog events.32:55Or config changes on that devices.33:15So together this kind of dashboards give the teams two steps workflow to get all nodes for anomalies, then zoom into the specific devices to understand what's happening in the field. So that workflow allows significantly reduce the investigation time compared to the manually checking.33:18Each device is in its own management console.33:37So that concludes our workflow, works through our Cisco and Splunk integration for netops. So r3 customer deployments, r3 different domains, all coveraging or same answer Splunk as the unified operational platform across your.33:46Cisco infrastructure.34:05So moving next over to isab King for the next section for the LLM integration for AI in powered offs. So over to you. Thank you Joe. Just just one please. Hello everyone. My name is Isab. Now we move to the next part.34:06Of the today's session for AI import of operation.34:27Splunk provides three AI related features. 1st, AI hosted for SPL. SPL is quality language for the splunk. It's powerful, but it takes time to learn. With this AI.34:36You can write SPL very easily. It can also explain complex SPL in a simple way. Splunk store data collects from many different infrastructure, e.g..34:53Let's say you want to analyze network issue by correlating file logs with raw data. If you write, if writing feels difficult, you can simply ask the AI Austin for help.35:09Second, LLM plus Splunk MCP. Splunk MCP server connect your own LLM or cloud LLM to Splunk. This means your team can use the LLM.35:25You already have to analyze the data granted by Splunk. 3rd, AI toolkit. With AI toolkit, you can run your own LLM directly on Splunk. If your team does not have its own LLM, you can use the.35:40Now AI took it to log LLM on Splunk.36:00It's the range of LM for different operation needs. Operators can choose the proper LM they need. Now let's talk about a bit more about second picture, Splunk MCP.36:05Possible what is MCP? MCP stands for border context protocol. In simple word, MCP like is like user adapter, it connects LLM to outside of system.36:26Splunk provide the data condition, every row, every metric, every event from your network and infrastructure is ready already in Splunk. The splunk FP server let the LLM read and analyze all of this data inside Splunk.36:46The data collected by Splunk is stored in different time of the indexes, the Splunk. Using Splunk MCP server, LLM can check which index exists in Splunk and.37:07It can read data stored in the each index. You can download the server from Splunk base.37:23Then just download the file through the Splunk GUI and installation is complete. After that, you only need to set up the authentication so that your LLM can connect to it now.37:39So let me explain the Splunk architecture that include the Splunk MCP server.37:59This is, this slide shows the full architecture. On the left of slide, we have the data sources. Here we have network devices such as firewall, airport switches.38:05And etc. And SDL family, SDL controller such as apic, and Nexus dashboard, which is our network analytic solution. All telemetry data is collected from here.38:20In the middle, we have Splunk, the portal, portal collected data from the devices using various protocols such as netconf, API, SSH.38:38Which were explained earlier by John and data is sent through the folder stored on indexers and searched through the search head. As server is installed on.38:53Such head. On the right, you have user interface. The operation teams can use the dashboard as a basic interface, but when the team's LL is connected to.39:09Operator operators can also check data status through the LLM without opening or using any dashboard. E.g., operator can ask the LLM. What is the status of the.39:27Devices, find where flow drops, activities in the last hour. The used MCP server, Splunk MC server to get the data from Splunk and analyze it.39:42And gives the gives the answer back to the operator any service can collect and store every row battery, every event from the network and infrastructure. When you connect to.39:57Operators can check each of the scenario show show on this slide without building or using any dashboard. Without the operators would need to build separate dashboard for every scenario.40:18So now let me show you a demo of the. And this screen you see now is entropics cloud desktop.40:36It is already connected to let me ask the simple question. Let me ask, any devices with high CPU, when I answer the question.40:52LLM used the Splunk MCP to check the CPU city of the each device. This kind of information can usually be checked through the NMS or Splunk dashboard, but when you connect the LLM.41:08We can get answer easily without comparing each device with this one by one next here is 2nd example.41:24The question is, which device is server IP connected to? This is very common question in daily operation for network operators. E.g., when there is a service issue.41:43The 1st thing which operator check is which device related host is connected to. In the past, the operator had to login to each device to check the and check the ARP table using CLI.42:00If you have Splunk dashboard, we can search the by search by IP address on the host dashboard or with LLM integration with Splunk, the answer comes back immediately. It shows the.42:19Connected device interface and other useful information. Next so the question is, let me know where traffic is spiking.42:35LLM will report to which interface or device are spiking. This kind of function is what every operator really wants, right? With this integration, you just ask the question to LLM.42:52And you get, you can see the result right away and in this demo, LLM tells us that there are there is NO traffic spike at the moment but it also mentioned a spike from the past time.43:10Because Splunk consciously collect all the data and so the can also analyze past event, not just current status and this is the last example.43:30E.g., question was the free pair the daily operation report. With one instruction, the LLM create a full daily report. The daily report should include many check items.43:48In this demo, the LLM checks their talk statusion, resource usage, and connection issue for each host. LLM check each item one by one from Splunk.44:06And then makes a clean report that is easy for operators to read.44:22I can see theory.44:42Through daily report. And normally this kind of report is created through automation, automated process, but when the report item or need to change, we also change, we also have to change the automation process.44:48With LLM integration with Splunk, adding new item to the existing report becomes very simple and generating one report takes only a few minutes. And so far I have shown you how connecting LLM with Splunk server can simply.45:06Apply the two operations.45:25Now the back to John. Thank you. Okay, thank you so for the impressive demo. So yeah, this is our 1st session for the Splunk and the introductions of how Splunk.45:27And Cisco products can integrate it.45:47So before we're moving open up for the Q and A, let me just leave you a few some key takeaways for today's sessions. The firstly, the end to end visibility across network domains is critical for stable operations. That means that the gray areas which.45:49Talking about intersection between domains, between campus and the backbone between SDM fabrics and upstream networks, where the incidents originate and where the traditional tools go blind. So using this kind of Splunk uni unified monitoring that spans all domains.46:09Isn't nice to have it it's the foundation of reliable IT operations and second part Cisco and Splunk integration announced the netops with deeper insights and faster troubleshooting. So we shared the three customer cases with today or demonstrated the or the same pattern. The Cisco plan.46:29Platform or other vendors will provide lots of rich operational data.46:49And Splunk provides the unified layer to correlate it or visualized and act on the data across all domains. So all together they closed the gap that NO single two could close on its own. And 3rd one, the LLM integration with Splunk actually enables the next.46:54Is for AI driven operation is means the AI ops. So we are now at the point where the operational data is collected and correlated. The next step is making the data accessible and actionable through the natural language. So that is very every member of the operation teams, not just.47:14To the power users can query and understand the networks. That is what integration delivers and is the direction we are actively moving with our custom today.47:34So yeah. Okay, so that brings us the Q and A sections. If you have questions, please using the Webex Q and A panel to submit it. We will walk through the questions from there and also we will sharing the all the materials and the recordings after the session.47:48Questions and if you have something regarding.48:08Today's topic regarding the Splunk integration or deep dive sessions or other kind of discussion you needed. Please I just shared one email address in the chat window. You can refer that email email, you can send us the email. We will follow it with the further discussion.48:12So with that let me check the current Q and A panels for the questions. Let me check.48:33So the 1st question we had the from customer, how long does it take to build this kind of network monitoring dashboard? Yeah, it's ok, can you answer for this one?48:48Okay, if the customer use the dashboard already provided this, you can the customer can start very quickly, but however, if customer need to need a custom dashboard.49:05We typically work with partner for requirement analysis, data collection design and dashboard-dashboard development. This usually take about one month I think.49:21Okay ok thank you yeah and 2nd one I see that is quite frequent FAQ in other customers engagement as well is regarding the data size. How much the data does the Splunk handle from a typical networking.49:42Do you have any insight for this one? Based of the reason our customer case for ACI fabric with about hundred switch and together with the dashboard.50:01We estimate about three gig, 3 gb for day per day that this include the flow data from dashboard and but actual volume can vary depending on how much flow data is generated.50:18Right, so this is the just one replicase, so the actual size actual sizing should be compound based on the the each customer specific environment.50:38Okay. Thank you, thank you for the question answer. The 3rd one will be how accurate are the answers. Can we trust them for the operations with a spoke MCP, the does not get the.50:53The answer. It runs real and data. So the answers are grounded in the customer's actual environment. In addition the by using lack.51:11To provide to provide Splunk better data to LLM the customer get more accurate and ok thank you and I see that there's some questions regarding the the demo and.51:28That the size we showed today. So to that answer that questions for today's sessions is only for the demo environments is only for net Cisco network internal lab. So there's NO actually NO.51:47Having the, that demo lab you can access or we can provide the credentials for your access. But if you need it for the support or discussion, please let us know, send me the email, we can follow up with the engagement for the further.52:05Discussion. Yeah, so I think another maybe last question would be.52:22Do you have used cases when integrated with other products when collected the log through the API or syslog or SNMP? So I think you mean the use cases from the other vendor?52:39Or if just for for common answers for API or other vendors supported, if the product supports the API or other formats of the data can be collected, then Splunk can covers the data format and to the dashboard.52:55Yeah. Okay, and yeah, let me check and also for the studio study material for this one.53:16Maybe 1st we will be sharing the this EIS sessions material recording as well to your to your perspective and we can refer and for the other sites we can having the more discussion and summarize the links if we have, we can have it.53:33Okay, so, yeah, if you have any other questions you can contact us for the with the email. We will follow that and.53:54That's it, yeah. So thank you for all for joining today's sessions. We hope this give you a one is overview and the one is practical and the concrete pictures of how Cisco and Splunk integrated looks like in the real production environments.54:13So I mentioned the recording and the materials from today's session will be made it available after the session. So if you found this useful, please do share this with your colleagues as well. So thank you again and we look forward to seeing you in the next experts series. Thank you. Thank you everyone.54:31