Monitoring Best Practices with Splunk Integrations

You're on mute Joe? Thanks. Hello and welcome everyone to today's Cisco expert insights series. Today's topic will be monitoring best practi.00:36Practices with Splunk integrations.00:56And we'll have our customer success specialist, Andrew Pignataro and JT Stam doing our presentation and demonstrations for you today. Now we will be using Slido.01:01For any questions you have, you can either ask them in the chat or preferably in Slido. If you go to slido.com, you can enter the join code that you see eIS-DC-Splunk number two.01:18And then the passcode of Splunk 123, or probably more easily you can use your mobile device such as a phone or tablet and scan the QR code, and that'll produce a hyperlink that you can click on to go directly to it.01:33Now we will have a couple of questions, so I please encourage you to go ahead and get started and joined in now. With that, I'll turn it over to Andrew and he will get us started. Over to you, Andrew. Great. Thank you Joe.01:50So that brings us to our 1st slide of question, and let me go ahead and share my screen.02:06Which of the following best describes your primary area of responsibility?02:26Think data center and compute, networking connectivity, cloud SRE or devops, security compliance, IT architecture and leadership or other. Let me give you guys just a minute to go ahead and fill out the Slido.02:31You see some results coming in from network connectivity secure.03:12Meeting compliance.03:15IT architecture and leadership then we'll go ahead and continue on. Looks like my.03:19Severe from networking and connectivity?03:39So just to give you kind of a quick overview on the full of this session today, we're gonna be going over Splunk's capabilities and within Cisco data center environment. So the 1st little section is gonna kind of be a show me how of showing how Splunk works.03:43And some of the different, we'll do a review over the different Splunk apps related to data center. Then we'll go to in exploring with us where we'll go over those data center apps and we'll have some demos for you. Then finally at the end, we'll kind of summarize what we went over.04:00Open it up to Q and A And of course if you still have further questions or want to meet with us, you can look to do a one on one consultation. Now, what slows operations down, we have a lot of different tools and a lot of different vendors.04:18Along with a lot of different teams managing them. This is gonna create visibility gaps. Teams are split into different silos managing their own platforms and their environments, right? And we're looking at the problem.04:35At different layers, but not at an overall view of what's going on. This is gonna make it hard to troubleshoot, I often end up ending up being reactive instead of proactive. When teams.04:51Sift through all the noise and finally understand that there's an issue. You have war rooms where everyone's looking at the data, but NO one's really agreeing on what's going on. And it gets worse when something outside your control happens, like an ISP suddenly redirecting traffic.05:07We lose visibility and the experience is gonna fall apart. So, what can we do about this? Thankfully thankfully, we have a solution in Splunk that can provide full stack observability.05:24They're all the different layers. So you can see down here we're starting with key storage and compute and the infrastructure layer with the network, right, working our way up to the visualization layer, with the hypervisor and operating system.05:40Along with Kubernetes, finally the application layer, then security wraps all the layers together. Then Splunk has the ability to ingest, analyze, and correlate all this information then Splunk's gonna gather all these different data metrics through.05:59These metrics, events logs, and traces, and Splunk is vendor agnostic, so there's many different apps and add ons for Cisco and other vendors and solutions as well. So Splunk provides.06:20A unified.06:40Observability experience, as you can see here. Today we're gonna be focusing on the ones highlighted in blue, which are gonna be infrastructure monitoring and network observability specifically related to data center networking. So if you're not familiar with Splunk already.06:42There's a couple different deployment options you have. The 1st one on the left being enterprise.07:02Networking, which is gonna be your on prem solution, and then on the right being your cloud option, right? And for Splunk Enterprise on prem, you can deploy it on a VM, on Linux, let's say a Windows server, bare metal server.07:07You have a bunch of different options. Then you can deploy Splunk folders, which are gonna act as proxies that's gonna forward the data. And these folders gather data through different API queries, CLI commands, syslog.07:23SSH or or even webhooks. Then Splunk is gonna act as a searchable database where all this data gets indexed and stored, allowing you for fast and optimal searches on your data. Then for the cloud.07:39The forwarders act are, the forwarders are gonna act as a secure point for you to connect your network to the cloud and it will transmit the data. For both of the options, the pricing models are based on how much data.07:57Is ingested per day or how much compute and the V CPUs that you're gonna be using in your instances consuming. And then if you're debating between whether you want to work with on prem or on cloud, every new feature.08:14Available on Splunk is gonna be cloud 1st. And for the cloud you don't have to worry about maintaining your environment such as upgrades or where you have to worry about capacity planning, and it's gonna be the cloud is gonna be updated and maintained by Cisco. So here.08:32Within Splunk you have apps and add ons.08:52Splunk provides an app store in a way called Splunk base. And you may see the term app or technology add on. And the difference between the two is an app is gonna be a fully packaged solution, which usually contains a visualization or a dashboard while technology add on.08:55Is more of gathering data inputs and this technology add on is used to help build other apps. Then every one of the apps on Splunk Base is inspected and vetted by Splunk. Make sure they follow security compliances and best practices.09:14Then speaking of the different apps, Cisco has a bunch of different Splunk integration solutions. As you can see here up in the top, we have an app for Meraki Catalyst Center, D Cisco DC networking.09:35And Cisco Intersight add on, which we'll be kind of getting into a little bit later today in the presentation. Then in order to collect this telemetry data, Splunk's gonna use netconf, APIs, SSH.09:51Right, to gather these metrics, events, logs, and traces between all these different environments. And as we talked about all these, all the visibility gaps you have from working with all these different teams, right? Splunk is gonna bring them all together in one place for you to view.10:06All at once, so it can help bridge the gap for that issue. Now, Splunk for Splunk, you can, if you never used it before, all you need to do is go into splunk.com and sign up for.10:24A, a login. Then once you have a login and username and password, right, you can start your free trial, you can start a free trial on Splunk enterprise or Splunk Cloud. The Splunk enterprise, you should be able to get a a trial enterprise license for 60 days.10:41Then after the 60 days it converts to a free license. For the cloud, I believe it is a two week, 14 day free trial that you can go ahead and test it out for your network. And all the apps that we're talking about are free today, so you should have NO issue downloading them and trying them out.11:00Yeah for the Splunk apps for Cisco Data Center, on the left hand side, we have the Cisco Data Center Networking app, which is comprised of ACI.11:16Nexus dashboard and standalone nexus nine thousands. And you can either go to that Splunk base that we mentioned before and search up Cisco DC networking app or you can use the URL. For the data center, we also have the inter site app.11:32The Cisco Intersight add on.11:52Then you also have the option to use the Splunk Cisco app navigator, which is a collection I believe about 30 or so different Splunk Cisco Splunk apps, in which it should have the following DC networking app and inter site add on app and as well you can check out all the other different offerings.11:54That we have. Now I want to talk a little bit about native Splunk and Nexus dashboard. Right, so a Splunk instance can run inside a Nexus dashboard cluster on version four.two of Nexus dashboard.12:13Kubernetes pods are deployed in order to run it. Then the 1st release supports a single node of G five which is a UCSM eight deployment only, right? So support for aG5 cluster is targeted for the four.four release.12:30At the end of the calendar year, the Spunk license is included with the DCN premiere, so work with your account team if you're already running it and they can help you get that set up. And if you don't have premiere, you also have to work with your account team for ordering the right licenses and SKUs.12:50So, once you have Splunk downloaded and enabled for four.two, Nexus dashboard is automatically gonna push the instance and start collecting the data. And you can see here in the bottom right hand side that the app is provided.13:13Under the analyze tab, then going into the app, we have a whole bunch of prebuilt dashboards and inputs already set up for you to use. Then this is exclusively for Nexus dashboard traffic. You're not able to add any other external apps or integrations.13:31Into this, then we'll go ahead and take a look at a live demo for the four.two. I'll pass it to JT, and he'll help you take a look. Thanks Andrew and hey everyone, well 1st of all, thank you for being here and excited to show you what.13:49Native Splunk looks like on Nexus dashboard four.two. So I have a4.two instance right here. So I'm running Nexus dashboard 42110I believe that's the one that's on cisco.com or the latest one that you'll find on cisco.com. Yeah, this is our 1st release that is supporting Splunk as a native instance on the same platform.14:10So you don't need separate hardware, you don't need to install a separate VM for this. However, as Andrew mentioned, there are some licensing considerations. So again, if you have premier, then you're entitled for this, but you'll need to work with your account team to obtain that license, and if you don't have premiere, then still work with your account team, but yeah, there would be an additional cost to that.14:29So to get started with this, you need to be running four.two, of course, this is only supported on the G5S single node deployment today, but as Andrew mentioned, towards the end of the calendar year, we're looking forward to our four.four release, which is expected to have cluster support.14:49Alright, so to get started, I'm gonna go here to admin and then system settings and here is where we're going to be able to enable Splunk. So under your advanced settings, so remember to scroll all the way down for this, you'll see my status for Splunk is enabled. So it's here that you will go to edit and then supply your license key and.15:06And then you can just simply click enable, give it a couple moments and then it'll spin up that Splunk instance. It's actually running on Kubernetes on the back end, just a little fun tip there. I mean as is the rest of Nexus dashboard, so it's a containerized instance of Splunk. Alright, once you're done with that, then you can go to analyze.15:26And then analysis hub. Now if you've been working with Nexus dashboard for a while, this, you know, this view probably familiar to you, but now we get an extra tile here, so we can launch into a Splunk instance. Now we are going to have to perform an additional authentication here. Now, for the native Splunk instance, this is going to essentially have the same login.15:46In a database like user database as Nexus dashboard itself. So if you create a local user on Nexus dashboard, then you'll be able to use that here on Splunk. Now there is a little bit of an additional detail there, so of course we have the local admin on Nexus dashboard. In this case for Splunk native Splunk.16:06You want to be sure that you prefix that with ND_admit, so, but otherwise it's gonna be the same password here.16:27So we'll go ahead and get logged in and you don't need to do any additional configuration to start consuming data from your Nexus dashboard instance. Today it is just data from your Nexus dashboard instance, just sort of keep that in mind. But you're gonna find this Nexus dashboard app pre installed on this installation, so.16:36Let's go ahead and browse to that.16:56And you're gonna get a couple of dashboards at your disposal right off the bat. So if we go to dashboards here, well, there's a couple of them that are built in, but you can build your own. That's like one of the main points of splunk, right? So here we can see anomalies, for instance.16:58And I mean yes, we can get anonymous within Nexus dashboard, but the key difference here is we get very deep, we can get very detailed in our searches, we can get very detailed in our custom dashboards. We can really make a view that's all our own here. So, I mean if you've worked with Nexus dashboard before, then, you know, the.17:16Values that you're seeing here are gonna look very familiar to you, but you get that level of data control. More on like the custom dashboards in a bit. I'm just kind of showing what native Splunk looks like currently. Now, going back to our dashboards, we can also see audits. So like what are folks doing on the Nexus dashboard cluster? So not a lot of activity here.17:36You do get some filters up here in the top left. So this is just for the course of the past 24 h. I can change this say to like week to date, for instance, just sort of see what folks are doing. It mostly just logins and.17:56Failure login there even as well. So this is just a quick preview of what native Splunk looks like. More to come on this in future Nexus dashboard releases, but if you're interested in this reach out to your account team and they can get you started.18:11Alright, Andrew I'm gonna go ahead and pass the ball back over to you. Great. Thank you JT for that demo. Okay, then.18:27That brings us back, going over to the flow of the session once again.18:46We did a little bit of an introduction into Splunk, now we'll be getting more focused into the demos and going through the different apps of the Cisco DC networking app and the Intersight app along with some MCP information.18:50So that brings us to our next Slido question back. Are you currently using Splunk to monitor your network and data center infrastructure?19:09So I see a lot of answers coming in with most of them being yes, we're using across the entire environment.19:39And some people are still evaluating it or getting started. Good to see. So we have one app for Cisco data center networking, which is gonna be comprised of the Nexus dashboard, ACI, and the standalone Nexus 9000.19:45And you can see for these specific areas, what information they're bringing in Nexus dashboard bringing in anomalies, advisories flows, ACI bring in fabrics, tenant tenants VM manager, and the standalone nexus, bringing in inventory power, performance switching etc.20:04Right, so this brings us to the question you maybe asking, do I use Splunk? What's the value when we already have Nexus dashboard? And if you're not familiar with Nexus dashboard, Nexus Dashboard is our visibility and observability analytics tool specifically for data center.20:25So and you will see similar data within Splunk and Nexus dashboard, but the value you really get is through the cross domain visibility, right? So the ability to correlate and adjust data from all the different layers at the same time. So let's say there's an issue with your application.20:41Experiencing delay. You're not gonna know where that issue is. It could be the application layer, compute side, could be on the networking, each team is gonna start looking at their separate systems. But the good thing about Splunk is you can collect and correlate all this different data at all the different layers.20:59Right, and it filter the info to help you find a solution. You can also correlate and bring in multiple information for alerts, and if for the alerts you the ability of course to send an email, set a script.21:18Log it within Spark itself. So you're able to trigger the alerts based on that cross domain automation. Now just a quick highlight level overview, but JT will get more to it in the demo.21:36Is when you 1st download the DC networking app, you're gonna want to go over to the configuration tab where you have your accounts and you'll go ahead and add your account for your data center networking profile between your nexus dashboard, ACI and Nexus nine case, which is gonna be a host name and your.21:50Authentication credentials, then you have access to a whole bunch of different inputs that are available, right? And these inputs are specific to ACI, Nexus and ND, but I won't go too much into this as we'll go over it more in the demo.22:09Then as well you have the Splunk Intersight add on, which you'll be pulling device inventory, events, alarms advisories, server profiles, CPU metrics, et cetera. Then adding an inter site account is a little bit different than adding a Nexus dashboard.22:29Account. Intersight is gonna be requiring OAuth token for your account. Then to navigate to get that OAuth token in intersight, you'll go over to settings and OAuth tokens, right? Then you'll go ahead and create that specific OAuth token.22:46In which you'll get a client ID and client secret, and in the configuration tab of the inter site app is where you'll go ahead and configure that to start getting inputs and gathering data. So let me go ahead and pass it back over to JT so we can go ahead and take a look at this too.23:02The different Splunk apps.23:21Alright, thanks Andrew. So let me go ahead and share my screen. Alright, welcome back everyone. So, a little bit of change pace here, so now I'm logged into a different Splunk instance, so previously we saw the native Splunk instance. However, this is just an install I have.23:23Have a running on an Ubuntu host in one of our labs. So, really nothing fancy here. However, what we do have are those two apps that Andrew was talking about. So we have our inter site add on, and we're 1st gonna take a look at our DC networking app in just a moment. Now, if you're new to Splunk, it's pretty easy to get started.23:43So Andrew talked about how you can get started with that and like the, you know, pre evaluation that you can do with it.24:02Once you get it installed to install these apps, you have, I'll show you two options basically. So you can go to simply find more apps and this is basically gonna open up like the Splunk store, like the Splunk base, information here natively within Splunk, so we can go ahead and.24:09And search for, I'll just go ahead and search for Cisco, but you'll see there's many other apps that are available too.24:28So we can get an idea of like what else is available from Cisco for different portfolios, so security cloud, enterprise networking, but here I already have my Cisco DC networking app. So this is the one that we're gonna be taking a look at today, already installed, so I just get an option to open the app. If you're installing it through the.24:35This way, you go ahead and click on the green install button, and then you'll just need to provide your Splunk.com credentials and then it'll automatically basically next next next sort of install. So that is one way. Now, if you're in a air gapped environment, let's say.24:54You can go to Splunk.com, you go to Splunk base, and then you can find the add on that you're looking for. You can just search up here. So here's the inter site add on. You can go ahead and download. You will need to be logged in, so you do need to have an account here and then that's gonna pull down a single file that you're then going.25:12You take back to your Splunk instance, and then you'll go to manage.25:31And then in the top right, you get an option to install app from file. So you just simply come in here, drag and drop or browse your file system, go ahead and upload the app and then just kind of follow the directions and then you'll get the app installed.25:37So once you're at that point, again, you should see something like I have here where I have those apps installed. 1st, we're gonna take a look at the DC networking app. Alright, now once you have this installed, if you're just getting started with it, there's two things that you need to consider. So you need to go to this configuration tab 1st.25:52And this is where you're going to provision your accounts. So we need to provide some level of authentication to access in this case the app has support for Nexus dashboard, ACI, and standalone Nexus 9000 here. And I do have all those in my particular environment. Maybe you only have one. Maybe you.26:12Have ACI nexus dashboard, for instance, but you prefer to pull everything through Nexus dashboard. That's totally valid, but you have the option of interacting with each of these platforms. So in my case, because I am interacting with each of these three platforms, I need to have a credential for each. So from my nexus dashboard accounts here, I just have a, you know, you plug in your IP address or host name.26:32Give it a credential to log in with. Maybe you create a separate like Splunk account, for instance, for our production environment. If I go to edit here, you can get an idea of the just the settings we need pretty straightforward, right? So you can specify whether this is for local user authentication, like locally provisioned users on Nexus dashboard or if you're doing some level of like remote.26:53No AAA authentication here, so like we use LDAP in our lab here, so you can define the particular login domain for that.27:12Pretty straightforward. For ACI, likewise, this is just logging into some of the apex in our lab as well as four and 9KI just have a single switch here today just for demo purposes, but this is one of my 90 3180 ycfxs in the environment. So we go ahead and add it.27:21Management IP address as well as the credential. So pretty straightforward, right? So that's the 1st step. Now, 2nd step is now we need to configure our data inputs. So this is in my opinion, this is one of the two big things that you get out of this app, out of this app. The 1st one being.27:41Here is we get a really nice framework to bring this data into Splunk. So of course you can set up like, you can consume syslog and you can do API interaction, of course, you can do that all manually. However, the apps provide a very convenient way with a lot of the inputs already defined for you to get the data that you want.28:00Into Splunk and you can get a fairly decent level of control just out of the box with it. So when you go to inputs, I'll admit 1st time I came in here, I saw a lot of inputs and you know couldn't make heads or tails of it. Little trick here, you can define a filter for one of those three platforms and that just sort.28:20That helps you get an idea of what you can collect out of the box. So for Nexus dashboard, advisories, anomalies, congestion endpoints etc here. When you 1st set this or when you 1st come in here, you're gonna see that all of these are going to be inactive, meaning Splunk is not collecting this information from Nexus dashboard.28:40So what I've done is I've selected what I want. You don't have to collect everything. You can collect everything if you want, but I've been a little bit choosy here, so I'm collecting information on advisories, anomalies, endpoints, and protocols, but it's really up to you. You have that level of control.29:00And then here under actions for each of these input types, you can further specify things here. So you can set an interval, you can set which index you want to pull this from. I'm just in a lab environment, so I'm just using the default one. In production you probably would have your own index there. Input type here for advisories, that's pretty populated, but category severity.29:17In time range. So these are given to you out of the box. Now, one of the other cool things we can do with this is define a new input. So if there's something you know you want to pull from the API, for instance, you can come in and define your own, say, custom.29:37Well custom API endpoint here, and you can get that data into Splunk. Now, next I'm going to show you some of the dashboards that you get out of the box with this app. However, of course, if you're defining a custom data type, don't expect any of those default dashboards are going to immediately present that data to you. You will be able to use the search function within Splunk.29:57Of course, but just sort of keep that in mind if you, you know, get custom data in, of course, you want to make a dashboard you gotta do that, kind of custom. But don't worry, I'll show you how you can get started with that.30:16All right. So these are our Nexus dashboard inputs just to show here, we also have ACI. So, this is more class based, you know, if you're familiar with the ACI API and the whole management information tree, so AAA model LR for audits, fault instance is that's pretty self explanatory.30:28Same idea, right? We can activate what we want and we can leave inactive anything that we don't want. And then for Nexus 9KI pretty much have everything enabled here, but, you know, version information, transceiver information, interfaces, host name, et cetera.30:46Alright, so that is the one, that's the 1st big thing that you get with this app is a really nice framework to get data into Splunk. Now the other big thing you get with that, with this is dashboards that are prepackaged for you that you can immediately start using with your teams for your own purposes up to you, right?31:03So we have the three platforms here, we have Nexus dashboard, we have ACI, we have Nexus nine K let's take a look at Nexus dashboard 1st. So I can hop in and go to say my anomalies.31:23And I mean, we saw this on the native Splunk instance, and of course you can get this within Nexus dashboard itself. But the difference is, of course, we can customize this to our needs. I'll show you that in a moment. And we can also share this with other teams. So we don't necessarily need to open up Nexus dashboard itself to.31:35All the other teams in our environment, we can be select, we can be a little bit more selective of that and do that through Splunk.31:55We sort of choose. Alright, now all these dashboards, I shouldn't say all, but most of them are going to have filters up here. So we can filter on the last 4 h, for instance. I just have the single Nexus dashboard cluster, so I, that's the only thing I have here, but you can select that if you so choose, as well as your sites, so I have these two.32:02Sites with my switches as well. As far as these individual tiles go, we can, in a lot of cases, we can click on things to help us further filter down into what we're curious about. So for instance, like.32:23I want to take a look at anomalies by, by category. I can click on connectivity and I can see all those connectivity anomalies down here towards the bottom. So looks like in this demo environment we leave a lot of vpcs partially configured, so I'm not too concerned about this to be honest with you. But you know depending on your.32:37In your case, you can use these filters to find what you're looking for, and you can specify like down to the individual node level as well.32:57All right. So just like a pretty, you know, pretty normal example of what a dashboard might look like. Now, getting into a little bit more detail, let's say we want to take a look at endpoints. So, you know, this might be a particularly handy dashboard to expose to say like the application team or the server team perhaps.33:05Up to you on that, but this can give users the ability to search for particular IP addresses or if they have like a certain tenant they want to take a look at or keep an eye on for their endpoints, you can filter on that up here. So e.g., like I have a couple of IPs in my environment, I have this 192.33:2581670.ten.33:45Endpoint and maybe I just want to make sure, hey, is this online in the network? So yes it is. I can, you know, it's just showing me one endpoint because that's what I filtered for, of course, but then down here in the endpoint table I can see which tenant is that a part of, what is its MAC address, the switch it's connected to, the interface bridge domain EPG.33:47Maybe I'm doing like a point A to point B sort of troubleshooting. I'm trying to see, hey, are both sides of a connection online in the environment? So I know that this guy should be talking to, 172.sixteen.0.ten.34:07So I just have a simple web app running between those two guys. And I can basically keep adding on to that and I can see that table down here. So yes, I am learning that other endpoints, so if I was troubleshooting connectivity between these two, at least I know where they are in the network, and that's you know I've really narrowed down where.34:21This traffic should be flowing, so between these two points, between these two leaf switches.34:41Now, that's one way you can I can go about this. I'm gonna remove those filters and maybe I want to take a look at one of my tenants. I'm gonna keep an eye on my, my own personal endpoint inventory. So you'll see here we have a lot of tenants in this environment. We use this environment pretty heavily for our demo purposes, but.34:47Maybe you have certain business units or certain organizations within your environment that you want to keep an eye on for you know their endpoints. So in my case, I have this local tenant.35:06And I have a couple of endpoints here, so I have about nine endpoints and I have my table down here. Maybe I wanna keep a persistent look on this. Maybe I want to create a my own custom dashboard for instance. Now one of the cool things we can do with this is hover our cursor over one of these tiles, and then you'll notice you get a couple of options here.35:18One of them is open in search. So when I click open in search, I can see what is the SPL or Splunk processing language query that is used to generate that tile. So if you've used Splunk before, this view probably looks very familiar to you. So this is essentially the source code behind.35:38Find that tile. So, if I'm just coming in here as a one off, I can edit this for my needs.35:58I'm pretty happy with this though. I just get a simple count here, so nothing flashy down here, but that could be useful information if I see a significant drop in this, then, you know, maybe there's a problem going on in the environment, just e.g.. Now let's say I want to save this, like maybe I want to keep an eye on this persistently. So up here in the top right.36:06You get a save as.36:26And I'm gonna create a new dashboard here, but you can also create alerts if you like, for instance, so I can trigger a threshold, like let's say a lot of these endpoints drop off of the environment. Well, maybe I want to get an email about that. That's maybe a pretty specific example, but you could do this for, you know, pretty much any query that you can come up with. So.36:28Some serious flexibility there. Now, I'm gonna create a new dashboard here and I'm gonna make this my home dashboard here in a in a moment here, but 1st we'll create that dashboard. I'll throw my username in there. I'll say this is just a demo dashboard. I can select my permissions so.36:48Maybe I want to keep this to myself, maybe I want to share this with other users.37:08I can select my dashboard type. So if I've worked with Splunk for a while, maybe I prefer the classic dashboard approach or I kind of like the dashboard studio and I can select my layout here. I kind of like the absolutes. I feel like I get a little bit more fine grained control with that. So I like that. And then panel title, someone that shows up on the dashboard, one.37:13Is that going to be presented as a little, you know, title text? This is just gonna be endpoint counts, pretty simple. So I'm gonna save that to a dashboard. I get the option to view this da, this dashboard, but I'm gonna do that in a moment. There's some more information I want to add to this dashboard. So we'll come back to that.37:32Alright, so I'm gonna go back to my endpoints view and I want to get that table actually. So let me go ahead and reinstate my filter for my tenant, just so the data is consistent, right? So I'll go down to this endpoint table.37:51I'll go back to.38:11Open in search, and then same process as before. I can edit this query if I so choose, like for instance, down here at the bottom I can modify, see, with that type table I can modify these table columns if I so choose, but I'm pretty happy with that, so I'm gonna go ahead and.38:12Click save as, and this time I'm going to add it to an existing dashboard. So I can go ahead and search for that and don't need to type it all in. It's just a search function there. So looks good and then this panel title will be endpoint table.38:31Save that. All right, now let's take a look at that dashboard. All right. So this is where you can do a little bit of like final like cosmetic customization here. So this is kind of taking up a lot of space here. Like this is just a simple count of nine endpoints. So.38:50Now what I can do is, go to edit and then maybe clean this up a bit. So I have two tiles here. This endpoint count, it doesn't need to be that large. So I'm gonna go ahead and condense that down and that leaves a lot of space for me to take my endpoint table, get it nice and cozied up here.39:09And then just sort of clean that edge up. So maybe I'm happy with that. I can go ahead and click save. I can go back to my view. And then I have my own custom dashboard that I built with tiles that I took out of this app.39:25Now, even better is maybe I want to keep an eye on this throughout the day or like when I 1st come in, I can go to actions, and then I can oop, I should be able to get an option to.39:42Add it to the dashboard, maybe I missed something there. Interesting. Should get an option to make this your own custom dashboard. I'll have to figure that out. Okay, apologies for that. So.39:57Anyway, let's move on to the next step. So that was Nexus dashboard, and an example of how you can create your own custom dashboards. Now, what else do we get? So let's take a look at, say, ACI. So ACI, you can get a lot of similar information, right? That you can get out of Nexus dashboard. Nexus Dashboard kind of adds additional.40:13My analysis layer on top of it. But maybe you don't have that Nexus dashboard. If you don't I seriously recommend you look into it. But if you don't, then, you know, you can absolutely use this app to get information out of your ACI environments or maybe you prefer just extracting data through the MIT or like through the API if you so choose, but.40:33For ACI, I can go to say like help desk. So we have this whole context in here for like an operation center or a help desk for some common things that they might be looking for. So e.g., system faults. So analogous to anomalies in Nexus dashboard.40:52You know, pretty similar views to what we got before. I can click on things to filter down if I so choose. You kind of get the point and of course I can create custom dashboards with this. Additionally out of ACI, if I go back to help desk, did you know that when you are working with say contracts and.41:10Policy we actually log whenever there is a policy drop by default actually. So that's very helpful when we're troubleshooting.41:30There we go. Now by default, if, as you know in ACI, if we don't have a contract or we don't have some policy to permit, we're going to drop that traffic in most cases or if our VRF is our vrfs are enforced by default, so that's the default behavior.41:41So very helpful for troubleshooting if we suspect that traffic is not getting through the network. Now you can also log for permits, so in ACI in your contracts, you can enable permit logging. Generally speaking, we don't really need to do that right? You have that capability, so that's why we do see some here. We do have some permit logging going on in the environment.42:00But we do have a history of L three drops, so I can come into this dashboard and then I can find a tile here for L three drop packets by node.42:20So that might be a little bit interesting to me. And I also get L three drop packets by tenant. So if I'm looking at a particular tenant and I'm troubleshooting something, I can come in here and see if there's any policy drops. That's what we would call these. And they're logged on a per node basis, which is why I can see for node two oh two, we do have a history of drops.42:30Here. And these were last logged back all the way back in April actually so it gives you an idea of like how far back we can check this sort of stuff. But I can click on this very fine grained bar graph there.42:51And I can get a list of logged packet drops. So, very helpful when we're troubleshooting. I might also recommend using the Elam assistant, but that's a topic for another day. However, finding these sorts of logs can really help narrow down where drops are occurring in the environment. And if you see something like this and if it's a drop here, then.43:02And that is a very good indication that we need to modify our contract configuration or otherwise ensure our policy is permitting the traffic as we, as we sort of choose.43:22Okay, so just giving you an idea of what's possible with ACI and then for Nexus nine K, let's take a, take a look at that real quick. So I can get say like inventory information, I can get a physical inventory. Now I have just added the single N9K here today, but I could add multiple and I can get a nice.43:33List view of what's in my environment. I can also get say like resource visibility information. So in this case I can go to resource summary.43:53And this actually gives you some pretty deep stats in terms of like memory utilization, CPU utilization, graphed over time. This is just for the past 16 min, but I can crank this up if I so choose, and it can really help me identify if there's like some outliers in the environment. Maybe some devices are experiencing a period of high.44:03CPU utilization for instance.44:23Now, of course, we can also get, say, interface information. So if I want to see how interfaces are doing on this switch, I can pop into my all interfaces view. Again, I get my filters up here at the top, and let's say I wanna keep an eye on like my 40 gig up links, for instance, so I can filter.44:26On my device, I just have the single one here, so this YCFX for my operational state.44:45Maybe I'm only interested in the up, you know, the up up interfaces for operational state and admin state. So I'll go ahead and filter those out and then for Ethernet speed here, maybe I just want to keep an eye on my 40 gig interfaces. So I get a nice list here. So these are my uplinks on this guy.44:51All right. So, again, this is our DC networking app. You get two big things with it. You get a really nice framework for bringing data into your Splunk instance, and you get a bunch of dashboards that you can use right out of the box or you can customize them for your purposes. Next, let's take a look at.45:10It's inter site.45:30So, we'll shift back to our home view. Now, our inter site add on, I already have that installed. The idea here is gonna be pretty much the same with a minor twist. So 1st we're gonna start off with our configuration tab.45:31As Andrew had mentioned, we're not gonna be doing like username authentication here. Here we're gonna be doing Oauth two tokens. So you do need to go to your inter site instance and be ready to provide that here. So I've already done that. So we're using the SAS based instance of inter site, so this is just home to intersight.com.45:51Come, but if you were to add, say, you know, add a new one here, you can give a friendly name for this account. You can specify which inter site instance you want to get information from. Now.46:11I'm in the US, so we're just using the regular inter site.com but if you have like, you know, certain compliance requirements like maybe you're located in the EU, maybe you need to ensure that data locality is considered here. Maybe you use the instance in the EU or if you have on prem deployments, then you can select that as well. You can provide the.46:25FQDN here. Now when you create that OAuth two token, now if memory serves when you go into intersight, basically you can find your OAuth OAuth two tokens over on your far left and then scroll all the way down. Sorry, just the mental picture for you. You can find your OAuth two tokens, create one.46:46You'll get two key pieces of information. You'll get the client ID and the client secret. Now that client secret, you'll see pretty clearly that you'll get one chance to see what that client secret is. If you lose it, then you basically have to, well, you can disable that token and then you can generate a new one, basically provide that information.47:04Here, so you just paste in those two pieces of information and that's pretty much it.47:24So we're done with the configuration tab there. Now, just like the DCN app, 2nd part of this is I need to define my inputs. And this is gonna work pretty much the same. So, everything's going to be inactive by default. I have most things turned on here, but you can be selective about it if you say.47:28Don't choose.47:48So, most of these are active here, so we can get alarms, audit logs, device metrics. Again, you can define your own inputs if there's data here, you know, if these seven inputs aren't enough for you, then you can create your own inputs if you so choose, and then for each of these you can further customize them. So you can define the interval.47:50Which index they're using, et cetera. All right. So once we have configuration set up and once we have our input set up, then we can start taking a look at the custom dashboards that are available to us. So for my.48:09Let's say like I'll go to inter site Insights here and we already took a look at say anomalies in exist dashboard and faults in ACI, so the analog to that in inter site is going to be alarms. So I can see if there's any unexpected alarms happening in the environment. So I do see I do have.48:24Five active alarms, I can get an eye on that and take appropriate action.48:44So pretty simple view there again you get your filters. Additionally, oh, one example that I particularly like with the Splunk or with the inter site add on here is it has a really cool example of being able to check your inventory.48:49There we go. So I get a breakdown of like four blades and eight rack mount servers, but I feel like this kind of gives you a really cool example of what SPL is capable of and what these dashboards are capable of. So I can get was essentially essentially a directory type view into my environment and.49:08And again, this is a case where you can see this, the SPL query that is responsible for this.49:27And you can use this for your needs. So I'll, I'll pick on this server here. So we have aC2 20. When I select that, then it filters out details for that node. I can check its coverage. This is just in the lab there. I can check the hardware compatibility list. So everything's validated, that looks good. And additionally.49:33I can get a nice little inventory of this node. So I get a list view of physical disks, ports, CPUs, as well as what dims are installed. So one more trick that we can do here is I showed you custom dashboards, but here we can also.49:53Download this information. So if there's something I like here.50:13I can hover my cursor over the tile as I did before, but now I can export, and in this case I can export as CSV, XML JSON. Maybe I'm happy with CSV. So I can give this a file name. It's called like mem inventory.50:16And I can go ahead and export. So just as simple as that. So I can export that table. Now I'm just on a Mac here, so this is just using the built in numbers app, but you can open this up in Excel or a text editor, your preferred tool, but really effective way of exporting this data.50:35All right, so that's gonna take us to the end of the main demo here. Remember, it's very easy to get started with these. You can find these on Splunk based.com. You can find them through the manage manage view in Splunk and install them natively within your Splunk instance. But anyway, we'll go ahead and end it there.50:53Andrew, let me pass the ball back over to you. Great, thank you so much JT for that fantastic demo. Then in the essence of time, I'm gonna pass over the Slido question for now, and if we have time we'll go and get back to it.51:11But what I wanna go over is Splunk MCP and just MCP servers in general. So MCP is gonna stand for model context protocol. It's gonna be an open standard framework, and what this framework does is it allows seamless and secure two way communication.51:31Between an AI application such as a large language model LLM, and your external data sources and tools. So you can kind of think of it as a universal adapter, then without it, you would have to create a custom code integration for each of the tools that you're using.51:50So, and if you're more of a visual learner, you can kind of hear, you can see here on the left without the MCP. If you have your LLM it can only talk to tool one, but if you haven't set up the integration between all the other different tools, it's not gonna be able to communicate with them. But with the MCP here on the right hand side.52:09You can see that all the different llmiss speak to the MCP, then talks to see what different tools are available and determine the best tool to utilize for your request. So just a high level overview on how Splunk MTP.52:27Server architecture works and the workflow is it's 1st gonna initialize the AI clients can reach out to the Splunk MCP server.52:47And the MCP server's gonna define a list of tools that's available such as Blunk run query that has other tools for get info, getting indexes, getting metadata, et cetera. Then the user's gonna type in request. In this example we have shown me the top five users by error account.52:57The IAI is gonna construct an SPL query to send over to the, to the MCP server, then the MCP server is gonna authenticate it and create that REST API request. In return.53:17We're gonna get that JSON result back to the MCP server, in which it's gonna get sent back to the AI. Once the AI is receives it, it will create a dashboard, visualization based on that. And this is the Splunk MCP server app you can.53:37Look it up in Splunk base as well. And I want to oh sorry before we get into a little bit of a demo, this is just a high level overview of everything and how it kind of fits together. You can see here on the left hand side, we have all our devices which connect to a Splunk folder.53:54This Splunk folder indexes the data and stores it. Then you have your large language model here on the right hand side where users can say, hey, give me the device, where can I find the drops? Right, the LLM is gonna talk with the Splunk MCP server.54:12Which will work with the search head to find where this data is living in your indexers. Once it finds that data, it's gonna return it back in some sort of a visual event such as the dashboard. Then I will pass it over to JT so we can explain a little bit.54:30And how the MCP server works as the demo. All right, thanks Andrea. I'll go ahead and share my screen here. Alright, so yeah, just to record demo here, I'm showing the MCP server. So we have four examples for you.54:48Our 1st one is just asking a very simple question to the LLM, so this is using plot, integrated with the MCP server. So we asked.55:08Any devices with high CPU. So this is where our LLM is going to reach out to the MCP server and select right tools to gather that information and generate that SPL query. Now, of course, one of the cool things that can happen with these llms, whether we specify specifically or maybe.55:18It's done through a system prompt is, get a very detailed view but consumable view with visualizations on well for the answer to this query. So right now Claude is representing the answers to that query in the form of a bar graph here. I'm actually gonna speak.55:38There we go. Yeah and so in this case, the leaf switches, they're all within acceptable boundaries for CPU, However, it does note that leaf one oh three did have the highest peak CPU and maybe we wanna take a closer look at that. So just an example of a pretty simple question.55:58But we'll go ahead and get into our next example here. So I showed you an example earlier on how you can get endpoint information out of Splunk or of course you can get endpoint information directly within Nexus dashboard or ACI. But in this case, we could just simply ask through natural language, hey, where's this IP?56:18Address connected to. So again, we're gonna reach out, well, the LM is going to reach out to the MCP server, gather that information, and then present that to us here. Now, interestingly.56:38It does find information, finds a record for that IP address and then presents it as like which switches it connected to, which interface was it connected to, but it makes the note that there actually hasn't been activity for this endpoint within the past 24 h, and that can be pretty valuable in a troubleshooting situation, because maybe something.56:49Has happened within the past 24 h for this endpoint, but we know where it was. So it really helps us narrow down if we're like say doing a root cause analysis.57:09Alright, next up a little bit more of a complex query here, very simple question, but let me know where traffic is spiking in the environment. So that could potentially have to cross reference many different pieces of information in Nexus dashboard. So in this case, the LLM is working through the problem.57:19It's generating multiple queries, so it's going the tool server multiple times, but ultimately it does identify several interfaces that have had you know relatively high traffic utilization. And in particular, it identifies that one interface ETH one nine on leaf one oh one did have a relatively signifi.57:38Microburst of traffic, so.58:00Yeah, pretty simple query, but maybe not the easiest to answer. All right. And then just for our final demo here, another use case for this is you prepare the daily operations report. So, think of this as like a daily health check that could be done. And again, you can like structure this through a like so.58:03System prompt or, you know, prompt the LLM prior to this. But in this case, it's gathering different pieces of information to build this report. So, I might have to give this a moment here, but ultimately.58:23Cloud will generate a report for us to consume. I'll go ahead and speed this up a little bit. And it gets to a point of generating a HTML report. It also gives a little text summary there at the bottom. But wouldn't this be kind of a cool thing to have just for your daily reference?58:37So we can open up an HTML report, we could say throw this up on an internal website and get a nice view into our faults anomalies and points all automatically generated for us. All right, so.58:56Pretty cool stuff there. Hey, Andrew I'm gonna go ahead and pass the ball back over to you and I think we're getting wrapped up here. Great, thanks so much. Let me go past our demos that we did. That's gonna bring us to our session slide. So.59:14Today we went over a bit about a bit about Splunk kind of covering over an introduction and what's available in the Cisco DC apps that we have available. We took a look at a couple of demos, and now we're finally getting towards the end here. I believe we have one final Slido question or maybe one more after based on what you see.59:31Today, are you planning on installing any of the Splunk apps in your environment? Looks like we have some, we're interested in evaluate with the Splunk team, which is good to see. And of course, if you still need more information, like a one on one with us.59:51Feel free to schedule something after the session.01:00:11Then I will pass the ball back to Joe and he'll help us close this out for today. Excellent. Great. Thank you very much, Andrew. Wonderful presentation today.01:00:31And a very good demonstration JT. For our audience, I'm going to share out some content here and I'll put some links in the chat if everyone can see that.01:00:47I've actually got a couple of more I'm going to send out now, so if you will, take a look at those links that I put in there and.01:01:06If anyone wants more technical information, as they indicated in the Slido earlier, feel free to explore those links and look at that. One of those is the FAQ, actually around Cisco IQ. And that may help.01:01:23For our topic today, you do have time, you can do some Q and A, ask some questions either in Webex chat or Slido, which is preferable. I'll put a link there to our Cisco community so you can explore.01:01:46More about topics, and we've got a session survey that is going to be started here in just a second that will allow you to give us feedback.01:02:02And to go in and answer questions, give us a rating. Again, we've got Slido.com there. If you're able to go to that on a web browser, you've got our join code and passcode. Otherwise that QR code there is available for you.01:02:17Now to talk to you about Cisco IQ, we know that Cisco has produced Cisco IQ for the landscape that's evolving around the methose era, a time of unprecedented complexity and scale. Cisco IQ is ready to meet this challenge.01:02:35Moving beyond traditional support to provide absolute clarity and intelligence. The Cisco IQ is a digital intelligence platform that leverages anonymized global data to provide you with personalized insights and benchmarks that are tailored to your specific IT environment.01:02:54It allows you to measure your current technology maturity against the industry standards, and it's gonna help you to identify exactly where to focus your optimization efforts in your environment. By transforming complex data into clear actionable recommendations, it empowers your team to make data driven decisions that accelerate your digital.01:03:11Transformation goals. Now the four points that we've got listed here speak to your ability to manage modern infrastructure, in way of it being like solving a 10000 piece puzzle without the box cover, right? So hard to do.01:03:31Now you're often flying blind until a failure occurs, but with Cisco IQ, that's gonna eliminate your blind spots that you have and provide a single absolute view of your entire Cisco IT landscape. Secondly, every outage leaves a digital trail. Most disruptions are preventable, stemming from unpatched vulnerability.01:03:48These or misconfigured items that go unnoticed. Cisco IQ identifies these risks before they ever impact your business shifting from reactive firefighting to proactive resilience.01:04:08Now when the service incident occurs, any minute of downtime erodes trust and revenue, we are collapsing the traditional resolution timeline from hours down to minutes by eliminating the manual documentation hunt and the complex escalation chains.01:04:23And lastly, complex deployments, of course often stall not due to lack of talent, but because critical knowledge is siloed. Cisco IQ brings over 40 years of institutional experience to your projects, ensuring we build.01:04:42Right the 1st time. So we want to know if you're interested. I put some links in the chat there. As you can see here, iQ.cisco.com. Now if you're an existing CX cloud customer, you'll be able to migrate easily. You'll simply want to follow the few steps here, navigate to Cisco IQ and.01:04:59For your cco ID.01:05:19Secondly name your Cisco IQ account, IQ account and select the CX cloud account where you want to migrate data from. Next you want to click the account and migrate data. The CX cloud data will migrate automatically the Cisco IQ, and lastly, once the migration is complete, you'll be in redirected over to the Cisco IQ launch.01:05:21Bad and welcome page.01:05:41Now if you're a brand new customer for Cisco IQ, then you'll want to follow just three simple steps. They're gonna be navigate to Cisco IQ and into your CCO account. You'll create a Cisco IQ account name, and you'll go through a self guided account configuration.01:05:44Where you'll add service contracts, you'll add your Cisco IQ link, you manage your users, and explore Cisco IQ.01:06:04Now in the links I put in chat earlier, includes this FAQ link frequently asked questions, so I invite you to explore that and give us feedback on the survey that has been opened up to you. We do have a few minutes left. If you've got questions, please ask those in chat.01:06:15Chat or Slido if you can.01:06:35Let me check for anything that's come up. No questions there. I don't see any questions in chat. We'll take a look. As Andrew said reach out to your Cisco account team.01:06:38If you would like, you know, more information than these links provide possible contacts, give everyone a few more minutes if they want to go in and vote on that survey and feedback.01:06:56Post any questions in chat looks like we had someone post a question around ACI Nexus dashboards.01:07:13And we had a couple of our panelists there including JT respond, yeah, please feel free to to post your contact information there and we could perhaps.01:07:31Schedule a follow up session with you. I don't see any more questions coming in, so to respect everybody's time, we will move on. Thank you for joining us today.01:07:48Again, this has been the expert insight series for Splunk monitoring and best practices, and integration with Intersight.01:08:05Thank you and have a great day.01:08:23