Unified vision: Introduction to Splunk Observability Cloud

Oh, we have a pretty good number of people joining today's session, which is good. In fact, guys, this is the 1st series of Splunk observability cloud.00:00You have.00:21Upcoming series on specific components of Splunk observability cloud, stay tuned and then attend those sessions. Today's agenda is to give an overview of what Splunk observability cloud is all about, the basic features, so you are familiar with those features.00:22We are not touching up on any of the configuration part today. So as a detailed drill down, those are all follow up sessions for for for your information next month starting we have a session on.00:43Kubernetes monitoring followed by infrastructure monitoring and then real user monitoring using Splunk observability cloud. So the coming months, the 1st week of the coming months is, important in terms of understanding more about Splunk observability cloud. All right?00:59So today's agenda, I will just mix this up in between because to make it interesting and exciting as we don't drill down deeper, I may have to switch to demos in between, and then talk about some features up front rather than keeping.01:16It's step step by step, right? So we will just talk about the overview of Splunk observability cloud. What exactly is that?01:36And then a glimpse of open telemetry, then the data ingestion models in Splunk observability cloud, the collector, and then interesting demo on the AI assistant and the automated root cause analysis using Splunk followed by the.01:45The overview of how you can access various tools and features in Splunk observability cloud. This is how the agenda is structured. Again, the expected outcome from this session to be clear is you are familiar with Splunk observability cloud cloud features, where to access what, and then at least.02:05To play around with the product. That's the agenda.02:25Let's get started. What exactly is Splunk observability cloud? In the initial, the opening remarks Kevin mentioned about product offering as well as overlaps etc. I'm just sorry for these faded or maybe the the unclear probably the screenshot, but I'll just.02:31To explain to you what exactly that real time. So Splunk observability cloud is Cisco's offering.02:49For cloud native and Kubernetes native applications, the observability solution for cloud native and Kubernetes native applications, it is the de facto observability solution from Cisco for modernized microservices as well. So to look at your tech stack, you have cloud native.02:58If you have Kubernetes workloads, you have microservices, then Splunk observability cloud is the solution for you for end to end monitoring. All right? This is a SaaS based offering.03:18The bottom line is Splunk observability cloud can be used to monitor metrics logs, traces, and events. All right? That being said, the observability cloud is entirely open telemetry native.03:33You, if your organization is all about open telemetry, your roadmap lies based on the open telemetry, then this is the solution for you with respect to observability. Or on a side note, if you have subscriptions to.03:51Cisco secure app for security vulnerability and attack as well as threat detection runtime in the runtime. Then Cisco Secure app can be integrated to Splunk observability cloud. Why I'm saying this.04:09As per cisco's, take, the observability going forward is incomplete without visibility into security. So going forward, security insights will be more and more embedded into observability solution. So as the 1st step, you have the cisco secure.04:26Have Integration available runtime, you can detect security vulnerabilities and threats, very important, right? If you have subscription, then you can integrate. So as the network visibility, if you have subscriptions to subscription to thousand eyes, which is the.04:46Synthetic network monitoring solution from Cisco. Of course you can integrate it with Splunk observability cloud. So you have end to end network visibility as well as with application performance visibility, right? I believe that's.05:02Just a context setting. Now, as I said, Splunk observability cloud is entirely open telemetry native. That means you don't have a vendor lock in here. You can use the open telemetry agents as it is the open telemetry collector.05:18As it is or we have we recommend or we recommend to you, you to use the Splunk distribution of these open telemetry collector or open telemetry agents. Why? Because.05:38As you are aware, you have the to to to in the open telemetry native open source project, there are two distribution. One is the open telemetry core distribution, which is the, the, the core SDKs and APIs for end to end observability, and then there is a contribution where.05:51Or the contribute version where different vendors customize or build upstream with, the endpoints and then configurations according to the different vendor products. So Splunk contributes to open telemetry and you can see in, in the Splunk GitHub, the.06:11Open telemetry distribution for Splunk collector as well as the various agents. So we always recommend you to use those because there is support from Splunk upstream on the upstream part of those, the distribution, right? And then enhancements, feature rollouts all happens to Splunk distributions rather than the.06:30Open telemetry core distribution. So, now the hot topic here, all of these insights what observability cloud is providing is equipped with the power of AI. You can do.06:51You have a, you have an AI assistant with which you can interact and then understand what is happening within your landscape. Similarly, you can just make use of the AI troubleshooting agents to understand, to, to pinpoint the root cause and generate an AI summary.07:09Alright? So that's from the open telemetry perspective. Now, how are we going to ingest data into Splunk observability cloud? Just bear with me for a moment on this one or two sliders and I'm just entirely going into the demo environment post this but just setting the context here.07:26I just mentioned in the beginning that.07:471 s. I just mentioned at the beginning that observability cloud is for for cloud native as well as the Kubernetes workloads, right? Now, if you use, you rely on AWS GCP or Azure Cloud.07:50Then don't worry about monitoring the SaaS and services provided by those vendors. We have Splunk observability cloud provides cloud connectors using which you can monitor your Azure services, your AWS services, GCP services.08:08So it's just a cloud connection, and, basically for Azure, it's a pull model where the metrics will be pulled from the respective monitoring APIs, say the Azure monitor APIs, and for AWS it is from the cloud watch API. And for AWS, there is even a push model where the.08:28Message stream can be used to to push metrics back into AWS observability cloud. So that is cloud connectors, so you are.08:48Yeah, you you are going to monitor your cloud resources end to end using observability cloud. Now the next component is the observability open telemetry collector, which is the heart of the deployment from the your premises or the customer's perspective. Open telemetry.08:58Entry collector.09:18Collects the the data, the metrics logs and traces and forwards it to Splunk observability cloud. I will just come back to that in a while, but then there are open telemetry agents. E.g., you have your java, microservices.net node.js Python based services, then open telemetry provides.09:20Agents and we have a Splunk distributed agents, which is upstream, which does the instrumentation for those services and then collector collects the details and send back to observability cloud. I'll make it that make that clear in a, in a while. The next mode of ingestion is via REST APIs. Come there is an.09:40Environment where you can't install your agent, install an agent or probably a serverless, e.g., then you can rely on Splunk APIs and those APIs can be used to send data directly to.10:00Observability cloud, right? So that is with respect to the data, ingestion. And here is a workflow or data flow with respect to Kubernetes data collection, right? So this is the open telemetry.10:12Three collector as I said, the heart of the monitoring, so you will have to install Splunk open telemetry collector in your Kubernetes cluster. The collector has distinct pro services embedded like the receiver service, processor service, exporters and the extension. Anyway, you can just get the detailed documentation in the.10:32Open telemetry documentation and I believe most of you are already aware of it. So the collector is installed in the Kubernetes cluster and then this collector itself will collect metrics and logs directly from Kubernetes cu Kubernetes cluster, which means that you don't require any additional forwarders.10:52For agents to collect the Kubernetes native metrics, the board metrics, the workload related metrics, the node metrics, the Kubernetes events, the Kubernetes container workload logs, et cetera or the platform logs because this collector is provided with.11:11Receivers to collect each of these data types. So you just you just install a collector, you are immediately going to get the metrics logs from the cluster. The metrics are sent to Splunk observability cloud, which is the topic today.11:30While the log is sent to Splunk's cloud platform or enterprise platform. So you must understand that logs are not saved in the observability cloud backend. It is on the Splunk cloud or the Splunk enterprise platform, which means that you need a subscription for logs to make use of.11:49Log ingestion and then troubleshooting using the ingested logs. The metrics and traces only are getting saved in the observability cloud just for your understanding. Now comes the back end services or the microservices deployed into the cluster to instrument the services, to get the trace detail.12:08Is the.12:28Call flow, the waterfall, and then the JVM or the CLR nodej's runtime metrics, you'll have to instrument the services using open telemetry agents, say open telemetry java agent.net agent, node.js, agent, etc. And we provide Splunk distribution of that agents as well. We recommend.12:30You to use those. Those agents will instrument your services and then prices are generated.12:50And then the runtime native metrics are sent to collector, which is again sent back to open telemetry oh sorry observability cloud. I believe this is very clear from the Kubernetes standpoint. It's the collector which is the key component and for the backend services, instrumentation, you'll have to rely.12:56My own agent, a specific agent.13:16Okay, what about, what about a flow in a native host? If you have if your estate contains the virtual machines or physical servers, say Linux, AIX, windows, etc. All that you require is.13:19To to install the collector and the collector itself can capture the host metrics directly. So you have a Linux box, you just install the open telemetry collector, the metrics are collected by by the collector directly because it has the host receivers.13:39And then it is sent to Splunk observability cloud. If you have a Java service running on that particular host, that need to be instrumented to get the trace for which you will install the Java agent and the traces are received by the collector and sent to Splunk observability cloud. I I I I think this flow is clear to all.13:57If you have any queries with respect to this, you probably can put a message in the chat. We have a Q and A at the end or we will we will get back to you directly over an email if you are unable to answer those right now.14:19Okay, now I'll just quickly switch to Splunk observability cloud UI. So this is 1 s, ok. So this is one of the instances of Splunk observability cloud which I use. This is a demo instance in fact.14:34All these are dummy applications, so the data may not be as relevant as you think, but I will just use it for the demo's ak.14:53Alright, so the observability cloud is hosted mainly on AWS instances on different regions. E.g., US east, US West E EU central, EU West, Singapore, Japan, Sydney, this.15:01These are the regions where as of now the instances are available. We call it as real real realm, and then you you your instance corresponds to one of these hosted instances. By the way, it's not just AWS. Some of the instances are hosted on GCP as well. So you can say it's AWS plus GCP. All right?15:21So now some hygiene practices which we just built into this UI, there is a contact support directly over here. If you have an issue with the platform, you can end up opening a support ticket directly by clicking this link.15:40There is a portal, support portal hyperlink using which you can raise a support ticket, so this is what you will use. 2nd thing, let me just go to say say some overview pages randomly I'm just picking some, some page here. Now you can see here the.16:01The documentation links as well as short demos. Now, if you have, if you, if you want to understand more about the feature, then there is short demos available which are kind of 5 min or 2 min and then direct link to the documentation, just click on it, you land onto the documentation link, learn about it and come back and then play around.16:20Play around with that, ok? Again, just going back to the homepage, if you want to see more about the release notes, the product updates et cetera, all those are here. Click on the hyperlink, understand what is newly released, what bugs got fixed, what are the known bugs et cetera.16:41Right, and here is the release note for you. Now what are the components of Splunk observability cloud? I said it is end to end, now what does that mean? End to end meaning you can monitor your application performance, which is the APM, then underlying infrastructure, be it kubernetes.17:00Clusters be it your cloud assets or your virtual machines or or physical machines, you can monitor those the infrastructure monitoring. Then comes the digital experience monitoring where you have.17:20The option to monitor your mobile apps as well as the browser apps. See all of these are, these are lined up as upcoming sessions, where we will drill down to the details. All right? Now, I I said the logs are going to Splunk Cloud or the Splunk enterprise platform.17:34Form, but you have an option.17:54Of log connector here using which you can create a connector to your, to, to your ingested logs so that the logs can be visualized on this particular platform itself. E.g., if I just click on this log explorer, you can see that the the list of just forget about the.17:56It may take some time by the way, but anyway, so the logs are.18:16Already published over here. Now, the good thing is when you just troubleshoot based on your traces that if the traces are linked to the logs, then directly you can land onto the error logs or the the error statements. Okay, so that is the beauty of the log connect integration here, make use of it.18:20By bringing your respective logs to Splunk platform.18:41Okay, so this is log connector. I'm just not mentioning about the configuration as I said, but the configuration itself is simple. If you want to go through it, maybe you can just go for a trial and then explore it. So this is log Explorer. All right, so that is those are the main pillars, APM. In fact, these are called.18:45The distinct tools within Splunk plat Splunk observability cloud. This is the APM tool, this is the infrastructure monitoring tool digital experience monitoring tool, and this is a log connector. All right? Now comes the metrics. So now you I I just want to highlight the metrics over here because.19:05The oh let me just go to the metrics finder over here. Now, the good thing with respect to the observability cloud is that the along with the metrics, the metadata is captured. E.g., you have a Kubernetes container container workload.19:22Running in a cluster, now you want to monitor that container closely.19:42So what you can do is you can make use of the Kubernetes primit primitives or the or the other or the law, the labels which you employ. Those labels are available as metadata, meaning I want to know the container memory limit usage of.19:47This particular port in this particular name space so and so, right? Or or or any service description you provide or any label you provide or I want to monitor the CPU usage of a specific Linux host which is in the staging environment which where my app now.20:05No one is hosted that you can do that much flexibility is there. So maybe if I just put a search over limit, then you can see CPU limit here. Memory limit here, I just put memory limit over here.20:25So all the, the mem the metrics corresponding to those are all all all listed over here. So you can just make use of the available metrics and then plot plot it. Let me just go to the limit again and then plot that CPU limit for you.20:41Was that CPU limit e.g. and then if I just search, probably I may get a lot of details over here. So you can see I can just view this in a chart, so that opens a new window here, it it gives me the limit CPU utilization of.21:00Certain ports or all of the ports there. Now I can add a filter saying I just want this metric from specific namespace or specific service or specific port. Similarly, if it is a host metric, I want this, this metrics from my staging environment or my production environment, so all sort of.21:20Finer filtering is available using this metadata. That's the importance of this particular metric finder. Make use of it while troubleshooting, if you are creating charts comparisons, et cetera, incident analysis etcetera, make use of this metrics explorer, ok?21:40So that is in a nutshell about various components. Now I believe you now know that you can monitor your APM infrastructure, browser, app, mobile app. You can connect your logs.21:54And then you can play around with the metrics using Splunk observability cloud. Let me up the excitement here by bringing something new. We just told you that observability cloud is fully equipped with the AI. Now let us see the power of AI. I'm going to highlight this feature not just to.22:09Not just simply to highlight that we have AI capability but to exemplify the power of that AI feature. All right? What I'm going to do here, some more context be to to set, you look at these two components here, dashboards and alerts, that means.22:29You, you can build your own custom dashboards, you can create health rules. Health rules are called detectors in the observability cloud, and you are going to get a bunch of built in dashboards which will cater to almost all the use cases unless you.22:49Have a specific requirement, you need not to create a custom dashboard here. So let's check that once. If you just go to the dashboards part of it, right? And you can see a bunch of dashboard groups already available and.23:09I think I will have to select, which one I'm just checking something here. Kubernetes, e.g.. Yeah, if I just take the Kubernetes overview, then I am going to land onto the Kubernetes overview.23:27Dashboard.23:44Which gives me the cluster details, the node details etc. So these are all built in. The moment you install your collector and the moment the collector starts sending data into observability cloud, these dashboards will get auto populated, right? So it gives you the, the distinct nodes available in the cluster, the.23:44The ports available in the cluster, you can individually go into each of these ports and then analyze drill down to the port utilization, port metrics et cetera, right? So what I am saying is the dashboards are all available over here. You can readily start using it. So as the.24:04Alerts, there are some default alerts or the default detectors created, but you are free to create your own detectors.24:24Alright, so this is how dashboards and alerts work. Now, let me just go to I'm not going to explain you how alerts can be created as well as, going into the existing alerts because that's not.24:31That's beyond the scope of this session. What I'm going to do is I'm just going to click on active alerts. So anyway, from the last eight days, the active alerts are being highlighted over here. Now you can see, we have the AI root cause troubleshooting agent embedded in only cloud. So.24:49If you have enabled the feature for your detector, then you can get the root cause analysis done using the the feature. E.g., you can see set of alerts here for which the AI enhanced the icon is there, which you can just readily double click.25:08And that alert description is over here and then it is finding out the root cause. Now.25:28I must say this is of course AI generated, you have to involve human in the loop as a best practice, go through the root cause evaluate and then confirm it and then take the next step rather than blindly using it. But to based on my experience on.25:36Some of the detected root causes, this is giving you an inclusive and exhaustive root cause analysis because it is checking the entity dependencies. It is checked if it is a Kubernetes entity, it is checking the Kubernetes primitives and entities Kubernetes events, then it is checking the logs if.25:55There are, if the logs are in ingested, then log errors are checked for and then it gives you a comprehensive root cause analysis. All alright. So this analysis may take few minutes, say a minute or two.26:15But the same one to to save the time I have opened it on another tab, I believe, yeah, this this is the same alert for which the root cause analysis was done and it says the confidence is medium and I mean based on the available entities as well as the.26:29The evidences, sometimes it maybe statically high as well. So we can review the root cause. It says some host is down, some PG host is down for some reason. Now, I'm not bothered about the alert here but just want to go into this root cause analysis, so it says that some node in the Kubernetes.26:49Cluster is operationally degraded and it's it even went into the GPU operator upgrade module messages here and it went into the the logs and it find out, it found out that.27:08This is stuck because some GPU upgrade failed state is present in the logs, right? Now you can create a we AI generated action plan as well based on this, if interested, by the way, and it gives you a step by step.27:25Next steps in fact, this is not about the AI root cause analysis. I'll just copy this operator this node name for a, for a moment. You can see your chat board here, which is the AI assistant. By the way.27:40If you, if you are, if you have some query on the model being used, this root cause analysis is using a friendier model LLM, which is relying on Azure AI services, but oh, just a minute, I may have to plug in my charger. I'm sorry for that.28:00In the meantime, if anyone is interested in playing around, the product, the platform yet, you are not too sure. Splunk has recently released a 15 host cloud free.28:23Solution for you to play around. There's NO limitation on how long you want to use it. Ongoing access, you don't have any procurement required. Just use it as much as you can, as long as you're within that 15 host limit.28:39I will post a link on the chat shortly. Yeah, alright thank you Kevin. That's something which I forgot to mention here. Alright, so here this chat assistant, by the way, you know, here, not just the friendrier model, but this is a rag based, you can say this is a grounded.28:54Output based, it checks the, the knowledge bases. It uses the Splunk query language and it creates the search queries and then analyze the entities. It uses Cisco foundational models as well.29:16To get you the results, so you can just make use of this AI assistant to understand the issues on this particular, on, on this environment in fact. Suppose you get an alert, you just mentioned the alert ID over here and then and ask ask you to summarize. So it gives you the details over here, right?29:32I can just do this and anyway, it takes some time, but all that I'm going to show you is this. So you can see some of the links over my note, the OneNote here.29:52So there is a, this is well documented. If you want to more, know more about how the AI is working, what models are being used or go to this blog or, go to this blog or this.30:06Docs docs site link to get guidance on prompting, you have good examples of poor prompting as well as good prompting over here. Maybe I can just put this in the chat.30:22Okay so I just pasted it in the chat, go through that so we don't have time to explain all those things, right? So this is about the AI feature. I believe this is going to be quite useful in your environment. All right. It's not about that. So let's.30:41Just go ahead and then play around with that infrastructure overview. We have already data reporting for the.30:56The host environments over here. Now, if you have installed the collectors in the respective environments, then you will start seeing the data like this. Now, this is in fact an inventory what I would say. So you can see I'm monitoring 70 active hosts to Redis instances here, load balancers here, auto scaling groups are over here.31:05Then EBS, EC to SQS ECS, these are all AWS services, which means that you can monitor all of these cloud services, ok? Now the good thing is if I just click on this active host, e.g., it opens me the inventory of the host and I, if I am a Linux administrator, I just.31:25Want only to see my Linux in infrastructure, ok? How, how is this coming? These are all labels.31:45Which the collector sends to the only cloud. All right, so this is the, this is my Linux state and these are my various charts depicting the resource utilization. And now you can, you can build your own charts as well, right? So this is.31:52This is about it now, if I just want to monitor monitor my Kubernetes cluster, go to infrastructure, go to the Kubernetes overview, there you go, your Kubernetes workloads, which I already touched upon. You have two K ports running and if I want to monitor the ports, you are, you are here.32:08Okay, alright. Now I just want to highlight one more feature over here. Let me just go to the APM overview and that is the one of the strengths here. We just talked about metrics logs and traces, right? Now.32:25Let me show you how tracing works here. So go to APM tool, I'm at the overview and you can see the distinct services which all, which all are reporting here. Let me go to last.32:454 h and you can see the request errors throughput and late latency for the ok respective services over here. The these are the services and you can see the transactions, business transactions evolved evolved from this evolve from these services.33:04And then you have an application tagging as well. Now you can create an application and tag the services to the application. So the moment you can select your application here, you can put the application filter over here so you will get only what you want.33:24The environment suppose you want to see only staging environment details, production environment details, then, you make use of these filters, right? Provided you have proper tagging. I just go to the service map over here and it loads me, so this is.33:39Alright, so this is how all of my microservices interacts with each other, ok? I I am just randomly selecting something over here, nothing of my interest here. So, end to end flow map is there here. Some of the microservices are unhealthy, probably.33:58See you can start the investigation from here. I mean, ideal place is the alerts, active alerts and make use of the AI.34:18Or you can just make use of these, what, the flow map over here so you just can just go to one of the services over here. I just just clicked on one service here and then you can see that.34:26The that particular service, the recommendation service related stats are over here that request errors and all, but what I'm interested is this tag spotlight. I just bear with me, all of these tiles may not be populated, but some maybe. All I want to say is if the tags are.34:43Assigned properly to the respective backend services. In fact, the tags are as if the prices are having tags, then you can have a bird's eye view of the prospective service using this spotlight. E.g..35:03Yeah, this is what exactly I wanted to show you. The service.version is one such tag and in the spotlight it, we, in this case we only have one service. What if you have version one, version two, et cetera?35:20And if the issue is specific to version two, then the spotlight will highlight it. So in version two, you have ten K calls out of which two K are having errors. So easily you can understand my specific version is having the problem or not all. Similarly, which particular.35:35Particular here endpoint out of this particular service is problematic in which environment the problem is, which operation the get to or post.35:55Somewhere there here. Oh, that's not there there there ok forget. Get post request, which one is problematic? Which endpoint is problematic, all of those will be available on this tag spotlight very useful troubleshooting tool.36:06Spotlight. In fact, one stop shop to infer insights from the, from the for particular services. All right. Now you can just move to the traces. We just want to understand there are 02:24 errors over here and then you can just move on to the traces view.36:22And it lists you the traces with I expect this is a shallow Trace we can just check something else as well, but just understand the flow there, right? We just went to the service map, pinpointed one service and then tagged the spotlight on that service and understood or got some insight.36:40Rights, and then we went to the traces view. Here is a long trace probably hotspot or latency, and then there is an error trace as well. You can view more trace in your trace analyzer or just click on the trace and it will give you.37:00The open up the trace analyzer, ok? So this is one trace and then if I just click on the trace another window just opened and it will show me the waterfall and where exactly the error happened. Okay, so this is a shallow trace by the way, as I said, if it is a deeper one, then the entire course.37:19Trace the waterfall will be over here with the span performance as well.37:39If there are specific and now you can see the error stack trace as well over here, and unfortunately, I don't think this service is just a minute, is, is tagged to or ingesting the logs here over here. The good thing here is if the service is ingesting logs, click just click on this.37:43Widget, you will land onto the specific logger entry where the errors are highlighted.38:04Massively improving your top troubleshooting time, ok? So it's all about analyzing the trace, locating the error, and then do go going deeper into the logs and pinpointing what exactly is the error.38:09So this is the generalized workflow when you troubleshoot. So in this we just leverage the traces, logs, and of course metrics for alerting and metrics for comparison as well. I mean.38:27Simplistic one. So let me go to the service map once again. And in this case.38:45I'll expect something over here, another service. So what we did, I just want to tag the spotlight 1st on this service to understand or maybe get some insights. Okay. For that service, not all endpoints are problematic, but this post call, this post endpoint endpoint.39:03Point is the problematic one it has 2708 requests, 2708 errors.39:21We simplify to a massive extent over here. A service is problematic, but one post API call is the one which is having the problem, ok? So that's the importance of the, the, the spotlight. Now we can go to the traces.39:26There are many error traces over here because there are 2708 errors and there are some lengthy traces over here. I just click on this particular error trace to understand the waterfall again.39:45Okay, that's again a shallow one by the way, but it is a clear cut error. Somewhere down there are details would be there and if logs are tagged then the logs would be over here. So that in a nutshell is what.40:00The troubleshooting generalized flow looks like. I just want to show you one specific demo instance over here. So this is how right the logs. Now, this is just this is an automated demo demo. See look, we are at the exact same point of the spotlight.40:16And, now the, the, the, the spotlight has captured the log entries. You just click on the next one. See all the log entries are all listed over here. You can now for focus on the error entries over here. The good thing over here is all the queries.40:33Are automatically populated by only cloud rather than you framing your search queries over here. So the error entries are there, you can specifically note what exactly is there, some payment failure and then then proceed accordingly. So that's the beauty of it.40:52Alright, so I I think I it's, it's almost towards the end of the session. Again, I don't think this is an over all overwhelming one, but, expected outcome is you now know what are the different components and then how to just play around with it.41:11Before the closure note, if you have, if you are building your AI apps, AI agents and your llmiss, you can monitor your AI, AI agents as well as the LM, the token consumption, the AI hallucination, et cetera.41:31Using only cloud, the feature is already released. You have the AI trace, AI agent monitoring and A AI overview. I will just go to the dashboard and then just quickly show somebody has built some AI dashboard. E.g., this AI port.41:50It maybe not be complete ok by the way, so so this is the GPU utilization, then GPU nodes similarly, if you want to, there is something called tokenization hard economics. So this is even an interesting.42:12Dashboard, which is giving you the consumption of the input and output rock and the trending. So all, I mean the moment you build the AI apps, then you can leverage.42:30AI agents you can leverage only cloud to monitor the performance, the token consumption, the hallucination, etc. And the GPU underlying infrastructure as well. So that's AI monitoring. This is what I told about security monitoring. If you have secure app subscription, then all.42:40That you need to do is just come here and then understand what are the potential attacks real time attacks happening on this particular service or application. So remote code execution, e.g., log for J vulnerability is there scripting forgery is there. Oh, so the attacks vulnerabilities.43:00And then potential vulnerabilities on the libraries all are highlighted over here. Good feature in the, in the AI error, this is much, much needed feature, the security vulnerability detection, alright?43:20Before I conclude just one statement over there, how are we getting this data from? Configuration, of course is a next session, but you have to come to data management.43:36Go to available integration, it's as simple as that. Splunk only cloud has a massive set of integrations over here. You can monitor almost all of your assets in the estate by clicking on one of the tiles and then proceed, proceed, right? E.g., I, if I want to monitor my.43:51My Kubernetes environment I'll just click on this Kubernetes tile and then the documentation over over here.44:11Specify your provider. Mine is red Hat openshift. So I specify others and specify red Hat openshift over here. This is my cluster. All right. If you want to go for log collection, go for it. For now it is not. Now.44:18For these services, you can do auto instrumentation, meaning via via an annotation on the services and in each container, an agent in each container will be automatically added to the respective workload upon.44:36A fresh deployment or a fresh update. So this is called what? Mutating webhook. In fact, I just forgot mutate mutating admission webhook. So it doesn't affect standalone services, but whenever there is a an API.44:53The state change, then Kubernetes API server informs the Splunk operators. Splunk operator injects an init container into the workload. So this is called why I'm mutating admission a webhook. The webhook is not active when a service is running and in the desired state. All right? Next.45:13That's it.45:34You got the helm command, add the helm repo and then do a helm install. All installation can be managed from the centralized UI, but if you want to do it via your CICD pipeline as a terraform in automation or puppet, you can do it. The documentation has the command.45:35And the steps for doing that. So it's not that you always need to need to do it from UI, but this is the most easiest way. Helm commands are there just run this helm command in your cluster. The, the collector will be installed and then data will be sent.45:55Similarly, you install agents on the virtual machines etc. I'm just I conclude there and let's open it for QA, Kevin? Yeah, not many today. I am also understanding this is a new platform and we used to use this EIS for.46:10At dynamics mainly and.46:30I think many of us will be absorbing these information and use them accordingly. Just a few things I have questions around, ok, you guys seeking support, is it the still AppDynamics supporting team and things like that? Well, us as a Sujith and I and the team.46:32Team are still we are also supporting.46:52Splunk only means observability cloud in the same way, as how we support app dynamics. So feel free to ask us, ok, can I do this? How do you do that? We'll help you on that. Or if you have a dedicated person, then that will be that. But yeah, I think we don't have many questions, so.46:55So I think it's time to let everyone go. We're a bit running late. So, thank you very much. Thank you Sujith and feel free to register the latest EIS.47:15Yeah, just for for I just pinged you one more link there. If you if you're interested you can just go for it. That explains the Splunk service level description, the compliances, the realms available regions etc. Right? All right. Thank you everyone.47:27