Beyond reaction: Proactive monitoring with Splunk AppDynamics

Oh, let's start. I believe we have few attendees here and.00:00We have the recording going on now, so the ones who are not able to attend from the beginning, they can use the recording to see what they have missed. Right, so my name is Sanjeev. I am one of the names CSS at AppDynamics.00:16And I have Kevin who is also another expert CSS in our team. He's based out of Sydney, Australia. So let's dive in. Kevin, could you please start your presentation?00:32Sure, thank you. Hope everyone can see my screen. Again, my name is Kevin. I am the expert CSS basically Sanjeev's technical counterpart. Some of you guys who is in the account with, with a CSS will be familiar with their colleagues, and so.00:48Some of them I have attended previously CSS.01:08Sessions which you might met people like Paul or Sujiv, and other folks. As usual, video is recorded and if you have any questions, feel free to leave it in the chat. We will be either answered on the goal in the chat or we'll leave it at the end for the que.01:11Q and A depending on the length of the answer or the complexity of it or if it's worth discussing going forward.01:31Other than that, today's topic is beyond reaction proactive monitoring. It's one of those topics where we usually run on the enablement session from introduction to all the way beyond to, new use case. So.01:39So we previously might frame it as.01:58Our proactive model, but I want to expand this topic to more about strategic way of thinking of proactive monitoring, not only about appdynamic itself, but probably a bit more thinking along with.02:04What is proactive monitoring? What sort of boundaries should we draw? And I have a lot of customer that is really drilling into the rabbit hole of, we need to be very proactive. We cannot be reactive.02:23Being really bounded by this, that boundary. I just want to put a few words in there, then we'll kick start our standard session. So the way I see proactive monitoring is we have four separate levels that you can think of as a discreet.02:39Discrete, how should I put it, like a discrete kind of, process. So level one will be something we almost won't be able to see, which is there is a change in a very low level infrastructure level problem.03:00In nowadays it might be on your cloud vendor, which they might not be even noticing that kind of micro anomaly, then it pass it on to one problem to multi problem because it's kind of cascade down into a more specific node within that architect.03:20Structure CPU that is spiking or database start to have a bit more, it takes longer time to get queried.03:40And then it walks into something we're more familiar in the appdynamics world where, ok, there is some sort of response time or error rate that has crossed a certain threshold Whereas if you're familiar with health rule that will kind of trigger errors, and then obviously the.03:47Last step will be when the actual user internal external will experience a problem, then they feed it back to you, raise a ticket, give you a call. It's it's something that, ok, the reactive auditoring start to happen, right?04:07I would say from experience observability applications are working mainly around the level three and four and there is an exception of having a synthetic monitoring where you create level four and then you kind of being react.04:25Actively proactive.04:45And kind of fix the problem. Obviously, the best advice would be write better code, having better deployment, having a better cycle, better QA, but this is not part of the discussion. We want to stretch level three and level four as much as possible and even trying to.04:47Stretch to level two.05:07If possible, but at the same time we acknowledge the fact that if you are monitoring everything on the very low level, on the very marginal, say e.g. you want to look at every CPU that's above 71.2 % usage, things like that, then you'll be.05:09Putting all the effort into that doesn't really worth it. So we acknowledge that limitation, but at the same time, we also want to know that how far we can go.05:29Is there anything, e.g., AI can help me to kind of understand or kind of you kind of lay down that labor work to them? On the other spectrum, you also want to know, you also want to acknowledge that.05:39The more detailed, the more low level, the more singular rate, the more singular metrics that you look at, the less likely you can figure out what's happening, but you can only acknowledge the degradation and not understanding the root cause. You can understand the root cause but yet it doesn't understand the bigger picture on what's happening, so you need correlation on different things. They're kind of stretch it to level.05:54Through again because the bigger the system kind of being impact.06:14Then the e the the bigger the problem is. So you want to acknowledge the fact that if you want to know the bigger picture, you can look at the smaller picture. That's the theory. Now in appdynamics, we, our basic principle is business transaction as you may know, then we want to show you what you want to know.06:18That's where the original founder and concept that has begun. So we're trying to show you at level three what you want to know, but yet we want to have those metrics on level two that's behind the scene. So you can kind of play with that. E.g., my advice will be if you want.06:38The Structure level two.06:57You can try to monitor the rate of change, not just the specific rate threshold. Obviously the baseline will really help, but, trying to have more creative mind on if you want to stretch more, how far can you stretch and what can you do, what you want to do. I will also talk about anomaly detection.06:59That's from our latest, not the latest but the recently released and knowledge detection root cause analysis, these kind of things will be within today's topic. But yeah, I just want to put that out for you.07:19Probably food for thought food for thoughts for you guys to have a think about and yeah, that's the forward. Sanjeev, do you want to kick start with some of the agendas overviews and things like that? Sure Kevin. Thank you.07:33Alright, so today what we are going to cover here is introduction to the monitoring in AppDynamics and how we can then understand how we can do that proactively to understand how our metrics are.07:52Knowing how our components are being watched et cetera, and accordingly we can take actions. So another part of that is alert and respond overview. We will look at how the alert and respond section in, in the AppDynamics control that looks like.08:10We have anomaly detection and health rules policies, actions. So as in how the health rules are defined, what are those, what are the default settings and how policies are adjusted and what actions are suggested and what, how, how we can take them.08:28To fix the issues that we encountered yeah and then in the in the end we'll go through a demo to understand the practical of what we have learned so far in the theory in the upcoming slides.08:46Yeah, next slide please. All right. So let's cover alert and respond overview. I'd like to show you how you know an overview of the alerts and respond the architecture of it.09:01So on this screen, this is for your understanding of, you know, how the proactive monitoring is done so you can understand from this diagram. If you look at the left side of it, here are the components that are being washed or evaluated.09:19And in the center of the screen then we have triggers. This is this comes under health rule violation. We will in the later slides will understand how the health rules can be defined and when they are violated, how the trigger, how the events are triggered and.09:37That depends on the policies which you can see on the right side of the screen. Policies are, They they tie the triggers and the actions together. When these health rules are violated, then the policies they trigger and then suggest the actions that that need to be taken.09:57So those can be notifications HTTP request or it can be custom actions or some cloud auto scaling. All right, next slide please. Now this is the architecture.10:14Of alert and respond overview and we will talk about these in detail in the next slides, but here you can understand from this architecture how this whole process works. So we have to define metrics and thresholds.10:31And once that is done, then you know we have to take care of the health rules, define them. There's another part of it that is anomaly detection, which is part of AI ops. It basically works with ML. So we'll talk about that too.10:48Now in appdynamics, you know, we have a lot of metrics that reflect the performance of our applications, so we can monitor those and then define the health rules accordingly.11:04So, in the next part you can see when we collect our data source, the health rules have their conditions which we configure, we'll be looking at that also, how we do that.11:20And then like I said, the anomaly detection part is the AI ops. So we monitor these events and under health rules once these are the condition that is met. So the condition can be.11:35It can vary depending on environment, so we have to define what's normal for our environment what's the threshold that's considered normal and.11:55What's not? So we define that in the health rules and policies. The policies then help trigger if a threshold is broken or if a normal event is the the threshold of that NO normal event is broken and then.12:06A trigger is through which would then in the end suggest some actions. So these actions can also be defined. Now what kind of notifications for for such an event, what kind of action would you suggest so that can be also defined in our.12:24Environment in apti. So we can look at that in our demo also. Yeah, and there can be diagnostic remediation or it can be HTTP request or custom.12:44Actions and so on. Next slide please. Alright, so 1st part of this, of, this overview would be health rules defining them. So let's have a look at that, how that is done.12:58How does it look like and where in our AppDynamics control of.13:19Screen where do we change that? And how do we create these health rules? So we have some default settings, we can edit them or we can create our new rules as well. So let's have a look at that. This is the sample screen.13:25So you can see on this particular screen what we are showing is how we navigate through these health rules. So when you click on the alert and respond on the left, you can see under that we have health rules. There are a set of health rules that are shown in on the screen, which can be.13:41E.g., it can be a CPU usage or memory usage et cetera. You can double click that and then edit those or you can even create your own new rules from the top of this particular page. You can see there's a plus sign under health rules that can be used to create new rules.14:01And there's a toggle switch that you can also see see on the screen, which is, it applies to all the health rules. If we turn that on like it is shown on the screen now, right now, all the health rules that we have defined will be working and they will be enabled. But if we toggle that off, none of the health rules will work. So.14:19So that could be.14:39In a condition where it's a test environment, we just want to know or or maybe you know that there's a downtime. We do not want to be alerted for any thresholds that are broken, but we want to know how these maybe it's if it is server monitoring, how it's being performed.14:40Or if it's database how how that is working, so all that can be done while toggling off the health rules which and yeah so that's, you can see there are default health rules, then we can create new health rules, we can edit them.15:00And you can also see the status of these. So the ones that are not enabled would show you on on this screen, we can see that there's one that is disabled, the others are enabled, so you can also do that. And when you open these.15:19Health rules. You can then edit more rules into that you can make changes as per requirements of your environment. So yeah, let's dive into that and look at more closely. I'll pass it back to Kevin.15:37Thank you. All right. All good. Just very quickly, healthrow is comes in out of the box with a few of them that we think that is important. Surely people who has been using on a daily basis will know that it's usually utilization to a.15:55Like, business transaction in terms of.16:14Error per minute, things like that. The most important part I will personally suggest is go into there, look into these default error default health rules, if they are important to you, turn it on, turn it off. If they're important to you, does the actual works for you?16:19That's the most important part. Is it going to be a static threshold? Is it going to be a dynamic threshold? Once you've toggled that, that basically gives you a good basic monitoring given your BT is very tidy. And then from there you can start to make new health rules.16:38You can make your own policy, things like that. But just remember recommendation out of the box is to make adjustment on the default rules and after that you make new rules. I'll show you in the environment very shortly, so don't worry about looking at the screen on that part.16:56As I said, the static dynamics, health rule, people who are familiar with the baselining dynamic is basically that. We use a lot of time we use standard deviation to do that.17:14But static most of the time is sort of very, very popular, that is if you are aligning with some sort of SLA with your customer internally externally static is very popular While dynamics dynamic is a lot better when it comes to, when you are determining.17:31On, ok, I want to figure out the problem, but then this environment is dynamic. How should I see the differences? But yeah, I think you would just have to consider what is the strict rule then you apply static rule.17:51What is diagnostical, then you use dynamic, that's the general rule. Now in terms of configuring health through, it is easy, but there is quite a bit of a details you would have fill in in terms of what metric, what value, again, is it dynamic?18:09Is it based on baseline? As you can see in this particular screenshot right hand side, you can see second row, you can see this based on standard deviation, which is which is which I'm a big fan of if you're using baseline already in your dashboard.18:29Continue to use that and there is a very it's a recently relatively more new feature where you can trigger when the violation happens, how many times for the last 30 min. This is very helpful.18:47When you have alert storm in most of the case, you can make it really big and then slowly reduce it. That's what I would recommend. And then obviously after you have those out of the box health rule, you can start to.19:04Look at, ok, I am trying to roll out this batch of code, this batch of new application or not application but these batch of new codes, and I want to look at that specific part of the application, how is it going to happen? Then you can build a health around that specifically specifically, and these.19:21Kind of, recurring theme can be saved as a template. So whenever you roll up a new code base then you can use the same template, turn it on and off, which I can show you later. Now this.19:40Particular slide dragged from baselining if just for a reference of how baselining work, what can it do? And then you can apply it on the health rule. You can do it on monthly, weekly, daily roll.19:58And then you can compare it to, week for week, things like that. This is basically for you to, again, if you have a cycle that releasing quote you want to compare, this is how you wanna do it, but then you can do it in a health rule way.20:15Where I want to compare before the COVID release, then I will I and then after if that has that amount of increase in say CPU usage, then there's a problem. That's one of the things that people would do. Another thing would be, ok, seasonally what's happening in that particular time.20:31When it's peak, I want the thressel to be a particular standard deviation away from my off season or you use last year's.20:51More simulation to do it this year. You can do that too. The tools is here for you to use and then obviously you're bound by the static rules that for your SLA.21:02Now, this is probably the same thing baseline deviation, how, how should your health rule be using standard deviation? The script asked me to tell you how these works in terms of the calculation. I'm not going to bore you out of it.21:18Everyone knows the deviation how it works, so I'm not gonna talk too much about this. The one thing I would recommend is if you're really driving into the weeds of it, you can go to the metric browser and investigate the baseline by yourself using a separate screen.21:39Looking into exactly what should be the range that's in morning, does this has a problem, then you can come back to build that health rule. That is quite helpful as well, but it takes time. I would recommend that if you are a specific administrator.21:55When there is a health rule violation, we will have what is normal warning critical, if you look at the new relatively newer screen on an event that happens that.22:16Triggered a health rule, you were able to see when does it start, when is it upgrading from what?22:35Health normal warning, and then critical and how the event ends or sort of action has been done. It's a lot more comprehensive in depending on your version of your controller, you might have something called root cause analysis.22:43In your customized health rule or in your AI knowledge detection. These even give you summary of the RCA and also action taken potential problem, which give you a comprehensive view.22:59Helping your reporting and also investigating. Policy, I'll say we call it a playbook of the whole model where you want to, as Sanjeev said, tie in action with health through tie in action and health through together.23:16I'm not gonna put too much time into click here, click there, but basically I will recommend this is the 1st thing you want to do when you're thinking of alert and respond model. You want to make sure that, ok.23:36This is the playbook I want to have, what action when some sort of things happen, and I want to I want to put a name on the specific item, say, ok, these are all about CPU usage of this application or database query for this particular business transaction.23:53Then I'll put it up, what sort of action I want it to be? What sort of trigger which is the health rule you want it to use. Then you build that kind of playbook together with the right naming convention.24:13When you come back to, say e.g., I want to redo a new playbook for a new application. This is where the template is at. You just changed the name and you can just use the same thing. That will be the best for scaling a bit more.24:29Generally speaking, if you keep this very clean, then your replica you when you replicate these kind of playbook will be very easy. If you want to change small little things, it's not too hard.24:48But getting policy right is the beginning of getting alert and respond model right? Now when you create a policy, it takes a lot of time because you want to make sure everything is accurate for what it is. E.g..25:02You want this policy to involve.25:23Both warning and critical when it's triggered, you want to make sure the scope is in the right one with the right object like which note or business transaction you want it to be involved. You even want to add say I also want to look at if the business transaction as a whole is slow.25:26You can do that or if you want to look at, when it comes to user experience, that's also that. So this takes time. We have a kind of easier mode of user friendly mode where.25:47You only click a few things and then it automatically give you notification. You don't choose a lot of things that will be a good start. I understand a lot of our customer will only use email appli email notification. There is NO customer action or there's NO ticketing. You can do that, but obviously if you want to.26:02Have a more comprehensive way of, ok, I want to have these three healthy triggered on these few things specifically to a scope and I wanted to pass on a pager duty system.26:22Then you will need to use this menu mode. In terms of action, I think it's quite straightforward. The more than 80 % our customer would just use, send an email.26:35The started diagnostic session is also very popular, which you target that business transaction and have snapshots or even take a threat dump, but just be aware of the workload. If you have script that you can run.26:52You can do that. Sometimes if you need to do certain things internally, you can restart something you can. Jira is natively integrated so you can use that directly to Jira.27:10HTTP requests basically expand the scope of that integration to other vendors and obviously you have custom action which is everything else. This is basically a notification that you will find.27:26When, when you have that toggle on, I'm not gonna put too much effort into it. It's basically telling exactly how it happened. Snapshot is basically it.27:44Threat dump. Again remember it will have overheads and things like that you need to be aware of. Remediate remediation action is relatively rare for my customers as far as I can see.28:02Because it will have some script involved, Sometimes it's not everyone's flavor, but it's there for you if you really run, want to run some simple script. And you can see we have this option of email address for approval. That's is.28:19That basically explain why NO one wants not a lot of people wants to do it. Jira action, very straightforward, go straight to a ticket. HTTP requests basically gives you that ability to expand to other vendors for the ticketing.28:36ITSM systems.28:56Okay, now let's put some effort into this anomaly detection. Previously we have talked about as Angeez had talked about this particular screen. You can see in your 1st page of alert and respond overview. On the right hand side, there's this anomaly detection.28:58It is a new pathway in this particular model where we use basically machine learning to grab metrics. It's not rocket science. We basically grab certain metrics, trying to correlate with each other, have some condition ready, say e.g..29:19If that particular note is not active, there's NO data, then we will ignore it, things like that. And then we'll run something like this, correlate the variance of a few metrics.29:36And then we try to understand, ok, is this really an event? You don't need to create a health rule around that, which you can, but it will become an event. You just need to create an action for it. Generally speaking, notification will be enough.29:54You can see the assumptions here like we see at least 20 min of this duration, you can see, and normally in error per minute, average response time, calls per minute, then we'll kind of, ok, that might be a problem. This applies nowadays.30:10Is to not just APM also database.30:29User experience, obviously they collect different metrics. You can toggle it into different kind of sensitivity. My experience is always normal or above, otherwise it's not quite meaningful.30:33Unless your environment is very, very, very sensitive, otherwise I would put it as at least normal. You can see how they differ from each other. Obviously healthy is something that it comes up from your head. You need to consider everything and then set it up.30:50Or normally detection is purely numerical. We use mathematic equation machine learning to understand. There's NO any toggle, NO configuration beside of the toggle.31:10I Normally anomaly associated with business and now it's expanded to a lot more frontier. So, I will show you that, which is very exciting. Sorry before we go ahead, I also want to talk about anomaly detections availability.31:26So at the moment obviously customer will have it and expanding. For on prem on premises customer, traditional on premises will still be pending. Simply. You can understand because of the machine learning capability.31:45That it's hard to put it on your system. There is a, what we call an OVA adoption, which is a newer version, newer architecture of on premises. That part we can have anomaly detection built into it. It's, it's a new architecture.32:02It's VM based, not even VM based.32:22There is a lot of movement in that aspect. So if you're an on premises customer and you're interested in that, we are happy to talk to you about if you want to move to a newer on premises architecture. Or you can talk to your customer success team.32:28If that works for you. All right, break pack best sorry best practices very very quickly running through and I will run demos and Q and A So you want just these are this general advice.32:45Most of the customer has problem of having a lot of noises, alert storms, but if you want to kind of future it up very quickly in the level of alert monitoring, you want to configure.33:04Say you want it to be three times in a 5 min window to make it a bit more concrete, that can expand based on your usage. But also you also need to remember on the other spectrum of if you do it too in a too many times.33:20Times.33:40Then you will be missing the level two as we described. Second one is the interesting one. This is why an AI anomaly detection exists. You want to have multiple signal before you kind of, ok, this is a problem. So you want to look at errors.33:41As well as the calls per minute, you want to combine them together and then.34:01Okay, if these two health rules start to happen or in one health rule you have these things, all happens together, then that means a problem. You can do that and a longer detection basically trying to do that in just the metrics and obviously minimum threshold.34:06Especially in dynamic baselining, you will need to be realistic about, ok, how many transaction is happening, handles if how much you can handle, what if it is ten times? Is it anomaly?34:26Is it an anomaly? Is it really not? What sort of time frame? Then you kind of make your, more specific change or low level changes on your health rule that kind of fine tune it into a more more accurate.34:42Meaningful notification, but in general, some very basic things like naming, making sure that your health rules are tagging with.35:00It's just some entity, not everything, some specific object that's also very helpful. I want to show you this page before you go ahead that shows you why meaning why the naming matters a lot. On the left hand side is where you set up in a controller. On the right hand side is the email notifi.35:14Chris is one of our US counterpart. If you don't set your policy health through, if you nope it's ok and also the condition properly on the email when you wake up in the middle of the night, look at the notification.35:35You can't get a lot of context out of it and you need to head to your computer and do this and do that and turns out it's nothing. Naming in business transaction or in here in in the health rule monitor.35:54Proactive monitoring are also very, very important. So these are basic, but you want to do that before this happens and it screws up your night. So that's the presentation. I will move to.36:09The controller environment demo controller environment. If you have any questions, feel free to leave it in a chat and Sanjeev will monitor it for us. Yeah, we have one in the chat.36:27Oh yeah, let's look. I have a question. If all the notes in an application go down at the same time, assuming AppDynamics agents lose network connectivity to an internal proxy issue with the AppDynamics controller, it takes three to 4 min to trigger an HTTP action on.36:45For email notification.37:05This impacts proactive monitoring primarily due to the 2 min metric reporting latency from the current time. Here the client says why they received this alert so late. Now you mentioned this is I mean.37:07This is a very specific scenario. I'll give you I'll I'll say you if you 1st comes with the agent has NO network content connectivity due to.37:28A proxy issue that has predom previously set up, then that will be, we are already pointing out the problem. That is some explanation you might need to wrap up nicely and explain.37:44In an ideal world, we don't want that to happen. I think it's pretty obvious that you just need to phrase it properly to the client, explain why this happened, as you mentioned.38:03On the internal proxy, if it's our controller, like if we have problem we generally have 99.9 a, if I'm not wrong, that might be a wrong number but we have quite a SLA uptime holding.38:17If you see the latest data, we could see there is 2 min old metrics in the controller. I'm not sure why, definitely that's not something that we want. If you can see there is a lag.38:39I really want to see what's happening with, our controller, not your controller, but the controller's controller. I want to see what's happening. That's definitely not a general case. We will have to look at, ok, is there any specific, what sort of proxy issue it is? Is there some.38:53Something on our side that we need to fix, is it a security problem?39:13But, e.g., I am in ANZ. The standing ANZ is quite stable most of the time there's one big problem I ever expected. This is something we will need to discuss with support with your customer success team.39:18But I don't think this is a normal thing to have 2 min old metrics in the controller, but yeah, I would recommend that. Sorry, I can't really fully answer your question with like this is the answer, but, I don't think that.39:36This is something happens.39:55Expected for all controllers. Oh, Sujan, if you can, send me, I will send me send you my email address. Feel free to shoot me an email.39:58Tell me what's happening. I want to have a look yeah that'd be great. So Susan, like Kevin said let's describe the whole situation in an email if you could.40:16Share that with us.40:36Maybe we'll be able to provide a better resolution or answer to that. Yeah pretty specific situation here. Yeah. And also I want to send, I want to send this particular case to the right person as well, one of our team members that.40:37That might be in your region and things like that.40:57Okay, so again we always start from the page, the very front page. You can quickly jump on on the left hand side to alert and respond. But remember you always want to get into an application or a specific.41:00UX database that you want to change to reduce your scope. The easiest part we'll be looking at is our demo application e commerce. So as I mentioned on the right hand side, you can see there is this.41:21Policy set up wizard. This is what I was mentioning the easy way of setting up policy that will cover once you've set up, you have covered everything, like every application, like everything within this application that will have this policy included, then you can go into the.41:37That and kind of narrow it down. Unfortunately, I don't have privilege to create an email address to receive alert in this controller, but that's how it is. Just fill in your name, the name, if you haven't configured an email.41:57Then you need to, otherwise put an email address save, then you'll be ready. Alright, so you can come go to create policy manually here. If not, we can go to the normal pathway, which is here. Then you can create a policy.42:17This we have looked into, what sort of things you want it to be together to have this happen. This health rule scope won't won't be available unless you have health rule events, which you need to define what health rule.42:34Object scope is basically what do you want to be involved, that will be all the below business transaction tiers and stuff. And action wise, you will have to create these previously, if not you can create it here.42:52Most of the people would just use email and you can also s change the email template in the alerting template. As I said, the diagnostic section is also very popular, which will take snapshots.43:12And then in terms of yep, on the right hand side, you can see this save as save config as template. This is very useful when you're replicating the playbook, the policies.43:30From application to application, you want to save a new template and then overwrite them whenever you want to. And obviously you have this mass button to do mass actions. Health rule.43:45As I mentioned, these, say e.g. these CPU utilization and above, these are all out of the box. 1st thing I'll do as someone that's ok, I want to start monitoring more, make it a bit more detailed, make sure health rules are right, policies are correct, I'll go into here.44:00I'll stop playing with this.44:21Because these are something generally speaking, most of the people will want to see. We want to look into, most of the time, affected entity page you want to reduce to start with, to make sure you, these are meaningful and you minimize alert storm, which is too many alerts.44:23And then criteria you want to slowly slowly change it according to your use, and add condition to cross check, as a whole, this becomes a bit more complex, but it's very helpful and if you can do it in within one particular health rule, that would be great.44:43Warning and critical you can swap between copy and change that will help you a lot. But I wouldn't recommend start with custom right away, start with the out of the box and then slowly change it there.45:05And worth mentioning, you can set up scheduled times and, you also want to set things set set this wait time after violation.45:21This basically gives you some time before it updates its status. So it makes sure that these are not just a very short timing spike.45:36Quite straightforward, just keep anomaly detection for a quick 2 min action. If you want to create a s a list of action 1st before you go back you can, but.45:53As you see and helpful you can do it. This is very helpful when you are trying to segregate different email lists and things like that or you want to have specific ticketing system for here and there. Okay.46:06A knowledge detection, you only, all you need to do is to toggle this to on, then the, and then obviously you, if you want everything to be involved, you can have that. Otherwise you can limit the scope.46:22But once you toggle it on, it will start to be trained and it will say ready in here. Obviously if any other separate problem, we can look into that. This is obviously under.46:39The auto traffic bracket, so we are not including them as you have locked down your other business transaction. Once these are ready, you can look at anomalies. This will only include the events that is detected by anomaly detection, which is not the event you look into the.46:55Application screen and an event. It will include these, but this page will only include the one within anomaly detection.47:16Obviously within that application. Now when you double click on it, it just run machine learning, it just run the AI summarization by itself. It tells you a summary.47:23And then it was try to, obviously when you think of troubleshooting, it comes with the whole idea of following the red, going into drilling down, the suspected root cause is doing that.47:39So it will tell you which particular node or what's, what's the problem. Then you click into that, it will explain to you why does it think that way? And what recommendation does it have? I'll give you, I'll tell you that recommendation does not always tell, always.47:54Give you all the answers, but it's a good start. If there is any snapshot that you can reference to, it will. Then you can jump from here to a snapshot, which I personally think is really helpful, then you can do your manual way of drilling down.48:15Obviously that's part of troubleshooting in the in another session. I won't talk about it too much. If there's an action that's attached in your policy, you can do that, then you will execute action. It will tell you that.48:34And then if you want to, this is also very, very helpful when you want to take screenshot and report, tap up view dashboard during the anomaly, it will show you exactly that timing that particular.48:50This application dashboard and you can see the load and all that. This is very helpful. Ok. The last bit will be alerting template. This is basically.49:07When you're trying to have an email sending over, what sort of template it will be, how do you want it to look? You can do that. Other than that, that's pretty much on a very high level demo.49:25I want to see if, say, the reason why I'm clicking the database is simply because I want to know, if it's available, if anomaly detection is available. Luckily it is.49:44So yeah, so if I'm not wrong or the detection capability is now deployed, you can go into every single section here. They will be available, feel free to use it and then use that as a crouch to help you going for your proactive monitoring.49:58Oh alright that's everything we have for today. As always, if you have any questions, feel free to leave it here and.50:16If you have any topics you want to discuss or you want to have an extended session with us after this specifically on, say, ok, I want to look at anomaly detection, I want to know more mechanics about this. Feel free to do that. Just send us email or just.50:32Put it in your survey after. I'm happy to reach out and have or have one of my teammates in your region to reach out. Recording is available on the Cisco community page. If your colleague wants to, they have missed it, feel free to have that. Going forward, we'll see.50:51To have our.51:11Fortnightly EI session, now we'll be involving not just appdynamics. Also, we'll be involving a lot more other Splunk platforms. Feel free to subscribe feel free to register and we look forward for your next attendance. Thank you very much.51:14Thank you Kevin, that was great. All right, do we want to take any further questions while we have a couple of minutes left? Yeah, ok, I have one question send it to me.51:33On premises with anomaly detection is something that might not be soon especially in traditional, as you can think of, you can't put the whole AI into your on premises traditionally.51:51Otherwise you can imagine the resources that needs to be taken. That's the problem we have, we're trying to tackle. That's why we have this VMware or we call it OVA version of new ways of using on premises. That's.52:07Will likely to be the future when it comes to.52:27Any AI capabilities. Unfortunately at the moment, if you're on traditional on premises, you will have to consider a migration towards a new architecture. Obviously there's a lot of prerequisite and all that you want to discuss. But yeah, I don't think.52:31In a foreseeable future traditional on premises will be having this particular feature, same as RCA, same as any, say e.g. the upcoming potential AI features. That's my answer.52:49And yeah, I don't have any other questions sent to me. Anything from you Sanjeev? No I didn't see any other questions after Susan's questions? All right. Yeah.53:08All right. Thanks again Kevin and thank you all for participating in this webinar. Really great having you here and we hope to see your participation in our next webinars and you will be notified about those in the future. Thanks again, have a great day ahead.53:21