Splunk AppDynamics: Alert and Response Part 2

Oh. So welcome back everybody. We are back for another observability expert insight series, and this series that we're on currently in.00:00Question number two or episode number two is the achieving proactive monitoring and today's session is gonna be about Splunk AppDynamics alert and response level two. We built a good foundation last time on the health rules and and how alert and response works within appdynamics.00:20You know ultimately trying to get you to be more proactive, being re reactive, you know, obviously we're still a good tool for that, but being proactive is gonna be where you want to be. You don't want to be woken up at 02:00 in the morning and and not have any idea of what's going on. So meaningful actionable alerts, you know, and.00:40Is what we're after and we are lucky to have our expert speaker Drew Osborne back again for this session who's gonna be able to build on what he went through last time. And then we also have Eugene Brain on as a panelist, so if you guys have any questions along the way, and we'll cover that on the next slide.00:59Let's go ahead and get into some house cleaning items. So again, welcome. We are part of the customer success organization here at Cisco. So everything that we are working on today, everything that we are going to cover, the.01:19Content that you see before you, it's all created for and by Cisco, so please NO funny business with our content, but it is meant for, for, IT professionals using Cisco observability and Splunk observability suite, you know, as a part of that, we're gonna be focusing.01:39A lot on app dynamics, but, you know, we want to encourage you to ask questions along the way, either via the chat or the Q and A section, and that Q and A section is actually.01:59Down in the bottom right corner, there's three dots on Webex. If you click on that, you'll see Q and A the benefit of using that section is going to be, you know, if you put in a question, we answer in line, that question and answer will be preserved there for everybody else to see. So they, that.02:13That's how you get the most out of a Q and A, but at the same time too, if you put it in the chat or have any feedback or wanna share your experiences.02:33You know, keep it more of a discussion, we can do that. We will have everybody on mute though, so, you know, we, for privacy reasons, we need to keep it that way, but at the same time, you know, if you have a, customer success engineer assigned on your account, you can always pull us in, just let your.02:42Your account owner know and we can come in and have working sessions where it's more of a back and forth discussion versus you having to chat. But for right now, for today, we're gonna keep you on mute, but we encourage you to give us feedback, ask questions, share your experiences, and we'll definitely try to have.03:02Have a Q and A section at the end of, of this as well. So if you have questions and you want to hold off until then you can or if you wanna ask throughout our panelists will will try to get it for us or get you guys situated and then anything left over we can bring up towards the end in the Q and A but.03:22Without further ado, I will go ahead and hand it over to Drew.03:42Hey, thank you Chris, for the introduction and great to have everybody here today. It's a pleasure to be able to continue our talk track and present for you today. To to just quickly cover the agenda, most of the stuff we're gonna do today is gonna be.03:50Directly in the controller, but I'm gonna start with a quick alert and respond recap, and after that, we'll, we'll go into the controller and I'll recap as well the the collateral that we made in the.04:09In the 1st session to kind of bring people up to speed if you were not there, and then we're gonna pivot and talk a little bit about baselines in shape, as well as some static qualifier stuff and finally kind of wrap up with some metric expression stuff, hopefully showing some.04:23Some real world examples that if you're not doing these things today, maybe they help, you know, help level up how you're doing things or help, you know, enhance how, how we're we're doing certain conditions within our health rules today.04:43So with that being said, quick, high level recap of the alert and respond module. We have three main components that that make up that module. Health rules, health rules are gonna let us specify what parameters, what.05:01Metrics we're collecting and what normal or abnormal means. Policies are are basically something consisting of a trigger based on one or more events and, and they tie in action to, to, to that trigger.05:20And, and finally actions, they're predefined, they're predefined thing that we can use within policies, things like send an email. We have the ability to, to make an HTTP call, call a webhook, integrate with other tools, things like that. But essentially policies tie.05:37By a health rule to an action. That's that the easiest way I've found to explain it and there's some advanced features you can do of course within the policies. So with that in mind.05:57This is just a kind of a graphical representation of what the alert and respond module kind of flow looks like. We have health rules key here. The top blue dotted line contains the health rule component. The middle one contains the policy component and the bottom one contains the act.06:10Components. And so you can see policies really tie actions and health rules together. You can have multiple policies, multiple health rules inside multiple actions inside of an app, and they can be distinct within each app if you wanted them to.06:31So with that, let's jump over to the controller. I'm gonna quickly call out for those that were on the 1st call, about a month ago, we, we started in the AD air app today we're gonna be working in a an.06:47An application called sock shop, but I want to be clear we've mirrored the the health rules, the actions and the policies from the AD air app over. And why did we switch to this sock shop app for the sake of this call? I wanted there to be a little bit of a, a uniqueness that.07:07It was pretty much all normal calls or all error calls in the other there was NO slow, very slow stall, and so I've done a little work to.07:27Have the tool slice and dice up transactions a little bit differently so that we could, I could show you how, how could somebody perhaps alert on these things later? So, with that being said, I'm gonna dive right into alert and respond. We're gonna start with health rules.07:37We, the two health rules that, that we crew that I consider collateral that we created in our last talk are gonna be these two. So I'll take you through each one of them. The rest of them I'm gonna take you through.07:56Today and we're gonna actually build the V two health rule based on what we've done in response together as well. So high level 1st one, app agent availability. Sometimes people call this application availability, but this is essentially is your.08:14Our app reporting metrics to the controller. That's how I describe it to my.08:33The the people that I work with and and I say because it can be a communicate, your app can be up, but your we can have a communication problem that's preventing the metrics or any telemetry about the performance from making it to the controller. And that's how I like to describ.08:39Drive this, so you'll see I I name it agent/app/network. You don't always know which of those categories it is, but it's.08:59Indicative it's indicative of a problem when it fires if it's properly tuned. So in here, both of our health rules are gonna follow the same, overview configuration. They're both enabled always 24 by seven. We're gonna use the data from the last 30 min.09:09That's of time.09:29And wait time after violation is also gonna use the default of 30. In this 1st health rule, the affected entities are gonna be the tier node health. Again, cause we're targeting app agent availability and that is the, the, the area within app dynamics where that metric exists and.09:30And we wanted to affect the tier health, I only have one node in a tier, so if that node goes down, the tier is not functioning, and that's why I wanted to impact tiers. Say I had a highly available app that had five nodes in a tier.09:50And one was down. Maybe I only wanted to impact that node cause four online out of five is still up. That's the type of, of things I want you to understand these, these settings toggle, right? And we really kind of tried to drive home all that in the 1st meeting so.10:09If you missed it, I encourage you to go back to that recording because we talk a little bit about why you might select and change these things. We're not gonna get into that so much today. I'm gonna talk about a few key things though.10:29Critical criteria. We selected our metric. We're just looking at a single metric and we're saying it's less than one. I can I wrote this metric reports a one when my application is up. I'm I'm also what this means in practice, I wanna go back to the overview.10:44To call this out.11:03Because we're looking at 30 min of data, this is saying right now if one out of the 30 data points cause I'm always gonna have 30 data points in 30 min within appd, if one of those data points is less than one, I'm gonna fire. I call.11:04That out because if you end up implementing this in practice and you start to see a lot of flapping.11:25Meaning it does, it fires, but when it fires it only fires for a minute and it goes back to healthy. That's where we can say, well, now 15 out of the last, so half of the data points in 30 min need to be.11:31Less than one for this to fail. So you can impact the flapping with this. I just want to call that out again. We're gonna go to application response time is too high again always 3030.11:47Affected entities is overall app health. And I want you to remember these numbers on the call. Critical criteria 500 and warning criteria 400. So that's the, the static threshold that we started with.12:05And we're gonna come back to this health rule, we're gonna make aV2 version of it, but I want you as we pivot to talk about baselines, I want you to remember that our critical and warning criteria are 405 hundred, ok?12:23So let's go back to the controller here and the now a static threshold is a very, it's a reactive alert ultimately cross this threshold fire this alert. Taking in mind what we're talking what we're.12:41Trying to achieve with the talk track here.13:02We want to try to make things a little bit more proactive rather than reactive. I think what's always in our best interest if we can do that. And baselines are kind of a step in that direction. They're not the final step.13:05Towards becoming completely proactive. I want to be clear about that. They might be one step around along your crawl walk walk run journey to get there, right? So there's, there's other things we can do besides just health rules static.13:22Dynamic thresholds, things like that to help get more, more in a proactive phase. But I want to call out if I, if I go over the last one month in the tool, you can see all of my.13:42All of my numeric values now are mapped over time, and my average response in this app is 111 milliseconds. Remember my health rule had 405 hundred. Let's double click in there and see if those health rules make sense.13:58That we had that we made in our 1st one within this app. Remember AD air had a little bit different performance. If I can, if I map this over the last one month, you can see the highest value I have that I've really captured.14:15And a mapping at this, this metric zoom level, right? Cause we're crunching a month worth some metrics into a little screen trying to draw this line really nicely. There could be spikes in there if we look closer, but it's pretty safe to say.14:35If we're, if we're ab but you know perhaps this crossing of 1705 was an issue. So we want to know that as a warning. So remember we used 405 hundred. I just wanna call out we're probably we could lower that, but let's talk about baselining.14:53The dotted line here represents the baseline, and you can impact what type of baselining, you know, monthly trend, daily trend, weekly trend, which one is applied to your data here. And I just, I encourage you if you're, if this setting, this is.15:13There's always gonna generally be if you have never logged in or have never done this.15:32It's gonna, it's gonna be set to this. Don't compare against baseline. So I encourage you to set it against the baseline, right? That's kind of the secret goods that the automatic out of box stuff that appd is pretty good at giving you. So we're let's just set a daily trend. Just know you might have multiple options.15:39In here and now I get a dotted line because my baseline.15:59Checkbox down here is checked. I want to call that out. Once I turned on baseline, there's also this other checkbox called shade. And if I go up here baseline shading and I check these boxes under configure, when I turn this.16:03Band on. That tells me.16:23If I want to convert this from a static value to deviation from normal, which you'll hear a lot about when we talk appd, that's where we're gonna see where would I have to fall into to be in one of those bands.16:27Right, and so I just want to show you it's kind of an easy button. It can be a lot harder to figure out what number is for deviate if you're not a math expert, right? This makes it easy for you to see well visually what values do I need to fall into?16:45So I want to call that out. If you're not using shading, great feature to tell you, well, where do we need to go to look for what value is appropriate to implement for standard deviation. So with this in mind, remember we had a rule that.17:02That's 400 milliseconds for warning.17:21500 milliseconds per critical. Let's go back. I'm gonna make a copy of this health rule, and we are gonna call it V two. Just so we're gonna build the V two app. I've called that out because if you have a good framework of a health.17:24Health rule to start with and you want to make adjustments and you're not sure if those adjustments are gonna work.17:44I recommend you make a copy and, and, and separate it from your working health rule. And again, when I go to my policy, I just wanna call this out as well. My policy is very specific to the work, two working health rules that I have.17:51So now my copy that I'm working on will not go through my policy and it will not give somebody a false positive. I wanna call that out as I think very important as you're working on health rules.18:09I'm gonna go into the V two application and right away, we know we can go down to 200. And so I'm gonna change and and that's based on what we saw here, right? We're using, I don't know that. I'm I'm using the tool.18:27To tell me what the right number is and and then you know applying what I know about the app to to to determine, does that, does that make logical sense to do this?18:47So now we have a little bit tighter static values, but I want to go and I want to make this a little bit more proactive by using baseline. And so I'm gonna add a condition. I'm gonna label the condition what we're trying to do. I'm gonna select the metric of.19:02Of average response time, and we're gonna say it's greater than three standard.19:22Deviations. And I'm just gonna abbreviate for the sake of this. So single metric greater than baseline and I I like to use default baseline here. You can pick a baseline.19:33Line below and the but default means whatever you use in the app is the default baseline, the health rule will use. Just know you can, you can pick a different baseline.19:50For a longer you know as as you've had an app instrumented longer, that monthly trend is gonna be more perhaps more valuable as you've instrumented it year over year, right? So default baseline is what we're gonna use here, three deviations from normal.20:03Again, what do if there was NO data, what, what will we want that to do? Well, I'm gonna show you a little real world example of when this becomes when what how would I visually know.20:20What what I would wanna select here. And I'm gonna show you a use case after this that kind of covers that. So again, warning criteria, I'm going to add, and here we're gonna use two deviations to stagger it.20:35Of course I'm gonna test this in the tool, right? I'm not just gonna send it in a policy to the person who might already be getting these and have them get too many of these, right? I want to now save this, and of course I'm gonna use the tool in the future now.20:53How to come back in here and over time see, well, is it too noisy for me to implement in production?21:12Do I eh, do I So I'm gonna use the view here eventually to tell me, yeah, it's fired 30 times this week. That's that I'm not comfortable that 30 alerts every week going forward for this one health rule is gonna be sustainable. If.21:19Especially if that's not truly a problem. If it's not truly a problem, we wanna tune it and get get it so it's not generating a lot, a lot of alerts. And then we're gonna go back to the policy.21:39Now that we've tested it, ok, we know we validated, we've done our due diligence to determine it's not too noisy to implement into production. Now I'm gonna add this explicit health rule into my policy, and that's the final step to to to lighten.21:51Make it production connected, ok? And then I'd probably remove my V one so that I don't have two separate alerts telling me two different things. So there's kind of a cleanup that you'll want to do as you evolve your health rules as well, ok?22:11So we're not gonna add this to this, but I illustrate this as a workflow because I do think it's important to be very thoughtful about what step and how, what step you're doing with health rules and and will this send an alert to the person who's configured in my policy?22:30Today.22:50And I don't want them to have a poor experience as I'm dialing in health rules. So with that being said, let's go back over, you know, that that's kind of implementing baselines. I want to call out, we, you know.22:51Because we talked agenda on the agenda static values and baselines. Before I move on, I like to couple conditions. A lot of times.23:11I use, I like to have two qualifiers. One generally is the main metric I care about, which is average response time. But I may, I might like to also have like a call per minute qualifier cause do I care?23:22If my, if I only get one call during an off peak time in the middle of the night that's above 200, maybe not is important as when I'm getting 50 calls a day during my peak time, right? That's where we could impact.23:41With s that's where additional static qualifiers, you know, a load qualifier, a load check that we're getting a minimum load requirement as a qualifier is a great additional thing to help make you more proactive.23:58So I'm gonna show that a little more in practice with these other health rules. Let's see you wanna use a metric expression rather than we know the tool can do standard deviation. We know we can do single static values.24:18Okay, I'm gonna show you because obviously if we go back to the main view, we, we show you percentage based things, but the metrics themselves, you can see our numeric values. They're not stored in a ratio.24:36So I'm gonna go back. How does the tool do that? In this case, application error ratio health rule. Again, we're gonna use always 3030, affected entities. We're just gonna target high level application. I'm gonna show you how would I do a percentage based alert?24:53And I want to show an example like this with the idea of how do I go from proactive to reactive? What do I mean or reactive to proactive? Pardon me. What do I mean by that? Maybe I want to implement some, some high level SLA standards or at least as a.25:13As a command center, I want to offer as as a service or a menu item, the ability to report.25:33You, you as an app team can tell me what is the error percentage you want to for sure be notified when you go above. In this case, we have a really high error rate. Remember, I would have tuned that with my app team to remove errors that do not matter.25:43But regardless, it's always at 44 %. I want to call out, it's a very it's a very stable number that doesn't change by 30 points in a day, right? The lower your error rate, the better based on error detection.26:03Action.26:22Filtering, get rid of the things that aren't errors so we can have a low as low of a percentage as possible. But let's say we want to alert on a flat SLA percentage. I'm gonna go back to the health rule and instead of metrics single metric, we're gonna select metric expression.26:23If you're not doing this in your environment today, it's pretty powerful. We can combine the metrics that we store into custom expressions. I'm gonna edit and go in and show you that I've added two variables.26:43One is tied to the errors per minute metric, and one is tied to the calls per minute metric. I've labeled the errors per minute metric as EPM. I've labeled the calls per minute metric as CPM.27:01And you can then see when you look at the bottom, I've I've generated a metric expression. I'm taking errors per minute divided by calls per minute, ensuring order of operations complete on that and.27:19Multiplying it by a hundred to change it into a percentage based alert. And this is where I could say if the error rates over 90 then obviously this is not a real world example in terms of the number that we're using, but the framework of the health rule.27:35Can be used in production with a good number. Keep be clear here. We're using 95 % of course we could lower that to 50 % e.g.. And here's that in a, in a, on a lower staggered value.27:52I like if I'm for SLA style alerts, I really like, if I'm gonna tell a developer who doesn't know the tool to go look at something, I want the number that they're gonna get alerted on to be a number that's easy to find in the tool. Error percent is right.28:09Here, and it would be very clear.28:29If this spike, let's say I I've done my due diligence, I'm tuned down to 2 % error rate and today it went up to 10 %, they're gonna see that in like one click when they get into the tool. So they don't have to dig deep to see, ok, yeah, we have a dramatic change in the.28:32Way our application's performing today. So again, let's say I want to do that with slow or very slow. Let's go back over and I'm gonna show you this in another health rule. We're gonna call this performance degraded.28:52And this, we're doing something very similar. I have much more conditions in here. There's five conditions total, and this is how we can get really smart. I've got an average response time static qualifier.29:12A check that we're getting a minimum load requirement and it has to be true. I have a calls per minute, e.g., and you know, I could say is greater, we're getting at least five calls per minute and the average response time is above.29:2950, right? Like the the so we can make more intelligent choices and then one of these three also has to be true. And here's the percentage of slow, very slow install. And why did I build this use case out? I I want you, I want to show you a real.29:49World example of why you might change this radio in a health rule.30:09I, if I am making a metric expression based health rule for a person, I will always build a companion dashboard to check my work. And so I'm gonna pivot to a final screen here where I have mapped and I've done this work in advance and.30:17I illustrate this for the way that you guys can think, how can I make this proactive? I can auto deploy this dashboard across every one of my ten critical apps. It's gonna auto populate pretty simple, and now I know the SLA values to put in all ten apps.30:37In my new SLA health rules to help me get proactive, you know, little bit more proactive. And they can intelligently pick those numbers ideally based on ideally the app teams telling me yes, that number makes sense based on this dashboard. So what I've done.30:56In here to make sure my health rule calculations are right, I go in and I I map the same expression. And so now I can't see this, I can't see slow transactions mapped over time or very slow transactions as easy.31:16Unless I do something like this and this allows me to ensure that my my metric expression is returning the value that I expect in a health rule for it to fire. I build this to help me populate this.31:38And again, if they're flapping, this is where I can say it's gotta be above, you know, this percentage for half of the, for 15 data points out of the last 30 min. So that'd be half of the data points. These are ways I can make it you know.31:56More of a sustained failure, not only that, the user who responds that might might be accountable for the app, they can easily see that these numbers have spiked. You can see they they they remain very tight for this app. They don't deviate.32:14Which tells me when there is a problem, they are going to.32:34With that being said then, I just want to call out. Remember I talked about one last thing. It can be hard to know when I need to use this evaluation for NO data scenarios. It may not always be.32:42Apparent to you examples where this could be used as let's say we have a, a highly ephemeral applic, you know, an application that has many pods that are spinning up and spinning down, we're scaling, auto scaling often.33:00So maybe when there a pod goes down, it's not a bad thing and it's expected. I want to show you, here's what this looks like, here's my visual queue. I'm I'm telling the health rule, what do I want to happen for this specific scenario? And what do I mean?33:17Okay, if I took stalls divided by calls per minute, stalls being zero, I can't finish the mathematical equation and so it fails. That's where I'm telling it.33:37What do I want it to do? Do I want the health rule to say it's critical during that time? Do I want it to say it's unknown or do I want it to be healthy? In this case, I want it to be unknown because that what that represents to me.33:54Is that the dashboard that I've created for validation can't do the calc either. The number it's not a real number, but if I go back to eight days, I can see when the calc did work, so I know the calc works, and so that's where I would probably.34:09I would want it, I would want to know the health rules in an unknown state if it couldn't calculate stall.34:30That's where I would control or if it couldn't calculate, maybe I want that healthy, right? Cause it's ok if we're, if that zero is in the top spot, make it healthy, right? That's where I can make that choice to help me.34:37You know, it's all about helping me make intelligent choices so that I'm not responding to alerts that are a false positive. So with that being said, that is really, what I had from.34:57A demo standpoint I'm gonna pass it back over to Chris and I am happy to answer any questions. Let me know. Great job Drew, and I honestly wish that I I had.35:13I wish I went to this session back when I was a customer, it would have really helped. No, the experience that you have in your explanations just really helps a lot and I hope I think we got a lot of value out of that. But this is the time if you guys have been holding back your questions and wanting to.35:33Bring that up now, go ahead and throw your questions either into the Q and A or in the chat.35:53I do see there is a question, where can we find the recording from the 1st session? Oh, so that is actually in the chat. Let me just go ahead and repost all of those links that I provided. Maybe you joined a little bit later, and we can.36:00Make sure that you have everything see if it's gonna be two, and then were there any other questions?36:20Just put that in line there's it.36:38My Q and A seems to be messed up, ok, so to answer the, oh, there we go.36:58There it is. So I answered in line. If you ask the question, please go ahead and, it should be in there. That's the link to the 1st one. And then I also am gonna just repost all of these links back into the chat. It makes it a little easier to grab them all.37:09And then also I see, you know, Eugene Breen, who is our panelist on here, he is also an expert when it comes to these things, former customer, awesome engineer. He wanted to reinforce that using the dashboard, especially when you're dealing with, with metric expression.37:30Was dealing with a dashboard that checked to make sure that there's a metric showing up in the place that you're thinking is going to be a really good best practice and make sure that you're.37:50You know, what you're putting in the health rule actually has value and has a metric there. So I just wanted to point that out and I'm not seeing any other questions in the Q and A or any other mentions, discussions or loud, loud out bursts in the chats.38:02So we can go ahead and start going. But at the same time folks, if there's any questions along the way before if even if we're talking, you can still bring up any sort of question. Oh, I do see another one. Okay, so the customers provide performance baseline.38:22So, usually customers will provide performance baseline and when it, when it is time to change or recommend it to change it. Okay, so I'm if I'm reading this correctly, when a person is, when a customer is using the baseline, when would it be a.38:41Good time or recommend it to change it and then how would you set it or recommend it? So I think you kind of touched on this Drew, but.39:01Yeah, great, great question, Mike. I, I to to to answer it, I, I used so a couple things. 1st off, I would have a couple questions couple couple things I'd look for. Is it firing too much?39:10Okay, I would like a decision tree that I might follow. I'm gonna look at I'm gonna say, ok, does this health rule fired too much? And what do I mean by too much? And if I'm tuning for a customer for production or I was a consultant, you know, with my s put back on my consul.39:30Faulting hat.39:50I don't want them to get more than two or three alerts, two or three, four alerts a day, right? And they, and they, and when I say an alert, though, that's even tough to manage four outages in a day. Alert to me is action.39:51Outage. True action needs to be taken. So how, how would I decision tree determine do I need to make a recommendation and how might I go about that, right? That's my thought process here. 1st off I'm gonna look at how many times has the health rule fired over a seven.40:12Day period. And if that number is more than a few times a day.40:31Already it's on my radar as an opportunity for improvement. 1st thing I'm gonna do is go to that person and I'm gonna say, hey, I noticed this fired 15 times this week. Can you tell me a little bit more about the 15 outages you had?40:36You know, I might use language like that or what was the, what was Did we have 15 actual outages or or performance customer impacting events and if they tell me NO, ok great what you know just so I know, what are you doing with those alerts today? Do we have an opportunity.40:52Opportunity to reduce that noise if you're not responding to them, right? And, and, and they will know, hey, I've got we we.41:12My, my users or even me as a person who might be consuming alerts, I'm gonna know which ones might be firing that haven't been a problem in the past so that I might, if I've never told somebody to fix it.41:24The, the, the usual behavior is that they will just kind of like ignore that one or have a rule to filter it into another email box and they'll look at that when there's a problem to determine did it fire then? Oh yeah it did. So 1st off, how many times is it firing? If the answer's two.41:41Too many, that's a caveat or that immediately flags it for me that we have an opportunity for enhancement. But also if it has never fired.42:01Then I'm gonna look at it and I'm gonna question, well, did like my real world example where we reduce the threshold from 500 to 200, did somebody just pick a value that's it we would need an absolutely catastrophic failure to even hit?42:12Or is it even in line with reality for their application? And how would I determine that?42:32Okay, I'm gonna I I don't necessarily need to go I'm gonna 1st if if either opportunity for enhancement or I'm concerned we may not have a legitimate value so that it will ever fire. 1st thing I'm gonna do is go into the tool in the metric.42:40Browser and I'm gonna find that exact metric that we're keying on and I myself am gonna look at that data over time. We can't, we've we have the data for a year.43:00Down to every numeric value that we collect for you. So then the discussion is, is I'm always at a high level gonna map the metric over time. I kind of demoed that workflow for you guys today. I want to know, well, if they have 3000.43:13Was in.43:33In this response time for the last, I'm never gonna hit that in a health rule. That health rule, while it would tell me something if it, in reality I'm never gonna hit that number. And this just told me that simply by look, so the a lot of the answers.43:34I get without going to them my 1st look at the metric over time and and ask myself some questions, do the thresholds in my health rule?43:54Makes sense. For the numbers that I'm seeing that appd is actually collecting as as my source of truth for the performance. And if the answer is NO, then I'm making intelligent choices by the numbers I see here.44:05And, and that's where I'd start to make recommendations that, hey, I think we could make this more effective either by reducing the noise or or tightening the numbers so that it, it could fire if there was a problem.44:23Usually it's gonna be either too noisy. One of two scenarios is probably is gonna cover almost 95 % of these use cases I think in reality is probably gonna be either the numbers are too low.44:42And they've started to ignore it, meaning it's devalued. It's, it's devalued the information it's even telling them to the point where they're not responding, and that's an opportunity or it's, it's got thresholds that literally could never make it fire. And that was purposely.44:59Done probably because of the 1st scenario being in place for too long.45:19Meaning, for a year I got too many meaningless alerts, so now I've I went, if I, if I could finally get somebody to change it for me, I I change it to a number so high it won't ever fire. Does that help a little bit?45:27And to be clear, there Chris I don't know if there's mechanisms like call a consultant type stuff for them to ask more, right? Or NO, there's there's also, you know.45:44Comments and and things you can see you can put on the YouTube or on the community page for Cisco. And I've I have all those links put out there. The, the question did seem to be answered the the.46:00The person that had the question said that you you answered it and and it's very relevant to their support. So they're, they're happy to get that answer. If there's any other questions, by all means, please bring them in and then.46:16If there is a call consultant flow where if you guys wanted to put in a request for, a, you know, a a Cisco customer success engineer like ourselves, you're welcome to, to request some time with us, you know.46:32Separate from a greater audience, Eugene, if you don't mind putting that link in the chat for us. Oh, you're awesome. Thank you sir. So yeah, if you guys want, go ahead and bookmark that. That is a good resource for you all.46:50If you're having any questions getting stuck and you don't find any good videos on our YouTube or or anything like that where you can find it yourself and just need a little extra nudge, you can either contact your account team.47:10And and say, hey, you know, we want to sit with a customer success engineer or go ahead and use that form that you need just sent out and it'll, it'll prompt us to get in touch with you. Alright, so I don't see any other questions, but like I said, my I'm not.47:25I've taken my offer back. If as we're progressing, if we're still talking and we're still on this call, feel free, you know, if the question rattles out or made you think of it now, go ahead and put it in the chat or in the Q and A and we will stop and kind of go back and answer that question. And then.47:45Real quick before I go further, I'm also going to on top of what Eugene sent, thank you for the feedback, really appreciate it. And then I'm also going to just put this back out there again, all the links and resources for you all.48:04All right, so back to the slide here, now in addition to your enablement with customer success and just, you know, trying to build and adopt adoption within your organization, within your company, or, you know, if you're just an IT professional trying to better your skills and knowledge and, maybe.48:20You want to start going towards certifications, maybe you want things that you can call out on your resume to start building you up as a an observability professional. Cisco University is a great resource for that. There's instructor led courses, live live demos in addition to labs, and you know, you would be able to.48:40Get your hands on it and you know have that instructor and labs to follow, you know, getting that hands on practice, getting acclimated and then certification is going to definitely help you in a professional level, whether you're already a customer or just an IT professional, you know.49:00Seek Seeking advancement outside of what you normally do.49:21You.cisco.com is an awesome resource, not even just for observability either, you know, there's a lot of appdynamics things on here but also you know Splunk is into, I mean Cisco is into a lot more than than just observability. There's AI, data centers, networking, security.49:26All the major buzzwords of today that you want to get savvy on, definitely check out Cisco University for those, and let's go ahead and go to the next slide.49:45And then if you've enjoyed this video, you want to see the past recordings in addition to, you know, just the last session that we had on this one topic, they're all out on our YouTube page, which is youTube.com at or/at Cisco observability. We have a lot of short videos like.49:56Long videos, recorded long sessions like this.50:16And we try to have, you know, meaningful playlists out there as well. So like subscribes, support us, where you can. We enjoy making these videos to you know help and benefit you all. So let us know how we can make them better. If there's any ideas or desired content, let us know.50:20And then that follows into another.50:40Another way of providing feedback. We have a Slido link here if you want to do, give us feedback anonymously, you can do so, you know, by either using your phone and using that QR code or grabbing the the link there, the link and password, which I also.50:45Also paste it into the chat so you don't have to stare and try to grab it with a screenshot or anything, but feel free if that's the way that you want to go. And we really appreciate everyone coming out and helping us make these a success and we will see you all next time so take care.51:04And then I do see there is one more question for me. I don't have that in front of me. They're asking what specific app dynamic certifications are available. I don't have that available if we could come up with a link and put it out in the chat.51:24Any of. We'll get you a link and it'll give you kind of a certificate or certification explanation of what you can get on there hopefully, so.51:45Just bear with us. Sorry for the the long silence there all we're we're just trying to get the link. We had one last question where they were asking what certifications are available for app dynamics specifically on the Cisco university. So we just wanted to try to get that.52:01For you.52:21I'm gonna send two links here. Here's an example of one that's called AppDynamics Certified Implementation Professional and the why it took me so long is I'm gonna send you another link, these, these.52:26These are still the right names to give you it. But the posting is from back in 2001. That's why I was paused paused for a little bit, but I have posted.52:42The link that kind of describes the technical badges that are available and that 1st link. Second link is what's, what's kind of available sit. The the 1st link that I posted is direct information on one of the three.52:53That are referenced in 2nd link. Alright, thanks a lot Drew, and hopefully that helps, but I don't want to keep I know that we were scheduled for an hour and a half, but we really try to.53:09You know, not take up a ton of your your time in the day so please grab all the re the resources and the links from the chat. Let us know if you have any other questions and thanks everybody for joining. Take care everybody. Have a good, good rest of your week.53:28