Setting up Baselines, Errors and AI

Think not a critical. If this is expected behavior of your application, this is great. And this is something that would be really hard to judge if you're using static threshold.00:10And remember this, we didn't AppDynamics will learn this automatically and we didn't have to configure any of these ranges. The next thing I want to talk about is configuring your default baselines and talk about kind of some of the interesting facts of what our baselines use.00:20So one thing, you can, one thing you can do here is actually create custom baselines or edit the ones that are default, and you can name them whatever you want, which will be useful when I bring up a scenario later, and the next thing, the key decision here when you're.00:40Creating these is which trend to use?00:59So you can configure a daily trend, a weekly trend, a monthly trend or use NO trends, and each one will tell you how AppDynamics decides what normal looks like. So for daily it uses the last 30 days of data, and it compares the same time of day across 30 days.01:02So 02:00 P.M. today is compared to 02:00 P.M. yesterday and then the day before, and this is usually the default baseline. Then we use the weekly, and this uses the last 90 days and it compares the same hour on the same day of the week over the last 90 days.01:21So Tuesday at 02:00 P.M. is compared to every other Tuesday at 02:00 p.m.. And finally monthly which uses the last year, and it compares the same hour on the same calendar day of the month for the last 365 days. So 5 January at 1030 is compared to 5 December at 1030.01:38Now, what does that actually mean in real life? Again, this is a very important thing to understand because one, it's a great way to set your default baseline for your application. And two, this really helps you create good health rules that actually match what your application needs.01:59So for our daily trend, you want to best use this when your traffic is fairly consistent day to day. If your Monday looks like a Tuesday and looks like a Wednesday, this gives you the tightest, most responsive baseline because it has 30 data points at each hour.02:14And this is good for industries such as internal enterprise applications such as HR portals, health core healthcare scheduling systems, manufacturing floor monitoring, and anywhere where usage is driven by business hours and doesn't vary much between weekdays.02:30And all these systems what predictable repeatable daily patterns. Now the weekly trend is best used when the weekdays differ from weekends. If your B to B platform gets heavy traffic Monday through Friday.02:49And almost nothing on Saturday, a daily baseline would average those together and a weekly trend keeps Wednesday separated from Sunday. So each day of the week has its own normal, and this is why it's the default for most teams. This is great for industries such as e commerce, SaaS platforms, financial.03:03Services food delivery apps where there might be a Sunday or Friday a rush or media and streaming where you'll see weekend bench patterns. And this matters most for most real world applications, as most of them have a weekday and weekend split.03:24Then finally, we have our monthly trend, and this is best for seasonal or billing cycle driven applications. If you're in retail and your traffic spikes on the 1st and the 15th of every month, aka payday shopping, a monthly trend captures those patterns without flagging them as anomalies.03:42And again, this is great for industries such as retail, insurance, taxes, payroll or HR systems. And this is because these businesses have a strong calendar driven pattern that repeats monthly, now I recommend for most teams that they use the weekly 90 day trend.04:01As a good starting point, as it handles that normal weekday weekend split that most applications have.04:21And then you'd only have to switch to daily if your traffic is truly uniform or monthly if you have a strong seasonal pattern. Now the final thing I want to talk about here in the configuration is a time period setting, as you see here, we have two options which are dynamic and fixed.04:27So dynamic uses the auto learn baseline and AppDynamics will continuously recalculate what's normal based on historical data using the trend that you configured. The counter to that is the fixed. So this is a snapshot, a snapshot and time baseline, so you pick a specific time window.04:45Such as last Tuesday 09:00 A.M. to 05:00 P.M. and freeze it. And this is useful for creating named baselines like weekday peak, weekend or holiday season/black friday. I think the big one here would be creating a black Friday one, so you can name it black Friday.05:05And then use the fixed and then select the last year's black Friday. I know Black Friday's almost a month now, but you can select your expected be expected period for that holiday.05:24And then once the next holiday come or your next black Friday comes up, you can use that last last black Friday to give to create that baseline so that you're not seeing spikes or getting false alerts during the holiday period. Now the next thing I want to talk about is moving from.05:40Baselines and health rules as they are a great start to proactive monitoring, but now I want to talk more about error configurations. And the 1st thing you want to know is how to understand errors and exceptions. So we'll go to troubleshoot here.06:00And then click on errors. And you see here we have two separate tabs for errors and exceptions. In the errors tab, these are business transactions that AppDynamics has flagged as errors. Each of these is a transaction snapshot that we can dive into deeper.06:15And each of these represent a user facing failure, a checkout that didn't complete or a search that returned to 500 an API or an API that timed out. The next is our exceptions tab.06:34Now usually this is an extremely full list, and if I tried to fix every one of these, I'd never sleep, and some of these don't even need fixing. But if you look at the error transactions 1st, I can actually see which exceptions are causing the business to lose money.06:48And just as a kind of an ah moment, an exception is any throwned error in the code. Some are handled gracefully, some are transient, and some are expected. Meanwhile an error transaction is the business transaction that AppDynamics has decided that represents an actual field.07:07Earlier, the error transaction tells you what matters and the exceptions tab tells you what happens under the hood. So always start with your error transactions as they give you the real indicator of how your business is doing. The next thing I want to do is go into my configuration.07:26And instrumentation and then error detection. And then you do see here at the top that you have different options for different languages. I'll spend my time here in Java since this is a Java application. Now you want to make sure your errors are configured correctly, and to do so, we have our custom configuration.07:44Because out of the box, AppDynamics catches a lot.08:04But doesn't catch everything and it might catch too much depending on your application. So I'll walk through some step by step use cases for this error configuration. Now out of the box, app dynamics captures HTTP error codes 400 to five oh five.08:08And it also catches all thrown exceptions, which is a good starting point, but notice the defaults range ends at five oh five, and if your app uses higher 500 codes like five oh seven or 522, you need to add those manually. And to do so, you'll create a description here in the HTTP error code.08:24And then you'll use a lower bounce, so in this case we're only doing five oh five to five five oh seven to five oh seven, but you could do it five oh seven to five oh nine and capture five oh seven five oh eight or and five oh nine. And just to show you what this looks like for the rest of them, everything.08:46For all of the configurations that I'm showing you, you do want to configure it to specific business transactions, whether it's all of them or the specific or very specific ones that need that specific rule, you'll decide that, and I'll talk through some use cases of why you'd want to do one way or the other.09:04And then once you're done configuring, don't forget to hit the save error configuration. You can actually find it at the top and bottom, just for convenience. A lot of the time people will spend, you know, 30 to 40 min creating error configurations, great, they don't hit save, they walk away.09:21And they lose all that progress, so don't be that person. The next thing I want to talk about is ignoring HTTP error codes. A popular use case I see is a lot of customers will have four oh four errors inflating their error rate significantly. But if you think about it in an e commerce.09:38Application, four oh fours happen all the time.09:58Users will mistype URLs, web crawlers will hit dead links, and old bookmark points will point to remove products. That means a four oh four isn't a bug. It's really a normal behavior in our case. So to do it, you'll do the same thing we did here. We'll enter a description lower and upper.10:00The trick here is to hit enable and then hit it again to disable it, and this will make it so that we ignore four oh fours and won't log them as errors. Now another use case will go into our ignored exceptions, and I'll just click on edit for one of them.10:19Just to show you what it looks like. So we saw some exceptions in the tab when we were looking at the troubleshoot, such as NO point exception or out of memory errors, and these are valid ones that we want to track.10:38But before I was seeing connection pool exhaust exception and socket timeout, and since for this application, we consider that just noise, and every time we get traffic, we see a massive spike in them, but to us it doesn't actually affect the end user.10:52We ignored these so that they don't show up in the exception list and create a lot of false positives for alerting. Now, the important thing to understand here is that when we do this, it will still show up in the exceptions tab.11:10But you won't, it just won't show up in the air as an actual business transaction error, so we didn't lose visibility, but they're NO longer flagged as error transactions, and we separate the noise the app handles from the problem that hurts users because of it.11:26Now, another scenario that we see customers deal with constantly is having a vendor do maintenance, and when that happens, half the downstreams calls fail with connection refused. Now the error rate goes through the roof, and it's not our code's problem, our R code's fault.11:45So we can't, we don't want to ping our developers every time it happens depending on the use case. So what we can do is implement message based filtering. So you see here that we're looking for any logged message that shows connection refused.12:04And this message based filtering is surgical. We're not just ignoring the entire exception class. We're just focusing on the ones with this specific message. Now I'm gonna walk through a more tricky situation.12:21You might have dealt with this before, but AppDynamics shows everything that shows that everything is green. There's a 0 % error rate and our dashboards are clean, but 30 % of our orders are silently failing. And what's happening is the app HTTP code is 200.12:39But in the Json body, we're getting a business error equals true. The payment didn't actually process and the order didn't actually go through, but the HTV status is still 200. So AppDynamics correctly doesn't flag it as an error, at least in its mind.12:56And in production, this could mean 30 % of your revenue is silently disappearing and NO one gets paged. Now there are two solutions to this, and the 1st fix is really the long term fix for it and the right way to do it.13:13But you might not be able to, and this is making the except application actually throw an exception when the business error occurs so the agent can detect it. But the problem is you might not be able to change the application's code right away. Maybe it's a 3rd party service or the release cycle is six weeks.13:28Out instead, AppDynamics will let you detect errors purely through configuration with NO code change.13:48Now our app has a business error logger with a class called log error, as you see here that gets called out on every business error, and it was already there for audit purposes, so we can tell AppDynamics to watch this method.13:55So you see here that we're grabbing our class name, our method name, and then we are grabbing the exception parameter here with our method parameters. Now once this is applied, we should see our error rate climbed 30 % up for those missed calls that we saw.14:12In this way we didn't have to change a single line of code. We told AppDynamics when this method gets called during this business transaction, that transaction is an error, and the 1st parameter becomes an error details in the AppDynamics error dashboard.14:30Just to quickly show you what that looks like, if we go to our exceptions list here, we can see our custom error message, and we can see our custom logger here and that the business logic error or order validation failed. And before our AppDynamics wouldn't have caught this.14:45And we wouldn't have been able to see what's going on and going back to our error detection, the next thing we want to talk about is our payment failure payment failure logger. Now we just configured a general purpose business error logger, but in a real organization, different teams need different visibility.15:04Ability and our payment team wants their own error category, they need to see declined cards, insufficient funds, and frauds hold separately from general order validation failures. And they already just have a dedicated payment failure logger.15:24Class that gets called when a payment failure payment fails. So same thing we define our class method and method parameters. Now once we save this, we have two custom logger definitions that work side by side.15:40So we have our business error logger and our payment failure logger. Our business error will catch all business errors such as order validations or inventory mismatches, and our payment will catch payment specific failures such as declined cards or insufficient funds.15:55And if we looked at our error dashboard, you'll see errors from both loggers. The payment will include transaction IDs and dollar amounts right in the error details if we want. And the payment team can filter on their logger and see exactly which transaction failed or for how much by how much.16:11Without sifting through unrelated order validation errors. Now the key talking point on this is that in production, different teams have owned different services and custom logger definitions let each team define what counts as an error.16:29For their own domain without changing each other's configuration, and you can layer this as much as you need. And the pro tip here is you can also ignore specific loggers. So we see here our request audit loggers being ignored, and this is because it fires on every request.16:46And it proactively add, so proactively adding them to ignored loggers prevents them from accidentally being picked up as a custom logger definition, which would mark every single BT as an error. Now another one, and this one's gonna be specific.17:05Is there's another blind spot some enterprise applications especially older ones with server style rendering have, which is redirecting users to an error page instead of returning an HTTP 500. And from app dynamics perspective, the response is a three oh two redirect to a 200 page.17:22Which looks like a success from our end.17:43Now to fix this, we'll go to our redirect page, and we'll look for any patterns that match error/page and enable it. Now, once you configure this, those missing errors will reappear in the error page and count towards real errors to your total count.17:46In this case, any redirect to a URL containing error flagged page is flagged as an error, and this is incredibly common in spring Boot applications.net or PHP apps.18:05And now we've covered baselines and error detections, so I want to finish this off with our AI ops or anomaly detection. So we'll click to alert and respond here on the left and click on anomaly detection.18:22Now baselines will be covered in part one is seasonal normal and measure standard deviations from it. An anomaly detection uses a completely different machine learning algorithm. It correlates the variance of multiple metrics against each other to determine if something is abnormal.18:40Now, for different entity types, we do monitor different things. So for application servers, we focus on CPU utilization and memory utilization. For business transactions or APM, we focus on errors per minute, average response time.19:01And those are correlated against calls per minute. Then for browser, we focused on end user response time. For databases, we focus on the number of connections, the time spent in execution, and the calls per minute. And then finally for mobile, we focus on network request times.19:17HTTP errors and network errors, and you want to notice that each of those are purpose built for entity type. It's not a generic watch everything. It knows what metrics matter most from our experience of learning for each type and how to correlate them.19:36And the cool thing here is it does perfect or does really good noise reduction techniques, including disregarding temporary spikes and periods of NO data. It also normalizes metric data, e.g., using error rate.19:52By dividing it by the calls per minute. And then it also correlates variance across related metrics rather than looking at any single metric isolation. Now in this anomaly page, it's really easy to enable anomaly detection. So all you'll do here is click enable here in the.20:08Top, and then once enabled, it just works, but there is an important detail. It takes 48 h for anomaly detection to be available once you toggle it on. You see here that I have quite a few in training and I'll talk about what that means.20:28So during those 48 h, the machine learning model is training on your application's traffic, and you can check the training status here in the models training. You can see statuses such as in training, ready or not available for each entity.20:46And just to talk about those not available, BTs with very low calls per minute won't get models trained at all, as there's not enough data for a reliable model and configuration is optional beyond that point, but let me show you what you can control when it comes to anomaly detection.21:04So we'll click configure here in the top, and the 1st thing you can select here is whether you want to monitor business transactions and servers. So for business transactions, it's checked by default and you can scope it to specific business transactions or leave it on the default, which is all business transactions in the application.21:23For servers, this is checked by default, but you can scope it to specific servers or leave it on all servers. The next thing I want to talk about is detection sensitivity. This is really the key one that I would change depending on your use case.21:44For high, this is when early detection is critical, and you're willing to accept more false positives because of it. The next one is medium, this is a balanced and it's a good starting point for most applications. And then finally we have low, and this is minimizing noise and only getting alert.22:01On high confidence anomaly detections.22:21We also have severity level, and you can select which select severity levels generate alerts, whether you select all severities, critical or warning. And then finally, oh sorry, before we go into that, most teams for the detection sensitivity should.22:24Start at medium and then just adjust from there.22:44If you're getting too many anomaly alerts, that turn out to be nothing, I would drop it to low. And if you're missing real incidents bump it to high. And the final setting here is a test mode checkbox, and are the, you wanna ask yourself, are the monitored entities in a non production environment?22:47The regular machine learning model is designed for production traffic volumes, and if you're in a staging or dev environment with low traffic, you want to turn this on, and it uses a model optimized for low traffic environments, so you still get accurate anomaly alerts despite lower request volumes.23:06Now anomalies by themselves are informational, but to make them actionable, we'll go to our policies, and when we configure our policies, we'll go into the bottom part where it'll fire on specific events and we'll open our anomaly tab.23:28And you can create policies that trigger on our anomaly detection events, and once you, once those anomaly events occur, you can trigger an action such as sending an email or opening a ServiceNow ticket.23:45And the workflow is simple. Once an anomaly is detected, it will alert the policy, the policy will evaluate and then it will send any action here. Now the last thing I wanna talk about today is when we have a planned maintenance window.24:00And someone on your team suggests turning off anomaly detection, don't do it. If you turn it off, the machine machine model loses its ability to learn and loses its training. Instead we can use action suppression, which we talked about previously on our EIS sessions.24:18So you'll go to alert and respond actions, and then action suppression here at the top. An action suppression keeps anomaly detection running and learning, but suppresses the actions such as paging or emails and ticketing during a defined maintenance window.24:37That means the machine learning model can continue to collect data and re refining its understanding of your metrics, so you and you won't get woken up at 03:00 a.m., just because there's a planned database migration, and that is it for my demo, and I will pass it off to.24:54Yes.25:14All right, good job. Can you get me? Okay, so good job man, that was a really good demo, learned a lot and I do see that there are some questions, so let's go ahead and go over those. So the.25:231st one is.25:43Can you, can you have ignored errors per tier rather than all of the app? So when you're dealing with error detection and you're dealing with ignored errors, can you put them per tier rather than.25:47All of the app. So it's not per tier, but you can do per business transaction and you can set your business transaction scope to be a specific tier. So depending on how you have it configured yes.26:04Although I would say business transaction is more useful since most of the time when you're creating ignored exceptions, you're doing it for specific, class, which would more align with what a business transaction would cover.26:20Please let me know if you have a follow up to that question. And then the other one that I see in the chat is if you turn off anomaly detection, oh, they're asking, they're asking 1st, how do you do that? They're sending it to me.26:39Privately.26:59But yeah, is if you have time, do you wanna go back and show or do we have a documentation link that we can just send them? On how to turn it off for I'm sorry. It's it's back to the 1st question about the ignored error.27:02Per tier or business transaction. Yeah, of course. So for any of these, you can configure the business transaction.27:22And then select business transactions that are using this air configuration. This is almost better than, this is better than tier because it's more granular. Like say that you have multiple business transactions that use that class and method.27:40You know, in one tier, but you want it from one call but not another, the business transaction's better cause it's more granular honestly. So hopefully you get some use out of that. The next question is, if you turn off anomaly detection and later turn it on, will it basically start over.27:57Over on the, the learning, the training.28:17I'm double checking this. I My my immediate guess is it will restart on it, but I need to double check, please. I think when you turn it off it, it loses that.28:27That training and then when you turn it back on it has to go back through it. But there's probably some sort of scrubbing period where it holds it for a couple of weeks or something I would plan for.28:40Yes, I would plan for yes that you have to restart the the learning and training.28:59You do have to restart the learning and training period. I can't remember if it keeps the historical training, but I, my guess, in my re my remembering it is that it does not. So I would just recommend never turning it off personally and use action suppression instead.29:05Or make it less less strict or or tone it down with the the three different levels alright, any other questions?29:25Please put them in either the chat or the Q and A And as we go as we continue on, if, if we, you end up thinking of something, you know, last minute, feel free, still, still throw it in the Q and A or chat and we'll backtrack a little.29:45Little bit.30:05If you're still writing it or still trying to think of how to articulate it or, and it comes up late, we'll, we'll backtrack, ok? So let's go ahead and go back over to the slides.30:06I'm sorry Chris, we can't hear you if you're talking.30:32Mark, can you hear me? Yeah, I can hear you now. Okay, let's go ahead and, so if you guys still have questions, go ahead and please you know put them in the chat or in the Q and A We can go back if that's needed.30:36Otherwise we're gonna continue on. All right, so this is meant for enablement, it's meant for you guys to be successful with the products that you purchased from Cisco. That is what our role is. We are, you know, customer success engineers. We work with customers regularly.30:51So for you to get the benefit of the kind of the patterns that we see, the best practices we run into, that's what we throw together for these enablements, say that you are an IT professional and wanting to get certification and you're wanting to build your resume and kind of progress your, your career a little further and in your.31:08You need that piece of paper or that validation.31:28You know, go over to the Cisco U, there's a lot more than just observability on there. There's AI, there's data centers, there's networking, you know, everything that Cisco is into, which is quite a bit. If you go to u.cisco.com, Cisco university.31:32There's a lot of, you know, already ready to take on demand courses that you can, that you can take as you, you know, at your own pace kind of a thing. And then there's also instructor led courses there. Now those, those instructor led courses and some of the.31:48You know, the on demand courses, they do have some requirements in terms of, you know, subscriptions or depending on, you know, if you're you're a customer already and you have some of those credits towards those things.32:07But if you are already a customer and you're interested in seeing what you have, definitely go on there, check it out, use your company's login, you know, your email, and try to see what you have and then talk to the account team on either your Cisco rep internally or.32:22If you're familiar with who's on the account through the Cisco side, you know, maybe you can arrange to where you can get an instructor led course or start going towards the certification. So go definitely check that out, and let's go ahead and go to the next slide.32:42So beyond that, with the recordings that we do, for these EIS sessions as well, there's a lot of other short videos and, you know, different types of webinars, long and short, different variety of topics.32:59Some things are more follow along demos, some things are just walking you through the concepts. We have a lot of really good content over on YouTube, so go check us out on youTube.com/at Cisco observability, and you'll get to see the content that we create not only for Splunk app dynamics, but also.33:15So for.33:35Splunk ITSI as well for Ali. You know, we're slowly getting some content created for that. Right now we do have primarily a lot of the Splunk app dynamics and you know, there's just a lot of really good stuff on there. So definitely check that out. Comments are opened up.33:36So if you have feedback or you want to tell us, you know, what more you would want to know.33:56In terms of, you know, the, hey, we, we really like that you, you showed this. Can you show this, this scenario or hey we wanna know more about error detection, can you do a deeper dive on that? Whatever the case maybe, whatever the feedback is.34:01Let us know. Largely these expert insight series are driven by your guys's feedback. And so we really value things that you tell us, whether it's approach scenarios or use cases, whatever.34:18The case maybe we want to hear, we wanna make this content better for you, we want you guys to be successful. So chime in and and let us know and it helps you drive your own learning from us. So, let's go ahead and start closing up and you know one last call to action, we do have a.34:33Slido open.34:54So if you would like to let us know how we did today or how to make it better or what you'd want to learn in future sessions, this is a great opportunity for you to do so anonymously, so go ahead and we'll, we'll paste this link into the chat. There's the, you need the link and the password to do so.34:55Or you can use your phone and go over to the.35:16The QR code there, and we will be able to, you know, get that feedback, start applying it towards future sessions. Here's the link, the chat as well.35:20All right, there's the password, and we really appreciate you all joining today. We had a a great turnout and the the next one will be in mid April, so look, look for the invites for that one.35:38As well, check out the link to our YouTube page, which I posted earlier on, and that will go to all of the the past recordings as well. This recording will be on there. So again, thanks so much and have a great rest of your day.35:56Oh, please don't forget. Okay, one moment, let me make sure that those, I see a question saying to please make sure we send out the, the link to the the past one. One moment.36:15May not have gone through on the chat so just grab a bunch of links and post them all to you guys towards the end here. Alright, that 1st link in the the chunk that I just sent out.36:31That is for, the past recordings as well the recordings or this one will be on there.36:51Obviously if you guys registered for this, you will get an email on it, which will give you more links for our community page, where there's also recordings there and explanations of it, and then the, the registration will be posted on underneath the comments of each of those as well. So, again.36:58Thank you all for joining any feedback that you, you have given or will give.37:18And we'll see you all next time. Have a great rest of your week and day.37:25