Achieve Proactive Monitoring with Cisco AppDynamics

Alright, welcome back to another observability expert insight. I'm your host Christopher Banano joined with my teammates, Rob Carlin, who's gonna be.00:09We are expert speaker today and he's gonna be going over our last session of the achieving proactive monitoring series and what this series has accomplished so far, you know, we had Drew Osborne who's also on here as a panelist, and Mark Kirpile who should be joining.00:20Joining us later.00:40Who walked you guys through the 1st four videos or the 1st four sessions where we started breaking down the health rules, the policies, and all of the the custom actions that you can do with that. We went in depth trying to get you guys to understand what you can do.00:41And why you would do it to be more proactive and then turning on the AI functionalities, such as baselining, also anomaly detection, which really sets you up for the, analysis and having kind of that AI assistant within the.01:01Controller which you're going to actually see put to the test and actually see the full functionality and see it at work, see it, see it working today. And Rob is a, has been, you know, with our team for a very long time before that, you know, as a customer and myself, he.01:21He taught me a ton, so I'm excited to have him here before you guys today.01:41We also have Bill Harper on the line as a panelist as well. So if you guys have questions along the way, and we'll get more into that. Actually, that's a good segue. Let's go ahead and get into the housekeeping items. Next slide please.01:45Alright, so we do assume that you guys have some prior knowledge to that you know appdynamics Splunk appdynamics. We don't necessarily expect we're teaching you from the beginning yada yada, you know that this is the last episode of the series, so, bear with me. It's just the part of the flow here.02:05We will expect that everyone is muted and we're going to be taking questions in the chat as well if you have a good or if you have the ability to go down to the bottom right corner of Webex and there's actually a Q and A section there, that Q and A.02:26May section benefits everybody if you put your question there because.02:45Sometimes your question get lots, gets lost in the chat, and you know obviously we want to be able to say hi to you guys, interact with you guys in the chat or for you to let us know, hey, audio's kind of choppy or can you zoom in? Things like that, where the Q and A if you have an actual question about the.02:49Topics, it makes it easier for us to either go back to it at the end or to answer it along the way with our panelists. So, just keep that in mind, but we'll, we'll try to get you guys questions answered either way. And then also, this content is created by Cisco's you know observability.03:09The customer success engineers, you know, it's not meant for you to take it and say that it's yours, it is owned by Cisco, but it's for you guys. It's meant for helping you be successful with the products that you purchased from Cisco. Also if you're dealing with observability and you're trying to be proactive, you'd probably.03:29Trying to get more sleep. So, obviously this is for you and we want you to, you know, have nothing but success with it. And that's the purpose of it. So let's go ahead and without further ado, I will hand off to Rob.03:49Thanks Chris. This is our agenda today. And by the way, welcome everyone. I appreciate it. This session is really gonna focus on kind of two aspects of using AppDynamics for troubleshooting your applications. We're going to go ahead and start off with a, you know, a traditional.04:05Reactive approach where we're navigating inside the controller user interface to be able to identify issues that are emanating and impacting our application. And this could be from the application itself in terms of code issues, this could be from the infrastructure side, is it a, is it applies to.04:25Server or some network, type of traffic that's having an issue.04:45There is also the database side as well. So we're gonna look at all the aspects that could potentially impacting your, your, your application. The other thing we're gonna go through is once we go through the reactor side, we're gonna show you some of our newer features for proactive.04:51Of troubleshooting. And this takes a very AI approach using anomaly detection.05:10To help you identify unusual behavior across your applications. And when I say applications, I'm talking about the anomaly detection features we have now that allow you to identify capabilities and issues on business transactions, infrastructure, meaning servers, databases, as well as browser and mobile applications.05:17As well. So we've got that new feature set for AI directed troubleshooting on all of those entities that I just mentioned. And we'll finally go through a Q and A session, but don't be shy, please reach out and if you want to interrupt me or just excuse me, if you want to send any type of message on the chat.05:37Or anything in Slido, please go ahead and do that. So let's get started. 1st time is I want to introduce kind of a process for you and how we like to go about troubleshooting traditional issues, application issues in app dynamics.05:56We have three types of stages that we like to do in our troubleshooting C act, and know. C is really understanding the problem within the right scope and the time frame. So being able to isolate the time frame in your time picker of where you wanted or you saw the issue and, you know, do.06:13Do any type of root cause drill into the, you know, the red, meaning the the critical issue or if it's a yellow, it's a warning on the dashboard.06:33Leverage the business transaction snapshots that are coming from that particular application to be able to drill down from the business transaction snapshot into the call graph where you can determine the root cause of the application. 2nd part is acting or setting up alerting models and actions.06:43This will eliminate problems before they impact you, so we want to make sure you get to the right, the issue to the right technology team and being able to optimize your health rules and setting up your alerting and your policies that work with those health rules so you can get the right amount of violation from.07:03Information as well as the alerting automatically established and working so you can get proact or you can get identification of your applications in a very timely fashion, and then now is determining the events impact to your organization and being able to correlate.07:23Your application performance with business metrics. So this is the approach. Now, in terms of the way I like to go about, we like to go about using kind of a reactive examples is kind of a five step process. We want to start out with the application dashboard.07:42We want to look at the dashboard in terms of where do we see problems emanating? Are they from the application side, other than the infrastructure side, other than the database side? We want to follow the red. We want to drill down to more specific views.08:00To be able to isolate the issues and then elevate these in abnormal transaction snapshots. So we want to find out the snapshots that are related to the problem that we're seeing on the dashboard. So these snapshots allow us again the business transaction snapshots are the user's experience.08:17With.08:36A particular use case within their, within the application. And from the snapshots, we might have been and link those and save those, maybe archive those and save those and send those to the development team or to the SRE team so they can have a better idea of, you know, where the problem is limited.08:37From and where they need to be able to go ahead and provide remediation. So when we do this.08:57We'd like to take a very deliberate approach and I'm gonna get into the controller and show you, but this is a snapshot of the AppDynamics controller, application dashboard. For those that are familiar and and the majority of you are familiar with the application dashboard, it's broken down into a flow map and then a variety of.09:04The channels over here on the right hand side.09:23With a summary of your load or calls per minute, average response time and errors, and then a variety of tools over here in the panel on the left hand side that lets you drill down deeper and get further information on some of the issues. So I like to take a very muscle memory approach to this.09:25Dashboard and I like to work in a clockwise fashion. If you're a SRE or if you're a, production support engineer, this is specifically for you. If you're a developer, I'm gonna show you a little bit different approach on how you can use this dashboard. But one of the 1st things I want to make sure you're doing when you use this.09:46Just go to the very top and make sure your baseline is on. Then flow, look over here into the flow map. Again, this is showing a summary.10:06Of all the applications or services that are being currently measured or monitored in this particular application we call e commerce here. Look for anything that's red or yellow and drill down into that to figure out whether or not that's giving, gonna give you what type of information. Oops, what type.10:15Getting information that's giving you.10:35The tools over here are the channels on the right hand side is another way and in other words, when I'm working here is I'm trying to show you how to work in a very clockwise fashion. These channels over here provide a summary of all the entities that you have as well as a transaction scorecard, excuse me, that show me, show you an example of.10:37Where the problems are emanating and where I can really focus. So I might want to look at my very slow errors, slope slow behavior, slow behavior, very slow or any of my errors and exceptions from here. I can drill down into those.10:57If I see an unusual percentile or a quantity that's, you know, fairly differentiated from what's normal. Down here at the bottom are the summaries of my three time series graphs for calls per minute, average response time and errors per minute. And then over here.11:11Is my tr or my troubleshooting tools. And when the troubleshooting tools over here on the left hand side, we have three categories. Slow responders, errors, and health rule violations. These are the troubleshooting tools that I like to be able to use when I'm starting to to do an analysis of where my application is.11:30Deviating from normal performance.11:50And how I might be able to drill down and get further evidence of where to be able to look for that problem, ok? So this is where I like to start. Let me give you a little bit of a demo so we can get into it and I'm gonna hover here and show you.11:52Okay, I'm gonna start here with the, with, with the end user monitoring, and I'm gonna use browser and user monitoring example, and whether or not you're using browser user monitoring or mobile user monitoring, the approach is very similar. We're gonna drill down to the issue on the browser side if it's a browser request on the net on the mobile side.12:10Slide.12:30You have network requests coming, but the the UI and the metaphor in how we drill down is is very is very similar. And I'm gonna show you an end to an example of how I can drill down into an issue that's causing a problem with response time, elevation, for that's impacting my end user.12:31Users, how I can use browser snapshots to be able to give me an idea of what the users are experiencing. In other words.12:51The browser snapshots will show me the actual user experience and the, the propagation of what they're experiencing from a, a browser per browser or a, a, a screen to screen perspective and how that's how those services are being used or how that experience is being used with so.13:00Services in the backend. So the whole idea of browser monitoring here is to enable you to measure your application and user experience.13:21By providing application performance visibility from browser all the way to the application server and the database for both real and synthetic user monitoring. I'm gonna start over here with the end user response time trend. Notice the dashboard gives me a lot of very useful information.13:28On how things are performing, what browsers are really, you know, where our problems are coming from a certain browser or device. I can drill down into the end user response time and I can also geographically get a determination of where.13:45My highest latency is on my end user response time pages etc. And I can drill down to each one of these geo areas to determine and provide context on a particular area that I might want to focus on within, within my analysis. I also have browser snapshots over here at the top, and this is really.14:03Key, this provides me with a, a camera or a kind of like a picture of how our users are interacting with our HTML pages.14:23Our Ajax requests, things like that, any type of identification or even all the resource files that are contained within a page or within an Ajax request, e.g.. I can use my filter icon to identify the slow and very slow criteria user experience.14:33And also down here toward the middle of the property sheet is a checkbox for bringing up the server side snapshots. Remember we have the ability in AppDynamics to add a header into the request of all of our captured snapshots, and that gives us the ability of providing end to end visibility with the browser.14:50Snapshots and the business transaction snapshots that are correlated with the actual pages or H or or HTML page or our HX request in the back end. So I can get over here and I can look at, e.g..15:10I want to look at, add to cart. Again, we have a a good example. Let me get and filter on the highest performers here. There I've got one for checkout that's 11 s. Again, you can see all the meta information that's coming over here on the right hand side and I can double click on that and get a variety of information. One is with a.15:25The waterfall graph that actually shows me a variety of metrics.15:45Including the amount of time, the network time, and also the server time as well. Included here is the resource fetch time, how much time it's taking to provide resource files onto the page like cascading style sheets JPEGs, including things like that.15:50But what's really cool here is it's mirroring not only my user's experience from the HTML page in this example with the business transaction snapshot, which is also called checkout on the back end. So I can drill down.16:05Into that snapshot, and I can see, ok, yeah, we have a Firefox browser experience. It's got a 9.9 s call into this service. It's taking that amount of time, and we can see what's occupying the majority of that time is this call to the database over here.16:20That's also reflected in the potential issues on the left hand side. Now this is where we get down to the end to end troubleshooting capabilities of AppDynamics very quickly cause I can drill down into this specific service using a hint here called start.16:39And this allows me to go ahead and drill down from the flow map. In other words, this is an actual user experience. This is the same information that's included in the browser snapshot and the business transaction snapshot together, which is really cool. This is giving me all of the information on how that user is getting serviced from.16:55That, that front end and the services that are being called. In other words, the business transaction snapshot on the back end is, is involving three different services and these three different services are giving up.17:15The checkout transaction of the back end as well as the correlated request that's coming in from the front end browser, right? So I can drill down here from this particular area and then determine where I'm having my problems. And this is the call graph and this shows me the actual.17:27Object. In this case it's a plain old java object that's actually executing a query at line 52. And if I drill down into that, I can see the actual name, class, method, and line number of the code that's my developers really need to look at that's causing this amount of latency.17:48Within my application, ok? So this is a very quick example of of using end to end monitoring and AppDynamics pulls this all together again by correlating all of the header information or adding a header request, what we call our, our header, which is our secret sauce into.18:07Into the request and marries that information with the back end. Now what I also get is a bonus, I get information here on the server side and the network side as well, as well as the database. So I can go down here to the server side.18:28And I can look at the way that snapshot that I just was looking at is, is occupying aware in the time frame and is there any type of latency that's being added on my, from my application server? So this is actually looking at the server.18:42As well as the processes that are consuming CPU and memory within that specific snapshot time frame, and it shows me if there's any elevation in CPU, which there isn't a memory, but if I had any type of elevation or any type of latency on the infrastructure side, it would go ahead and pop up and show me, ok, yeah, this is a definitely.18:59Influencing the latency of your application.19:19Business transaction and this is something we need to look at. But this case it looks pretty healthy and I'm able to go ahead and, and move on. We also have a network agent that also provides me with any type of throughput, RTO, any type of TCP loss or retransmission time.19:24And this also associated with that snapshot as well. So I can see if there's any type of network.19:43Latency or any type of performance impact events, which we call them pie that are happening in impacting that particular business transaction snapshot at the same time, ok? I've also got database visibility as well.19:50So I've got a correlated view with my database instance in terms of the actual query that's being invoked on the back end, right? And I can look at the JDBC query I can see the queries being made over here to the database from this application service called e commerce services. It's calling in.20:06To the database and it's executing this particular query, right? Now, in conjunction with that, I've got a really cool capability through our database agent, which automatically gets correlated.20:26Over here and indicated in the, in the flow map for the business transaction snapshot, where I can drill down into database visibility and actually I've got that over here on the left hand side. I can go down here and look at it and drill down or I can I actually have the slide over here.20:37Done for me, the dashboard we brought up for me. And then over here I can look at all the impact of that particular query so I can get back into my business transactions, I can go to add to cart, and you can see the business transactions that are correlated with my database.20:56So I can look at the amount of weight that business transaction is taking up compared to other business transactions that are using that instance of the database and I can drill down into that particular business transaction, see all the queries that are being invoked.21:14And then drill down into the query that's taking an unusual amount of time to get details on it, its consumption, its wait states, and all of the other information like which client you're using it, the actual query itself down here at the bottom, and any type of session information. So this is again providing that rich.21:32Activity information that that rich correlated information that we have from the database side, from the infrastructure side, and the application side under one roof for us. So now I've got the ability of looking at.21:52Any type of or they can look at things like execution and explain plans as well, which is more DBA information that can be that can be retrieved from this particular query as well. But it might show, you know, where we, where the DBA might want to.22:06Optimize this particular query to be able to, you know, get us or away from maybe any late wait state waiting latency or any type of resource consumption that might be impacting our users experience.22:27Okay? So again, this is a little bit of, again, moving out from the database side and I could have gotten to it over here if I've got database visibility correlated or turned on and you can edit that capability. If you got the database agent engaged.22:43You essentially want to be able to, link that database agent or link that instance to your database over here. If you're an appdynamics administrator, so this allows me to jump right into the visibility of the databases which I'm looking at and I can, you know, determine root cause.22:59Alright, so that's a little bit about how we kind of measure from an end to end perspective and use the complete variety of tools both from the infrastructure side, network side, database side, to be able to give me a full end to end picture about what is going on. Now.23:20When I'm looking over here at the application dashboard, let me go back into the dashboard, excuse me.23:40I'm gonna go into the dashboard and I wanted to show you how we can actually work at that level. So I'm gonna go over here and skip into the application. I'm gonna go into my application called AD commerce. And again, when I talk to the slide that.23:46Shows you kind of like, how do I approach the application dashboard from a, from a, you know, a blockwise view here. I want to make sure 1st of all, my application or my or my threshold is on here or my, my.24:03My baseline is on. Cause without my baseline on, I'm not gonna get a lot of interesting information down here at the bottom. We get a lot of metrics, but I'm not gonna understand whether these spikes or any of this performance down here at the bottom is really, you know, causing a problem until I turn the baseline on.24:22I turn the baseline on, now I get that little dotted line down here, you can barely see it, but it does provide me with the baseline trend about what is normal we should be seeing, and that's really important when we're 1st, you know, using the dashboard here to look for issues. I like to again look over here in the middle and.24:40And if you've ever turned on the legend at the top of the flow map, this will give you kind of a an understanding of where things are in the flow map or how to read the flow map in terms of all these tiers that are appearing here that each one of these represents either kind of a web service or microservice of your application that's being managed for the entire app.25:00E commerce application over here. The middle of the tier represents the response time to errors versus the baseline or the outer part of the tier that parked that's in yellow over here.25:20Represents the actual health of the nodes. So if I click on this, I can see I see two of my four nodes are in yellow, ok? I've got servers here, ok?25:31But I've got two of my four nodes that are appearing oops, that are appearing here and in yellow. And I want to maybe drill down into that. I wanted to and look at maybe the nodes that are that are having issues. I can also look over here on the right side.25:47And look at my health rule events. I've got a variety of violations that have started over the last day. I've got some application changes. I've got some anomaly detections that have started. So the anomaly detection is working and it is working in conjunction with my, with my traditional health rules to be able to understand.26:05Where my anomalies might be, and I might wanna isolate those anomalies and look at them. I generally like to start at the business transaction health. That really gives me a really good understanding of where the problem is, but whether you start here or whether you start at the transaction scorecard and drill down into.26:24Any one of these areas that have a particular high rate of percentage is up to you. I like to also focus down here at the bottom and see if any traditional peaks and valleys in my calls, if I have a dip in calls and I haven't rised an elevation of average response time, it probably is an area I want to be able to drill down in.26:42Two. So if I have a certain area of.27:02Of interest. E.g., if I want to expand my time picker into maybe three oh I'm gonna go back to 3 h. Well, I have an elevation here of response time, but I don't have any, I've got some pretty steady traffic, so it's probably ok, but if I wanted to, I could hold my left mouse button down.27:05And I can view, set the time range here to that particular time range or I can view the snapshots that are being gathered that support the elevation response time. So I can go ahead and look at that particular time range in context to everything else in the dashboard. So everything else turns into that time range and you can see the time range.27:24Over her at the very top of the screen, right? So a tradition, it it changed from a, like a 2 h time range into the time range of 08:52 to 09:37 central, which I'm looking at right here, and this might allow me oops, that might not allow me to more isolate the issue a little bit more effectively. And finally, as I'm working in a clockwise fashion.27:44I can also.28:04Work up here into the troubleshooting tools, ok? These are the three tools that I wanted to concentrate on. Slow response times, errors, violations and anomalies. And this will give me an idea of for those that have been using AppDynamics for a while, every time you get a an alert on a violation.28:05That you subscribed to, you would get a an email notification that would look something like this, and that would be a health rule violation that is a representation of the email that you would receive, so I can go back to the presentation.28:24And I can look at a couple slides here that I wanted to look at, where did I put that right here? Yeah, so you would traditionally get some type of email notification, right? And you would drill down into that violation, and that violation would allow you to go ahead and look at any type of notification that's coming out.28:41Right here that looks something like this.29:01And then you can drill down into the issue, right? And that would provide you with the ability to be able to drill down into, into the, into, you know, into the violation and get an understanding of, you know, where that violation is eminating. E.g., if it's the checkout business transaction, I can go into this and I can obviously go in.29:03To the business transaction for checkout.29:23I can look at my checkout BT. Again, this is the flow map that we were kind of looking at before during my end to end example, except there's NO browser here. I'm just looking at it from the back end, but it's the same example minus the actual, icon that represents the browser from the, from the end user migrate, but I can look in here and drill down.29:26Not into the snapshots. One of the useful features I want to point out when we are doing troubleshooting, is this transaction scorecard Cause the transaction scorecard will give me an understanding of when I'm seeing an unusual amount in this case an unusual amount of errors.29:46Or if I'm seeing an unusual amount of slowness that's elevating the blue line, which is represents my average response time or my latency, I can drill down here and determine how those are impacting, you know, my end user response time. Let me get back to my 2 h example here cause I got something interesting I wanted to show you.30:02I'm gonna turn off normal.30:22Errors and I'm gonna focus only on slow and very slow business transaction. Actually, I'm gonna turn off stalls and slow. You can see how the transaction scorecard nicely lights up how the elevation in response time during this particular time interval is impacted by the slowness of the application. And I can isolate this.30:24And look into.30:44The events and the snapshots that are supporting this particular transaction score view. This is really nice for helping me identify and drill down into the snapshots that are most impactful and then drilling down into that 19 s call, realizing there is an issue here with the database.30:45Again, it's emanating from some method call over here on the right hand side, but the result in we are having a slow JDBC query that's resulting. And the best thing to do if you're a, a developer or sport engineer, I can certainly drill down over here, but I like going to the slow calls and errors.31:05And looking at my slow database calls and remote services as well as my slow method calls. This is gonna give me kind of a, a nice summary breakout of everything that I'm seeing within the flow map over here, but it's gonna detail it. This gets me down into the call graph.31:23There's nothing wrong with going over here, but if I want to look at the entire landscape of all of my slow calls going to the database or anything that's going to remote service calls or my all of my slow method calls, I can get to them through these links down here at the very top of the page too. And this provides me with the ability of getting into this query.31:39Drilling down into, oops, into the, into the call graph and then through the call graph link to the database, I can really determine the root cause. So this is another way of getting into the controller, and providing you with a few clicks to resolution, but again, this is a very.31:59This is a more of a reactive approach, and I wanted to show you this because this is a a workman like approach to being able to go ahead and troubleshoot.32:19Your application issues again looking at the server, looking at your network, looking at your slow costs and your application as well as the database side. Alright, now within doing this, it's always important and I always will tell all of my customers, customers there is NO substitute.32:27For a good transaction reporting and understanding which business transactions are the most important for your business. Sometimes when you turn out auto discovery or when auto discovery does its work and goes out there and discovers what APIs or endpoints you.32:46You have for your business transactions, it pulls out a lot of useful information. It also pulls out a lot of information you probably don't need at all. So there also needs to be some grooming or combing of your business transactions and which ones are the most priority, right? You can use auto discovery grouping here.33:05As, as Drew has mentioned, we also have a health rule to be able to build out your inventory so you can identify which business transactions are the most important and how they're coming out of your application through your endpoints. Okay, so it's always really good to be able to go ahead and make sure.33:25Whether you're using a proactive or reactive approach to app dynamics, to make sure your business transaction inventory is well groomed to represent what's most impactful to your application. And then obviously in your health rules, we want to make sure your health rules.33:45Are also being, optimized to make sure that those health rule conditions within each one of your health rules reflect the actual metrics that are being pulled out and being examined by your.34:01By your baselines. We want to make sure that all the conditions that are reflected in your health rules, whether it be a business transaction, whether it be something from the database side or anything to do with JVM monitoring or.net CLR monitoring, if it's a.NET application is well.34:20Managed and well optimized over here. This is gonna give you your best bang for the dollar in terms of helping you understand and really separate the truly impactful health rule deviations within your application from those that are just causing a lot of noise and potentially.34:37Are false positives. So this is very, very, very important. Especially when you're taking a traditional approach and it's always a good idea to make sure that these are optimized for your applications. Okay, this is just 123 ABC type of.34:54You know, work in app dynamics, but it's always important to start off with, you know, optimizing your BTs and making sure your health rules are optimized as well. Okay, so now that we've got the a little bit of approach from a, a, a reactive monitoring approach, and I've shown you a few tools here.35:11That might be of interest to you. Remember the transaction scorecard, the transaction snapshots? Remember how to navigate the controller start with the dashboard and then make sure your, your trend line is on, your baseline line is on, and then navigate, I don't know this is going to full screen, navigate to the dashboard.35:32Work into the flow map, go over to the right side and take a very clockwise approach working all the way up on the left side here into your troubleshooting tools, right? Okay, now if you're an application developer.35:51I mentioned that earlier. You might want to take a little bit different approach to the dashboard here. Rather than starting from the application dashboard, I would recommend starting over here in cheers and notes. Look at your application. Again, these are your services, some of these, so if I'm responsible for building and maintaining the e commerce services application, I want to start over here.36:06And look at its flow map, and I want to have everything in context, my snapshots, my errors, my slow response times.36:26You know what servers are impacting my application in terms of any type of infrastructure latency. I want to be able to work over here in a world that's sensitive to me or or or context to what I'm working at. This is very nice maybe from a production support engineer or maybe from an SRE perspective, but if I'm a developer, I want to go.36:33Over here, I want to look into the services that I'm building and maintaining and I'm not maybe interested in any of these other services, but I am interested in the e commerce. That's what I'm, you know, that's what I'm responsible for, and I want to understand how that is interfacing with the rest of the world that.36:53I'm calling or I'm consuming from, ok? So just a little navigation tip if you're a developer, always, you know, try to start with the tiers and notes for you as well. Great. All right, now I'm gonna switch gears a little bit. I'm gonna move over here gingerly.37:09I'm gonna move up to the slide deck because hopefully what I've done for you is I've given you kind of an understanding of what we just did in reactive mode of starting with the application dashboard, following the red, drilling down to more specific views, finding the abnormal business transaction snapshot, and by the way, linking and sending them. That's one thing I didn't.37:28Can show you, there's a little icon in the.37:48In the title bar that lets you it's a little representation of a paperclip icon and that allows you to link and send that information to other team members like development team members that want you want to be able to go ahead and look at the issue that you're investigating.37:51Okay, we've looked at this, we've demoed that. Now let's, you know, go to something completely different. All right, so I mentioned this at the very end of the reactive demo troubleshooting. Make sure your health rules.38:09An anomaly, your your health rules are in good shape. Make sure your policies are set up to be able to trigger on on observed violations and be able to go ahead and set those triggers on observed metrics custom events health rules and anomalies.38:26Make sure your alerting is, email notifications are set up, so it's going to the right aliases or the right people, and then the custom actions. Being able to set up the actions that provide HTTP response code or response messages to or request messages to.38:42Of your enterprise alerting systems, ServiceNow, mooxoft, pager duty, things like that, making sure that's all set up for you. Cause this is gonna set you up for success, not only for reactive monitoring, but it's also going to be setting up for further success.39:01For proactive monitoring that we're gonna be talking about right now. So proactive monitoring th the the APM troubleshooting workflow is a little bit different than what we showed you for reactive monitoring. You'll be starting from an e email alert notification which you're currently very familiar with, and I kind of showed you.39:18Put that email alert notification looked like earlier, you certainly get it in in in any type of monitoring you're doing in AppDynamics, and this is gonna allow you to drill down to a new dashboard that provides context to troubleshooting and expediting your mean time to resolution and mean time to identify.39:37Of the issue. We want to use that for drilling down into our RCA and we wanna be able to use that to help us make.39:57And advise teams on what is going on with the our particular issue on the application side, ok? So I'm gonna go ahead and give you an example of how that works. I'm gonna move over here to reactive or proactive. So when I drill down into an email.40:04Notification, right? I get immediately, I will get immediately this type of interface. Now you're usually, you showed your, the previous one we were looking at. If you looked at a few slides before, remember I showed you this one? This is the one you probably are traditionally used to.40:23There is a new feature called AI troubleshooting agent. It's, we call that our AI directed troubleshooting. And what that does right now, that allows you from that same link in your email to drill down into this health rule of violation dashboard, right? The launch of that dashboard is the same as what you've seen before. It's coming out.40:42Down here from violations and anomalies, and it's focusing on business transactions, right? So over here, any one of these, I can also drill down and I can get the same interface. Now, this is a new feature that allows us to be able to identify unusual behavior across business transactions.41:02Browser and mobile applications, we have the same feature available on the EUM side as well as infrastructure and database. And it works out of the box with NO configuration required and it allows you to go ahead and explain the why. In other words, it goes beyond our traditional anomaly detection capabilities.41:21To explain the why. It provides suspected root cause analysis, guided recommendations, and transaction snapshots, as you can see in the bottom for supporting evidence evidence. So again, this new feature provides the capability of identifying unusual behavior in your application. You can see there's two diff.41:42Different, areas that you're looking at within the context of the dashboard, so provides an area for.42:02An AI generated view. And this view will allow me to go ahead and identify what health rule violations am I seeing that are impacting my health rule violation and my application. So this AI generated summary at the very top of the page provides the overview of the issues and.42:10Including details of the impacted.42:30E.g., in this case, the business transaction started end time of the violation, the violation states for the example, whether it's critical or warning and information about the deviated metrics, right? And this will leverage AI to analyze and correlate all the metrics related to the health rule violations and generate an info info.42:32An informative summary for you. So these.42:52Very slow percentage and number percentage calls. This is coming right from our health rule of violation, so if I was to click on this, I would see these two as conditions inside my health rule of violations, right? So this is summarizing what is being violated or away from the baseline or away from a specific value within the health rule.42:55Down here at the bottom is really the helpful information. This is the suspected root cause, and this really shows us the most probable reasons behind the health rule violation. And you can view, it'll help you view up to three suspected root causes. It's giving me one here, but you can see up to three of them depending on the violation.43:15And each of these will it be accompanied by an AI generated summary that explains the underlying issue. The section also provides AI generated recommendations as well as the summary that help you identify and resolve the identified problems. The sources.43:36Component at the bottom is something you're very familiar with. These are the, the, these are capturing the business transaction snapshots to examine the potential issues related to each of the violations that you would see up here at the top. So if you have a violation for database, maybe you have another one here. Number two would be maybe for infrastructure, e.g..43:56Number three would be something to do with network could be. Each one of these would provide the summary of the, of the suspected root cause where it thinks it, it, it, the root cause is eminating from and then supporting snapshots over here down at the bottom. So I can click on any one of these snapshots and I can go down and exam.44:16Root cause as well that supports the app, again, optimization of the identified query database structure is the recommended so the RCA is being provided for you. As we enrich this feature, we're gonna provide even more RCA.44:36Enriched content as well. But this allows me to go ahead and summarize and go right into the snapshot. I can load the snapshot here. I can go right into the the the snapshot which is really the same use case I was using before, but the nice thing is it's doing the work for me where I don't have to search for a variety of snapshots.44:50And I can drill down into the suspected root cause to be able to find the issue that I'm looking for. In other words, in a JDBC issue here.45:10I also will have issue with, with the, with it on the database side as well, ok? Here we go. Here's a spring and I can always isolate that as well. I can set that to root, expand that, and I can also look at any type of other related issues that are being invoked from either the GDBC I see.45:18Side in this case from the middleware infrastructure side as well.45:38Okay, so that is nicely providing for me. This is a feature that is available, in the controller version 2510 as a feature that you can get turned on, through a support ticket, just tell support you want this new.45:42AI troubleshooting agent turned on and they'll turn it on for your controller. You probably wanna, for those that don't have it turned on, you probably wanna examine it in the documentation and run it by your, AI governance committee before you actually turn it on. You want to make sure it's cleared and everybody has an understanding what this feature does.46:01So it's clear with your AI governance committee before it gets turned up, ok? But this also gives you the ability of drilling down into the issue and also identifying the event as well, the issue as well. Okay, that's one of the new capabilities. The other one that we have is.46:22Is if I close this or I get back over here, and I put that one, then this is a new feature, actually not as new, but it's been out for a little bit, but I want to direct it to you. This is called app application transaction diagnostics or ATD. This is another AI.46:42A feature related feature that allows you and looks at how your metrics have been performing traditionally and whether or not it is a an issue that is probably gonna raise its ugly head in the future, so it's looking at past.47:00The president information is looking at where it's emanating from. In this case I've got, I'm over here in a business transaction called get credit summary. Now if I look into and drill down into transaction under transaction snapshots, there's a new link called transaction diagnostics, ok? And this transaction diagnostics gives me.47:16The.47:36A, a summary of, you know, how my application response time has been performing on that particular business transaction. It gives me areas I can drill down into, that I can have suspected root cause. So this is also showing me that there is potentially an issue coming.47:36From the credit services web service or credit services microservice, and these are the potential snapshots that I might want to look at. Now, these are all gathering useful information and probably elevating information that's already in context to a violation or a health rule alert of some sort, but it's showing.47:56The where I'm probably getting.48:16Behavior that's going to be repeating in the future if I don't maybe turn some attention to it. So the whole idea of the transaction diagnostic is more of a proactive feature that shows me, yeah, ok, I I have a problem here, I'm probably gonna based on, you know, what I'm seeing.48:19Going down the tracks or going and what I'm seeing that in the future, I'm probably gonna see a very similar issue unless it's not some attention isn't applied for remediating the issue today.48:39So these are the two new features that are AI related. One is brand new, which is the AI directed troubleshooting or AI troubleshooting agent. The other one that's not so new that it's been out for a couple years is transaction diagnostics or automated transaction diagnostics, ok? So I wanted to introduce you to those, those new features, and that is really providing.48:50You with.49:10More of a proactive approach in how we're, how we're addressing root cause for the issue, ok? I just demoed that for you. And again, just kind of just summarizing this. I really love this slide because it really shows the elegance, the architecture behind the magic.49:11The inputs, we have dual detection here. We're combining machine learning anomalies without with with zero configuration and that's the AI automated detection of the performance strips using our health rules. Previously with anomaly detection and app dynamics, which has been out for the last three or four years, you would turn on anomaly detection and would.49:30Use.49:51Unsupervised learning to detect patterns of average response time and errors per minute. This troubleshooting feature, this intelligent troubleshooting feature leverages your health rules that you've worked so hard and you've built over the past, you know, years of your reactive efforts in act.49:52Appdynamics, ok, for position alerting on your specific, on your business logic. So it's the best of both worlds.50:12The AI finds what you don't know to look for and the health rules protect what you already know which is critical. Those is the engine. This is where the agenda AI works, it's magic. I'd like it to call you GPS for troubleshooting, but here is how the GPS actually works. The engine traverses the entire call path, not automatically walk.50:18Walk you through or walks through.50:38The the all the services, backends and cross application dependencies to isolate the fault. And then it specifically analyzes state transitions. It looks for the exact millisecond, a warning became critical to correlate the timing with deviating metrics. So you don't need to manually map.50:39Every correlation the AI is doing that it capture work for you.51:00And finally the output. This is all our actionable intelligence. In the summary section we saw within our AI troubleshooting example, you get natural language root cause analysis reporting that replace the cryptic alerts explaining.51:03With with explanations anyone can understand, but we also go a step further. We identify the AI top three suspected causes. In this case I only saw one, but there's up to three causes giving each one its own individual summary and remediation steps, right? So this is really nice. This is giving you an intelligent.51:20That way.51:40A proactive way to go through troubleshooting with appdynamics. So out of the box with this feature, there's NO configuration, there's NO new agents to installed, there's NO manual tuning. It leverages your existing health rules and all the capabilities that you've built out within your business transactions, inventory or.51:41Manifest and works what you have, right? Leverages your logic.52:01And provides instant time to value. Okay. Hopefully this gave you a little bit of an idea of how you can use some of the newer features of how we've got going using troubleshooting used using traditional what they call reactive troubleshooting techniques that I've shown you.52:05And gives you, you know, a little bit more clues on how you can go about traversing the controller as well. Okay, ok Chris, I'm going to find out if you've got any type of questions from the panel.52:24Or in that have been maybe addressed by the panel or anything that you want to discuss here, ok? Somebody wanna get off mute? Rob? No, we're actually, we actually have to keep it on mute for I know we're almost at time, right? So I want to be respectful of that, right?52:44That as well, that as well. So, very good job and this was a, you know, kind of a culmination of everything that has been built up to it. We'll put the recording or the, the YouTube link in the chat if you guys want to watch the previous recordings, but I do.53:04Do you want to, we're ok. I do wanna stop and if there's any other questions that have not been reached, oh, there was, there was another question.53:24How, how could I use AI features with only synthetics hosted and API and Brem? Oh, ok. So, there is a, a new capability or 1st of all, for EUM.53:38For both mobile and browser, you want to turn on your, your, your anomaly detection feature, ok, which is something you've already had, and then you're gonna get an AI troubleshooting dialogue like we saw from the APM side and the example demo for EUM as well, ok?53:56So, that feature, again, I believe is available in 2020 05:10 or 261I can't remember. I think it's 2061 that the EUM AI troubleshooting agent AI troubleshooting feature is available.54:16Okay in that controller version. All right, now for synthetics, we don't have that capability for synthetic chat. There's another question on and it was related to EUM troubleshooting.54:35Anomaly detection. If I'll ask it this way. If a client has questions on how to do something like do EUM troubleshooting, how could they, if they don't how how could they ask an account contact or whatever to have an accelerator with us?54:54Oh, again, that's something that we could, we'd be more than happy to entertain for you. You would start off with your, probably your Cisco account manager, I would think through, right?55:14And that person get a hold of us where the customer support or success team for app dynamics. And we'd be more than happy to entertain that. Yeah, yeah, absolutely. So we are the customer success engineers that can go out and and help customers. So if you guys.55:28Ever have a question or need kind of more of a working session or, you know, and you guys have.55:49So if you guys are affiliated with your Cisco account team, you know, or or your app d admin might be the one that kind of sets you up with us. You just go through those channels internally to set it up for a customer success engineer to meet with you guys and, and work with you. There's also.55:56Links for caller consultants.56:16Alright, there was another question, how, oh, is there a way to customize application dashboards? No, there isn't, as far as I know, please correct me if i'm wrong guys. You can you so what I'll say is that.56:22That application dashboard is a depiction of the real user traffic, the actual.56:39Traffic we're seeing flow through your system. So the flow you're gonna see is simply by us tagging and following stuff. You can ungroup like say you make calls to 50 databases, you can group those so it looks like a call to 50 back end.56:45In one bucket and you can ungroup those into the 50 unique back ends. You can do the same for remote service, but otherwise NO.57:05You, you can make custom views though that filter things perhaps in an like if you care only about one tier within an app, you can make a custom flow that shows.57:16Only that tier, but it's your, it's not modification up to the default. It's, it's creation of a new one. So, when, when you're saying icons, you know, and what what Drew's saying, he's he's mentioned or he's talking about the flow map, you can adjust.57:31Just how you're, what you're looking at on the flow map or you can refine it to be only certain connections as if you were drilling down on it, which is not an option on the, you know, custom dashboards, but you can do that on the application dashboard. But in terms of like which charts you see and which, which data is.57:51Being aggregated based on what you have selected, you can't necessarily like modify those specifically, but the the flow, the diagram that you definitely can modify that. And I'm pulling, if you want, like I'll pull up a example really quick. Sure, you want me to stop?58:11Yeah, give just don't give me 1 s. I'll pull it up and grab them and if any of you want to, have help with that or or any type of assistance, we're able to help you with that too. Okay, we can hop on a bridge and help you with that too.58:31And then we had another question that's pending too, which is how do we use the war rooms? Oh, the war room feature in app dynamics. Yeah, that's interesting. That I I have to, I I I I think.58:46That's a great question. That was a feature that we, we actually put in the product. We've never taken it out and we've had it since the inception of the product.59:06And I'll, I'll I'll be I'll be transparent. I've never used it. It allows you to be able to supposedly draw in other app dynamics users and have a, a consensus meeting, so to speak.59:15But, I haven't actually used it. I mean, I I would say try it out in, in your test environment and see how it works. But I I'll be transparent in all my years I've been with Apti for eight years. I've never used a war room. Appdynamics has been out for, for so long that there's things in there that you know.59:30You know, later on bridges became a lot more popular, so everyone just goes on like a Webex bridge and they talk it through there as opposed to using it in there or or using it in the actual tool.59:50So it's kind of like something it's more like an obsolete approach, so I wouldn't worry about the war room so much. Yeah, it's a feature that came out when the product was introduced in 2010, and I think it kind of like kind of just nobody took it out of it and kind of stayed there, but you know, like I say you could play with it. I just don't know how it works today.01:00:04I'll be transparent with you, yeah. But in most cases though you'll see people go on a bridge and you'll see people sharing sharing the links, the, the linking feature like at the top of the snapshot and they'll just send each other the links and be able to talk over chat or conference on a, on some sort of a video chat like this.01:00:22Yep.01:00:42And just so just to call out, can everybody see my screen? Yeah, we can see it. I don't, and I'm sorry I don't have a very interesting app to show this in. I'm looking where I'm I was hoping I had one that would call more than one endpoint, but let's see.01:00:45Just maybe one of these will. Hi dr. Ohio here's an example. Here's what I wanted to show you. So here's what I was referring to yet. You cannot change any of this stuff.01:01:05This in the middle is a depiction of your real user traffic, but notice this is a bucket, so what type of change can I make is, is what I'm gonna show you. If I have more than one backend, the default behavior is gonna sometimes going to be to group them. I can.01:01:16I can ungroup them to see the flow uniquely and I can do this for databases, and I can do this for remote services, anything that's gonna show in this bucket separately. I also have the ability to group some of them. If I had 30 of them, I could leave two or three that I really want to see out of the group.01:01:36And I could group the other ones. Those are the things you can manipulate and then they make a new custom, you need the proper permissions, of course.01:01:56In your environment to be able to manage flow maps, but you also, you could hypothetically do a filtered view of what you want to see here. I'll say from a best practice standpoint, most people just drill into what's important for them rather than get into making.01:02:07Get into an exercise of needing to make custom views for each team. I would say best practice is more in line like if you only care about one tier to Rob's earlier point, just go right to that tier rather than.01:02:27If you're an app D admin having to make custom dashboards everybody, we are. Thank you, Drew. Yep. Sorry, if you guys heard that audio, I was trying to get the.01:02:43The link for somebody there, the the last question which is they're needing the, the playlist where the recordings are. So I'll I'll send that link out to you in just a second. Let me see. Are there any other questions even intervication question came?01:03:02Yeah, and I provided the, the, documentation for the war room, but I'll I'll let you know that it's, it's not as commonly used as it as it once was. So, but if, if it's in the certification, it's, it's one of those.01:03:25You know, it's it's part of the product so they expect you to know, but at the same time it's it's not really commonly used so in terms of actually using it, I would I would skip it and just go on a bridge.01:03:41Alright, so, this is the, this is the location where all of the, all of the recordings are as well. Oh, go back, please. Right, my bad? Go too back, actually.01:03:56So, 1st of all, you guys are talking about certifications in the chat, so you guys are probably more than aware of the Cisco university. But at the same time, it's, it's very important if you guys are wanting to build your resume, look into not only appdynamics, but otherwonderful products.01:04:12Coming from Cisco, you know, we are, you know, very innovative with the AI bubble and that, you know, you need to have networking, you need to have data centers and we're a huge part of that. So, check out everything we're into at the u.cisco.com, and.01:04:32As well if you want to start building up your, you know, resume with certifications, et cetera, you would be able to accomplish that there.01:04:51So go ahead and go to the next. If you've enjoyed this video and you want to see our other content in addition to the recordings, check out our, YouTube channel, which is youTube.com/at Cisco observability. We have a lot of short content.01:05:00Tent long content. It's all gonna be on demand and you know we do polls, we have comments available where you can post you know what.01:05:20What your guys's feedback is, on our videos, how we can make them better or what kind of topics you want to see. And then beyond that, there's also, a Slido at the end here where we would love to hear back from you guys, anything that we could do, do better, anything.01:05:31We we can see if that will we'll we'll pass that pass, we will pass that feedback along to the education team and and yeah keep that keep that kind of.01:05:56Keep that kind of feedback coming. We, we need to know that stuff. We are obviously, you know, working with customers every day and more caught up with, you know, what people are doing and, you know, sometimes the education part, you know, it, it should be caught up, but we'll, we'll let them know about that as well while you're on your phone.01:06:14Feedback. So without further ado, let's go ahead and go to the last, and if you guys would give us any extra feedback on how we perform today, if you want to let us know about educ your your any other feedback towards education as well, you can put it in there and we'll, we can relay that, but we we are capturing this here as well.01:06:34Well so it won't be lost on us, and we will take the time of letting them know about the, the certification updates that you guys are asking about. But in terms of, you know, how you liked our session today, what kind of content you guys would wanna learn next or.01:06:54You know, would like to see more videos on YouTube next. Whatever we can do to help you, let us know and then we will also.01:07:14We we just really appreciate you guys being here. Any feedback you guys would have would be grateful. We would be grateful for. And then now before we go, let me go ahead and grab the playlist for, where all of these videos are going to be for the.01:07:25For this series and let me just grab that real quick. Do you want to share Chris? No, NO, just leave it leaving here. Okay, sure.01:07:44Alright, I posted where the recordings will be, and then also.01:08:18I think I wanted to provide the, ok. No, that's it.01:08:26We, that folder will be updated this weekend, so we'll make sure that this video's on there and any others that need to get thrown on there as well. So, and then also check out, check us out on the community. We're gonna have a, another.01:08:43Series starting next quarter, so please check us out on there or check out our invitations for the next round. Like I said, your feedback is what drives us, you know, so anything that you guys end up telling us, you know, might be put into the next round or or.01:09:01For future series. So, we really appreciate you there. And, thank you all and we'll go ahead and let you go. Have a great rest of your day.01:09:20And great job Rob and team. Thanks everyone.01:09:31