Observability Onboarding: Mapping the Realm

Okay, welcome back everybody to another observability expert insight series. This series on this quarter is going to be the observability.00:07Forwarding speed run, unlocking easy mode, and this is gonna be kind of level one mapping the realm.00:20You know, when you're 1st starting out your journey with monitoring with Splunk app dynamics, there seems to be, you know, quite a bit of information that you guys have to learn and in addition to, you know, thinking about what you have to plan, what you have to deploy, what order you should deploy it in, and then, you know, it's a very critical time.00:26Time because it sets the tone for the whole rest of your journey, you know, there's templating, there's, you know, pipelines that you're going to be subscribing to or setting up. So coming up with the good plan up front, understanding kind of some of the best practices be behind that, you know.00:46You know, that's gonna set you up for success. Now, before going too much into the content, I'm gonna be your host today, Christopher Banano, and I'm, we are joined today by our, you know, good friend and colleagues, good friends and colleagues Drew Osborne, who's gonna be our expert speaker leading us to.01:06With today's content, and then also Calvin Miller. We've all had some skin in the game on this, so, you know, we all have some, some good insight that we can provide and you know it's gonna be an awesome session, awesome series, and we really look forward to working with you guys on it. So, let's keep this going.01:26That one.01:46Now some housekeeping items, we're going to assume that you guys underst have a general understanding of what Splunk AppDynamics is, its core concepts, its core components, and then you already have a controller, and you're just starting to, you know, think about how you're gonna roll this all out, how you're going to.01:48To, you know, scale at your enterprise level, you know, what agents are gonna be involved?02:07And things like that, what things to start tuning and thinking about standardizing et cetera. During this call, the guest microphones will be on mute the entire session, not only for privacy reasons, but also for us to kind of keep track of you know the the questions coming.02:13In versus, you know, this is gonna be a recorded session that's not only going to be out on, the, you know, Cisco community page, it's also going to be sent out to you guys via email if you registered and then it's also going to live live on, YouTube. So, you know, definitely check us out there.02:33We'll give you the link a little bit later. And then if you have a question, please use the Q and A options. If you're looking at Webex and you're like wait a minute I don't quite see it. Go down to the bottom right, three dots, you'll see the Q and A button there. That just helps, so if you have a question, it persists and then we answer.02:53People can go back and look and and not ask the same question because the chat kind of, you know, might shift through or shift through where, you know, we might ask a question, you guys might be interacting with us giving us feedback. Hey, the microphone is low or hey, I don't see the slide, something like that. Give us feedback.03:13That give us interaction through the chat and try to ask your questions in the Q and A obviously if you, if you put your question in the chat or still gonna try to answer you, so don't worry about that. And then as a disclaimer, we do make this content for you, but it is not meant for you to go and take it.03:33And go to your, your other customers and try to offer the same services. This is owned by Cisco for our customers and in the in the spirit of enabling you all. But at the same time, you know, we are.03:53I'm here to help and at the same time it is our content, so please don't take it and claim it as your own. All right, so without further ado, and now that we've got the those housekeeping pieces out of the way, let's go ahead and hand off to Drew Osborne.04:08Hey Chris, thank you so much. Yeah, again, real quick, my name's Drew Osborne. Great to have you with us today. Here's the agenda for today. We're gonna start with some planning considerations including understanding your application architecture and monitoring needs. And we're also gonna.04:29Cover team access, you know, maybe some best practices on naming conventions. The actual agent deployment will be covered in a follow up session. So you can see, solution mapping what we'll talk about middleware communication, some network firewall considerations, and then of course, you know, operational readiness.04:49At the end. So that's good. All right, understanding your application architecture is critical. Let's discuss how many servers you have in production and non production environments. What roles do these servers play?05:09E.g., are some web servers and some application servers? Do they run unique code? If you have architecture diagrams, this is where we would want to see them. It's also important to understand the business purpose of the application and how it delivers value. Ideally, have your business team.05:28Demonstrate this for you live. And so you could see that's really what this slide's talking about. I want to turn it over to the the other panelists though that we have on here. Do you guys have anything you'd want to add to that? Yeah, when you're looking at your application flow and what kind of.05:47Of infrastructure, is behind it, you know, and then, you know, so you can start to think, you know, do I have multiple of, you know, multiple applications that are structured in the same way? Are they, you know, all on the same kind of server infrastructure? Are they in cloud? You know, understand.06:06Obviously the environment and and everything, it starts to help you templatize so that you don't have to, you know, reinvent the wheel with every application deployment. So it's real important to start understanding the architecture and then understanding, you know, what value you need out of it, what, what holes are.06:27Are in monitoring and then start to replicate those steps and you know right out the gate.06:46You know, you can start thinking like this. And I just wanna add Chris brought up a good word in what he was saying that was templates. Once you get your architecture form, you can start creating those templates so you can easily roll out this type of information onto your servers, to make it just easier.06:52For you to instrument these, the app the monitoring app architecture onto your application. So setting up templates is a good thing. Having your templates so you can easily roll things out is a big plus when it when it comes down to looking at your architecture and how to.07:12Get everything instrumented.07:32And, and I'll just add one more quick thing about that is with those templates, you can use them as checklists as well, right? So you can mark which servers are, you know, when you have a good template, you can mark which servers are actually done with the installation. Maybe one had a problem.07:37Like a one off. It's nice to be able to track that stuff because then you can perhaps not have certain blockers actually block you. If you, you know, if only one host didn't install right, you could troubleshoot that and move forward with the other ones with good tracking.07:57On a good template. Oh true. And so thinks, so let's map your monitoring needs now with appdynamics. You know, are you looking for basic application performance monitoring? Do you have a browser or a mobile?08:16App, do you need to monitor end user performance or browser real user performance? Is synthetic monitoring required for availability or API checks? Do you want any server visibility on it? What about network flow with the network visibility agent? We can also do database instance.08:34It's monitoring, we can really tailor our solution to meet your need. And so it's about understanding again from the previous slide, what application architecture you have.08:54Man, how does these six components essentially fit into that and where would you use these features? And looking at the, the diagram on the right, the flow is actually makes sense where you usually start with the APM and infrastruct.09:06Structured then you're looking at database and browser mobile. From there you go to your network monitoring, then you probably want to end up doing some synthetic monitoring just doing some.09:25Automated monitoring of jobs within your system. So this is a good flow of how you would just start off adding in your agents so you can get that monitoring of your environment. Yeah, great, great point. Just to add one thing.09:41Or one additional point to that, right? Most, most clients are gonna start off their deployment journey by doing one module at a time, right? But, you can significantly increase your deployment agility.10:00If you, in the future as you mature with the deployment categories and they become, they become standardized and we're confident in our org that, that the deployments tend to work, you can actually try to do all agent deployments up front.10:16It's a lot harder to manage that though. I'll just call out if you, if you don't have good, you know, tooling and automation, but it can increase your agility to implementation significantly if you wanted to look at a strategy that works that way.10:35So Chris, is there anything you want to add there? All right, middleware communication in the in the your backends. We want to consider what type of calls your application makes. Are they web calls? Do they use message.10:55Aqs like MQ. Are you calling 3rd party APIs or cloud based APIs?11:15That might be serving a function that your app uses. What database back ends do you use? Do you have a, you know, Oracle or SQL back end that we need to monitor it? Knowing this helps us plan the right monitoring strategies.11:21And plan for the right agent configurations and deployments so true. If you're, you know, the beauty of AppDynamics is largely when you start deploying, you start to automatically start seeing, you know, the connections, but then if.11:38If there's something that's missing, then you can immediately start, you know.11:57Going into refining transactions, identifying additional backends that you need to, to, you know, call extra attention to and separate out from maybe a big group of them. So it's gonna be real important for you to understand how the application works. So if you are an implementer.12:02Versus somebody that fully understands that application. Make sure that you have the right people on board. Make sure you have, you know, either an application owner or you know the architect that can help them, you know, set up all of their communication or developer that is, you know, maybe hard coding or or.12:23Dynamically coding their, their application so that they're calling out to these other services externally. So.12:42It's gonna be real important for you to get those those correct people up front so you understand the context that that Drew's spelling out for you here. Yeah, I will I'll just add just following what Christia said, that that way you make sure you get that full picture of that end to end communication throughout the trans.12:49Transactions that's going through your application by making sure those backend, that back end infrastructure is being reached and and being able to be displayed within AppDynamics.13:09Yeah, I I just it's and Chris used the term when something's missing and Kelvin reiterated there without a complete picture and knowing, you know, the context of the application, this is where we identify maybe appd didn't auto discover something.13:20That we need.13:40And so getting this information up front helps the validation phase again. Alright, network and firewall configuration, we depending on your agent types, you want to verify that your agents have outbound.13:41Connectivity to app dynamics endpoints. You could be using SAS.14:02Right, and so that would be going over the internet if you're on prem, you can, you can be a little bit more specific for sure but you want to configure your firewall rules to allow the required ports and IP ranges if if you are using SAS to avoid connectivity issues. And we want to really make sure.14:07We've verified that the right port requirements are open. Just we don't want to deploy to prod and have agent data not report cause we forgot a firewall rule, e.g.. And then there's different.14:27There's different connection points, you know, just to kind of add on to what Drew's saying, you know, it's it could just be a simple tech test with a curl command or SCP or if you have some sort of, you know, wireshark or you know whatever you guys normally would use to, to check, you know, you, you probably have.14:44Have your most critical applications running in some sort of demilitarized zone, right? Your DMZ, you know, and then you also probably have disaster recovery. You know, if something happens where, you know, your servers are going down or you get some crazy influx.15:04You want to make sure that you account for not only the the key players, the key servers and resources, NO matter where they're running hybrid cloud, on prem, you know what like what Drew was was giving you examples of, but you also want to account for things that aren't going to always be reporting but are going to need to.15:21Be reporting if, should there ever be a need? Like, like we were saying with the disaster recovery, you know, if you, if you have disaster recovery servers laying their dormant and then they spin up, you know, you're gonna not be happy if, you know, they get added to the, the.15:41The load balancer, but you can't get any monitoring on those because you didn't account for doing the testing up front. So, you know, it becomes very important, definitely go with some sort of automation script, you know, if you have a ton of servers, you know, the enterprise scale, obviously.16:01Or if you have.16:21You know connections from, you know, various cloud services. It's, you know, just do your due diligence, have somebody go out there and, and, you know, test all the connectivity, make sure all the, the firewalls are open or and you know that you're going through, you know, the proper channels between security and networking.16:23And we're, we're gonna touch on this again later on, but it's it's a.16:43It's not a difficult step, but it is a tedious one that you do not wanna miss. I could mention one instance that happened to me, what you want to make sure especially in your controller info.xml. The SSL line in there has to be set to true when.16:48Go on the CSAS and the port 443. So you want to make sure that that property is set properly. I had an instance where a customer was trying to connect agents and it wouldn't connect and the problem was they did not have SSL enabled, so that just gotcha I just wanna mention when it.17:08Looking at network ports and, having the agents connect to the app D controller.17:28Good point, it's definitely got you.17:37Hi Drew Drew, if you're speaking you're on mute, sorry. Thank you. I was.18:03Additional planning considerations. Consider your deployment model, SAS or on premise as this can affect your planning. Ensure network and firewall ports are open for connectivity, plan for SSL and secure communication to protect your data. Review integration points like extensions and resting.18:11Api's and decide on agent rollout strategies manual, automated or scripted.18:30Also plan for agent upgrades and compatibility in the future. Version control is very good here. And again, good credential management using a secure credential store helps manage things the most securely. Just ensure you're doing secure credential management.18:35Yeah, very important considerations and, you know, understanding these things up front allow you to kind of put in requests for things, you know, in parallel or, you know, more up front so that, you know, with network access, maybe you guys have, you know.18:54With with our controllers for SAS being up in AWS, maybe you're in, maybe, maybe your company has.19:13It's own ADF or I'm sorry AWS or maybe it's on Google's platform or maybe it's on one of the other major players and providers like Azure or openshift, you know, what et cetera. No matter where you guys are, you guys, your networking team is probably gonna want to review things with you.19:23So it's gonna want, they're gonna want to prioritize probably cloud to cloud, you know, it.19:43As long as you start getting the wheels in motion going towards that way, you know, they'll, they'll guide you, you know, it's not gonna be everything you have to architect. It's just you have to explain and articulate what, what it is that the controller's doing. We have great documentation available to you. We can drop it into the chat, so that.19:49You can kind of follow along with that and bring that back to the networking and security teams. But that's probably gonna be where you're gonna have a lot of focus there. And then, you know, just to kind of reiterate, you know, security, you know, along with it, with the networking, you might have certain connections that are going.20:09Up to the SAS controller where they're gonna want to validate SSL at some sort of a, you know, maybe it's a load balancer or some sort of a proxy in between as opposed to maybe your APM versus your analytics traffic going up to the controller, and.20:29In certain cases, they might prefer, hey, you know, we have dedicated IPs for our controller. You know, that one, it can just go straight out, up and out. But since the enterprise console or you know maybe things for enterprise console or event, event services, which is in elastic.20:49You know, we have them divvied out for our customers, but because it's dynamic, the, the IP range changes a bit and you might, they might prefer there to be a little bit more checks and balances along the way of that traffic, so then, you know, you're gonna wanna see where that SSL verification's gonna be happening.21:09Thing within your network, you know, going, going out or in or coming in, and in addition to, you know, just making sure that, you know, your integrations and everything are working, I don't necessarily want to reiterate everything that that Drew went over very succinctly, I just, but I did.21:29You want to really cover the the networking gotchas that I've run into as both a customer and as a customer success engineer. Sorry Calvin, what would you want to say? What?21:49I wanted to mention item four and five because of course I'm doing a deployment next week on the agent, EIS own deployment, but I think that is so important because you want to have an automated way of deploying your agent so you can keep them up to date so you can take advantage of those new features.21:59Coming out with new agents. So I I just wanted to point those two out because I think they're very important because we're constantly putting new features in our agent and having a way to upgrade those agents fairly quickly and with a dependable way of putting them out there, is a.22:19Big deal so you can stay up to date on the new features that's coming out within the AppDynamics.22:39Real quick on that, what it it support will support agents up to two years old, right? And then, you know, beyond that, you know, we just recommend within a year or what what's the recommendation there, Galvin?22:46I'm not totally certain. I think it's one year support go back one year on the agents, not two years. But I have to check into that. I'm not a hundred percent sure. That's my short as well. Yep. One year. So.23:02So in short, if you don't want to put a support ticket in and get the easy answer of, oh, did you try updating your agents 1st if you don't want to just get the easy answer from support, then, you know, make sure that they're up to date.23:17So now we should have a pretty good picture of what the application landscape is gonna look like for the app that we've, we've designated for implementation of app dynamics. We're gonna know, you know, essentially architecturally, does it have web servers? How do those maybe.23:40Talk to various back end servers.23:59Services, they might be distinct services, are they Kubernetes? Are they distinct VMO hosts? Our discovery to this point should have captured all that information. We're gonna have a really good picture of all the dependencies for that that are gonna need to be mapped for end to end monitoring.24:02And the end result is, you know, once we've installed the agents or scoped them properly, we're gonna hope we see an accurate representation of that, that real user flow or real real traffic real work through the systems.24:24All right, team access and our back considerations. Defining team access early is essential. Role based access control ensures users have appropriate permissions, that they're programmatically assigned. We don't have any security concerns, and we align with standards in our organizations.24:52We recommend integrating with external authentications providers. Saml's generally the the choice for streamlined user management and security these days.25:12Especially when you're 1st starting out, even if you have to make some like dummy shell groups and, and have some people validate, you know, if you're an implementer, if you're an if you're an appdynamics admin and you're implementing things for other people, you don't necessarily always know what you're looking for. You have a good.25:27The idea.25:47Particularly everything that's supposed to be connecting. Hopefully you have some diagrams and you have an idea, but you know one can only hope versus if you have that app, that app owner actually validating for you, then you know that you're on the right track and you don't have, you're gonna have less rework, you know.25:52Later down the road. And then in addition, you know, around when you're doing the, the, I mentioned earlier the request that you can submit internally in parallel, you know, so when you're doing your firewalls and you're doing things like that, you can also start connecting with the teams that going to be supporting the application.26:11Going to be using what you're building, so that you can find out.26:31You know, what's their team structure? Do they have kind of like a tier one group that, you know, does sanity checks and then, you know, they would just need like read only versus are they gonna, do you have like a tier two group or a more advanced group that starts doing some of the monitoring governance, like setting up health rules or.26:37Building out dashboards and kind of baking things more into their flow and then do you have like a tier three group who's gonna be, you know, not only wanting to be able to create health rules and alerts and, and things like that, they're also gonna wanna get.26:57You know, be able to adjust business transactions and you know how the actual flow map is looking or additional, maybe they're gonna reassign jmx M beans to point to the correct metric that you know maybe AppDynamics is picking up incorrectly. You know, people that are going to.27:13You're refining what everybody else is gonna be looking at so.27:33You know, obviously having some you know checks and balances in place of who can do what, even past the point of validation and you know when you're just deploying, you know, it becomes a very critical point, and at the same time, it's not something that you have to put off until it's.27:37It's already all deployed. Like Drew said, you know, start setting up that SSO and and then you can start getting those those buckets created and like some sort of active directory or whatever flavor you're using and then associate that with some sort of ticketing system so then you as an admin or an implementer.27:57Where you're not sitting there manually putting users in there, you want that to be done kind of automated in the background. Maybe you have a, you know, some sort of approver that says, Yeah, hey, this is my application. I want this person that's putting the request in to be able to modify health rules.28:17Or modify the flow map but everybody else is looking at or hey you know what? This person is a tier one. I just want them to be able to look at it and help us monitor the environment, but I don't want them to be able to adjust other stuff. So let that be on the other group, it's less liability on yourself. And at the same time, there's, it's very.28:37Important for compliance internally to every company. Especially if you have, you know, secondary customers or, you know, depending how, how deep your model goes there, you're gonna wanna make sure that you safeguard yourself and the information going through your applications.28:57I totally agree Chris, you covered that perfect. I think you hit every point I could think of, great job on that. Thank you. All right.29:16Naming conventions and limits. Here we're referring to the appdynamic, the application name, the tier name and the node name. That's really the the main objects of APM monitoring. Appdynamics has a character limit of course for the application and tier names. That can be up to a hundred.29:34Characters and agent node names can be longer. I think it's important for you to think about.29:54How you want to standardize things across the board when you're picking things like these. So you're gonna, you're gonna follow generally a standard for your app name you know tier names or your node names. And my colleagues here will talk about some of those.30:01Well, I'm going to talk about tagging. We have it listed down at the bottom. Some items you might not want to put in the agent note name or the tier name. You can also use tiers and with tier, I mean, I'm sorry, you can also use tagging to determine certain.30:20Items so you can cross over the application and group things within health rules and dashboards so that you can have a central view of items that may bleed over into different app applications and you can always group them together using tags. So tagging is a.30:38New item within appdynamics, but it is very helpful when it comes down to crane health rules and dashboards. Great call out, right, yeah, that tagging is a newer feature. You know, back when I was a customer, this.30:58The naming convention was a major gotcha, you know, when we tell you guys the limits, the character limits, we're not, we're not trying to tell you to overachieve and, and try to try to get a hundred characters in an application name, but at the same time, you know, you wanna name things that are meaningful.31:18To your guys's, you know, not only internal terminology, but things that maybe are not that maybe people that aren't the app owner, maybe they want to have some sort of context. So you want to have meaningful names that explain the relationship of what that, that overall goal.31:38Role is.31:58So for the example that we have here, there's e commerce, where, you know, the variable that we were giving as an example is the business application name, and then the the example of it would be it's an e commerce website for your company. And then within there, you have multiple tiers and one of those.31:59Most tiers would be, you know, the, the grouping of.32:19Of nodes that all have the same code and they're grouped together under tier name and you know it would be like check inventory. Now, when you're dealing with the tuning, the business transactions and stuff like that, it's doing it through the tier level. It's doing it at the tier level.32:23Is a better way to put it. So if you start overly diversifying things at the tier level, you're gonna end up creating a lot more work for yourself. You're going to have to create the same business transactions for, you know, four different tiers, say that you break it up by.32:44Data center. You have four data centers.33:04And you have, you know, check inventory 123, and four as opposed to just having them all under one tier check inventory, and then if you want to call out those extra data centers, you can put them in the node name, so that way when you apply that.33:06Business transaction detection rules, it applies to all of them because even though they're across different data centers, they're not gonna have different calls usually. They're gonna be running the same code just from a different location.33:25And it's gonna be based on like how the load balancing is working or failover, high availability, whatever the case maybe. It's not likely that you're going to need different transaction rules based on different data centers. And these are just some examples, right? So, you know, if you have, we'll we'll kind of shift.33:40Down into the node name. This is where I, I see most people and most value.34:00Adding extra little tagging or or breadcrumbs per se in the naming. You know, if you have certain namespaces, if you want to have it dynamically name, the, the node after the.34:07You know, the the instance that's running if it's a, you know, a kind of like a Kubernetes where it's coming up and you need to be able to see that container ID and namespace that it's associated with there you know or, you know, do you need to call out, is it a VM? Is it.34:23In, you know, Arizona or is it in Rhode island? Is it in.34:43You know, production, is there you know several of these so you want to number it so or a host name, you know, these are just some examples here, as you can see, and then this would be kind of an example. The P Z is saying which data center it is. So I don't need to add more into it.34:47It's also associated with the host name, so sometimes the host name, sometimes the things that you already have standardized and named in your environment, sometimes it already captures it for you. And that's what we tried to give an example here. So even though we're calling out the limits, I am not telling you to overachieve and try to reach those limits, but I do.35:08Do recommend adding some breadcrumbs in there for you to associate what you see at a tier node level under a business application, you know, back to what it's running in so that it's easier for you to differentiate when you're creating.35:27Dashboards and health rules or if you're just simply trying to associate it back, to, you know, what it's running in. But like I said or like Kelvin said rather, there are more there are more things that you could utilize nowadays where before that was the only option.35:44Option was your tier and node naming. Now you also have some tagging availability. So, you know, definitely try to to figure out, you know, what's gonna be better to put in the name versus tagging. And I highly recommend when you're doing the deployment, test it out.36:04And try it a few different ways before and then set that as your template going forward for everything else. You know, see how it comes in, see how reports, see if it's as granular as you want it to be, see if it's easy to follow for your support teams. And then once they check off and say yes, this is exactly how we want it.36:23Roll it out to everything and then keep that set standard, that naming, you know, becomes very important especially for visibility in how you're going to see the application.36:45Represented in appdynamics. Nope, nope. Couple best practices I want to call out because there's a lot of information that was shared there guys. Chris mentioned if you split data centers tiers by data centers, direct, what, what that'll directly output for you is twice.36:56It's the amount of business transactions. If you have three data centers and you split by three data centers, you'll have three times the amount of business because he, as he called out, they're, they're discovered distinctly per tier.37:16So we would The best practice is to group those and use something in the node name to tell you which data center it comes from. And another thing I would call out when you talk node name, make sure you include a segment in there that is exactly how I'd already.37:31Key to the host, your SSH to the host.37:51I have seen instances where people pick a st an alias standard, and when that host goes down, then you need a key to figure out what that alias means. So I recommend you, you, you have host name in there somewhere.37:55Definitely, alright, prioritized transactions and monitoring. This is really focused on business transaction monitoring.38:14Okay, so when we call out priority transactions, what we mean is we want you out to identify key business transactions for end to end monitoring that are tied directly to business outcomes. We want to plan how to collect end user monitoring and analytics.38:32This data if needed.38:51That's gonna allow us to segment and do other types of custom metrics perhaps. We want to configure alerting in health rules or pardon me. We want to I'm gonna Chris I'm gonna start over here because my slide, my.38:53This got updated and my script doesn't align with it. So let me start that over. No worries. So prioritize transactions and monitoring. This is where we want you to identify key business transactions for end to end monitoring that are tied directly to business outcomes. Your most critical business transactions are.39:12The ones that we want to ensure that they're all monitoring. We're also gonna plan how to connect end user monitoring and analytics data. This is gonna allow you to define custom metrics and perhaps do some unique segmentation strategies that that is really powerful at enabling you to do.39:31Now we also want to integrate your database in any back end hardware monitoring to give you comprehensive monitoring. We can monitor the back end instance from the perspective of the instance and the front end or with APM, we're monitoring what is the perspective of the.39:51The application.40:11That's reaching out to the database. We stitch that then together to allow you to make intelligent decisions on is the problem the database or my app? If you have both modules, then we want to configure alerting. And that's where we're, that's where we're gonna configure alerting in the health rules that are directly aligned with.40:12The critical transactions that we identified in the 1st box. That way we're alerting on things that we know is actionable and tied to business outcomes. And this this segment, I mean this whole series, this whole, this session, you know, we're not necessarily.40:32You know, intending to.40:52Have you reinvent the wheel in your learning in terms of, you know, understanding what all of these like business transactions and SEPs which are service endpoints, you know, we're, we're not necessarily trying to reteach all of that to you, but just as a quick refresher.40:55You know, a business transaction like Drew gave a perfect example of if they are your most critical transactions going through your application, but say that you are also look trying to, you know, as a developer or in prod support, you're also wanting to do a comparison of maybe.41:13The application that just got redeployed, you have version one and version two, you have maybe a single node of version two out there where you're wanting to kind of compare the single hop of, you know, say that you have a full up, upstream to downstream flow, which is super important and a great use.41:31This case for a business transaction, you want snapshots being taken, things like that. But then you also want to, you know, monitor that one hop to see, hey, you know, what's the performance difference between version one and version two? I want that specifically called out.41:51And you can do that very easily with a, you know, service endpoint and you can just monitor that one hop going from, you know, you know, something upstream to that version one application versus the upstream to that version two, and you can have them sitting side by side, but at the same.42:08Time you're not.42:28Having this extra load on the controller, you're not having all these snapshots being taken, but at the same time, you know, the full flow from start to finish is capturing all of that and taking snapshots and then allows you to dig in deeper, but then you can have independent views from a service endpoint.42:30Where it also collectively it gives you a collection of all of the snapshots that would be associated with those same service endpoints too underneath it. So it just really helps put that all together for you.42:50I, I found that as a big gotcha when I was starting things out where I thought everything needed to be a business transaction or they're not. So, the very important part to for me when I was starting out and wanted to share that experience with you. Well Chris I'd like to talk about identifying key transactions also because there is a limitation.43:02In it for business transactions. So that is the reason why you want to identify your key transactions. And plus with auto discovery it can discover a lot of noise that's not relevant excuse me, to your monitoring needs. So you want to exclude those so you can stay below that limit, get rid of our.43:22All that extra knowledge and be focused on key performance indicators so you can be able to discover issues faster and lower your MTTR.43:42Yeah, great, great point. Just to add to that, the limit itself is 200 per application and 50 per node. That can be changed, but I want to call out one other key point. When we identify our key business transactions, we want to make sure that those are not bucket.43:53It's and what I mean by a bucket in app D is appd might group URLs based on the 3rd segment of the, the URL.44:13But the actual important endpoint, like the API might be the 5th segment, and we might have ten different APIs. If that is now bucketized in app D, we're grouping ten distinct APIs that might be business critical into one.44:24One transaction based on the 1st three segments. If the all ten, we can probably safely assume have very different performance. And so it's really critical that your when you're identifying your business critical transactions, that those become distinct.44:43Transactions and not buckets in app D or you're going to struggle with alerting. You need, you need the tool to be able to baseline the distinct critical call and not be polluted with other things in the bucket if it's a critical transaction.45:03And then it only then will you be successful in tuning alerts for that endpoint. Tuning alerts on a bucket is very hard and usually ends up creating situations where your health rules stopped providing value because thresholds were pushed so high.45:22That they almost don't fire anymore or when they fire they're not actionable. Buckets are generally the culprit for that, alerting on a bucket, and we can provide, you know, links to, you know, more in depth business transaction tuning.45:42In the chat, as well, you know, when this is posted out in the.46:04And our YouTube, definitely take the opportunity to look for business transaction tuning videos there. You know, there's there's it can be pretty in depth and it could turn into a gotcha like Drew's calling out, you know, where things are rolling up under something else and you, you miss another call.46:10That you want prioritized and separated out. So, good call outs.46:30Things I've definitely experienced as a customer and and over on this side at Cisco. And one last thing I will mention on the last block configure configure alerting, not only configuring alerting, but you also with business transactions, you probably want to turn on another.46:36Detection to catch those outliers that.46:56The system just picked up through AI so you can have that machine learning and understanding of what's going on if it's So it can discover those anomalies. So, that's a big thing that I would turn on as far as configuring along with configuring alerts is turning on anomaly.47:00It's a great point, Kelvin and just if you're brand new to app D, that is a controller side setting that doesn't impact the application performance. All right.47:22Operational readiness. This is a big one for me. Identify your monitoring team, the key stakeholders responsible for ongoing maintenance. Do they have the necessary skills and resources to know how to use the tool based on what we configured for them? We want to plan to tune our service.47:40Entry points and health rules after deployment to operate, optimize monitoring effectiveness. These are all things that are, are, you know, we're gonna be in a pre prod stage before we actually go live and connect it with our ITSM. Tools.48:00For auto incident tracking as a best practices, e.g., make sure we know when that alert fires people know what it means, what they need to look at, and if there's not a problem with the health of the application and there was a problem.48:18With the health rule that indicated or like a false positive, a feedback loop to get us.48:38From pre production to to ready to go live in a week or so is a great thing to think about. And to help you in those endeavors, you know, you have a lot of resources, you know, as a customer of Cisco's, you have the ability to not only.48:44We send people through the Cisco university and take courses, you know, depending on, you know, what, what type of role that they're gonna play in using this tool. But you also have an account team with dedicated resources and.49:04They can connect you or that group with somebody like us where the customer success, you know, experts where we're engineers that have been using the tool. We are familiar with it, we can explain to you not only.49:24How to set things up, but also how to use it and how to think about it and how to work it into your process. So if you're unsure as an implementer, if you're unsure as an admin, an app owner, and you're, you know that this is needed and you're trying to wrap your head around how this group is gonna use it, maybe you're a shared services team.49:42Maybe, you know, whatever the case maybe of, of your role, you know, but you're also gonna wanna make sure that the tool gets adopted or all of this work that you put in setting things up is for nothing, right? And then at the same time, the monitoring of key key business assets, you know, or or.50:02Your infrastructure.50:22Is gonna be extremely important. So you want people using it, you want it baked into their process. We have a saying where, you know, when you're rolling out app dynamics, you want to align the three Ps, it's people, process and products. You know, in this case we were talking about your monitoring solutions that, that you have available for, from.50:23From AppDynamics.50:43AppDynamics and mapping them out based on, you know, the, you know, what you guys have versus what solution you need in place, and then you want to start aligning the people behind it and and prove their processes, you know, based on what their feedback is, maybe they have SOPs already existing and you.50:44You can just think of how to fit the tool into that. You know, this becomes very important and at the same time, you're not alone.51:04You, like I said, you have the Cisco university, you have your account team that is represented by Cisco. You know, your implementers should have access to talk to those folks, they can set you up with customer success engineers and, and experts, as well, you know, there's our YouTube page.51:14Which which will give more information. There's community pages. We'll give you more, more resources like this at the end, but you're not alone. Don't stress out, but it is a very important point to start baking it in and and getting people in the know of how to use a new tool you have.51:33I'd like to discuss this last one on the list here, the plan for tuning service in entry points and health rules post deployment. Appdynamically you just don't put it in place and it takes care of everything. You you're gonna have to revisit due to code changes and just.51:51Performance changes do could be due to hardware changes. So this is the ongoing thing because you can have a new key performance indicator introduced into your application that you want to monitor. You may have to do some configuration of that later on in the process. So it's not just put it in place and let it run.52:10You do have to pay attention to it and understand what you want to monitor in case you have code upgrades or new things introduced into the application that you want to monitor, so it's something that you have to revisit and make sure that your key performance indicators are being monitored over time.52:30So I would say you probably want to set up maybe a monthly talk with the app team to make sure that the, business transactions that they want to monitor are being monitored or any key performance indicators, you know, are actively being monitored.52:49Okay, great point. All right, documentation and communication. You want to make sure we maintain updated architecture diagrams and documentation throughout the deployment. If you have.53:10Have a playbook or consistent communication plans, you know, make sure that you keep those documents updated. That way you can establish communication plans for things like deployment status updates and escalation paths to ensure smooth coordination.53:28Yeah. And this is part of, you know, the, what Kelvin and and or what all all three of us were saying on the last slide too, you know, you want to make sure that everyone's kind of talking to each other, you want to make sure if you're an app the admin or an implementer, you know, you're gonna wanna revisit things every so often.53:46You know.54:06Obviously the developers have a lot going on in their sprints, so, you know, having being you know being setting a standard and precedence ahead of time and helping them understand why, you know, will help them justify, you know, creating a little bit of space.54:07Face or wiggle room for, you know, letting you guys know of any changes or, you know, sometimes it's just simple, you know, there's a new call coming out that's really, really important for a new flow in an existing application and you just want to make they wanna make sure that they're able to monitor that correctly and you're gonna.54:27Be able to help them and so if you want them to be able to instrument that change themselves or meaning like through the UI, just setting up a new endpoint, you know, or a business transaction, then, you know, at least they will know how to do that. They could be.54:47You know, self regulated, but at the same time, you know, if they're going crazy and they're, they're putting over like what Kelvin said earlier, the 200 limit for business transactions, then there there's probably a chance that they're missing stuff. So by having documentation of what's going on.55:04The changes.55:24You know, maybe they have their own site of things that you can look at as an apti admin and then just regular communication like like also like Kelvin and and Drew were saying where you might wanna re reconvene with them every so often, you know, it just kind of keeps everybody in lockstep going towards having.55:25The full end to end view of the environment, all of the IT assets, and it it just doesn't get away from you, you know, out of sight, out of mind, but if people are using this regularly, you know, you want to think about what they're looking at regularly and make sure that they have what they do they need because you know NO one likes.55:45Been woken up at three in the morning and to find out that somebody else changed stuff without letting anybody know. Think of it that way.56:05Be considerate and at the same time, if you're the owner of AppDynamics, you also want this to be something that is usable, so do the right thing for your customers by, you know, keeping up on that communication as well.56:13All right, to summarize our our best practices, if, if you're a new, you, you know, we have free trials that are available of our SAS controller or, you know, we, you can reach out and discuss doing an on prem proof of concept.56:36So we, again, we we want to use our flow maps to and custom dashboards early to visualize application behavior to know that our map looks correct. We want to leverage things like the smart agent for life cycle management or agent version control.56:55So if you can't do that perhaps agent, if you're not gonna use the smart agent, you know, other CICD pipeline tools or configuration management tools can be very helpful here.57:14To automate things. Regularly review and upgrade agents, right? You want to stay in support and if you identify a bug with or identify a reason to upgrade, try not to end up with 20 different upgrade versions across across.57:27Across your applications. Version control is very helpful.57:47You want to customize your health rules and alerts to reduce noise. Generally you can have a standard framework for alerting, but the values that are gonna be inside the actual health rules themselves are distinct per application. So just know that and make sure you tune them to make sure.57:52They're they're only actionable. We definitely don't want noise. You want to encrypt sensitive credentials to maintain security compliance, and we really want to engage business stakeholders continuously to prioritize observability focused monitoring.58:12When we tie it to business value, we can always quantify the results and the effort that it took to get us there and tie it directly to business value.58:30Next steps in our next session, Kelvin will be doing an agent deployment overview and specifically talking about the smart agent, and we're gonna discuss the app tier node hierarchy in more detail.58:50Thanks for coming. Alright, let's go ahead and great job Drew. Great job everybody, and let's go ahead and open the floor for Q and A Excuse me.59:05I have a question. I'm just kidding. Great is that way. Please please make sure you fix just before we forget, what slide did I mess up on? It would be slide.59:22I think it was team at name it was slide twelve, slide twelve, business transactions and monitoring strategy. I think you changed it to prioritized just if you can cut my fumble out.59:45Oh, NO worries. Oh fodge. I just screwed up the whole recording. No, I can cut it all out. You're good. All right, let me get, let me get into this though. So, awesome job guys. You know, we we went over a lot of information. We all come from various backgrounds where.01:00:03We help customers, we work customers, you know, we all have a lot of insights to give you. So hopefully we didn't overwhelm you with our chat.01:00:23We thought it would be a little bit better for you to get extra context. We usually try to focus these, expert insight series a little bit more around demos, but with it being a planning, you know, all about the importance of planning and what to plan for, it's a little bit different.01:00:34And then, you know, just going into the controller and and showing you things that you've already seen of what you could get or what you can't get. We want to help you guys plan correctly, so we wanted to spend a little extra time.01:00:54We all had a lot of insight, like I said, we all have a lot of background to kind of back that up. But what do you do beyond that? The additional resources that you have is our We have multiple communities that you can join as well, learning opportunities. So I would highly recommend getting into the community dot.01:01:06Dot cisco.com, which, you know, has a lot of discussion especially around the integration of tools with both app dynamics and and other Cisco products. And then, you know, Cisco also owns Splunk but and Splunk is the observability suite.01:01:26Under Cisco's flag there, so also you can check out the community.splunk.com which also has appdynamic a lot of the appdynamics discussions that used to be on the old AppDynamics website, it they're all moved over and migrated over to the Splunk website now.01:01:46You know, there's a lot of really good and powerful, integr integration discussions, people that have had problems that other people in the community have helped them work out or custom needs that, that they, you know, were trying to go for that they, you know, again, they had a community, they had people behind them because.01:02:06It's very important, you know, dealing with these kinds of things, you know, where if you're doing something that NO one else has done, get some help, get make it a discussion. Talk, work with your community, work with your, and then of course.01:02:26You have the account team that I mentioned earlier, people actually that are representatives of Cisco working with you guys as our customers that could set you up with, you know, you might have professional services, you might have, you know, customer success experts like, like ourselves, where we come in.01:02:40And we help, you know, our job is to help you guys be successful. So if you guys need something like that, definitely put in a request for that. And then if you'd like this content that we created for you here, and you would like to see not only these longer discussions, but also maybe short videos.01:03:00That, you know, you can go on to our YouTube page, which is YouTube.com/at Cisco observability. We have a, a good amount of information there. We have playlists, again we have long videos, short videos, and at the same time comments are turned on, so let's.01:03:20Just know what you like, what you don't like, you know, how we can make things better, like subscribe, we appreciate it, it supports us, and then at the same time too, say that you're going towards wanting to get certified, you're wanting to, you know, show the world that, you know, you have these badges, you can put it on.01:03:39On your resume, you know, whatever the case maybe, go check out U.cisco.com where you can get.01:03:59You know, additional training and go towards certification tracks that it like I said, becomes very powerful to put on your resume and and get yourself certified. As always, we really appreciate you guys being here.01:04:09And we would love to know how you think that we did, you know, please check out this our slido. It's a quick survey, there's only three questions. Largely, you know, what we get from you guys is how we develop our, our next quarter's content or, you know, decide what topic to cover next or.01:04:29Decide what which YouTube video we're gonna make next for you guys, you know, you guys are in control of the content that you want to see from us in the under the you know the flag of enablement. And then, you know, you can use this link in passcode or if you wanted and you can.01:04:49You know, either give your information of, of who's giving that feedback or you can answer anonymously and or if you wanna, you know, keep it completely separate from your work laptop or whatever computer you're on, you know, grab your phone, scan that QR and do it on your phone and again you can be anonymous or.01:05:09You can give some information there. Whatever you can do, we really appreciate it.01:05:29But ultimately, thank you for being here. Thank you for making this success and we will see you next week with Calvin delivering on how to deploy and you know start getting that automation set up when you're dealing with you know a deployment strategy for appdynamics. You're monitoring.01:05:33Your monitoring solution of choice, right? All right, thanks all and and good job team on our side.01:05:54