Observability Onboarding: Deployment Velocity

Hey, welcome back, everybody. My name is Christopher Bardono, your host today, and we're back again for another observability expert insight series. This quarter's series.00:09This is all about.00:20The observability onboarding and kind of making it more of a speed run, you know, if you're more into video games, you might appreciate the terminology of a speed run or unlocking easy mode or, you know, what's the cheat codes to it kind of a feel, ultimately it is refined and boiled down.00:21Content based on what we've seen with other customers based on what we've experienced as customers, as customer success engineers, and today we're gonna be focusing largely on level two, which is the deployment velocity. So, when you're trying to onboard, you know, Drew set you up last week.00:41Talking about all of the considerations and all the prep work that goes into it, our back, setting up networking and making sure you have the interconnectivity that's required as well, going into the.01:01The naming and making sure that you name things and structure things correctly before you even actually deploy, but you're gonna inevitably probably deploy a little bit, do some testing. And then when you start really ramping up, you start going up to scale, this is where this content's gonna be really helpful.01:17For you and we are lucky to have my friend and colleague Kelvin Miller, as well as Drew Osborne again. We will turn this a little bit more into a discussion so we can all share our insights. We've all had experience in the field, not only with customers but as customers or any professional.01:36Professional services sense. So, a lot of variation of.01:56Perspectives all offered to you for you to take forward and and use, you know, in your own situation. So let's go ahead and do some housekeeping items. So we want to automatically assume that you guys have a pretty good understanding of what AppDynamics is conceptually.02:04We don't necessarily want to break down every component and, and everything on this call. Absolutely if you have a question, please put it into the Q and A If you hit the.02:23Three dots down at the bottom, there's a Q and A section. What this does is it per, it has a question that persists the entire conversation and an inline answer, so then everybody that's here could be looking at that as well, you know, they don't have to type it in chat whether might roll.02:37Love the chat definitely, if you put a question in there, we're not gonna ignore you, we're still gonna try to answer you, but the chat's gonna be better for feedback if our mic is messed up or if you want to interact with us, you can.02:57But feedback is gonna be best in in there as well and then we also are gonna have a Slido at the end, so if you want to provide feedback or if you want to put drop us a comment on the recording on YouTube later on, you can do that. Speaking of that, we do have a YouTube channel where all of these are, all of all of these.03:09Recordings are dropped in as soon as they are recorded and and modified and make sure that we don't have any sort of privacy issues and then it'll be a part of a playlist, an ongoing playlist, so this, this will be 2nd episode essentially.03:29Of this series, and we will have, you know, they will progress on another on one another, but at the same time they're broken down by information that's most relative to that one topic. So you could watch them individually, best to watch them progressively, and then all of this content that we have.03:47Here for you today.04:07It's meant for you to be successful. It is meant for you to start off on the right foot if you're just starting your journey with AppDynamics or if it's a refresher later on and you've, you've been with AppDynamics, you, you love it, you want to get better with it, you want to refine your process. Sometimes it's good to go back and start looking at what you.04:08What processes you can refine. But all of this information is created for and by Cisco. We don't expect you to take it and make it your own, but at the same time it's for you and we hope that it helps.04:28So without further ado I'm gonna hand off to Kelvin and he's gonna walk you through the velocity of deployment. Thank you Chris. We'll start off today with talking about the agenda, the full life cycle of appdynamic agent. It's, it's real.04:41Really critical to get this right, this right at the beginning because you don't want to have to go back and do the rework. What we're gonna cover is an overview. Then we're gonna talk about the the deployment strategy.05:00Strategies and best practices before moving into application modeling, planning, and troubleshooting. And then we're gonna finish up with the smart agent, our internal tool for managing the deployment and upgrades of agents, then conclude with an actual real life demo of the smart agent.05:13Introduction to appdynamics agent, app appdynamics agent monitors the application performance in real time across devices, containers, hosts and applications. They send detailed metrics to the controller which aggregates and creates visualizations for perform.05:37Performance data.05:56These agents support multiple languages including java.net Node.JS, Python, PHP, go, and also C++. This broads support enables comprehensive monitoring of diverse application environments helping teams quickly identify and resolve.05:57Issues that maintain optimal performance.06:17Agent types and overview. There are several types of agents available within app dynamics. Each service specific monitoring purposes. Let's start with the application agent, application agent perform monitors the performance of your application code across various programming language that was.06:24I also mentioned those languages on the previous slide, so I'm not gonna go through them again. Next is the infrastructure agent. The infrastructure agent provides visibility into the underlying machine network and container clusters through components like the machine agent, network agent.06:44And cluster agent.07:04After that is the user experience agent, which focuses on the tracking of end user interaction and experience including browser, real user, and synthetics, and at the end we see the database and analytic agent offers deeper insight into database.07:07Performance and analytic data helping you understand that data layer, that data layer behavior and the application impact. Together these agents provide comprehensive observability across your entire application ecosystem enabling proactive performance management and faster issue resolution.07:25Agent rollout strategies vary from manual to fully automated approaches. Manual rollout involves downloading and configuring agent by hand. This is suitable for small environments or testing. Script rollout.07:50Uses tools like curl ourself installing scripts enable a more consistent deployment. Automated rollout is, the one that is preferred especially for a large environment. It integrates the agent into the CICD pipeline using tools such as ansible.08:09Jenkins or our smart agent providing scalability, repeatability, and reduce human errors. It's important to integrate the agent rollout process closely with your application deployment to ensure monitoring is enabled as part of the release cycle, allowing immediately immediate.08:30Get visibility into your application performance and faster issue detection. So I want to discuss this with the panelists a little bit more and see how these strategies, how they use these strategies and environments in the past and where you see, where they can see them see success in integrating these process in.08:49Meet in their workflow.09:09So, if anybody Chris Ordrew if you want to step in and just give an example of how you use the automated or the manual rollouts strategies in the past. Sure, NO, very well put and and set up. So, you know, you guys have a lot of options, obviously.09:11You know being in an.09:31Instrumenter or a an admin, you guys probably start out with the testing things out, getting the feel for it. So you might, you know, start out as a manual, it might go into more of a scripted, but really the goal is to go into the automation, you know, and a lot of times.09:33Teams that own applications or that are deploying applications or, you know, sometimes there's middleware involved or a cloud infrastructure team. Sometimes there's already an existing.09:53Already preferred existing and used pipeline that you can just piggyback on. Why, why reinvent the wheel or the gear at this point, and why why reinvent it if you could just reuse what they have? We have a lot of really good documentation.10:06I will put a link out on the chat as well into the, when I post the recording on YouTube, then, you know, down in the details section, we'll have some documentation links there as well. Definitely pick that up. There's a lot of ansible scripts and everything that's already there.10:25That people have used and it can support you in doing that. And then beyond that, you know, I think Kelvin, what Kelvin's gonna share with you today is gonna just make that all the better. You know, these are for places where you already have things in, where you already have a pipeline and you're just gonna piggyback off of it. But then there's other options that we offer.10:44And it's gonna be great to hear that going in the future slides here.11:04Yeah go ahead, oh sorry Kelvin. Yeah I'll just add like I've had great like real world success at clients where, you know, if they, if they have a large scale.net footprint, we will actually use a tool like SCCM. We'll, we'll deploy.11:08The the windows packages via SCCM will use, you know, as some sort of deployment mechanism to tag by app and ensure we push those updates at a in a, you know, in line with the change window. You know, I've also used, you know, package managers for, for Unix Linux.11:28Ansible's very popular, at at some of my existing clients.11:48And some are now as advanced just from an art of the possible, when we start to talk, you know, dual homing agents perhaps to also report to an hotel backend. There I actually have clients that are, are testing all the pipelines.11:54That are possible through the Otel collector and sending a metric and confirming that metric exists and gets written to the backend platform all in their deployment package so you know the deployment works without having to go to the UI for any sort of validation at all.12:14So there that, you know, gives you an idea of the art of the possible. And I was just gonna mention that when I was a customer, and this was pretty smart agent, we use Ansible scripts to roll out agents and it worked great. I mean, we was able to roll out both Linux and Windows agents.12:35Without using these scripts and using a a host file to pull out the hosting just.12:55Have the Ansible script push it out to all of the hosts and everything pretty much works seamlessly. So, I really liked and by the way, smart agent is written in ansible, too, so.13:02Just to let you know that. Anything else before I move on? I'm gonna say silence, so let's move on to actual agent deployments and considerations. Before deploying an app dynamic agent, it is crucial to verify that the agents are compatible with.13:19With the specific application environment to ensure proper functionality to make sure you have a user account with sufficient privileges to install the agent and restart the application if necessary. Network configurations such as proxies and firewall must be properly set up to allow seamless.13:39Less communication between the agent and the controller. Additionally carefully planning your name and strategy for application tiers and those to maintain a clear and unique identification within your monitoring system. Follow these considerations help ensure a smooth deployment process.13:59And reliable monitoring results.14:19So going back to the panelists, but based on your experience, what are some of the best practices for naming tiers and nodes to ensure that the controller remain organized and the environment scales? I know we mentioned this last week and we talked about this last week, but the importance of making sure you get.14:21Just right the 1st time. I want to go just kind of hear about your past experience on this just so the audience can have an understanding of ideas of ways they can name their tears and notes to make it scalable.14:41So this is, you know, this was something that burned me as a customer, so I I really like making sure that people know about it, and it it's not something that is overly difficult to.14:58To fix if you do get burned on it, but it is rework and who wants that? Especially you have a large scale of agents and, and deployments. So obviously you have your application name, you have your tier node tier and node names, and when you do that, it actually starts to structure that in your flow ma.15:16So, understanding the connectivity, you know, some gotchas would be, you know, maybe if you have a.15:36A proxy tier, say that you have, you know, some something that provides the traffic to everything else and you know naturally, because there's one team that is monitoring that one application that proxies information out to everybody else, you want to just put that all in.15:46Zone one application and and and some level that's ok. But at the same time, if you have a specific tier in there that it's like the the proxy for e commerce, and you have the proxy for the pharmacy, you have the proxy for.16:05You know, whatever other service is going on in your company, then it becomes a matter of, you know, do I really want to have just the proxy sitting there on its own little island and it might be useful for the team that's monitoring it, but the information that's going through it connects to another.16:21Application completely. So why not have that tier that's specified for what other supporting app gives it gives traffic to? Why not have that tier added into.16:41The other application, and you split that one application in a sense of, you know, supported infrastructure and you, you actually put it in a better context of AppDynamics business application. So then you get.16:54Everything flowing in through the proxy, going out to all the different services, the queues, the databases, and you get a more, you get a better understanding of what the traffic is actually doing from its origination at the highest upstream you could, as opposed to.17:10Having things chunked up in two different applications where you you might miss some some things or you might have to work harder to kind of bring those things together. There is tagging involved nowadays, you know, the the newer versions of the controller support tagging, which is really useful and it helps a little bit more with that cross.17:29App, you know, experience or correlation, but if you can struct your applications correctly up front and with that consideration of thinking what actually talks to one another?17:48And you're gonna set yourself up for success early on and it largely falls under, you know, a lot of the the standards that Drew was Drew and all of us were talking about last week in the mapping the realm episode where you know we were trying to explain all the ins and outs of and little breadcrumbs you can put in the node names.18:00To kind of help with that, but this is like when you're actually structuring it and you're deploying it.18:20Another thing that that I really liked to do was to do some testing. You know, we mentioned it on the last slide where we're talking about doing manual processes, testing it out, seeing how it looks, making sure that it makes sense on the transactions and then, you know, going full board, doing all of them all at once.18:26And then making that the template, you know, that's the kind of methodology you want. You don't want to just deploy it all and then be upset that you did, you know, a thousand deployments and now you need to redo them, do some testing 1st. Do, do what's right by the internal customer and by the.18:45App teams that are having you as an app the admin, you know, deploy their, deploy your agents so that they can monitor stuff. Sorry, I long winded, this is a passionate thing for me because of, some, some scars I had from my past life, but you know this it's really important to look at it up front and.19:02Once you'd think of that.19:22And consider it becomes easy. Yeah, so just a couple of things that I would add like from an app name like when as a, as an owner of monitoring, you're also a steward of metadata and tagging standardization. So have a naming standard. I recommend your naming standard.19:23Be the old, the application name, the top level be the only place that you put the environment in, right? And so you you're gonna have the same.19:43You'll have app name-environment, right? That's kind of a good naming standard. Align that app name with something meaningful meaningful in your CMDB, right? If you have a special name that you generate incidents for that app team, align those two values, right? So that your incident management.19:53In alignment. I like to have standard tier naming. I don't put environment in there.20:13Chris mentioned things like web or pharmacy or proxy, right? Those are all good dest descriptive things that say what a tier does. I don't put environment in that that way I can use the same tier name in the dev app as I can use in the prod app.20:19I everything is normalized and standardized once I go inside the app, the only difference is-dev or-prod and what controller it reports to. And then to Chris's point, how do you know.20:37When you have too many tiers, right? Like, how do you know? Here's the actual numbers. In AppDynamics, we discover business transactions distinctly per tier. So the more tiers you have in an app, the more transactions you will have. I do not recommend you.20:53Split tiers, e.g., by data center.21:13If you have a web tier by data center one and a web tier by data center two, you double the amount of transactions that you discover because it's distinct per tier. Combining data centers into a single tier means well, you know, whether it goes through data center.21:17One or two, we're still gonna track it as one transaction and we're gonna use the node name to tell you what data center it went through.21:37And so you don't want over 200 business transactions in your app if you can help it. Apps that get too monolithic and have too many business transactions or too many tiers, you'll hit limits that actually hinder your visibility in the future or not enable the most gran.21:45Reporting that you might, might be looking for. So just some things to keep in mind.22:06And while they were speaking, I was just looking at this slide and I was looking at the number one item here. Verify agent compatibility with application environment. Our documentation has all the compatibility that each agent is compatible with.22:11As far as the application, it's good to take a look at that because I've been burnt several times where I just see a application I know I can actually load the agent on it. I don't look, go through the documentation, make sure it's compatible and I spend hours trying to get it to work and finally go back to the document and realize that the same.22:31And it's not compatible with this application. So make sure you reference the documentation when you're doing your initial deployment to make sure everything is compatible.22:51Oh, you know, since you mentioned that Kelvin, one other thing I think is really another good call out is not to have like ten different versions of the agent deployed across multiple different places in your environment. Try to have some sort of version controlling so that you know, you don't.23:01End up with possible different versions that you have to actually troubleshoot differently with support. Try to standardize on.23:22You know, one version of these, you know, of your your app agent for.net, one version of Java, one version of the machine agent, and then upgrade every six months to an the next supported version.23:30That of your of your choosing. I think that will also help from an administrative overhead standpoint. Great point, Drew, and using automation, you should be able to easily keep those versions in sync. So, I think that's a great point.23:45To cover that.24:05On the versions best practice for agent deployment, following the best practice is essential to maintain a reliable and efficient agent deployment recording plan and perform agent upgrades ensuring that these upgrades are 1st.24:07Tested thoroughly in a development environment before being applied to production systems. Next, second step is schedule the application restarts when required helps.24:27To minimize downtime and ensure that new agents versions are properly loaded. Always review and re always review the release notes carefully to understand new features, bugs, fixes, and any changes that might affect your environment. Additionally, verify agent.24:41Controller version compatibility to prevent communication issues.25:01Properly log, properly configured login directories is crucial for effective troubleshooting enabling you to quickly identify and resolve deployment problems adhering to these practices help maintain system stability and performance over time.25:06We all know the importance of testing upgrades in a development environment, but in fast paced CICD pipelines, that stage is often the 1st that to get squeezed. So I'm gonna ask Chris and Drew, how would you balance the need to.25:23For rearrous testing with depression of rapid deployment. I mean, not only do you want to test with with live or live like, you want to test the agents and how things are? I mean our agents.25:40That's a very stable.26:00You know, you also want to Okay multiple things here, right? You want to not only test how things look and how they they look and feel, you know, Drew brought it up and and started eye about and and we have all brought it up on how we need to see how things are working together, how they're talking.26:02And then also, you wanna make sure that you test the agent. Sometimes there are little flags and features that get added to them, and if you have an older version of Java, if you have an older application as opposed to something that's more bleeding edge and, and in a container and things like that, sometimes the behaviors.26:23They're slightly different and the resource utilization is slightly different. And then also you wanna make sure that you are running things through like your Qualys and your, you want to make sure that it's running through your security checks. Making sure that you have the testing and a process of.26:43Front is going to save you trouble in the long run. Think of like the log for J You know, when we, when we told everybody everyone that had pipelines that were already set up, they immediately ran into doing testing or, you know, plugging the holes and then immediately scaling a a mass rollout.27:03To make sure that they avoided an incident.27:23When the CVE came out, you know, CVEs aren't necessarily anybody's fault. It's a, it's a happens happenstance situation where, you know, things get exploited, things are found, you know, library, there's so many millions of libraries that are being used in all of these applications.27:25You know.27:45I would always make sure you're doing proper testing before you start getting stressed out about making sure you need a mass rollout, and it saved me a lot as a customer and the customers that I've helped. Yeah, and kind of what I've done in practice just to add on is, you know, when I owned it, I might do a.27:47Three month burn in of a new agent and non broad, it would have been a requirement that.28:07You know we we would have monitored prod and then two lower environments dev and test, right? That was the, the, the requirement and so we have the ability to roll out to, to the lowest environment, let that bake in.28:14All the app teams were, were using the tool for validation as well, right? And we had normalized configs across all, all applications, so prod looked like non prod in terms of naming, and that gave us an opportunity to let the agents burn in. They had a period of time, one month, two months.28:30Months.28:49On whatever where they could report any issue, right? And so then we would roll through the next environment in a month or two later we'd roll through prod. So most applications when you load the agent you have to do a restart. What strategies are.28:50Best practices have you found most effective for minimizing the impact for on these restarts to the end user?29:10I know, I know what I used to do as a customer is I used to, as I said in previous slide we use Ansible scripts to push out the actual agent. But what we would do is we would have a symbolic link on to the old agent, and when we get ready to do the restart, we'll just switch that.29:19Symbolic link over to the new agent and then we can easily schedule the restart so it doesn't affect and we can do it during off hours. Do you have any strategies that you used in the past?29:39Definitely, you know, we, we used to very similar, we used to change the sim link to the new agent, and then, you know, we would ripple it. So there was never something that the load balancer couldn't route traffic to or if somebody you know got dropped, it would, you know, they would refresh the screen, it would.29:54Push them over to one of the machines that are up. Load balancing, it becomes a very important one.30:14You know, sometimes you can take the certain machines out of the pool and say, ok, we're gonna restart all of these. And then, you know, once those are all solid, you know, cut over the other half to the other half of the pool that didn't get upgraded.30:20Or that already got upgraded rather, and then go do the same thing to the side that didn't get upgraded. So load balancing and rippling and doing things one by one, stuff like that, usually as a part of your automation, and if it's not, you just have, you know, hey, network engineer, can you take these out of the pool?30:37Will Ripple those.30:57Make sure that, you know, obviously you're gonna have to do it at nighttime when traffic's not so heavy and you don't need, say you have eight boxes taking traffic and you want to do four at a time, you take four out of the pool, ripple those four, you know, switch the, the load balancer again to the the other four that weren't upgraded.30:59I'm kind of repeating myself a little bit, but you see the pattern and it becomes a big point. And then like what Calvin said, if you're doing shortcuts like sim links, it just makes it all the easier, and all you gotta do is restart.31:18AppDynamics have application model strategies, understanding how applications mapped to AppDynamics models is crucial for effective monitoring.31:39Oops I'm sorry. Went back. The model consists of three key concepts, application tier and tiers and nodes. The application represents the overall business application tier group.31:50Group nodes that perform similar functions and notes are similar runtime instances such as servers or jvmiss Defining a clear business transaction strategy allows you to monitor critical end to end processes that impact your business outcome.32:07Planning for end users monitoring and analytic data collection helps capture user interactions and performance insights from the user's perspective. Additionally considering database and hardware monitoring strategies provide comprehensive visibility into back end components and infrastructure health.32:27Together these strategies enable a holistic view of your application environment for facilitating proactive performance management and troubleshooting.32:47In the past, as a customer, I used, we used to have lucid charts of our technical technical diagrams of our application flows. When I was creating applications within app D I used those to kind of separate my tiers in the app in my app D app.32:57Applications. I just want to know from the panelists, is there any suggestions on how to to group?33:16These tiers and applica appdynamic applications to make sure that they get that full end to end view once they start monitoring their applications.33:22So is there any special grouping or any way, anything you all use or you know going to developers or to the product managers to get an understanding of which, what's the best way to group applications to get that full end to end view of what's going on?33:38So a lot of times that there to me I really nowadays categorize, I think of application mapping is really kind of two two different there there's kind of two main groups are gonna fall into.33:57And, and one's gonna be what I'd call your traditional infrastructure, you know, you may have, I don't like the term monolithic anymore cause that might not be the case, but a lot of times it's gonna be you know standard VM infrastructure as the back back end, right? And then you'll have, you know, things that are.34:13Deployed in containers. For my things that are deployed on VMs, more or less.34:33Usually the architecture diagrams that I get from the client, are, are for while they don't define app dynamics, app and tier strategy, they usually fit well into how appd would group things, right? Usually.34:39An infrastructure map is going to have your web servers in a separate box a little bit and those would represent one tier and then you'll have your app servers and they maybe they may have different roles. They may, do they all perform the same function or do they have different roles? I might give those names.34:58Is that represent what, what they functionally do. And those could have multiple nodes in each of those tiers. For Kubernetes though, things change a little bit because I find a lot of my.35:19Orgs will use like like the just straight mapping to a name space, like an application becomes a name space, in Kubernetes and and a lot of times to that works really well. The caveat there is you don't want a hundred apps that all.35:31All talk to a hundred different apps, right? Like one, you don't want one tier.35:51That literally doesn't talk to anything in an app other than another application. Those, if you have lots of those where you just have one tier and it doesn't, it really doesn't communicate to anything, those, you know.35:57Can can hinder your your visibility a little bit, but I've I've found namespace in the Kubernetes side for a lot of my clients ends up being like something that can be standardized on and auto instrumented that way in the future.36:16Great Point so you always want to find where that sweet spot, you know, adding on to what Drew said and what Calvin was, was going over with you guys, you know, you always want to find that sweet spot of having end to end visibility. So you want to have, you know, your app.36:33Application in a way that makes sense transactions coming through it, you know, business transactions are a good way to define and, you know, follow that flow map.36:52But you also wanna make sure that you have your M out front, you're connecting the right data from, from it and your business transaction analytics to have what you need to make whatever business context decisions you're making as well you want to show the, you know, the.37:02Database and hardware and everything. But I also want to cover, you know, a little bit on the custom, you know, sometimes you have an extension or sometimes you have a backend call that is not well defined in your flow map and you need to also be able to, you know, it's.37:21Not so much the application modeling, but making sure that when you're looking at the flow map, it makes sense that you also are looking at something else that's very important. There's gonna be situations that, you know, hey, maybe there's this other service, it's a black box. It's not something that you can add an agent to.37:40But you want to really call out attention to it and be able to monitor and create health rules and dashboards and everything to it, so you create a custom back end or sometimes you use an extension to grab those and you, you assign that to a custom tier. There's extra little, little things that you can do.37:59And it really helps, you know, bring the whole application together and it becomes a larger part of your strategy. That's good info. Let's move along to deployment planning and overview. Let's discuss the deployment.38:19Planning phase, a successful implementation relies on solid foundations, starting with a thorough assessment of your network requirements and firewalls configurations to ensure secure and seamless agent connectivity. Next you must define your rollout strategy.38:38Whether you choose a manual approach, scripted installation or fully automated deployments consistency is key to a smooth transition. It is also critical to identify your monitoring team and key stakeholders early on to ensure everyone's aligned on the project goals and responsibilities.38:56Finally, we focus on maintaining a long term operational feasibility.39:17This involves pro proactively plan planning your learning thresholding, and health rules, but also ensuring your custom metrics are retained during upgrades. By mapping out these report reporting strategies before you go live, you ensure that you're, you maintain.39:23Full visibility into your application performance and can catch potential issues before that they impact your business outcome. So parents, what are the most effective process or checks your team.39:42In the past used to ensure alerting threshold health rules and custom metrics don't get lost or misconfigured during the upgrade during the upgrade process. I will tell you, example that I've got.39:57Bitten up well happened to me, and that's with custom extensions doing upgrade of machine agents and we had several custom extensions and we couldn't figure out what happened. You have to make sure you copy the custom extension directory over which all your custom extensions follow.40:15In the machine agent up under the monitor's directory, make sure that monitor directory without the customer extensions is copied over to your.40:34New machine agent, otherwise you would lose those custom metrics. That's just a gotcha that happened to me in the past. We did upgrade of the machine agent and couldn't figure out what happened to our custom.40:45Metrics. Do y'all have any kind of example of rollouts that, you know, you had some gotchas that you'd like to mention? There are, you know, in addition to custom extensions, there's also, you know, some.41:00The Customization you can do to say like a java agent. Say that you're hooking up a Java agent to an application where you're only able to get pojos. There's some translation that you can add to the agent.41:20And say, hey, you know, we know that you don't normally talk to this product. In the example that, that I was running, it was, API gateway and you know they were, you know, we, it was a black box, but we were able to see the pojos and we wanted.41:35The servelets which come out a lot better as.41:55Business transactions. So we had to not only account for when we deployed our own agents, we also had to account for when there was an upgrade on their end. So, if you have any sort of customization, which a lot of people love about appdynamics because it's so versatile, you also have to account for, you know.41:58What things are being upgraded, not only the agents but also the application itself.42:18And, and I I'll just say, you know, for, from my standpoint, I what's worked well for me as, is having standard health rules. So I, that doesn't mean the numbers that I have in the health rules aren't gonna be different per application, but I standardize on a.42:27A framework for alerting that might be availability, high level app, BT, and critical backend alerting, and and all of that standard is built in a way that assumes the data's reporting.42:48Right, and so you I have intelligence in the health rules for null values, right? To me, I should never have a null value unless I know that metric returns null values, right? And so it's a matter for me of knowing what the metric type.43:08Deep is how the metric reports.43:27And then having a set of standards that accounts for that metric always reporting, right? If you're some red flags that would, that I would see if availability alerting is not working in your organization, e.g..43:30If you can't trust that, it's very hard to then trust that you're not gonna have a metric, a custom metric stop reporting. And so a good example, you know, I'll give you guys a real world example.net windows. Dot net IIS has a setting called idle timeout.43:49It is a default app pool setting that's set to 20 min, ok? When you install IIS, the IIS role, Idle timeout on your app pool is 20 min. That means after 20 min of inactivity, NO traffic to that IIS worker process.44:09IISS is going to spin it down. Now, in practice, when that goes down, you can NO longer report on availability for that worker cause it's down until it spins back up. And so there are strategies of like turning that setting to zero.44:28Turning it from 20 to default to zero means the IIS worker process will never spin down due to inactivity and thus will always be up. And so if you have flapping of your availability reporting.44:49Or your availability metric in windows. The problem isn't usually app D it's usually a setting that might be spinning things down. Knowing all of that enables me to ensure I can trust 1st off the availability alert. If I can't trust that.45:04It's tough to trust anything else from the tool. You know, standard up down is a big one to get right. I would say that then enables, you know, you you're gonna know when an MQ extension stops reporting metrics to a critical queue.45:21Based on NO values going to that queue when they should be expected. And so we can build that intelligence into it. I do, I I tend to build the intelligence on the health rule side with the intent that I expect the metric is going to be reporting.45:40And if it's not a metric that reports regularly, I need to have a strategy that's slightly different. I need to use metrics that report regularly to be my flag for a problem.46:01When they don't all of a sudden report anymore. So Chris, I have a question for you. I'm looking at this slide and I'm thinking about documentation in the center of S excellence and being able to put this these.46:16Those these four items into a document and maybe store them in a si system of excellence for future reference. For instance, the rollout strategy so you can understand later on if just say you're a new employee and you want to go out and see.46:35See the rollout strategy and how everything is rolled out. Would, would you consider this something that you would put in the center document and put in the center of excellence?46:55Absolutely, absolutely great and great call out. Yeah, you definitely want to make sure that you have a, if you have a center of excellence or if you have a wiki or whatever you guys want to call it, you know, a place where you have all of your rollout strategy, your templates.47:05You know, you don't want to go reinvent the wheel every time you have a deployment.47:24If you have an application that's similar in nature or function from one to another, then you could reuse the deployment strategy. You could reuse the health rules, you can reuse certain things, and then like what Drew was calling out before of, you know, non prod to prod and testing.47:29Along the way, you also want to make sure that you get those, those kind of.47:49Points documented as well. Best practices become very important and it helps you start gaining that velocity where you're deploying quickly and and efficiently and not having to spend so much time where the 1st few deployments, you might spend a lot of time doing it, but at the at.47:55As you go, you get.48:14Better and better and you have documented flows and then you just figure out which flow the next deployment's gonna be in. So yeah, NO, I definitely agree with that and documenting in center of excellence is a great way to do that. Okay. So let's look at troubleshooting. When troubleshooting the agent deployment.48:16At 1st check the JVM arguments verify all directory's path points to the correct agent directory. Next inspect the controller info.XML file for the the settings making sure the controller URL, account name and account key is set to correct to the correct controller. I also confirm that.48:36Agent CA search.48:56File includes the, the proper service certificates. If none are configured, the agent uses the JRE defaults trust store. For IPM Webser environments, note that the JRE home is inside the webser installation directory requiring addition.48:59Additional configuration. Enable agent debug logging by adjusting the agent log settings to capture detailed diagnostics. Does anyone on the panel have any other troubleshooting tips that they've used in the past when having issues with agent deployment?49:18I know agent deployment is pretty straightforward. It's, I'll just add one. I think it's a good thing to know, and of course the agent needs to be communicating for this to work. So caveat here, right? This.49:38This doesn't help you when it's just not connecting. You know, make sure you, you use telnet, right? Try to hit your ports, verify those things if, if the things If connectivity's not working and you don't know why, but I think it is a good thing to learn how to capture debug logs.49:58From the UI by using the the what is it com singularity and then setting it to debug. Do you know what I'm referring to Kelvin? Yes, I do. It's just a nice little thing when it, when it works, if it works, it's very helpful to be able to run.50:17A debug session on the fly for like 10 min.50:37And be able to grab those logs through the agent UI screen. And the good thing about making those changes, those are hot changes. There's NO restart needed. You just go in there and change it, the enabled from I think it's from false to true. I can't remember that exact.50:41Wording, but I know it's a hot change and it can go from info info login to the debug login alright so let's go into the smart agent. Now let's dive into the smart agent.51:00Now, it is designed to centralize and streamline the entire life cycle of your agent rather than managing the agent individually, you now have the unified control plane accessible via both the UI and command line giving you full oversight of the entire fleet.51:20The smart agent excels in auto discovery automatically identifying your application and deploying the appropriate agent exactly where they, they are needed. By leveraging policy based control for installation, upgrades, and configurations, you ensure that your environment remains consistent.51:38Standardized and secure. Ultimately, this shifts all the, all, this shifts away from manual repetitive tasks, not only save your team significant time but also strengthen your overall compliance posture by ensuring every agent is deployed and configured according.51:58Into your organization's best practices.52:18Here's a walkthrough of the smart agent deployment workflow, which is designed to remove the complexity from the agent installation. The process began by installing the smart agent directly onto the host machine. Once actively the smart agent performs an intelligent scan of the environment to identify.52:25All running processes rather than require manual intervention for each application, the system references your predefined policies to automatically determine which agent are required and how they should be configured for your specific environment.52:45Finally, once the application is restarted, the agent are seamlessly also attached and monitoring data begins to flow directly to the controller. This automated workflow not only accelerates your time to value, but it also ensure that your entire infrastructure is monitored consistently and.53:03Accurately without the risk of manual configuration errors.53:22So now let's look at the demo in the demo. 1st let's go into the actual controller and look at the agent map and look at agent management to get the agent management. I'm gonna go up here to my main.53:30Me you are a controller menu and I'm gonna click on agent management, which takes me to the agent management screen. And let's just look at some of the tabs up on the agent management before we actually go into the installation of the smart agent. So the 1st tab is agents and you see the 1st four items app agent.53:50Agent machine agent, smart agent, and database agent. All of these can be monitored and controlled by the actual smart agent. Your analytic agent.54:10And network agent, you still have to come up with a deployment process for those. I'm looking at those to eventually be put in smart agent just because of the way they're configured. I'm I would think that those would eventually be added into the smart agent. Also with the analytic agent.54:22You can turn it on through the machine agent as an extension, so that could be another way you can use for deployment of the analytic agent. Then we have the cluster agent at the end, you know, the cluster agent is used to monitor items such as Kubernetes objects like pods, NO.54:42Namespaces and deployment. This installation is done through using Ham charts or standard YAML manifests. This allows you to integrate the agent installation directly into the CICD pipeline ensuring that monitoring is baked into the cluster configuration.55:02From the start. So I don't foresee the cluster agent being put in the smart agent since we can.55:22Easily bake it into the actual cluster configuration. Next we have the task in progress. When you load a agent through smart agent, you can go and task in progress and you can actually see whether it's been triggered and when it's completed.55:30Once it's complete, you have history.55:50On whether it was successful or failed, next is the config management. Config management would be used with, your.net and this allows you to manage your config.xml file from the actual controller. So if I click.55:54On pet clinic demo instrumentation and click on view config, you'll see that I've imported this config.XML and now I can drag and drop a server over into, this one, this.56:13Config.xml file and it'll copy it over and all I have to do is restart the application. It'll tell you to wait 1 min once you drag and drop, so as you can see, it's just a simple drag and drop. It'll tell you to wait 1 min while it copy copy the file over and once that 1 min over, you can go over to.56:33Your host server into an IISS reset to pick up the new configuration.56:53Processes is when you load the smart agent, the smart agent can, if you have it configured, it'll go out and search for running processes that can actually be used to install the agent. So it picks up those processes and you can see which processes you can install another.57:00App agent on.57:19Then we have deployment groups. In deployment groups, you can set up groups of app agents or hosts that you want to deploy the agent onto and schedule a time for it to be deployed. So if I hit create new.57:21I can name the deployment group, I can schedule the deployment, then I can go next, I can pick which agent I want to deploy. Then I can pick this, the host that I want to deploy the agents onto. So.57:39If you want to set your deployment on a schedule, you can do it using deployment groups.57:59So let's go into installing the agent. The 1st agent I'm gonna do an install on the smart agent install on is the Linux Java I mean, it's the is a Linux box, I'm sorry. So let's do a LS, you'll see that I have the smart agent already downloaded.58:09I'm gonna run the command to unzip the smart agent. I've, I have the command in my documentation, so I'm just gonna paste it over and hit enter.58:29And before I go any further I'm just gonna show you inside of the smart agent files what needs to be configured. If I go into the config.in I file, you mainly need to configure this 1st section up here and it's the same informations that's in the controller.58:47So that XML file for a regular app agent, you'll do the controller URL, the port.59:07The account access key, the account name and if you're on, if you see you have SSL set to true. I've already configured one of these files, so I'm just gonna copy it over.59:16And now I'm gonna run the command to start smart agent and it gives me the smart agent ID.59:38Now I'm gonna go back into my controller, which we remember D14T So when I go back into my controller, I click on agent, go to smart agent.59:59And look at here we have D14T now reporting. So this small agent is installing Linux. You can use the system CTL, which is the CLI command line interface, and you can actually set up host files so you can roll this out remotely.01:00:14So you don't have to touch each box to do this. I'm just doing it manually since this is just a quick demo, but this is how you would install it on a Linux box. The next one is I'm gonna go to a.01:00:34Windows box and do the same installation on the Windows box. So here's my zip file. I'm gonna unzip it.01:00:46All right, I'm gonna copy this pre configured config.INNI file. It's the same file that we just seen in Linux, but I have a pre configured one. So I'm gonna copy it over here paste replace, and now I'm gonna go into a.01:01:06Windows prompt, and I'm gonna go into this directory and now if I run the following command.01:01:26Oh. You see that the app dynamic smart agent started and I can go into the directory and I see an IDTXT. I'm gonna edit that file.01:01:52And it shows me the smart agent ID. So now I should be able to go into the smart agent ID. I'm into the controller and see that it's now reporting. So now we have the smart agent reporting for a.01:02:08Linux box and a windows box. Let's go into loan agents for both of them. So let's start off with the Linux box. I'm just gonna do install agent. I'm gonna do the machine agent 1st.01:02:26I'm gonna select host at span, I believe this was the Linux box. Shifted over, apply. And you can see right here you can actually change the location of your machine agent where you want it to run, but I'm gonna.01:02:45Leave it as the default, hit next. I'm gonna turn on sim and since this is let me check. Is this the what was the windows box?01:03:05Okay, for a second I got mixed up, I thought this may have been the Windows box. So I'm just gonna click send next and submit. And now I'm gonna do the same thing with the and you see it puts me in.01:03:25Task in progress and it shows me that it's triggered. So let's do now load the java agent, same thing, install java agent, and I'm doing them one at a time, but if you was doing a fleet, you can choose as many smart smart agents as you want.01:03:43Over here, but since I'm just doing this as a demo, I'm only picking one, and now I'm gonna hit apply. And now I can change the application tier and note name.01:04:03To change the application tiers a little tricky because you click on it, you don't see it, it doesn't highlight it, but you actually click on it and then click edit edit. I'm gonna put this into a demo app, created called clinic.01:04:21Dash demo. Hit enter to save it. I'm gonna change the tier name, I'm just gonna say EIS.01:04:39Dash test and I'll leave the note name the same. Just hit next. And here you can I just want to mention this too. You can actually, I'm using the download app D portal, but you can create your own portal. You can create your, just say if you have to scan the.01:04:55The agent before you can use it, do a security some kind of security scan and you want to create your own repository, you can create your own local.01:05:16Repository that you could put in so you don't have to pull down from the app D portal if you don't want to. So since I'm using all just defaults, I don't have to make any changes here. I can just click next.01:05:27And submit and now you see that the Java agent has been submitted. The same process goes, go with the Windows agent now since we already have the agent reporting, we can just click on.01:05:43Install agent, machine agent, next, and we can pick the actual window box now right here, and I'm going through this pretty fast since it's pretty much the same process.01:06:03And I'm gonna hit next and I'm gonna sim enabled and I'm gonna do.net enabled since this is a net.net application that's running on this server. I'm gonna click next.01:06:25And submit and now you see the Java machine agent has completed so it disappeared, but the Windows machine agent just started up. Let's do the same thing for.net.01:06:42And you notice when I do that net, the only boxes that shows up are the boxes, the window boxes. So it separates the window boxes away from the Java boxes. I'm gonna select it.01:07:01Hit apply done. I'm gonna just leave everything the same. Hit next, and I'm gonna next and submit. So the.net boxes, as I mentioned before, you can use with.net.01:07:20Net config management. So when you're using config management, your actual boxes would show down here, but just say if you're just managing your config XML files, I'm gonna just show you in windows. If I go to.01:07:40This server right here, I believe it is, and I go into Windows Explorer, and go into C drive, program data.01:07:59AppDynamics, and I go to the config and I take a look at the config.XML, you'll notice that it's going to pick clinic KLM right now.01:08:19Once I make the change into the controller to change it from KLM, and I drag it down to a pet clinic demo.01:08:39It tells me here, wait a wait a minute before I restart my IS, but it's copying the new config.XML over to the one to the windows box. So if I go back into this box.01:08:56And let's see 48. It may have already copied. Let's look. Now it's showing pet clinic demo. So it copied the config.xml over. I can do a is reset and it should start reporting over into the new application.01:09:14And that's mentioned in the config.xml.01:09:34So now let's look at, we've installed the application smart agent. We also did install install Java agents and.net agents. Let's look into what you would do.01:09:39To do an upgrade of the agent within the controller.01:09:57Using the smart agent, which is really simple and very convenient. You just go into your, let's do machine agent. So I can go into machine agent. I could search for.01:10:01Let's search, do a filter for manage service that's managed by the smart agent, so I'm just gonna click manage apply. Done. Now, when you do upgrade, you have to do by architecture. So if I want.01:10:21And upgrade all of these Linux boxes. I could easily look at the ones that's showing update available and click on those.01:10:38And now I can click upgrade. I can do a custom upgrade if I want to make changes to like the controller info.XML, I can just do a default upgrade which keep everything in place. So I'm just gonna click next.01:10:52And submit. And now if I look up on the task in progress, I see that there's a new task in progress that's working on three hosts to upgrade those hosts. That's pretty much.01:11:10How the smart agent works, and the different processes you can run through the smart agent, other items that you can do, if you do an upgrade and you wanted to do a rollback, you can go in here and you can do.01:11:27You can do your rollbacks, you can do book reset, book disabled or book enabled, so there's other features that you can do in agent management as far as the smart agent, our agents.01:11:45That's been reported into this control into the controller. So Chris, that's pretty much basically what I want to show as far as the agent smart agent and its functionality, being able to actually install the smart agent, install install the application agent.01:12:03A machine agent along with doing upgrades. One I did forget one thing. I also want to mention that it we talked about how the smart agent, use LD preload, so this for instance, if I look at this smart agent here.01:12:23I'm gonna go, I believe to this one. If I go in here, I'm able to go into the smart agent directory, well.01:12:43Let's go into three. There is a file up on the lib called LD preload JSON. This file allows the smart agent to auto attach to your processes.01:13:00This one coming up. So when it auto attached you see the different type of processes that it can attach to, for instance, Java.01:13:20Tomcat, glassfish. And you notice that you probably say, well, if it automatically attached to these processes, the machine agent is a is a Java process why it doesn't match.01:13:31Attached to this. So if I search for machine agent, you'll see that it's defined within this file, but you'll also notice that instrumentation is set to false where all the rest of them are set to true. This is so.01:13:47Is how the smart agent know which process to attach to. So if I go into the actual at this application, which is just a little small Java application called pet clinic. I want to make sure it's not wrong.01:14:03Running.01:14:23Which it is, so I'm gonna run a stop to stop it and I won't want a VI to start line so you can see the actual start line does not have the Java application JVM arguments that you usually have to add into a java.01:14:26Application for the agent to start up. Since it's using LD preload, it knows.01:14:47To attach to this agent. So if I start it back up, you'll see the 1st thing it does now is it attaches the Java agent to the Java process and then it starts the application. Another thing is right after you load the smart agent and you try LD.01:14:55Preload, you have to exit out and come come back in so it can resource the environment variables in order so it can pick up LD preload. But that's what I wanted to cover today. I'm gonna hand it back over to Chris for Q and A.01:15:15Great job, Alvin, that was it was really informative, it's good to see those things done. You get a feel for it. So some call outs that I wanted to bring up, you know, we brought up the, the, well, I'm seeing some questions come in.01:15:38About the cluster agent and dealing with containerized environments. Now, you know, from what I understand, the best way to deploy those things are either through the existing pipelines that, you know, are already there and existing.01:15:55But you also have the option of auto instrumentation through the cluster agent. So the say that you have a bunch of you know.01:16:15Pods in a name space that are, you know, being monitored by that cluster agent already for more infrastructure pod level metrics. You also want to be able to deploy those agents quickly. You can set up auto instrumentation through that cluster agent if there's not already an existing pipeline.01:16:23Involved. So that was one of the, some of the comments that I saw come in.01:16:43Everything else was, you know, I I felt that you, you delivered that really comprehensively, gave a lot of good examples, showed the ins and out both from the server side as well, the UI. So I don't see that there's any other questions coming in. If any questions come in before we.01:16:50We End this session, we will go ahead and revert back, but for timekeeping purposes, let's go ahead and continue.01:17:10And both in the chat as well in the description of the video, we will have some extra information on both EOM deployment, MRM deployment as well, things related to the like we were talking about with the cluster agent and the auto deployment of your.01:17:22APM on containerized environments. So additional resources.01:17:43That are available. You have the Cisco and Splunk communities that have a lot of good information as not only in regards to, you know, Splunk core or Cisco's you know networking switches, you know, if there is any sort of integration or anything as it relates to appdynamics, there's a lot of.01:17:48Good information in both of those as well. Appdynamics is a part of the Splunk.01:18:08Product line offered by Cisco, and then in addition to, to those community sites, and we make a lot of videos in addition to just these EIS sessions. We have short videos, longer videos, discussion videos, ones that are just, you know, getting you from point A to point B almost.01:18:14It's like a follow along demo.01:18:34A lot of different options for you there on YouTube, so definitely check this out at YouTube.com/at cisco observability and we're right now we have a lot of app dynamics stuff, but we also are going to be ramping up not only on these EIS sessions but also on the YouTube video.01:18:36Creation for Splunk Ollie and ITSI things that relate to you know connecting all of your observability into one place. And then there's also the u.cisco.com, which it has a.01:18:56You know, a trove of information for Splunk appdynamics, getting certified. You don't want to, you know, just say you know things, you want to get certified on them. You want to be able to show on your resume, have the cool little badges in your emails.01:19:13Etc. So whether you're a, you know, a DIY person or a person working for a major company, both of those are gonna be, you know, very important. One also to mention is School of Splunk, which we don't have on here on this slide yet in future sessions. I'll make sure that it's on there. So check out.01:19:28Tools Plunk as well the U.cisco.com for the Cisco university. And let's go ahead and start wrapping up, guys.01:19:48Thank you all so much for joining. It's, it's you guys that make this a success, it keeps us going, it makes us want to keep making good content for you all to follow along and be successful as we've had success with the products that we talk about and believe very very highly in as.01:20:02So if you have any feedback for us, you know, a lot of the changes that we made, a lot of the content we have been creating and continue to do so, are based off of your guys feedback from not only Slido but chats that you, you provide in here.01:20:22As well, you know, comments that you make on our YouTube page under the videos, so feel free to make a, you know, let your voice be heard if you want, if you want it to be more anonymous, just scan the QR code or the link in the password, you know, it's up to you if you want to say who you are or not. But.01:20:39Otherwise, thank you all for joining and we'll see you all next time. The next one's gonna be on 11 June, and we'll we'll see you then.01:20:59