Observability Onboarding: Expanding the Load Out

Welcome back to another Observability Expert insight series. I'm your host, Christopher Banano, and we are going into our 3rd episode of.00:10Of our observability onboarding speed run, unlocking easy mode. For those of you that grew up playing video games, you understand probably, you know, you know, cheat codes or just doing things enough times, you end up finding processes to fix and improve along.00:20You know along the way and make sure that you can actually get through the content faster, where it normally takes a lot of people a long time to do, and you have perfected that challenge so much that it becomes a speed run. Same thing when you're dealing with observability, you know, there are the, we spent the 1st.00:40Two episodes going over where you're, you know, looking at what actually is the goal at hand and how to achieve it. And then you're also trying to get into, you know, how is the best way to deploy your agents. So once you figure out everything that you need to monitor and.01:00The monitoring solutions that you're going to apply to those things, how do you deploy them? How do you make it efficient, how do you make it repeatable? And then, you know, beyond that, it leads into today's topic, which is expanding the load out. So what comes out of the box is often great for a lot of people, but at the same time.01:19There's going to be gaps and there's going to be extra things that you guys are gonna wanna get to complete those flow maps, to complete those.01:39The business context analytics that you're gonna want to be able to, you know, dive into and make business decisions from, all of that ties in with today's topic, and I have a full panel. I already introduced myself, my friends and colleagues that joined us today are gonna be Eugene Breen Calvin Miller and.01:48Little Harper, we have all.02:08Been in the field of observability for a long time with various aspects of it and now we're here as customer success engineers trying to make sure that you all have the same level of success that we experience in our multiple experiences outside of our current role, and that we help current customers with today.02:10Okay.02:30Let's go ahead and jump into some cleanup items or housekeeping items. So we're going to have some assumptions as you being our audience. We are going to assume that you have a pretty good understanding of app dynamics conceptually. You already have a control.02:31Or and you guys are kind of starting your journey with appdynamics. We will have the mics muted for the entire session. It's better for security as well. You know, it makes it go a little bit faster. This will be recorded, it's not mentioned on here, but this will be.02:50Recorded and posted up on our YouTube page, which I will link down in the chat. But if you have any questions, definitely go down into the bottom right, press the three dots and hit the Q and A button there. If you put your question in the Q and A, it makes it persist through, you know, multiple, with other.03:10The people have questions as well, it kind of preserves it in place so that we can respond and everybody can see the question and answer in line versus if you have any feedback like my audio's cutting out or the text is too small, things like that, let us know in the chat so we can handle that in real time. And then obviously this information.03:30Is made for made for you guys to be successful, but it is owned by Cisco and is not meant to be taken and, and, you know, marketed as your own. But like I said, it's for you guys. It's going to be on YouTube and we are happy to be here presenting it to you today, helping you guys be successful.03:51Tools the products that we help support. Now, to kick us off into the agenda and going into some of the good content that we have today, let's go ahead and hand off to Eugene.04:10Thanks very much. So today's agenda is going to be light enough, in terms of the number of items to get through. However, we will still be taking our time getting through them all the same with this.04:29Session today being an hour.04:47We want to make sure that we can cover everything equally, so we're gonna start with extending app dynamics. Chris, could you lead into the next page, please? Thank you. Okay, so why extend app dynamics? Well, there's a lot of text on the screen here that a lot of you.04:49Can read in your own time, but in the background, why, why do we want to extend that dynamics?05:10It's it's not because of default monitoring being weak, it's because every customer's environment and their their processes, business priorities are all different. So the default view gives application performance data, but extensions and further configurations allow us to enrich.05:17That data with context that matters. So e.g., data collection and information points that we're going to be looking at today, which I'm gonna bring you guys through next, that's an example of how to extend that dynamics and there's further options as well with integrations, and this is what will help our teams.05:37And your teams move from simply seeing an issue to understanding who or what is affected and what action should be taken. As well as that there's a number of areas around app dynamics. If you're a 1st time user and you navigate around the controller, where you will see out of the box configuration, ok? Where you have that with about.05:56Configuration. That's our, that's our standard as you start up with appdynamics, but there's so many areas that you can extend extend to and build upon. And we're gonna cover a lot of those areas today. Next slide please Chris.06:16Thank you. So I mentioned data collectors and information points on the last slide when we were covering the agenda and covering extending app dynamics and these are two pretty good analogies. You can think of data collectors. It's like putting a small notepad beside a.06:30Check out tail and saying whenever a customer transaction happens here.06:49Write down this extra detail, customer name, account type, basket value, order ID, your cart value. Its purpose is to collect business context data from a transaction snapshot, and HTTP request data collectors, you also get that on top of with your, your.06:54Method invocation, data collectors and appdynamics. Two of them are different. I'll show you guys examples in a demo, how, how both of them work and how they operate, but there are differences between them. So e.g., we have HTTP request data collectors.07:14They might be in the form of, they let AppDynamics capture values that are already present in incoming HTTP requests, right? Whereas for method invocation, data collectors, they would operate a bit different. The value is inside, let's say jabber.07:32Our.net code as a method parameter return value or object property. Now, information points that you see on the right, there's there's a small bit of overlap between data collectors and information points. However, information points is more like putting the counter above a door and saying.07:52Every time someone walks through the store counter.08:12And also tell me how long this spend passing through. So the purpose of it is to monitor specific method or code path over time, even if it's not the main business transaction entry points. Information points throughout my time using AppDynamics has been really well used by.08:15Your your developers, right? Very useful to, to let you know join any tests and how many times specific methods is being hit and so forth. So if you have a bit of time and you have access to your dev controller in the background, feel free to play around with information points. Now I.08:35I am going to share my screen and bring you guys through a quick demo of both data collectors and information points. So I suppose the 1st area that we can have a look into is, information points. Thankfully there's a.08:55Application here that one of my colleagues was working on, so that's gonna be a useful example for looking at information points. So firstly, where is information points located within your controller? If you come into your business application, you will see it down here under more, right? Here's your information points. How can we set up information.09:15Points. Let's take a look. We can select information points here and we can click on add, ok? You see there's various different data that we have to put in here. The naming is important to give it a bit of context and let people know what the information point is set up on. You need the.09:35Fully qualified class and method, and then you can also extend information points out of it to get custom metrics, which is a really cool option that we have. So when we're setting up information points, there's again a number of ways we can go about it, ok? So if I duplicate.09:53Get this guy here and I go into some of the business transactions when this.10:12Loads. If I go into a business transaction and I go into a transaction snapshot, if I just select one of these guys here and drill down hopefully it has a bit of data for me. It doesn't, let me go to another one that has a bit more to it. Here we go. I'll get this guy here.10:19Now if I go and drill down, I should see classes and methods, ok? I might be able to right click on this guy and go configure instrumentation for this class and method. And you see here I get options for information points, right? I also get options for method invocation data collector, so I won't have to overlap on that later, ok?10:38That's one way we can set it up.10:58I'll close down this guy and go back into information points. So let's take a little look at information points and how this is set up. So here we go, we have get credit score, right? It's letting us know exactly what we want to get. We can see here it's getting calls, calls per minute. There's a low number of metrics that you get.11:00Information points and here on the right has shown us custom metrics. So this is a good example to show us the credit score.11:20And you can see some of the metrics that it's coming out with calls per minute. However, take a look at this guy as well. This has given us the credit score of every time that someone is coming in and hitting this method in the code, it's it's returning back a credit score and we're capturing that and we're showing it here in the.11:28In the in the way of a time series graph, ok? So we can see the performance of this over time.11:48Now, let's take a look at it and how it's configured, so you can see get credit score, good meaningful name. We have the class method, and here's where we input some more information to get the actual credit score itself.11:54This is the application up on GitHub and you can see here, get credit score, right? Return credit score. So that's easy how it is configured here. In my experience when I'm setting up.12:10Information points and data collectors, doing it with someone that has visibility of the code is always the the best way forward. However, as you've seen earlier, there are options to try and play around with it yourself, but working with a developer, developer definitely helps here, ok, if you have any questions on it or.12:25You have any difficulty reading code. Again, AppDynamics is a supplemental tool. We're not all experts, ok? So we can lean into people to help us get forward. Now, I'm gonna go on to one more thing before we progress the data collectors, and that is the metric browser. That's another place where you can see the performance.12:45Of your information points over time in a time series graph. So we see here the get credit score, right? We have the credit score here. We see the performance of overtime or you can see the performance of the calls per minute, ok, to see how it actually performs. Again, good for your developers and good for people if you're wanting to know.13:05The performance of a key area being hit in your application, right?13:25Data collectors then. So data collectors, we were talking about two, two different areas in, data collectors, there's your HTTP and your method invocation. We're gonna have a look at your method invocation data collectors today. As you can see, it's located in configuration and.13:30Data collectors, right? So data collectors again give us supplemental data and and context to issues. When let's say we get a high response time on the transaction and we dive down into the transaction snapshot, right? We can see the high execution time.13:50Against a particular method, right? However, we, we don't know who or what that high response time is affecting, right? And that's what data collectors do. It gives you context, and this is where you set it up. You can see all the meaningful names here, how they're set up, which is really good. You can see options here with data collector.14:10So apply to new transactions, any new any transactions that's discovered by appdynamics, again, will take on whatever way you have set up this.14:30Data collector. You can enable it for snapshots, so as it was saying, when you're troubleshooting, you can create the snapshots and you can see a bit more context there, but you can also send it into analytics, and that helps you slice and dice things a bit more, create custom metrics, and kind of give a more business context view.14:40Against the performance view, which is really cool and we probably show something like that at a later date. Let's have a look at how it's set up. It's pretty similar to information points in a number of ways. So you see here we have the.15:00Class and method, and then we have this guy here, ok? Loan ID, so that's gonna specify what we want to pull out. We have credit score here as well. So this guy here, get credit score, the class and method and then we see this guy here, ok, it's a returned credit score.15:15And we can have a look at the code as well up here. I think I have an example already saved, so we can see the actual where is it, this guy here, instrumentation. Let me close this. Okay, so we can see here the class of method right AppDynamics loan model.15:35Customer, close this. All right. Loan model customer, ok? So that's what we're looking at now. We go in here and we see get credit score, have a look. We can search for it, which is quite useful, so we go down here again, this is where credit score has been returned.15:55Okay, so this is going to give credit score back to us any time that we go into a snapshot. We should be able to get the the credit score when we view data collectors. So let's have a little look at that. We go on here to our business transactions.16:15In this application, we'll see if this is the data collector that's returning this, and there we go, you see the credit score. So I'm gonna go back on that a little bit, ok? So we go to business transactions, right? We go to our customer login, and this is the path that a lot of us will take when we're analyzing business transactions.16:32For performance issues or just to be curious how they are performing. So if you double click in here, this is actually one that's performing very slow, we'll go to data collectors. So not only do we get to see, ok, what, why is this.16:52Business transaction performing poorly, but we'll also see who has been affected, the credit score that they have here. We'll be able to narrow down on individual IDs if we need to do searches in the background, all, all very useful information again supplemental. It's not only asking or showing that there's a high response time, but who is it affecting.17:07So data collectors, short.17:27Sure story long or sorry, long story short. It is something that is something that you should be using if you're not using now. Okay Chris I'm gonna stop sharing my screen and hand it on over to you. All right, so some more or additional options to.17:33Getting more out of the product and getting more information into the tool for you guys to be able to utilize as a business, as a developer, as a product support person, NO matter what hat you wear, it's gonna be important for you to get the right context and the right information that you need. Now in addition to the data.17:51And information points, you also have an option from the agent level, which are custom SDKs. Custom SDKs are translators, if you will. It could be that you want to very similar to what Eugene showed where you're, you're wanting to use the.18:12Class and method, you know, to, to be able to make sure that those come up as their own.18:31Data point, you can do that from the agent side using these custom SDKs, but then you can also expand on that. So say that there is a product that is, you know, a propriet that has proprietary code, something like, you know, the example I like to use is like an API gateway.18:38You know, if it's a product that, you know, they're, they have really good documentation. They have all of their, their information written out on their website or you can get it from their account teams or whatever, and then you're able to deduce from that, hey, you know, these classes and methods are not.18:57Standard, they're not things that aren't the agent normally would catch out of the box. So then you can and you you can indirectly start translating those and creating data points instead of, you know, just picking up standard pojo items, you know, of whatever you can see, you can specify.19:17By.19:37And really, and really make your monitoring experience much better with our APM tools at where you can start to get servelets instead. And you know, it's gonna be kind of hard to show an example of that, but we will look at an example script together to kind of understand that more.19:38And, but before we do that, let's go ahead and try to get a little bit more into some other options here. So another option is say that you have some serverless services that are running in, you know, a container.19:58Some lambdas and you need to be able to monitor those. Obviously our APM tools trying to reach out to them, you know, probably won't have a connection. They don't have Java installed, but you're able to see the traffic coming in and out and you're able to build telemetry off of that. And with the, you know, the, the new.20:18That standard for monitoring in a lot of cases, open telemetry becomes very applicable and desirable within all the organizations. And AppDynamics sees this and and adapted as well. Not only do you have the out of the box java, node.js.net.20:38You know Python, whatever you are normally using, go. We have agents for each one of those things, but then if they're running in a lambda, you might not have a supported environment for that. So to capture that, you're able to now use open telemetry functions to bring that into the controller and kind of complete that thought, complete that.20:58Flow map. And you have multiple ways of doing that. There are.21:18Dual agents where, you know, you, and not only are you able to, you know, go out and start getting open telemetry information on things that you're already getting from a dedicated agent, as well the, the lambdas. You also are a.21:25Able to point it, you know, there's a lot of our customers right now going from AppDynamics over to Splunk Ollie that has a little bit more support for native cloud.21:43Or say that you guys have another tool that you're running in parallel to app dynamics. You know, it happens. A lot of people have or a lot of companies we, we work with have multiple products looking at similar things so that they can assess everything, you know, all, all together and they can figure out.21:54Okay, so, our APM side is not picking this up. We see from open telemetry that there's an issue.22:14Or vice versa. You want to be able to cover all of your, your bases, you want to make sure that nothing's getting by that's not being traced or spanned or or being monitored by bite code instrumentation. It's different approaches, so that's why, that's why it's so good to have them combined.22:21And then you also have with that, like I said, you have the ability to send the open telemetry data not only into appdynamics, where you it might be a little bit more limited, you can send it out to something a little bit more polished for open telemetry like the Spark observability cloud. So, a bunch of different.22:41And options, a lot of supported types of transactions that you can get.23:01It does work with open source, open telemetry as it is a, you know, a set standard that's originated from open source origins and you know at the same time, we have some dual options where you can download it and and work it pretty quickly, especially since.23:07It's we're all under that Splunk flag. If you're using Splunk observability cloud, it makes it even easier, and we can go a little bit more into that as well. But before we go into the demo, let's go ahead and there's one more slide. So now just to kind of wrap your head around the open telemetry versus a traditional.23:27So with byte code instrumentation, there's an agent running, it's looking at the it's usually a part of the same heap or it's running some side by side as a proxy with an existing agent. And then once that, you know, anything's going in or out of that.23:48Or if it's following along, it's able to see, you know, what's actually happening in the code, and sends that information over to the controller. Now, if you have that agent and you have the dual reporting on, that agent's there, it's also sending information over to the open telemetry.24:05The collector, which sends it over to the service and then, you know, sometimes you can have it go back over to the controller, you can complete those flow maps or missing pieces that you would not be able to see with bite code instrumentation. Maybe it's not a good fit, maybe it's something that isn't supported or it's a little bit more of a.24:25Black box, it gives you more options to be able to complete those, those flows.24:45And then lastly, we're going to speak a little bit about server the server monitoring. Now with server infrastructure monitoring, also known as SIM. Once you have that enabled, you get a lot of information, as it's it is its own Java process sitting there running. It's not just a bunch of shell scripts running.24:52Top and and putting that information into the hardware section of the controller. It's actually looking at the processes. So if you have.25:12Or processes, you know, network utilization based on which port it is or there might be you know five disks that are or or volumes rather that are being monitored and they might not be something as important for you, where, you know, if you want to refine those settings.25:22Things I'm gonna show you where you'd be able to do that, so then you'd be able to, you know, break up your, say that you have processes that are being completely ignored. It grabs the top ten and your batch job that's running in the background to update whatever, whatever it is, it's not being looked.25:42Yeah, so you want to be able to look at that, you can whitelist it.26:02You can even do a regex so that the class is, is seen as a priority and it's named by itself, as like an example, if you have, you know, three, proprietary or self built self run Java processes there.26:06Under the controller, it might only show you one one Java line and then three PIDs underneath it and they just say Java. They're, it's hard to understand what it is. It's, that PID is gonna change each time. So it doesn't give you something measurable that you can create an alert on, you know, and feel good about that.26:24Outside of maybe ok maybe three processes are running as if you have two, but then you're still gonna have to figure out, well, which one's down and you have to go over that box where if you want to have an explicit.26:44Error call out or an alert rather, showing, hey, this is down. We need to get that back up right away, send it to this team who operates that, something more granular, then you can actually use a re regex renaming. I'll show you where to do that in the actual machine agent director.26:57And then volume mapping. Say that, you know, out of the box, we only capture the the 1st five that we see. And it's usually the devs. If you have you know if you're unfamiliar with this or if you have a specific drive or a network drive that.27:18Is not called out out of the box, that's not gonna be as useful for you. So you might want to specify which.27:37Volumes you want to actually be monitoring or or with your server infrastructure monitoring without, say that you have a whole bunch of drives or volumes created and attached. You don't necessarily want all of them, you want to specify pick and choose which one you want, which one's important. And the same thing goes for the.27:44Network adapters. All of this is contained in the monitoring.yaml of a machine agent. I will show you where that is and kind of where those settings are so that if you have this struggle, you'll be able to address it.28:04Now let's go ahead and start talking a little bit more about the, let's start talking a little bit more about the, you know, how you're going to start seeing things and how you can refine it. Now, 1st of all.28:20Let's go ahead and get a little bit more into, ok, let's go in order. So we were talking about the custom SDKs. This is an example of, a, a custom SDK that you would actually put in the configuration directory of your Java agent, and you.28:34You would create a script, something like this, and now you're.28:54Importing those certain calls and then you kind of help it decide, you know, what it is that it's actually looking at and how you want to be presented within the controller. Much like the data collectors, it works the same way, but it's from the agent side, but like I said, you can really build on this because it's the script and the.28:58The background and it's not a million clicks in the controller.29:18You can make it an actual translator for proprietary code of a service that you guys use, whether it be open source or something that your company pays for, and then you can even, you know, put in there specify when you want it to take snapshots, things like that, becomes a very pow.29:21Powerful tool. When I worked at a an insurance company at like the example that I gave you before, you know, the agent did a good job of trying to get what it could, and it pulled out everything that it could as pojos, which I don't know if you guys are hopefully you guys are familiar with those plain old Java calls versus.29:41Is, you know, if you have a servelet, it does a much better job. Our, our agent does a much better better job of seeing everything downstream from those calls. So if you have, if you want really rich, data streams where you're looking at the call graph, you know, and from way upstream.30:01All the way downstream, and you're able to see all the calls in between. Pojo does not always do a great job of following that along, or all the asynchronous calls, all the JDBC calls.30:21You know, to really master this, to really get the most out of it, you know, what we found when in when I was a customer was we needed these to come in as survelence. So we actually built this these data collectors essentially from the agent side to say, hey, whenever you see this, know that it's.30:34It's gonna translate to to this class and and method, and we want to be able to get this information in the controller as a servelet. We wanted the full support of snapshots, and there's really good documentation on this as well, you know, this is a topic, this is kind of a heavy topic so.30:54So.31:14You know, to spare you guys, we're going to create a separate video on this. It is something that could easily take up the whole session just showing this. So just to expose you to it, let you know it's an option both inside the controller, outside of the controller on the agent side. And then let's go ahead and start touch.31:14Thing more into the open telemetry.31:34So with the open telemetry, I'm sorry to just kind of show you that some, some documentation, but we do have really good documentation on this stuff. So I do want to show you, you know, you can walk through here on how to get started, and where you would generate your access token or access your, your access key and.31:38Etc. And then once it starts coming into the controller and you start going through these steps of getting it set up, you would be able to start seeing.31:57You would be able to start seeing those lambdas. You would be able to start seeing connections through open telemetry. And you know, like I said before, sometimes these are serverless. If you see obviously the open telemetry, these are done from a Java agent.32:12And then if it's lambda, you will see the Lambda function right here as well. But you know being able to complete this view, being able to get everything that you're actually needing to be able to monitor and understand the connect interconnectivity between all of your services, and the calls that are being made between.32:29Them Becomes very powerful NO matter what chat you're wearing. If you're looking at something like this, then you need to know as much as you could. And you know, open telemetry opens those doors. Again, this is gonna be something that we would want to spend more time on. I'm exposing it to you. I'm showing you.32:49Kind of what could be done with it and where to get started, and but I'm going to.33:09We're actually gonna create a separate video on this because it it could get very heavy, very in the weeds, you have to set up a, a collector, and a service, kind of like what I showed you in the diagram in the beginning, and to do that, you know, could take up a whole one of these sessions, so.33:16And, you know, for me to expose it as much as possible, we'll have to skip some of that on the demo, and like I said, I will, we will make some videos and we will put it underneath this one, in the description so that it's easy to follow and you can say oh you know what, I wanted to click in more on this. It's more important for me to.33:36You know, get dive dive deeper into it now that I've been exposed to all of my options versus dive deep right now and not know what other options there are.33:56And now, some of the newer options that we have with the, the dual agent, is you can report it in two places and now I'm really sorry this is my windows kind of came up in a, in the wrong order.34:07So.34:27This is a an environment where we have it already set up with an app with an application. The application is a, a spring application, Java, AKA java. It's just running in a, you know, singular, it's not containerized.34:28Is nothing's super fancy, but I just want to show you, you know, once you start going through those getting started steps that I was, I was showing you where they were in the documentation.34:48It honestly becomes very easy. You, you come in here, you run your integration with, you know, in this case we're using Splunk Olly or observability cloud. We went through the integration steps over there so that when we downloaded the dual agent from appdynamics, we were also able to get the collector.34:59And everything set up with the keys. Now, obviously for security purposes, I can't show you that part, so you know bear with me, but we want to show you, you know, not only, you know, some of the caveats that we found that there were because the documentation is pretty good on setting up the door.35:19Agent, but there are some pieces missing and I felt that it would be more important to show you those things so if you're struggling, it'll get you through those. So 1st of all, we're gonna go ahead and check out what the actual dual agent structure looks like in terms of when it's on a, when it's on the server.35:39For.35:59So let's go ahead and do that. Dual two, we'll do an LS. Now, when you're in here, you know, it looks pretty standard, right? It looks like a regular Java agent, but something might call out to you. There's an Hotel, there's an hotel folder. Let's go ahead and, and check what's in there. So CD Otel.35:59And now if you notice there's a completely different jar that we're not used to seeing. Obviously we do, we normally see this Java agent jar. You know, if you guys have been doing this for a while, this will look super familiar. There's a configuration for it yada yada if you're using the specific version.36:20You know, the same thing's gonna be in there. But this is kind of a gotcha that we ran into and that I want to share with you guys. So now when we start up the agent or I mean when we start up the, the application with the agent applied and I'll show you the configurations.36:40For the, you know, what we put it as the startup arguments over there, to get this all running. It does assume that you are using this java agent jar and not the one in the version folder, you know, the.36:56Let's just go back out of here. So in a lot of cases, we'd recommend that you use this Java agent jar within the version directory. But if you do that.37:13You will not, it will not lead you over to the correct location of the hotel directory and therefore the hotel side, basically the dual agent won't actually initiate. What will happen is it'll just run on, you know, the normal Java agent.37:28Or just hotel, and that's not gonna be well obviously not the hotel won't, the hotel will won't work, but the java agent will.37:48So it'll be your normal reporting into AppDynamics, but if you're expecting to see, you know, traces and spans coming up on the Splunk side of things or the, the Splunk observability cloud sides of things, it will not load this because it is looking for this directory here, but in the context of it.37:55Being in the same place as the Java agent that's getting loaded into the heap of the application. So if you don't have basically if you don't use this java agent, it looks for the hotel in the wrong spot and then it doesn't load the.38:15The actual dual agent, which is this one, the Splunk Hotel java agent, which is within the Hotel directory. So here, let me give another view. This is the hotel.38:33This is the job agent jar that you actually want. A lot of this stuff is covered in the read me, but it does not give you the correct arguments, so let's go ahead and correct that. Let's go one directory up and we'll go into that clinic singular jar.38:49And then I'm going to, I had to sanitize one of these a little bit for you so that, you know, obviously I don't want to put a bunch of blurs all over it by showing you something that you're not supposed to see. So we're just gonna go ahead and look at this one here.39:13Now, this is essentially the same thing that I I have running currently. These are all the start up startup arguments. It's basically saying, alright, I want to load this jar file as normal with all of the normal stuff that you would pass with it. And then.39:33This is where it's usually different. So right here is where it's going to, you're going to split it up and you're gonna say, alright, not only do I want the Java agent jar to run and provide appdynamics, all the normal good stuff, you're also going to put it in dual mode. And then when you put in du.39:51Tool mode is starts to take these credentials as well. Now, the.40:11The key, the public key will get generated when you go through the integration service of the, you know, in this case it was Splunk observability cloud. If you're using something else or something, you know, it's pretty set standard. It gets put into your ETC directory, but you can.40:16Can also specify a different place where that public key's gonna go. For security purposes, I'm not gonna necessarily show you that, but we will find a way around that to show you more of that step when we're doing a dedicated video to it. For right now, just know that this is something that you would do differently.40:35When you're deploying the java agent, and then once you actually start up the, the application, you'll be able to see the, not only what you normally would.40:55Within app dynamics, I, you know, like right here, pet pet clinic, tier dual here, we'll just come in here. We can see the tiers of nodes.41:07So not only do you get to see your normal APM information that you're used to, you're accustomed to, and you want to keep keep going, but you also have the ability to start breaking things up into traces and spans through the Splunk observability cloud.41:28So now it's reporting into both platforms, both Splunk AppDynamics and Splunk Observability Cloud. And like I said before, this benefits you a lot, and with it being a dual agent, there's less less involved with actually setting up.41:44The transactions and what things are gonna need to be picked up. The agent will kind of apply it for you, and then hands it over to the Hotel side of things as opposed to.42:05You know, having your developers have to put in a bunch of extra code to make, you know, to, to show the Otel collector what it should be collecting. You know, largely the app D side helps the HOTEL side. There's less work for you to get started and.42:16And also you get it reporting in both platforms right away, so then you can either start.42:36Phasing into using more Ollie and start building out more, and then you might want a dedicated observability or hotel collector and I mean agent running there as opposed to, you know, running the dual agent, but at the same time, you know.42:42Like I said, a lot of customers start out with this, they find out if hotel's right for them or if they want to stay with the bite code instrument, bite code instrumentation, which our normal agent does. Alright, so let's go ahead and get back over to.42:59The command line, and we're gonna go over where you would go to get the, the information or additional information out of a server infrastructure monitoring or a SIM agent. Now when you're in here, it, it's kind of.43:16Misleading because we all know that extensions, let me see if I can make this a little bigger without it being too wonky. We all know that if you're running an extension, it goes under this monitors.43:36Directory. Now, to save time, we're not gonna dive into that, but if you have custom extensions, you know, that's where that goes. For this, a built in extension, you're actually gonna go under the extensions directory. So CD extensions, and now this was the, I started at the base of where you're.43:48Where I installed the machine agent, it's gonna be different for everybody. You know, for me, it's under opt AppDynamics machine agent.44:08You know, our machine agents since I have multiple, it I'm put I put it into version 26 and now I'm in the extensions directory. Let's go ahead and do an LS, get you guys familiar with what's in here. Now there are, you know, other options here of things that you can expand on within.44:18Crash guard or if you have it on a hypervisor or a docker monitoring or agent server, these all have their different use cases. What we're focusing in on today is what's most used, which is the server monitoring.44:37And then I think there's a config directory within there. VI see server monitoring. Now this is where the magic happens. This is where I really want you guys to focus on especially if you guys have the server infrastructure monitoring agents there. We want to make sure that.44:55That you are actually.45:15Looking at not just a long string of Java processes, you wanted to actually call out which ones, you know, which things you actually want. So let's go back to here. Now there's a lot in here, obviously if you don't like things to show up as a percentile.45:17You know, if it's not a good fit for you guys, you guys can change that here. If it's something about volumes.45:37Like I said, out of the box, it captures the 1st five that it sees, but if you want to expand that out to ten or or if you want to blacklist some or if you want to whitelist some, you can do that here. You know, especially if you have different drives for different, different purposes, yeah.45:45You want to make sure that, you know.46:05Of the ten that are showing you only care about the main three, you can do that here. You can make sure that this is showing, you can kind of save your overhead on the agent as well on the controller by blacklisting what you don't need or, you know, setting it up by limits. The network configuration, same kind of view.46:07Deal. You know, if you, there's something that's going on in the background that you don't care as much about or it's a virtual virtual network adapter that you don't care about or it's not being utilized, you don't necessarily need it sitting there showing up within your server. You're.46:28Your server tabs or your server views within the controller, so you might want to blacklist or whitelist that. And then this is where I really wanted to, to get into.46:47Is right here. So now to actually do a renaming of processes, this is where I was explaining if you have those three proprietary Java agents or I'm sorry Java processes that are doing things in the background, say that they're running in our sync or they're running some sort of batch command that.47:01Moves all of the latest forms that were filled out from your customers.47:20And and you're wanting them to actually be called out as opposed to just something general that says, Oh, it's java. I want it to actually show up as the name. So like, you know, that we gave an example here. So machine agent tasks. Nowadays the the machine agent will actually be called as its own.47:28Where before it used to be, you know, it used to be rolled up under just Java.47:49We acknowledge that, you know, for us to be, to be promoting this and then expect you guys to call out your own monitoring solutions on every control or on every controller on every server that's being monitored is a little unfair, so this is actually done automatically though. So but at the same time.47:54Time, those proprietary ones are not. So for sake of example, if I was I was looking under the processes view, which I'll show you in the controller in just a moment, this is the name that it'll show and it's using this information to catch it.48:13Within, you know, the processes that are running on the machine. And that's where it becomes very important and it's it I found it to be a little hack because, you know, why, why, in some things you don't need APM for, some things you just need to make sure that it's running and you want to know when it's running.48:30Or if it's in idle or, you know, whatever the case maybe. Sometimes you just need to know process opera operational, you know, operating system processes that are running as opposed to necessarily like the response times and and all of that. And then, and the actual controller.48:49Come on. Then in the actual controller, if you're under the servers view, there's a lot in here, so bear with me.49:10If you're actually in here and you're looking at the processes, it won't, it won't just put it all under Java with a whole bunch of PIDs, it'll actually break it down into a specific.49:26Process named whatever you put in there. So that is.49:46All that I had for you today for our demo, and these are extra ways of you getting more information out of AppDynamics. With that, I'm gonna go ahead and hand it over to Bill Harper who has a lot of experience from multiple different ways dealing with integrations, which.49:51Is another very powerful way for you to expand what AppDynamics gives you.50:11From just out of the box. Thank you Chris, if you could advance the slide, which it looks like you just did. So talking about some of the integrations that AppDynamics has.50:16There's a few that are built in, the namely like the thousand eyes one is built in, there's one built in for Splunk and there's a few other that are built in. You get to them by going to administration and then integrations. But if we go across the top there, one of the main ways to integrate into AppDynamics is using its.50:32Successful API.50:52There's a number of robust API calls out there that you can use. These are leveraged by lots of customers to automate tasks, but there's also a, a grouping of partners that use AppDynamics as sort of a, a performance judge. And these.50:53Tend to be in the development framework under CICD pipelines. So you'll see lots of tools out there. I listed a few at the bottom there, microfocus. They own load runner.51:13A couple of different versions of load runner, which is probably one of the oldest load generation and and judging products out there. Appdance is a small company in Santa Clara that does API stuff, and then Harness also has their own integration with AppDynamics using their CICD pipeline. So that's a big.51:25Use case for using the APIs. There's others, you know, building dashboards like with Grafana or or similar things. But like I said, a lot of customers use the APIs to automate tasks. One example that I've shown multiple people is there's some maintenance activity going on and what you'd like to do is not get any.51:46Alerts.52:06And but you don't want to go on the tool and and click off, you know, 15 health rules or whatever. You can click one button, click them all off at once, but a lot of customers would just like to run a script. So I've shown people, customers how to make the API call to basically disable the business transaction so they don't fire and then re ename.52:07Table them when the the maintenance window is done. So that's a, that's a use case for using the APIs. There's a lot of other ones out there, but those are some of the the more prominent ones.52:26If we go to 2nd pillar there custom extension, so AppDynamics has the ability to build into the machine agent. Chris was just going through the machine agent, what we call an extension, and an extension brings in, at the end of the day, brings in custom metrics and what an extension does is it, it typically will go make API.52:37Calls to another system, and bring those metrics into appd through the extension, and once they're brought into app D, you can actually see them under custom metrics, and you can set health rules, you can use them in dashboards, you can do all the normal things you would do with any metric in in appd by bringing in.52:57These custom metrics through through an extension. There's a couple examples there in this column here. Redis, Redis MQ, data power. There's a lot more. If you go out to cisco.devnet.com or devnet.cisco.com, excuse me, and search for app dynamics extensions, you'll see probably.53:17Be about 80 to a hundred of them out there that people have developed. There's one, another real common one that we see out there was like an uptime one where customers want to keep track of the uptime of a specific server, how long has it been running for? Has it been a year? Has it been a month? Has it been a few hours? And there's quite a few others out there and customers extensively use.53:37Use these to create metrics that they want to bring into dashboards or use for health rules to help them manage their environment or better manage their environment. There's the back end of appd, this is a very powerful part of app dynamics as well. We have basically an outbound webhook that can be triggered.53:57Through an application health rule, through a policy and through what we call an action. And we always refer to these in the real world as alerts, but the web hook can be configured to basically one of the biggest use cases is to drive ticketing systems. I've listed a few down there below that have integrations with AppDynamics, BMC pager duty.54:17ServiceNow, those are probably the the big ones in the market. A lot of smaller ones out there have done this as well. It's fairly easy to create a webhook integration, and we allow you to actually create a custom payload in that integration using what we call our templating variables. And if you.54:37Click on our YouTube channel, I did a, a video on those templating variables and how they work. They're a bit tricky, but once you understand kind of the format and the syntax, they're pretty straightforward to get information out of there. So ticketing systems is a big consumer of webhooks, also event correlation.54:55That would be something like, you know, Splunk ITSI or mooxoft or few others out there, cloud fabric out there in the market, as well as big pandas another big one out there that does event correlation and generally event correlation system sits in between, the APM and the ticketing system.55:15So it correlates other signals coming in from other solutions and provides more information when the ticket needs to be opened in the back end incident management system. Chat, there's a lot of things that you can do with webhooks and chat. I built one for slack. It's in our community if you want to look, look it up. I find every time.55:35My work with the customer that the customer, if they have Slack internally, they've actually integrated that, that.55:55That integration I built for, for Slack into their environment. I see it every time I get into customers controllers. You can do this for Microsoft Teams as well as I think there's an older one out there for Webex. So those are some really good use cases for the outbound webhook. And again, it can be triggered from a health rule violation through a policy and through an action.56:03Now we come to thousand eyes. So this is like an in product integration. Thousandeyes is the standalone product on its own. It provides network insights. It provides tests so you can run tests against different endpoints and there's an Apti integrated widget underneath our browser.56:24Monitoring solution that allows you to see thousand eyes data. What I always like to tell customers is, you know, our browser monitoring or our brem, it can tell you when things start to slow down and where when you're having issues and when customers are, are struggling with response time, it can tell you that that's happening.56:43Thing but it can't tell you where. And that's what thousandeyes can do is it can actually tell you where in the network that bottleneck exists so that you can go work with your IT team to get it remediated. There are a number of other Cisco integrations with thousand eyes. I know we're supposed to be talking about appd here, but I want to just put a plug in there for thousands.57:03It's integrated into all kinds of Cisco network gear because of what it does in terms of finding where the bottleneck is. And the name thousand eyes comes from the fact that they have thousands of eyes on the network, watching every major provider out there, watching every major, you know.57:23Congestion point, so they can see all this traffic and they correlate it together and provide insight into where bottlenecks are occurring in the path from like A to B on a network. And then lastly there's Splunk. This again is an in product integration with what we call log observer connect.57:43It's an integration with ITSI and Chris talked about the Hotel dual reporting. And so we have this under Splunk because the dual reporting that Chris was talking about was showing the application reported to appd as well as Splunk only cloud under Otel. So it's sending out two signals.58:03The appd signal to app D and the hotel signal to Ali Cloud. Log observer connect is actually a a really nice improvement in app dynamics. It allows you to drill into logs as you go through and you troubleshoot your application and dig down.58:23The further you dig down, the smaller the log footprint becomes. So it makes it really easy to go in and understand what error messages were thrown when that anomaly occurred or when that health rule violation occurred. And again, ITSI I said this earlier, this would fit under the.58:39The webhooks bucket because ITSI is sort of an event correlation pro.58:59Middleman just like big Panda is and just like some of the others I mentioned. So there's a integration to bring Apti data into ITSI and to use it to further make decisions based on what other signals are coming in and then when to go open the ticket. So Chris, that's about it for the integration slide. I will hand it my handed.59:05Get back to you or am I handing it to someone else?59:25Back to me, he's ok. Back to you. There you go Chris. Back to you. Hopefully that was that was an ok overview. That was excellent. Thank you Bill. So, and just to reiterate, you know, Bill Bill brought a couple of good points up here about, you know, we have other content on all of these, you know.59:29Bill has actually done a whole series of videos on integrations. Kelvin who's on here as a panelist, he's done a lot of good videos on the custom extensions, and you using them with APIs, you know, powering your analytics in the background.59:48And then we've also done videos on the, you know, the networking insights and everything from thousand eyes. So some of the examples that you'd get within the controller or within just thousand eyes, and then also how to set up on the splunk side, how to set up the log observability connect.01:00:08Or in ITSI you know we've, we've done videos on, on a lot of those internal integrations like thousand Eyes and Splunk you know very specifically, so definitely check them out. I will try to, maybe create little, little clickable links for you within the video.01:00:28On on this YouTube recording and, you know, without further ado, let's go ahead and jump into what questions you guys have.01:00:48For us. I saw that some were already some of the questions were already answered from our panelists along the way.01:00:55It doesn't seem like there's any more, but, by all means, if you end up having a question, go ahead and put in the Q and A now before we finish up and we will come back and, and address those as we go. So now.01:01:11In terms of getting set up for, you know, getting you're ready to use the tool, trying to get better at it, refining your your skills, honing in so that you can do a speed run as opposed to a crawl, you know, trying to figure out how to use the product. It these enablement sessions are exactly.01:01:28Really what that's for. It's meant for you to get familiarized, understand all the possibilities of it. And then you have the ability to start reaching out to your Cisco resources, you know, hey, we need help doing this, whether it's professional services or account team, and you can actually reach out and ask for.01:01:48A customer success engineer and we can engage.01:02:08To support you along the way. There are a lot of community discussions both on Cisco and on Splunk depending on where your journey is starting from or where it's going. And you know a lot of questions get answered just from our community. Other people that have been.01:02:11Doing this for a while and using the products for a long time.01:02:31A lot of them are very knowledgeable or in addition to that, we also have people like ourselves that are on there answering questions for customers as well. So check those out. Also the, our YouTube page that we keep mentioning, we have a lot of good videos on there both long and short.01:02:34We have, you know, put a lot of time into those to help you guys in making sure that you are successful with the product. And then if you guys are looking to get some badges and kind of expand your technical knowledge and credibility of knowing AppDynamics, we also want to.01:02:52You know, promote the U.cisco.com. There are certification courses in both are both instructor led and on demand content depending on, you know, what kind of membership you guys have. I would highly recommend checking those out and start going towards a way that you can get certified.01:03:12It looks great on your resume, tells your employer you're serious about the product or the monitoring solution that they have you using. So always a good, good point and good sign to to be developing yourself, not just the codes that you support.01:03:32Alright, I don't see any additional questions coming in still, so let's go ahead and start wrapping up. Thank you all very much for joining both the panelists and speakers, as well the customers and and you know people that that joined the call. We really appreciate.01:03:50Appreciate you helping me make this a these a success, and also we want to give you a chance to provide feedback if you guys have any content that you guys are hoping to see. A lot of a lot of the content that we come up with for these are based on your guys's feedback. So, if you guys have any.01:04:08That you would like to, any extra stuff that you would like to either see YouTube videos on or that you would like us to go over in these observability expert insights series episodes, we would love to cover those for you, make sure that you're getting the most out of the product, most out of your time with us.01:04:28And, you know, otherwise, let us know how we did. Was it, was it too fast, too slow? Do you wish there was more demo? Do you wish it was a little bit more in depth, do you wish it was more and more service level, even if it's just a matter of our approach, we we appreciate the feedback. We wanna make things better for you guys and make these useful, so.01:04:47Thank you all and have a great rest of your day.01:05:08