Mastering APM: Splunk AppDynamics best practices for Observability

Today we are gonna be talking about mastering APM with Splunk AppDynamics and best practices for observability.00:11I am Shashank. I am observability customer success manager for Splunk. I am joined by my technical counterpart Aditya Kumar, who's gonna be leading this session. He's going to be taking you through what is APM and why it matters. He's gonna be talking about.00:19Strategies used in business transactions.00:40Best practices for tiers and notes and again best practices for agent deployments and snapshots and diagnostics. We'll talk about health rules and alerts, dashboards and KPIs and how AppDynamics and Splunk is being integrated as we speak.00:43If time permits, we'll also have a short UI demo and if you have any questions, you can post it on the chat. We'll take it up at the end of the session. If or you can just raise your hand in the middle and we'll see if we can take it up while the session is going on.01:05We're also joined by Kevin Huee who is the architect of this webinar series. He's also our customer success specialist. He's joined us from Australia. With that.01:22I'll hand it over to you. Let's get this started. All right, thank you everyone for joining in. Just a second. Kevin, are we going to record this? We are recording as this week. Okay, sure.01:37Got it. Yeah, let's get started. Yeah, alright. Thank you. Thank you Shank for a brief introduction and good morning, good evening, good afternoon, everyone. And this Ishank has introduced me. I am customer success specialist based out in Singapore. I look after all the technical.01:52Engagement, post sales engagement basically if I would say, and I'm working.02:12You know some of the, you know, critical customer is spread across APJC region. And today our agenda for this webinar is to talk about what are the standard best practices which Cisco or appd has, you know, introduced to our customer.02:18Who are, whoever is in their journey to implement Abdi or any of the APM solution in their environment. So during today's session we will be talking, you know, just a little bit about APM, what is APM, why it matters, and then, you know, during the implementation, there are several.02:37You know operational stuff which a particular implementation team would be doing like, you know, discovering application business transactions, you know, configuring tiers and node, what are the best practices of agent deployment, how we can streamline all the process and automate it, and then.02:57And what are the best practices of capturing diagnostic data if we have to do it any RCA and, you know, find a root cause of any problem and then what are the standard and best practices of, you know, configuring proactive health alert, health rule and alerting your application so that it will be, you know, more meaningful.03:16Tool and more productive.03:36To the user who are monitoring the application and what are the standard dashboard and KPI which we should take into consideration while configuring it to any critical application. And then lately we will be touching base on the, you know, one topic which is how we can get more value.03:38Out of integrating AppDynamics with other Splunk product.03:58Which comes across in a observability, you know, space only. So on that note, I would start and while I will be discussing on this point, if any one of you have any questions then please feel free to put it on chat window. Would be monitoring that and you know.04:02And once we complete all the topic in our agenda, we'll be taking one by one. And during our agenda, firstly we will be talking about some theoretical stuff and then if time permits at said, we'll be going to our UI demo controller. There we can discuss about what are the different.04:22Options and where we can do that configuration which we are talking about.04:42All right, so let's discuss about what is APM, ok? So observability is basically, you know, ability to monitor your application, you know, infrastructure or I would say eco hole your application delivery ecosystem where you are tend to.04:49Monitor your critical application which are delivery delivering to your customer, your businesses. And when I talk about application performance monitoring, it is not just monitoring the code only.05:09Since Splunk AppDynamics is a fully stack observability solution, so we cover from end user to the application services to the infrastructure and database. So that's why we call it as a full stack observability, so we are touching base on.05:22Exposing all those, you know, underlying components which are, you know, responsible to deliver that application. We have a full visibility across all the, you know, infrastructure database and application, and once you onboard each and everything, everything would be, you know, auto correlatedly.05:36You don't have to go relate it manually. So more precisely today we will be talking about APM piece of it, application performance monitoring.05:56So once we say application performance monitoring, why it matters and why we need to monitor that. So because of APM performance monitoring, we used to get end to end visibility. It tracks the user request workflow completely cutting across various distributed, whether it is microservice or a server based application.06:05Or a monolith application?06:25So once APM agent would be sitting in your application and your user would be interacting to your application right in your business hours or non business hours also, then let us start tracking each and every individual application and captures whatever happens in your application code, right? So.06:27It goes to your microservices to the, you know, servers to the database calls or the cloud API also if if your application might be consuming. Then apart from that, we are getting code level diagnostic also. So.06:47At at any point whenever your user might be facing any issue or your application would be encountering any, you know, production issue, then it has sufficient code level diagnostic data, which is basically quite evident to find out the root cause that where that particular problem started and what are.07:05For the, you know, class method or a particular services get invoked, whoever is responsible to, you know, slow down that particular business call or or that error, right? And then dependency mapping as as soon as you onboard your application services.07:25All the exit call and all the entry call on that particular services get discovered by that agent. So our agent will help you to get the, you know, correct dependency mapping of your, of your application, right? So it dynamically discover all the integration between backend services, whether it is database or the.07:45Our queuing system or maybe any.08:04Cloud API or any particular, you know, any particular microservice which you might be accessing. So those are the three factor why what APM brings into onto the table, right? And by this matter, it basically helps customer to reduce MTDR, ok.08:06Slash down the web downtime will prevent.08:27Expensive business disruption. So just imagine a condition if you don't have any authority solution in place, then the only thing which you might be relying on to that your customer would come and, you know, sending an email or might be contacting via any other mode and saying that your application is not accessible.08:29And they are not able to function with it, right? And then.08:49You will check in your system why this application is broken, what all services are impacted. So all these things are quite, I would say reactive, right? And by the time you would be knowing about the root cause, you will end up losing your, your business and your all digital user experience would be you know.08:53By the time. So what APM solution bring onto the table, it actually proactively monitor your application throughout the day, 24 into seven. It is not about that once your customer would be interacting, then only.09:13You know, you will get to know about the about the issue. So it's a proactive and it use some, you know, AI ML methodology where your system will predict your potential issues in your application, right? Before actually your user might be consuming your application, so that you can proactively take a decision and.09:28And fix it to prevent your outages in your application, right? And.09:48And the other piece of that, when you are developing or, you know, staging your application, then only you will start monitoring your application and see the performance of your front end of your application so that you can optimize the UX performance also apart from the backend services, right?09:53And so basically if if let's say there is a situation where you get, you know, issue in your application and there will be a lot of diagnostic data, whether it is traces, logs or system event or any services disruption.10:11So once you have this solution in place that, you know, it actually cut the noise through and correlate all the related events and, you know, alerts and put it in a simple context to make it easier for a user to start their troubleshooting, right? So you don't have to manually go hands and forth.10:30To the different sort of data, logs, metrics, event, and traces. It will all correlate it into single entity that is called, you know, your traces.10:50And you're gonna start your troubleshooting from there. In your snapshot you will end up correlating everything, right? So on that note, you know, as you know that in appdynamics, each and every user interaction would be counted as a business transaction, and if I have to put it on a simpler term, so.10:59Business transaction is nothing but a, you know, services endpoint which your application might be catering. Let's assume if you are coming from a financial back application background or a e commerce application background. So typical user interaction for flow would be your user might be logging to your application then searching some something in your application may.11:19Maybe any item, then they will add to cart and then.11:39Check out and then logging out to that application. So all these steps which I have just now, you know, elaborated. So those steps would be counted as a business transaction. So login is one business transaction at two, another business transaction search item is another business transaction. So likewise.11:43Every service endpoint which your application might be exposing to the user to consume it, that will count it as a business transaction. So in normal scenario what happen is, let's say we have put in our APM agent and start tracking all those user interaction.12:04So by default, what it will do, it will take all those URL system name and, you know, mark it as a business transaction. So this we call as a default naming of business transaction, ok, and that naming would be convenient or maybe understandable to a developer or whoever created a.12:23Application, but it might not sound very user friendly to the operational guy or to the business or some, somebody else or maybe a partner, whoever is responsible to monitor that application or you might be sharing the data, right? So.12:43What business transaction strategy says is you have you need to align those business transaction to your business journey. You don't have to rely on the default naming and everything. So in appd there is a flexibility where you can, where you can explicitly go and name your business transaction to the user friendly naming, right?12:57So rather than we say/application domain/login, we are simply saying that this is a checkout business transaction or a payment business transaction or user login as you see on my screen, right? So we are just giving a meaningful business transaction naming that aligns your tech team to the operation team basically.13:17And apart from just renaming those business transactions, as you know that APM agent tracks each and every endpoint, so not everything is as important to your business. So there is a flag flexibility where you can define which all business transactions are super critical for your application.13:37And rest all you can exclude or you are not looking at them. So what you can do is you can simply go and create an exclude rule to remove the unnecessary noise or I would say the the business transition which you are not interested to look at and.13:57Clean your business transaction list whichever is are important. There might be a case where agent would not be able automatically discover that endpoint so you can explicitly go and configure a rule to include or discover it.14:14Once your business transactions rules are ok and you are able to see all the business transaction in your application and you have renamed them and it is as clean as possible for operational or business guy, then you can lock those business transaction list that is called business transaction lockdown.14:31And why we are doing it because as you know that every time agent will restart, then agent will again.14:51You know discover unnecessary endpoint and will impact your finalized business transaction list to avoid that situation. What you are saying you are just locking down your business transaction list and saying that these are the final business transactions which I am interested to check or monitor apart.14:59From that all should be going into all other traffic, I don't want to monitor them.15:19To enable that functionality, you need to lock your business transaction listing, right? Whichever you are monitoring in your application. So that's it. We are avoiding all the user URL sessions and BT noise, whichever we are not able to, you know, we are not interested to look at and we are just simply.15:23Segregating all the critical business transactions, renaming them, locking them, including whichever is missing and excluding which we are not interested to see. So this is all together a part, you know, standard business transaction strategy for an application. And you don't have to do it, you know, manually for all those applications. Whoever is identical.15:44Services and identical application, you can import and export those policy from one to another application as well, right? But that depends upon.16:04That architecture of an application or the, you know, particular services would be the same, right? So now we go to tier and node best practices. So another important piece is, let's say you are onboarding your services and in appd as you know that.16:12Everything or be model application in appd in terms of tier and node.16:33So tier is basically, is a logical grouping of your services and node is basically, you know, minimum minimal unit where you can instrument FD agent, right? So node is equal to one CLR or one jbn, as you can say it in a simpler term and.16:39There is a logical grouping of those interrelated or I would say correlated notes. So let's assume you have login.16:58Service running in your application and that service might be hosting into, you know, a cluster of five hosts or, you know, container of five container or ten container, you know, port in your Kubernetes environment and you are interested to onboard each and every node.17:07And what is the best practices would be you just onboard those five login services and club it together in a single group that is called tier, which is login. So login tier is nothing but a logical grouping and the logic could be depend upon whether your application environment, it is a staging.17:25Development or production or in a best case scenario, it would definitely be a name of your service, right? And under the name of services, how many instances are running in your application that will represent as a node.17:45So standard naming would be the clean name of your service, right? The name should be, you know, very self explanatory to the user that that particular services represent to a login service or a two guard or something, then logical grouping them under a tier.17:59And then you need to isolate them further in an environment whether it is, right? So likewise, once you do all these practices while implementing appd in your environment, then your environment would be, you know, very user friendly to the end user because we are not onboarding and we are not doing all this instrumentation for us.18:17To understand, right? It is for the operation team, business or the other partner where we are intended to share the data later.18:37So now we talk about agent deployment life cycle. That is another major aspect of appd implementation. So we need to find a way where we can inject APM agent straight into your infrastructure images using CICD pipeline. So.18:46In earlier days we used to manually implement all your machine agent or your APM agent individually one host to another host, right? And that is quite time taking or, you know, heavy lifting I would say for any appd or APM implementer, right?19:06So now as the time has evolved, ok, so Abdi has also come up some of the, you know, automation techniques where you can actually use APM or any agent binary and, you know, package it through your CICD pipeline, you can use it Ansible or Puppet or whichever is automated.19:25Tool you might be using.19:46Or a normal system script would also do a job, right? And then you can trigger those job to put up those agent into the, you know, targeted host. Okay, so you can deploy your machine agent or or APM agent directly into the system base image, right? So that.19:47Once the system would be spinning up, then agent would automatically inject it into that system. And not just implementation by the technique you can maintain all the upgrade rollback and every administration administrative stuff right using CXD pipeline.20:07So what it actually helps us to standardize the versioning also because manually it is very hard to check. Let's say if you have thousands of hosts, it is very difficult to check each and every host has the same, same APM agent version or not. So while you are using CICD pipeline.20:26So it will help us to standardize the, you know, agent version across the board, right? So now this is you know out of the box CACD pipeline I'm talking about you might be using Jenkins or any, any other automation tool, right?20:44If at all you are not having those sort of stuff, right? So Abdi has come up with one of the, you know, better way of managing all your agent that is called smart agent, right? Smart agent is just a single agent which is a base layer of your host. Okay, once you install smart agent onto the host.21:01And then that particular agent ID and agent name would be reflecting onto the controller.21:21Then from the UI itself, you can directly deploy a machine agent and APM agent onto that particular host and not just implementation or installation, you can upgrade rollback or maybe, you know, do all sort of administrative stuff using a smart agent. So smart agent is just a hook.21:27Or the basic layer layer which has to present onto the host and once it is there, so all other agent will use that particular base layer to deploy.21:46Those agent down to the host. It helped us to regular audit what version APM agent would be running on a single host and machine agent version and we can quickly upgrade the agent, roll back the agent using a smart agent.21:56So this is basically, you know, best practices that is recommended from appdynamics side. So now we talk about what is the practices to capture a snapshot and code insight.22:12So as you know, by default AppDynamics uses and wait either number of calls or number of time to take a particular transaction snapshot, right? So let's say, by default setting would be after ten call.22:30Of a particular business transaction every agent would take one snapshot, right? Or if the response time of a particular service would go more than a threshold, then agent will start taking a snapshot, right?22:48So, but in most of the scenario or a or a particular scenario, it might not be true for all the customer environment, right? And they would be more interested to take snapshot more often than not, right? So what they can do it, they can customly go into application configuration and.23:04And select when they want to take a snapshot and it could be as aggressive as they want. But the only thing which remains is and.23:24You should be cautious about as much as a snapshot you would be asking agent to take for a particular service or a business direction, it will increase overhead onto that particular host because all the snapshot and every, everything would consume system resources more, right? So I would highly recommend.23:33Before you actually go and change any configuration to take a snapshot, you should 1st consider the system requirement and prerequisite.23:53Then only depends upon your environment, just go and you know change the default configuration to take more screen snapshot. It will help you to collect more, you know, widen your diagnostic data and in case of, you know, any outages or any issue occurs in your application, you will have.24:01Have more, you know, diagnostic data within your hand, it will ease up your, you know, troubleshooting and finding an RCA.24:21So like, you know, after we onboard application and we are ok, we have finalized your our tier node, business transaction naming and everything. Now what next? Okay, so we need to find actionable insights from an application and for a proactively monitoring them, the major.24:32Chunk of, you know, monitoring data would be going as a health rule and alerting, right? So that is the final.24:52Agenda of onboarding an application, right? So we wanted to 1st of all expose all the performance matrices, whether it is metric trace logs, event, anything, right? And then we wanted to observe it for a period of time and then once we are sure about that what we wanted to progress.24:59Monitor whether it is response time, calls per minute, errors, any particular traces, event or the log messages.25:20So we wanted to create an alert on those particular KPI, so we call it as a performance KPI. And we have four golden signals that is availability response errors and calls, right? So those are four standard metrics which every application should have a.25:28Active alerting on that.25:48And what is the best practices in case of health rule and dynamic alerting? So as you can see, and you might have also observed in as the time is being passing, ok, that static alerting is obsolete, right? So earlier days what we used to do, we used to capture one KPI and then putting.25:50Some static threshold and giving a threshold, any static value, right? So, but, you know, we are not tend to monitor our application only in our business hours, right? So now the time has changed and application monitoring has to be 24 cross seven, right? And.26:10The threshold which you are setting up in a static threshold, it might not be true for your non business hours, right? And it is at stagnant as possible. So just to be little flexible and beat, you know, on a dynamic side, what we wanted to do is we are using dynamic baselining.26:28To create.26:48Our health rule rather than using a standard static threshold, right? So it's a mix and match basically. So wherever our requirement is to be very stagnant and using a static threshold, that's fine. But in most of the cases dynamics baseline would be the answer and it gives us.26:49It's more flexibility to cut through the alert.27:09Noise and spikes, right? So there are a lot of scenario where you are actually knowing that, you know, currently my system performance would be going up or going down and that is, you know, known to you. But as you have set up a threshold which is a static, your system would not be having an intelligence to adjust.27:12Tested according to the time, right? So they will.27:32Keep on sending you false alert, that is false positive. So unnecessary it will just flooded up the user email or you, you would be annoyed for that number of, you know, alert you are getting in. So just to cut through those false positive scenario alert noise or the correlating the actual.27:35Alert into one single actionable insight. So we highly recommend it recommend you to use dynamic baseline. So as Abdi is doing all the historical data, you know, inside appdynamics, it is storing and once you enable.27:55And a dynamic baseline, it actually compare the current time metric value to the historical data of it, hourly data and accordingly, that baseline value would get changed, right? Depending upon the historical value whatever it is. So that actually give you flexibility.28:13To dynamically adjust your threshold value depends upon the time. So you won't be getting any false positive or alert noise during that time. So that would be, you know, more efficient and I would say more flexible approach of you configuring health rule and alerting. So last pieces for.28:33For creating a dashboard once you have created health rule alerts and everything. So now our final agenda of onboarding any application would be creating a dashboard. So.28:53Why we are creating a dashboard. It's a sing simple logic is that. Okay, so being as Abdi implementation implementer, I would be knowing that where to look out for metrics, event basis logs, and everything, right? Where I can go for transaction performance, where I should be going for tier node performance.29:04A snapshot or a service level or host level performance, but being as operational folks or the application owner, right? Whoever is going to monitor that, they might not be, you know, bothering to.29:24Switch to a different screen or different context to find out data. So what is recommended is to create one single dashboard for an application, which will give you, you know, three 60 view about that application, how that particular application services are functioning, what are the UX experience.29:38Or or the behavior or what are the infrastructure level of matrices, how their hosts are behaving? Is there any pro.29:58Production issues for that critical services or how their, you know, application services are functioning. So all this thing together we are putting in a single dashboard along with dashboard or 3rd party services also. And then we are asking them to monitor that single dashboard and all those widgets and everything.30:07Applicable. So whenever there is a issue or there is a color coding change from green to red, so they can directly click and go inside to the level two dashboard to get more, you know, drill down to.30:26To get more insight about that particular component. So while we are configuring dashboard, so 1st of all, we shouldn't be creating a single dashboard for all the audience, as we know that typically in an organization we have.30:39Audience, right? Audience, as in the the team, whoever is looking at those dashboards. So I'll start with the engineering team Engineering team or you can say it as a developer. So they would be more interested to see their class perform Java class method or service level performance. So they.30:57To be bothering about what are the business revenue impact and all. So they are mainly bothered about how that that particular class is functioning, what is the response time of that method or if there is any error, then how to fix it, right? Then goes to operational team.31:17So they are in between, so they have to look the system level metrics also along with the business level, right? And they are more concerned about the system availability, what are the active incident, what are the uptime, what are the response time of that application? And they are not bothering about the underlying, what are the method who is impacting the risk.31:33You know response time or availability. Then it goes to the top tier where your business leaders or executive are seeing your application performance. They are the one who are looking for the business matrix where what is my business revenue which got impacted due to any outage?31:53They are, again, they are not bothering about which class, which method has, has actually impacted that business. So they are just, you know, then they are asking operation team to look at and then they are asking to engineering team to look at the system level metrics. So the idea of configuring role based dashboard.32:10Is to, you know, create it or curate a particular dashboard according to the target audience who is whoever is going to, you know, monitor that dashboard. And the last but not least is basically how we can get more value while we are integrating our Splunk appdynamics with the other.32:29Splunk product, right? So, as you know that, you know, appd has joined hands with the Splunk absolutely.32:50I would say domain and there are other product also which are integrating seamlessly with Abdi to get more insight. So one is one very key aspect of integration is to have log observer connect and why we are having it. Okay let's assume your your service is getting impacted, right?32:58And you know that the service is impacted, but you want the raw system level data or the log level data to conclude, right? But alone FD cannot serve that particular use case as efficient as it can do it with log observer connect. So log observer connect is nothing but a Splunk instance.33:17That's where all the, all your services logs, application logs are getting piped.33:38And once you integrate your appdynamics with Splunk log component, then through log observer connect log observer connect is just an integration way and everything your all your business transaction snapshot and application services would have the raw raw log level data also along.33:43With it and it will automatically keep the context that.34:03While you are searching the particular, you know, log entries or or any data into it. And then accelerate root cause how you can do it. Okay, so while you are integrating with appd with the Splunk product, so.34:08This Plunk core indexing engine will will help you to, you know, have the data and, you know, correlate all the errors, logs and everything which will help you to cut down the mean time to detection, right? And give you more optimized workflow to find a root cause in case of any system outages.34:24On that note, I'm just, you know.34:44Done with all the best practices. I will just hand it over to Kevin for a minute to take up whatever questions you might be posting. Kevin? Yeah, just answer one question around out of the box default alerts.34:49Where we do have some default alert that's out of the box based on the baseline. I also want to point out the other point where we also have anomaly detection using.35:08Our AI learning model which is relatively new and it also trigger anomaly detection and those generates alert as well. Just one last thing is even if you have alert trigger, your health rule needs to have more actions involved.35:21Otherwise there's NO point of triggering alert, but you need to go into the particular part.35:41To check out, ok, this is an alert, instead you will trigger a notification, emails, digests or even a script to manage that, this will be covered as a separate topic in 21st of July proactive monitoring.35:47Which you can registered from the link I've posted as well.36:07I think the AI anomaly detection is also worth talking about. If you have some time you can quickly show people around as well if you can. Yeah, sure. No problem. Okay, so I'm just ending up this slide show and we'll go to the browsers.36:11Okay, just there are a couple of other questions as well. So sorry for that. It's fine. Can we can we take actually? Yeah, I can answer that. You can just show them.36:30No, I'm just reviewing the questions. I'm saying that can we take one, one question at a time and I'll be able to answer that or did we answer all of them? Okay so 1st question.36:48Yeah, 1st question in my chat which I can see is, can apply trigger alert out of the box based upon dynamic baseline for the key issues or do we need to manually define the alert or trigger? So Abdi does have, ok, some out of the box health rule where this these are pretty standard.37:04Okay and it starts monitoring our application major KPI as I say there are some golden signal let's say availability response time errors and calls.37:24And every does have all the default health rule configured on each of that, not just application level, all the business transactions, peer node, everything are covered, but those are pretty standard, ok? And it might not be very helpful in a particular customer environment.37:35So what is standard practices be before we actually turn on that those health rule?37:55We have to manually, you know, review those health rules and if that works, then if if they don't, then we have to adjust the threshold value accordingly, whether it is a static or dynamic baselining or s or other and once we are satisfied with the KPI which we are intended to monitor and.38:02And then what threshold value we wanted to keep, then we can enable them in, in our environment. And from then onwards it will start, you know.38:22Be active, right? And whenever there will be outlier, then they will, you know, highlight and send you notification. Does that answer your question any? Yeah, and.38:32Appd has already has a capability to collect log what is the benefit of have the observer connect enabled in addition. As I mentioned that Abdi is, has a capability of monitoring log.38:47Log monitoring and log analytics are two different things. 1st of all, I would like to clarify. Log monitoring is something we wanted to monitor a number of occurrence of a particular pattern, right? Let's say in log we wanted to see that particular entry are coming up or not. And we are seeing number of occurrence, right? That is log monitoring.39:03Log analytics is something where we are just seeing the live streaming of the data and based upon that we are writing some queries to check up and configure alert on, on top of it, right? So it's a live streaming of the data. So every does have that log analytics, but it is not as efficient as.39:23Splunk has. Splunk is basically.39:43Is dedicated product for log monitoring I would say. And since we have the more extensive and, you know, more mature capability in Splunk for log monitoring per se, so log observer connect is basically a bridge to connect AppDynamics instance to the splunk, right?39:46And whenever there is a outlier in your application service or business transaction or any error in your in your services.40:06So, once you have the log observer connect in place, so you can actually drill down to the application level data to the log level, right? And the context will be, you know, secured and preserved, right? You don't have to go and choose the time window, what service, what application, nothing.40:14Once you click, everything will be pres preserved and whatever will will be the log entries for that particular service and the during that time duration, so everything will be putting in a context and you can easily, you know, pinpoint the actual root cause which might be coming as a log, right? So.40:33So the so the idea of integrating both product is to bring both capabilities together in a tight correlation.40:53And give a user a flexibility of navigating from application data log level. Where is the data correlate for APM plus Splunk is the data APM sent to Splunk which has log data or correlation and the correlation? No data resides on two different.41:02Platform. Log data resides from APM resides on APM and the log observer connect would do a trick, right? So data would not be going from one instance to another instance.41:22So those are locally stored. Kevin, does that is that correct or you want to add anything? Yeah, I think it it is separate. But in terms of how the correlation is done, basically there is some sort of tag system that we, that we put in where both are under.41:341st then this is the log that's correlate with the business transaction.41:54So, in a nutshell, that's how it works. Yeah. And, and the tag which Kevin is referring to, that is particularly particularly about the metadata of your services, right? What is what is the node DR, anything, right?41:58Yeah. That is actually metadata. Metadata is basically used to, you know, tag the data in Splunk as well as appd and log observer connector use that metadata to actually correlate both kind of data, application plus log.42:14Yeah, for that particular part we also have sorry we also have at EIS that's gonna cover very soon. Just be sure to check out the calendar yeah and can every send the log to Splunk or we need.42:31Universal forwarder. I think every agent would not send data to log, ok? I think your Splunk universal forwarder would, would take that log data from the system and send it to Splunk instance. So this particular one I have experienced so I can tell you how that works.42:48Yeah, you need a middleware to make that work. Our official support is universal forwarder, which you just need token based to make it work. It's a lot straightforward. And if unofficially we can also have other forwarders to make it work. E.g., you can use.43:09Hotel collector that will make it work as well. However, there will be some menu additions say the metadata needs to be included extra column, things like that to make the format correct.43:29So which we are highlighting, that actually works in a alley, right? Observability instance, right? But just to answer this question individually appd agent like APM or or something, machine agent, that doesn't.43:43Don't have the capability of sending log data to Splunk as I know, but I would I would be happy to go and check and confirm it.44:03Yeah, you you need some sort, some form of forwarder to make that work. Universal forwarder is the official support and you can use other forwarder as a workaround. Yeah, but but again once we move to only which is again a part of our observability.44:12You know, authority scheme also, so only has this what is that? Sorry, let me check once.44:33Open telemetry collector so that hotel collector is basically capable of monitoring your APM data as well as sending log data to universal forwarder or the Splunk. Yeah, yeah, yeah. But that's a topic for another time. Yeah, yeah, yeah, correct. That is not being covered here, right?44:48So I think we are done with the questions. We don't have anything, so let's see if we get more questions now? Sure. Did you have any small demo that you wanted to show?45:08Yeah I can. So let me quickly share my screen and I would, I would tell, ok, so yeah. So let's go to that one application where I would like to see how our business direction should look like and how we are doing.45:24So I would go.45:44It's not here. Next I think that application. Yeah, you see this business direction name loans credit check, right? So it is not system name. It is a user friendly name right for a business. Let me take other application.45:50These are the system name, right? Slash web API/underword, so these are, you know, understandable for a developer, but those naming convention would not be very efficient to understand for the operation or business guy, right?46:11Let's go to other AD commerce and go to the business connection. Yeah, you see the business connection that add to cart login, view item update address and how we are doing it. If you go to the configuration, instrumentation.46:25Transaction detection, then you see all this naming? So we are individually monitoring all those business transaction. This custom include, right? This is a custom custom, I would say.46:42Custom method of monitoring business transaction. If I go inside that, ok, you can see all those configurations. So we are naming that business transaction as a view item, and this is the system path or I would say URL for that BT, and I'm.46:57Individually going and monitoring that business transaction naming it, right? View item. Likewise I have done it for all those business transaction update orders and all and whichever I don't want to monitor, so I am configuring custom exclude rule. You can go and check that, right? So this is exclude rule, not the include.47:13Right? And this is the system name or the URL and here I'm just naming it as a resource form, and I'm excluding it. Why am excluding? I don't want to monitor this.47:33I don't want to, you know, impact my business transaction list. I only wanted to concentrate on the business transaction which are very important for my application. And if I don't want to impact this finalized list of business transaction, I need to enable this log, right? So locking freeze the list of business.47:46All additional traffic will go into overflow. This will not impact your business transaction list once you are locking this. So that's one thing and how we can name them TSM node. So you can see this inventory services are the tier and inside that there are.48:07Four services which are running inventory service one, this, these are blue green deployment, right? So two instance we have in green environment, two in blue and all four instances are, you know, putting into one single.48:26Group that is called inventory service tier, right? So now the 3rd thing is snapshot. So if I go slow transaction threshold, ok, and there is a snapshot configuration. 1 s slow transaction.48:42Yeah, you see configure periodic snapshot collection. So there are two way of taking a snapshot and taking a snapshot every 10 min or we can take a snapshot every hundred execution of a business transaction, right? And we can choose whether we wanted to apply to all business transaction or some.49:03Individual transaction and these values are, you know, configurable. You can change it to every 1 min 5 min 3 min, whichever is permissible limit. But again, just wanted to highlight that once you are changing it, go and check.49:23Your system resources and what are the recommended way of doing it before you actually change anything on production because it will put up ahead, you know, it will use more system resources on your node.49:40It's a overhead basically. So once we go to the alert and, I would say alert and respond. So if you go to the health rules, so you see some of them are by default health rules, some of them we have created and.49:58Let's say confirm order response time is too high. This is custom. If I go and check this health rule, so these are the standard configuration affected entity we are choosing, and here we are choosing some static threshold, right?50:17So other than a static threshold, you can choose this one greater than baseline and you can choose your baseline, which baseline you wanted to use, right? So currently appd is doing 30 days of baselining.50:32I, my application matters is I can use daily trend of 30 days by default and I can use, by three and then you can use standard deviation, right? So whenever your current value is going to be in it is going to be, you know.50:47Deviate from your baselining by three degree, then this health rule will trigger and send you a notification.51:07That's how you can do is dynamic baselining for, for all your alert, right? And apart from that, there's a anomaly detection also, which is again works along with health rule violations. Health rule violation look after individual metrics or a set of metrics to decide whether it is.51:15It is violating or not, but to have more intelligence and to look other matrices also along with the that particular matrix and to decide whether it is actually an alert or it just a, you know, noise.51:35So you need to enable the anomaly detection. Once you enable anomaly detection for any business transactions. So, so what it will do, it will take three, four metrics into control consideration, let's say average response time.51:55It calls per minute and error and it will decide let's assume if your average response time is high and if you don't have anomaly detection, then simply your health rule will see your average response time is high, let's send a notification and trigger an alert. It won't actually go and check whether the calls are also high, right?52:15So it's a very.52:35You know, I would say common scenario when your application is is experiencing high volume of calls, then your application response time could be high at that moment, right? And that is quite common scenario. But if you don't have anomaly detection, those two metrics would be treating as a tool.52:37Individual health rule, which has nothing correlation in between and if those two metrics would be violating their threshold, then they will trigger an alert, right? While anomaly detection will take those two metrics and and correlate the data.52:57If let's say your average response time is getting high or peaking at the moment, then the system will check other particular metrics also, whether calls are also high. If the calls are high and every response time is high, that means it's a noise, right? So it won't trigger an alert.53:17Whenever your calls is down and your every response time is still peaking, that means there is a problem, then only it will, you know, trigger an alert or it will throw an anomaly. So it will put up a lot more intelligence rather than just relying on the standard health rule.53:35And, and for a clarification both are individual, nothing is being replacing to the other one. They both work in conjunction to give more intelligence to a particular application monitoring.53:53That's it from my side. There is one more question says can you simulate how appdynamics work? Yeah, let me quickly do that. Okay.54:10Let me share it. Okay, so I'll go to my instance and I'm using different instance this time around because I don't have permission onto other instance to showcase you.54:28Okay, if I go to agent management, so while it is loading, let me just, you know, take a time to actually, you know, talk about the work methodology of smart agent. You just assume a smart agent as a CLI or a utility.54:42Don't confuse with the name, ok, the smart agent is a, is just a utility which you are installing onto particular system, right? Once you install it on a system, ok, you can see if I go to the smart agent, right? So I have, I have installed one smart agent in my environment. This is.55:03The host name and this is the architecture. It is active. This is a smart agent ID, right? So once you.55:23Once you install it on a system, and it starts reflecting onto controller, you can choose this smart agent ID and what you want, whether you want to upgrade your smart agent or you want to install the new agent. If I say install agent, then I can choose which agent I want to use, whether I want to use Java agent or.55:31Which one? Let's assume I want to do machine agent. Next.55:51Select where to deploy agent, right? So this smart agents is tagged to that particular host, and in this directory I want to do it. Okay, let's go next. Okay, I want to enable SSL, I want to download from portal. This time around I'm choosing the option to download it from download dot.55:57Appdynamics.com, but in case.56:17If you want, not, if you don't have internet access and you want to use your local directory to fetch the agent, you can choose this option and give the directory path. Or if you have J for any any artifactory installed in your environment, then you can choose this HTTP URL and give the jfrog URL or any other.56:19Get URL anywhere where you are hosting that agent.56:39You need to choose the agent, which agent you want to install, right? Let's say I'm choosing this one. You you can enable same.net compatibility mode. There are a lot of configuration options and default extensions are there and say next, right? It will install directly machine agent onto that host. So what it.56:43Bringing onto the table is you just install one simple CLI or smart agent into the system and all your agent management of machine agent, APM agent, DB agent, everything, whichever are supported can be done through UI only. So let's assume a task where you have hundred of agent.57:04And one fine day you have to upgrade the agent version. What you will do, the manual approach is to go all the hundred agents one by one and start upgrading your machine agent or apm agent. And it will be a heavy lifting and overnight job for an engineer, right? But with smart agent what you can do, you can select.57:24Get all those smart agent at once, ok?57:44In in here, ok? Select all the smart agent right here and you can start upgrading, ok? And you can upgrade your machine agent, everything from UI itself, right? So it gives great flexibility and automation to manage all your.57:47Agent. So smart agent is just a base layer and on top of that, there is we using smart agent for APM, yeah, application restart would require. So, yeah, just for a clarification, when I say installing java agent using a smart agent, so all the down.58:04Loading agent, configuring agent, everything, application tier node, everything naming would be done through smart agent, but to inject that agent into startup or JVM startup file, right? That you have to do it manually because auto attach is not.58:24Available as of now, it is under development. Okay, in our next phase, that is the only target. Once you put the java agent, it will auto attach to your Java services, right? So as of now, as, as for my understanding, auto attach is not available. You have to give it manually, but once auto attach will be there.58:42It will be nearly, you know, zero manual effort, I would say.59:01Does that answer your question or you need more clarification on that? Okay then we should go on. Okay Susank.59:08I am done for today. Thank you for helping for hosting the CIS. Thank you Kevin. There's just two more questions which have answered one, it's about anomaly detection, but can you just wrap up with smart agent the question about smart agent around.59:25Using smart agent for APM, the application will need a restart I can say you can answer that. Yeah, that I have covered that I have covered. Okay. Okay, I think that's everything then. Yeah, then we are done now, today.59:45Any other question, please feel free to send out an email to any one of us and then we will be happy to take it forward. Yes, thank you everybody for joining today and we look forward to having you on our next one.01:00:01I think that is in a couple of weeks, so still there. Thank you. Thank you. Thank you. Thanks. Thanks much. Bye.01:00:18