Splunk Observability Cloud APM & RUM monitoring

Without further ado, let's get started. Aditya over to you. Yeah. Hi hi, thank you. Thanks Deepali for introducing and.00:10Before I started the technical discussion, so Asi has introduced me, I Amity Kumar, I am part of APJC customer success team. So I look after for the customer in APGC from technical standpoint and I do technical adoption.00:20Enablement and all the technical post sales activity with all my customers. So today we are here to discuss about, you know, how Splunk of the cloud APM and basically works. Okay, so we'll be, you know, the agenda for this call.00:40For today's we will be starting upon, you know, touch base on how APM.01:00Works in observability cloud, we will look after some of the, you know, the data interpretation, what do we have on only and how the data gets correlated, how we can see the data and what are the difference in, in different sort of views. And then we switch to our real user monitoring, you know, use case.01:06Where we have three different ways of doing real I mean real user monitoring can do in in in two ways.01:261st Isram and other one is the and then the last topic which we wanted to touch base on the synthetic test, right? So without ado, let me share my screen. So I don't have much PPT or deck to present. Okay, so we'll be voicing it over and we'll discuss.01:34Discuss about, you know, the different use case and whatever question you might be having in your mind while I'll be presenting about these topics, right? I mean, we have a couple of session earlier before and not sure about the people who have joined this. One of my colleague has already, you know, hosted a session where he talked about.01:54The Kubernetes monitoring, how we can do that, what is.02:14I mean open telemetry collector and agents, but just for a recap, I'll take, you know, quick one or 2 min to discuss about it, how we actually onboard application in Splunk only ok let me share my screen. So before I actually start talking about this view and further.02:18The you know.02:38The services and the transaction and what metrics it actually, you know, capture from the application. So let me take a simple example as you could see my on my screen. So we have, you know, this application, ok, where front end is talking to some different services and it is quite evident that we are talking about the microservice.02:40App application which is deployed in any sort of Kubernetes setup. So just to onboard this application, what Splunk observability will do? So it uses open telemetry language specific agent which actually instrument your application and the data will pipe down to.03:00Splunk observability collector other than the open community collector also, we support both, but due to the enterprise level of support and you know the support from splunk side for all the troubleshooting and all, it is highly advisable to use the Splunk distribution of open telemetry collector, which actually takes all the.03:20Telemetry from your application services and send data to Splunk observability cloud, right? And as soon as the services will get onboarded.03:39Then this whole, you know, view will be created automatically. It is not, you know, explicitly created. It is dynamic view as soon as the services would be start discovering and talking to the, to the application services, whether it is your upstream service or downstream service all will be coming under, you know, one.03:49Good which we call as a, you know, application overview that is typical a flow map of your application, how each one of the services are talking to the services, right? And here you could see what are the service health, what are the service error rate, I mean what are the sources of error and what is the latency of the services?04:09Right, how fast they are, you know, responding and if there is any latency in that service, then basically it is taking a percentile out of it and, you know, categorizing.04:29Them accordingly. So if I go to the services page, it gives me an overview of what all services are basically part of that application and some of the key performance indicator which we say as a request and errors and what are the latency. Let's go to the one service.04:40For a minute. Then we get this home view. So it takes out all the other services out of the equation for now. It is just showing me that this what is the upstream and downstream for that particular service. So the front end is directly talking to this edge service for now, ok? And, and this.04:59This can be, you know, changed according to the time what you are picking up here, right? So this maybe, you know, possible that in last 15 min this service might not be calling to any other service, right? Only this front end is calling. And these, these are some, you know, basic KPI which talks about success rate, what is the success rate of that service.05:19Last 15 min then some of the SLI, then total number of requests over a resolution of 10 s in last 15 min and, you know, if there is any error, right? And there are other metrics also where you can see the error breakdown if there is any. So I I'll switch to other.05:40Your screen which is tag spotlight. Again, these, the difference between these two view, let me tell you, ok, so this all metrics which we are, you know, having it here. So those are all the, you know, cumulative number of requests, you know, NO matter what kind of span or NO.05:59No Matter what kind of, you know, endpoint which this service would be listening or calling to, ok? So if we wanted to do deep down analysis or slice and dice the data out of these number of requests, then we go to tag spotlight. What it does is it actually bifurcate all the total number of records.06:191st based upon the endpoint, I mean, one service can listen to the different endpoints from different services, right? Let's say service A is exposed to the, you know, some three, four services in your application architecture, so all services are talking to service A on a different endpoint. So for with respect.06:39To this add service, it takes all the requests as in like total number of requests, but if I have to bifurcate that out of those requests, how many requests are coming from this particular endpoint and how many coming from a different endpoint? So this all bifurcation, everything will be done in the spotlight.06:59It makes your troubleshooting and, you know, RCA is pretty much easy, right? Where you can put a lot of filter what endpoint you wanted to check right now, ok, what kind of.07:20You know span kind you wanted to troubleshoot whether you wanted to troubleshoot an internal client or server, you know, different kind of span you can start troubleshooting, right? So, and it it gives all the three major KPI with regards to that bifurcation, right? Say right now we are looking only the server kind of span.07:31And the requests which are coming from the outside, right? Or the user are requesting for, right? What is the total number of requests for that? What are the error? What are the, you know, what is the, you know, response time for aP90 latency, right? Now we go to the error tab. Okay, it it says all the.07:51That if this particular service is having let's say in two days, let's see if there is any error. We don't have any error for this service, ok? So I can change the service also if I want and.08:11If I wanted to see any particular service has any error, let's see quickly if we have any. No, I don't find any error. All right. So.08:23Let me go to the service. Yeah, we have some error, right? So these are the errors which we have, how many errors we have for this particular service, then whatever endpoint these services maybe, you know, listening to, the data will.08:39We filtered out for that particular service, what are the different endpoints, and all these matrices would be, you know, bifurcated based upon the endpoint, right? And what is their latency, what is the B 99 latency error rate, errors and all and one last thing which we talk about, you know, very important is.08:58The traces, right? The traces will have all the diagnostic data for about this service. Okay, we can take any any trace which you wanted to troubleshoot, maybe the trace which we see the error, so if we click on this traces, so it opens up the, all the, you know, oh basically.09:18The execution of that particular service, what all services are involved and what are the spends which were part of this, this particular trace and if there is any error, then it will clearly highlight, right? You can click on that and if it is known error or a known error, you'll get to know, right? You'll see that, you know, span details.09:38If there is any error, this is an exception message for that.09:58Particular span. So all the spans which are correlated to a particular trace, ok, we can click on the trace ID and, you know, all sort of span, right? Not just root span or the child span, everything will get correlated in that particular trace and it will actually.10:01Give you a kind of timeline view where it talks about that if a trace actually took a particular time then what all the spents are being executed and and how much time it takes and if there is any error, then it will definitely go and highlight that, right?10:21So this is all about APM in a high level view. I'll take a pause, maybe if you guys have any questions then feel free to ask. Kevin, you're there?10:37Do we have any question? No questions so far. All right, then I'll move to the, you know, the real user monitoring part, ok? So this what we talk about APM is predominantly we are talking about the business service and the beg.10:57The backend of our application. Now we wanted to monitor the end user experience which you called the digital experience monitoring.11:16Oh, Code level visibility, yeah, we do have we we can actually highlight that which particular class of the method is, is basically taking how much time, right? So this is the idea of having this space is actually for that only, right? So.11:25It talks about which particular services are being executed and and how much time it is taking.11:45It will give you all the detail. Let me get it back again checkbox.11:55Thanks. Process.12:27Oh me get if there is any class and method are being also exposed up.12:36Sure we need to enable the code profiler or something to get the class and method detail out of those services. For the demo instance I could not see it. Okay, it talks about the, you know, which all services are being executed.13:29For that particular traces, and performance performance ok.13:49All right. Okay, so we'll move to the real user monitoring. Okay, so while we talk about real user monitoring, so we'll be, you know, starting with, you know, which is predominantly browser real user monitoring. So I'll go.14:07So each one of the steps very quickly, how we instrument it, how is it works. It's pretty simple to the you know standard instrumentation.14:26If I click on the next, then I need to select a token, whichever token we have to ingest the data on, you know, Splunk only cloud endpoint for browser monitoring. If I need to put.14:36The name of my application, then I say the environment, which is basically, you know, metadata for my application to correlate all different kind of network request and and you know webpage accesses. So let's click to the next. So this is basically a script which is generated.14:51That is called as a JavaScript agent or or or SDK which you need to enable and inject this particular script to the heads head section of your, you know, web pages of your application and then you need to, you know, start sending a request to your application, right? So this is NO.15:11Nothing but you know it it talks about the endpoint Splunk only.15:31Endpoint where this this script is hosted and it will, you know, start instrumenting all of your user request or the webpages interaction and start sending the data to the, you know, endpoint.15:37So once you instrument it, ok, so we you will start seeing those web application which are sending the data to olly. So we do have some of the application in our demo environment that is Anthony 2D79 is still.15:53And you could see that it shows about the page views and the JavaScript error for that application and it is highlighting all the JavaScript error also and what are the core web vitals for that particular application.16:12And if I want to go inside that application, then it, it will give me, you know, this is basically a home dashboard for any application which you would be seeing about, right? So it talks about what are the page views, ok, what are the name of the page? Okay, and.16:27What are the page load basically?16:47Okay, and some of the other metrics, right? So these are the UX metrics which are coming from your application. If I have to check front end health of my application, then I'll be switching here and it will, you know, show all the JavaScript error by page which are coming in my application.16:50And right now.17:10And, you know, then it is segregating between the different sort of JavaScript error which is coming in my application. These are from page page wise, and these are from the type of the JavaScript error which comes in my application right now. And these are reflecting all the JavaScript, like cannot read property of this.17:11And then uncort type error and some, what are the long pages which are or the network requests which are running in my application?17:31Which is taking more time and what are the, what are the count for those particular long running tasks basically in my application? Task is nothing but the, you know, the pages, that pages of the application.17:40So backend health will, will correlate all the, you know, the network requests which just talks to, and these are nothing but a API call which my front end application is making to the backend application. And again, you can go and check the page performance here.17:57Also.18:17It gives you, you know, complete view about the number of requests, number of requests per minute, then page load time. Again, it is taking the, you know, latency about P 75, you know, latency, the how much time it is taking, and then total number of error.18:18And all of that, right? And a network request is again the API call which is is happening to my backend application from the front end side. I can go and click on the tag spotlight. Let's say I am taking what any.18:37Yeah, this one showcard or this one view text spotlight, so this page request will again go and bifurcate based upon the user ID based upon the end user role based upon the HTTP URL based upon the HTTP status code.18:56Total number of requests are for this particular page will get, you know, classified into different particular tag.19:16Then what is the F version? The request from browser, from country, from OS version, from OS name, and there are a lot number of texts which you can actually, you know, classify all the, all the requests coming to your application web pages.19:23And one major thing which we talk about the user session, right? So this, these are the real user sessions for my browser application. Okay, so I have taken an example of, of a user session which is happening today basically, right? And.19:43It gives about what are the error or the events for that particular session. It has the session replay also which you can.20:03Execute and see what user was doing in my application and where that particular application start facing an issue if there is any, right? So some of the details have been marked for a security reason. And if I go down, then this is particularly particular.20:11The whole session detail, right? What user has started doing it? Okay, it, it is going to the.20:31Card service then it is checking out to from my application. If I click any of the, you know, web service which the user would be calling, so wherever there is an any error that's, it says that card page has start seeing some error. If I click on that, it will show me what is the error it.20:37It's talking about, right? All the data would be there, right? So this is the raw data about that particular error and this is a person, let's open other error. Okay, so it says these are there, right?20:58So source, some of the fields are missing and there is some error while rendering the final price. So if I add the source code while instrumenting this application, then it will exactly tell me which line or which particular section of the code is throwing that.21:15So right now we have not added any source map file to that application onboarding, so that's why it is just saying that these are the errors which your webpages are throwing or that particular service are throwing, but.21:36It is not still able to, you know, pinpoint where that particular error is coming from, right? So if I add the source map file to that application instrumentation, then that add, that detail will also be added here.21:51So it will also correlate to the, to the backend also if you can see that this particular, this session detail and it is calling to checkout service and this service seems to be ok and if you wanted to see any, you know, trace or something in case of any.22:08Issue, then you can directly.22:27You know, switch to the APM context and start troubleshooting, right? So it's a context driven and all the data will, will definitely will be, you know, correlated automatically, right? So you don't have to manually correlate. So once we have onboarded front end of the application as well as the.22:29You know, the backend that all front end request will be mapped to the backend services, right? And this will you know ease up all the RC and troubleshooting issues, right? Wherever we are seeing the issue. So it is taking that, you know, user.22:47Interaction into cons consideration and you see that this is a rum call basically which is happening from here and here onwards it is start executing those particular services and wherever it is throwing an error, it will just, you know, highlight that.23:08Which container, which container image, and all the metadata will also be there, right? So, so that we can take the data and discuss with the developer and start, you know, fixing those issues. So this is another, you know, example of session only. Okay, so.23:26So if I see the.23:46Yeah, session replay, then again the detail are being masked and you know we can go ahead and see what user are doing. Again, it is searching some product and then going to add it into the card and then call into the checkout service, and this is the trace which is actually.23:49You know, core gets correlated to that particular network request.24:09So that's all about. Any question on that before we get started to the mobile instrumentation?24:14Kevin, help me to take question if there is any. There is one question, but then I was just thinking we might just push it till the end because it's about lock observe connect and how to troubleshoot and issues when you have too many traces.24:33During that time that's a bit more relevant to something else, to be honest. Oh alright alright. We'll, we'll take it later. All right then I'll go back to the let me close some of the tab to make it easier to.24:48Right ok that's all. Okay, so we'll be talking about 2nd piece of real user monitoring which we basically do on the mobile for the mobile user and here there is a simple instrumentation step.25:04For an Android application and same goes for the iOS also. We'll quickly look at how it does and what we need to do it. Okay, if I click next, again, we need to search for the ingestion token, what we are going to use it for authentication purpose, then we will fill all the detail, what application, what environment, what.25:23The version of that app. Then if I click next, then it's a, it's a SDK basically which you need to enable while building the mobile app, ok? Once you enable it from the maven, from the I mean Coatland and any any of the, you know, suitable.25:43Way of shipping that particular SDK and you build your application, then once the application will be stored in the play Store and user will download and start interacting to your app, then all the user data will start getting into.26:03Only cloud and if I go to that, you know, application for the mobile app, just a minute. I have not selected the let me select to the mobile. Yeah. You see all the all the, I'm selecting astronomia.26:19App.26:39For this one. Okay, so these are some of the UX metrics like app launch statistics, what OS version, what is the app version, what is the device? It is just bifurcating all the launches based upon these different values, how much percentage these calls are coming from or the app are launching from.26:40And if I again go to the front end help, then it will tell me about the app error, how many errors are coming for that.26:59Application per minute, then again it is bifurcating all the errors based upon the different sort of, you know, exceptions and error, and then what are the crashes for the mobile application per minute? Okay, if there is any. So we have just put up some load and then it is bifurcating.27:08All the crashes.27:28Based upon the category, right? Which category those crashes are belong to, right? Again, this backend health will talk about what are the services it is talking to, what are the back end resource distribution request duration, I mean how what is the response time for the back end service?27:30This which this application is talking to, and network request is again, you know, telling about those, you know, endpoint, ok? What are the number of requests or what are the response time?27:50For that particular request and it is P 50 latency latency time and this is P 75 latency time, right? If there is any error, then it will highlight that this this particular checkout service is having ten network error, we can go and view in tag spotlight.28:04Again, this particular network request will again get, you know, bifurcated into the component, what kind of request, what are the deployment type and user role, you know, what is the org ID and what are the deployment type? It's a blue green or what are the user roles?28:23What are the URL?28:43I have built application and all. So there are a lot of tags where this request will actually go and bifurcate it upon, right? So, so the idea of spotlight is just to highlight and and distribute the total number of requests into the different bucket.28:45So if user want, say user wanted to troubleshoot a specific kind of, you know, error code or a HTTP request or an endpoint, so they don't have to go and search it, right? It's already there into the filter. They can click and start troubleshooting. So, again, we are taking into some of the more.29:03While a real user session into the consideration. So if I go and start.29:23Seeing the, seeing the user session replay, let's go to the replay. You can see what user is doing in my application, they click some product, then add it to the card.29:31And they're going to the cart and and they're making payment. So there are some error, right? While doing some error, then this is some error while talking to some HTTP post method.29:50So we'll see in the session detail what error those were, ok? And then again the the user is doing something for now we'll just stop it and we'll go what error the user might be facing. So this is a particularly session timeline, ok, and the sequence of activity.30:07See which user might be doing. So let's.30:27Open one by one and if we see any error, then we'll actually go and so these all are, are your backend call which are getting triggered and wherever there is a APM correlation out there you can go and directly search for the traces also in case see, there is 204.30:30Right, it's not 200, so let's go and see the trace, right? So this proxy is calling to front end, then it is calling to card service and then radius. Okay, now go and see why this.30:48200 204 is coming 11 s oh I mean.31:04Yeah Card expire and this is I think this is the error ok by which this particular 204 error is coming, right?31:20So whatever user might have selected in their cart, so it get expired while they are doing checkout. Now go back to the session as let's highlight this. Okay, let's go.31:40Yeah, there's payment error they have some error during that time. What is that?32:02Let me see if there is any any other seems to be a problem with your order, please contact to a sales representative. Yeah, this is the exception message or error what user is getting while doing payment.32:16So likewise, there is one checkout service which user wanted to execute and we got some 500 error for that. Let's go in the APM and see, oh, what is causing that particular error, right?32:34Let's go and see payment ok.32:54I wanted to see the error. Okay.33:23Okay this is saying this this particular error is for unauthorized.33:26In a payment. It didn't get authorized while doing the payment. Okay, so this is all about any question Kevin or should I move to the last topic for today that is synthetic?33:44Oh, good man. Okay, so so whatever we have discussed about the real user monitoring, we have talked about browser real user monitoring and mobile real user monitoring, but we do have certain requirement where we need to run some.34:04Checks to, you know.34:24To see whether our API application service or a particular, you know, user action in our application are available 24 cross seven across the world. So how do we do that in, in non working hours for a, for a matter if that application is, is globally.34:26Accessible, then we create some robotic checks on behalf of a, you know, user and then see whether those APIs are or the services are are accessible. All right? So, so we have created some of the test. Let's see.34:45Okay, so if you create any test, it will ask you what kind of test you wanted to do. If you click on API test, then you need to name it. What name you wanted to put, let's say test and here you can add your request to what whether it is get request or.35:03Or what kind of request this is, right? And you need to put your API, then you need to set up, it will ask you some of the, you know, different parameter for the configuration. And once you've done that, ok, so your AP.35:23Only cloud will start monitoring your API availability and then you can choose the location where you wanted to do that, right? Which all location, it has some synthetic agent over there then from there you can see your.35:41API availability like that we have some Spotify search, this particular API we wanted to check. And if you click on that, it will check about what is the run duration and then you see availability of that API, it is it's a hundred percent available.35:58And which all application we are looking at it, we are looking from the AWS Osafa location, right? And it, it says about the, you know, result of that API availability. Whenever it gets filled or or there is any issue it will get highlight, you can create detector and alert on top of.36:18Of it so that it notifies and you can start troubleshooting why that API is failing from that location.36:38There are performance KPI also which you can look at. Okay, these are the test result again, right? So let's take another example for the browser test. Okay, let's go to this one. There is another API which is checking about the uptime.36:47And the availability of that API from these many location and it says.37:06About the result where we can select particular location also if we if we don't want to see from all the location, we check from AWS Mumbai and it says whether this, whether this synthetic test runs and and what is the result for that, right? Whether it is available or.37:12Not if it is not available, a hundred percent then the test will fail and you'll get a result over here, right? And you can see why this is failing, so let's go and select all of them and.37:32I'm going to see if there is any result which is failing. Let's go and see and ok so I'll go in that result and see why this.37:46This failing, this is the reason assert custom variable track or this verification failed and this is failed here. Yeah, this is the reason right this is the error while this.38:06This particular API availability field actually. So there is one more example which we have for, for the browser test. Let me open it once.38:22Okay, so now go to the, go to the synthetic again and I'll take some browser test. So this is a checkout, you know, service which we wanted to see whether it is available or not. Okay, so these are the same KPI which.38:40We were looking for the, for the API, these are the uptime trend in one day it was 84.38 % was up and let's go down and see.38:59So these are the result, right? So let's go to the failure one and we'll see what user was doing. So this is whole recording what user was doing in that application. Let's play that.39:11Yeah, yeah.39:34Oh yeah so see why they were doing something it gets. Let's see what happened sorry this seems to be problem with the order, please contact the sales representative. This is a demonstration. All right, so that means this particular.39:50The browser test gets failed. Let's see where the problem started.40:08We see some of the payment service get failed, so while checkout, we saw that this post checkout service gets failed with 500 error. This is the URL which it actually triggers.40:14And we can go and check the apmiss right at that time right let's go what is that checkout service again the same I think.40:30Verified unauthorized payment or something, right?40:50It is related to, and if I go and check this 500 error, yeah, that is basically the correlation what we have from synthetic to APM also. And these tests you can go and create.40:54From, says synthetic test, create and you can add your browser test and there is a way of importing that a particular test also maybe you can.41:12Recorded a user action from a browser SDK and import it here to schedule it or maybe you can do it in in some scripting language like Python or some some other language. And then you can select the location, which all location you wanted to do, what devices you want.41:27To execute those and what is the frequency and what.41:47Fashion those particular location will execute that particular test, right? And if you wanted to create a detector also, you can create an and enable it. Yeah, that's it from my side for today. Ali over to you.41:53Yeah, thank you so much, thanks everybody for joining us today. If you have any further queries, please feel free to reach out to us. You can reach out to us via your account managers or your csmiss and yep we will be.42:09Be really glad to have if you want any focused one on one sessions, we can have those sessions or any queries will be, answered from our end.42:28I don't see any questions in the chat window Aditya so I think we are good to wrap up the session for today. Just if you Aditya, do you want to answer the lock up server connect question.42:40What was the question Kevin? Let me it seems like they are a partner, so they have, they have lot of serve connect set up and, trying to troubleshoot and they.43:00Problem is that the related lock are thousands because of the trace have multiple services and multiple operational spends. This is quite more about I think Splunk platform itself.43:17But maybe we can try to answer 1 s Kevin, I'm reviewing that. Yeah, so the answer is like, how do we go to the log level detail, right? If there is any trace or services failing.43:34There are too many. It's very hard to troubleshoot with that amount of traces and spans. Yeah, I mean that I get it. There are too many traces and spans, ok? But if, I mean the question is how do we get from traces and services to the log level, right? For that particular traces.43:53ID. Yeah yeah yeah I mean that that is quite possible, ok, as you know as you know that open telemetry does, you know, for all the signals, metric trace and logs, ok? So right now we are talking about the traces and probably metrics, we haven't touched base.44:13On the log, but hotel collector will also take that log signal and send it to the, you know, the other endpoint which we say as a Splunk enterprise or or some other, you know, where the log is going to be ingest, and that particular thing will get correlate or the.44:33Integrate with Splunk only with the help of log observers connect and.44:54Once you will be doing a troubleshooting for a specific trace or a service or let's say on a span level, and if there is any underlying issue which which can be answered and more investigated through the log level, then you can seamlessly switch from the span to the log detail and the context will remain.44:59There, right? The con the trace ID or the span ID will, will help you to filter down all the logs, right? With which are related to that particular span. You will not be losing any context in that case and everything will be get correlated from the start from the metric to the trace to the, to the logs or probably any other service.45:19As well, the question is more related towards how do they narrow it down? There are multiple services which are involved and there are thousands of it. They're trying to see how.45:39How we can make it easier for them to narrow down when something i am answering for that. Okay, so whenever this portal does all the telemetry collection, right? So, say metric trace or logs, right? So everything will have.45:57The context propagation which has the trace ID or the span ID with that also right so while the data will be ingesting to the only as well as the, you know, log stream or the other endpoint, and we integrate both system together.46:17So, NO matter if we are switching from the metric or trace to the log, ok, that particular span ID or the host ID or or any context information which brings those two telemetry together will have the context type, right? So we don't have to go to the log blindly and look for a span ID.46:33Your race ID to understand which log is basically related to what particular span. So we don't have to do it manually. We just have to click to the span ID and it will have a link to go to the logs and that link will lend you to the all the logs which are related to that particular span ID only.46:53So that context will be coming from the open telemetry agent or the collector itself, right? So we don't have to do it from the UI level. So once we integrate.47:16Everything will be there. Yeah, it's Unless it's something specific in their case or depends on if there is any specific use case then we would definitely be planned.47:28And the volume, the time frame and everything, but yeah, it's something if we can get more details on, then we can also help you with that. Yeah, one, one more thing, ok, I would like to highlight, ok, this context propagation which I'm talking about is is by default, right? You don't have to do it. Okay.47:47So trace ID, span ID, everything will will be, you know, you know, flowing until to that particular call, whether it is logs, metrics or traces to bring them together. That's how the spans are being correlated to a particular trace. But if you want more metadata to ensure that, you know, your.48:06Data will not be spillover and it will not be, you know, uncorrelated. Then there is a concept called baggage where you can define some custom values or the texts to get the data.48:26You know, even more correlated Does that answer the question, Deepali or do we have any following up?48:38I think yeah it has helped, he said thanks. I want to know if logs can be filtered at specific span level as my traces have multiple spans which are generating multiple logs. Got it, thank you for the I think yeah it it has helped. Thank you. All right.48:55Is there any other question I'm missing? Anything Kevin? Did I miss anything? I saw you have responded to some of the questions about. No, NO, I think these are quite simple answer those I don't think we need to answer over the call and we have a bit up run out of time as well, I think.49:16Alright, thank you so much. Thank you for joining in. If there is any follow up question, then feel free to, you know, send out an email to any one of us, maybe Kevin, Deepali or me, and whoever is working for your account with you, then we'll be happy to take, take up.49:36Alright, thank you so much. Have a good day.49:57