Splunk AppDynamics: Setting up Alert and Response Part 3

Thank you all for joining and welcome to another observability expert insight series and with this series, it is achieving proactive monitoring.00:10This is our Splunk app dynamics Alert and Response three. And I'm your host Christopher Banano here with my good friends and colleagues. Drew Osborne, who's been here this whole time you know presenting the 1st two sessions, which I'll put the link to in, in the chat very.00:20Pretty soon if you have any.00:40You know, if you want to go back and rewatch any of the old videos or if you didn't get to and you're just kind of catching up now on on the the whole series where this is the 3rd episode, you can go back and watch those recordings and kind of get brought up to speed. But at the same time, each one of them is independent of each other. They are building on each other, but at the same time.00:42There is some, some content that is separate from all of them. You should be learning something new on each one. So, don't, don't fred if you're watching them out of order. And then we do have Kelvin Miller and Mark Curpile here as panelists, as well myself are going to be able to answer questions along the way.01:01And let's go ahead and start going into some housekeeping items. So, it is assumed that there is some, you know, what Splunk AppDynamics is and some basic knowledge of the product and monitoring observability in general.01:21So we're not going to reinvent the wheel or anything like that, but at the same time, if you have questions or if you, you know, are feeling kind of stuck or overwhelmed with what we're showing, let us know. We'll, we'll slow it down. We can try to you know address it or explain it in a different way or or talk about it more.01:39And we are going to be having a Q and A section more towards the end, so whatever we can't answer along the way so we don't break up Drew's flow while explaining things. You know, we would really appreciate if you would go down and use the three dots in the bottom corner of Webex.02:00And open up the Q and A section. That way everyone can see and benefit from your question. It'll, it'll persist and then everyone will see the answer. Obviously, if you put in the chat or if you have any feedback like hey the volume's too low, the, the slide's not moved, something silly like that.02:19Definitely let us know in the chat. That's feedback we really value. We want to make sure you get the most out of this session. And for for those reasons, we're gonna keep the mics muted this entire time. Sorry for that in advance, we are going to publicly release this and we will not mention any names in the recording.02:38Or, you know, we, we scrub things out before putting it out on YouTube.02:58And we do have a dedicated YouTube page, and a playlist for this session and I, again, I'll send that out in just a moment. And then everything that we create, you know, it is, it is created by Cisco employees for the purpose of Cisco and Splunk and observability.03:03But at the same time, obviously, you know, it's, it's meant for you guys, it's meant for you all to get the most out of the products you have and we're here to help. So, but we don't expect that you're gonna take it and, you know, claim it as your own or anything like that. But without further ado, I'll hand off the mic to Drew. Thank you.03:21Thanks Chris and great to be here again and have everybody back. Thanks for attending. My name is Drew Osborne. I'll be the presenter today. We've done two previous sessions that kind of build on each other. So I'm gonna do a quick recap of alert and with the alert and respond module.03:41The three components. And then I am gonna show you some collateral in the controller to bring you up to speed. The main three things that we want to show today, I'm gonna trigger a custom event. So I'm gonna show you actually how to write a custom event to AppDynamics.04:01The.04:21And then use a policy to trigger and send that, that custom event or or an alert based on that custom event. Why might you want to use a custom event? The the example that I'm going to show today is, is a simple API call to create an event. So hypothetic.04:22Scenarios. You perhaps if you you're doing an application upgrade, you, you can write a custom event at the end that that was successfully completed.04:43Or if there's a deployment failure, you could, you know, include a custom event indicating something, you know, went wrong in the build process or deployment process. The example we're gonna show you is just a basic custom event and then how to use the policy to, to filter upon that and, and send that.04:54We're gonna actually send that to 2nd piece Slack integration. I'm also gonna show you how to send health rules through a Slack integration. That's just a basic HTTP request template in app dynamics. As we do the Slack integration, I, you know, think about what would be the tool you might want to integrate.05:14Your events with in the future in AppDynamics. We have a lot of flexibility with HTTP request templates. Servicenow is a, is a common integration that we do. I am not going to show you how I built the payload or how the payload was built today for.05:34Slack integration, I'm gonna demo the integration. We do have separate videos that that the team can link to if you do wanna see how this, how is that payload created for Slack specifically. But again, I'm gonna show you the collateral and then show you it function.05:54And then finally, we're gonna step into a new application that we haven't been in, in any of our previous sessions, and I'm gonna show you guys a remediation action. When our agent availability health rule, is fires essentially, you remember we we may, and I'll show that.06:12Collateral today, we're gonna use an agent availability health rule.06:32That has a remediation action to restart our app if it goes down. And then I'm gonna show, I can't show the adjudication, the action adjudication directly live. That is email arbitration or email approval to do the remediation action.06:37But I have two screenshots for you to see exactly what that looks like and I'll show you where you would configure action adjudication as well inside of the policy. So with that being said, we're gonna dive right into a really basic recap of the alert and respond module.06:57These components are we're gonna be talking a little bit about today. I mean they're, they're key to the the the talk track or the demonstrations that I'm doing. So we have health rules. These let's let us specify the parameters that repre represent what we consider normal or abnormal and and they.07:17Change the color of our app from healthy to unhealthy warning or critical. We then have policies. Those consist of a trigger which is based on one or more events and then action. So policies tie health rules and actions together for the sake of of this conversation.07:36Station and actions which we're gonna talk a little a little bit about today. I'm gonna show you a couple different ones are predefined and reusable. In the 1st two sessions that we've done, we primarily focused on email action, ok? Today we're gonna look at more what I would call common production.07:56Alerting workflows.08:17With that being said, I'm gonna quick recap this picture if you've been in my previous talk tracks looks, should look familiar. This, the top blue box represents the health rules, the middle box here represents policies, and the bottom box represents.08:19The actions and and you can see the policies tie, health rules, and actions together. This is just a visual flow of of kind of how events, event management works through the alert and respond module. And again, this is available right in the controller. So with that being said, I'm gonna have a couple.08:37Of different windows open. Pet clinic is an app that we have not yet been in. This is the one that I'm gonna demo the remediation action. And we're gonna come back to this, ok?08:57Within sock shop, here's where I'm, we've previously done a lot of our work. This is where we're gonna do the custom event, and I'm gonna show you a Slack integration. And then I'm logged into the actual host where the remediation.09:09Action script is going to be run, and this is also the the host, which I'm going to write the custom action. I'm running a script to write the custom action to make or custom event. I'll show you that script today, and then again.09:26I just want you to see here that we, we also have a Slack integration. I've been doing a little testing in advance of this meeting, so we'll just add a page break in here. So I'll be able to show you that stuff live. What what I'm gonna do really quickly here.09:45Is I'm gonna show the HTTP action template for the Slack integration, but I'm not going to talk in detail about it. So we allow alert action HTTP request templates. If you have not used one of these before, you can see we're just calling.10:05A webhook in, in a Slack environment. We're doing a post. We're adding a payload body. Again, this is the details I'm not gonna go into the the specifics of how this was built. We have a separate video, but for the sake of of knowledge, I want.10:24I want you to know that we're using this Slack new integration HTTP request template.10:44I now have an action that is tied to that policy. So here is my action. Rather than send an email, I simply selected the HTTP request template that I want to use. And so for all actions, I would always use that request template.10:50Then I would format my messages the same.11:10Okay, so we're gonna go ahead and save this or just cancel out of this. I want you to wanted you to at least see the HTTP request template and the action. So really quickly, the, the health rule that we're gonna focus on today, I have a new, if you remember, we made an application.11:13An error rate health rule, which is just a percentage based, it's essentially a percentage based alert. Sorry, here's the right one. The ratio, if you've if you're not or have never used a metric expression.11:33I showed how to build this in our previous two talk tracks. This is a simple metric expression to give you the ratio of errors, the percent of errors in the app, ok? I've made a copy of this health rule.11:49Which I've disabled, ok, for testing, and I've engineered it so that it will automatically fail, so that when I turn this on, I am engineering a failure to validate.12:05That my HTTP request template and action are functioning properly. So I'm actually with this saying if it's not a hundred fire. So obviously that, you know, this is not a health rule I would use in.12:19In in a production environment, but for testing my integration, that is the condition we're gonna say if it's under a hundred percent error rate, it's in a critical state. I'm gonna go back to the overview tab. If you weren't on my previous two top calls, I go into detail.12:35On why you might want to use each of these settings. Why would you change them? What impact do they have on the health rule? And so if you were not in my previous two calls, I highly encourage you to, to go through that.12:54Those are, it's just great information to know how you, not only how to make a health rule, but all of the knobs and dials that you can tune to make it so that it's not noisy. I talk about a lot of these. Where if you'll remember, we always use 30 min of.13:11Data in this setting in all of our health rules. Today, I'm only gonna use 1 min because I don't want to wait 30 min for each time for this to fire when I'm doing validation. So I'm telling it only look at 1 min of data.13:30And if it's less than a hundred, make it fire, ok? 1st thing we're gonna do, I've got the policy, I'm gonna show you our policy, ok? We've got a policy that says if a violation starts.13:46Or ends in any way ends are canceled. Send that data to Slack. But I also want you to see this. If you've never done custom events, in this case, you can see we have a custom event type of SAP integration, and we're doing some property matches. This is how.14:04I configure a policy.14:24To look for a custom event and match on the criteria within the custom event and have that go through the policy. So we're gonna test two things at once here, ok? I'm gonna 1st go up to the health rule. We're gonna enable this health rule, which means it's.14:26Going to fire.14:47In a minute. After a minute's worth of data. We're gonna come back to this window and I'm gonna go to the host. In here, I'm just gonna simply show you and, and, for the sake of the people on this call, you can see we're right. We're just doing a, a script.14:48Yep.15:08To write a custom event. Simple shell script, it's doing a post to our API, and, and we'll we'll have to clean that up a little bit, obviously, but then once we run this, it's going to post data to our controller. That's.15:09Trained in there. So again, I'm gonna write this custom event. You can see the success, the success criteria came out there. I'm gonna go back to the controller. We're gonna go back to the health rule. So we've generated a custom event, and we've got our health rule going.15:291st I'm gonna show you the custom event. If, if I go to dashboards and I go last 5 min, you can see we have a notification for our custom event. Here's what a custom event looks like in app dynamics. You can see our script.15:49That ran.16:09It sent these values over and again these are the values I'm doing a property match in my policy, if you remember. So again, we've seen that that fires. I'm gonna go back to the health rule as well. And we have our 1 min.16:12Let's just refresh. It hasn't been a full minute. I expect once we have a minute's worth of data that cause we're using data from last 1 min, this is going to go unhealthy, so we're just, we're just waiting for the tool to catch up. Pardon me.16:31So we can see that it went unhealthy. If I go to evaluation events, this may take another 30 s or so, but you can see that we fired because our error percentage was, was, too low, which again it was an engineer.16:50Health rule to make it fire. I'm gonna go over to Slack now. 1st alert you can see. This is our SAP integrate. This is the custom event alert. We matched on.17:10You know, this being error, sorry, we matched on, on status fail and artifact sales order ID or sales order_sync V one. So again, Slack integration showing the custom event went through.17:29Slack integration, showing that our 1 min health rule fire example also fired. So right, there we go. We've now tested that our Slack integration works as intended. I'm gonna go back over here and we are going to this.17:48Able this health rule, ok? And because I'm gonna go back to the policy because my policy includes these events ended or canceled when that because we've disabled this health rule.18:08We are now going to get a canceled event, as you can see, I'm using the evalu So I'm simply using key key UI features to validate that certain event criteria changed as expected, so this enabled made this fire.18:27Disabling the health rule made made it cancel, ok? And I'm gonna go back and we'll have to wait a second, but that cancel should come through. So again, we're we're just giving it a little bit of time. Policies you know a.18:46Evaluate each minute. So let's, we should.19:06There is the canceled event. Why is it canceled? Well, because the difference between canceled and and closed. Because I disabled the health rule, I've turned off evaluation.19:14I've said NO longer evaluate that ratio, and because I'm NO longer evaluating it, we NO longer know it's true. And so what appd does is it clears the alert state so that if it becomes true the next time we have data, it will make a new health rule via.19:32That is what the canceled event means versus ended. And so I expected to see that.19:53So that at a high level, we we've kind of covered basic Slack integration and a custom event. I just want to go back to my policy. I wanna call a few things out here that we've talked a little bit about in previous.20:03Meetings, but, again, notice my policy has a specific health rule scope because I'm testing, I always recommend you, I like, my production policy won't just add any health rule that a person creates.20:22Gates and start sending it to ServiceNow or sending it to Slack. When I'm in full blown production mode, I'll have one production policy that manages my HTTP request template and I explicitly pick the health rules that I want to go through that template and I.20:41Use that process as to know the health rule has been tested and is ready for production. So that's another way you can kind of, you know, if I, if I had any health rule, right? It's you you want to make sure that when you're testing new things.21:01We're not sending through your existing production policy creating noise that, that, that could make people ask questions. Do your testing in a sandbox. I'll say try to sandbox it as much as possible.21:21That is really the 1st two pieces. So you can see Slack integration. Once you have the HTTP action template, then it just adheres to whatever health rules you want here. And it's going to follow. This format is defined in the body, in the in the.21:38Payload of the HTTP request template. So that format will be standardized for any type of event, whether it's an app alert and a BT alert, a tier alert or whatever.21:56So here again, we're, we're, we're gonna pivot a little bit, and we're gonna go to a different application. We NO longer need sock shop. We've demoed the custom event creation, the script. So you saw what that looks like. We've got the policy to do the match.22:11Criteria on the custom event and we've shown the Slack integration. This one, I'm gonna show a remediation action. We have a basic application. It's a one tier app, we call it pet clinic. It's one Java node, ok? If you remember the collateral that we made previously.22:31Pardon me, I have.22:51An availability health rule in here that essentially says this is the exact same health rule that we used in sock shop. So if I go back to sock shop, it's called it's named something different, but I'm gonna just show this is collateral we.22:56Made in a different meeting.23:15And I've described what it does. So we're just looking at the app agent availability metric, ok? And in this health rule, we're saying if that drops below one, this is going to be our action. And this is what a remediation action looks like.23:17When you configure.23:37For those who've never done a remediation action, want to call this out. Remediation actions are this specific selection in this in the create actions view. I want to call out that you have a script, you essentially need a relative path.23:39Off to the script, you name it. This goes into a machine agent. This does not, a remediation action runs as part of the machine agent, not the APM agent. That is how we are gonna be able to restart the app, even though the app agent is down because we're using the machinery.23:59Urgent process to do it.24:19Okay. In ours, we got a 5 min timeout, and I want to show you this. If I put an email address in here and check this box, that is how I enable adjudication, ok? For the sake of our demo, we're gonna leave this off.24:20Unchecked in ours, but I have two screenshots for you that will show you if it was checked, the exact screens you're gonna see, and I'm gonna show you those in a slide.24:40Again, but I'm gonna we're gonna do this remediation action live, ok? And watch what happens. So again, I'm in this app and, and this, this may take five to 10 min to do from start to finish, ok? I'm gonna go last 5 min. You can see my app is running like.24:53Last 15 min, not really worried about any of the events here, ok? I'm gonna go back.25:13To my host. I'm gonna do a PSEF and I'm gonna grab for Java process. Grab for java. And as you'll see, we have two Java processes running.25:19That, well, this is the one I care about, ok? This is the application. We are gonna kill this process ID, and the result will be that health rule will pick up that the availability went from one to some other value. It's NO longer one.25:35It's going to run our remediation action. That remediation action then is going to restart the app automatically, ok? So to engineer the failure, so the remediation action runs, we're gonna do akill-nine and Akil.25:53The pet clinic app. So, pseudo it ok?26:11I'm gonna rerun our grep command and you can see this one is NO longer running, ok? The app is now down. Now we're gonna let appd do its thing, ok? We're gonna see over the last 5 min, this data is going to slowly.26:26Trickle out, the app is, is going down. So bear with me the rest of this presentation is a little bit more waity to watch the collateral that I just showed you, the remediation action.26:46Will not, will now run, ok? But this might take another 10 min or so for everything. So this is full automated remediation based on a known a known issue if you had a script that needed to be or or something could be run to.27:04Mix it.27:23And it's always that every time this happens, you can use a remediation action to, to do some more advanced stuff. So I'm gonna go to tiers and nodes, bear with me here. We should start to see the app agents going down.27:24Just gonna go back over here. You can see the the process.27:56When I go to the host, this process is still not running, so the app is, is definitely not running right now. So, ok, now that the app agent status went from 100 % to 75 in the last 5 min, the only way it could go from 100 to 75 in last 5 min.28:02Next is if we're in a downward trend.28:22Or an upward trend, obviously, in that same but it's going to go down to 50 shortly as we get more data and it's eventually gonna go to zero, ok? So you can see the data though has stopped flowing in. We didn't get the last 1 min.28:25Of data out of the 5 min total. So I know that appd is properly telling us that the availability is going down. If I go to alert and respond and I look at this health rule.28:44I just wanna call out. You can tell the health rule knows that our criteria is unhealthy. And because we have a policy that ties this health rule to the remediation action specifically because this went unhealthy.28:59Appd is doing its thing in the background. It's going to take a little bit. But again, we're now what we're waiting for, the app is down, it start up it's still down. You can see the process, the remediation script has not yet run.29:17When it runs, what we're going to see is we're going to see this come back when I do my job, show me what Java processes are out there. We're gonna see two when the remediation script is done. We're going to see this go back to healthy once the app.29:35Is actually back up and I'm gonna see data reporting back in the controller. Just doing a quick time check here. Okay, now I've run pS-EF rep java.29:54And you can see pet clinic is now returning, we have a new process ID. Our remediation script has actually successfully run in the background. I validated that on the host.30:10Let's look at what happens here. Eventually, this is going to go back to healthy. It won't be critical anymore. It's gonna take a little bit of time and we are gonna start to see data points front loading again. So we're just, we're just giving it a little.30:26Bit more time but I wanted to talk you through how I'm, what am I doing to validate that the process is working again?30:46Our app is good here. Have we started to get any new data points? We're gonna start to see this fill out and in the health rule, we're gonna see let's check if our app status. It's still unhealthy, so we'll give it a little bit more.30:55What time as things catch up?31:14Alright, have we still haven't went healthy yet?31:33Coming back online.31:37Okay, we've got our 1st data point as the app has come back up, so things are flowing as expected. Let's check app status. Okay, notice that this health rule, it now sees that it's already back online and healthy. So we have now, our remediation action script has.32:04Run and we now see app agent availability is properly reporting for the pet clinic tier and if I go back.32:24Again, we've, we've got one data point in another minute. We're gonna have our second, we're gonna have our second data point. And so what we've essentially done now with this 3rd thing is a remediation action, ok? What if I had had.32:33This area checked though. Okay I wanna show you again because we didn't have for our remediation action, we did not have this checked a time we we had a 5 min timeout, but I did not send this to an email for approval.32:51Had I done that? This timeout matters, ok? I just want to call this out. If if the person who's going to respond to that email cannot, you know, cannot get to this fast enough, it will, it will not run. But what will that look like? And in the UI.33:11If I had had my email, address in there, I would have gotten an email that contained this 1st top left message. I would have clicked this box to run the action and then I would have had to log into app D be a valid user and I.33:30I would have had to run this, click the run this action button. So if I had had email adjudication on, I would have gotten this, click the.33:50Click the run action link here and then yes, run this action. Those, the the bottom two boxes, if an email is there in a remediation action script, that is what functionality it will add. And at a high level, that's what I have for you guys today, for the group today. I hope that.34:00You saw some cool stuff that, gets you thinking about how, how we can enhance your environment. With that, I'm gonna pass it back over to Chris.34:20Awesome job Drew. That was, I mean, actually seeing those things, you know, happening before your eyes as opposed to just conceptually talking about it. You know, there's always some risk involved with us doing that because, you know, some things might not go right and everything. So I just really appreciate you.34:31You taking the time getting that working. Calvin and Drew, that's, that's a really good example.34:51Now you guys got to see, you know, a remediation script running, you got to see a custom event pushing something in and then tying that to a policy, as well. You got to see a, a an integration with slack. But there's also a lot more than just Slack that we can integrate with.34:57Right, so if you have, you know, incident management, you know, like, you know, pager duty or ops Genie or big panda, you know, any of these incident management systems where it observes an on call.35:17Situation, you know, those are all going to be things that you're gonna wanna tie into. And you don't, you don't wanna necessarily be reliant on, if you have a a company standard, appd wants to fit into that. So we have a very versatile and, and diverse selection of what we can tie in and how we can.35:32Do or so, I provided a bunch of links for not only just incident management but also for you know message alerts like you saw with the slack, but maybe it's Microsoft Teams, maybe it's Webex teams. And then there's also for ticketing, so you can, you know, create a ticket in ServiceNow or Jira.35:52Some of those, you know, more advanced action items, and then there's also, you know, good explanations on our YouTube page where there's a little bit deeper dives on the integrations where we wanted to show you more on how this relates to.36:12The health rules, and, you know, setting up policies for them and the custom actions that would be involved and, and, you know, the various web hooks or various ways of doing so. So, awesome job on, on all of that Drew, and I want to give some time for everyone to ask some questions. We did.36:28Good to have a couple of questions come through on the Q and A and thank you for using the Q and A but we This is time to open up the floor. If you guys have any more questions, please type them in, either in the chat or Q and A Q and A preferably.36:48And again, to find that it's the three dots in the bottom right corner of Webex, and you can go ahead and type that in and everyone will get to see it and it will persist, but at the same time, if you put in the chat, that's fine too.37:03Right, I don't see any coming in so far, but you know if we progress and something comes in, don't worry, we'll we'll take a step back. We'll still answer your question. But you know until I see something come in, we're gonna go ahead and keep.37:26Going so that we're respectful to everyone's time.37:43Oh, yes, very much so. It, it is going to be record we did have a question, sorry. Yes, this is recorded. You can definitely watch it again. I know it's kind of a a heavy topic, but it's something that doesn't get enough attention. That link that I put in for this series is a playlist, and that.37:47That this video will be a part of that playlist. So you will be able to see that. Also, if you registered for it, you will get an email of where it's.38:07Saved on the, NO problem. There's a lot of stuff there, so it's, it's hard to, it's hard to make sure you you digest it all, so I definitely rewatched them as well. No problem. All right, and then I'm sorry I I was saying the recording.38:15Is if you registered for it to be here, oh NO worries, NO worries, I'm coming in late. If you registered for this session or for this series, you will get an email with the link for these videos also on the Cisco community page, which is a perfect segue for the next slide.38:35So, in addition to the community page and also, you know, everything Cisco, you know, we're into AI, data centers, observability obviously, networking, you know, we're groundbreaking on, on networking technology, so definitely.38:56Only check out the Cisco university. There's a lot of really good self paced courses that you can take. A lot of times if you're already a customer of Cisco's in one way shape or or form, NO matter what kind of product, you probably have a largely a free access to.39:16A huge amount of self self paced courses. And then if you're starting to want to start getting certified, there's certification tests that you can take there and then leading up to that certification test, there's always instructor led courses that you can sign up for, where you get hands on labs, you know, we, we give you.39:35New enablement. We show you things that you can follow along with technically, but, their formal training going towards certification is always gonna be useful building up that resume. So I highly recommend checking out the u.cisco.com and then like I said in terms of the recordings, they are going to be on our U.39:55Two page, which is YouTube.com/at Cisco observability.40:15And it is the, the channel for the customer success team of Cisco observability essentially and you know we cover a lot of topics, on, you know, the various, you know, adaptions and integrations and, and, everything Splunk.40:21Gap dynamics, and then how it starts to connect into Ali and ITSI.40:41Things like that. So highly recommend checking that out. You know, we, we do appreciate any likes, subscribes, comments, feedback, you know, we, we wanna make this for you guys. A lot of these topics that we pick are based off of your feedback. So think of it as.40:46You know, you can control the playlist, you can control what we're going to, to present and also control how help helping us making it better. You know, we we are a part of the community and you we want you guys to be successful. That's the point of of our role and we take it serious. So, definitely check that out.41:06I will go ahead. Oh, can't speak to that 1III, alright, and then we also are really like like I said, we really appreciate your guys.41:25Feedback if you yeah we would definitely like that. We we definitely appreciate your feedback, your participation. We have a Slido here. I'm also gonna put it into the chat. It's a little easier if you are on your computer and you wanna do it through there.41:45You can just go click on the link, put the passcode in there, let us know how we did today. There's a few questions there.42:06Or if you wanna do it, you know, from your phone and a little bit more anonymously or, you know, if you want to provide feed feedback with an email, you know, whichever you is your preference, you know, we would just love to hear how, how we did, how we can make it better.42:13And ultimately what you would want to see in future sessions. So, check that out, let us know, and otherwise, thank you all for joining again your participation, your being here and and checking out what we have to offer. Drew, awesome job.42:30And thank you panelists for your help and I guess have a great rest of your day. See you next time. Oh, and the next session is actually next Thursday. Take care. Bye.42:50