Have an account?

  •   Personalized content
  •   Your products and support

Need an account?

Create an account

Fighting Malware to the End: How We Tested and Deployed AMP for Endpoints


Published: September 2019

We use multiple solutions to keep malware off of our 125,000 Windows and Mac devices. The endpoint protection solution we’ve used for years blocks malware before it enters our network: point-in-time detection. But some malware manages to sneak through and to detect that, we also need what’s called retrospective detection.

Our earliest testing with Cisco Advanced Malware Protection (AMP) for Endpoints, against that endpoint solution, began after Cisco’s Sourcefire acquisition in 2013.  After the business unit steadily added capabilities to its threat intelligence platform, we in 2018 decided to fully evaluate AMP for Endpoints. “One motivation was to combine point-in-time and retrospective detection in one solution,” says Steve Vida, information security architect. “Another was moving endpoint protection to the cloud to reduce the on-premises infrastructure we need to manage.”

Here’s how we evaluated and deployed AMP for Endpoints, working closely with the Cisco Computer Security Incident Response Team (CSIRT). “We’re testing to extreme limits, so we can share our feedback with the business unit,” says Oliver Parvin, Cisco IT manager.

Phase 1: Monitoring only, to look for false positives

For the first three months of testing, we ran AMP for Endpoints parallel with the previous endpoint-protection software in monitor mode. We left the blocking to our previous software. “Running AMP for Endpoints in monitor mode during the test gave us confidence that when we turned on protection, we’d minimize false positives that can interfere with work and increase IT caseload,” says Parvin.

We installed connector software on 1,500 Mac and Windows devices using our standard device management software: Jamf PRO for Mac and Microsoft System Center Configuration Manager (SCCM) for Windows. We configured the connector to check the cloud for updates every 15 minutes. “Testing confirmed that the connector was stable, lightweight, and didn’t interfere with the user experience,” Vida says. “We also saw an uptick in indicators of compromise, especially adware on Macs that sneaks in on third-party applications and installers.”

The next two phases happened in parallel…

Phase 2: Mac deployment

After completing the test in monitor mode, we fast-tracked the migration of 50,000 Macs to Cisco AMP for Endpoints in protect mode and removed the previous protection software. “We felt comfortable going directly to general availability on Mac because AMP for Endpoints provides the same capabilities as the previous solution, and the test showed higher detection rates with low impact on the user experience,” Parvin says.

Transitioning all 50,000 Macs to AMP for Endpoints took less than a week. Upgrades--a single policy for the entire environment--are also quick. For example, a connector upgrade in June 2019 took just two days, compared to several weeks with our previous solution.

Phase 3: 10,000-person Windows pilot in protect mode

In parallel with the Mac migration, we conducted a 3-month, 10,000-client pilot for Windows devices. This time, we used AMP for Endpoints in protect mode, removing the previous endpoint-protection software. We made sure to include devices in regions with low-bandwidth connections and high rates of malware. We also included different functions--engineering, legal, IT, finance, etc.--to identify false positives for specialized applications. At the conclusion of the pilot, we made the decision to move forward with general deployment.

Windows pilot results:

• Doubled malware detection rate. AMP for Endpoints detected 100% more infected Windows hosts than the previous solution detected in the three months preceding the pilot. During the same time period, we compared the malware detection rate of the pilot group against that for a control group of hosts running the existing endpoint protection software and detected no increase in infection rate for the control group.

• Proactively detected software vulnerabilities. AMP for Endpoints identified critically vulnerable software in 2% of the Windows clients in the pilot. Our previous solution lacked this capability--it looked only for point-in-time threat events and said nothing about potential vulnerabilities that could be compromised by drive-by or phishing exploits. “Now on a single pane of glass, we can see threat data as well as vulnerable running processes,” says Lawrence Dsouza, security investigator for CSIRT. During the pilot, CSIRT saw that printer software package had become vulnerable. They immediately went to work to identify the root cause, which turned out to be the bundled Java runtime. “We didn’t have to wait for a scan or for another team to tell us we had a vulnerable version of Java,” says Dsouza. “With AMP for Endpoints, we saw the problem as it emerged.”

• New insights. The availability of device trajectory data was another new addition to our capability. This data was invaluable in giving us the ability to perform root cause analysis in large infection outbreaks or recurring patterns of infections in the threat data. For example, in an investigation that was conducted when a very specific malware detection was occurring daily across scores of hosts, device trajectory data (in addition to other data sources) was leveraged, leading us to the discovery that users were falling for a very simple/common social engineering ploy.

Deployment decisions

Staggered deployments

For the 10,000-user Windows pilot, we started with an early adopters’ group of a few hundred people. Next, we expanded to our user acceptance group, which has 500 people in different business units and different regions. Following that, we began general deployment, moving 5,000 users at a time.

Reducing false positives by allowing exceptions for certain groups

Some Cisco groups need to allow endpoint behavior that other groups don’t. Examples include our extranet, data center, and business units that develop security software. We met this need by creating separate groups on the AMP for Endpoints management console. Within these groups, IT administrators can control certain types of exceptions. We roll up the results from these groups when we create reports.

Turning on features one by one

“We’re turning on features gradually and observing how they work in our environment rather than turning on all the bells and whistles at once,” Parvin says. If a new feature conflicts with other features or results in false positives, we build exceptions before putting the new feature into production. A new feature we turned on recently is malicious activity protection (MAP), which detects ransomware by identifying processes that attempt to encrypt user data.

Proactive alerts

We configured AMP for Endpoints to send alerts to the IT team managing our top executives’ desktops whenever an application vulnerability is detected. Promptly patching or upgrading the application helps to prevent spear phishing attacks targeting executives.

Custom reports with Splunk integration CSIRT uses the AMP for Endpoints API to create custom reports in Splunk. The team built a dashboard that shows total events, blocked threats over different time periods, most vulnerable groups and operating systems, and the number of threats detected that are file-based, behavior-based, etc. “The dashboard highlights trends, so we can see where to focus our efforts,” Dsouza says.

Lessons learned

We offer the following suggestions for deploying AMP for Endpoints:

  • Test in-network locations with different uplink speeds to ensure AMP connector upgrades don’t interrupt user productivity. Set it to monitor mode initially. Include different types of users--like power users, task workers, and administrators in a variety of business units--to identify false positives associated with different applications, then write exceptions.
  • Turn on new security features gradually to identify system conflicts or false positives that might interfere with the user experience.
  • Stagger deployments. We typically start with a few hundred people. When we move to general deployment, we transition 5,000 users at a time.
  • Next steps

    We’re getting ready to use AMP for Endpoints exclusively on all Windows and Mac clients.

    We plan to use the AMP for Endpoints API to integrate with other security solutions. Integrating our threat intelligence platform with AMP for Endpoints, for example, will enable it to automatically enter new hashes for infected files. Currently we add the hashes manually.

    For more information

    Fighting Malware to the End: How We Tested and Deployed AMP for Endpoints (PDF)

    Cisco AMP for Endpoints

    To read additional Cisco IT business solution case studies, visit Cisco on Cisco: Inside Cisco IT