The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This document describes how to configure URL Filtering on the Cisco Email Security Appliance (ESA) and best practices for its use.
Control and protection against malicious or undesirable links are incorporated into the anti-spam, outbreak, content, and message filtering processes in the work queue. These controls:
When you configure URL Filtering on the ESA, you must also configure other features dependent upon your desired functionality. Here are some typical features that are enabled alongside URL Filtering:
Note: As of AsyncOS 11.1 for Email Security, support for URL scanning in attachments is now available. You can now configure your appliance to scan for URLs in message attachments and perform configured actions on such messages. You can use the URL Reputation and URL Category content and message filters to scan for URLs in message attachments. For more details, see the “Using Message Filters to Enforce Email Policies”, “Content Filters” and “Protecting Against Malicious or Undesirable URLs” chapters in the user guide or online help.
Note: Additionally as of AsyncOS 11.1 for Email Security, support for URL filtering support for shortened URLs now available. You can now configure your appliance to perform URL filtering on shortened URIs, and retrieve the actual URL from the shortened URL. Based on the URL reputation score of the original URL, a configured action is taken on the shortened URL. To enable URL filtering for shortened URLs in your appliance, see the “Protecting Against Malicious or Undesirable URLs” chapter in the user guide or online help and the CLI Reference Guide for AsyncOS for Cisco Email Security Appliance.
In order to implement URL Filtering on the ESA, you must first enable the feature. URL Filtering can be enabled from GUI or CLI by the ESA administrator.
To enable URL Filtering with the use of the GUI, navigate to Security Services > URL Filtering > Enable:
From the CLI, run the command, websecurityconfig:
myesa.local> websecurityconfig
Enable URL Filtering? [N]> y
Note: URL Logging is a sub-feature from with-in VOF. This is a CLI-only feature that must be enabled as shown here, using outbreakconfig:
myesa.local> outbreakconfig
Outbreak Filters: Enabled
Choose the operation you want to perform:
- SETUP - Change Outbreak Filters settings.
- CLUSTERSET - Set how the Outbreak Filters are configured in a cluster.
- CLUSTERSHOW - Display how the Outbreak Filters are configured in a cluster.
[]> setup
Outbreak Filters: Enabled
Would you like to use Outbreak Filters? [Y]>
Outbreak Filters enabled.
Outbreak Filter alerts are sent when outbreak rules cross the threshold (go above or
back down below), meaning that new messages of certain types could be quarantined
or will no longer be quarantined, respectively.
...
Logging of URLs is currently disabled.
Do you wish to enable logging of URL's? [N]> y
Logging of URLs has been enabled.
The Outbreak Filters feature is now globally enabled on the system. You must use the
'policyconfig' command in the CLI or the Email Security Manager in the GUI to enable
Outbreak Filters for the desired Incoming and Outgoing Mail Policies.
Note: Ensure that you commit any and all changes to your configuration before you proceed from either the GUI or the CLI on your ESA.
Enabling URL filtering support for shortened URLs is able to be done by CLI only, using websecurityadvancedconfig:
myesa.local> websecurityadvancedconfig
...
Do you want to enable URL filtering for shortened URLs? [N]> Y
For shortened URL support to work, please ensure that ESA is able to connect to following domains:
bit.ly, tinyurl.com, ow.ly, tumblr.com, ff.im, youtu.be, tl.gd, plurk.com, url4.eu, j.mp, goo.gl, yfrog.com, fb.me, alturl.com, wp.me, chatter.com, tiny.cc, ur.ly
Cisco recommends having this enabled for URL filtering configuration best practices. Once enabled, the mail logs will reflect anytime a shortened URL is used with-in the message:
Mon Aug 27 14:56:49 2018 Info: MID 1810 having URL: http://bit.ly/2tztQUi has been expanded to https://www.wired.com/?p=2270330&drafts-for-friends=js-1036023628&post_type=non-editorial
Once URL filtering is enabled as described in this article, from the mail logs example above, we can see the bit.ly link recorded AND the original link that it expands out to also recorded.
When you enable URL filtering alone, it does not take action against messages that might contain live and valid URLs.
The URLs included in inbound and outbound messages are evaluated. Any valid string for a URL is evaluated, to include strings with these components:
When the system evaluates URLs in order to determine whether a message is spam, if necessary for load management, it prioritizes and screens inbound messages over outbound messages.
You can perform actions on messages based on the reputation or category of URLs in the message body and message attachments. If you want to perform any action other than modifying URLs or their behavior, add a URL Reputation or URL Category condition and select the reputation scores or URL categories for which you want to apply the action.
For example, if you want to apply the Drop (Final Action) action to all messages that include URLs in the Adult category, add a condition of type URL Category with the Adult category selected.
If you do not specify a category, the action you choose is applied to all messages.
URL reputation score ranges for clean, neutral, and malicious URLs are predefined and not editable. However, you can specify a custom range instead. The specified endpoints are included in the range you specify. For example, if you create a custom range from -8 to -10, then -8 and -10 are included in the range. Use “No Score” for URLs for which a reputation score cannot be determined.
In order to quickly scan URLs and take action, you can create a content filter so that if the message has a valid URL, then the action is applied. From the GUI, navigate to Mail Policies > Incoming Content Filters > Add Filter.
This example shows a scan for malicious URLs with the implementation of this inbound content filter:
With this filter in place, the system scans for a URL with a Malicious reputation (-10.00 to -6.00), adds a log entry to the mail logs, uses the defang action in order to make the link un-clickable, and places this into a URL Filtering quarantine. Here is an example from the mail logs:
Wed Nov 5 21:27:18 2014 Info: Start MID 186 ICID 606
Wed Nov 5 21:27:18 2014 Info: MID 186 ICID 606 From: <bad_user@that.domain.net>
Wed Nov 5 21:27:18 2014 Info: MID 186 ICID 606 RID 0 To: <joe.user@goodmailguys.com>
Wed Nov 5 21:27:18 2014 Info: MID 186 Message-ID '<COL128-W95DE5520A96FD9D69FAC2D9D840@phx.gbl>'
Wed Nov 5 21:27:18 2014 Info: MID 186 Subject 'URL Filter test malicious'
Wed Nov 5 21:27:18 2014 Info: MID 186 ready 2230 bytes from <bad_user@that.domain.net>
Wed Nov 5 21:27:18 2014 Info: MID 186 matched all recipients for per-recipient policy DEFAULT in the inbound table
Wed Nov 5 21:27:18 2014 Info: ICID 606 close
Wed Nov 5 21:27:19 2014 Info: MID 186 interim verdict using engine: CASE spam positive
Wed Nov 5 21:27:19 2014 Info: MID 186 using engine: CASE spam positive
Wed Nov 5 21:27:19 2014 Info: ISQ: Tagging MID 186 for quarantine
Wed Nov 5 21:27:19 2014 Info: MID 186 interim AV verdict using Sophos CLEAN
Wed Nov 5 21:27:19 2014 Info: MID 186 antivirus negative
Wed Nov 5 21:27:19 2014 Info: MID 186 URL http:// peekquick .com /sdeu/cr.sedin/sdac/denc.php has reputation -6.77 matched url-reputation-rule
Wed Nov 5 21:27:19 2014 Info: MID 186 Custom Log Entry: <===> MALICIOUS URL! <===>
Wed Nov 5 21:27:19 2014 Info: MID 186 URL http:// peekquick .com/sdeu/cr.sedin/sdac/denc.php has reputation -6.77 matched url-reputation-defang-action
Wed Nov 5 21:27:19 2014 Info: MID 186 URL http:// peekquick .com /sdeu/cr.sedin/sdac/denc.php has reputation -6.77 matched url-reputation-defang-action
Wed Nov 5 21:27:19 2014 Info: MID 186 rewritten to MID 187 by url-reputation-defang-action filter '__MALICIOUS_URL__'
Wed Nov 5 21:27:19 2014 Info: Message finished MID 186 done
Wed Nov 5 21:27:19 2014 Info: MID 187 Outbreak Filters: verdict positive
Wed Nov 5 21:27:19 2014 Info: MID 187 Threat Level=5 Category=Phish Type=Phish
Wed Nov 5 21:27:19 2014 Info: MID 187 rewritten URL u'http:// peekquick .com/sdeu/cr.sedin/sdac/denc.php-Robert'
Wed Nov 5 21:27:19 2014 Info: MID 187 rewritten to MID 188 by url-threat-protection filter 'Threat Protection'
Wed Nov 5 21:27:19 2014 Info: Message finished MID 187 done
Wed Nov 5 21:27:19 2014 Info: MID 188 Virus Threat Level=5
Wed Nov 5 21:27:19 2014 Info: MID 188 quarantined to "Outbreak" (Outbreak rule:Phish: Phish)
Wed Nov 5 21:27:19 2014 Info: MID 188 quarantined to "URL Filtering Quarantine" (content filter:__MALICIOUS_URL__)
Wed Nov 5 21:28:20 2014 Info: SDS_CLIENT: Generated URL scanner configuration
Wed Nov 5 21:28:21 2014 Info: SDS_CLIENT: URL scanner enabled=1
Wed Nov 5 21:28:21 2014 Info: SDS_CLIENT: Generated URL scanner configuration
Wed Nov 5 21:28:21 2014 Info: SDS_CLIENT: URL scanner enabled=1
Note: The URL that is embedded in the previous example has extra spaces included in the URL body, so it does not trip any web scans or proxy detection.
This URL for peekquick.com is MALICIOUS and scored at a -6.77. An entry is made in the mail logs, where you can see all of the processes in action. The URL filter detected the malicious URL, defanged, and quarantined it. The VOF also scored it positive based on its rule set and provided details that this was a related Phish.
If VOF is not enabled, the same message is processed through, but URL scans are not acted upon without the added ability of VOF to drive scans and action. However, in this example the message body is scanned by the Cisco Anti-Spam Engine (CASE) and deemed as spam-positive:
Wed Nov 5 21:40:49 2014 Info: Start MID 194 ICID 612
Wed Nov 5 21:40:49 2014 Info: MID 194 ICID 612 From: <bad_user@that.domain.net>
Wed Nov 5 21:40:49 2014 Info: MID 194 ICID 612 RID 0 To: <joe.user@goodmailguys.com>
Wed Nov 5 21:40:49 2014 Info: MID 194 Message-ID '<COL128-W145FD8B772C824CEF33F859D840@phx.gbl>'
Wed Nov 5 21:40:49 2014 Info: MID 194 Subject 'URL Filter test malicious'
Wed Nov 5 21:40:49 2014 Info: MID 194 ready 2230 bytes from <bad_user@that.domain.net>
Wed Nov 5 21:40:49 2014 Info: MID 194 matched all recipients for per-recipient policy DEFAULT in the inbound table
Wed Nov 5 21:40:50 2014 Info: ICID 612 close
Wed Nov 5 21:40:50 2014 Info: MID 194 interim verdict using engine: CASE spam positive
Wed Nov 5 21:40:50 2014 Info: MID 194 using engine: CASE spam positive
Wed Nov 5 21:40:50 2014 Info: ISQ: Tagging MID 194 for quarantine
Wed Nov 5 21:40:50 2014 Info: MID 194 interim AV verdict using Sophos CLEAN
Wed Nov 5 21:40:50 2014 Info: MID 194 antivirus negative
Wed Nov 5 21:40:50 2014 Info: MID 194 queued for delivery
Wed Nov 5 21:40:52 2014 Info: RPC Delivery start RCID 20 MID 194 to local IronPort Spam Quarantine
Wed Nov 5 21:40:52 2014 Info: ISQ: Quarantined MID 194
Wed Nov 5 21:40:52 2014 Info: RPC Message done RCID 20 MID 194
Wed Nov 5 21:40:52 2014 Info: Message finished MID 194 done
This detection via CASE alone does not always occur. There are times when CASE and IPAS rules might contain that match against a certain sender, domain, or message contents in order to detect this threat alone.
Neutral URL reputation means that URLs are currently clean, but may turn malicious in the future, as they are prone to attacks. For such URLs, administrators can create non-blocking policies, for example, redirecting them to the Cisco Web Security Proxy for click-time evaluation.
Note: In AsyncOS 9.7 for Email Security and later, URLs that were formerly labeled “Suspicious” are now labeled “Neutral.” Only the labeling has changed; the underlying logic and processing have not changed.
This example shows a scan for neutral URLs with the implementation of this inbound content filter:
With this filter in place, the system searches for a URL with a Neutral reputation (-5.90 to 5.90) and adds a log entry to the mail logs. This example shows a modified subject in order to prepend "[NEUTRAL URL!]". Here is an example from the mail logs:
Wed Nov 5 21:22:23 2014 Info: Start MID 185 ICID 605
Wed Nov 5 21:22:23 2014 Info: MID 185 ICID 605 From: <bad_user@that.domain.net>
Wed Nov 5 21:22:23 2014 Info: MID 185 ICID 605 RID 0 To: <joe.user@goodmailguys.com>
Wed Nov 5 21:22:23 2014 Info: MID 185 Message-ID '<D0804586.24BAE%bad_user@that.domain.net>'
Wed Nov 5 21:22:23 2014 Info: MID 185 Subject 'Middle of the road?'
Wed Nov 5 21:22:23 2014 Info: MID 185 ready 4598 bytes from <bad_user@that.domain.net>
Wed Nov 5 21:22:23 2014 Info: MID 185 matched all recipients for per-recipient policy DEFAULT in the inbound table
Wed Nov 5 21:22:24 2014 Info: MID 185 interim AV verdict using Sophos CLEAN
Wed Nov 5 21:22:24 2014 Info: MID 185 antivirus negative
Wed Nov 5 21:22:24 2014 Info: MID 185 URL https:// www. udemy.com/official-udemy-instructor-course/?refcode=slfgiacoitvbfgl7tawqoxwqrdqcerbhub1flhsmfilcfku1te5xofictyrmwfcfxcvfgdkobgbcjv4bxcqbfmzcrymamwauxcuydtksayhpovebpvmdllxgxsu5vx8wzkjhiwazhg5m&utm_campaign=email&utm_source=sendgrid.com&utm_medium=email has reputation -5.08 matched url-reputation-rule
Wed Nov 5 21:22:24 2014 Info: MID 185 Custom Log Entry: <===> NEUTRAL URL! <===>
Wed Nov 5 21:22:24 2014 Info: MID 185 Outbreak Filters: verdict negative
Wed Nov 5 21:22:24 2014 Info: MID 185 queued for delivery
Wed Nov 5 21:22:24 2014 Info: New SMTP DCID 26 interface 192.168.0.199 address 192.168.0.200 port 25
Wed Nov 5 21:22:24 2014 Info: Delivery start DCID 26 MID 185 to RID [0]
Wed Nov 5 21:22:24 2014 Info: Message done DCID 26 MID 185 to RID [0] [('X-IronPort-AV', 'E=Sophos;i="5.07,323,1413259200"; \r\n d="scan\'208,217";a="185"'), ('x-ironport-av', 'E=Sophos;i="5.07,323,1413244800"; \r\n d="scan\'208,217";a="93843786"')]
Wed Nov 5 21:22:24 2014 Info: MID 185 RID [0] Response '2.0.0 Ok: queued as 0F8F9801C2'
Wed Nov 5 21:22:24 2014 Info: Message finished MID 185 done
Note: The URL that is embedded in the previous example has extra spaces included in the URL body, so it does not trip any web scans or proxy detection.
The Udemy link in the previous example does not appear clean, and it is scored NEUTRAL at -5.08. As shown in the mail logs entry, this message is allowed to be delivered to the end-user.
The administrator may not wish to take the broad range of neutral (-5.90 to 5.90) as an indicator. It may be more appropriate to have a custom range with a smaller range to lean more towards negative neutral scoring, as to not trigger against all URLs that fall within the neutral range and possibly creating a false negative/false positive action.
This example shows a scan for clean URLs with the implementation of this inbound content filter:
With this filter in place, the system searches for a URL with a clean reputation (6.00 to 10.00) and simply adds a log entry to the mail logs in order to trigger and record the Web-Based Reputation Score (WBRS). This log entry also helps to identify the process that is triggered. Here is an example from the mail logs:
Wed Nov 5 21:11:10 2014 Info: Start MID 182 ICID 602
Wed Nov 5 21:11:10 2014 Info: MID 182 ICID 602 From: <bad_user@that.domain.net>
Wed Nov 5 21:11:10 2014 Info: MID 182 ICID 602 RID 0 To: <joe.user@goodmailguys.com>
Wed Nov 5 21:11:10 2014 Info: MID 182 Message-ID '<D08042EA.24BA4%bad_user@that.domain.net>'
Wed Nov 5 21:11:10 2014 Info: MID 182 Subject 'Starting at the start!'
Wed Nov 5 21:11:10 2014 Info: MID 182 ready 2798 bytes from <bad_user@that.domain.net>
Wed Nov 5 21:11:10 2014 Info: MID 182 matched all recipients for per-recipient policy DEFAULT in the inbound table
Wed Nov 5 21:11:11 2014 Info: MID 182 interim AV verdict using Sophos CLEAN
Wed Nov 5 21:11:11 2014 Info: MID 182 antivirus negative
Wed Nov 5 21:11:11 2014 Info: MID 182 URL http:// www .yahoo.com has reputation 8.39 matched url-reputation-rule
Wed Nov 5 21:11:11 2014 Info: MID 182 Custom Log Entry: <===> CLEAN URL! <===>
Wed Nov 5 21:11:11 2014 Info: MID 182 Outbreak Filters: verdict negative
Wed Nov 5 21:11:11 2014 Info: MID 182 queued for delivery
Wed Nov 5 21:11:11 2014 Info: New SMTP DCID 23 interface 192.168.0.199 address 192.168.0.200 port 25
Wed Nov 5 21:11:11 2014 Info: Delivery start DCID 23 MID 182 to RID [0]
Wed Nov 5 21:11:11 2014 Info: Message done DCID 23 MID 182 to RID [0] [('X-IronPort-AV', 'E=Sophos;i="5.07,323,1413259200"; \r\n d="scan\'208,217";a="182"'), ('x-ironport-av', 'E=Sophos;i="5.07,323,1413244800"; \r\n d="scan\'208,217";a="93839309"')]
Wed Nov 5 21:11:11 2014 Info: MID 182 RID [0] Response '2.0.0 Ok: queued as 7BAF5801C2'
Wed Nov 5 21:11:11 2014 Info: Message finished MID 182 done
Wed Nov 5 21:11:16 2014 Info: ICID 602 close
Wed Nov 5 21:11:16 2014 Info: DCID 23 close
Note: The URL that is embedded in the previous example has extra spaces included in the URL body, so it does not trip any web scans or proxy detection.
As shown in the example, Yahoo.com is deemed CLEAN and given a score of 8.39, is noted in the mail logs, and is delivered to the end-user.
“No Score” is given for URLs when a reputation score cannot be determined. These may be URLs that contain new domains, or URLs that have seen little to no traffic and are not able to have a current score.
Administrators may wish to handle URLs with no score at their own discretion. If there is a seen increase in Phish-related emails and attachments, please review the URL score associated. Administrators may wish to have no score URLs redirected to the Cisco Cloud Web Security proxy service for click-time evaluation.
At times, a URL might not be classified yet, or it might be miscategorized. In order to report URLs that have been miscategorized, and URLs that are not categorized but should be, visit the Cisco URL categorization requests page.
You might also desire to check the status of submitted URLs. In order to do this, click the Status on the Submitted URLs tab of this page.
This can occur because the web site reputation and category are only two criteria among many that anti-spam and outbreak filters use in order to determine their verdicts. In order to increase the sensitivity of these filters, lower the thresholds that are required to take action, such as rewriting or replacing URLs with text, or quarantining or dropping messages.
Alternatively, you can create content or message filters based on the URL reputation score.