Content Filtering and Blacklisting

This chapter describes Content Filtering and Blacklisting policy on WEM.

This chapter provides information on the following:

Content Filtering

Content Filtering is an in-line service that enables operators to filter and control the content that an individual subscriber can access, so that subscribers are inadvertently not exposed to universally unacceptable content and/or content inappropriate as per the subscribers' preferences.

Content Filtering policy enforcement is the process of deciding if a subscriber should be able to receive some content. Typical options are to allow, block, replace or redirect the content based on the rating of the content and the policy defined for that content. For example, block of pornographic content.

Content Filtering Database Management Functions at WEM

  • Imports SFMDB/SFMDB-INC files from the MCRDBS on a time-configurable schedule
  • Processes SFMDB-INC updates from MCRDBS maintaining an updated SFMDB file
  • Merge the operator’s WBLIST database with the most recent SFMDB creating a Starent Format Customer Master Database (SFCMDB)
  • Converts the SFMDB into an Optimized Customer Master Database (OPTCMDB) suitable for distribution to CF-ST16s that need a full database
  • Computes an incremental update to the Optimized Customer Master Database (OPTCMDB-INC) suitable for updating CF-STxx and merges with OPTCMDB
  • Collects the Unknown URLs file (SFUNKDB) fromASR5K periodically and merges the files and forwards the combined SFUNKDB to the MCRDBS

URL Blacklisting

In URL Blacklisting, all HTTP/WAP URLs are matched against a database of blacklisted URLs. If there is a match, the flow is discarded, redirected, or terminated as configured. If there is no match, subscribers view content as they would normally view.URL Blacklisting policy enforcement is the process of denying access to restricted URLs.

Blacklisting Management Functions at WEM

  • Imports iwf files from the MCRDBS on a time-configurable schedule
  • Imports NCMEC blacklisted urls from NCMEC https link
  • Optimized Customer Master blacklist Database(OPTCMBLDB) suitable for distribution to CF-STxx is formed by merging iwf and ncmec bin files

Configuring Test URL for Blacklisting

You can add test URLs to the blacklist database when NCMEC is enabled at WEM. If the test URLs are present in the testurldb.pub file, WEM merges the URLs whenever it downloads the URLs from NCMEC.

You must add the test URLs to the testurldb.pub file in the <WEM_Install_Dir>/flash/blacklist/testurldb directory. For example, www.xyz.com and www.abc.com. When you add URLs to the testurldb.pub file, make sure that you add one URL per line without any space. If space is added in the URL, WEM ignores the complete URL which has space in it.

Flow diagram of Content Filtering and Blacklisting


Figure 1. WEM Content Filtering

Flow Explanation

WEM receives the CF related information from vendor rulespace through MCRDBS which is the intermediate server between WEM and Rulespace. MCRDBS converts the vendor specific format to Starent specific format.

The information of restricted URLs or blacklisted is shared by vendors NCMEC & iwffiles.WEM directly fetches URL information from NCMEC and Iwf bin files from MCRDBS. MCRDBS polls the iwf server to fetch the iwf bin files.

Directory Structure and File Format for CF and BL under the WEM Server Directory

File Formats

  1. Sfmdb flash/cf/cfdatabases/sfmdb 
    
    Format:
     sfmdb<timestamp>_<vendorno>_<majorno>_<minorno>.pub 
    
    Example:
     sfmdb20130805151040_1_023_1697.pub
    
  2. Sfmincdb flash/cf/cfdatabases/sfmincdb
    
    Format:
    sfmincdb<timestamp>_<vendorno>_<majorno>_<minorno>.pub
    Example:
    sfmincdb20130805151040_1_023_1697.pub
  3. Optcmdb flash/cf/cfdatabases/optcmdb/
    Format:
    optcmdb<timestamp>_<vendorno>_<majorno>_<minorno>.bin
    Example:
    optcmdb20130805151040_1_023_1697.bin
  4. Optcmincdb flash/cf/cfdatabases/optcmincdb
    Format:
    optcmincdb<timestamp>_<vendorno>_<majorno>_<minorno>.bin
    Example:
    optcmincdb20130805151040_1_023_1697.bin
  5. Ncmec flash/cf/blacklistdb/optcmbldb/ncmec
    Format:
    ncmec<timestamp>_<uniqueno>.bin
    Example:
    ncmec20130806035404_43.bin
  6. Iwf: flash/cf/blacklistdb/optcmbldb/iwf Format: ncmec<timestampepoch>_<uniqueno>.bin Example: iwf1374696564_83.bin
  7. Optcmbldb /flash/cf/blacklistdb/optcmbldb
    Format:
    optcmbldb<timestamp>_<uniqueno>.bin
    Example:
    optcmbldb20130731234542_60.bin
  8. Wblistdb flash/cf/cfdatabases/wblistdb
  9. Sfmdb-latest: ( This folder is used for internal purpose during conversion of starent specific format to binary file) flash/cf/cfdatabases/ sfmdb-latest/
  10. Sfmincdb-haspath: ( This folder is used for internal purpose during conversion of starent specific format to binary file) flash/cf/cfdatabases/ sfmdb-latest/

Statistics for Content Filtering and Blacklisting

Statistics for Content filtering and blacklisting are stored at the below path: <wem installation dir>/server/flash/cf/statistics

For generating the content statistics at a given interval, you must configure the cfstats time interval. This cfstats interval should be configured in the cf.cfg file located at <wem installation>/<server>/etc/cf.cfg. This time interval is configured in seconds. After you configure the cfstat time interval, statistics is generated every cfstat interval for Content Filtering. For example, if you configure CFStatsInterval = 300, then WEM collects the following statistics every 300 seconds. The default value, min value, and maximum value for the cfstats is as follows:
  • Default: 300 sec
  • Min: 60 sec
  • Max: 1800 sec

The below are the statistics information stored:

Content Filtering Statistics

  1. No. of times FULL DB files fetched from MCRDBS
  2. No. of times failed to fetch FULL DB files from MCRDBS
  3. No. of times Incremental DB files fetched from MCRDBS
  4. No. of times failed to fetch Incremental DB files from MCRDBS
  5. No. of times SFMDB converted to OPTCMDB
  6. No. of times failed to convert SFMDB to OPTCMDB
  7. No. of times SFMDB-INC converted to OPTCMDB-INC
  8. No. of times failed to convert SFMDB-INC to OPTCMDB-INC
  9. No. of times OPTCMDB file sent to IMG
  10. No. of times failed to send OPTCMDB to IMG
  11. No. of times OPTCMDB-INC file sent to IMG
  12. No. of times failed to send OPTCMDB-INC file to IMG
  13. No. of times FULL database files purged
  14. No. of times incremental database files purged
  15. No. of times FTP connection failed
  16. No. of times WBLIST merged with SFMDB
  17. No. of times WBLIST merged with SFMDB-INC
  18. No. of times failed to merge WBLIST with SFMDB
  19. No. of times failed to merge WBLIST with SFMDB-INC

Blacklisting Statistics

  1. No. of NCMEC BLACKLIST imported
  2. No. of times failed to import NCMEC BLACKLIST
  3. No. of IWF BLACKLIST imported
  4. No. of times failed to import IWF BLACKLIST
  5. No. of times Blacklist converted to OPTCMDB-BL
  6. No. of times failed to convert Blacklist to OPTCMDB-BL
  7. No. of times OPTCMDB-BL file sent to IMG
  8. No. of times failed to send OPTCMDB-BL to IMG
  9. No. of ncmec database files purged
  10. No. of iwf database files purged

Unknown URL Statistics

  1. No. of unknown URL database imported
  2. No. of unknown URL database import failed
  3. No. of unknown URL database exported

Configuring WBLIST by Operator

Below are the steps to be followed by the operator to configure WBLIST.

  1. Create a WBLIST source file as below
    • vi wblist.txt
    • Enter the URLs to be categorized in the below format (URL category)
      playboy.com DYNAM
      colt.com DYNAM
      dollarsgambling.com DYNAM
      erowid.org DYNAM
      sexetc.org DYNAM
  2. Build wblist.pub file
    • wblist tool is available at location <EMS_Server>/server/tools/
    • Generate wblist.pub file as below [root@pnqaextappsucs210-05 tools]# ./wblist -help
      WHITE BLACK LIST Utility
      Usage :
      
      -f  <path>  : path
      is wblist source file path
      
      -o  <output path>
      : WBLIST will be created at <path>/wblist.pub      
                  : [default : current path]
      
      -v  <version>  :
      WBLIST version number   : [default 1]
      
      -h : Help--------------------------------------------------------
      
      [root@pnqaextappsucs210-05 tools]#Example:[root@pnqaextappsucs210-05
      tools]#
      
      [root@pnqaextappsucs210-05
      tools]# ./wblist -f wblist.txt -o /export/home/SQA/ems/server/flash/cf/cfdatabases/wblistdb/ -v 2
      
      WHITE BLACK LIST Utility
      Setting WBLIST file
      path : /export/home/SQA/ems/server/flash/cf/cfdatabases/wblistdb//wblist.pub
      
      --------------------------------------------------------
      
      Source File: wblist.txtWBLIST
      file path: 
      
      /export/home/SQA/ems/server/flash/cf/cfdatabases/wblistdb//wblist.pub--------------------------------------------------------
      
      [root@pnqaextappsucs210-05
      tools]# 
      
      This wblist.pub automatically
      gets merged with the SFMDB if created at
      
       <EMS Server>/ server/flash/cf/cfdatabases/wblistdb/
      

Category List

The Category-based Content Filtering solution uses categories to categorize content and URLs for content filtering.

The following table lists the category codes used in the Category-based Content Filtering application. Apart from these categories, in the Content Filtering Policy Configuration Mode, runtime categories not present in the CLI can also be configured for rating.


Table 1. Category Codes and Descriptions
Category Description

ABOR

Abortion

ADULT

Adult Related Material

ADVERT

Advertising site

ANON

Anonymizer

ART

Art, Museums

AUTO

Automotive site

BACKUP

Online Backup and file storage

BLACK

Inappropriate Content

BLOG

Blogging

BUSI

Business

CAR

Career, Job Search

CDN

Content Delivery Network

CHAT

Chatting site

CMC

Virtual Community

CRIME

Criminal Skills

CULT

Cult

DRUG

Drug

DYNAM

Dynamic site, Wireless

EDU

Educational site

ENERGY

Energy

ENT

Entertainment, Music

FIN

Finance

FORUM

Forum and Messageboard

GAMB

Gambling

GAME

Gaming

GLAM

Glamour

GOVERN

Government site

HACK

Hacking

HATE

Hate Site, Hate Speech

HEALTH

Health site

HOBBY

Hobby

HOSTS

Hosting site

KIDS

Kids

LEGAL

Legal, Law site

LIFES

Lifestyle

MAIL

Webmail

MIL

Military

NEWS

News site

OCCULT

Occult

PEER

File Sharing

PERS

Personals and Dating

PHOTO

Photography

PLAG

Plagiarism

POLTIC

Politics

PORN

Pornography, Nudism, Naturism

PORTAL

Portal site

PROXY

Proxy, Test

REF

Reference site

REL

Religion

SCI

Science

SEARCH

Search, Web Search

SHOP

Online Shopping

SPORT

Sport

STREAM

Streaming Media

SUIC

Suicide

SXED

Sexual Education

TECH

Technology and Telecommunication

TRAV

Travel

VIOL

Violence

VOIP

Internet Telephony

WEAP

Weapons

WHITE

Clean Content

UNKNOW

Unknown URLs which are not present in the optimized content category database



Glossary Terms

Following are the Glossary terms used in this chapter:

  • SFMDB: Starent Format Master Database (Content filtering full database in starent format)
  • SFMDB-INC: Starent Format Master Database Incremental Update file(incremental starent format content filtering database)
  • SFUNKDB: Starent Format Unknown URL Database
  • MCRDBS: Master Content Rating Database Server - Unix server that converts vendor-supplied content rating information into a Starent format that is vendor independent. Handles both full and incremental updates. The server is responsible for distribution of Starent format data files to CFEMS servers in the field
  • OPTCMDB: Optimized Customer Master Database(Content filtering full Database in ASR5K format)
  • OPTCMDB-INC: Optimized Customer Master Database Incremental(Content filtering incremental Database in ASR5K format)
  • OPTCMBLDB: Optimized Customer Master blacklist Database