eStreamer eNcore for Sentinel Operations Guide v4.0.9

eStreamer eNcore for Microsoft Sentinel 3.6.8

First Published: June 1, 2017

NOTE: Works with document’s Advanced Properties “First Published” property. Click File | Properties | Advanced Properties | Custom.

Last Updated: Oct  29 2020

 

 

Table of Contents

Table of Contents. 2

About This eStreamer eNcore Operations Guide v3.6.8. 4

Revision History. 4

Conventions. 4

1  Introduction. 6

1.1  Document Purpose. 6

1.2  Background. 6

1.3  Application Summary. 6

1.3.1  eStreamer-eNcore CLI. 6

1.3.2  Cisco eStreamer eNcore for Splunk (TA-eStreamer) 6

1.3.3  Cisco eStreamer eNcore Dashboard for Splunk (eStreamer Dashboard) 7

2  eNcore CLI Prerequisites. 7

2.1  Python 2.7 Installation. 7

2.2  pyOpenSSL.. 8

2.3  EPEL Repo Dependency for RHEL.. 8

24  Running eNcore CLI on Windows. 8

3  Installing eStreamer eNcore CLI. 8

3.1  Download eStreamer-eNcore-cli-X.YY.tar.gz. 8

3.2  Extract Files. 8

3.3  Create (or copy existing) PKCS12 file. 8

3.4  Install the PKCS12 File. 8

3.6.8  Test 9

4. Running eNcore CLI. 11

5.  Configuration Options. 12

5.1  Essential Configuration. 12

5.2  Advanced Configuration Options. 14

5.3  Execution. 17

5.4  Logging. 18

6  Troubleshooting and questions. 19

6.1  Error messages. 19

6.2  Frequently Asked Questions. 19

7  Cisco Support 20

8  Appendix A: 20

8.1  FMC eStreamer Certificate Creation. 20

8.2  Example Configuration File. 23

Trademarks and Disclaimers. 25

 

 

About This eStreamer eNcore Operations Guide v3.6.8

 

Revision History

 

Conventions

This document uses the following conventions.

 

Note: Means reader take note. Notes contain helpful suggestions or references to material not covered in the manual.

Caution: Means reader be careful. In this situation, you might perform an action that could result in equipment damage or loss of data.

Warning: IMPORTANT SAFETY INSTRUCTIONS

Means danger. You are in a situation that could cause bodily injury. Before you work on any equipment, be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents. Use the statement number provided at the end of each warning to locate its translation in the translated safety warnings that accompanied this device.

SAVE THESE INSTRUCTIONS

Regulatory: Provided for additional information and to comply with regulatory and customer requirements.

 

NOTE: Available paragraph styles are listed in the Quick Styles Gallery in the Styles group on the Home tab. Alternatively, they can be accessed via the Styles window (press Alt + Ctrl + Shift + S).

1  Introduction

1.1  Document Purpose

This document seeks to outline the background and usage of the eStreamer eNcore client in order to assist users with installation and execution.

1.2  Background

The Cisco Event Streamer (i.e. eStreamer) allows users to stream system intrusion, discovery, and connection data from Firepower Management Center or managed device (i.e., the eStreamer server) to external client applications. eStreamer responds to client requests with terse, compact, binary encoded messages that facilitate high performance.

Historically, the eStreamer SDK has been wrapped with some additional code to create separate Perl applications (e.g., the Cisco eStreamer for Splunk app and the CEF agent).

eStreamer eNcore is a multi-platform, multi-process Python application that is compatible with FMC versions 6.0 and above.

1.3  Application Summary

eNcore is an all-purpose client, which requests all possible events from eStreamer, parses the binary content, and outputs events in various formats to support other SIEMs. eNcore was built from scratch in Python with a scalable and fast multi-process architecture. It supports version 6.0 of Firepower Management Center. It was built and tested on CentOS 7, but should work with any Linux distribution that supports the pre-requisites. The software will run on Windows, although, it has not been made production-ready yet.

There are three packages associated with eStreamer eNcore.

1.3.1  eStreamer-eNcore CLI for Sentinel

This is a command line interface for eStreamer eNcore. It runs standalone to request data from the FMC eStreamer server and output its data. The output data format can be:

—      key-value pairs designed to maintain compatibility with previous Splunk collectors

—      JSON

—      CEF which maintains backwards compatibility with the previous cef-agent.

The output can be streamed to files, a TCP or UDP network port, stdout.

 

2  eNcore CLI Prerequisites

The CLI version of eNcore can be run on either Python 2.7 or Python 3.6+. You must also have a means of splitting the FMC’s PKCS12 file. The default approach is to install pyOpenSSL and let eNcore do the work for you.

Note: The encore.sh script should guide you through all these points if you wish to get going immediately, but it is worth being familiar with these points prior to install.

 

To check whether Python2.7 is present, use following command:

which python

 

To test where Python2.7 is present, use the following command.

whereis python

 

Note: If you are installing the CLI version on a device running Splunk, then it is worth noting that Splunk has its own version of Python. The Splunk Python has been compiled differently from the normal distribution – specifically, it is built with PyUnicodeUCS2. The encore.sh script will detect this and warn you. If you encounter this problem, then you will need to create a new user and run eStreamer-eNcore as that user. You should consider running the Splunk add on instead.

To check for pyOpenSSL, use the following command:

pip list | grep -i pyOpenSSL

Alternatively using the python3 version will no longer require the pyUnicodeUS4 complication.  To access the python3 branch perform the following

git checkout python3

 

2.1  Python 2.7 Installation

Use the following command to install Python on CentOS:

sudo yum install python

 

2.2  pyOpenSSL

Install pyOpenSSL as follows:

sudo yum install python-pip python-devel openssl-devel gcc
sudo pip install pyOpenSSL

If using python3 branch then run the following

sudo pip3 install pyOpenSSL

2.3  EPEL Repo Dependency for RHEL

If you are having problems installing these packages, then you may need to enable the EPEL repository. Instructions for installing and enabling the EPEL repository are available on the World Wide Web.

2.4  Running eNcore CLI on Azure

Create a new Linux resource such as Ubuntu 18.04 LTS:

 

Assign CPU(s) to the Virtual Instance. eNcore CLI can support up to 12 threads, we recommend 8-16 cores compute optimized, eNcore CLI can support up to 7k events/second using 16 CPU F16s_v2 option.  Scale according to expected volume of your organization, the minimum recommended number of CPUs is 4 for low volume (>500 ev/sec) operations.

 

Name your instance and download the pem certificates

Make note of the Public IP assigned to your instance, you will use this to create a certificate in the FMC eStreamer

Connect to the Command Line version of your instance using the pem file. Now you are ready to proceed with the installation.  Azure also has a shortcut to enable a quick command line connection.

ssh -I <private key path> azureuser@<public ip>

2.5  Running eNcore CLI on Windows

Warning:       Windows is not yet supported for production execution. If, however, you wish to attempt an install for the CLI version, then you will need to run the following commands.

pip install pyOpenSSL
pip install win-inet-pton

 

 

3  Installing eStreamer eNcore CLI

3.1  Build the eNcore Client from Source

Use the following command to copy the file from your local machine to the target device:

git clone https://github.com/CiscoSecurity/fp-05-microsoft-sentinel-connector.git

The project can also be downloaded to zip or

3.3  Create (or copy existing) PKCS12 file

See Appendix A for instructions on how to create a PKCS12 file in the FMC and download it.

3.4  Install the PKCS12 File

Use the following command to securely copy the pkcs12 file to the eNcore CLI installation.

scp -i /path/to/pem/encore-demo-2_key.pem /local/path/<public ip>.pkcs12 azureuser@<Public     Ip>:/tmp/

Copy the certificate from /tmp to the runtime path of the git project
   
cp /tmp/client.pkcs12 ~/fp-05-microsoft-sentinel-connector

3.6.8  Test

Change the working directory to /using the following command:

cd ~/fp-05-microsoft-sentinel-connector

Then, run the encore shell script – you will be guided through any additional configuration:

./encore.sh test

 

The script will verify that you have the pre-requisites installed, notably:

—      Python 2.7, Python 3.6+ requires “python3” branch from git

—      the correct build of Python

—      pyOpenSSL

—      a client.pkcs12 file

—      a valid host

—      It will prompt you to choose whether to output data for Splunk, CEF or JSON, in this guide we use the CEF outputter, however future versions may use JSON or other custom formats on depending on the Sentinel Connector being used

 

If there are any missing items, you will be presented with an explanation. An example explanation is in the following figure.

 

Figure 1. Choosing your output

 

Figure 2: Missing pkcs12 File

 

 

You will then be prompted to enter the IP / FQDN of the FMC and the PKCS12 file password.

 

Figure 3: Enter Password

 

 

Figure 4: Successful Test

4. Running eNcore CLI

If you run encore.sh without any parameters, you will be presented with brief instructions.

 

Figure 5: Help Screen

 

 

For your first run, run it in the foreground so you can see what is happening. Every two minutes, the screen will update with a note of how many records have been processed. If you wish to change the update frequency, see the monitor.period configuration setting.

 

Figure 6: Running in the Foreground with Monitor Status

 

 

Note: To stop the foreground process, press ctrl-c.

5.  Configuration Options

5.1  Essential Configuration

The default configuration file is set up to run out of the box. Following is a brief explanation of each setting in case you wish to customize.

5.1.1  Subscription Server

This is the FMC host and associated information. If you encounter TLS difficulties and are willing to downgrade, then you can change tlsVersion to 1.0.

       

Note: Note that downgrading the TLS version is useful for debugging and seeing the software work but it is not a recommended long-term strategy. It is recommended instead to fix the root cause.

 

Figure 8: Subscription Server Screen

    "subscription": {
        "servers": [
            {
                "host": "1.2.3.4",
                "port": 8302,
                "pkcs12Filepath": "client.pkcs12",
                "@comment": "Valid values are 1.0 and 1.2",
                "tlsVersion": 1.2
            }
        ], …

5.1.2  Monitor

The monitor is a separate thread that runs monitoring and maintenance tasks. By default, it runs every two minutes. It will report the number of events received and handled and will check the status of sub-processes. If there have been any problems, the monitor will place the client into an error state and the client will shut itself down.

 

Figure 9: Monitor Screen

    "monitor": {
        "period": 120,
        "velocity": false,
        "bookmark": false,
        "subscribed": true,
        "handled": true
    },

5.1.3  Start

The eStreamer server expects requests to state their chosen start time. There are broadly three options:

•     0: Return all data from the earliest point available on the FMC

•     1: Return all data from now onwards

•     2: Use a bookmark to pick up where we left off. First run is from 0

 

Figure 10: Start Screen

    "@startComment": "0 for genesis, 1 for now, 2 for bookmark",
    "start": 2,

5.14  Outputters (Output Data Location)

Two examples of outputters are given in the figure below. Although only one outputter is required – one that sends CEF events to the Sentinel connector, it is often useful to write CEF output to local files. The second outputter shown in the figure below writes the CEF events to local files.

Figure 11: Outputters Screen

        "outputters": [
            {
                "name": "CEF",
                "adapter": "cef",
                "enabled": true,
                "stream": {
                    "uri": "udp://10.0.1.2:514",
                }
            },

            {
                "name": "CEFfile",
                "adapter": "cef",
                "enabled": true,
                "stream": {
                    "uri": "relfile:///data/data.{0}.cef",
                    "options": {
                        "rotate": false,
                        "maxLogs": 9999
                    }
                }
            }

        ]

5.2  Advanced Configuration Options

 

5.3  Execution

Various shell scripts options are available.

During installation and initial setup – or perhaps for debugging purposes it is useful to run the following commands.

./encore.sh test

And

./encore.sh foreground

In all other cases, it is expected that encore will be run in the background, for which the following commands are pertinent.

./encore.sh start
./encore.sh stop
./encore.sh restart

Figure 12: Start, Tail Log, Stop

5.4  Logging

By default, eNcore will output an estreamer.log application log in its working directory with a log level of INFO. The format of the log file can be adjusted using the logging.format configuration setting. The level can also be adjusted. It is recommended that the default settings are left in place for production execution.

6  Sending data to Sentinel

6.1 Configuring Encore to Stream UDP

        Configure encore to stream CEF data using UDP on port 514

        If encore is already in process use the encore.sh stop/start command to restart encore

6.2 Creating a Sentinel Workspace

 

Once you’ve established a working eNcore client between the FMC and your Azure instance you can route your data outputs to Sentinel using an agent collector

If you don’t have a Sentinel Workspace proceed with the following.

6.2  Setting up the CEF Data Connector

Once you’ve established a working eNcore client between the FMC and your Azure instance you can route your data outputs to Sentinel using an agent collector

Please refer the official Microsoft guide (https://docs.microsoft.com/en-us/azure/sentinel/connect-cef-agent?tabs=rsyslog) , accessing

Accessing the connector documentation guide directly from Sentinel is preferred as the docs and prepopulated commands will contain workspace and primary key information specific to your Azure instance.  The following steps below are directly from the Azure Sentinel setup guide for reference, again it is better to use direct documentation with the Sentinel platform since it contains the exact command and workspace/primary ids that will need to be run when installing the agent collector.

Run the deployment script

1.      From the Azure Sentinel navigation menu, click Data connectors. From the list of connectors, click the Common Event Format (CEF) tile, and then the Open connector page button on the lower right.

2.      Under 1.2 Install the CEF collector on the Linux machine, copy the link provided under Run the following script to install and apply the CEF collector, or from the text below:

sudo wget https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/CEF/cef_installer.py&&sudo python cef_installer.py [WorkspaceID] [Workspace Primary Key]

3.      While the script is running, check to make sure you don't get any error or warning messages.

 Note

Using the same machine to forward both plain Syslog and CEF messages

If you plan to use this log forwarder machine to forward Syslog messages as well as CEF, then in order to avoid the duplication of events to the Syslog and CommonSecurityLog tables:

1.      On each source machine that sends logs to the forwarder in CEF format, you must edit the Syslog configuration file to remove the facilities that are being used to send CEF messages. This way, the facilities that are sent in CEF won't also be sent in Syslog. See Configure Syslog on Linux agent for detailed instructions on how to do this.

2.      You must run the following command on those machines to disable the synchronization of the agent with the Syslog configuration in Azure Sentinel. This ensures that the configuration change you made in the previous step does not get overwritten.
sudo su omsagent -c 'python /opt/microsoft/omsconfig/Scripts/OMS_MetaConfigHelper.py --disable'

https://docs.microsoft.com/en-us/azure/sentinel/connect-cef-agent?tabs=rsyslog

After running the validation script you should be able to see data coming into the Azure Sentinel Analytics screen

Note:  Seeing the messing Received CEF message in agent (incoming port 25226) is an indicator that the validation and configuration of the agent was successful

7  Troubleshooting and questions

7.1  Error messages

As far as possible, eNcore has been engineered to provide meaningful error messages. Below is an example error message.

Figure 13: Example Error Message

The eStreamer service has closed the connection. There are a number of possible causes which may show above in the error log.

If you see no errors then this could be that
 * the server is shutting down
 * there has been a client authentication failure (please check that your outbound IP address matches that associated with your certificate - note that if your device is subject to NAT then the certificate IP must match the upstream NAT IP)
 * there is a problem with the server. If you are running FMC v6.0, you may need to install "Sourcefire 3D Defense Center S3 Hotfix AZ 6.1.0.3-1")

 

If you encounter errors that do not make sense or require further explanation, please contact support so that we can fix the problem and improve the error messages.

Microsoft Sentinel Agent install:   If you encounter issues install the Microsoft agent on Azure then  try reinstalling the OMS
 https://support.microsoft.com/en-us/help/4131455/how-to-reinstall-operations-management-suite-oms-agent-for-linux

7.2  Frequently Asked Questions

Can I output my data to a different server?

Yes. Currently eNcore only writes to the filesystem, but you could mount an NFS or SMB share and specify its path as above. This may impact performance.

 

Can I run more than one instance?

Yes, using the CLI version. Although currently the encore.sh shell script only supports one instance. The underlying Python program prefixes temporary files (e.g., metadata, certificates, bookmarks) with the host and port. You will also need to update the outputter locations (e.g., [Splunk] … directory = splunk) in order to avoid data collision. If you wish to run more than one instance we recommend you extract additional copies of eStreamer-eNcore and configure separately in order to avoid changing encore.sh.

 

Can I connect to more than one FMC?

Currently not within a single instance. However, you can configure multiple instances as above.

 

Can eNcore de-duplicate data to keep my SIEM costs lower?

Not today. It is on the roadmap.

 

Can I run two instances of eNcore in a HA pair?

Yes and no. It is technically possible to run two side-by-side, but they will be completely ignorant of each other and output double the data. It may be preferable to run them in a hot-stand-by configuration where the primary client’s state and configuration data is regularly copied to the secondary client. The state and configuration data in question is estreamer.conf; x.x.x.x-port_bookmark.dat; x.x.x.x-port_cache.dat; x.x.x.x-port_pkcs.cert; x.x.x.x-port_pkcs.key; x.x.x.x-port_status.dat

 

Can I increase the logging granularity?

Yes, change logging.level in the conf file. Please note that while it is possible to increase this level to VERBOSE, the performance impact will be crippling. DEBUG may be useful but slow. We strongly recommend not going above INFO for standard production execution.

 

8 Cisco Support

Support is provided by Cisco TAC.

9  Appendix A:

9.1  FMC eStreamer Certificate Creation

Steps to generate an eStreamer client certificate are as follows:

Navigate to the web interface of the FMC – https://fmc-ip-address and log in with your FMC credentials.

In the FMC 6.x GUI, navigate to System > Integration > eStreamer

Figure 14: FMC eStreamer Certificate Creation

 

 

Click Create Client. Provide the Hostname and password.

Note: This should be the IP of the client, which will be collecting the event data from the FMC. This password will be required when you first execute eStreamer eNcore.
Please note that the IP address you enter here must be the IP address of the eStreamer-eNcore client from the perspective of the FMC. In other words, if the client is behind a NAT device, then the IP address must be that of the upstream NAT interface.

 

Figure 15: Create Client Hostname and Password Screen

 

 

Click Save.

Figure 16: Create Client Save Screen

 

 

Download the pkcs12 file.

 

Figure 17: Download Screen

 

 

Copy the pkcs12 file to the desired location in the target device. By default, eStreamer-eNcore will look for /path/eStreamer_eNcore/client.pkcs12. If you wish to use a different filename, then you must edit the estreamer.conf file.

9.2  Example Configuration File

Figure 18: Example Configuration File

{
    "connectTimeout":  10,
    "responseTimeout":  10,

    "@startComment": "0 for genesis, 1 for now, 2 for bookmark",
    "start": 2,

    "monitor": {
        "period": 120,
        "velocity": false,
        "bookmark": false,
        "subscribed": true,
        "handled": true
    },

    "logging": {
        "@comment": "Levels include FATAL, ERROR, WARNING, INFO, DEBUG, VERBOSE and TRACE",
        "level": "INFO",
        "format": "%(asctime)s %(name)-12s %(levelname)-8s %(message)s",
        "stdOut": true,
        "filepath": "estreamer.log"
    },

    "@queueComment": [
        "Maximum number of messages buffered before throttling takes place. The more powerful",
        "your CPU and more RAM you have, the larger this number can be. It's essentially a",
        "buffer size. Beyond a certain size you won't see any performance gain and it will",
        "just take longer to stop"
    ],

    "maxQueueSize": 100,

    "subscription": {
        "servers": [
            {
                "host": "1.2.3.4",
                "port": 8302,
                "pkcs12Filepath": "client.pkcs12",
                "@comment": "Valid values are 1.0 and 1.2",
                "tlsVersion": 1.2
            }
        ],

        "records": {
            "@comment": [
                "Just because we subscribe doesn't mean the server is sending. Nor does it mean",
                "we are writing the records either. See handler.records[]"
            ],
            "packetData": true,
            "extended": true,
            "metadata": true,
            "eventExtraData": true,
            "impactEventAlerts": true,
            "intrusion": true,
            "archiveTimestamps": true
        }
    },

    "handler": {
        "records": {
            "core": true,
            "metadata": true,
            "flows": true,
            "packets": true,
            "intrusion": true,
            "rua": true,
            "rna": true,

            "@includeComment": "These records will be included regardless of above",
            "include": [],

            "@excludeComment": [
                "These records will be excluded regardless of above (overrides 'include')",
                "e.g. to exclude flow and IPS events use [ 71, 400 ]"
            ],
            "exclude": []
        },

        "@comment": "If you disable all outputters it behaves as a sink",
        "outputters": [
           

            {
                "name": "CEF",
                "adapter": "cef",
                "enabled": true,
                "stream": {
                    "uri": "udp://10.0.1.2:514",
                }
            },

            {
                "name": "CEFfile",
                "adapter": "cef",
                "enabled": true,
                "stream": {
                    "uri": "relfile:///data/data.{0}.cef",
                    "options": {
                        "rotate": false,
                        "maxLogs": 9999
                    }
                }
            }

        ]
    }
}

Trademarks and Disclaimers

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.

All printed copies and duplicate soft copies are considered un-Controlled copies and the original on-line version should be referred to for latest version.

Cisco has more than 200 offices worldwide. Addresses, phone numbers, and fax numbers are listed on the Cisco website at www.cisco.com/go/offices.

 

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1110R)

 

© 2020 Cisco Systems, Inc. All rights reserved.