By default, the proxy cache paths are stored in the file system. We recommend changing them to in-memory drives by creating a cache location in tmpfs as shown here.

1. Create directories for the different proxy cache paths under /home.

   As an example, these directories must be created for the primary Unified CCX and Customer Collaboration Platform servers. The same steps should be followed for the secondary Unified CCX server.

   mkdir -p /home/primaryCCX/rest 
   mkdir -p /home/primaryCCX/desktop 
   mkdir -p /home/primaryCCX/shindig 
   mkdir -p /home/primaryCCX/openfire
   mkdir -p /home/CCP/ccp
   mkdir -p /home/CCP/ccpopenfire
   mkdir -p /home/primaryCUIC/cuic 
   mkdir -p /home/primaryCUIC/cuicdoc 
   mkdir -p /home/client_temp 
   mkdir -p /home/proxy_temp
   
   echo "tmpfs /home/primaryFinesse/rest tmpfs size=1510M,rw,auto,noexec,nodev,nosuid,gid=root,uid=root,mode=1700 0 0" >> /etc/fstab echo "tmpfs /home/primaryFinesse/desktop tmpfs size=20M,rw,auto,noexec,nodev,nosuid,gid=root,uid=root,mode=1700 0 0" >> /etc/fstab echo "tmpfs /home/primaryFinesse/shindig tmpfs size=500M,rw,auto,noexec,nodev,nosuid,gid=root,uid=root,mode=1700 0 0" >>
   /etc/fstab echo "tmpfs /home/primaryFinesse/openfire tmpfs size=10M,rw,auto,noexec,nodev,nosuid,gid=root,uid=root,mode=1700 0 0" >> /etc/fstab echo "tmpfs /home/primaryCUIC/cuic tmpfs size=100M,rw,auto,noexec,nodev,nosuid,gid=root,uid=root,mode=1700 0 0" >> /etc/fstab echo "tmpfs /home/primaryCUIC/cuicdoc tmpfs size=100M,rw,auto,noexec,nodev,nosuid,gid=root,uid=root,mode=1700 0 0" >> /etc/fstab echo "tmpfs /home/client_temp tmpfs size=2048M,rw,auto,noexec,nodev,nosuid,gid=root,uid=root,mode=1700 0 0" >> /etc/fstab echo "tmpfs /home/proxy_temp tmpfs
   size=2048M,rw,auto,noexec,nodev,nosuid,gid=root,uid=root,mode=1700 0 0" >> /etc/fstab

   Note	Increase the client and proxy_temp caches by 1 GB for each new Finesse cluster added to the configuration.

2. Mount the new mount points with the mount -av command.

3. Validate that the file system has mounted the new mount points with the df -h command.

4. Change the proxy_cache_path locations in the Unified CCX and Customer Collaboration Platform cache configuration files. For example, to change the paths for the Finesse primary, go to to<nginx-installdirectory>/conf.d/finesse/caches and change the existing cache location /etc/nginx/cache/<ccx_server_name>/ to the newly created filesystem location /home/primaryCCX.

   ##Must-change /etc/nginx/cache/ location would change depending on folder extraction ## Nginx config file to cache the desktop/shindig and notification service related static files. 
   proxy_cache_path /home/primaryCCX/desktop levels=1:2 use_temp_path=on 
   keys_zone=desktop_cache_primary:10m max_size=15m inactive=3y use_temp_path=off; 
   proxy_cache_path /home/primaryCCX/shindig levels=1:2 use_temp_path=on 
   keys_zone=shindig_cache_primary:10m max_size=500m inactive=3y use_temp_path=off; 
   proxy_cache_path /home/primaryCCX/openfire levels=1:2 use_temp_path=on 
   keys_zone=openfire_cache_primary:10m max_size=10m inactive=3y use_temp_path=off; 
   proxy_cache_path /home/primaryCCX/rest levels=1:2 use_temp_path=on 
   keys_zone=rest_cache:10m max_size=1500m inactive=40m use_temp_path=off;

5. Follow the same steps for the Unified CCX secondary server.

   Note	Ensure that sum of all the tmpfs drive sizings created in all the previous steps is added to the final memory sizing for the deployment. This is because these drives are memory blocks that are configured to look like disks to the application and they consume that much memory space.

Use self-signed certificates until the reverse-proxy is ready to be rolled out into production. On a production deployment, use only a Certificate Authority-signed (CA-signed) certificate.

1. Generate OpenResty Nginx certificates for SSL folder content. Before you generate certificates, you must create a folder called ssl under /usr/local/openresty/nginx. Generate two certificates (one for <reverseproxy_primary_fqdn> and another for <reverseproxy_secondary_fqdn>) with the help of the following commands:

   1. sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /usr/local/openresty/nginx/ssl/nginx.key -out /usr/local/openresty/nginx/ssl/nginx.crt (pass hostname as: <reverseproxy_primary_fqdn>)

   2. sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /usr/local/openresty/nginx/ssl/nginxnode2.key -out /usr/local/openresty/nginx/ssl/nginxnode2.crt (pass hostname as: <reverseproxy_secondary_fqdn>)

   3. Ensure that the certificate path is /usr/local/openresty/nginx/ssl/nginx.crt and /usr/local/openresty/nginx/ssl/nginxnode2.crt.

2. Change the permission of the private key 400 (r--------).

3. Configure the firewall and iptables on the reverse-proxy to enable the firewall to communicate with the ports that are configured to listen to the OpenResty Nginx server.

4. Add the IP address and hostname of all the configured servers in the /etc/hosts file of the reverse-proxy server.

Note	The provided configuration is for a sample 400 agent deployment and must be expanded appropriately for larger deployments.

A CA-signed certificate can be installed on the reverse-proxy with these steps:

1. Generate the certificate signing request (CSR).

   To generate the CSR and private key, enter openssl req -new -newkey rsa:4096 -keyout nginx.key -out nginx.csr after you log in to the proxy. Follow the prompt, and provide the details. This generates the CSR (nginx.csr in the example) and the RSA private key (nginx.key in the example) of 4096 bits.

   For example:

   
   [root@reverseproxyhost.companyname.com ssl]# openssl req -new -newkey rsa:4096 -keyout nginx.key -out nginx.csr
   Generating a RSA private key
   .....+++++
   ...................................................................................................................................................+++++
   writing new private key to 'nginx.key'
   Enter PEM pass phrase:passphrase
   Verifying - Enter PEM pass phrase:passphrase
   -----
   You are about to be asked to enter information that will be incorporated
   into your certificate request.
   What you are about to enter is what is called a Distinguished Name or a DN.
   There are quite a few fields but you can leave some blank
   For some fields there will be a default value,
   If you enter '.', the field will be left blank.
   -----
   Country Name (2 letter code) [XX]:US
   State or Province Name (full name) []:CA
   Locality Name (eg, city) [Default City]:Orange County
   Organization Name (eg, company) [Default Company Ltd]:CompanyName
   Organizational Unit Name (eg, section) []:BusinessUnit
   Common Name (eg, your name or your server's hostname) []:reverseproxyhostname.companydomain.com
   Email Address []:john.doe@comapnydomain.com
   Please enter the following 'extra' attributes
   to be sent with your certificate request
   A challenge password []:challengePWD
   An optional company name []:CompanyName

   Write down the PEM passphrase. This is used to decrypt the private key during the deployment.

2. Obtain the signed certificate from the CA.

   Send the CSR to the certificate authority and obtain the signed certificate.

   Note	If the certificate received from the CA is not a certificate chain containing all the respective certificates, compose all the relevant certificates into a single certificate chain file.

3. Deploy the certificate and key.

   Decrypt the key generated in the first step with the openssl rsa -in nginx.key -out nginx_decrypted.key command. Place the CA-signed certificate and the decrypted key inside the folder /usr/local/openresty/nginx/ssl in the reverse-proxy machine. Update or add the following SSL configurations related to the certificate in the OpenResty Nginx configurations in the following file: /usr/local/openresty/nginx/conf/conf.d/ssl/ssl.conf:

   
   ssl_certificate /usr/local/openresty/nginx/ssl/ca_signed_cert.crt;
   ssl_certificate_key/usr/local/openresty/nginx/ssl/nginx_decrypted.key;

4. Configure permissions for the certificates.

   Enter chmod 400 /usr/local/openresty/nginx/ssl/ca_signed_cert.crt and chmod 400 /usr/local/openresty/nginx/ssl/nginx_decrypted.key, so that the certificate has read-only permission and is restricted to the owner.

5. Reload OpenResty Nginx.

1. Create a custom Diffie-Hellman parameter by using the following commands:

   openssl dhparam -out /usr/local/openresty/nginx/ssl/dhparam.pem 2048

   chmod 400 /usr/local/openresty/nginx/ssl/dhparam.pem

2. Modify the server configuration to use the new parameters in the file /usr/local/openresty/nginx/conf/conf.d/ssl/ssl.conf by using the following command:

   ssl_dhparam /usr/local/openresty/nginx/ssl/dhparam.pem;

Note	In order to enable the Online Certificate Status Protocol (OCSP) stapling, the server should be using a CA-signed certificate and the server should have access to the CA which signed the certificate.

Add or update this configuration in the file: /usr/local/openresty/nginx/conf/conf.d/ssl/ssl.conf:

• ssl_stapling on;

• ssl_stapling_verify on;

The default OpenResty Nginx configuration file (/usr/local/openresty/nginx/conf/nginx.conf) has to be modified to contain these entries to enforce security and enhance performance. This content should be used to modify the default configuration file which is created during the OpenResty Nginx installation.

#user  nobody;
# Increasing number of worker processes will not increase the processing the request. The number of worker process will be same as number of cores
# in system CPU. Nginx provides "auto" option to automate this, which will spawn one worker for each CPU core.
worker_processes  auto;

# Process id file location
pid /usr/local/openresty/nginx/logs/nginx.pid;

# Binds each worker process to a separate CPU
worker_cpu_affinity auto;

#Defines the scheduling priority for worker processes. This should be calculated by "nice" command. In our proxy set up the value is 0
worker_priority 0;


error_log  /usr/local/openresty/nginx/logs/error.log info;
error_log  /usr/local/openresty/nginx/logs/blocking.log emerg;

#user root root;
#user nginxuser nginxuser;
# current limit on the maximum number of open files by worker processes, keeping 10 times of worker_connections
worker_rlimit_nofile 102400;

events {
    multi_accept on;
    
    # Sets the maximum number of simultaneous connections that can be opened by a worker process.
    # This should not be more the current limit on the maximum number of open files i.e. hard limit of the maximum number of open files for the user (ulimit -Hn) 
    # The appropriate setting depends on the size of the server and the nature of the traffic, and can be discovered through testing.
    worker_connections  10240;
    #debug_connection 10.78.95.21;
}


http {
    include       mime.types;
    
    default_type  text/plain;
    
    ## Must-change Change with DNS resolver ip in deployment
    ;
    resolver X.X.X.X;
    lua_package_path  "/usr/local/openresty/lualib/resty/?.lua;/usr/local/openresty/nginx/lua/?.lua;;";
    

    ## Must-change change proxy_temp folder as per cache directory configurations
    proxy_temp_path /usr/local/openresty/nginx/cache/proxy_temp 1 2;
    ## Must-change change client_temp folder as per cache directory configurations
    client_body_temp_path /usr/local/openresty/nginx/cache/client_temp 1 2;


    lua_shared_dict userlist 50m;
    lua_shared_dict credentialsstore 100m;
    lua_shared_dict userscount 100k;
    lua_shared_dict clientstorage 100m;
    lua_shared_dict blockingresources 100m;
    lua_shared_dict tokencache_ccxproxy 10M;
    lua_shared_dict tokencache_ccxproxy99 10M;
    lua_shared_dict ipstore 10m;
    lua_shared_dict desktopurllist 10m;
    lua_shared_dict desktopurlcount 100k;
    lua_shared_dict thirdpartygadgeturllist 10m;
    lua_shared_dict thirdpartygadgeturlcount 100k;
    lua_shared_dict corsheadersstore 100k;
    lua_shared_dict timerthreadsstore 100k;

    init_worker_by_lua_block {
        local UsersListManager = require('users_list_manager')
        local UnauthenticatedDesktopResourcesManager = require("unauthenticated_desktopresources_manager")
        local UnauthenticatedResourcesManager = require("unauthenticated_thirdpartyresources_manager")
        --  Must-change Replace saproxy.cisco.com with reverseproxy fqdn
        -- This is done so that all the apis required are prepopulated with data before the first request by starting required timers.
        if  ngx.worker.id() == 0 then
        UsersListManager.getUserList("ccx-rproxy-finesse.cisco.com", "https://ccx-rproxy-finesse.cisco.com:8445/finesse/api/Users")
        UnauthenticatedDesktopResourcesManager.getDesktopResources("ccx-rproxy-finesse.cisco.com", "https://ccx-rproxy-finesse.cisco.com:8445/desktop/api/ResourceURLs?type=desktop") 
        UnauthenticatedResourcesManager.getThirdPartyGadgetResources("ccx-rproxy-finesse.cisco.com", "https://ccx-rproxy-finesse.cisco.com:8445/desktop/api/ResourceURLs?type=3rdParty")
        end	
    }

    
    include conf.d/*.conf;

    sendfile        on;
    
    tcp_nopush     on;
     
    server_names_hash_bucket_size 512; 

}

By default, the OpenResty Nginx configuration listens on port 8445 for Unified CCX requests. At a time, only one port can be enabled from a reverse proxy to support finesse requests e.g., 8445. If 443 port needs to be supported, then check the <NGINX_HOME>/conf.d/finesse.conf in order to enable listening on 443 and disable listening on 8445.

Client SSL certificate authentication for connections from reverse-proxy hosts can be enabled on Unified CCX and Customer Collaboration Platform by using the new CVOS CLI option utils system reverse-proxy client-auth enable/disable/status.

By default this is disabled and has to be enabled by the administrator by executing the CLI on each upstream server independently. After this option is enabled, Cisco Web proxy Service running on upstream host will start authenticating client certificates in TLS handshake for connections originating from trusted reverse-proxy hosts added by using the CLI utils system reverse-proxy allowed-hosts add <proxy-host>.

The following is the configuration block for the same in proxy config files named ssl.conf and ssl2.conf.

#Must-change /usr/local/openresty/nginx/ssl/nginx.crt change this location accordingly
proxy_ssl_certificate /usr/local/openresty/nginx/ssl/nginx.crt;
#Must-change /usr/local/openresty/nginx/ssl/nginx.key change this location accordingly
proxy_ssl_certificate_key /usr/local/openresty/nginx/ssl/nginx.key;

The SSL certificate used for outbound traffic (proxy to upstream) can be the same as the SSL certificate that is configured for inbound traffic (SSL connector for component server blocks). If self-signed certificate is used as proxy_ssl_certificate, it has to be uploaded to the tomcat trust store of the upstream components (Finesse/IdS/Cisco Unified Intelligence Center/Livedata) for it to be authenticated successfully.

Upstream server certificate validation by reverse-proxy is optional and disabled by default. If you wish to achieve full TLS mutual auth between reverse-proxy and upstream hosts, the following configuration has to be uncommented in the ssl.conf and ssl2.conf files.

#Enforce upstream server certificate validation at proxy ->
#this is not mandated as per CIS buit definitely adds to security.
#It requires the administrator to upload all upstream server certificates to the proxy certificate store
#Must-Change Uncomment below lines IF need to enforce upstream server certificate validation at proxy
#proxy_ssl_verify on;
#proxy_ssl_trusted_certificate /usr/local/openresty/nginx/ssl/finesse25.crt;
proxy_ssl_trusted_certificate: This file should contain the all upstream certificate enteries concatenated together

Caveats for Configuring mutual TLS Authentication

mutual TLS (mTLS) is a standard security requirement for connections established from DMZ into the data center. For more information, see Nginx CIS behcmarks-https://www.cisecurity.org/benchmark/nginx

mTLS requires that both the server and client be pre-configured with mutual information about each other, as well as that the mutual certificates be properly verified. Hence the term Mutual TLS. A properly configured proxy server will be able to circumvent TCP rate limits and provide the client IP to the server for logging purposes. As a result, it is critical that the proxy identity be verified before connecting as a reverse proxy. For security reasons, it is therefore recommended that this feature be used and turned on.

This requires the upstream component certificates to be made available to the proxy and vice versa. Reverse proxy by default establishes verified TLS connections to the upstream server and it is the proxy verification at the client which is optional. Therefore this needs to be enabled at the upstream client server as described below

Enabling mutual TLS

The mutual TLS needs to be enabled at the upstream component servers using the provided CLI.

Use the CLI "utils system reverse-proxy client-auth enable" to enable proxy certificate verification at the upstream component server.

Then, upload the proxy SSL certificate corresponding to the reverse proxy hostname used to connect to the same server, so that it can be used to verify TLS connections when the reverse proxy attempts to establish an upstream connection.