Setting up a jCustomer cluster (Elasticsearch and Unomi)

Note: This topic applies to older versions of the stack (jExperience 1.8, jCustomer 1.2, Elasticsearch 5.6.3). You can find more recent instructions on this page.

Here are the steps followed to install an ElasticSearch and jCustomer clusters for our preproduction environment. This cluster will be used for jExperience but also for every Jahia product or module needing ElasticSearch.

General

ElasticSearch and jCustomer will be installed on a cluster of three nodes:

unomipp01
unomipp02
unomipp03

Each node has :

Ubuntu 16.04 LTS
40 GB of hard drive
4 CPUs
7 GB of memory

ES cluster

Install

At the moment, only ElasticSearch 5.6.3 can be used with Jahia. The Debian package can be downloaded here
To install it:
```
sudo dpkg -i elasticsearch-5.6.3.deb
```

To configure it, you have to edit the file /etc/elasticsearch/elasticsearch.yml

# Name of the cluster. It has to be uniq per environment
cluster.name: JahiaClusterPP
# Name of the node
node.name: unomipp01
# Host name
network.host: unomipp01.int.jahia.com
# Name of the hosts being part of the cluster
discovery.zen.ping.unicast.hosts: ["unomipp01.int.jahia.com", "unomipp02.int.jahia.com", "unomipp03.int.jahia.com"]
# Compress the TCP transport
transport.tcp.compress: true

Enable the service ElasticSearch in order to start it automatically after an unexpected shutdown of the server
```
sudo systemctl enable elasticsearch
```
Start the service elasticsearch
```
sudo service elasticsearch start
```

Securing

By default, ElasticSearch does not ask for any kind of authentication. To correct that, you can :

Install the Elastic Stack extension X-Pack
Install Search Guard, a free, open-source alternative to X-Pack
Install a front-end and use the HTTP basic authentication to restrict access (chosen method)

First, you have to restrict access to the Firewall level on each node:

Deny all access to the TCP ports 9200 (used for the Rest communication) and 9300 (used for the transport communication).
- With UFW:
```
sudo ufw insert 1 deny 9300/tcp;
sudo ufw insert 1 deny 9200/tcp;
```

Grant access for the TCP port 9300 to all cluster members

sudo ufw insert 1 allow from IP_UNOMIPP01 to any port 9300;
sudo ufw insert 1 allow from IP_UNOMIPP02 to any port 9300;
sudo ufw insert 1 allow from IP_UNOMIPP03 to any port 9300;

Now, on each node, we can install the front-end in order to allow an authenticated access to the port 9200:

Install Nginx
```
sudo apt install nginx
```

Modify the file /etc/nginx/sites-enabled/default

# Redirect all insecure requests to the secure port
server {
  listen *:80 ;
  server_name unomipp01.int.jahia.com;

  return 301 https://$server_name$request_uri;
}

# Serve SSL encrypted data
server {
  listen *:443 default_server ssl;
  server_name unomipp01.int.jahia.com;

  # You'll have to generate these files (self-signed certificate or real one: https://www.digitalocean.com/community/tutorials/how-to-create-a-self-signed-ssl-certificate-for-nginx-in-ubuntu-16-04)
  ssl_certificate /etc/nginx/ssl/nginx.crt;
  ssl_certificate_key /etc/nginx/ssl/nginx.key;

  location / {
    if ($request_method = 'OPTIONS') {
      add_header 'Access-Control-Allow-Origin' '*';
      add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
      add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range,authorization';
      add_header 'Access-Control-Max-Age' 1728000;
      add_header 'Content-Type' 'text/plain; charset=utf-8';
      add_header 'Content-Length' 0;
      return 204;
    }

    # You'll have to generate the password file : https://www.digitalocean.com/community/tutorials/how-to-set-up-password-authentication-with-nginx-on-ubuntu-14-04
    auth_basic "UnomiPP ES";
    auth_basic_user_file /etc/nginx/conf.d/search.htpasswd;
    # Send everything to the Elasticsearch endpoint
    try_files $uri @elasticsearch;
  }

  # Endpoint to pass Elasticsearch queries to
  location @elasticsearch {
    proxy_pass http://unomipp01.int.jahia.com:9200;
      proxy_read_timeout 90;
  }
}

Restart the Nginx service
```
sudo service nginx restart
```

Test that the cluster is working fine:

curl http://unomipp01.int.jahia.com:9200/_cluster/health

Monitoring

Like before, you have multiple solutions:

Install the Elastic Stack extension X-Pack
Use other tools to monitor as ElastAlert or Sentinl
Use your own scripts (chosen methods)

Here are some of the tests we're doing every 5 minutes:

Check the status of the ES cluster

#!/bin/bash
fqdn=$(hostname -f)
response=$(curl -s http://$fqdn:9200/_cluster/health | jq -r .status)

if [ "$response" == "green" ]
then
    printf "elasticsearch_cluster.value 1"
exit 0
fi

printf "elasticsearch_cluster.value 0"

Check the amount of available disk space
Monitor the logs of ElasticSearch in /var/log/elasticsearch in order to find any warn or error messages

You can also install elasticsearch-head in order to have a UI to see the state of your cluster

Backup

To backup ES, the only officially supported way is to take snapshots. To enable them, you can follow this documentation. To do so, you can schedule a script like this one each night

#/bin/bash
curr_date=`date +%Y_%m_%d`
curr_date_minus_7=`date +%Y:%m:%d -d "7 day ago"`
echo $curr_date
curl -XPUT "http://unomipp01.int.jahia.com:9200/_snapshot/my_backup/$curr_date?wait_for_completion=true&pretty"
curl -XDELETE "http://unomipp01.int.jahia.com:9200/_snapshot/my_backup/$curr_date_minus_7"

Then you can archive the content of the folder related to the snapshots repository.

jCustomer cluster

Install

For each node:

Create the user and group unomi

sudo adduser --quiet --disabled-password --disabled-login --home /opt/unomi-default --gecos "" unomi

The last version of jCustomer can be found there
Extract the content of the archive in the folder /opt
```
sudo tar xzvf unomi-1.2.2-jahia.tar.gz -C /opt/
```

Create a symbolic link to the path /opt/unomi-default

sudo ln -s /opt/unomi-1.2.2-jahia/ /opt/unomi-default

Change the owner of the folder

sudo chown -R unomi:unomi /opt/unomi-default
sudo chown -R unomi:unomi /opt/unomi-default/

Change the URLs used by jCustomer by modifying the file /opt/unomi-default/etc/org.apache.unomi.cluster.cfg

group=default

contextserver.publicAddress=https://unomipp.jahia.com
contextserver.internalAddress=https://unomipp01.int.jahia.com:9443

nodeStatisticsUpdateFrequency=10000
contextserver.domain=jahia.com

Change the members of the jCustomer cluster by modifying the file /opt/unomi-default/etc/hazelcast.xml

<join>
    <multicast enabled="false">
        <multicast-group>224.2.2.3</multicast-group>
        <multicast-port>54327</multicast-port>
    </multicast>
    <tcp-ip enabled="true">
        <member>unomipp01.int.jahia.com</member>
        <member>unomipp02.int.jahia.com</member>
        <member>unomipp03.int.jahia.com</member>
    </tcp-ip>
</join>

Configure the ElasticSearch persistence by modifying the file /opt/unomi-default/org.apache.unomi.persistence.elasticsearch.cfg

# Name of the ElasticSearch cluster
cluster.name=JahiaClusterPP

# List of the ElasticSearch nosts
elasticSearchAddresses=unomipp01.int.jahia.com:9300,unomipp02.int.jahia.com:9300,unomipp03.int.jahia.com:9300

index.name=context
monthlyIndex.numberOfShards=3
monthlyIndex.numberOfReplicas=1
numberOfShards=5
numberOfReplicas=1
defaultQueryLimit=10

bulkProcessor.concurrentRequests=1
bulkProcessor.bulkActions=1000
bulkProcessor.bulkSize=5MB
bulkProcessor.flushInterval=5s
bulkProcessor.backoffPolicy=exponential

minimalElasticSearchVersion=5.0.0
maximalElasticSearchVersion=5.7.0

aggregateQueryBucketSize=5000

Disable automic cluster synchronisation on each jCustomer node by modifying the file /etc/org.apache.karaf.cellar.groups.cfg

#
# The following properties define the behavior to use when the node joins the cluster (the usage of the bootstrap
# synchronizer), per cluster group and per resource.
# The following values are accepted:
# disabled: means that the synchronizer is not used, meaning the node or the cluster are not updated at all
# cluster: if the node is the first one in the cluster, it pushes its local state to the cluster, else it's not the
#       first node of the cluster, the node will update its local state with the cluster one (meaning that the cluster
#       is the master)
# node: in this case, the node is the master, it means that the cluster state will be overwritten by the node state.
#
default.bundle.sync = disabled
default.config.sync = disabled
default.feature.sync = disabled
default.obr.urls.sync = disabled

The install of GeoIP and Geonames databases is described there

Create the file /lib/systemd/system/unomi.service in order to have a daemon for jCustomer

[Unit]
Description=Unomi
After=syslog.target network.target

[Service]
ExecStart=/opt/unomi-default/bin/karaf daemon
ExecStop=/opt/unomi-default/bin/karaf stop

User=unomi
Group=unomi

SuccessExitStatus=0 143
RestartSec=15
Restart=on-failure

LimitNOFILE=102642

[Install]
WantedBy=multi-user.target

Reload the configuration
```
sudo systemctl daemon-reload
```
Enable the service
```
sudo systemctl enable unomi
```
Start the service
```
sudo service unomi start
```
Connect to the Karaf console
```
ssh -p 8102 karaf@unomipp01
```
Start the jCustomer packages
```
unomi:start
```
Verify that the node everything is ok by checking the following URLs:
- http://unomipp01.int.jahia.com:8181/cxs/cluster
- http://unomipp01.int.jahia.com:8181/context.js?sessionId=test

Securing

Change the password of the SSH user karaf by modifying the file /opt/unomi-default/etc/users.properies
```
karaf = XXXXXXXXX,_g_:admingroup
_g_\:admingroup = group,admin,manager,viewer,webconsole
```

Activate the encryption of the password by modifying the file /opt/unomi-default/etc/org.apache.karaf.jaas.cfg

################################################################################
#
#    Licensed to the Apache Software Foundation (ASF) under one or more
#    contributor license agreements.  See the NOTICE file distributed with
#    this work for additional information regarding copyright ownership.
#    The ASF licenses this file to You under the Apache License, Version 2.0
#    (the "License"); you may not use this file except in compliance with
#    the License.  You may obtain a copy of the License at
#
#       http://www.apache.org/licenses/LICENSE-2.0
#
#    Unless required by applicable law or agreed to in writing, software
#    distributed under the License is distributed on an "AS IS" BASIS,
#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#    See the License for the specific language governing permissions and
#    limitations under the License.
#
################################################################################

#
# Boolean enabling / disabling encrypted passwords
#
encryption.enabled = true

#
# Encryption Service name
#   the default one is 'basic'
#   a more powerful one named 'jasypt' is available
#       when installing the encryption feature
#
encryption.name = basic

#
# Encryption prefix
#
encryption.prefix = {CRYPT}

#
# Encryption suffix
#
encryption.suffix = {CRYPT}

#
# Set the encryption algorithm to use in Karaf JAAS login module
# Supported encryption algorithms follow:
#   MD2
#   MD5
#   SHA-1
#   SHA-256
#   SHA-384
#   SHA-512
#
encryption.algorithm = SHA-512

#
# Encoding of the encrypted password.
# Can be:
#   hexadecimal
#   base64
#
encryption.encoding = hexadecimal

Change the password of the Hazelcast group by modifying the file /opt/unomi-default/etc/hazelcast.xml

<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.2.xsd"
           xmlns="http://www.hazelcast.com/schema/config"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <group>
        <name>jahia_preproduction</name>
        <password>XXXXXX</password>
    </group>

Change the third party key and the IPs authorized to execute some events by modifying the file /opt/unomi-default/etc/org.apache.unomi-thirdparty.cfg

thirdparty.provider1.key=DEFAULT_UNOMI_KEY_CHANGED
thirdparty.provider1.ipAddresses=127.0.0.1,::1,DX_NODE_1,DX_NODE_2
thirdparty.provider1.allowedEvents=login,updateProperties

Do not forget to restart jCustomer thanks to the command:

sudo service unomi restart

Monitoring

Here are some of the tests we're doing every 5 minutes:

Check the amount of available disk space
Monitor the log of jCustomer in /opt/unomi-default/data/log/karaf.log in order to find any warn or error messages

Check the status of the jCustomer node:

#!/bin/bash

response=$(curl --connect-timeout 1 --max-time 3 -sS http://localhost:8181/context.json?sessionId=munin >> /dev/null | wc -l)
if [ "$response" != "0" ]
then
  printf "unomi_service.value 0"
exit 0
fi

response=$(curl -o -I -L -s -w "%{http_code}\n" --connect-timeout 1 --max-time 3 -sS --connect-timeout 1 --max-time 3 -sS http://localhost:8181/context.json?sessionId=munin)
if [ "$response" != "200" ]
then
  printf "unomi_service.value 0"
exit 0
fi

Backup

To create a backup, you can now archive to an outside location the content of the folder /opt/unomi-default

jCustomer front-end

All nodes are now working but the public URL for jCustomer is not yet available. The related front-end will be installed on another node (for example web.int.jahia.com) which has also a public IP. The software Apache2 needs to be installed.

Add the file /etc/apache2/sites-enabled/unomipp.jahia.com.conf with the following content:

<VirtualHost *:80>
    ServerName unomipp.jahia.com
    
    CustomLog /var/log/apache2/access-unomipp.jahia.com.log combined
    ErrorLog /var/log/apache2/error-unomipp.jahia.com.log
    
    RewriteEngine On
    RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
    RewriteRule .* - [F]
    
    RewriteRule ^(.*)$ https://%{SERVER_NAME}%{REQUEST_URI} [R=301,L]
</VirtualHost>
<IfModule mod_ssl.c>
    <VirtualHost *:443>
        ServerName unomipp.jahia.com
        ServerAdmin monitor@jahia.com
        DocumentRoot /var/www/vhosts/unomipp.jahia.com/html/
        CustomLog /var/log/apache2/access-unomipp.jahia.com.log combined
        ErrorLog /var/log/apache2/error-unomipp.jahia.com.log
        ModPagespeed off
        
        <Directory />
            Options FollowSymLinks
            AllowOverride None
        </Directory>
        
        <Directory /var/www/html>
            Options FollowSymLinks MultiViews
            AllowOverride None
            Order allow,deny
            allow from all
        </Directory>
        
        <Location /cxs>
            #List of the IPs that have the right to access the context "/cxs". It should be only localhost and internal IPs
            Require ip 127.0.0.1 10.100
        </Location>
        
        RewriteEngine On
        RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
        RewriteRule .* - [F]
        ProxyPreserveHost On
        ProxyPass /server-status !
        ProxyPass /robots.txt !
        ProxyPass /elasticsearch-head !
        RewriteCond %{HTTP_USER_AGENT} Googlebot [OR]
        RewriteCond %{HTTP_USER_AGENT} msnbot [OR]
        RewriteCond %{HTTP_USER_AGENT} Slurp
        RewriteRule ^.* - [F,L]
        
        ProxyPass / balancer://unomi_cluster/
        ProxyPassReverse / balancer://unomi_cluster/
        Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
        <Proxy balancer://unomi_cluster>
            BalancerMember http://unomipp01.int.jahia.com:8181 route=1 connectiontimeout=20 timeout=300 ttl=120 ping=500ms
            BalancerMember http://unomipp02.int.jahia.com:8181 route=2 connectiontimeout=20 timeout=300 ttl=120 ping=500ms
            BalancerMember http://unomipp03.int.jahia.com:8181 route=3 connectiontimeout=20 timeout=300 ttl=120 ping=500ms
            ProxySet lbmethod=bytraffic stickysession=ROUTEID
        </Proxy>
        
        RemoteIPHeader X-Forwarded-For
        
        Include ssl-common.conf
        
        <FilesMatch "\.(cgi|shtml|phtml|php)$">
            SSLOptions +StdEnvVars
        </FilesMatch>
        <Directory /usr/lib/cgi-bin>
            SSLOptions +StdEnvVars
        </Directory>
        BrowserMatch "MSIE [2-6]" \
            nokeepalive ssl-unclean-shutdown \
            downgrade-1.0 force-response-1.0
        BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
    </VirtualHost>
</IfModule>

The file ssl-common.conf only contains that:

SSLEngine on
SSLCertificateKeyFile /etc/apache2/ssl/privkey.pem
SSLCertificateFile /etc/apache2/ssl/fullchain.pem
RequestHeader set X-Forwarded-Proto "https"

The generic SSL configuration is defined in the file /etc/apache2/mods-enabled/ssl.conf

<IfModule mod_ssl.c>
    SSLCipherSuite HIGH:!aNULL:!MD5:!ADH:!RC4:!DH
    SSLProtocol All -SSLv2 -SSLv3 -TLSv1
    SSLHonorCipherOrder On
    Header always set Strict-Transport-Security "max-age=63072000; includeSubdomains; preload"
    Header always set X-Content-Type-Options nosniff
    # Requires Apache >= 2.4
    SSLCompression off
    SSLSessionTickets Off
    SSLUseStapling on
    SSLStaplingCache "shmcb:logs/stapling-cache(150000)"
</IfModule>

jExperience

Now, you only have to configure jExperience to use the jCustomer front-end. You should end up with the following result: