Setting up a jCustomer cluster (Elasticsearch and Unomi)

November 14, 2023
Note: This topic applies to older versions of the stack (jExperience 1.8, jCustomer 1.2, Elasticsearch 5.6.3). You can find more recent instructions on this page.

Here are the steps followed to install an ElasticSearch and jCustomer clusters for our preproduction environment. This cluster will be used for jExperience but also for every Jahia product or module needing ElasticSearch.

General

ElasticSearch and jCustomer will be installed on a cluster of three nodes:

  • unomipp01
  • unomipp02
  • unomipp03

Each node has :

  • Ubuntu 16.04 LTS
  • 40 GB of hard drive
  • 4 CPUs
  • 7 GB of memory

ES cluster

Install

  • At the moment, only ElasticSearch 5.6.3 can be used with Jahia. The Debian package can be downloaded here
  • To install it:
    sudo dpkg -i elasticsearch-5.6.3.deb
  • To configure it, you have to edit the file /etc/elasticsearch/elasticsearch.yml
    # Name of the cluster. It has to be uniq per environment
    cluster.name: JahiaClusterPP
    # Name of the node
    node.name: unomipp01
    # Host name
    network.host: unomipp01.int.jahia.com
    # Name of the hosts being part of the cluster
    discovery.zen.ping.unicast.hosts: ["unomipp01.int.jahia.com", "unomipp02.int.jahia.com", "unomipp03.int.jahia.com"]
    # Compress the TCP transport
    transport.tcp.compress: true
  • Enable the service ElasticSearch in order to start it automatically after an unexpected shutdown of the server
    sudo systemctl enable elasticsearch
  • Start the service elasticsearch
    sudo service elasticsearch start

Securing

By default, ElasticSearch does not ask for any kind of authentication. To correct that, you can :

First, you have to restrict access to the Firewall level on each node:

  • Deny all access to the TCP ports 9200 (used for the Rest communication) and 9300 (used for the transport communication).
    • With UFW:
      sudo ufw insert 1 deny 9300/tcp;
      sudo ufw insert 1 deny 9200/tcp;
  • Grant access for the TCP port 9300 to all cluster members
    sudo ufw insert 1 allow from IP_UNOMIPP01 to any port 9300;
    sudo ufw insert 1 allow from IP_UNOMIPP02 to any port 9300;
    sudo ufw insert 1 allow from IP_UNOMIPP03 to any port 9300;
Now, on each node, we can install the front-end in order to allow an authenticated access to the port 9200:
  • Install Nginx
    sudo apt install nginx
  • Modify the file /etc/nginx/sites-enabled/default 
    # Redirect all insecure requests to the secure port
    server {
      listen *:80 ;
      server_name unomipp01.int.jahia.com;
    
      return 301 https://$server_name$request_uri;
    }
    
    # Serve SSL encrypted data
    server {
      listen *:443 default_server ssl;
      server_name unomipp01.int.jahia.com;
    
      # You'll have to generate these files (self-signed certificate or real one: https://www.digitalocean.com/community/tutorials/how-to-create-a-self-signed-ssl-certificate-for-nginx-in-ubuntu-16-04)
      ssl_certificate /etc/nginx/ssl/nginx.crt;
      ssl_certificate_key /etc/nginx/ssl/nginx.key;
    
      location / {
        if ($request_method = 'OPTIONS') {
          add_header 'Access-Control-Allow-Origin' '*';
          add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
          add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range,authorization';
          add_header 'Access-Control-Max-Age' 1728000;
          add_header 'Content-Type' 'text/plain; charset=utf-8';
          add_header 'Content-Length' 0;
          return 204;
        }
    
        # You'll have to generate the password file : https://www.digitalocean.com/community/tutorials/how-to-set-up-password-authentication-with-nginx-on-ubuntu-14-04
        auth_basic "UnomiPP ES";
        auth_basic_user_file /etc/nginx/conf.d/search.htpasswd;
        # Send everything to the Elasticsearch endpoint
        try_files $uri @elasticsearch;
      }
    
      # Endpoint to pass Elasticsearch queries to
      location @elasticsearch {
        proxy_pass http://unomipp01.int.jahia.com:9200;
          proxy_read_timeout 90;
      }
    }
  • Restart the Nginx service
    sudo service nginx restart
  • Test that the cluster is working fine:
    curl http://unomipp01.int.jahia.com:9200/_cluster/health

Monitoring

Like before, you have multiple solutions:

Here are some of the tests we're doing every 5 minutes:

  • Check the status of the ES cluster
    #!/bin/bash
    fqdn=$(hostname -f)
    response=$(curl -s http://$fqdn:9200/_cluster/health | jq -r .status)
    
    if [ "$response" == "green" ]
    then
        printf "elasticsearch_cluster.value 1"
    exit 0
    fi
    
    printf "elasticsearch_cluster.value 0"
  • Check the amount of available disk space
  • Monitor the logs of ElasticSearch in /var/log/elasticsearch in order to find any warn or error messages

You can also install elasticsearch-head in order to have a UI to see the state of your cluster

Backup

To backup ES, the only officially supported way is to take snapshots. To enable them, you can follow this documentation. To do so, you can schedule a script like this one each night

#/bin/bash
curr_date=`date +%Y_%m_%d`
curr_date_minus_7=`date +%Y:%m:%d -d "7 day ago"`
echo $curr_date
curl -XPUT "http://unomipp01.int.jahia.com:9200/_snapshot/my_backup/$curr_date?wait_for_completion=true&pretty"
curl -XDELETE "http://unomipp01.int.jahia.com:9200/_snapshot/my_backup/$curr_date_minus_7"

Then you can archive the content of the folder related to the snapshots repository.

jCustomer cluster

Install

For each node:

  • Create the user and group unomi
    sudo adduser --quiet --disabled-password --disabled-login --home /opt/unomi-default --gecos "" unomi
  • The last version of jCustomer can be found there
  • Extract the content of the archive in the folder /opt
    sudo tar xzvf unomi-1.2.2-jahia.tar.gz -C /opt/
  • Create a symbolic link to the path /opt/unomi-default
    sudo ln -s /opt/unomi-1.2.2-jahia/ /opt/unomi-default
  • Change the owner of the folder
    sudo chown -R unomi:unomi /opt/unomi-default
    sudo chown -R unomi:unomi /opt/unomi-default/
    
  • Change the URLs used by jCustomer by modifying the file /opt/unomi-default/etc/org.apache.unomi.cluster.cfg
    group=default
    
    contextserver.publicAddress=https://unomipp.jahia.com
    contextserver.internalAddress=https://unomipp01.int.jahia.com:9443
    
    nodeStatisticsUpdateFrequency=10000
    contextserver.domain=jahia.com
  • Change the members of the jCustomer cluster by modifying the file /opt/unomi-default/etc/hazelcast.xml
    <join>
        <multicast enabled="false">
            <multicast-group>224.2.2.3</multicast-group>
            <multicast-port>54327</multicast-port>
        </multicast>
        <tcp-ip enabled="true">
            <member>unomipp01.int.jahia.com</member>
            <member>unomipp02.int.jahia.com</member>
            <member>unomipp03.int.jahia.com</member>
        </tcp-ip>
    </join>
  • Configure the ElasticSearch persistence by modifying the file /opt/unomi-default/org.apache.unomi.persistence.elasticsearch.cfg
    # Name of the ElasticSearch cluster
    cluster.name=JahiaClusterPP
    
    # List of the ElasticSearch nosts
    elasticSearchAddresses=unomipp01.int.jahia.com:9300,unomipp02.int.jahia.com:9300,unomipp03.int.jahia.com:9300
    
    index.name=context
    monthlyIndex.numberOfShards=3
    monthlyIndex.numberOfReplicas=1
    numberOfShards=5
    numberOfReplicas=1
    defaultQueryLimit=10
    
    bulkProcessor.concurrentRequests=1
    bulkProcessor.bulkActions=1000
    bulkProcessor.bulkSize=5MB
    bulkProcessor.flushInterval=5s
    bulkProcessor.backoffPolicy=exponential
    
    minimalElasticSearchVersion=5.0.0
    maximalElasticSearchVersion=5.7.0
    
    aggregateQueryBucketSize=5000
  • Disable automic cluster synchronisation on each jCustomer node by modifying the file /etc/org.apache.karaf.cellar.groups.cfg
    #
    # The following properties define the behavior to use when the node joins the cluster (the usage of the bootstrap
    # synchronizer), per cluster group and per resource.
    # The following values are accepted:
    # disabled: means that the synchronizer is not used, meaning the node or the cluster are not updated at all
    # cluster: if the node is the first one in the cluster, it pushes its local state to the cluster, else it's not the
    #       first node of the cluster, the node will update its local state with the cluster one (meaning that the cluster
    #       is the master)
    # node: in this case, the node is the master, it means that the cluster state will be overwritten by the node state.
    #
    default.bundle.sync = disabled
    default.config.sync = disabled
    default.feature.sync = disabled
    default.obr.urls.sync = disabled
  • The install of GeoIP and Geonames databases is described there
  • Create the file /lib/systemd/system/unomi.service in order to have a daemon for jCustomer
    [Unit]
    Description=Unomi
    After=syslog.target network.target
    
    [Service]
    ExecStart=/opt/unomi-default/bin/karaf daemon
    ExecStop=/opt/unomi-default/bin/karaf stop
    
    User=unomi
    Group=unomi
    
    SuccessExitStatus=0 143
    RestartSec=15
    Restart=on-failure
    
    LimitNOFILE=102642
    
    [Install]
    WantedBy=multi-user.target
  • Reload the configuration
    sudo systemctl daemon-reload
  • Enable the service
    sudo systemctl enable unomi
  • Start the service
    sudo service unomi start
  • Connect to the Karaf console
    ssh -p 8102 karaf@unomipp01
  • Start the jCustomer packages
    unomi:start
  • Verify that the node everything is ok by checking the following URLs:
    • http://unomipp01.int.jahia.com:8181/cxs/cluster
    • http://unomipp01.int.jahia.com:8181/context.js?sessionId=test

Securing

  • Change the password of the SSH user karaf by modifying the file /opt/unomi-default/etc/users.properies
    karaf = XXXXXXXXX,_g_:admingroup
    _g_\:admingroup = group,admin,manager,viewer,webconsole
  • Activate the encryption of the password by modifying the file /opt/unomi-default/etc/org.apache.karaf.jaas.cfg
    ################################################################################
    #
    #    Licensed to the Apache Software Foundation (ASF) under one or more
    #    contributor license agreements.  See the NOTICE file distributed with
    #    this work for additional information regarding copyright ownership.
    #    The ASF licenses this file to You under the Apache License, Version 2.0
    #    (the "License"); you may not use this file except in compliance with
    #    the License.  You may obtain a copy of the License at
    #
    #       http://www.apache.org/licenses/LICENSE-2.0
    #
    #    Unless required by applicable law or agreed to in writing, software
    #    distributed under the License is distributed on an "AS IS" BASIS,
    #    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    #    See the License for the specific language governing permissions and
    #    limitations under the License.
    #
    ################################################################################
    
    #
    # Boolean enabling / disabling encrypted passwords
    #
    encryption.enabled = true
    
    #
    # Encryption Service name
    #   the default one is 'basic'
    #   a more powerful one named 'jasypt' is available
    #       when installing the encryption feature
    #
    encryption.name = basic
    
    #
    # Encryption prefix
    #
    encryption.prefix = {CRYPT}
    
    #
    # Encryption suffix
    #
    encryption.suffix = {CRYPT}
    
    #
    # Set the encryption algorithm to use in Karaf JAAS login module
    # Supported encryption algorithms follow:
    #   MD2
    #   MD5
    #   SHA-1
    #   SHA-256
    #   SHA-384
    #   SHA-512
    #
    encryption.algorithm = SHA-512
    
    #
    # Encoding of the encrypted password.
    # Can be:
    #   hexadecimal
    #   base64
    #
    encryption.encoding = hexadecimal
    
  • Change the password of the Hazelcast group by modifying the file /opt/unomi-default/etc/hazelcast.xml
  • <hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.2.xsd"
               xmlns="http://www.hazelcast.com/schema/config"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
        <group>
            <name>jahia_preproduction</name>
            <password>XXXXXX</password>
        </group>
  • Change the third party key and the IPs authorized to execute some events by modifying the file /opt/unomi-default/etc/org.apache.unomi-thirdparty.cfg
    thirdparty.provider1.key=DEFAULT_UNOMI_KEY_CHANGED
    thirdparty.provider1.ipAddresses=127.0.0.1,::1,DX_NODE_1,DX_NODE_2
    thirdparty.provider1.allowedEvents=login,updateProperties

Do not forget to restart jCustomer thanks to the command:

sudo service unomi restart

Monitoring

Here are some of the tests we're doing every 5 minutes:

  • Check the amount of available disk space
  • Monitor the log of jCustomer in /opt/unomi-default/data/log/karaf.log in order to find any warn or error messages
  • Check the status of the jCustomer node:
    #!/bin/bash
    
    response=$(curl --connect-timeout 1 --max-time 3 -sS http://localhost:8181/context.json?sessionId=munin >> /dev/null | wc -l)
    if [ "$response" != "0" ]
    then
      printf "unomi_service.value 0"
    exit 0
    fi
    
    response=$(curl -o -I -L -s -w "%{http_code}\n" --connect-timeout 1 --max-time 3 -sS --connect-timeout 1 --max-time 3 -sS http://localhost:8181/context.json?sessionId=munin)
    if [ "$response" != "200" ]
    then
      printf "unomi_service.value 0"
    exit 0
    fi

Backup

To create a backup, you can now archive to an outside location the content of the folder /opt/unomi-default

jCustomer front-end

All nodes are now working but the public URL for jCustomer is not yet available. The related front-end will be installed on another node (for example web.int.jahia.com) which has also a public IP. The software Apache2 needs to be installed.

  • Add the file /etc/apache2/sites-enabled/unomipp.jahia.com.conf with the following content:
    <VirtualHost *:80>
        ServerName unomipp.jahia.com
        
        CustomLog /var/log/apache2/access-unomipp.jahia.com.log combined
        ErrorLog /var/log/apache2/error-unomipp.jahia.com.log
        
        RewriteEngine On
        RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
        RewriteRule .* - [F]
        
        RewriteRule ^(.*)$ https://%{SERVER_NAME}%{REQUEST_URI} [R=301,L]
    </VirtualHost>
    <IfModule mod_ssl.c>
        <VirtualHost *:443>
            ServerName unomipp.jahia.com
            ServerAdmin monitor@jahia.com
            DocumentRoot /var/www/vhosts/unomipp.jahia.com/html/
            CustomLog /var/log/apache2/access-unomipp.jahia.com.log combined
            ErrorLog /var/log/apache2/error-unomipp.jahia.com.log
            ModPagespeed off
            
            <Directory />
                Options FollowSymLinks
                AllowOverride None
            </Directory>
            
            <Directory /var/www/html>
                Options FollowSymLinks MultiViews
                AllowOverride None
                Order allow,deny
                allow from all
            </Directory>
            
            <Location /cxs>
                #List of the IPs that have the right to access the context "/cxs". It should be only localhost and internal IPs
                Require ip 127.0.0.1 10.100
            </Location>
            
            RewriteEngine On
            RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
            RewriteRule .* - [F]
            ProxyPreserveHost On
            ProxyPass /server-status !
            ProxyPass /robots.txt !
            ProxyPass /elasticsearch-head !
            RewriteCond %{HTTP_USER_AGENT} Googlebot [OR]
            RewriteCond %{HTTP_USER_AGENT} msnbot [OR]
            RewriteCond %{HTTP_USER_AGENT} Slurp
            RewriteRule ^.* - [F,L]
            
            ProxyPass / balancer://unomi_cluster/
            ProxyPassReverse / balancer://unomi_cluster/
            Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
            <Proxy balancer://unomi_cluster>
                BalancerMember http://unomipp01.int.jahia.com:8181 route=1 connectiontimeout=20 timeout=300 ttl=120 ping=500ms
                BalancerMember http://unomipp02.int.jahia.com:8181 route=2 connectiontimeout=20 timeout=300 ttl=120 ping=500ms
                BalancerMember http://unomipp03.int.jahia.com:8181 route=3 connectiontimeout=20 timeout=300 ttl=120 ping=500ms
                ProxySet lbmethod=bytraffic stickysession=ROUTEID
            </Proxy>
            
            RemoteIPHeader X-Forwarded-For
            
            Include ssl-common.conf
            
            <FilesMatch "\.(cgi|shtml|phtml|php)$">
                SSLOptions +StdEnvVars
            </FilesMatch>
            <Directory /usr/lib/cgi-bin>
                SSLOptions +StdEnvVars
            </Directory>
            BrowserMatch "MSIE [2-6]" \
                nokeepalive ssl-unclean-shutdown \
                downgrade-1.0 force-response-1.0
            BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
        </VirtualHost>
    </IfModule>
    
  • The file ssl-common.conf only contains that:
    SSLEngine on
    SSLCertificateKeyFile /etc/apache2/ssl/privkey.pem
    SSLCertificateFile /etc/apache2/ssl/fullchain.pem
    RequestHeader set X-Forwarded-Proto "https"
  • The generic SSL configuration is defined in the file /etc/apache2/mods-enabled/ssl.conf
    <IfModule mod_ssl.c>
        SSLCipherSuite HIGH:!aNULL:!MD5:!ADH:!RC4:!DH
        SSLProtocol All -SSLv2 -SSLv3 -TLSv1
        SSLHonorCipherOrder On
        Header always set Strict-Transport-Security "max-age=63072000; includeSubdomains; preload"
        Header always set X-Content-Type-Options nosniff
        # Requires Apache >= 2.4
        SSLCompression off
        SSLSessionTickets Off
        SSLUseStapling on
        SSLStaplingCache "shmcb:logs/stapling-cache(150000)"
    </IfModule>

jExperience

Now, you only have to configure jExperience to use the jCustomer front-end. You should end up with the following result:

mf.png

  •  
  •