ElasticSearch / Unomi cluster setup

  Written by The Jahia Team
 
Sysadmins
   Estimated reading time:
unomi 1.2.2

Here are the steps followed to install an ElasticSearch and Unomi clusters for our preproduction environment. This cluster will be used for Marketing Factory but also for every Jahia product or module needing ElasticSearch.

General

ElasticSearch and Unomi will be installed on a cluster of three nodes:

  • unomipp01
  • unomipp02
  • unomipp03

Each node has :

  • Ubuntu 16.04 LTS
  • 40 GB of hard drive
  • 4 CPUs
  • 7 GB of memory

ES cluster

Install

  • At the moment, only ElasticSearch 5.6.3 can be used with DX. The Debian package can be downloaded here
  • To install it:
    sudo dpkg -i elasticsearch-5.6.3.deb
  • To configure it, you have to edit the file /etc/elasticsearch/elasticsearch.yml
    # Name of the cluster. It has to be uniq per environment
    cluster.name: JahiaClusterPP
    # Name of the node
    node.name: unomipp01
    # Host name
    network.host: unomipp01.int.jahia.com
    # Name of the hosts being part of the cluster
    discovery.zen.ping.unicast.hosts: ["unomipp01.int.jahia.com", "unomipp02.int.jahia.com", "unomipp03.int.jahia.com"]
    # Compress the TCP transport
    transport.tcp.compress: true
  • Enable the service ElasticSearch in order to start it automatically after an unexpected shutdown of the server
    sudo systemctl enable elasticsearch
  • Start the service elasticsearch
    sudo service elasticsearch start

Securing

By default, ElasticSearch does not ask for any kind of authentication. To correct that, you can :

First, you have to restrict access to the Firewall level on each node:

  • Deny all access to the TCP ports 9200 (used for the Rest communication) and 9300 (used for the transport communication).
    • With UFW:
      sudo ufw insert 1 deny 9300/tcp;
      sudo ufw insert 1 deny 9200/tcp;
  • Grant access for the TCP port 9300 to all cluster members
    sudo ufw insert 1 allow from IP_UNOMIPP01 to any port 9300;
    sudo ufw insert 1 allow from IP_UNOMIPP02 to any port 9300;
    sudo ufw insert 1 allow from IP_UNOMIPP03 to any port 9300;
Now, on each node, we can install the front-end in order to allow an authenticated access to the port 9200:
  • Install Nginx
    sudo apt install nginx
  • Modify the file /etc/nginx/sites-enabled/default 
    # Redirect all insecure requests to the secure port
    server {
      listen *:80 ;
      server_name unomipp01.int.jahia.com;
    
      return 301 https://$server_name$request_uri;
    }
    
    # Serve SSL encrypted data
    server {
      listen *:443 default_server ssl;
      server_name unomipp01.int.jahia.com;
    
      # You'll have to generate these files (self-signed certificate or real one: https://www.digitalocean.com/community/tutorials/how-to-create-a-self-signed-ssl-certificate-for-nginx-in-ubuntu-16-04)
      ssl_certificate /etc/nginx/ssl/nginx.crt;
      ssl_certificate_key /etc/nginx/ssl/nginx.key;
    
      location / {
        if ($request_method = 'OPTIONS') {
          add_header 'Access-Control-Allow-Origin' '*';
          add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
          add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range,authorization';
          add_header 'Access-Control-Max-Age' 1728000;
          add_header 'Content-Type' 'text/plain; charset=utf-8';
          add_header 'Content-Length' 0;
          return 204;
        }
    
        # You'll have to generate the password file : https://www.digitalocean.com/community/tutorials/how-to-set-up-password-authentication-with-nginx-on-ubuntu-14-04
        auth_basic "UnomiPP ES";
        auth_basic_user_file /etc/nginx/conf.d/search.htpasswd;
        # Send everything to the Elasticsearch endpoint
        try_files $uri @elasticsearch;
      }
    
      # Endpoint to pass Elasticsearch queries to
      location @elasticsearch {
        proxy_pass http://unomipp01.int.jahia.com:9200;
          proxy_read_timeout 90;
      }
    }
  • Restart the Nginx service
    sudo service nginx restart
  • Test that the cluster is working fine:
    curl http://unomipp01.int.jahia.com:9200/_cluster/health

Monitoring

Like before, you have multiple solutions:

Here are some of the tests we're doing every 5 minutes:

  • Check the status of the ES cluster
    #!/bin/bash
    fqdn=$(hostname -f)
    response=$(curl -s http://$fqdn:9200/_cluster/health | jq -r .status)
    
    if [ "$response" == "green" ]
    then
        printf "elasticsearch_cluster.value 1"
    exit 0
    fi
    
    printf "elasticsearch_cluster.value 0"
  • Check the amount of available disk space
  • Monitor the logs of ElasticSearch in /var/log/elasticsearch in order to find any warn or error messages

You can also install elasticsearch-head in order to have a UI to see the state of your cluster

Backup

To backup ES, the only officially supported way is to take snapshots. To enable them, you can follow this documentation. To do so, you can schedule a script like this one each night

#/bin/bash
curr_date=`date +%Y_%m_%d`
curr_date_minus_7=`date +%Y:%m:%d -d "7 day ago"`
echo $curr_date
curl -XPUT "http://unomipp01.int.jahia.com:9200/_snapshot/my_backup/$curr_date?wait_for_completion=true&pretty"
curl -XDELETE "http://unomipp01.int.jahia.com:9200/_snapshot/my_backup/$curr_date_minus_7"

Then you can archive the content of the folder related to the snapshots repository.

Unomi cluster

Install

For each node:

  • Create the user and group unomi
    sudo adduser --quiet --disabled-password --disabled-login --home /opt/unomi-default --gecos "" unomi
  • The last version of Apache Unomi can be found there
  • Extract the content of the archive in the folder /opt
    sudo tar xzvf unomi-1.2.2-jahia.tar.gz -C /opt/
  • Create a symbolic link to the path /opt/unomi-default
    sudo ln -s /opt/unomi-1.2.2-jahia/ /opt/unomi-default
  • Change the owner of the folder
    sudo chown -R unomi:unomi ~/opt/unomi-default
    sudo chown -R unomi:unomi ~/opt/unomi-default/
    
  • Change the URLs used by Unomi by modifying the file /opt/unomi-default/etc/org.apache.unomi.cluster.cfg
    group=default
    
    contextserver.publicAddress=https://unomipp.jahia.com
    contextserver.internalAddress=https://unomipp01.int.jahia.com:9443
    
    nodeStatisticsUpdateFrequency=10000
    contextserver.domain=jahia.com
  • Change the members of the Unomi cluster by modifying the file /opt/unomi-default/etc/hazelcast.xml
    <join>
        <multicast enabled="false">
            <multicast-group>224.2.2.3</multicast-group>
            <multicast-port>54327</multicast-port>
        </multicast>
        <tcp-ip enabled="true">
            <member>unomipp01.int.jahia.com</member>
            <member>unomipp02.int.jahia.com</member>
            <member>unomipp03.int.jahia.com</member>
        </tcp-ip>
    </join>
  • Configure the ElasticSearch persistence by modifying the file /opt/unomi-default/org.apache.unomi.persistence.elasticsearch.cfg
    # Name of the ElasticSearch cluster
    cluster.name=JahiaClusterPP
    
    # List of the ElasticSearch nosts
    elasticSearchAddresses=unomipp01.int.jahia.com:9300,unomipp02.int.jahia.com:9300,unomipp03.int.jahia.com:9300
    
    index.name=context
    monthlyIndex.numberOfShards=3
    monthlyIndex.numberOfReplicas=1
    numberOfShards=5
    numberOfReplicas=1
    defaultQueryLimit=10
    
    bulkProcessor.concurrentRequests=1
    bulkProcessor.bulkActions=1000
    bulkProcessor.bulkSize=5MB
    bulkProcessor.flushInterval=5s
    bulkProcessor.backoffPolicy=exponential
    
    minimalElasticSearchVersion=5.0.0
    maximalElasticSearchVersion=5.7.0
    
    aggregateQueryBucketSize=5000
  • The install of GeoIP and Geonames databases is described there
  • Create the file /lib/systemd/system/unomi.service in order to have a daemon for Unomi
    [Unit]
    Description=Unomi
    After=syslog.target network.target
    
    [Service]
    ExecStart=/opt/unomi-default/bin/karaf daemon
    ExecStop=/opt/unomi-default/bin/karaf stop
    
    User=unomi
    Group=unomi
    
    SuccessExitStatus=0 143
    RestartSec=15
    Restart=on-failure
    
    LimitNOFILE=102642
    
    [Install]
    WantedBy=multi-user.target
  • Reload the configuration
    sudo systemctl daemon-reload
  • Enable the service
    sudo systemctl enable unomi
  • Start the service
    sudo service unomi start
  • Connect to the Karaf console
    ssh -p 8102 karaf@unomipp01
  • Start the Unomi packages
    unomi:start
  • Verify that the node everything is ok by checking the following URLs:
    • http://unomipp01.int.jahia.com:8181/cxs/cluster
    • http://unomipp01.int.jahia.com:8181/context.js?sessionId=test

Securing

  • Change the password of the SSH user karaf by modifying the file /opt/unomi-default/etc/users.properies
    karaf = XXXXXXXXX,_g_:admingroup
    _g_\:admingroup = group,admin,manager,viewer,webconsole
  • Change the password of the Hazelcast group by modifying the file /opt/unomi-default/etc/hazelcast.xml
    <hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.2.xsd"
               xmlns="http://www.hazelcast.com/schema/config"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
        <group>
            <name>jahia_preproduction</name>
            <password>XXXXXX</password>
        </group>
  • Change the third party key and the IPs authorized to execute some events by modifying the file /opt/unomi-default/etc/org.apache.unomi-thirdparty.cfg
    thirdparty.provider1.key=DEFAULT_UNOMI_KEY_CHANGED
    thirdparty.provider1.ipAddresses=127.0.0.1,::1,DX_NODE_1,DX_NODE_2
    thirdparty.provider1.allowedEvents=login,updateProperties

Do not forget to restart Unomi thanks to the command:

sudo service unomi restart

Monitoring

Here are some of the tests we're doing every 5 minutes:

  • Check the amount of available disk space
  • Monitor the log of Unomi in /opt/unomi-default/data/log/karaf.log in order to find any warn or error messages
  • Check the status of the Unomi node:
    #!/bin/bash
    
    response=$(curl --connect-timeout 1 --max-time 3 -sS http://localhost:8181/context.json?sessionId=munin >> /dev/null | wc -l)
    if [ "$response" != "0" ]
    then
      printf "unomi_service.value 0"
    exit 0
    fi
    
    response=$(curl -o -I -L -s -w "%{http_code}\n" --connect-timeout 1 --max-time 3 -sS --connect-timeout 1 --max-time 3 -sS http://localhost:8181/context.json?sessionId=munin)
    if [ "$response" != "200" ]
    then
      printf "unomi_service.value 0"
    exit 0
    fi

Backup

To create a backup, you can now archive to an outside location the content of the folder /opt/unomi-default

Unomi front-end

All nodes are now working but the public URL for Unomi is not yet available. The related front-end will be installed on another node (for example web.int.jahia.com) which has also a public IP. The software Apache2 needs to be installed.

  • Add the file /etc/apache2/sites-enabled/unomipp.jahia.com.conf with the following content:
    <VirtualHost *:80>
        ServerName unomipp.jahia.com
        
        CustomLog /var/log/apache2/access-unomipp.jahia.com.log combined
        ErrorLog /var/log/apache2/error-unomipp.jahia.com.log
        
        RewriteEngine On
        RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
        RewriteRule .* - [F]
        
        RewriteRule ^(.*)$ https://%{SERVER_NAME}%{REQUEST_URI} [R=301,L]
    </VirtualHost>
    <IfModule mod_ssl.c>
        <VirtualHost *:443>
            ServerName unomipp.jahia.com
            ServerAdmin monitor@jahia.com
            DocumentRoot /var/www/vhosts/unomipp.jahia.com/html/
            CustomLog /var/log/apache2/access-unomipp.jahia.com.log combined
            ErrorLog /var/log/apache2/error-unomipp.jahia.com.log
            ModPagespeed off
            
            <Directory />
                Options FollowSymLinks
                AllowOverride None
            </Directory>
            
            <Directory /var/www/html>
                Options FollowSymLinks MultiViews
                AllowOverride None
                Order allow,deny
                allow from all
            </Directory>
            
            <Location /cxs>
                #List of the IPs that have the right to access the context "/cxs". It should be only localhost and internal IPs
                Require ip 127.0.0.1 10.100
            </Location>
            
            RewriteEngine On
            RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
            RewriteRule .* - [F]
            ProxyPreserveHost On
            ProxyPass /server-status !
            ProxyPass /robots.txt !
            ProxyPass /elasticsearch-head !
            RewriteCond %{HTTP_USER_AGENT} Googlebot [OR]
            RewriteCond %{HTTP_USER_AGENT} msnbot [OR]
            RewriteCond %{HTTP_USER_AGENT} Slurp
            RewriteRule ^.* - [F,L]
            
            ProxyPass / balancer://unomi_cluster/
            ProxyPassReverse / balancer://unomi_cluster/
            Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
            <Proxy balancer://unomi_cluster>
                BalancerMember http://unomipp01.int.jahia.com:8181 route=1 connectiontimeout=20 timeout=300 ttl=120 ping=500ms
                BalancerMember http://unomipp02.int.jahia.com:8181 route=2 connectiontimeout=20 timeout=300 ttl=120 ping=500ms
                BalancerMember http://unomipp03.int.jahia.com:8181 route=3 connectiontimeout=20 timeout=300 ttl=120 ping=500ms
                ProxySet lbmethod=bytraffic stickysession=ROUTEID
            </Proxy>
            
            RemoteIPHeader X-Forwarded-For
            
            Include ssl-common.conf
            
            <FilesMatch "\.(cgi|shtml|phtml|php)$">
                SSLOptions +StdEnvVars
            </FilesMatch>
            <Directory /usr/lib/cgi-bin>
                SSLOptions +StdEnvVars
            </Directory>
            BrowserMatch "MSIE [2-6]" \
                nokeepalive ssl-unclean-shutdown \
                downgrade-1.0 force-response-1.0
            BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
        </VirtualHost>
    </IfModule>
    
  • The file ssl-common.conf only contains that:
    SSLEngine on
    SSLCertificateKeyFile /etc/apache2/ssl/privkey.pem
    SSLCertificateFile /etc/apache2/ssl/fullchain.pem
    RequestHeader set X-Forwarded-Proto "https"
  • The generic SSL configuration is defined in the file /etc/apache2/mods-enabled/ssl.conf
    <IfModule mod_ssl.c>
        SSLCipherSuite HIGH:!aNULL:!MD5:!ADH:!RC4:!DH
        SSLProtocol All -SSLv2 -SSLv3 -TLSv1
        SSLHonorCipherOrder On
        Header always set Strict-Transport-Security "max-age=63072000; includeSubdomains; preload"
        Header always set X-Content-Type-Options nosniff
        # Requires Apache >= 2.4
        SSLCompression off
        SSLSessionTickets Off
        SSLUseStapling on
        SSLStaplingCache "shmcb:logs/stapling-cache(150000)"
    </IfModule>

Marketing Factory

Now, you only have to configure MF to use the Unomi front-end. You should end up with the following result:

mf.png

  •