Monitoring your server for performance
There are multiple ways of monitoring a Digital Experience Manager installation's behavior in real-time; we will present it in this chapter. Also, if you have identified an issue with a Digital Experience Manager installation and want to communicate it back to us, we have a section below that describes what is required to efficiently provide us with the data that will help us assist you in a timely manner.
1 Stack trace dumps
Stack trace dumps are a very useful way of figuring out exactly what the JVM is executing at a specific point in time. Basically, the JVM has a way of dumping onto the console output a list of all the threads currently executing with, for each thread, a detailed stack trace of where in the code each thread is currently If errors occur, Digital Experience Manager automatically generates thread dumps. To create thread dumps on demand you can also use the “System Health -> Memory and thread dumps“ panel in the Server Settings or a “Thread State Information” Tool available in Jahia Tools Area (see chapter “6.4 Tools”), which can also automatically create multiple thread dumps in an interval. If you want to analyze the thread dumps created by Digital Experience Manager with a tool, you may have to switch the useJstackForThreadDumps in jahia.properties to true, provided that the jstack command (from Oracle Java Platform SE package) is available in your PATH. That allows you to generate more accurate thread dumps (although the generation process is slightly slower), and it is guaranteed that in this case a dump can be read by any thread dump analyzer tool available on the market. You may also trigger such standard thread dumps manually in a Java standard way. Performing a stack trace dump is different on various platforms:
On UNIX platforms, you can send a signal to a program by using the kill command. This is the quit signal, which is handled by the JVM. For example, on Linux you can use the command kill -QUIT process_id, where process_id is the process number of your JVM. Don't be alarmed by the fact that the command is called "kill", despite the name, all this command will do is perform a stack trace dump and the JVM will continue executing. Alternatively, you can enter the key sequence <ctrl>\ in the window where the JVM was started (this works only if the java process is running in foreground in this window, not if you are doing a tail on the log file). Sending this signal instructs a signal handler in the JVM to recursively print out all the information on the threads and monitors inside the JVM.
To generate a stack trace on Windows platforms, enter the key sequence <ctrl><break> in the window where the Java program is running The output of the stack trace will go to the console output, so under Windows it will be displayed in the JVM window, and under UNIX it will be usually in tomcat/logs/catalina.out. Once the dump has been performed, you can look for threads that are blocked, or see the amount of threads that are performing some operations, which might not be expected.
A more convenient way to generate the stacktrace on all platforms is to use the JVM’s “jstack <pid>” command if you are using an Oracle Java. This will render the thread dump in your console or you could redirect an output into a file.
2 Memory dumps
In order to analyze the memory usage of a JVM, it is possible to perform memory dumps that can then later be analyzed to determine if the application is behaving as expected, or if a data structure is eating up too many resources. There are two ways of performing memory dumps with the JVM:
- via Java VM parameters:
-XX:+HeapDumpOnOutOfMemoryError writes heap dump on OutOfMemoryError (recommended)
-XX:+HeapDumpOnCtrlBreak writes heap dump together with thread dump on CTRL+BREAK
- via tools: o Oracle JMap: jmap.exe -dump:format=b,file=HeapDump.hprof<pid>
- Oracle JConsole: Launch jconsole.exe and invoke operation
- Oracle JConsole: Launch jconsole.exe and invoke operation
The heap dump will be written to the working directory.
Once you have the heap dump, you can use a Java profiler (see below) to load up the dump, but they usually have problems analyzing large files.
You could use dedicated tools, like e.g.:
- Eclipse Memory Analyzer Tool: http://www.eclipse.org/mat/
- SAP Memory Analyzer: http://wiki.scn.sap.com/wiki/display/Java/Java%20Memory%20Analysis?original_fqdn=wiki.sdn.sap.com&bc=true
What you will be looking for in memory dumps is the largest structures in memory. Usually these will be cached objects, but they may also be objects referenced from the sessions.
3 Java profilers
The most powerful tool to analyze in real-time what is going on inside a Digital Experience Manager installation is a JVM profiler. There are multiple tools that exist, but we recommend YourKit Java Profiler (http://www.yourkit.com/), which is a commercial tool that can be used even in production with lesser performance impacts.
You can find a more extensive list of profilers here:
- Free Profilers: http://www.javaperformancetuning.com/resources.shtml#ProfilingToolsFree
- Commercial Profilers: http://www.javaperformancetuning.com/resources.shtml#ProfilingToolsNotFree
Digital Experience Manager provides several tools as JSP's files that you can call to run certain commands on your server (activate Maintenance Mode, get information about the system, display thread dump, view the cache, cluster statistics etc.)
Those tools are password protected by a security realm with the Jahia Tool Manager user. Its username and password are configured during the installation wizard (defaults are: jahia/password).
The list of tools can be found after Jahia installation at http://localhost:8080/tools (adapt the URL, if you use other domains, ports or server contexts).
4.1 System and Maintenance
The tools under system and maintenance allow you to see the status of your platform. They also allow you to put your system under maintenance. This mode will display a nice page of information while you update your server (Jahia needs to be running, otherwise use a HTTP server in front to deliver a static maintenance page). The JSP pre-compiler should be run after deploying new releases of modules to pre-compile the JSPs, so that this will not happen once the server is already under load.
|System information||gather system information to analyze all the current settings|
|Thread state information||create one or multiple thread dumps|
|Memory information||show the current memory status|
|JCR sessions information||displays the information about currently running JCR sessions|
|System maintenance||set system into Maintenance mode to block access|
|JSP pre-compilation||trigger the precompiling of JSPs after deployment|
|System benchmarks||this tool will benchmark the database read performance as well as perform both read and write performance checks for the filesystem|
|Export support information||allows to export useful support information (including DX configuration files) as a ZIP file [since DX 126.96.36.199]|
These tools are here to manage your log4j configuration (change the log level for certain categories) over a user interface. Notice that these settings then only apply to the current runtime – they are not persistent, so on the next server startup the settings will be taken from log4j.xml.
You can also control the activation of the error file dumper.
|Log4j administration||tool to change log levels immediately|
|Error file dumper||ability to switch on/off error dumping to files|
4.3 Administration and Guidance
The tools in this section give you an overview of the currently running or scheduled background jobs in Digital Experience Manager, allow you to trigger re-indexing of the content and run SQL statements or Groovy scripts. As you may update a runtime database with that, you have to be very cautious and do backups before manipulations.
|OSGi console||The management and monitoring tool for the OSGi bundles|
|Background job administration||view active or scheduled background jobs|
|Search engine management||various indexing related actions and integrity checks|
|DB query tool||run SQL queries/updates using a connection from the configured Jahia data source|
|Groovy console||paste Groovy code you would like to execute against Digital Experience Manager|
|Workflow monitoring||View currently running workflow processes|
|Business rules||tool for viewing registered business rules, temporary disabling/enabling them for troubleshooting/debugging [since DX 188.8.131.52]|
4.4 Enterprise Tools – Cluster view
This section in the Jahia Tools Area is only available when the clustering is activated for Digital Experience Manager.
|Cluster view||Shows the current cluster membership, communication statistics and configuration|
4.5 JCR Data
The data tools contain a JCR repository browser that can be really helpful to browse your JCR content and have all data displayed in a particular node. You can also run JCR queries and Groovy scripts within a JCRTemplate.
Furthermore, there are housekeeping tools to clean-up the version history and to run the data store garbage collector.
|JCR repository browser||browse the JCR content tree in a simple UI.|
|JCR query tool||run JCR queries with SQL-2, XPath or SQL syntax|
|JCR query statistics||provides information about slow queries and most popular queries|
|JCR console||paste Groovy code to execute in a JCRTemplate against the JCR repository|
|JCR DataStore garbage collection||run the JCR DataStore garbage collector|
|JCR version history management||perform cleanup tasks on the version store|
|JCR integrity tools||perform some integrity checks on the JCR repository, and also implements some fixes|
|JCR external providers||Lists active external providers and current mount points|
|JCR components and nodetypes integrity tools||perform integrity checks and fixes for component nodes and node types|
4.6 JCR Rendering
Some tools to display information about the installed modules, definitions and render filters:
|Installed modules browser||display details of installed modules|
|Installed definitions browser||display details of installed node/property definitions|
|Render filters||display details of installed render filters|
|Actions||list of registered render actions|
|Choicelist initializers & renderers||Information about registered choicelist initializers and renderers|
Cache monitoring and management tools. You can also access the content of the HTML output caches if needed by accessing the following tools.
|Cache management||view the statistics of all available caches; flush particular caches or all at once|
|Output cache statistics||view the stats of the HTML cache|
|Output cache||view and search HTML cache elements|
|Output dependencies cache||view and search HTML dependencies cache elements|
4.8 Miscellaneous Tools
These are various tools that could not be classified into the other categories.
|Password encryption||tool to encrypt passwords|
|Document converter||convert documents into other formats if the conversion service is active|
|Document text extractor||check the text extraction from documents|
|WCAG checker||paste HTML to validate against Web Content Accessibility Guidelines|
|URL rewriting rules||view the rules for the UrlRewriteFilter|
|CKEditor configuration||allows creating and deploying custom CKEditor (rich text editor) configuration|
5 Other Issues
The best way to get support for your issues is to contact us for a support agreement. Please see the following page for more information: https://www.jahia.com/enterprise-now/support
If you have a commercial support contract, you will get your own space to submit issues that will be handled according to our SLA (https://support.jahia.com/). Otherwise, you can report issues to the general JIRA projects (https://jira.jahia.org/), but here there will be no guarantee as to how and when the issue will be handled. When submitting an issue to our JIRA Issue tracker, make sure you include as much information as possible, including:
- A detailed description of your environment with the version number and patches (J2EE server, JDK, OS) as well as memory and architecture (32-bit, 64-bit).
- A detailed (or complete) log file, including date and times at which the problem occurs, to be able to corroborate with log file.
- A list of steps to reproduce the problem (if not random).
- A stack trace dump or, in case of performance issues, multiple thread dumps in intervals (see chapter “6.1 Stack trace dumps”).
- If dealing with an OutOfMemory issue, please include a memory dump (see chapter “6.2 Memory dumps”).
A convenient way to get all the relevant system, Digital Experience Manager and environment information is to use the “Jahia Tools Area -> System information -> download as a file” action, which will allow you to download and later attach to the JIRA ticket the relevant information.
As a basic rule, we also prefer to have too much information than too little.