JavaScript modules under the hood / How it works

In this section, we will go under the hood of how JavaScript modules actually work in Jahia’s DXP. We will also dive deeper into some related technologies.

About NPM packaging

The central NPM packaging descriptor, the package.json file, can contain scripts that are basically used to perform tasks such as building, deploying, or even watching the current project.

The NPM package manager is responsible for using the package.json file, downloading all the required dependencies, and then executing the requested scripts.

The package manager can offer some useful tooling, such as the watch feature that makes it possible to have the package manager watch the project for any changes, and then launch scripts upon detected changes, that could rebuild and redeploy the changes to a server for testing.

Jahia, starting with version 8.2, is now capable of deploying JavaScript modules built using a package manager. Jahia by default uses the Yarn package manager, and although it is possible to use another one, it is usually a good idea to only use a single package manager across multiple projects to avoid maintenance complexities over time.

NPM Package (tgz) to OSGi Bundle Transformation

On deployment, NPM packages (in tgz file format) are dynamically transformed into OSGi bundles and then treated exactly the same way that Jahia handles regular OSGi bundles. This transformation has two main parts: file/folder relocation and metadata transformation.

File/Folder relocation

During the transformation process, the first step is to move some of the files coming from the NPM tgz package into directories inside an OSGi bundle that Jahia can then process and understand. The table below details which relocations are performed.

NPM Source path	OSGi Module Destination Path
`package.json`	`package.json`
`import.xml`	`META-INF/import.xml`
`settings/*`	`META-INF/*`

Metadata transformation

The metadata transformation process extracts the data from the package.json file and copies it to the OSGi bundle META-INF/MANIFEST.MF descriptor file. The table below explains how the data is transformed:

package.json property	MANIFEST.MF Clause	Default value	Notes
`name`	`Bundle-Name`	`N/A`	appends (npm module) after the name
`description`	`Bundle-Description`	`N/A`
`name`	`Bundle-SymbolicName`	`N/A`	(by removing all the unallowed characters such as @, / and replacing any other characters with _
`author`	`Bundle-Vendor`	`N/A`
`version`	`Bundle-Version`	`N/A`
`license`	`Bundle-License`	`N/A`
`jahia.category`	`Bundle-Category`	`jahia-npm-module`
`jahia.module-dependencies`	`Jahia-Depends`	`default`
`jahia.deploy-on-site`	`Jahia-Deploy-On-Site`	`N/A`
`jahia.group-id`	`Jahia-GroupId`	`org.jahia.npm`
`jahia.module-signature`	`Jahia-Module-Signature`	`N/A`
`jahia.module-priority`	`Jahia-Module-Priority`	`N/A`
`jahia.module-type`	`Jahia-Module-Type`	`module`
`jahia.required-version`	`Jahia-Required-Version`	`8.2.0.0`
`jahia.server`	`Jahia-NPM-InitScript`	`N/A`
`jahia.static-resources`	`Jahia-Static-Resources`	`/css,/icons,/images,/img,/javascript`	Used to define which directories to expose publicly

About Yarn

Yarn is an established open-source package manager who manages dependencies in JavaScript projects. It assists with installing, updating, configuring, and removing package dependencies.

Other package managers exist, the most notable one being NPM, which is provided directly by the NodeJS project. However, Yarn is usually more innovative and faster than NPM. You can find below a quick comparison table of NPM and Yarn.

Be careful with Yarn versions.

Yarn is a project that is evolving quickly, and it has undergone a major rewrite in version 2. Currently, the most recent version of Yarn is version 4, and it is the recommended one to use for Jahia JavaScript modules. It is not recommended to use Yarn version 1 anymore.

Yarn 4 uses by default a new “linker” (a way to reference dependencies) that is called PnP, which might cause some issues with tools that don’t yet support Yarn 4 (such as the Cypress test framework), so PnP is deactivated in jahia's projects. But you can configure it by removing this line from the .yarnrc.yml file :

nodeLinker: node-modules

You can find more information about nodeLinkers here.

About Webpack

From WebPack’s official website: “At its core, Webpack is a static module bundler for modern JavaScript applications. When Webpack processes your application, it internally builds a dependency graph from one or more entry points and then combines every module your project needs into one or more bundles, which are static assets from which to serve your content.”

The official definition is a little difficult to understand for people with less experience with JavaScript, so we will present a more generic one. WebPack is basically a script that is launched from an NPM package manager such as Yarn, and that will analyze all the project’s dependencies and resources and optimize them to build a more optimized version that will allow faster loading from a browser. It uses a configuration that can define different rules depending on the type of resources to “compile” such as JavaScript, CSS, SASS, and others. For example, it might aggregate all the JavaScript located in different files across a project to generate a single JavaScript file that will be faster to load as it requires fewer HTTP requests.

Jahia’s NPX project creator will automatically create projects that include WebPack as the default bundler and provide a default configuration that should fit most simple to medium complex projects (it even includes support for client-side JavaScript bundling).

About GraalVM, GraalVM Native Image and GraalJS

The GraalVM JDK, and especially its GraalJS sub-project, are the underlying technologies that enable the functionality of Jahia’s JavaScript modules. We will now introduce all the different parts of these technologies, including some that are not used and why.

GraalVM

GraalVM is a high-performance virtual machine designed to improve the efficiency and performance of Java applications. It extends the Java Virtual Machine (JVM) by introducing a new just-in-time compiler that is written in Java. GraalVM supports not only Java but also additional programming languages such as JavaScript, Ruby, Python, and R. It can run standalone or be embedded into other systems.

GraalVM Native Image

GraalVM Native Image, part of GraalVM, allows for ahead-of-time (AOT) compilation of Java applications into standalone executable files. These native executables do not require a JVM to run, which reduces applications' startup time and memory footprint. However, since these executables compile at build time, they do not dynamically optimize at runtime, which traditional JVMs do. Also, they cannot be compatible with dynamic class-loading applications, such as OSGi applications and therefore Jahia DXP is not compatible with Native Image.

GraalJS

GraalJS is a high-performance JavaScript runtime implemented on top of the GraalVM. It aims to provide better performance compared to Nashorn, the previous JavaScript engine bundled with JDK. GraalJS is fully ECMAScript 2019 compliant and supports Node.js applications and libraries.

Comparison with OpenJDK

OpenJDK is the open-source implementation of the Java Platform, Standard Edition. It includes the Java Runtime Environment (JRE), the Java Development Kit (JDK), and the JVM. OpenJDK operates only Java applications and uses the HotSpot JVM with the Just-In-Time (JIT) compiler. In contrast, GraalVM offers a pluggable architecture that supports multiple languages and enables both JIT and AOT compilation strategies.

Comparison with Nashorn

Nashorn was introduced in Java 8 as a replacement for the older Rhino JavaScript engine. It provided a JavaScript runtime in Java and was implemented as part of the JDK. However, Nashorn has been deprecated since Java 11 due to the advent of more efficient alternatives like GraalJS. Unlike Nashorn, GraalJS can interoperate with code written in other languages supported by GraalVM and provides better performance and support for modern JavaScript features.

GraalVM in Jahia JavaScript modules

For Jahia JavaScript modules, the DXP server must be configured to execute with GraalVM to take advantage of modern JavaScript features and improved performance. Although GraalVM offers the GraalVM Native Image feature, it is not compatible with OSGi (a fundamental framework used by Jahia for modularity). Consequently, Jahia does not utilize GraalVM Native Image. Although it is technically possible to use GraalJS without the full GraalVM JDK on top of OpenJDK, this combination comes at a performance cost and is not recommended for production.

Jahia Docker Images

The default Jahia Docker images come fully packaged with all necessary components for usage, including the configured GraalVM environment. This setup ensures that developers can use Jahia with minimal initial configuration and harness the full capabilities of GraalVM and its supported technologies right out of the box.

Npm-modules-engine Initialization Process

The initialization process of the npm-modules-engine module contains a lot of different steps that are summarized in the following sequence diagram:

Here are the details of the steps:

The OSGi Framework requests the global variables factory from the GraalVMEngine. This factory is crucial for initializing global variables across JavaScript contexts.
The OSGi Framework calls the activate method of the GraalVMEngine, which triggers the engine to start its services and prepare for execution.
Within the GraalVMEngine, the main initialization script is retrieved. This script contains essential startup procedures for the engine. Initialization properties are also fetched. These properties configure the behavior and settings of the GraalVMEngine.
The GraalVMEngine initializes the GenericObjectPool, which manages the pooling of ContextProviders, which is essential for managing multiple JavaScript execution contexts.
The GenericObjectPool instructs the ContextPoolFactory to create ContextProviders. These providers handle the creation and lifecycle management of JavaScript contexts.
The ContextPoolFactory initializes JavaScript Context instances. These contexts are environments where JavaScript code is executed.
ContextProviders are initialized with a JavaScript Context. Each provider manages a specific JavaScript Context, ensuring isolated execution environments.
Global variables are added to each JavaScript Context. These variables are typically shared across various scripts and provider helper classes to JavaScript code.
All relevant initialization scripts are executed within each JavaScript Context. This step is critical for setting up the environment, loading libraries, and performing startup tasks specific to the application's needs.
The OSGi Framework activates the NpmModuleListener object, which listens for changes in JavaScript modules, such as their loading and unloading.
The NpmModuleListener communicates with the GraalVMEngine to enable or disable modules.
The GraalVMEngine registers or unregisters the initialization scripts associated with different bundles. This ensures that each bundle's JavaScript environment is correctly set up or torn down.
The GraalVMEngine increments its version number whenever significant changes occur, such as adding or removing scripts. The version number is used to invalidate previously created ContextProviders that might not contain the latest context for the currently deployed scripts.
The NpmModuleListener registers itself with the OSGi Framework as a BundleListener so that it can reactivate to module deployment/undeployment.
Upon a bundle change event signaled by the OSGi Framework, the NpmModuleListener reacts by enabling or disabling modules again, as necessary, ensuring that the system remains responsive to the changes in the module's lifecycle.
Similar to earlier steps, the GraalVMEngine continues to manage the registration of scripts and increment its version to reflect ongoing changes and ensure system integrity.

Request execution

Detailed steps:

Browser to Jahia servlet: a request comes in for a page with a specific URL
Jahia Servlet to TemplateNodeFilter: the template node filter retrieves the template identifier for the page
The TemplateNodeFilter asks the (GraalVMEngine) ViewsRegistrar is asked to resolve the JavaScript for the template
The returned JSScript asks the GraalVMEngine to render the template
The GraalVMEngine retrieves a ContextProvider from its pool and renders the template using the GraalJS JavaScript engine
While the template is rendered, it might call the <Area> or a <Render> React Component that may call the RenderService to render some views.
The RenderService will call the ViewsRegistrar to resolve any views for the content that needs to be displayed.
The ViewsRegistrar will return a JSScript to be executed
The JSScript will call the GraalVMEngine to render the view
The GraalVMEngine will retrieve a ContextProvider from its pool to render the view
The ContextProvider will render the view using the GraalJS JavaScript runtime
Once all the views for the template have been executed, they are aggregated in the template, and the final HTML is returned to the browser

Npm-modules-engine main components

GraalVM Engine

The GraalVM engine serves as the core component for executing NPM plugin modules. It facilitates the creation of a pool of polyglot contexts, enabling the execution of JavaScript or any other language supported by GraalVM.

Engine and Context Pool

The service consists of a shared GraalVM Engine and a pool of JavaScript (JS) polyglot contexts (aka Context Pool). JavaScript code can be executed within these polyglot contexts. It's important to note that polyglot contexts are not thread-safe and should only be utilized by a single thread at a time. To accommodate multiple threads executing JavaScript in different contexts, we provide a pool of ContextProvider instances, each containing a polyglot context. When a new context is created, global variables from Java objects are bound, facilitated by instances of JSGlobalVariableFactory. Initialization scripts - including the main script from src/javascript/index.js which initializes available frameworks and provides polyfills - are executed with each new context creation. Additionally, every NPM plugin provides its own initialization script executed on context creation. In order to properly manage script changes, a local version number is incremented whenever a new script is added or removed, indicating outdated contexts that should be removed from the pool.

Configuration

The engine can be configured using the org.jahia.modules.npm.modules.engine.jsengine.GraalVMEngine.cfg configuration file. Keys prefixed by polyglot. are passed as options to the engine builder. For instance, experimental enables experimental features, while polyglot.inspect enables the JavaScript debugger. All available options can be explored using polyglot --help:all shell commands installed by the GraalVM SDK.

Module Registration and Initialization Scripts

A bundle listener, NpmModuleListener, listens to starting and stopping bundles. When a bundle is started, the following flow is executed:

Check for the presence of a package.json at the root level with a jahia.server entry, indicating an NPM module.
The initialization script that is referenced by the jahia.server entry is a single executable JS file compiled with Webpack and is added to the list of scripts for execution.
An internal version number is incremented to invalidate existing contexts in the pool.
A new context is obtained from the pool, and all available registrars are called to register Jahia extensions by transforming JS objects in the registry into OSGi services.

When a bundle is stopped, the script is unregistered, and the version number is incremented to invalidate existing contexts.

Note: Initialization scripts initialize the JavaScript context immediately after creation, rather than executing code on module start. These scripts are called whenever a JavaScript context needs to be created.

PAX-URL Protocol

Jahia uses PAX-URL to handle URLs to OSGi bundles. The NpmProtocolStreamHandler introduces a new npm protocol (npm://). This protocol acts as a wrapper around any other protocol, enabling the installation of NPM modules with URLs like npm://file://xxx.tgz or npm://http://xxx/xx/tgz. You can learn more about PAX URL here: https://ops4j1.jira.com/wiki/spaces/paxurl/pages/12058914/Documentation

I18N Setup

This setup enables the use of i18next in the JavaScript context. Translations are directly read from locales/<locale>.json within the current bundle.

The JavaScript server-side registry

Jahia uses a Javascript object registry to share objects between various scripts. The registry is similar to an OSGi service registry for those familiar with OSGi. Here is the interface of the registry:

package org.jahia.modules.npm.modules.engine.jsengine;

import java.util.List;
import java.util.Map;

public interface Registry {

    /**
     * Retrieves the map of objects stored under the combined type and key.
     * 
     * @param type The type category of the object.
     * @param key The unique key identifying the object within the type.
     * @return Map of objects if found, null otherwise.
     */
    Map<String, Object> get(String type, String key);

    /**
     * Finds all entries in the registry that match the provided filter criteria.
     * 
     * @param filter A map representing filter criteria where each key-value pair specifies a required property.
     * @return A list of maps that match the filter criteria.
     */
    List<Map<String, Object>> find(Map<String, Object> filter);

    /**
     * Finds all entries in the registry that match the provided filter criteria and sorts them based on the specified order.
     * 
     * @param filter A map representing filter criteria where each key-value pair specifies a required property.
     * @param orderBy The attribute name to sort the results by. If null, sorting is not performed.
     * @return A sorted list of maps that match the filter criteria.
     */
    List<Map<String, Object>> find(Map<String, Object> filter, String orderBy);

    /**
     * Adds a new entry to the registry. If an entry with the same type and key already exists, an exception is thrown.
     * 
     * @param type The type category under which the entry will be stored.
     * @param key The unique key for identifying the entry.
     * @param arguments Variable number of maps containing the properties of the entry.
     * @throws IllegalArgumentException if an entry with the same type and key already exists.
     */
    void add(String type, String key, Map<String, Object>... arguments);

    /**
     * Adds a new entry to the registry or replaces an existing entry with the same type and key.
     * 
     * @param type The type category under which the entry will be stored.
     * @param key The unique key for identifying the entry.
     * @param arguments Variable number of maps containing the properties of the entry.
     */
    void addOrReplace(String type, String key, Map<String, Object>... arguments);

    /**
     * Removes an entry from the registry based on its type and key.
     * 
     * @param type The type category of the entry to be removed.
     * @param key The unique key identifying the entry to be removed.
     */
    void remove(String type, String key);

    /**
     * Combines multiple sets of properties into a single map. This method is typically used when creating or updating entries.
     * 
     * @param arguments Variable number of maps that need to be merged into one.
     * @return A single map containing all the merged properties.
     */
    Map<String, Object> composeServices(Map<String, Object>... arguments);
}

Registrars

Registrars play a crucial role in parsing NPM modules to detect and register specific extensions, typically written in JavaScript or any language requiring the GraalVM engine. They are invoked when an NPM plugin is started, following the execution of the initialization script.

Here is the interface for a Registrar:

/**
 * The Registrar interface provides the methods required for registering
 * and unregistering resources associated with a specific OSGi bundle.
 * Implementations of this interface are responsible for managing the
 * lifecycle of resources such as render filters, views, or any other
 * entities that need to be dynamically registered and unregistered
 * in response to bundle lifecycle events.
 *
 * <p>This interface is typically implemented by classes that handle
 * the integration of JavaScript-defined components (such as views or
 * render filters) into a Java-based application, leveraging the
 * GraalVM's polyglot capabilities to execute JavaScript within the
 * context of an OSGi-based application.</p>
 */

public interface Registrar {

    /**
     * Registers resources associated with the specified bundle.
     * This method is called to handle the initialization and registration
     * of resources when a bundle is started or when the resources need
     * to be refreshed.
     *
     * @param bundle the OSGi bundle whose resources are to be registered.
     *               This is not null and is the bundle containing the resources
     *               that need to be managed by this registrar.
     */
    void register(Bundle bundle);

    /**
     * Unregisters all resources associated with the specified bundle.
     * This method is called to clean up resources associated with a bundle
     * when the bundle is stopped or when the resources are no longer needed.
     *
     * @param bundle the OSGi bundle whose resources are to be unregistered.
     *               This is not null and is the bundle containing the resources
     *               that have been previously registered and now need to be
     *               cleaned up.
     */
    void unregister(Bundle bundle);
}

Jahia’s npm-modules-engine provides two Registrar implementations:

The RenderFilterRegistrar dynamically registers and unregisters JavaScript-defined render filters. It manages lifecycle events to ensure filters are available as services when needed and are properly cleaned up on bundle deactivation.
The ViewsRegistrar handles the registration and management of JavaScript views and templates, integrating them into Jahia's rendering pipeline as part of the template and script resolution processes.

Render Filters Registrar

The render filter registrar's objective is to register "render filters" written in JavaScript, sourced from the registry with the render-filter type. It accomplishes this by creating an OSGi service (RenderFilterBridge) that delegates to the JS prepare/execute functions.

Render filters can be added to the registry in the form of a pair of two methods: execute and prepare.

registry.add("render-filter", "test", renderFilterTest, {
    target: 'render:50',
    applyOnNodeTypes: 'jnt:bigText',

    prepare: (renderContext, resource, chain) => {
        // Preparation logic here
    },
    execute: (previousOut, renderContext, resource, chain) => {
        return previousOut.replace('toto', 'tutu');
    }
})

The target must be rendered, followed by the filter priority.

Views Registrar

The ViewsRegistrar in Jahia’s npm-modules-engine is a versatile class responsible for dynamically managing JavaScript views within the context of OSGi bundles. It functions as both a ScriptResolver and TemplateResolver, identifying and applying appropriate JavaScript scripts and templates to content resources based on their context. Upon bundle activation, it registers JavaScript views by scanning the bundle, parsing files, and storing representations, which are then made available for rendering. Conversely, it unregisters these views upon bundle deactivation to maintain system integrity and free resources. The registrar optimizes performance through caching mechanisms, speeding up the resolution process and responding efficiently to system events that require cache updates. This integration allows for a seamless blend of traditional Java-based management and modern JavaScript-driven web technologies within Jahia’s platform, enhancing content management capabilities and rendering flexibility.

JavaScript to Java seamless integration

Although it is possible to use Java classes from JavaScript, there are some limitations around this functionality as well as some good reasons as to why this shouldn’t be done systematically. In general it is better to provide additional functionality to JavaScript modules through Java modules provides new JavaScript helper functions (see next section about how to do this) for security and maintenance reasons.

The built-in mechanism that GraalJS provides for accessing Java class is illustrated in the following example:

// Import the Java types from the Log4J library
const LogManager = Java.type('org.apache.logging.log4j.LogManager');
const Level = Java.type('org.apache.logging.log4j.Level');

// Create a logger instance
const logger = LogManager.getLogger("JavaScriptLogger");

// Log messages at different levels
logger.info("This is an informational message from JavaScript!");
logger.error("This is an error message from JavaScript!");
logger.log(Level.DEBUG, "This is a debug message from JavaScript!");

We can see in this example that a Java.type statement is provided by GraalJS to enable access to a Java class.

Important class accessibility limitation: This mechanism only works with Java classes provided by Jahia’s core. It will not work with classes packaged and exposed by modules.