Take thread dumps periodically for bottleneck investigation

Question

How to take periodically thread dumps to help investigate performance/bottleneck issues?

Answer

In Jahia Cloud, one of our ways to do that is to use the following script, scheduled to run each minute:

#!/bin/bash

# Be extremely careful when updating this variable as many rm -rf are based on its value
REPO="/root/thread_dumps"

# add jstack to /usr/bin if not present
if [[ ! -f "/usr/bin/jstack" ]]; then
    jstack=$(find /usr/java/openjdk-*/bin -name jstack)
    ln -s $jstack /usr/bin/
fi

# Rotate yesterday dumps if not already done
yesterday=$(date -d "yesterday" +'%m-%d-%Y')
if [[ -d "$REPO/$yesterday" ]]
then
    tar -zcf "$REPO/$yesterday.tar.gz" "$REPO/$yesterday"
    rm -rf "$REPO/$yesterday"
fi

PID=$(pgrep -u tomcat java.orig)
today=$(date +'%m-%d-%Y')
hour=$(date +'%Hh')
minute=$(date +'%M')
mkdir -p "$REPO/$today/$hour"

jstack -l $PID > "$REPO/$today/$hour/$minute"

# remove files older than 7 days
find $REPO -type f -mtime +7 -exec rm -f {} \;