Capturing a Java heap dump when Application Server memory spikes in PaperCut NG/MF

Before you start

• Ensure adequate disk space. A heap dump is roughly the size of the memory Java is using when the dump is taken, and it’s written uncompressed. Make sure the drive has room for the dump, the logs, and normal server activity, and check the free space again just before you capture.

• Set permissions: The account the Application Server runs under must have write access to the folder you’re dumping into:

  • Windows: the account is under the Log On tab of the PaperCut Application Server service in services.msc.
  • Linux and macOS: the Application Server runs as the papercut account.

  Keep the folder locked down to that account.

• Determine the best time to perform the dump: Writing a heap dump pauses the Application Server for as long as the write takes, and it puts a heavy load on the disk. On a test server, a 316 MB heap took a little over two seconds, so scale that up for your own heap size and disk speed. On a large heap, the pause is noticeable to users, so agree on a sensible window with your Support contact before you start.

• Turn on debug mode: Before you reproduce the problem or start waiting for it, turn on debug mode and note the capture time and time zone. Turning debug on afterwards doesn’t add any detail to what has already happened. See How to Collect Application Server Debug Logs.

Method 1: Take a memory snapshot from the admin interface

To take a memory snapshot from the admin interface:

1. In the admin interface, go to the Options > Advanced > Diagnostics section and click Show advanced diagnostic options.

2. While memory is still high, click Save Memory Snapshot and let it finish.

   Note: If memory was already on its way back down when you clicked, your Support contact will ask for another snapshot while it’s high.

3. Make a copy of the snapshot from the primary Application Server: [app-path]\server\logs\memory-snapshots\memory-snapshot.zip

   Note: Each snapshot replaces the last one, so copy the file somewhere else first if you want to keep it.

4. Send the file to your Support contact using the upload link they provide.

5. Delete your local copy once they confirm it has arrived.

For more on snapshots, including thread snapshots, see Using the Snapshots to Gather Information from PaperCut NG and PaperCut MF.

Method 2: Configure an automatic dump when the server runs out of memory

This adds two Java arguments to the Application Server. When the server runs out of Java heap, it writes a .hprof heap dump before it reports the error, so you get the evidence without having to be at the keyboard. This method doesn’t fire for every out-of-memory error, though. Running out of Metaspace writes a dump, but the server not being able to start a new thread doesn’t.

Before you start

Two things are worth knowing before you set up this method:

• An out-of-memory error doesn’t always stop the server, so the server can write a dump and carry on running.
• The Application Server also only writes one dump per start, so after the first dump, restart the server before you wait for the next one.

Windows and Linux

How to configure a dump to automatically start when the server runs out of memory

To configure an automatic dump:

1. Create the folder for the dumps, for example:

   • Windows: C:\PaperCut\heapdumps
   • Linux: /var/tmp/papercut-heapdumps

2. Give the Application Server account write access to the folder. Use a path with no spaces.

3. Stop the Application Server and take a backup of [app-path]/server/custom/service.conf.

4. Have a look at the base configuration file for your platform, but don’t change it:

   • Windows: [app-path]\server\bin\win\service.conf
   • Linux: [app-path]/server/bin/linux-x64/app-monitor.conf

5. Find the numbered wrapper.java.additional lines. PaperCut keeps several of these numbers spare for you, and they hold the placeholder value -Dpc-reserved=X.

   Note: For v24.1, the spare lines are 14 to 21. For v25.0 and v26.0 they are 11 to 18.

6. Pick two consecutive placeholder numbers that your custom/service.conf isn’t already using. The numbering changes between versions, so check the file on your server rather than copying numbers from somewhere else.

7. Add the two arguments to [app-path]/server/custom/service.conf, using the two numbers you picked.

   • On Windows, add:

     wrapper.java.additional.14=-XX:+HeapDumpOnOutOfMemoryError
     wrapper.java.additional.15=-XX:HeapDumpPath=C:\PaperCut\heapdumps
     

   • On Linux, add:

     wrapper.java.additional.14=-XX:+HeapDumpOnOutOfMemoryError
     wrapper.java.additional.15=-XX:HeapDumpPath=/var/tmp/papercut-heapdumps
     

   Important: Always add your changes to the custom file rather than the base file, because the custom file survives an upgrade.

8. Start the Application Server and check the settings have been applied.

9. After the dump has been saved, send it to your Support contact.

Note: It’s worth checking the settings again after you upgrade PaperCut NG/MF, because the base file and the spare numbers can move.

macOS

There are no numbered lines on macOS. The launch script passes whatever is in PC_CUSTOM_SERVER_ARG after PaperCut’s own arguments.

To configure a dump to automatically start when the server runs out of memory:

1. Create the folder for the dumps, for example, /Library/Logs/PaperCut/heapdumps, and give the papercut account write access to it.

2. Stop the Application Server and take a backup of [app-path]/server/custom/launch-app-server.conf.

3. Open launch-app-server.conf and add the following two arguments to PC_CUSTOM_SERVER_ARG.

   export PC_CUSTOM_SERVER_ARG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/Library/Logs/PaperCut/heapdumps"
   

   Note: If that variable already has settings in it, keep them and add yours to the same line, because a second export PC_CUSTOM_SERVER_ARG line replaces the first rather than adding to it.

4. Start the Application Server and check the settings have been applied.

5. After the dump has been saved, send it to your Support contact.

How to check the settings have been applied

Look at the arguments the running server was started with.

• On Windows, in PowerShell:

  Get-CimInstance Win32_Process -Filter "Name='pc-app.exe'" | Select-Object -ExpandProperty CommandLine
  

• On Linux and macOS:

  ps -ef | grep pc-app
  

Both of your arguments appear in that line. If they don’t, your changes haven’t been picked up, so go back and check the numbers you used on Windows and Linux, or the PC_CUSTOM_SERVER_ARG line on macOS.

Where the memory dump is written

Give -XX:HeapDumpPath a folder rather than a file name. The dump is then written into that folder as java_pid<pid>.hprof.

The operating system reuses process IDs, so a later dump can end up with the same file name as an earlier one. Java doesn’t overwrite an existing dump file. If the folder already contains a file with the same name, the server doesn’t write the new dump and you lose the capture you were waiting for. To avoid this, move or delete each dump after your Support contact confirms they have it.

The same happens if you set -XX:HeapDumpPath to a full file name instead of a folder and that file already exists. The server logs Unable to create ...: File exists and doesn’t write the dump.

Follow the steps in How to send the dump to your support contact at the end of this article.

What to do after your Support contact has the dump

After your Support contact confirms they have the dump, stop the Application Server, remove the two lines you added, and start it again.

On macOS, keep anything that was already in PC_CUSTOM_SERVER_ARG.

On all operating systems, delete any local dumps you no longer need and turn debug logging back off.

Method 3: Create a script to automatically dump memory when it passes a set threshold

If memory climbs and then falls back without an out-of-memory error, Method 2 has no error to trigger on. The scripts below watch memory for you and take the dump the moment it goes past a level you choose.

Every five seconds, the scripts use jcmd to read the memory in use. When memory passes the threshold, they use jcmd to take the dump. jcmd ships with PaperCut, in the Java folder underneath the install:

• Windows: [app-path]\runtime\win64\jre\bin
• Linux: [app-path]/runtime/linux-x64/jre/bin
• macOS: [app-path]/runtime/mac/jre/Contents/Home/jre/bin

Use PaperCut’s copy of jcmd from the folder above, not one from elsewhere on the machine, so it matches the Java version the server is running, which differs for each release line.

If the paths above don’t match your install, search the runtime folder for jcmd.

Before you start

Agree with your Support contact on the threshold. It should be above the level memory normally settles back to, not as a fraction of the maximum heap size. For example, if the server usually settles around 3 GB on an 8 GB heap, then 6 GB is a sensible trigger. 7.5 GB is too close to the ceiling, because the server may already be in trouble before the dump starts.

The scripts read and report in GB, the same unit Java uses for the heap size, so whatever number you agree is the number you type in. The script output labels this unit GiB.

How to create a script to automatically dump at a set memory threshold

To create and run the script:

1. If you haven’t already, create the folder for the dumps and give the Application Server account write access to it, as in Method 2. Use a path with no spaces.
2. Find the Application Server’s process ID. The script needs it so it knows which process to watch. See How to find the process ID below.
3. Save the script for your operating system as a new file. The scripts and the steps to run them are in the Windows and the Linux and macOS sections below.
4. Fill in the four values at the top of the script:
   • The process ID you found in step 2.
   • The threshold you agreed with your Support contact. Use whole numbers or decimals. For example, for a trigger at 5.5 GB, set THRESHOLD_GB=5.5 in the shell script, or $ThresholdGB = 5.5 in the PowerShell script.
   • PaperCut’s Java folder, the one that holds jcmd.
   • The dump path. The script takes one dump and then stops, so point it at a new file name inside the dumps folder, with no spaces in the path.
5. Run the script as the account the Application Server runs under, following the steps for your operating system below.
6. Leave the script running until it finishes, or press Ctrl+C to stop it early.
7. Send the dump to your Support contact. See How to send the dump to your support contact.

Note: A short spike can still rise and fall between two checks, so the dump isn’t guaranteed. If the first attempt misses it, shorten the five second wait at the bottom of the script (Start-Sleep -Seconds 5 or sleep 5) and run it again.

Note: Confirm the process ID again after any restart, because the script stops if the process it was watching goes away.

How to find the process ID

On Windows, in PowerShell:

Get-CimInstance Win32_Process -Filter "Name='pc-app.exe'" | Where-Object { $_.CommandLine -like '*WrapperSimpleApp*' } | Select-Object ProcessId

There can be more than one pc-app.exe, which is why the command filters on the wrapper class.

On Linux and macOS, run ps -ef | grep pc-app. The process ID is in the second column of the output.

You can also find the process ID on any OS using PaperCut’s copy of jcmd with -l (a lowercase L), which lists the Java processes. The Application Server is in the line showing:

• Windows and Linux: org.tanukisoftware.wrapper.WrapperSimpleApp
• macOS: biz.papercut.pcng.server.AppServer

Run jcmd -l as the account the server runs under, because each account only sees its own Java processes. On Windows, an administrator window lists no PaperCut process at all.

Windows

The service runs as Local System by default, and an administrator window can’t read memory from a process owned by SYSTEM. If you run the script as yourself, it stops with sun.jvmstat.monitor.MonitorException: <pid> not found, even though the account is an administrator.

To run the script on Windows:

1. Save the script at the end of this section as papercut-heapdump-watch.ps1 in a folder such as C:\PaperCut, and fill in the four values at the top.

2. Check which account the service runs as, on the Log On tab of the PaperCut Application Server service in services.msc.

3. Open a PowerShell window as that account:

   • Local System: start PowerShell as SYSTEM using PsExec from Microsoft Sysinternals, which you can get here: https://learn.microsoft.com/sysinternals/downloads/psexec Start it like this:

     psexec -s -i powershell.exe
     

   • A named account: log in as that account, or start PowerShell as it.

4. In that PowerShell window, change to the folder that holds the script and run it:

   powershell -ExecutionPolicy Bypass -File .\papercut-heapdump-watch.ps1
   

5. Leave the window open until it finishes, or press Ctrl+C to stop it early.

If you would rather not install PsExec, a scheduled task set to run as SYSTEM does the same job as steps 3 and 4:

schtasks /create /tn PaperCutHeapDump /tr "C:\PaperCut\run-watcher.cmd" /sc once /st 23:59 /ru SYSTEM /rl HIGHEST /f
schtasks /run /tn PaperCutHeapDump
schtasks /delete /tn PaperCutHeapDump /f

where run-watcher.cmd holds one line starting the script and sending its output to a file:

powershell -NoProfile -ExecutionPolicy Bypass -File "C:\PaperCut\papercut-heapdump-watch.ps1" > "C:\PaperCut\watcher-output.txt" 2>&1

Windows script:

# Fill in the four values below. Agree the process ID and the threshold with your
# PaperCut support contact before you run this.
$TargetPid   = 0   # The Application Server's process ID
$ThresholdGB = 0   # Heap used, in GiB, that you want the dump taken at
$JavaBin     = ''  # PaperCut's own Java bin folder, the one holding jcmd.exe
                   # e.g. C:\Program Files\PaperCut MF\runtime\win64\jre\bin
$DumpPath    = 'C:\PaperCut\heapdumps\MemoryDump.hprof'   # A file in the folder you created


# Runs a tool and gives up if it stops responding, so the watcher cannot hang forever.
function Invoke-Tool {
    param([string]$FilePath, [string[]]$Arguments, [int]$TimeoutSeconds)
    $proc = New-Object System.Diagnostics.Process
    $proc.StartInfo.FileName               = $FilePath
    $proc.StartInfo.Arguments              = ($Arguments -join ' ')
    $proc.StartInfo.UseShellExecute        = $false
    $proc.StartInfo.CreateNoWindow         = $true
    $proc.StartInfo.RedirectStandardOutput = $true
    $proc.StartInfo.RedirectStandardError  = $true
    try {
        [void]$proc.Start()
        $out = $proc.StandardOutput.ReadToEndAsync()
        $err = $proc.StandardError.ReadToEndAsync()
        if (-not $proc.WaitForExit($TimeoutSeconds * 1000)) {
            try { $proc.Kill() } catch { }
            return [pscustomobject]@{ TimedOut = $true; ExitCode = -1; Output = '' }
        }
        $text = $out.GetAwaiter().GetResult() + $err.GetAwaiter().GetResult()
        return [pscustomobject]@{ TimedOut = $false; ExitCode = $proc.ExitCode; Output = $text }
    } finally {
        try { if (-not $proc.HasExited) { $proc.Kill() } } catch { }
        $proc.Dispose()
    }
}

$ErrorActionPreference = 'Stop'
try {
    if ([string]$TargetPid -notmatch '\A[1-9][0-9]*\z') {
        throw 'Set TargetPid to the process ID of the Application Server, as a whole number.'
    }
    $TargetPid   = [int]$TargetPid
    $ThresholdGB = [double]$ThresholdGB
    if ([double]::IsNaN($ThresholdGB) -or [double]::IsInfinity($ThresholdGB) -or $ThresholdGB -le 0) {
        throw 'Set ThresholdGB to a number greater than zero.'
    }
    $jcmd = Join-Path $JavaBin 'jcmd.exe'
    if (-not (Test-Path -LiteralPath $jcmd -PathType Leaf)) { throw "jcmd.exe was not found in $JavaBin." }
    if ($DumpPath -notmatch '\A[A-Za-z]:[\\/]' -or $DumpPath -match '[\s"''=]') {
        throw 'Use a full path on a local drive, with no spaces, quotation marks or equals signs.'
    }
    $dumpFolder = Split-Path -Parent $DumpPath
    if (-not (Test-Path -LiteralPath $dumpFolder -PathType Container)) {
        throw 'Create the dump folder and give the service account write access.'
    }
    if (Test-Path -LiteralPath $DumpPath) { throw "$DumpPath already exists. Choose a new file name." }

    $probe = Invoke-Tool -FilePath $jcmd -Arguments @("$TargetPid", 'VM.version') -TimeoutSeconds 30
    if ($probe.TimedOut -or $probe.ExitCode -ne 0) {
        throw "Cannot attach to PID $TargetPid. Check the PID, and that you are running as the account the server runs under."
    }
    $targetProcess = Get-Process -Id $TargetPid
    $null = $targetProcess.Handle
    $thresholdKB = $ThresholdGB * 1MB
    if ([double]::IsInfinity($thresholdKB)) { throw 'ThresholdGB is too large.' }
    if ([math]::Round($thresholdKB) -lt 1) { throw 'ThresholdGB is too small to use.' }
    Write-Host "Watching PID $TargetPid for Java heap usage above $ThresholdGB GiB. Press Ctrl+C to stop."

    while ($true) {
        $targetProcess.Refresh()
        if ($targetProcess.HasExited) {
            throw "Process $TargetPid has exited. Find the new process ID and start again."
        }

        $reading = Invoke-Tool -FilePath $jcmd -Arguments @("$TargetPid", 'GC.heap_info') -TimeoutSeconds 15
        if ($reading.TimedOut) {
            throw 'jcmd stopped responding. The server may be paused. Try again, and tell your support contact if it keeps happening.'
        }
        if ($reading.ExitCode -ne 0) { throw ('jcmd could not read the heap: ' + $reading.Output) }
        $info = $reading.Output
        $heapKB = 0
        $zgc = [regex]::Match($info, '(?m)ZHeap\s+used\s+(\d+)M')
        $g1  = [regex]::Match($info, '(?m)garbage-first heap.*?used\s+(\d+)K')
        if ($zgc.Success) {
            $heapKB = [double]$zgc.Groups[1].Value * 1024
        } elseif ($g1.Success) {
            $heapKB = [double]$g1.Groups[1].Value
        } else {
            foreach ($m in [regex]::Matches($info, '(?m)(?:PSYoungGen|ParOldGen|DefNew|[Tt]enured)\s+total\s+\d+K,\s+used\s+(\d+)K')) {
                $heapKB += [double]$m.Groups[1].Value
            }
        }
        if ($heapKB -le 0) { throw ('Could not read the heap size from jcmd. Send your support contact this output: ' + $info) }
        Write-Host ('{0:yyyy-MM-dd HH:mm:ss zzz} | {1:N3} GiB heap used' -f (Get-Date), ($heapKB / 1MB))

        if ($heapKB -ge $thresholdKB) {
            $targetProcess.Refresh()
            if ($targetProcess.HasExited) {
                throw "Process $TargetPid has exited. Find the new process ID and start again."
            }
            if (Test-Path -LiteralPath $DumpPath) { throw "$DumpPath now exists. Capture cancelled." }
            $result = Invoke-Tool -FilePath $jcmd -Arguments @("$TargetPid", 'GC.heap_dump', '-all=true', $DumpPath) -TimeoutSeconds 900
            $capture = $result.Output
            Write-Host $capture
            if ($result.TimedOut) {
                throw 'The capture stopped responding. Keep this output and any partial file for your support contact.'
            }
            if ($result.ExitCode -ne 0 -or -not (Test-Path -LiteralPath $DumpPath -PathType Leaf)) {
                throw 'The capture failed. Keep this output for your support contact.'
            }
            if ((Get-Item -LiteralPath $DumpPath).Length -eq 0 -or
                $capture -notmatch 'Heap dump file created') {
                throw 'The capture could not be confirmed. Keep this output and any partial file.'
            }
            Write-Host "Heap dump written to $DumpPath. Send it to your support contact."
            break
        }
        Start-Sleep -Seconds 5
    }
} catch {
    Write-Error $_ -ErrorAction Continue
    exit 1
}

Linux and macOS

To run the script on Linux or macOS:

1. Save the script at the end of this section as papercut-heapdump-watch.sh, somewhere the papercut account can read, such as /var/tmp.

2. Fill in the four values at the top. Set DUMP_PATH to a file inside the folder you created earlier, so /var/tmp/papercut-heapdumps on Linux or /Library/Logs/PaperCut/heapdumps on macOS.

3. Run the script as the papercut account:

   sudo -u papercut bash ./papercut-heapdump-watch.sh
   

4. Leave it running until it finishes, or press Ctrl+C to stop it early.

Linux and macOS script:

#!/bin/bash
# Fill in the four values below. Agree the process ID and the threshold with your
# PaperCut support contact before you run this.
TARGET_PID=0     # The Application Server's process ID
THRESHOLD_GB=0   # Heap used, in GiB, that you want the dump taken at
JAVA_BIN=""      # PaperCut's own Java bin folder, the one holding jcmd
                 # e.g. /Applications/PaperCut MF/runtime/mac/jre/Contents/Home/jre/bin
DUMP_PATH="/var/tmp/papercut-heapdumps/MemoryDump.hprof"   # A file in the folder you created

fail() { echo "ERROR: $*" >&2; exit 1; }

# Runs a tool and gives up if it stops responding, so the watcher cannot hang forever.
run_tool() (
  limit="$1"; shift
  out=$(mktemp "${TMPDIR:-/tmp}/papercut-watch.XXXXXX") || exit 1
  child=""

  cleanup() {
    if [ -n "$child" ]; then
      kill -KILL "$child" 2>/dev/null
      wait "$child" 2>/dev/null
    fi
    rm -f "$out"
  }

  trap cleanup EXIT
  trap 'exit 130' INT
  trap 'exit 143' TERM
  trap 'exit 129' HUP

  "$@" > "$out" 2>&1 &
  child=$!
  waited=0

  while kill -0 "$child" 2>/dev/null; do
    if [ "$waited" -ge "$limit" ]; then
      kill -KILL "$child" 2>/dev/null
      wait "$child" 2>/dev/null
      child=""
      cat "$out"
      exit 124
    fi
    sleep 1
    waited=$((waited + 1))
  done

  wait "$child"; rc=$?
  child=""
  cat "$out"
  exit "$rc"
)

[[ "$TARGET_PID" =~ ^[1-9][0-9]*$ ]] || fail "Set TARGET_PID to the Application Server's process ID."
awk -v n="$THRESHOLD_GB" 'BEGIN {exit !(n ~ /^[0-9]+([.][0-9]+)?$/ && n > 0)}' ||
  fail "Set THRESHOLD_GB to the value you agreed with support."
[ -x "$JAVA_BIN/jcmd" ] || fail "jcmd was not found in $JAVA_BIN."
[[ "$DUMP_PATH" = /* ]] || fail "Use a full dump path."
[[ "$DUMP_PATH" != *[[:space:]\"\'=]* ]] || fail "Use a dump path with no spaces, quotation marks or equals signs."
[ -d "$(dirname "$DUMP_PATH")" ] || fail "Create the dump folder first."
[ -w "$(dirname "$DUMP_PATH")" ] || fail "This account cannot write to the dump folder."
[[ ! -e "$DUMP_PATH" && ! -L "$DUMP_PATH" ]] || fail "$DUMP_PATH already exists. Choose a new file name."

run_tool 30 "$JAVA_BIN/jcmd" "$TARGET_PID" VM.version > /dev/null ||
  fail "Cannot attach to PID $TARGET_PID. Check the PID, and that you are running as the account the server runs under."
STARTED=$(LC_ALL=C ps -p "$TARGET_PID" -o lstart=) || fail "Process $TARGET_PID was not found."
[ -n "$STARTED" ] || fail "Process $TARGET_PID was not found."
THRESHOLD_KB=$(awk -v n="$THRESHOLD_GB" 'BEGIN {printf "%.0f", n * 1024 * 1024}') ||
  fail "Could not work out the threshold from THRESHOLD_GB."
[[ "$THRESHOLD_KB" =~ ^[1-9][0-9]*$ ]] || fail "THRESHOLD_GB is too small or too large to use."
echo "Watching PID $TARGET_PID for Java heap usage above $THRESHOLD_GB GiB. Press Ctrl+C to stop."

while true; do
  CURRENT_START=$(LC_ALL=C ps -p "$TARGET_PID" -o lstart=) ||
    fail "Process $TARGET_PID has exited. Find the new process ID and start again."
  [ "$CURRENT_START" = "$STARTED" ] ||
    fail "PID $TARGET_PID belongs to a different process now. Find the new process ID and start again."

  HEAP_OUT=$(run_tool 15 "$JAVA_BIN/jcmd" "$TARGET_PID" GC.heap_info)
  HEAP_EXIT=$?
  [ "$HEAP_EXIT" -ne 124 ] || fail "jcmd stopped responding. The server may be paused. Try again, and tell your support contact if it keeps happening."
  [ "$HEAP_EXIT" -eq 0 ] || fail "jcmd could not read the heap. Keep this output for your support contact: $HEAP_OUT"
  HEAP_KB=$(printf '%s\n' "$HEAP_OUT" | awk '
    /ZHeap[[:space:]]+used[[:space:]]+[0-9]+M/ {
      for (i=1; i<NF; i++) if ($i=="used") { v=$(i+1); sub(/M,?$/,"",v); zgc=v*1024 } }
    /garbage-first heap/ {
      for (i=1; i<NF; i++) if ($i=="used") { v=$(i+1); sub(/K,?$/,"",v); g1=v } }
    /PSYoungGen|ParOldGen|DefNew|[Tt]enured/ {
      for (i=1; i<NF; i++) if ($i=="used") { v=$(i+1); sub(/K,?$/,"",v); if (v ~ /^[0-9]+$/) gens+=v } }
    END {
      if (zgc > 0) printf "%.0f", zgc
      else if (g1 > 0) printf "%.0f", g1
      else if (gens > 0) printf "%.0f", gens
    }')
  [[ "$HEAP_KB" =~ ^[0-9]+$ ]] && [ "$HEAP_KB" -gt 0 ] ||
    fail "Could not read the heap size from jcmd. Send your support contact this output: $HEAP_OUT"
  HEAP_GB=$(awk -v n="$HEAP_KB" 'BEGIN {printf "%.3f", n / 1024 / 1024}')
  echo "$(date '+%Y-%m-%d %H:%M:%S %z') | $HEAP_GB GiB heap used"

  if [ "$HEAP_KB" -ge "$THRESHOLD_KB" ]; then
    RECHECK=$(LC_ALL=C ps -p "$TARGET_PID" -o lstart=) ||
      fail "Process $TARGET_PID has exited. Find the new process ID and start again."
    [ "$RECHECK" = "$STARTED" ] ||
      fail "PID $TARGET_PID belongs to a different process now. Find the new process ID and start again."
    [[ ! -e "$DUMP_PATH" && ! -L "$DUMP_PATH" ]] || fail "$DUMP_PATH now exists. Capture cancelled."
    CAPTURE=$(run_tool 900 "$JAVA_BIN/jcmd" "$TARGET_PID" GC.heap_dump -all=true "$DUMP_PATH" 2>&1)
    CAPTURE_EXIT=$?
    printf '%s\n' "$CAPTURE"
    [ "$CAPTURE_EXIT" -eq 0 ] && [ -f "$DUMP_PATH" ] && [ ! -L "$DUMP_PATH" ] && [ -s "$DUMP_PATH" ] ||
      fail "The capture failed or stopped responding. Keep this output and any partial file for your support contact."
    [[ "$CAPTURE" = *"Heap dump file created"* ]] ||
      fail "The capture could not be confirmed. Keep this output and any partial file."
    echo "Heap dump written to $DUMP_PATH. Send it to your support contact."
    break
  fi
  sleep 5
done

How to send the dump to your support contact

To send the dump:

1. Upload the dump and the logs using the link your Support contact provides. Heap dumps are almost always too large to email.
2. Include in the upload:
   • Application Server logs covering the same period (How to Collect Application Server Debug Logs)
   • the gc.log files from [app-path]/server/logs
   • the output the script printed, if you used Method 3.
3. When you reply to your support ticket, tell your Support contact the:
   • PaperCut NG/MF version and build
   • operating system
   • total RAM on the server
   • maximum heap size
   • date, time, and time zone of the capture
   • process ID and threshold if you used Method 3
   • anything that was running at the time, such as a User/Group Sync or a database backup.
4. After they confirm everything has arrived:
   • Delete your local copy.
   • Remove the settings and the script.
   • Turn debug logging back off.

Still have questions?

Let us know! We love chatting about what’s going on under the hood. Feel free to leave a comment below or visit our Support Portal for further assistance.