Troubleshooting Server Performance Issues
““Help! Our PaperCut Application server is acting unpredictably, and we see lots of intermittent issues especially during peak usage times; how should we go about troubleshooting this problem?”
If you’ve already tried “turning it off and on again” and you still see some of the symptoms in the list below, then this might be the right troubleshooting article for you.
- The PaperCut server’s web interface is slow to load.
- Logging into MFDs may be slow or time out. These devices may display an error similar to “unable to connect to server” or “connection timed out” when users try to log in, especially in the middle of the day.
- The server might show consistently high CPU or Memory usage, plateauing at 100%.
- The common thread is that the problem often cannot be reproduced consistently, and is usually reported during the busiest times of the day for printing.
- You may see the error
Warning: Print Provider “printsrv01|126.96.36.199”: Print job “Document-1.pdf” printed by “user1” is offline and has been temporarily hidden from the Print Release screen.as described in the article Why was this job ‘Temporarily Hidden’? but you have ruled out the other causes.
You may have already gone through the suggestions in our article on Keeping a healthy system, and you’re still having trouble. So what’s the next thing to check?
Every server OS has a built-in utility for checking which apps are using the greatest amount of CPU or Memory. Here’s how to find how much CPU and memory is getting used by PaperCut on each type of server:
- Windows - in the native OS
Task Managerapplications, look for
- macOS - in the native OS’
Activity Monitorapplication, look for the process named
Javarunning as the user
- Linux - open the
Terminalcommand window and run the
topcommand, look for the process named
Is there anything else running on the PaperCut NG/MF server that is consuming excessive resources? As an IT best practice, we strongly recommend having a dedicated server in your environment for PaperCut NG/MF.
However, if you see that one of those PaperCut processes is the sole culprit then read on…
Check our Server Sizing Guide to make sure that the server is scaled with the appropriate resources. Remember that Site Servers and Secondary PaperCut servers need adequate resources as well to function as part of the printing ecosystem.
One extreme sign of a performance issue is to look in the
service.log file found on any PaperCut Application server for messages like
JVM Process has not received any CPU time for X seconds. This indicates that the PaperCut application, which runs as a Java Virtual Machine (JVM), is not getting enough resources from the operating system or virtual host. The
service.log file can be found in the
INFO | jvm 1 | 2020/04/14 13:11:31 | JVM Process has not received any CPU time for 23 seconds. Extending timeouts. INFO | jvm 1 | 2020/04/15 05:08:17 | JVM Process has not received any CPU time for 21 seconds. Extending timeouts. INFO | jvm 1 | 2020/04/15 20:56:36 | JVM Process has not received any CPU time for 19 seconds. Extending timeouts.
It is possible to have too much of a good thing. Follow our Server Sizing Guide to give your PaperCut server the right amount of memory, and don’t go overboard.
Why? PaperCut NG and PaperCut MF are Java applications, and hence subject to Java “garbage collection” activities, which maintain proper memory allocation. When too much memory is available to the Application Server, full garbage collections will happen less frequently but take longer to complete, creating a drain on resources.
By default the PaperCut server will use one quarter (¼) of system memory. If you want to fine-tune this value, but retain the same amount of physical memory allocated to this server, then follow our guide on how to adjust the amount of memory available to PaperCut NG/MF.
This point is extremely important and often overlooked. If the PaperCut server is running on a virtual machine, make sure the memory is reserved and not dynamically allocated. We have seen that the Java Virtual Machine that PaperCut runs on top of does not like it when memory is part of a pool that can be dynamically allocated to other servers.
For a VM running in a VMware ESXi host, this setting is called “Reserve all guest memory” and you want it enabled. For a VM running in a Microsoft Hyper-V host, this setting is called “Dynamic Memory” and you want it disabled.
We describe this problem in more detail in this article which answers the question, Does PaperCut work on a Virtual Machine (VM)?.
When troubleshooting a performance issue for Virtual Machines, don’t forget to make sure that your virtual host is happy and healthy as well.
PaperCut NG/MF has to track each print job held on the server, and more print jobs held means more server resources used. You might start to run into a problem after configuring PaperCut NG/MF to hold print jobs for longer than the default of 2–4 hours as this can cause too many print jobs to begin to accumulate that PaperCut NG/MF must continually account for.
To see how many print jobs are being held, log into the PaperCut server as administrator and look for the number of “Hold/release jobs” on the dashboard. More than 1500 held jobs at any moment should be considered a red flag and you should strongly consider changing the Job Timeout period to a lower value to reduce this number. This change will not take effect immediately, because it will only apply to any new print jobs that reach the server so it may be a few hours before you start to see the benefits.
The PaperCut Direct Print Monitor may be installed on user’s workstations to track printing to local or USB printers. However when there are several hundred or thousands of clients running the Direct Print Monitor this may put a strain on the PaperCut server.
This scenario is discussed in much greater detail in our article Environments with large numbers of Direct Print Monitors.
Check in the logs for the term
DatabaseHealthChecker. In the example below the database connection takes 6000 ms (6 whole seconds!) which is not healthy.
DEBUG DatabaseHealthChecker:69 - Database connection test - success. Connect duration: 6017 ms. Query duration: 0 ms. [db-connection-test]
This particular log snippet was due to an issue we discovered with the default driver for SQL databases that comes bundled with Java, since resolved, that we have documented here.
If NG/MF is configured to use the Internal Derby Database (default configuration)…
- If this server has been around for many years, consider Optimizing your database. This operation is a bit like defragmenting a hard drive in that it might provide a slight performance boost, but is not going to help much if there is a larger underlying issue.
- It may be time to upgrade to an external database hosted on another server. When implemented properly this will take a load off the server running PaperCut.
If NG/MF is already configured to use an external database…
- Is the database server properly resourced and is it near enough to the PaperCut Application server to ensure a good connection?
- Run a 3rd party I/O test on your database server to help identify if there is a latency issue when writing to the disk array.
- To narrow down whether this is a problem with the specific database you are using, consider switching database types to see if the problem goes away.
We sometimes see that Debug Mode is left enabled on busy servers. While debug mode is important to catch and diagnose an issue, it shouldn’t be left on indefinitely because the additional read/write operations can cause a measurable increase in load at large sites (on the scale of tens of thousands of clients). If you are troubleshooting a performance issue and leave debug mode enabled, this can compound the problem while clients have to wait for the server to update the log file. Heed this advice:
- Standard logging in most cases will include the information we need to search for key messages.
- Leave debug logging disabled on the server, unless it’s required to diagnose an active problem.
- If planning a restart during high daytime load, disable debug logging before the restart.
- When troubleshooting a performance issue, check to make sure that the different types of debug modes have not been left on by accident, including Application Server Debug Mode, SSL Debug, and SMTP Debug.
Start a support ticket
Are things still not working great after following the above steps? Enable Debug mode temporarily to download the Diagnostic Bundle from your PaperCut server and head over to our Support Portal for further assistance. We can analyze these logs to see if anything stands out to us.