Troubleshooting and optimizing Server Performance
“Help! Our PaperCut Application server is acting unpredictably, and we see lots of intermittent issues especially during peak usage times; how should we go about troubleshooting this problem?”
If you’ve already tried “turning it off and on again” and you still see some of the symptoms in the list below, then this might be the right troubleshooting article for you.
- The PaperCut server’s web interface is slow to load.
- Logging into MFDs may be slow or time out. These devices may display an error similar to “unable to connect to server” or “connection timed out” when users try to log in, especially in the middle of the day.
- The server might show consistently high CPU or Memory usage, plateauing at 100%.
- The common thread is that the problem often cannot be reproduced consistently, and is usually reported during the busiest times of the day for printing.
- You may see the error
Warning: Print Provider “printsrv01|18.104.22.168”: Print job “Document-1.pdf” printed by “user1” is offline and has been temporarily hidden from the Print Release screen.as described in the article “Why was this job ‘temporarily hidden’?” after you have ruled out the other causes.
You may have already gone through the suggestions in our article on Keeping a healthy system, and you’re still having trouble. So what’s the next thing to check?
Every server OS has a built-in utility for checking which apps are using the greatest amount of CPU or Memory. Here’s how to find how much CPU and memory is getting used by PaperCut on each type of server:
- Windows - in the native OS
Task Managerapplications, look for
- macOS - in the native OS’
Activity Monitorapplication, look for the process named
Javarunning as the user
- Linux - open the
Terminalcommand window and run the
topcommand, look for the process named
Make sure the PaperCut NG/MF server has adequate CPU/Memory allocation
Check our Server Sizing Guide to make sure that the server is scaled with the appropriate resources. Remember that Site Servers and Secondary PaperCut servers need adequate resources as well to function as part of the printing ecosystem.
One extreme sign of a performance issue is to look in the
service.log file found on any PaperCut Application server for messages like
JVM Process has not received any CPU time for X seconds. This indicates that the PaperCut application, which runs as a Java Virtual Machine (JVM), is not getting enough resources from the operating system or virtual host. The
service.log file can be found in the
INFO | jvm 1 | 2020/04/14 13:11:31 | JVM Process has not received any CPU time for 23 seconds. Extending timeouts. INFO | jvm 1 | 2020/04/15 05:08:17 | JVM Process has not received any CPU time for 21 seconds. Extending timeouts. INFO | jvm 1 | 2020/04/15 20:56:36 | JVM Process has not received any CPU time for 19 seconds. Extending timeouts.
Make sure the server PaperCut NG/MF server doesn’t have too much memory allocation
It is possible to have too much of a good thing. And, surprisingly, RAM is definitely one of those things for PaperCut. PaperCut NG and PaperCut MF are Java applications. Java has a garbage collection routine that will use a percentage of the memory allocated to the application as a marker for when to trigger cleanup. When too much memory is available to the Application Server, full garbage collections will happen less frequently but take longer to complete, creating a drain on resources. Follow our Server Sizing Guide to give your PaperCut server the right amount of memory, and don’t go overboard. By default the PaperCut server will use one quarter (¼) of system memory. If you want to fine-tune this value, follow our guide on how to adjust the amount of memory available to PaperCut NG/MF.
Reserve the resource allocation for VMs
This point is extremely important and often overlooked. If the PaperCut server is running on a virtual machine, make sure the memory is reserved and not dynamically allocated. We have seen that the Java Virtual Machine that PaperCut runs on top of does not like it when memory is part of a pool that can be dynamically allocated to other servers.
For a VM running in a VMware ESXi host, this setting is called “Reserve all guest memory” and you want it enabled. For a VM running in a Microsoft Hyper-V host, this setting is called “Dynamic Memory” and you want it disabled.
See ”Does PaperCut work on a Virtual Machine (VM)?” for more information.
Check how many print jobs are being held
PaperCut NG/MF has to track each print job held on the server, and more print jobs held means more server resources used. You might start to run into a problem if you have configured PaperCut NG/MF to hold print jobs for longer than the default of 2–4 hours.
To see how many print jobs are being held, log into the PaperCut server as administrator and look for the number of “Hold/release jobs” on the dashboard. It will vary depending on how well-provisioned your server is, but generally, more than 1500 held jobs is cause for concern. Consider changing the Job Timeout period to a lower value to reduce this number. This change will not take effect immediately, because it will only apply to any new print jobs that reach the server so it may be a few hours before you start to see the benefits.
Too many clients running the Direct Print Monitor
The PaperCut Direct Print Monitor may be installed on user’s workstations to track printing to local or USB printers. However when there are several hundred or thousands of clients running the Direct Print Monitor this may put a strain on the PaperCut server.
This scenario is discussed in much greater detail in our article Environments with large numbers of Direct Print Monitors.
Check on the database, offload to another machine
For a small boost, you can look into optimizing your database
Optimization may help a bit, however, internal databases produce a fair amount of load on the application server regardless. Even if you are not experiencing issues with the performance of the database itself, it may be worth moving the database off to another machine to distribute the load a bit more - particularly if you already have a database server in your environment.
See our article on upgrading to an external database
If NG/MF is already configured to use an external database…
- Is the database server properly resourced and is it near enough to the PaperCut Application server to ensure a good connection?
- Run a 3rd party I/O test on your database server to help identify if there is a latency issue when writing to the disk array.
- To narrow down whether this is a problem with the specific database you are using, consider switching database types to see if the problem goes away.
Turn off debug when not needed
We sometimes see that Debug Mode is left enabled on busy servers. While debug mode is important to catch and diagnose an issue, it shouldn’t be left on indefinitely because the additional read/write operations can cause a measurable increase in load at large sites (on the scale of tens of thousands of clients). If you are troubleshooting a performance issue and leave debug mode enabled, this can compound the problem while clients have to wait for the server to update the log file. Heed this advice:
- Standard logging in most cases will include the information we need to search for key messages.
- Leave debug logging disabled on the server, unless it’s required to diagnose an active problem.
- If planning a restart during high daytime load, disable debug logging before the restart.
- When troubleshooting a performance issue, check to make sure that the different types of debug modes have not been left on by accident, including Application Server Debug Mode, SSL Debug, and SMTP Debug.
Check on your health API calls
Health API calls (such as the requests made by PRTG or Zabbix) can put a large amount of load on the server. Consider reducing the frequency of check-ins coming from your monitoring system to free up resource usage a bit.
Move auxiliary services to other machines
There are many optional features that can be run on the PaperCut server. The good news here is that you move many of these services to other machines. These reference other PaperCut components - however the same philosophy can be applied to apps from other vendors on the same hardware as PaperCut. As an IT best practice, we strongly recommend having a dedicated server in your environment for PaperCut NG/MF.
The following are simple and easy to move off of the application server to free up a bit of capacity:
- User Printing: Move the sharing, authentication, and actual print rendering off your Primary Application server with a PaperCut Secondary server
- Mobility Print: You can turn any Secondary Server into a PaperCut Mobility Print server by installing both services.
- Web Print: See our documentation on Setting up a web print sandbox!
We have a separate article on how to reduce the impact of Print Deploy on your application server. Take a look through it to further optimize your resource usage!
Start a support ticket
Are things still not working great after following the above steps? Enable Debug mode temporarily to download the Diagnostic Bundle from your PaperCut server and head over to our Support Portal for further assistance. We can analyze these logs to see if anything stands out to us.