Choose your language

Choose your login

Contact us

Application Server Failover

This page applies to:

This article covers how to configure PaperCut’s natively supported High Availability capability that we refer to as “Application Server Failover”.

Understanding PaperCut Application Server Failover implementation

If you can, we recommend that you leverage a solution that your organization already uses. This could be a hypervisor-level configuration such as those within Citrix or VMWare , or it could be a designated backup and recovery solution like Datto or Veeam .

PaperCut NG/MF supports an active/passive configuration. One server handles incoming requests, and in the event of that server’s outage, another server in the passive server queue will become active and begin to handle the requests.

Once active, the server will continue to handle requests until the next failover event. Resolving a failure on a server adds it back to the passive server pool. Since the queue of passive servers is set up automatically, you’ll need to confirm that any network a passive server resides on will be ready to take on the full traffic of your PaperCut environment at any time- this will be important when configuring the shared resources across data centers and networks.

3rd party components for Application Server Failover

PaperCut’s Application Server Failover requires a few 3rd party configurations to get started, such as an external database (RDBMS) , a network load balancer, and a persistent file share.

Configure these third party components with whatever you’re most comfortable and experienced with. Should troubleshooting need to occur, being familiar with each of the third party components is a great help! Additionally, PaperCut doesn’t have vendors that are specifically recommended for these customer-configured components.

Application Server Failover system and environmental requirements

  • PaperCut NG/MF version 20.0 or higher
  • A Network Load Balancer (NLB) - Layer 4 or Layer 7
  • Two or more Application Servers (virtual or physical)
  • An external database (RDBMS)
  • A persistent network drive accessible by all servers
  • An external printing environment (printing can not be hosted on the Primary Application Servers when Application Server Failover is enabled)

There are many different load balancing solutions you can use to enable Application Server Failover. These steps outline the generic process of setting up the load balancer for PaperCut to leverage. For specific insight into your particular load balancer vendor, do a quick search and see if they’ve already written a procedure.

Here are a few of the guides that we’re aware of:

Configure PaperCut Application Server Failover

Configure the first Application Server
  1. Install PaperCut NG/MF , and then stop the Application Server .
  2. Upsize to an external database . Each Application Server must use the same database instance.
  3. Start the Application Server and confirm that PaperCut NG/MF is fully functioning.
  4. Stop all of the PaperCut services (PaperCut Application Server service, PaperCut Print Provider service, etc.).
  5. On the Application Server, disable the printing and tracking services. (Stop CUPS or disable the Spooler, and disable the PaperCut Print Provider service )
  6. Go to [app-path]/server.
  7. Copy the following files/folders to the configured network storage:
    • server.properties file
    • security.properties file
    • the application.license file
    • the /data folder copying files between the PaperCut /server directory and the shared network storage.
  1. Open the server.properties file in a text editor.
  2. Make the following changes to the file:
    • Remove the “#” symbol to uncomment the following lines:
      • high-availability-guard.enabled = Y
      • high-availability-guard.fail-over-internal-secs = 30
      • high-availability-guard.startup-timeout-secs = 300
      • See “Configuration keys for App Server Failover” if you need more details.
    • Change the high-availability-guard.enabled config option to “Y” if it’s not already.
  3. Save the file.
  4. Rename the existing local /data folder in the \[app-path\]/server/ directory to data.bak.
  5. Make a folder-level symlink for the /data directory.
    • On Windows: mklink /D "C:\Program Files\PaperCut MF\server\data" //network-storage/papercut/data
    • On Linux: Edit the /etc/fstab to mount the /data folder on network drive. Remember to include a system.automount parameter in the options. Then, run ln -s /usr/local/papercut/server/data /mnt/network-share/data
  6. Make a file-level symlink for each of the remaining copied files:
    • application.license
    • security.properties
    • server.properties
  1. Start all of the PaperCut services (PaperCut Application server service, PaperCut Print Deploy Server service). Confirm that the now-active Application Server is still functioning as expected.

  2. Open the Admin web interface and navigate to Options > Advanced > Server Address.

  3. Set the Server Address to the DNS name or VIP of the NLB.

  4. Open the advanced config editor and find the key system.network-address. Configure this key to the IP or the hostname of the load balancer

  5. To make full use of the NLB and ensure that all PaperCut components work during a failover event, review the following articles:

  6. Set up the SSL certificate for your servers. (See “Additional details on load balancer configuration” below.)

Configure additional Application Servers

If using a Virtual Machine

  1. Stop the PaperCut Services on your pre-configured Application Server.
  2. Clone the Application Server host to make a second, identical app server. Your hypervisor may require you to shut down the virtual machine to clone.
  3. Ensure that the PaperCut services are still stopped on the new clone of the Application Server.
  4. In the newly created Application Server clone, navigate to the [app-path]/server folder.
  5. Delete the file called ‘server.uuid’.
  6. Start the PaperCut services.
  7. Repeat for each server behind the Network Load Balancer

If using physical hardware, or if cloning is not possible

  1. Install PaperCut and then stop all of the PaperCut services. (PaperCut Application Server service, PaperCut Print Provider service, etc.).
  2. In the [app-path]/server/ directory, rename the existing local /data folder to data.bak.
  3. Make a folder-level symlink for the data directory detailed within steps 12-13 in the Configure the Active App Server section.
  4. Make a file-level symlink for each of the remaining copied files:
    • application.license
    • security.properties
    • server.properties
  5. Make sure the PaperCut Application Server service is running under a user account that has Read/Write access to the shared network folder you set up earlier.
  6. Start all of the PaperCut services (PaperCut Application Server service, PaperCut Print Provider service, etc.).
  7. Repeat for each server behind the Network Load Balancer.

When finished, confirm when attempting to access the Admin web interface on your new server that the below message is displayed:

Message displayed on passive server - “High availability activated, server in passive monitoring mode”

Configure NLB Health Checks
  1. Log in to PaperCut NG/MF as an admin and go to Options > Advanced.
  2. In the System Health Monitoring section, copy the URL path (the info after :9191) in the GET query parameter field:
  3. In the load balancer settings, paste the above endpoint as the monitoring URL.
  4. If the load balancer allows a monitoring interval, we recommend setting it to 30 seconds. A lower number will unnecessarily increase the load on the Application Server.
Expected Responses
HTTP 200 OK The server is active and ready to handle PaperCut traffic
HTTP 503 The server is in passive mode and is not handling PaperCut traffic
  1. Configure all external components (secondary servers, user clients, MFDs, etc.) to point to the virtual service IP address or hostname that has been set up on the Network Load Balancer.
Additional details on load balancer configurations

SSL Configurations

If you want to configure SSL for this server, it’s best practice to use a common SSL certificate for the Applicaiton Server and the Network Load Balancer’s VIP. A single SSL certificate will remove the chance for SSL certificate date mismatching and should simplify the maintenance for you in the future.

If you choose to enforce SSL traffic on your load balancer, you’ll need to configure the PaperCut Print Provider to support HTTPS. This feature is available in version 23 of PaperCut NG/MF and above. HTTPS traffic will not be configured in the print provider by default.

Using a Layer 4 (Transport Layer) Load Balancer

Layer 4 / L4 / Transport layer NLBs will direct traffic at a very low-level on the network. They involve less compute power to work, but are a bit old-fashioned and can involve a fair amount of manual configuration. Layer 4 load balancing was what was initially used for the PaperCut App Server Failover. As such, this configuration is the most tested and widely documented.

Network adapters and loopbacks

Layer 4 will require that a network adapter is configured for the Virtual IP Address (VIP) of the NLB. This network adapter should be set up as a loopback so that the address can be held by the server without being used on the accessible network. Additionally, this setup will often require that you do not allow for ARP responses on the configured loopback adapters.

Example steps for Windows Device Manager
  1. In a Run prompt, type ‘hdwwiz’ to open the “Add hardware” wizard.
  2. Select “Install the hardware that I manually select from a list (Advanced) and click next.
  3. Find the “network adapters” section of the list.
  4. In the manufacturer list on the left side, find the Microsoft entry.
  5. Select Microsoft KM-TEST Loopback Adapter and complete the install.
  6. In the Network Connections manager, find the newly added network adapter.
  7. Select Properties and navigate to the TCP/IPv4 menu.
  8. Populate the IPv4 address settings with the VIP address of the NLB. Optionally, you can name the adapter something more intuitive than the default name that is assigned Disable auto metric on loopback interface - picture of the “advanced” properties menu in Windows. “The ‘automatic metric’ box is now unchecked, and the metric has beeen manually set to 254”
  9. In the TCP/IPv4 properties menu, click the Advanced button.
  10. Uncheck Automatic metric. Set the Interface Metric field to 254. A view of the TPC/IPv4 settings for the loopback adapter showing the loopback has been configured to use the VIP of the NLB endpoint
  11. Configure “Weak Host” parameters for each of the Network adapters using the following command format: netsh interface ipv4 set interface "[interface name]" weakhostreceive=enabled
Example steps for Linux
  1. Open terminal.
  2. Run the command ip addr add [virtual ip address] dev lo

Preserve client IP address

To make sure that the server is capable of full functionality, configure the network load balancer to preserve the client IP address. This allows the logs and access policies to work correctly for the IP of the client, instead of filtering against the IP of the NLB. Some vendors will use the terminology “Preserve Client IP”, “Use Source IP”, or it can be written in the inverse like “Disable SNAT”.

Direct server return

We recommend a direct-server-return configuration to lower the potential bottlenecks associated with handling all ingress and egress traffic with the load balancer. Depending on the load balancer, this could be known as Direct Server Return (DSR), n-path routing, Direct Return (DR), Asymmetric Routing, or SwitchBack among others.

This allows for the PaperCut Application Server to reply directly to the client that is sending the requests without the involvement of the NLB. In addition to the overhead savings with DSR, you can also avoid some configurations like static routes or default gateways on the application servers.

If your environment does not support direct return, or you don’t want to configure direct server return, you can configure the load balancer as a static route destination for any of the clients IP addresses that you want to interact with the PaperCut app server.

Using a Layer 7 (Application Layer) load balancer

Layer 7 / L7 / Application Layer NLBs will handle traffic at a much higher level than Layer 4. Because of the higher-level operation, L7 NLBs are relatively intensive on computing resources. However, many modern load balancing solutions will have no problem handling a substantial amount of PaperCut traffic before the L4 / L7 distinction will come into play.Working with HTTP instead of TCP allows PaperCut to leverage X-Forwarded-For headers for logging and responding to requests.

SSL Traffic and termination

Most Layer 7 solutions will not require an encrypted connection between the NLB and the backend server that’s hosting PaperCut, so it’s fine to continue using the PaperCut self-signed certificate, and allowing HTTP unencrypted traffic to the server. Otherwise, if you do want to encrypt traffic between the load balancer and the PaperCut Application Server, you should assign a common SSL certificate for both the NLB endpoint and the Application Server.

####App Server configuration for L7 Support Application load balancers will need to be configured to trust and reference the X-Forwarded-For header that is added by the NLB. This configuration will be set with the “trusted proxy servers” configuration in the advanced section of the options menu.

If you do not set the above configuration, you might start to see Cross Site Request Forgery errors. If you see these errors, you might need to follow our steps for CSRF Error resolution here.

Additional details on shared storage

PaperCut will need a shared network file system to store common configuration files that need to be referenced by the Application Servers. This will need to be mounted using the OS-native command line. The PaperCut Application Server will continue to reference its default file paths (for example, C:\Program Files\PaperCut MF\server\server.properties) - however, the actual location of these files should be on the network storage.

It’s best to use a protocol that you are familiar with and that fits with your Operating System’s supported standards. For example, Windows servers could likely leverage an SMB share for the data and configuration files, while Linux servers might benefit from an NFS share.

Additional details on external databases

The database is used as a means of communicating which node is the active server in the PaperCut Application Server pool. As such, you should ensure that the database is operating quick enough so to not accidentally trigger the Application Server’s failover. The default configuration is generally fine, but keep this consideration in mind when setting up bandwidth and allocations for the database.

If you need to configure a highly available database to meet your organization’s resiliency needs, the Application Server will support this. You don’t need to take additional configurations into account for the PaperCut server - you can implement whatever high availability configuration that is available for your RDBMS.

Configuration keys for App Server Failover

These configurations are stored within the server.properties file, which is stored on the network storage you configure.

Configuration key Configuration effect
high-availability-guard.enabled Enable or disabled the high availability feature.
The default is disabled.
high-availability-guard.fail-over-interval-secs The passive server communicates with the active server using the database.
If the passive server does not receive any communication from the active server for a certain period of time, the active server is deemed inactive.
This config value is the period of time in seconds that the active server can be inactive before the passive server takes over.
The default value is 30 seconds.
high-availability-guard.startup-timeout-secs A server will assume the role of an active server on startup if it determines that there are no other servers active. If this active server fails to start correctly, a timeout is in place to trigger a failover if the server isn’t running before the timeout completes.
This config value is the period of time in seconds that the active server has to reach the fully running state before the failover is triggered.
The default value is 300 seconds.

Additional details on PaperCut features Application Server Failover

Secondary Servers and App Server Failover
Secondary servers will function the same as they normally would, but will need to point to the VIP or Hostname of the Network Load Balancer. Additional information about load balancing printing environments can be found here in our knowledge base .
Site Servers and App Server Failover

Where do site servers fit into my failover environment

Site servers are useful for tiding over multifunction devices temporarily. Things like hold/release, find-me, filters and restrictions (conversions and policy), account selection, basic accounting and authentication will continue to work in the event of an outage. If you need, site servers should be configured locally and in-office within the network so that the location could feasibly go offline and continue to function. If there is not a place for hardware on the network, it is generally best to leverage Papercut Application Server Failover without the use of site servers to satisfy your high availability needs.

Extra considerations with site servers

Site servers will apply a load on both the database and the application server. Make sure that the database server and application servers have ample bandwidth to sustain both the syncing tasks of the Site server, and the heartbeat and data transactions of the application server.

Print Deploy and App Server Failover

Configuring Print Deploy to work with App Server Failover

As of the November 2022 Print Deploy server version (1.7.2214), the Print Deploy server natively supports automatic cutover for App Server Failover environments. If you plan on leveraging Print Deploy with your App Server Failover configuration, you should perform the following steps on each Application Server:

  1. Stop the PaperCut Print Deploy Server service.
  2. Open the [app-path]/providers/print-deploy/[os]/data/config/client.conf.toml file in a text editor with elevated permissions.
  3. Set the following config key to true - if the key does not exist, you can add it as it is displayed below to the file:
    • HighAvailabilityCheckEnabled=true
  4. Open the [app-path]/providers/print-deploy/[os]/default-pc-print-deploy.conf file in a text editor with elevated permissions.
  5. Set the StartupRandomDelaySecs value to 0 in the ”ScheduledTasks": section of the config file.
  6. Set the StartupRandomDelaySecs value to 0 in the ”StartupTasks": section of the config file.
  7. Save the file.
  8. Navigate to [app-path]/providers/print-deploy/[os]/
  9. Copy the /data folder to the network share in a manner similar to the application server data folder above.
  10. Change the “accessible host” configuration to the VIP or hostname for the network load balancer.
  11. Configure the Network load balancer to use the same health check as the application server to route to the Print Deploy ports. To do this, you can either:
    • Expand the used ports for your existing application
    • Create a separate VIP/endpoint for print deploy, and re-create the health check for the application server.

Optional configurations

  • If you have enabled auto-update already (this setting is off by default), open pc-print-deploy.conf in the /[app-path]/providers/print-deploy/[os]/ directory
    • Set the StartupRandomDelaySecs value to 0 in the "ScheduledTasks": section of the config file.
    • Set the StartupRandomDelaySecs value to 0 in the "StartupTasks": section of the config file<
    • Save the file.
  • In the /[app-path]/providers/print-deploy/[os]/data/config/ directory, open the client.conf.toml file and edit the HighAvailabilityAppServerProbeIntervalInSecs
    • Use this value to specify how often the PaperCut Print Deploy server should query the PaperCut NG/MF Application Server health API URL. The PaperCut Print Deploy server determines the status of the PaperCut NG/MF Application server using the HTTP code of the response.The default value is every 15 seconds.

Automatic updates

Automatic updates can work with Application Server Failover enabled, however, you need to confirm that the above optional steps are applied to each of the Print Deploy servers. These configurations will prevent the Print Deploy server version from being different for a notable duration of time enabling failover to occur without issue.

While the Application Server health check should be used for the routing direction in the NLB, you can run health checks on the Print Deploy endpoint to confirm that the server service is running correctly. If you do an http check to https://[application-server]:9192/deploy/ping, it should return an HTTP 200 if the Print Deploy server is responsive. You could use this check in a health monitoring application to track the uptime of PaperCut Print Deploy.

Web Print and App Server Failover

All potential web print servers must use the same web-print-hot-folder. The web-print-hot-folder is already included within the /data folder of the PaperCut NG/MF Application Server [app-path]\server\data\web-print-hot-folder and should be copied already from the steps in the Configure the Application Server(s) section, step 8 above.

If you are using sandbox mode , make sure all your web print servers have access to the shared directory in the data folder.

Custom branding and App Server Failover
For custom branding to persist across all potential servers, copy the [app-path]/PaperCut [NG or MF]/server/custom folder to the shared storage and create a folder-level symlink like you have in previous steps.
Custom reporting and App Server Failover

For custom reports to work across all potential servers, the [app-path]/PaperCut [NG or MF]/server/reports on each Application Server needs to be moved to the shared storage.

On each applicaiton server perform the following:

  1. Stop the PaperCut Application Server service.
  2. Navigate to the [app-path]/PaperCut [NG or MF]/server/reports.
  3. Rename the folder reports.bak.
  4. Copy the folder to the network drive.
  5. Create a folder-level symlink for the default reports directory to the network storage like you have in previous steps.
  6. Restart the PaperCut Application Server service.

Once complete, custom reports can be created and maintained in the same way, but with the network storage as the target path.

The user popup client and App Server Failover

If there’s a failover incident, the user clients need to seamlessly switch over to the new active server. For that to happen, user clients must point to the network load balancer’s IP address. The user client’s configuration is propagated when it’s installed from the server.
In both Application Servers, update the User Client config to point to the network load balancer’s IP address.

  1. Open [app-path]/PaperCut [NG or MF]/client/[os]/config.properties.
  2. Update the server-name and server-address and server-port to reflect the endpoints of the NLB.
  3. Save the file.

Other useful information

Set up system notifications for application failures

You can subscribe to application log updates so that if there’s a failover event, you’re automatically notified.

  1. In the Application Server Options > Notifications tab, under System Notifications, select Error level events and enter the email address of each recipient.

  2. Check the SMTP Server Options are set up so that the server can send out email notifications. If an Application Server fails, you’ll see this message in the Application Log:

  3. We also recommend setting up monitoring and notifications on the Network Load Balancer, so you are aware of any of the servers in the pool become unavailable.

Performing Application Server upgrades with App Server Failover configured

Follow the normal upgrade path with some small adjustments:

  1. Stop the PaperCut Application Server service on all servers.

  2. Upgrade all servers.

  3. Start the PaperCut Application Server services one at a time.

Comments