I recently helped out one of our biggest corporate customers to resolve issues with their print server. During the the last week of the financial year (when printing load is the highest) their print server became overloaded and stopped working. This sounds bad, but we quickly got things working smoothly again and learned that …

This customer had been running PaperCut for about 6 months without issue. Over this period they were gradually transitioning 100s of print queues from legacy print servers to the server hosting PaperCut. This single print server was hosting all queues for their offices country-wide. The extra load of these additional print queues combined with the end-of-year printing load pushed the server to the limit.

When analyzing the problem I noticed that this server was handling a huge print load. In the 30 day period prior the following printing occurred:

• 477,287 print jobs
• 2,021,454 pages printed
• Between 22,000 to 25,000 print jobs each week day

Wow! That’s a lot of printing!

They were also using hold/release queues and Find-Me printing (aka follow-me printing) to provide secure print release and to reduce paper wastage. The result was an average of around 500-600 print jobs waiting in the queue to be released.

The cause of the problem was under resourcing. Their setup was:

• A single server hosting the both the print queues and the PaperCut application server
• The server was a virtual machine assigned only a single processor
• Allocated 3GB of RAM
• Running on a 32-bit Windows Server operating system

My recommendation was to leave the print queues on the existing server, but move the PaperCut Application Server service to a server with 4GB of RAM, 2 or more processors, and running a 64-bit operating system with the 64-bit add-on pack. This configuration:

• Spreads the load between 2 servers
• Allows the PaperCut Application Server to take advantage of more memory (64-bit)
• More available processors allowed efficient processing of simultaneous print jobs

Since making these changes, their system has been running very smoothly. Their servers are now handling more load than ever, and without overloading the servers.

If you’re managing a large PaperCut installation, and in particular leveraging some of PaperCut’s advanced print management features such as secure print release, then there’s a few lessons to take from this:

• Don’t skimp on RAM or CPU resources
• Monitor your servers. Particularly if you’re adding print queues and increasing print load
• Consider running a 64-bit OS to allow for future expansion (e.g. more memory)
• Run PaperCut on an external database like SQL Server, Oracle, PostgreSQL or MySQL

CC image courtesy of Emilian Robert Vicol on flickr

Last week we released PaperCut 10.0 with the much requested “Printer Groups” feature. But we’re not standing still. All our developers are working hard to implement the new ideas that you’ve been asking for. As you can see from our release history the product continues to improve with each release. But we want to do more … much more … and faster!

And to do that we need another fantastic developer to join the PaperCut team at our head office in Melbourne. If you’re interested, or know any friends who might be … please read on and apply.

The full position description is included below. If you’re applying mention that you read our blog for bonus points!

(more…)

Here at PaperCut we develop software that’s used in every region of the world. Most of our customers are in USA, Canada, UK, Australia, Brazil and China. However there are thousands of customers in more exotic locations. We have customers located throughout the depths of Africa, the middle east, the channel islands, and even a small tropical island.

Developing software for the world represents unique challenges. The most obvious challenge is ensuring that every facet of the application is translatable. PaperCut is already translated into over 15 languages which shows that we’ve done this bit right. You also need to make sure that you are 100% Unicode aware so that you can handle all characters, like Chinese, Japanese, Korean, Russian, etc. Again, PaperCut handles all this with ease.

But occasionally we encounter a problem or unexpected behaviour that only affects users in a particular region in the world. A few weeks ago we had one of these problems. In this case the user was reporting a problem where they were unable to run any PaperCut reports. We obtained debug logs that showed the error details, but even with this information we could not explain the problem. In the end it was the customer that helped find the problem. They reported that if they changed their “language” to “English” the reports worked, and if they changed the locale to “Turkish” the reports failed. Aha!! With that information we reproduced the problem.

I won’t go into all the gory technical details, but it turns out that the problem was that in Turkish the letter “i” when converted to uppercase becomes “İ” (unicode character 0×0130) which is an “I” with a dot above it. Some of our code has assumed that if you uppercased “i” it would become “I”, and when this didn’t occur the report failed to run.

As they say, you learn something every day. And we did! We learned that when dealing with text in different locales around the world … assume nothing.

We’ve now fixed this problem and this fix is now available in our 9.3 release. And with this release PaperCut is available to another large group of users. Prior to this Turkish users needed to run PaperCut under an English/US locale.

Now all we need is a Turkish translation. If you’d like to help translate PaperCut into Turkish (or any other language!!!) then let us know. We’d really appreciate your help.

We’re very proud to announce that we’ve been made state finalists in the Telstra Business Awards 2008.

It’s a great honor be a finalist amongst so many other great companies in the country’s most prestigious business awards. It is recognition of all the hard work the PaperCut team has put into developing a great printer control product and providing great support to our customers world-wide.

Running a business is hard work, and you don’t often get time to reflect on the how far we’ve come and what we’ve achieved. These awards have allowed us to do that, and be proud of these achievements.

The winners will be announced at a formal dinner tonight (19th September 2008). The whole team and partners are coming along. It should be a fun and exciting night.

Wish us luck!!

PS: We’ll post some photos of the night here next week.

When we’re not working on new features, we’re often working on improving PaperCut to ensure it works well in a large variety of situations.  Sometimes customers use PaperCut in ways we never intended, or encounter scenarios which we never envisioned.  We take pride in listening to our customer experiences, and making improvements to ensure we meet their needs.  The result is a better product, and everyone benefits.

Today I’ll explain an issue raised recently by one of our largest PaperCut ChargeBack customers, and then describe how we designed and implemented the solution.

The Problem

This customer was running PaperCut in a challenging environment:

• Around 2,000 users all running the user client to charge printing to shared accounts;
• Users are located at a number of remote sites over slower network connections;
• Printing is charged to over 30,000 shared accounts;
• All users could charge printing to any of the 30,000 accounts;
• Each night an automated process added, removed or modified hundreds of accounts.

They reported the following problems:

• The user clients on remote sites were slow to popup when first printing.
• A large amount of bandwidth was used communicating with the primary PaperCut server.
• The PaperCut server was slow to respond in the mornings.

Analysis

It was clear that the cause of all these symptoms was the large amount of account data downloaded by each of the user clients.   The user client will only download the account list from the server when it detects accounts have changed.  But at this customer the account list was changed nightly.  This caused the client software to re-download the large account list when the user first printed each day.

The client software communicates with the server using XML web services.  After some analysis we found that the the XML representation of the 30,000 accounts was about 7.5 MB.  On a fast local network this size is not a problem, but with 100′s of clients running on remote sites over slower WAN connections this starts to be an issue.  It causes congestion on the network, the popup is slow to appear, and the server becomes overloaded.

After a few brainstorming sessions and consulting with the customer we approached the solution from two angles.

1. Reduce the size of the account download;
2. Allow the account data to be efficiently distributed to remote sites.

Solution Part 1: Reducing the download size

To reduce the size of the data downloaded we changed the account file format in two ways:

1. Change from verbose XML to a much more efficient binary format;
2. Compress the binary data

The net result was a reduction of approximately 25 times!   In the case of this customer the 7.5MB download was reduced to around 250KB. Quite an improvement!

This change alone dramatically reduced the popup speed, and bandwidth used over the WAN connections.  But we could do better!

Solution Part 2: Efficient distribution

We realized that it was very inefficient for multiple clients at the same remote site to download the identical account list.  What if we could only download the list once per site?

We implemented a server-side tool to save the account list to a file (in the identical format described above).  We added a new option to the client software to allow it to load the account list from a file instead of downloading from the server directly.  Then all that this customer need to do was:

1. Schedule a task to create the account file each night (after the accounts were modified), then copy the account file to a file share on each of the remote sites.
2. Configure the clients at each site to load the local account list file.

The result was that only a single 250KB transfer was required to delivery the new account list to all clients at a given remote site.

Conclusion

The result of these changes was dramatic.  Previously, on a site with 20 users the daily account download would consume around 150MB of bandwidth.  After these changes it was reduced to only 250KB.  What an improvement!

Not only was this a great result for this customer, but the product is improved for everyone.

I hope this gave you a little insight into the work we do behind the scenes.  This change wasn’t glamorous or exciting.  It will only be a single bullet point in our version 8.4 release notes.    However it’s often many of these small improvements that make the overall product great.

We often get requests from customers that want to be notified when we release new versions. They want to run the latest version to take advantage of all the new features (without having to check our web site every week).

So today I added news feeds to the website to notify you as soon as a release is available. There are news feeds for both PaperCut NG and PaperCut ChargeBack and are available from the release history and download pages. For the impatient, here are the feed links:

• PaperCut NG Releases
• PaperCut ChargeBack Releases

News feeds are a great way to keep up-to-date with blogs and news. To subscribe to a news feed you need to use a news reader (or aggregator) which is a piece of software (or web site) that retrieves news items from all sites your are subscribed to and notifies you when you have unread items.

Both Firefox 2 and Internet Explorer 7 have built-in feed readers, but they are pretty basic. If you’re interested in something more powerful we highly recommend Google Reader. All the developers at PaperCut use Google Reader daily to keep up-to-date with what’s new in the world.

Happy news reading!

I’ve just uploaded another build of the Net Control module. This release includes a number of changes:

• Add “enable/disable user net usage” option to the bulk user action page, so this setting can be set on users in bulk.
• Add support for the MS ISA Server log format, in addition to the preferred W3C log format.
• Improved detection of the latest log file based on the specified log file mask.
• Improved behaviour when processing MS ISA log format, when the W3C log format is specified in PaperCut.
• Do not log net usage performed by Squid users that haven’t authenticated correctly.
• Fix to error that occurred when running a transaction log report that contained Internet usage transactions.
• Fix bug in Linux installer that caused the Internet service config file to be overwritten when upgrading from a previous version.
• Improved the setup of the Internet Provider service on linux to warn the user if the specified log file directory is not readable.
• Add a config option to enable LDAP referrals.
• All the many changes included in the most recent public release of PaperCut NG 6.3. For more information see the release history page.
• Other small fixes.

You can download the latest release (Build 4006) from the Net Control module testing page. To upgrade from a previous release you can perform an install over the top procedure.

I’ve just uploaded another build of the Net Control module. This release includes the following changes:

• All the changes included in the most recent public release of PaperCut NG 6.2. For more information see the release history page.
• Other small fixes.

You can download the latest release (Build 3909) from the Net Control module testing page. To upgrade from a previous release you can perform an install over the top procedure.