I recently helped out one of our biggest corporate customers to resolve issues with their print server. During the the last week of the financial year (when printing load is the highest) their print server became overloaded and stopped working. This sounds bad, but we quickly got things working smoothly again and learned that …

This customer had been running PaperCut for about 6 months without issue. Over this period they were gradually transitioning 100s of print queues from legacy print servers to the server hosting PaperCut. This single print server was hosting all queues for their offices country-wide. The extra load of these additional print queues combined with the end-of-year printing load pushed the server to the limit.

When analyzing the problem I noticed that this server was handling a huge print load. In the 30 day period prior the following printing occurred:

• 477,287 print jobs
• 2,021,454 pages printed
• Between 22,000 to 25,000 print jobs each week day

Wow! That’s a lot of printing!

They were also using hold/release queues and Find-Me printing (aka follow-me printing) to provide secure print release and to reduce paper wastage. The result was an average of around 500-600 print jobs waiting in the queue to be released.

The cause of the problem was under resourcing. Their setup was:

• A single server hosting the both the print queues and the PaperCut application server
• The server was a virtual machine assigned only a single processor
• Allocated 3GB of RAM
• Running on a 32-bit Windows Server operating system

My recommendation was to leave the print queues on the existing server, but move the PaperCut Application Server service to a server with 4GB of RAM, 2 or more processors, and running a 64-bit operating system with the 64-bit add-on pack. This configuration:

• Spreads the load between 2 servers
• Allows the PaperCut Application Server to take advantage of more memory (64-bit)
• More available processors allowed efficient processing of simultaneous print jobs

Since making these changes, their system has been running very smoothly. Their servers are now handling more load than ever, and without overloading the servers.

If you’re managing a large PaperCut installation, and in particular leveraging some of PaperCut’s advanced print management features such as secure print release, then there’s a few lessons to take from this:

• Don’t skimp on RAM or CPU resources
• Monitor your servers. Particularly if you’re adding print queues and increasing print load
• Consider running a 64-bit OS to allow for future expansion (e.g. more memory)
• Run PaperCut on an external database like SQL Server, Oracle, PostgreSQL or MySQL

CC image courtesy of Emilian Robert Vicol on flickr

And to do that we need another fantastic developer to join the PaperCut team at our head office in Melbourne. If you’re interested, or know any friends who might be … please read on and apply.

The full position description is included below. If you’re applying mention that you read our blog for bonus points!

Senior Java Developer

• Smart customers, smart colleagues, smart code.
• International customer base – work on software that’s used all around the world.
• Influence the direction of our actively developed software.

The Company:

PaperCut Software is an Australian software development house based in Mount Waverley, Melbourne. We develop print, copier and internet control software that is used worldwide in over 60 countries and 20,000 organisations. We are a small team of IT adepts who love to talk tech. The work environment is casual and the organisational structure is flat.

Our customers include many high profile education and business organisations, such as Harvard University, Monash University, the University of Cambridge and Allianz.

The Technology:

Each month PaperCut is downloaded and evaluated by thousands of people world-wide. PaperCut must be dead simple to install, reliable, and have a great user interface. First impressions count! We’re very proud of the product and are looking for a developer who can make the product even better and has great pride in the quality of their work.

Our application is a cross-platform solution of around 1/2 a million lines of code and targets Windows, Mac, Linux and Novell. The majority of the code is written in Java (Hibernate, Spring, Tapestry stack) with large chunks in C/C++ (OS integration layer), and Perl, Python and Shell scripts.

Developing at PaperCut is challenging as our application is used in a variety of environments ranging from 5 user networks all the way to 350,000 users. As a result we need to get our hands dirty on technologies such as clustering, multi-threaded development, embedded device development and effective database design. You will work on all aspects of the product.

The Job:

We’re looking for an experienced developer with at least 5 years experience. Having extensive experience with Java, and an interest in cross-platform development using open source technologies would be an advantage (but if you’re a guru in any language then please apply). The ideal candidate will have a passion for technology, be a great communicator and enjoy working in small smart focused team. We have a flat structure and all developers are involved in all areas of the business. Your contribution and involvement will range from conception through to implementation and sale. We’re also looking forward to hiring someone with new fresh ideas, energy and the desire to implement them.

To Apply:

Express your interest to Tom Clift by sending your CV and cover letter to jobs@papercut.com. In your cover letter, please detail your passion for software development, and why you think you would be the perfect addition to our team.

First round interviews begin February 15th. Applications close March 3rd (or sooner if we find the right candidate).

Developing software for the world represents unique challenges. The most obvious challenge is ensuring that every facet of the application is translatable. PaperCut is already translated into over 15 languages which shows that we’ve done this bit right. You also need to make sure that you are 100% Unicode aware so that you can handle all characters, like Chinese, Japanese, Korean, Russian, etc. Again, PaperCut handles all this with ease.

But occasionally we encounter a problem or unexpected behaviour that only affects users in a particular region in the world. A few weeks ago we had one of these problems. In this case the user was reporting a problem where they were unable to run any PaperCut reports. We obtained debug logs that showed the error details, but even with this information we could not explain the problem. In the end it was the customer that helped find the problem. They reported that if they changed their “language” to “English” the reports worked, and if they changed the locale to “Turkish” the reports failed. Aha!! With that information we reproduced the problem.

I won’t go into all the gory technical details, but it turns out that the problem was that in Turkish the letter “i” when converted to uppercase becomes “İ” (unicode character 0×0130) which is an “I” with a dot above it. Some of our code has assumed that if you uppercased “i” it would become “I”, and when this didn’t occur the report failed to run.

As they say, you learn something every day. And we did! We learned that when dealing with text in different locales around the world … assume nothing.

We’ve now fixed this problem and this fix is now available in our 9.3 release. And with this release PaperCut is available to another large group of users. Prior to this Turkish users needed to run PaperCut under an English/US locale.

Now all we need is a Turkish translation. If you’d like to help translate PaperCut into Turkish (or any other language!!!) then let us know. We’d really appreciate your help.

It’s a great honor be a finalist amongst so many other great companies in the country’s most prestigious business awards. It is recognition of all the hard work the PaperCut team has put into developing a great printer control product and providing great support to our customers world-wide.

Running a business is hard work, and you don’t often get time to reflect on the how far we’ve come and what we’ve achieved. These awards have allowed us to do that, and be proud of these achievements.

The winners will be announced at a formal dinner tonight (19th September 2008). The whole team and partners are coming along. It should be a fun and exciting night.

Wish us luck!!

PS: We’ll post some photos of the night here next week.

Today I’ll explain an issue raised recently by one of our largest PaperCut ChargeBack customers, and then describe how we designed and implemented the solution.

The Problem

This customer was running PaperCut in a challenging environment:

• Around 2,000 users all running the user client to charge printing to shared accounts;
• Users are located at a number of remote sites over slower network connections;
• Printing is charged to over 30,000 shared accounts;
• All users could charge printing to any of the 30,000 accounts;
• Each night an automated process added, removed or modified hundreds of accounts.

They reported the following problems:

• The user clients on remote sites were slow to popup when first printing.
• A large amount of bandwidth was used communicating with the primary PaperCut server.
• The PaperCut server was slow to respond in the mornings.

Analysis

It was clear that the cause of all these symptoms was the large amount of account data downloaded by each of the user clients.   The user client will only download the account list from the server when it detects accounts have changed.  But at this customer the account list was changed nightly.  This caused the client software to re-download the large account list when the user first printed each day.

The client software communicates with the server using XML web services.  After some analysis we found that the the XML representation of the 30,000 accounts was about 7.5 MB.  On a fast local network this size is not a problem, but with 100′s of clients running on remote sites over slower WAN connections this starts to be an issue.  It causes congestion on the network, the popup is slow to appear, and the server becomes overloaded.

After a few brainstorming sessions and consulting with the customer we approached the solution from two angles.

1. Reduce the size of the account download;
2. Allow the account data to be efficiently distributed to remote sites.

Solution Part 1: Reducing the download size

To reduce the size of the data downloaded we changed the account file format in two ways:

1. Change from verbose XML to a much more efficient binary format;
2. Compress the binary data

The net result was a reduction of approximately 25 times!   In the case of this customer the 7.5MB download was reduced to around 250KB. Quite an improvement!

This change alone dramatically reduced the popup speed, and bandwidth used over the WAN connections.  But we could do better!

Solution Part 2: Efficient distribution

We realized that it was very inefficient for multiple clients at the same remote site to download the identical account list.  What if we could only download the list once per site?

We implemented a server-side tool to save the account list to a file (in the identical format described above).  We added a new option to the client software to allow it to load the account list from a file instead of downloading from the server directly.  Then all that this customer need to do was:

1. Schedule a task to create the account file each night (after the accounts were modified), then copy the account file to a file share on each of the remote sites.
2. Configure the clients at each site to load the local account list file.

The result was that only a single 250KB transfer was required to delivery the new account list to all clients at a given remote site.

Conclusion

The result of these changes was dramatic.  Previously, on a site with 20 users the daily account download would consume around 150MB of bandwidth.  After these changes it was reduced to only 250KB.  What an improvement!

Not only was this a great result for this customer, but the product is improved for everyone.

I hope this gave you a little insight into the work we do behind the scenes.  This change wasn’t glamorous or exciting.  It will only be a single bullet point in our version 8.4 release notes.    However it’s often many of these small improvements that make the overall product great.

So today I added news feeds to the website to notify you as soon as a release is available. There are news feeds for both PaperCut NG and PaperCut ChargeBack and are available from the release history and download pages. For the impatient, here are the feed links:

• PaperCut NG Releases
• PaperCut ChargeBack Releases

News feeds are a great way to keep up-to-date with blogs and news. To subscribe to a news feed you need to use a news reader (or aggregator) which is a piece of software (or web site) that retrieves news items from all sites your are subscribed to and notifies you when you have unread items.

Both Firefox 2 and Internet Explorer 7 have built-in feed readers, but they are pretty basic. If you’re interested in something more powerful we highly recommend Google Reader. All the developers at PaperCut use Google Reader daily to keep up-to-date with what’s new in the world.

Happy news reading!

• Add “enable/disable user net usage” option to the bulk user action page, so this setting can be set on users in bulk.
• Add support for the MS ISA Server log format, in addition to the preferred W3C log format.
• Improved detection of the latest log file based on the specified log file mask.
• Improved behaviour when processing MS ISA log format, when the W3C log format is specified in PaperCut.
• Do not log net usage performed by Squid users that haven’t authenticated correctly.
• Fix to error that occurred when running a transaction log report that contained Internet usage transactions.
• Fix bug in Linux installer that caused the Internet service config file to be overwritten when upgrading from a previous version.
• Improved the setup of the Internet Provider service on linux to warn the user if the specified log file directory is not readable.
• Add a config option to enable LDAP referrals.
• All the many changes included in the most recent public release of PaperCut NG 6.3. For more information see the release history page.
• Other small fixes.

You can download the latest release (Build 4006) from the Net Control module testing page. To upgrade from a previous release you can perform an install over the top procedure.

• All the changes included in the most recent public release of PaperCut NG 6.2. For more information see the release history page.
• Other small fixes.

You can download the latest release (Build 3909) from the Net Control module testing page. To upgrade from a previous release you can perform an install over the top procedure.

• Allow the internal admin password to be changed from within the user interface (on the Options->Advanced) tab.
• In the Internet Usage logs screens, allow the results to be sorted by the “MB received (ex cached)” column.
• Fix error that can occur when saving groups if only the Internet Control module is enabled.
• Other small fixes.

You can download the latest release (Build 3858) from the Net Control module testing page.

You can download the latest release (Build 3853) from the Net Control module testing page.
The latest build include the following changes:

• Allow the Internet usage for a single user to be disabled for a specific amount of time or indefinitely (time latches). This can be useful to discipline students who have abused Internet privileges, but you still want them to be able to print.
• Allow data costs to be defined “per gigabyte” in addition to “per megabyte”
• Allow the user client window position to be changed. This allows multiple user clients to be run when different PaperCut instances host Net and Print control.
• Clarify some of the terminology to make it easier to understand whether data includes cached data or not.
• Ignore machine accounts (e.g. username$) in proxy logs. These entries are recorded when a Windows machine downloads Windows updates.
• Improved LDAP support for large sites.
• Licensing changes to allow users who are registered for Print Control to be able to trial the Net Module for 40 days.
• Bumped the version number to 7.0.
• All the updates included in the latest 6.2 release (see here for full details).