Choose your language

Choose your login

Support

Blog

PaperCut security post-incident report from April 2023

In April of this year we had a significant security incident in PaperCut MF/NG that affected our customers. In this post today, I want to be courageous, live our company values, and transparently share what happened, what we’ve learned, and what our plans are to continuously improve our security posture for the future.

I personally learned a lot during this time, and I hope that the transparent approach also helps others get value from our experience too, such as other software vendors in our industry, our partners and resellers, and our customers alike.

Before we break into the details, as CEO and Co-founder, and as a Software Engineer who actually contributed to the code that caused the problem, I want to acknowledge the impact this issue has had. I wish to thank our customers and partners for the efforts they put in to patch vulnerable servers, and apologize for the impact. I understand that our products are mission critical to your organizations, and we don’t take that responsibility lightly. The code leading to the vulnerability in question was added almost 15 years ago. If I didn’t author this code, I definitely reviewed it as there were only a few people in the company in those days. The buck stops with me, and I’m personally committed to making sure we all learn from this.

For those customers affected, we are working to regain your trust, and the openness of this post is part of that. We understand that PaperCut, and printing in general, is now no longer  "below the radar" and in sight of hackers. We understand this will not disappear and we must step up our security posture, and support our customers to do the same. We are committed to making sure PaperCut remains a world-class platform.

What happened?

In January 2023, two vulnerabilities were identified in PaperCut Software’s MF/NG applications, brought to our attention by security researchers from Trend Micro.

We released a software patch on 8th March to address these issues, however, a month later by 18th April, the first attack was observed on a system that hadn’t been updated.

Though 98% of our customer base had either already implemented the patch or had firewall protection, around 2,000 organizations were at high risk. These unprotected systems, accessible via the internet, became prime targets for ransomware attacks (some of which were staged by Nation-state-backed hacker groups).

In response, PaperCut Software initiated an urgent outreach program to secure these public vulnerable systems.

Timeline

All dates and times are reported in AEST (Melbourne, Australia), unless otherwise noted

A TL;DR summary timeline is summarized below. A full timeline, with annotated learnings can be found at the bottom of the post.

10 January 2023

PaperCut acknowledges two vulnerabilities in PaperCut MF/NG that were reported by Trend Micro under their Zero Day Initiative.

7 February 2023

Key PaperCut Authorized Solution Centers and PaperCut Resellers are confidentially notified of the identified vulnerability and the upcoming patch so they can support any customers when the patch is released.

8 March 2023

PaperCut MF/NG software update released to address vulnerabilities. A security bulletin is published and all subscribers to the PaperCut security mailing list are emailed.

14 March 2023

Trend Micro publishes additional details including the Java Class name containing the vulnerability on their website via ZDI-CAN-18987 and ZDI-CAN-19226.

18 April 2023

The first customer to report “in the wild” suspicious activity contacts PaperCut. Their software raised an alert on their unpatched PaperCut server. Log analysis indicates possible entry via PaperCut MF.

19 April 2023

A second unpatched customer’s security software reports an alert. The Security Bulletin was updated indicating that we were seeing exploits in the wild. Customers, particularly those with servers open to the internet, are urged to upgrade immediately.

19 April 2023

PaperCut initiates a cross-functional, cross-region incident response team (this team meets twice a day).

20 April 2023

PaperCut’s external security company embeds two members into PaperCut’s Melbourne office to bolster the response team. Work starts on identifying unpatched public PaperCut servers using the same tools and database “the hackers use” to cross-link with PaperCut’s own data to help identify high-risk customers.

21 April 2023

High-risk customer identification list complete. Proactive outreach program commences on Friday aiming to contact approximately 2000 customers via phone, email, and fax. This continues uninterrupted for 48 hours using callout teams in PaperCut’s Australian, UK, and USA offices. Researchers at Huntress.com release additional findings on Indicators of Compromise (IoC). IoC detailed updates are added to the security bulletin, along with the date and time of the first reported exploit.

26 April 2023

The FBI contacts PaperCut. PaperCut liaises with the FBI, CISA, and ACSC to provide information to support advisory alerts.

27 April 2023

Microsoft Threat Intelligence attributes detected exploitation to Russian-speaking group “Lace Tempest”.

1 May 2023

Internet scan shows the majority of high-risk customers have servers patched.

6 May 2023

Microsoft Threat Intelligence attributes additional attacks to Iranian state-sponsored threat actors.

1 June 2023

Threat level reduces and PaperCut shifts to the learning phase.

What we learned from this incident and what we’ll do next

From our post-incident reflections, we’ve derived the following insights and action steps. They were collected from retrospectives conducted across multiple teams and were enriched by insights from our security consultants, partners, and customers. Our learnings have been categorized under Product, Organizational, and Data and Communications domains.

Product change learnings

1. Application notification and dashboard alerts are often ignored

The security update was issued via our typical release process which includes an in-app check-for-updates notification, release notes, RSS feed, in-app dashboard news, and partner portals. Because it was a security focused release, subscribed customers were also notified via the security mailing list. We learned:

  • Many customers did not notice the severity level - announcements were not generally visually different.
  • Some customers rarely log into the admin console, and hence requests to update or check-for-updates were missed.
  • In some workplaces, administrators had changed and the new admin was not aware of notification sources and locations.
  • Many messages were ignored in the noise of less urgent notifications.
  • Our customer-to-subscribe ratio on the security mailing list was low

Our action plan

  • Many software vendors are now starting to make security specific notifications inside the app that are highly visible and require action. A good example is Apple’s recent changes around “Rapid Security Response (RSR) updates”. These notifications are highly visible and require action. Our plan is to introduce this new style of alert in our applications.   (NOTE:  This has been actioned in the  23.0 release)
  • We’re actively promoting the security mailing list, and uptake has been significant. We plan to increase awareness further by introducing a new process in PaperCut MF/NG’s administrator console. For example, when a new admin is added to PaperCut MF/NG, they’ll be encouraged to ensure their organization contact information is current, and have their attention drawn to the security mailing list (also see Organization Changes Learnings below).

2. Our approach to penetration testing needs to step-change in line with the security landscape

It’s clear that PaperCut, and printing in general, is now “above the radar” in the hacker and malicious actor domain. This heightened visibility isn’t a transient phase—it’s the new reality. Recognizing this, we must intensify our penetration testing efforts and deepen our engagement with the InfoSec community.

Our action plan

We’ve incorporated several new top-tier pen testers into our white box penetration testing initiative, where testers have access to the source code under a confidential agreement. This not only increases the frequency of our testing but also brings varied expertise.

Furthermore, we’ve created a specialized internal security team responsible for coordinating with prominent InfoSec partners and vendors to assist more research on PaperCut MF/NG. Our objective is to intensify the focus on PaperCut’s security and expedite the journey from issue reporting to resolution. 

In tandem with this, we set up a group of our most experienced engineers who regularly need to look at ways of enhancing the security of the product. This has resulted in a series of security-centric releases (version 22.1 is an example) to facilitate this transition. 

All of this complements our ongoing efforts to enhance development practices in terms of tools, training, and procedures - a journey we embarked upon with our ISO27001 adoption and enhancement.

3. Familiarity with our suggested server security hardening measures was not widespread

Although a significant number of our customers remained protected due to their security strategies—even if they hadn’t applied the latest patches—we identified a segment that wasn’t acquainted with these measures. We provide Knowledge Base articles and dedicated sections in our user manual addressing this area. However, leveraging these resources necessitates proactive efforts to locate and adhere to them.

Our action plan

Similar to the in-app notification and alerts change, we plan to make a more visible link to security hardening practices within the product. Where possible, we will actively highlight potential actions that may apply to the environment. We will re-evaluate security defaults, for example, a public addressable server should restrict access to selected areas based on approved network address locations such as where printers are located. (Note:  This has been actioned in the  23.0 release)

We will also review application default sessions, and where appropriate consider more secure or “locked down” defaults. Keeping environments secure is not something we can do alone, it’s a joint responsibility with our partners and customers and we need to work closely with all parties to ensure that we are all staying on top of security.

Organization change learnings

1. Strengthening our communication and proactive stance with infosec vendors is crucial to ensure they grasp the nuances of our product, our customer demographics, and our partner ecosystem

In the wake of the incident, our review and retros were not confined to just PaperCut. We also included Trend Micro, who played a pivotal role in identifying the core vulnerability, in our process. This exploration revealed some gaps in understanding how our product is implemented and used. Notably, it’s frequently managed by the customer, with upgrades subject to processes like change controls, planned downtime periods, or updates timed around school vacation periods, among other nuances.

We learned there is actually a strong desire for more cooperation, and this is exemplified by this quote from a leader in Trend Micros’s ZDI team:

We thank you for the opportunity to collaborate with you on improving your processes and we are “fighting” for the same cause. I appreciate your coming forward with this request because it’s not too often that vendors will reach out for guidance like this. I can only think of a handful out of the hundreds we deal with each year. We are oftentimes seen as the “enemy” but in reality, we just want to ensure our Trend Micro customers are protected regardless of whether or not we are able to release protections for any particular vulnerability we acquire or discover.​​​

Our action plan

To bolster our security, we commit to amplifying our interactions with the InfoSec community and security resources. Our approach will be both anticipatory and responsive, especially during incident evaluations. Our initiatives will encompass the following:

  • Enlightening the community about the intricacies of our software’s deployment and usage
  • Pushing for a grace period before any complete disclosure of vulnerabilities or proof of concepts
  • Involving InfoSec experts in our post-incident analysis, ensuring comprehensive learnings for every issue.

2. Improved CVE reporting and coordination is crucial

Traditionally, we’ve depended on the InfoSec community for synchronizing updates to the CVE database, typically via the CNA registered security researcher. We recognized that a significant number of our customers rely on CVE notification services or deploy software scanning tools that utilize CVEs for insights. During this incident, there were noticeable lags in updating and publishing the relevant CVE, leading to customers using this process being alerted belatedly.

Our action plan

To protect our customers, we want to control CVEs so the publish date and information provided in them match our own advisories and releases. We’ve taken the action of joining MITRE as a CNA. By doing so, we’ll be directly responsible for overseeing this crucial process.

Data and communication learnings

1. Our customer contact information is often not current

During our active call out period we attempted to contact identified high-risk customers with publicly addressable unpatched PaperCut MF/NG servers. During this phase we identified that some customer contact information was stale or outdated. This was a hindrance and we often needed to resort to web search and LinkedIn to get current information. Most of our on-record contact information was collected at the time of purchase, or at best the customer’s latest support ticket. With many administrators and IT people changing roles in the last few years, a notable percentage of the contact details were incorrect.

NOTE: I should also acknowledge I’m proud of our team’s effort during this phase. Many team members went to great lengths to ensure the right person was contacted, and customers were patched quickly. A few stories:

  • A Canadian college’s overnight helpdesk person was surprised to receive a call from Melbourne. He usually deals with late night calls from stressed students trying to submit assignments on deadlines. He took it upon himself to help track down the new admin responsible for the server and made sure they were briefed on arrival in the morning.

  • Our team had issues navigating the switchboard of a Thai Government organization and was unable to leave a message. Their IT department had a fax number, and we got their attention quickly that way resulting in the server being upgraded soon after.

  • DMs via LinkedIn got us quickly in touch with a new admin, again resulting in quick server actions.

However we need to acknowledge that incorrect contact information was inefficient and impacted the time it took us to contact some of our high-risk customers.

Our action plan

We will add the ability to review and update customer contact information from within the application, via the About Page or equivalent. When a new administrator account is created, we’ll ensure this admin is encouraged to review this. We’ll also look at the feasibility of promoting the security mailing list to new admins.

2. We have inconsistent emergency response processes in our channel network of partners and resellers

We need parallel paths of communication into our customers. Many of our resellers are trusted partners of our customers and were able to get important messages to our customers quickly. However, in some cases the relationships didn’t exist the way we would have liked and so these messages didn’t get through.

Our partners can be extremely valuable to support our customers in cases like this. 
EXAMPLE: A channel partner called the security contact at a major university. The response was dismissive indicating they would get to it “in the next few weeks”. Clearly this isn’t good enough and so the partner called the head of IT and said “if it’s not upgraded this morning I’ll get in the car and do it myself”.

In other cases we have channel partners where security is regarded as “a new topic”  in our print domain. PrintNightmare put security on the radar in print but it was seen as a one-off event instead of a need to drive a systematic security posture improvement. In this incident, there were some cases where security messages or support were either not effectively conveyed to the customers by our partners, or when conveyed, they were not passed on to the appropriate individuals within the customer organizations.  This was a key learning for both us and our partners.

Our action plan

  • We will add security as a dedicated component in our partner and reseller training and certification program
  • We will define formal cybersecurity DEFCON levels, and expected responses for both PaperCut and our partners
  • We will work with a sample of partners to include them in our future table-top war game exercises and response drills
  • In addition to the product changes listed above to ensure contact information is current, we’ll encourage our partners to assist here too.

All the learnings, plans and actions listed above are owned by leadership, and I, as the CEO and co-founder will be overseeing them. While the vulnerability discussed in this review did not impact our next generation cloud products (PaperCut Hive and PaperCut Pocket), the learnings will be applied to all products and processes across PaperCut Software.

Closing thoughts

We’re well aware that there is never a convenient moment to respond to a security incident. A heartfelt thanks goes out to all our customers, partners, and resellers who worked diligently and swiftly during this challenging period to promptly update systems. The efforts and resilience displayed by our PaperCut support team, alongside our partners, filled me with immense pride. 

This incident, while testing, will undeniably make PaperCut stronger. We’re committed to deriving crucial lessons from this experience and, in our spirit of openness, will continue to share our learnings for the collective benefit of all. To every customer who stood by us: thank you. Your trust and support drive us forward.

Full timeline

This version of the timeline has more detail than the summary above. We have also annotated it with mixing in the learnings with a mindset of “If we had applied the learnings over the incident again, where would things be different”:

10 January 2023

Two vulnerabilities in PaperCut MF/NG were reported by Trend Micro under their Zero Day Initiative.

24 January 2023

Initial fix for ZDI-CAN-19226 committed to source code branch.

31 January 2023

Initial fix for ZDI-CAN-18987 committed to source code branch.

7 February 2023

Key partners and resellers were confidentially notified of the identified vulnerability and the incoming patch.

8 March 2023

Released a PaperCut MF/NG software update that addressed the following vulnerabilities: Remote Code Execution vulnerability (CVE-2023–27350 / ZDI-CAN-18987); User account data vulnerability (CVE-2023–27351 / ZDI-CAN-19226).

Due to the potential severity, releases were available for all currently supported versions of Papercut NG and PaperCut MF back to version 20 (20.1.7, 21.2.11, 22.0.9).

8 March 2023

We published a security bulletin KB article here: https://www.papercut.com/kb/Main/PO-1216-and-PO-1219 NOTE: This page was regularly updated (generally daily) as new information came to hand.

8 March 2023

We emailed subscribers to the PaperCut security mailing list to bring the new bulletin to their attention.

14 March 2023

Trend Micro published additional details of the vulnerability on their website: ZDI-CAN-18987 and ZDI-CAN-19226.

18 April 2023

We first received evidence to suggest that unpatched servers had been exploited ‘in the wild’. This was in the form of a security report from a customer whose security software had intercepted attempts to compromise the server. PaperCut was the primary software on that server. Our security team requested PaperCut server logs to conduct analysis and look for indicators of compromise (IoC).

18 April 2023

PaperCut sets up a cross-functional, cross-region incident response team.

19 April 2023

Closer inspection of logs suggested the server was compromised through PaperCut. The giveaways were that the server was unpatched at the time, and logs indicated that config keys disabling print script sandboxing were set by the user setup-wizard in quick succession. The speed of updates suggest it was scripted. We received a second report from a customer of security software raising an alert. We updated the Security Bulletin with the new information discovered declaring that exploits were happening in the wild. Indicators of compromise (IoC) added to guide administrations on detection and mitigation.

19 April 2023

We emailed channel partners (including our Authorized Solution Centers as well as individual PaperCut resellers) notifying them of the important updates to the security position (e.g. exploits in the wild) and reiterating that unpatched servers remain at risk of exploitation.

19 April 2023

Trend Micro updated the details of the vulnerability on their website.

20 April 2023

We emailed all direct PaperCut customers (PaperCut NG), and indirect PaperCut MF customers for whom we had validated email contact details provided by resellers, with notification of potential vulnerabilities and the evidence of unpatched servers having been exploited, and directed to install the appropriate maintenance at their earliest opportunity.

20 April 2023

PaperCut’s external security company embedded two members in PaperCut’s Melbourne office to bolster the response team. Data indicated that around 35% of customers had patched their servers at this time. Work started on identifying public unpatched PaperCut MF/NG servers using the same tools and database “the hackers use” to cross-link with PaperCut’s own data to help identify high risk customers.

21 April 2023

Identify high risk customer list complete. Proactive reach out program commenced on Friday and over the Weekend aiming to contact approx 2000 customers via phone, email and fax. This continued for 48h using callout teams in PaperCut’s Australian, UK and USA offices. Researchers at Huntress.com released additional findings on Indicators of Compromise (IoC). IoC detailed updates in the security bulletin, and along with the date-time of first reported exploit.

26 April 2023

The FBI contacts PaperCut. PaperCut liaised with multiple national agencies including the FBI, CISA, and Australian Signals Directorate to provide information to support advisory alerts.

27 April 2023

Microsoft Threat Intelligence attributes detected exploitation to Russian-speaking group “Lace Tempest”.

1 May 2023

Internet scan by the PaperCut security team shows the majority of high-risk customers have servers patched.

6 May 2023

Microsoft Threat Intelligence attributes additional attacks to Iranian state-sponsored threat actors.

1 June 2023

PaperCut begins post-incident learning phase. Retrospectives are scheduled across all business areas.

September 2023

Learnings consolidated and this writeup created and shared internally and to partners.

November 2023

Learnings shared to the public via the PaperCut website.

Newsletter

Subscribe for the latest in print management and product updates!

By filling out and submitting this form, you agree that you have read our Privacy Policy, and agree to PaperCut handling your data in accordance with its terms.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.