Skip to main content

Using a pilot to minimize risk and rapidly deliver new services

By Nicole Budzius, Product Manager

Nava is partnering with the Commonwealth of Massachusetts to support the launch of their brand new Paid Family Medical Leave (PFML) program. When the program launches in January 2021, we will have designed, tested, and built a new digital service, paidleave.mass.gov — in just 13 months — to help Massachusetts workers as they apply for and manage benefits. In tandem with this work, we’re also building the API integration layer to connect multiple state systems with paidleave.mass.gov and other paid family and medical leave systems.

In our shared long-term vision, Massachusetts workers, families, and employers will be able to confidently navigate life’s critical moments because paidleave.mass.gov’s personalized support makes managing paid leave simple.

Approach

We launched this project with several pilots. In a pilot, we incrementally develop pieces of the end-to-end experience and test them with real users (pilot participants) before the service launches to the public. This helps us identify and mitigate risks early on so we can affirm that what we ultimately deliver works for users, integrates with downstream systems, and collects the information needed to meet program goals. In a multi-vendor environment with a strict legislative deadline, pilots also serve as milestones for teams to align on so we can collaboratively build and validate new features.

For our first pilot, we designed and built an application that allowed claimants to create an account, log in, verify an account, reset the password, log out, and take the first steps toward creating a claim. We also tested a key technical integration with the Department of Revenue (DOR), which provides contribution and wage data that is essential to determining a claimant’s eligibility.

In the second and third pilots, we expanded the application to allow claimants to submit an application and ensured that Massachusetts administrators receive the complete and accurate data needed to adjudicate a claim. And in a future pilot, we’ll confirm that payments are made to the right person, in the right amount, at the right time.

Building and testing piece-by-piece, over multiple pilots, also allows us to test the application with a diverse set of participants to further ensure that it’s simple and easy to use for everyone.

Outcomes

Pilots let us build, test, and see results quickly. Within a few months, we confirmed that most claimants could easily navigate account creation and that the DOR integration was successful. Our initial findings:

  • 73 percent of participants were able to create an account

  • 90 percent could reset their password

  • Of the 73 percent able to create an account, 100 percent could take the first steps to create a claim

  • 28 items in the account creation flow were identified for improvement

  • 100 percent of DOR data was successfully delivered, received in the correct formatting, and added to a database with zero errors

In the short term, our partners in Massachusetts were assured of our ability to solve increasingly bigger parts of the picture. In addition, we could quickly prioritize our next steps — based on the 28 items we identified for improvement — to ensure that 100 percent of claimants will be able to navigate account creation and that paidleave.mass.gov would be accessible to all at launch.

Process

Mitigating risks with real users

Nava often works in multi-vendor environments with strict legislative deadlines. In Massachusetts, we have 13 months to build a service to support an estimated 161,000 claims annually — from scratch. And on this particular project, we onboarded before the other vendor teams. So we wouldn’t be stalled, we specifically chose a scope for our first pilot that was independent of other teams.

We focused on the account creation user flow, from account creation through the first step of making a claim. In addition to being something we could start immediately, the account creation flow is also high-impact; it’s one of the first things PFML claimants will experience. Getting this foundational component right is essential to building a seamless user experience.

A simple log in page helps make the first steps as easy as possible.

For the pilot, we used the production environment. Our 11 participants completed the testing remotely, either on a desktop computer over Zoom, or on a mobile device, using a tool called Validately. Participants were Massachusetts state employees (we didn’t yet have identity proofing enabled and needed to be able to otherwise verify participant identities) who will eventually be eligible to use the service.

The ability to test a service with the people who will actually use it is essential to a successful pilot. It lets us observe and identify user needs and fix negative user experiences that could ultimately hinder adoption and use of the new PFML service.

Each session had a participant, facilitator, and notetaker. Occasionally, someone from the Massachusetts program would also join as an observer. Seeing how participants interact with an application in real time gives program partners firsthand knowledge of how their programs are experienced by real claimants — it’s the most informative part of this process. So we encourage program partners to participate in pilot sessions whenever possible.

Identifying user needs

During this first pilot, a participant surfaced an unexpected error message when attempting to verify their account. When entering a verification code, they accidentally added an extra space at the end. The participant couldn’t easily see the mistake. And the consequent error message used confusing, technical language; only the developers on our team understood it.

Instead of just updating this confusing error message, we streamlined the user experience to avoid it all together.

Getting an error message that doesn’t help users solve the error is a frustrating experience that erodes trust. So, from this finding, we eliminated the error message issue by simply allowing users to enter a verification code even with an extra space, streamlining the user experience.

In another instance, we iterated on the dashboard based on participant feedback. We revised both the content and the layout until we had a version that met user needs. Ultimately, we discovered that users need to see: their progress through the application, what types of information they’d need to complete each step, and new applications and in-progress or completed applications under different tabs. These features — a direct result of the findings in our pilot — provided a more clear user experience.

To best organize the information on this page, we tested several different versions of both the content and the layout with users.

Reducing administrative burden

In another testing session, we discovered a loophole where participants were stuck because they were unable to reset their password. This was important because we learned from other states with PFML programs that password resets are a primary cause of high call volume at call centers. We knew that creating a simple self-service reset password flow would reduce the burden on the call center, saving staff time and resources. (And in “Administrative Burden,” Pamela Herd and Donald Moynihan emphasize the importance of reducing unnecessary, often time-consuming tasks, so that government agencies can better serve citizens.) In this case, a participant mistyped their email address when completing the flow for resetting their password. When they went back to the login screen to try again, the site had saved their mistyped email address and would only let them create a new account. So, we updated the flow to allow users to fix a mistyped email address and still successfully reset their password.

Conclusion

Pilots let us discover unforeseen and unpredictable errors like these in a low-risk environment, before they become a barrier to claimants and a burden to staff. In this first pilot, we were able to build and test the first steps of the account creation process — an essential part of the user experience — with real users. Working continuously and incrementally, we’re able to address issues as they appear so that we ultimately deliver a simple and effective experience for all users. This particular pilot was our first step in building Massachusetts’ brand new Paid Family Medical Leave (PFML) program at paidleave.mass.gov. This new service will ultimately support an estimated 161,000 claims annually and make managing paid leave simple. Our shared goal is to ensure that workers, families, and employers in the Commonwealth will be able to confidently navigate life’s critical moments.

Work with us

Let’s make government services simple, effective, and accessible to all. Get in touch.