Case Study

Impact stats

• 1.1 million lines of code analyzed in two weeks

• 30% reduction in average time needed to understand legacy code

Summary

Since 2022, Nava has partnered with the New Jersey Department of Labor & Workforce Development (NJDOL) to build a modern, flexible, and resilient end-to-end unemployment insurance (UI) system. Currently, we’re helping NJDOL replace their legacy mainframe with a cloud-based system built with a more modern architecture to improve systemwide performance, reliability, and importantly, observability.

Legacy mainframes for unemployment insurance programs were designed decades ago with a now outdated coding language. Since then, technology and tooling have advanced, enabling systems to handle higher volumes of claims, complex scenarios, and real-time integrations with other systems. Integrating these modern tools with legacy platforms is complex, time consuming, and risky, prompting many agencies to modernize their systems. 

The initial technical discovery process requires extensive interviews with subject matter experts (SMEs) and extensive, manual code reviews to confirm the current system’s functions. This must be done before designing the replacement system, ultimately adding more time to the modernization project. We’re working with NJDOL to evaluate whether AI-powered tools can streamline the process of analyzing and documenting the mainframe codebase.

Approach

We helped NJDOL conduct a two-week pilot to evaluate how Amazon Web Services (AWS) Transform, an agentic service, could speed up the process of technical discovery for four code modules. The pilot aimed to understand if the tool can identify what these modules do, how they connect to each other, and whether the tool’s code outputs are accurate and usable enough to support engineering developments and subject matter expert review. We developed a methodology for evaluating the tool, measuring how long engineers and SMEs took to understand a module and comparing it to time spent on the manual process.

Outcomes

After the two-week pilot, the team had plain-language summaries for each code module that included important pieces of business logic. Reviewers can access this information without having to return to the raw code, an important aspect to helping reduce time spent on technical discovery. Transform also produced maps to show the relationships between programs and data, illustrating how the modules rely on each other to function. Finally, NJDOL used Transform’s analysis to derive test cases for the new code. Overall, leveraging Transform reduced the time software engineers and reviewers needed to understand the modules by an average of 30%. 

Transform also provides a more comprehensive understanding of the legacy system by identifying dependencies, business rules, and integration points. This foundation helps NJDOL accelerate the process of analyzing and documenting the legacy codebase and designing the replacement solution.

Process

We helped NJDOL configure and operationalize Transform to analyze the legacy system’s codebase, uploading the source code to AWS data storage so Transform can process it.

NJDOL focused the Transform pilot on four code modules to gather valuable insights in a quick timeframe. Each module had varying amounts of existing prior analysis and documentation that helped inform and analyze the tool’s output for accuracy. 

We reviewed all Transform outputs with NJDOL for technical accuracy, operational importance, and usability for software engineering review. We helped document what the tool gets right and how any corrections, follow-ups, and gaps are identified, making the process reusable and scalable. We also measured how long it took to reach a usable understanding of the module and compared it to the manual process. 

Conclusion

Nava is partnering with NJDOL to evaluate if AI-powered tools can streamline the technical discovery of a legacy mainframe modernization project. After a two-week pilot, we found that Transform helped reduce time spent on technical discovery by about 30% and provided engineers and experts with technical assets to inform the next project phase: designing the replacement system. We also helped NJDOL document the process to inform future testing and evaluation.