By Jessica Paddock
Courtesy of StickyMinds
What happens when an enterprise with a feature-rich, web-based fulfillment platform wants to enhance performance by aggregating its multiple external functions into a single, unified application? The software team transitions the legacy system to a composite architecture. Although using a composite framework can greatly enhance application flexibility, reliability, and functionality in production, the transition to this complex structure is often difficult and may require a considerable amount of process and procedure restructuring—or even reinvention.
The challenge can be especially daunting for performance testing teams. In the best of situations, transitioning existing processes to meet the increased complexity and test data requirements for scripts and scenario development often requires a customized effort. When the platform incorporates siloed individual systems, layers of piecemeal changes, and intensely territorial company stakeholders, achieving a satisfactory outcome can be problematic. Yet this is precisely the environment in which many legacy, corporate-developed software platforms operate.
I recently worked on such a project, where the structure of an existing web platform, combined with the behaviors and attitudes of various teams, had made the transition to a composite application more difficult than it needed to be. Although the composite application had been launched, its design continued many of the problems inherent in the legacy system, resulting in a slew of defects in production. As a consultant, I was involved in identifying pre-existing problems that had hampered system performance and proposing solutions that would enable testers to resolve production defects. Our team was also charged with running a simulation with the new solutions in place to ensure final testing approaches would contribute to better application performance.
We identified four key areas where change was pivotal to success: test data reporting, user and load models, ongoing data management, and team approach and structure. Although we completed initiatives for each of these components, for this article I will focus on one: reporting, specifically key performance indicators (KPIs).
Project History and Challenges
For this particular enterprise, the legacy fulfillment system had been a web of more than fifteen applications, fifteen user interfaces, and thirty middleware systems—tied into a centralized platform with a custom-developed bus/messenger—that served both external and internal users. There was simplicity in the siloed approach, but it came at the expense of flexibility and speed to market for new product offerings. As technology advanced and site traffic escalated considerably, the organization’s business decision-makers determined that a composite application would be the best approach.
There were prior issues with the siloed structure that had led to the development of a composite application:
- Testing was often done at the individual application level, or sometimes with subsets of applications that shared back-end systems.
- Over time, application development teams had responded to user requests for additional functions by developing purpose-built solutions without considering how the changes would impact system performance as a whole.
- Quarterly releases often involved many changes across the entire platform, yet the disorganized development approach meant there was no comprehensive set of tests that could generate a view of overall system performance before each build release. With ten fulfillment systems (internal and external) sharing multiple back-end systems as well as the main fulfillment system records database, it was important to be able to test all the applications together. Only then could performance testers accurately create a baseline for system performance to assess the impact each release’s changes would have on the system, operating as a whole.
- Test data was extremely hard to find. Most test cases needed their data to be created. This had to be done manually by the testers, which consumed a great deal of their time.
As you can imagine, errors in production were common—and their sources were difficult to pinpoint in testing. In fact, even replicating performance issues was problematic, given the unwieldy nature of the platform.
Resolving these issues—and giving the performance testing team a unified mechanism for assessing the impact of changes to the system—would have been nearly impossible without re-evaluation of the current environment. With the move to a composite architecture, problems that occurred during the transition were in some ways making matters worse.
A prime example is scripting. When application testing teams of three to five people were testing one application, they might have needed to manage ten scripts per application, with ten transactions per script. In the new composite application, built of fifteen applications with the same number of scripts and transactions, there would now be fifteen hundred transaction timers—data that forms the building blocks of the KPIs. Even though the testing team had used transaction naming consistently across all test scenarios, such an enormous number of transactions was making test execution unwieldy, and performance evaluation and reporting next to impossible.
Imperative KPIs Rise to the Top
To alleviate the overload of information, our team determined that defining a meaningful set of KPIs for test results would be pivotal to success. We asked organizational stakeholders to settle on three to five KPIs that were most important to application performance. The goal was to keep the team focused on the slowdowns that mattered most, thereby allowing the most important transaction timers to float to the top.
The KPIs chosen by the stakeholders were “make or break” items that would cost the company sales, impact user satisfaction, or otherwise be problematic for the system:
Business impact: How the change impacted the business, its customers, and the composite application’s users (both internal and external)
Production volume: The transaction count for the change requested, with high-transaction count changes being evaluated more carefully than more simple ones
Known issues in the past: Whether the behavior or process being modified had caused problems previously and therefore might be likely to do so again
From a previous count of some five thousand KPIs, we ended up with approximately a hundred KPIs across the applications that had been integrated into the composite application. We would use these KPIs when simulating the fulfillment engine activities for the system, and they would also provide benchmarks for the resulting testing methods. KPIs that didn’t make the cut would still be timed, but testers wouldn’t be able to look at them when they were running a test.
This approach eliminated the “ghosts” that were bogging down testing at the time we engaged with the customer. For example, their prior performance goals had been based on taking the past three runs and adding 10 percent. If a goal were one second, it would be 1.1 second for the next release. There is no way a human can tell the difference between a transaction processing time of 1 second and 1.1 seconds, so KPIs like this were meaningless, and resolving them did not have enough impact on performance to be worth pursuing.
We also created a test data management (TDM) team to fulfill the almost impossible data needs. The TDM team handled the test data requests for all the performance testing, creating millions of records to satisfy our high-volume testing. The resulting data was delivered to the back-end systems, and the testers received spreadsheets with all the created data. This was incredibly helpful in keeping testers from stepping on each other’s toes.
Simulating Success
With the KPIs in place, we were ready to develop the load models teams would use to model performance testing workloads moving forward. We looked at production data to identify the “busiest day in the life” of the application, then mapped the KPIs against them to define user actions for the models. We also added pivot tables to the data sheets the TDM team delivered in order to gather the data sets for building the user community modeling language (UCML) diagrams.
The UCML diagrams included not only KPI actions, but also non-KPI user actions, such as an offer being explored or a search being conducted with no resulting follow-through. Once the UCML diagrams were complete (one for each application in the system), the team reviewed and confirmed each UCML diagram with business analysts and stakeholders and proceeded to develop a single, modularized script that emulated the composite application’s workflow.
With this script complete, testers were able to develop performance tests that were both efficient and effective and to propel the development effort toward a satisfactory result. Additional benefits of the project have been reduced script maintenance, defect reduction (due to having correct test data), an increase in testers’ availability (thanks to the new TDM team), and improved team communication. More importantly for the organization, the team has met its production goals of increased application throughput and a reduced incidence of outages, and has been able to increase coverage through the inclusion of additional applications that were not being tested previously.