Choosing this alternative medium to share with a wider community, I have extracted a small deck of slides from my course “Test Management: Leading Your Team To Success” and linked it below.  Hope you enjoy.

View the course extract…

We apply and tailor our test approaches to match our customers’ development and testing life-cycles, always keeping their project constraints and business goals in mind.

The “V-model” provides a rudimentary yet effective way to describe a software development lifecycle regardless of the organization’s or project team’s chosen methodology or processes.

A Scalable V-Model as a Tool

In the diagram below, the scalable model illustrated is used to facilitate discussion around the testing approach options.

In discussing the ability and need to scale the rigour of testing and which test planning and execution techniques can be employed, visibility is provided to the project team as to what challenges and risks that may exist for successfully assessing system quality – resulting in recommendations on how the test team will be able to maximize their contribution within the specific context of the project.

One crucial point to take from the V-model is the implication that the definition of the acceptance tests should be possible before test cases for technical testing are defined or code is written.

This encourages involvement and collaboration by all the parties early in the project when feedback regarding the scope of change could prevent serious issues later in the project cycle when it is much more expensive or even impossible to make changes – part of a “smart” approach to testing.

Finally, the V-model embodies the ability to describe and promote validation activities at multiple levels as well as verification.

Summary

PQA’s overall approach to testing is risk-driven (whether the project is Agile, Iterative or Waterfall) such that we examine the project and the system to be deployed for risks to system quality, risks to the test effort being successful within the context of the existing or assumed constraints, and any implied risks to the business/organization.

Using this risk-driven approach, PQA scales the level of formality and the depth or detail of the testing required for:

• Each area of the system
• Each phases of the project
• Each type of testing to be employed

Then we match the appropriate technique for testing to fit the pre-determined need for formality and detail.  Often the result is a hybrid of multiple techniques employed as the project progresses.

Consciously deciding your outsourcing approach is similar to choosing to use risk to drive your testing – Your resulting strategy will be customized to your specific needs and constraints.

Often the outsourcing solution is not decided by the team but rather mandated or inherited (including the choice not to have one), much like the expensive software tools that sit on the proverbial shelf – Purchased without a proper needs vs. solutions analysis, or a plan of how to implement and roll-out to gain the full breadth of promised benefits.

In the recent whitepaper “Survey Report – Project Success & Test Outsourcing”, it seemed that responses generally supported the feeling that the software industry has advanced in the last 10 years with respect to its attitude towards project success, customer satisfaction and quality, and including outsourcing of testing as part of the solution to that success.  But it was still clear that some organizations have the opportunity to see significant improvement benefits by reviewing how they are structured internally and how they conduct their quality and test efforts whether there is the intention to outsource or not.

In the following presentation, the software testing function of an organization is used to discuss crafting a multi-faceted sourcing solution model aimed at providing benefits on a combination of fronts including cost efficiency, capacity scalability, optimized communication, domain expertise capture/retention, specialized skills access, centralized strategic planning and accountability, etc.

Perhaps there is such an opportunity to identify and organize the teams, their activities and processes within your organization to realize such benefits as well, enabling a greater capability for overall project success.

Read the full article…

As testing is virtually always constrained with not enough time, not enough people, not enough infrastructure, etc it is vital that the testing effort is able to prioritize what is critical and what is not so the scarce resources can be applied in the most valuable manner possible.

New functionality is where the excitement is, and definitely where there is a high probability of defects.  And in the face of tight budgets and schedules, even new functionality testing is often short-changed.

But is it really the only place of high risk? How many times has the code been changed and something seemed to break somewhere else?  What if the impact is a data integrity issue?  What if the collateral damage is deep into a different business process?  What if performance of key tasks degrades after making a “small” change in a supposedly unrelated area of the system?

Regression testing is intended to uncover such errors in areas of the system that were previously considered to be working by re-testing the system after changes have been made (eg: defect fixes or new functionality).

Challenges

Regression testing is typically a challenge on any project because of ever-present time and resource constraints.  However the neglect of regression testing builds a potential for undetected failure into the system.  This potential typically only increases as time passes.

The following are some more specific reasons why regression testing is often left behind in the project plan.

Size of system grows while projects remain similarly constrained:

• The potential size of the regression suite is increasing over time as the system grows in terms of functionality and complexity
• As each new release typically faces the same constraints in terms of budget, schedule, team size a smaller and smaller percent of prior functionality gets re-tested because of the focus on new functionality and the growing size of the regression suite

Complexity of system grows while testing practices remain static:

• The inherent complexity of the system is increasing as more and more code is written for new features and the interactions of the system increase
• The system-level dependencies and interactions become less and less understood on average by each individual team member
• System-level knowledge of the system and how to test it typically remains uncaptured as investment in appropriately formalized tests is haphazard or unsustained, resulting in insufficient test coverage of the system or the inability to even gauge the degree of actual coverage

Automation efforts are prone to failure:

• Investments into test automation efforts for regression testing are prone to failure over time because of perceived maintenance overheads,  lack of an ROI-driven approach, lack of the right-skilled resources, lack of the understanding of what exactly is being tested, etc
• Rapid development cycles do not traditionally encourage/allow for the time to create well-designed test automation suites (eg: effort is perceived as slowing the ability to release the current new functionality, or an expense on the project when there will be little direct benefit to the project)

Solution

One of the crucial components to being “smart” in a constrained situation is to be able to correctly decide on or select which things are “must-haves”, which are “should-haves” and which are “nice-to-haves”.

For any testing effort on a maturing system, a regression testing approach is definitely a “must-have” in order to address that risk of collateral damage or service degradation.  At the same time, you should not expect to have to rerun all the tests you did when the functionality was new.  Thoughtfully sample the system functionality and associated tests for areas where a regression would matter most.  If/when such issues are found deeper testing can take place at that point.

In order to define what a reasonable sample of the tests should be:

• Identify traditionally risky areas in terms of criticality, sensitivity, complexity, interdependencies, and instability
• Determine the “breadth and depth of testing” required based on both traditionally risky areas and where the changes for this next release are planned
• Capture the strategic regression tests from the business/acceptance, technical/functional, ‘ilities/non-functional testing points of view in a formalized manner

If you already have comprehensive, well-defined, documented tests, then deliberately de-select less strategic tests that you used when testing new functionality.  Your pared-down regression test suite will help you mitigate the risk of missing things while you continue to put focus on the new development in your testing effort.

It may feel a bit painful at first to “throw-away” some of those tests, but in truth you are really just setting them aside and focusing on the high priority tests that make the most business sense to re-execute on a regular basis.

Strategically Keeping Up

Even with thoughtful consideration of the tests for the regression suite, the number of tests and the time it can take to execute them can be significant.  To help reduce the time for execution as well as reduce maintenance of test data and set-up/configuration costs, we can turn to test automation.

“Many people try to add automation to their projects, only to end up frustrated and annoyed. After one or two disastrous attempts, many just give up and stop trying. However, implementing automated testing is a basic cost-benefit analysis.”

“Automation is a method of testing, not a type of testing, and so should be applied to facilitate or accelerate those tests in the overall plan where there is a clear benefit for the project or the maintenance of the final system.”

- Potholes on the Road to Automation

The regression test suite should be automated where the business case of doing so makes sense. To help determine the actual ROI for this effort, be sure to take a system lifecycle view of the investment (not just a current project view):

• Automate where the business case is strong (eg: repetitive, difficult to do manually, data volume and/or set-up intensive, permutation volume)
• Selective creation of automated scripts based on high-use and high-risk functionality also helps maximizes ROI
• Use the previous build to automate tests to allow development of the scripts on a “stable” system and minimize rework
• Manually test the new functionality on a current build to actively remove defects and immediately add value while simultaneously identifying candidate automation scripts, data requirements, and dependencies for the next automation cycle

Summary

Is the risk tolerance of your project and organization so high that you can really afford not to have an efficient regression testing solution?

Reduce the level of risk in your system and organization by including a regression testing solution as part of your overall test strategy for any significant release.  Design your regression test suite in such a way that you are able to perform different levels of regression testing while also reducing resource and turnaround requirements.

If you carefully feed and care for your regression test suite you will reap the benefit of continually increasing confidence in the stability and quality of your system through maximized test coverage and ease of execution within your constraints.

The intention of the survey was to use the responses to paint a picture of the attitude or approach towards outsourcing of testing in comparison to how organizations seek success for their projects and teams.

In general it was hoped that the responses might indicate that:

• Organizations were looking beyond hourly rates to the total value that was being delivered by contract service providers (eg: productivity, increased capability/capacity, organizational planning and execution improvements, bottom-line impact).
• Organizations were seeking solutions that: were more than just bodies to add to a project team; provided strategic thought leadership rather than just execution on assigned tasks; built a relationship with a partner specialized in delivering specific value-adds.
• Organizations were open to looking beyond the confines of their own offices to find much needed support such that not all of the team members needed to be co-located in order to be effective.
• Organizations were extracting the benefit of up-front thinking about risks and constraint trade-offs by investing into test strategy and effort estimation planning activities.

Not all of these expectations were fully supported in the responses, suggesting that there is still significant opportunity for improvement benefits. However the responses provided a picture of a stronger focus on quality and testing activities than would have been expected 10 years ago.

Read the full article…

In general, I have found that a significant degree of value can be added to an organization’s software development process and bottom-line through the application of the expressed intentions of the Agile approach.

Consider:

Principles behind the Agile Manifesto
We follow these principles:
Our highest priority is to satisfy the customer
through early and continuous delivery
of valuable software.
…
http://agilemanifesto.org/principles.html

If one understands Agile as not a methodology or process but as a philosophical approach to trying to do better at something that has traditionally not worked as well as it could, substantial opportunity exists for improvement:

“This year’s results show a marked decrease in project success rates, with 32% of all projects succeeding which are delivered on time, on budget, with required features and functions, 44% were challenged which are late, over budget, and/or with less than the required features and functions and 24% failed which are cancelled prior to completion or delivered and never used.”, Jim Johnson, chairman of The Standish Group, CHAOS Summary 2009 – http://standishgroup.com/newsroom/chaos_2009.php

This is not to say that using Agile will overcome all the challenges inherent in your development methodology, but rather that using each of the principles included in the Agile Manifesto to change your approach to planning and executing projects can get you significant “Quick Wins”.

For example with the principle: “…satisfy the customer through early and continuous delivery of valuable software”:

• Organize your project with 5 or more ‘releases’ to the customer for acceptance, rather than just the typical final release at the end.
• Distribute your effort and functionality evenly across the releases being sure to be able to show useful, increasing value with each release.
• Have the customer sign-off on each release and ideally start using it in their business activities in order to start realizing the benefits of the investment as soon as possible.

This approach can bring all sorts of benefits in terms of lowering the Total Cost of Quality, but the one key performance indicator we shall make an observation on is the aspect of customer satisfaction.

By the time the Final Release is given to the customer for acceptance, the business should have already accepted and be ramped up on the previous releases. Therefore the project team already has an 80% acceptance rating going into this last cycle.

It may be a little light-hearted to look at this outcome as similar to entering the final exam with an “A” already guaranteed, but fundamentally the team has reached the end of the project proving with each release that they are in synch with the customer’s needs and are delivering the solution that fits that need.

Good grades aren’t everything when looking for a successful project, but without a happy customer…

Often in discussing the value of estimating test efforts, a comment is made by someone that there is no point in doing so as they will be given only 2 weeks (or some other arbitrary number) for testing anyway.

In such a tightly constrained situation, it will frequently feel that there is no ability for the test team to control or even influence the situation – it is clear to everyone on the team that there is not enough time to do the job, but that is all the time there is, and so…

It is the next thought after that which is critical: so… what do you do? What is your plan or strategy of testing? And how do you know that this approach is the most effective compared to another? How do you make it visible to the stakeholders and senior management what they will get for the 2 weeks (and what they won’t get)?

This is where estimation can help by allowing the rapid generation and comparison of different efforts, by allowing the effort required for a given test approach to be quantified, by translating a schedule constraint of working days into a budget of person days and vice versa, by revealing opportunities to get testing involved earlier in the SDLC where they won’t be on the critical path, etc.

Just because you are tightly constrained does not mean that you can not take control and ownership of the problem, think it through, come up with the best solution approach possible for the situation, and succeed together as a team.

The ability to give early feedback during testing is a cornerstone to providing value on a software project. A well designed smoke test is one part of a strategy that can enable that feedback.

How do you determine what to put in your smoke test? First, ask what are the critical behaviours that need to be working. Then consider the types of changes that have occurred since the last test cycle and what you need to check to gain necessary confidence that these things and what they impact are in “working order”. Finally, prioritize which other areas of the system you need to review in more detail than others (eg: what would you do next if you had some extra time).

In general, you should plan to execute the “breadth” of “breadth and depth” testing in a smoke test, reporting readiness and severe issues continuously as you go. For example:

• The first pass of the system should make sure that all the things that you want to test in this cycle are in fact testable (ie: can be accessed and can perform at least one ‘happy path’ behaviour or task). The time to complete this first pass should take you and/or your team no more than 30 minutes – half of that would be better.
• The second pass should be a more detailed investigation into any new functionality. Make sure that whatever has been promised to the customer or end-user in this release has been evaluated such that you can provide early feedback on whether features are working well or not (don’t want to find those issues late in the test cycle).
• The third pass would include regression tests of the high priority behaviours that you know were working last time; things that cannot be allowed to break from release to release.

Successive passes, outside of the smoke test, would then go deeper into riskier areas; areas that have had a lot of change, are relatively more complex, are business critical ($$$), have been problem areas in the past, or haven’t been tested in “depth” yet.

Testing would continue to delve deeper following these priorities, guided by the Test Strategy, as time/budget and resources permit — it is very unlikely you will have all the time you feel you need.

Note: Do not constrain yourself to documented detailed test cases in your smoke test. Utilize checklists, matrices, use cases, user stories, etc.

Make sure to customize the above outline to fit with the unique balance of quality requirements, scope, and constraints you need for your project.

Silverpath works with our customers’ in-house project teams to integrate our adaptable approach of Thinking Through Testing™ which emphasizes the inclusion of testing activities early in the project lifecycle.

In the recorded presentation linked below we will review some of the background to the value of this approach and the base principles of the approach itself.

Each of the aforementioned items is related to the organization’s perceived value of the testing group. Not only must the testing group be concerned with knowing and performing the activities of software testing to a high standard, but it must also understand how it should participate within the larger activities and priorities of the organization, making visible the value the testing group delivers in that regard.

Testing is typically a substantial portion of any software project, but it is still often an afterthought in many organizations where the testing group is an unequal stakeholder with respect to budget, resources and project planning.

While it is true that the degree of quality doesn’t directly impact market share, the lack of quality, competitively, will certainly have an adverse impact on reputation and revenues.

Investing in testing is expected to help to increase product quality, increase customer satisfaction, and improve the overall organizational profitability. Testing is part of a set of risk mitigation activities where the basic principle is to spend money wisely now in order to avoid greater costs later. Each organization and even each project must find the optimal mix of investment in upfront activities versus potential costs of inefficiency or failures later to minimize the Total Cost of Quality.

Perhaps the most important goal for a testing group within any organization is to move from the perception of being a cost center to being an effective and efficient cost-optimization center. This goal can be readily achieved through making visible the impact and value of the risk mitigation efforts of the testing group to the key stakeholders of each project and across the organization for review and improvement.

Read the full article…