Test Case, Test Suite, Test Run, What’s the Difference?

Summary: This article explains common testing terminology, including test cases (instructions to validate software functionality), test suites (grouping of test cases for organization), and test runs (determining which test cases to run by users at specific times), to help newcomers in the field of testing understand key concepts.

If you’re a newcomer in the world of testing, you easily become overwhelmed by all the testing terminology getting thrown around. For example, what is the difference between automated and manual testing? And what is a test script vs test case? In this article, we’ll try to explain some of the jargon you might encounter in your daily testing life, just to make it all a little bit more understandable. We’ll also provide a test plan example to demonstrate how these concepts come together. Let’s jump in!

Test Cases

So, we’ll start with test cases. Most importantly, they provide a series of instructions that validates some part of the software is doing its job as expected.

Say, for example, you require a contact form on your website that sends an e-mail when a visitor leaves a message, you’ll probably create a test case verifying the e-mail actually arrives when the form is filled and submitted. This will help you determine whether or not the software meets your requirements.

In some situations, a single test case can verify a simple requirement. However, it is more common practice to think of multiple situations that test your features more thoroughly. When your contact form offers an optional text field, try both filling and clearing the field in two separate cases. Even better: try to go beyond the “happy path” and think of things the software shouldn’t do. If your contact form accepts an e-mail address, why not check if a non-valid address is accepted too?

Test cases do not merely exist out of instructions – or test steps, as we’d like to call them. For example: simply performing a few steps without explicitly telling the expected outcome could cause misunderstandings. Should our contact form reject an invalid e-mail address? Perhaps – or perhaps, we really don’t care. A proper test case usually comes with a predefined expected result, telling you what to expect when completing it. Similarly, it is common good practice to define the initial expected state when writing a test case. You don’t want to cover every piece of preparation in your test steps, like accessing a test environment, setting up a browser, enabling mail notifications, etc. These pre-conditions are usually defined as a separate property of the test case. When a condition isn’t met, there’s no point in testing.

So, that’s it then? Well, there’s more. Like writing a blog post, creating test cases require certain writing skills, like imagining things from the end-user’s perspective. If you want to read more, be sure to read 7 tips for great test cases.

Test Suites

When you know about test cases, the concept of test suites should be really simple. With a growing amount of test cases, the need for categorizing them increases as well. You don’t want to throw all your books on a pile, you’ll organize them on shelves, for better accessibility. In a way, test suites are very similar to shelves, as they group test cases together.

In the process of planning and running tests, keeping track of hundreds of cases really becomes cumbersome. Thinking in terms of “We need to run a performance test tomorrow” or “Next week we should schedule a test for our website” forces you to know the purpose and domain of each test case. 

A test suite allows you to categorize test cases in such a way that they match your planning and analysis needs. Do you run functional and performance tests? Create two suites and label them accordingly. Do you test different applications and want to monitor them? Create a suite for every application. Are you still able to manage without test suites? Don’t use them! Remember, they are only there to help you organize when needed.

Test Runs

Finally, we end up with test runs. They are little crossroads in your manual testing project, where several aspects come together. Simply put, test runs determine which test cases are tested by which user(s) at what particular time. Quite a mouthful!

Imagine you have created your test cases for your contact form and its about to become available to your organization for testing. Are you going to test it for yourself, or do you pick someone else? When are you going to start testing – and, when do you want to gather the results? Which cases are you going to select – all of them, or just the ones marked “happy flow”? These questions should be answered when planning a test run.

When you have created a test run, you have reached an important milestone: people are able to run a test, using your test cases, grouped by your test suites. 

Bringing It All Together With Test Plans

A test plan, often created by a test manager, defines the test suites, test cases, and test runs, including how they will be organized, executed and managed within the project. But that’s not all. A comprehensive test plan includes the complete testing strategy, objectives, schedule, estimation, deliverables, and resources required to conduct the testing. 

Consider the test plan to be like a blueprint for a construction project. In the same way a blueprint provides detailed guidelines on how to build a house or a building, a test plan offers a detailed outline on how to test a software product. Perhaps the contact form you are testing is just one small part of a larger website testing project. Having a detailed plan helps test managers, testers and even stakeholders align around the same goals.

Software Test Plan Example

Here’s what a complete software test plan often looks like:

1. Introduction
   1. Purpose
   2. Scope
   3. Objectives
   4. Document Conventions
   5. References
      
      
2. Test Strategy
   1. Testing Approach
   2. Test Environment
   3. Test Schedule
   4. Test Deliverables
   5. Test Roles and Responsibilities
      
      
3. Test Design
   1. Test Suites
   2. Test Cases
   3. Test Data
   4. Test Dependencies
      
      
4. Test Planning
   1. Test Milestones
   2. TestRun Planning
      
      
5. Test Execution
   1. Test Execution Process
   2. Test Environment Setup
   3. Test Case Execution
   4. Test Result Tracking and Issue Reporting
   5. Test Status Reporting
      
      
6. Test Completion Criteria
   1. Exit Criteria
   2. Suspension and Resumption Criteria
   3. Acceptance Criteria
   4. Approval Criteria
      
      
7. Test Risks and Contingencies
   1. Identified Risks
   2. Mitigation Strategies
   3. Contingency Plans
      
      
8. Test Sign-Off
   1. UAT Sign-Off
   2. Stakeholder Sign-Off
   3. Test Closure Activities
      
      
9. Appendices
   1. Test Plan Revision History
   2. Glossary of Terms
   3. Supporting Documents

Test Case Software and More—Start Testing with TestMonitor

Not a TestMonitor user just yet? Good and easy testing is a key priority to quality assurance managers, test managers,  IT-managers, test managers and the release manager. TestMonitor makes testing easy and fun for test users as well.

So let's get you started. You might want to check out our video or download the product leaflet. But of course, with a free trial you can really experience the ease of use of TestMonitor yourself.  

 

 

Stay in touch? Follow TestMonitor on Twitter and LinkedIn.