The Future of Test Case Writing: Is ChatGPT Changing the Game?

Summary: This article discusses the impact of ChatGPT, an AI-driven chatbot, on software testing, focusing on how quality assurance (QA) professionals can utilize ChatGPT to generate answers, create code, and draft test case structures while highlighting the limitations of the technology.

ChatGPT and its impact seem to have come out of nowhere. Now, you can’t look anywhere without hearing about this powerful, dynamic, AI-driven chatbot developed by OpenAI.

From earning passing grades on Ivy-league MBA papers to crafting websites in seconds, ChatGPT’s impact isn’t around the corner—it’s already here.

With just a simple prompt, question, or series of criteria, users can generate seemingly endless outputs with surprising complexity. Naturally, as software developers and quality assurance (QA) professionals, we want to know what this innovative new technology means for our work.

Here’s TestMonitor’s take:

The Intersection Between ChatGPT and Software Testing

The “why” behind the quality assurance process that flows through the software development lifecycle will always be the same: Ensuring that software delivered to customers is fit-for-purpose, secure, and meets customer requirements.

However, the “how” has evolved. Testing has grown with technology, from creating and tracking test cases in spreadsheets and manually drafting testing documentation to leveraging test case templates and utilizing automated testing. 

In other words, it’s natural for the software development community to be curious about the impact that ChatGPT will have on the trajectory of software testing. 

Put simply, using the platform’s natural language processing and algorithms, QA professionals can use ChatGPT to:

• Generate answers to questions based on information from its vast database.
• Create properly formatted and logical code in several languages.
• Draft a rough test case structure based on best practices and user-defined criteria.

At the same time, ChatGPT is open about the platform’s limitations and displays them on its homepage. For example, ChatGPT’s processor relies on a database compiled with data through 2021 and can have trouble handling math calculations beyond simple functions.

Exploring ChatGPT for Test Case Writing

So what do all these factors mean for software testers? 

Ultimately, if QA professionals know the bounds of ChatGPT, it can be great for:

• Creating a starting point from which to build more specific test cases.
• Generating an array of generic test cases based on best practices.
• Creating testing-related documentation and sources based on user-defined input and specifications in various languages.

Here’s just an example of what ChatGPT can produce when asked to “Create a test case for a sign-up form”:

And here is the output from ChatGPT when prompted with "Create a test case for adding a new purchase order":

Although not perfect, the output from ChatGPT is a great starting point. The output includes the main elements of a test case, provides specific steps, and is written in straightforward language.

Where does ChatGPT test case writing have room to grow? 

Given the foundation from which ChatGPT was created, there will be limitations in using it for software test case writing. Some of the more prominent include:

The system struggles to create tailored test cases based on unique software designs.

ChatGPT creates outputs based on already existing content. In other words, it cannot create new content. ChatGPT will rely on test case best practices and information it has been exposed to created before 2022 to generate results. If the function is one of a kind, ChatGPT may struggle to create a test case to meet your needs.

Subjective or customer-experience-based test cases cannot be generated.

Like automated testing, ChatGPT cannot handle the subjective, emotional, or aesthetic elements of software testing. Software testers and potential end users will still need to evaluate how well a design meets the stated requirements across platforms.

There have been incidents of errors within the generated content.

Because of documented grammatical, logical, and numerical errors, the output generated by ChatGPT will still need to be reviewed to determine if it meets requirements and is easy to follow.

Users need to have an understanding of how ChatGPT works.

Software testers will need to understand how and what can be used to prompt ChatGPT. For example, ChatGPT is not designed to handle specific questions about the security requirements of a design element and cannot render code to test for bugs.

Bringing It All Together

There’s no denying the power of ChatGPT and its natural language model. Once you recognize the potential for AI to redefine many facets of your personal and professional life, there’s no looking back. 

However, the limitations of the technology will still require experienced QA professionals and developers to check its work, understand the subjective elements of design, utilize test management tools to manage the overall process, and deliver real-time, unique insights that only humans can provide.

At the end of the day, the TestMonitor team doesn’t think ChatGPT—and other AI-enabled platforms—will replace the need for sound development and QA, but rather be another tool in the toolbox professionals can use to amplify their impact.

Want to learn more about the latest innovations in software testing and test management? Then make sure to subscribe to the TestMonitor blog here!