Principles of Testing

Chapter 1

Principles of Testing

1.1 OVERVIEW

In this module you will get an overview of the most fundamental principles of testing.

Firstly, we point out that although you will come across different terminology throughout your career, professional testers are all pretty much agreed on these basic ideas that we describe in this module.

Secondly, we take a look at the need for proper testing look at the cost of getting it wrong and we show why exhaustive testing is neither possible not practical. What are errors and how do they get into the software?

Thirdly, we describe a fundamental test process, base on industry standards, and underline importance of planning) tests and determining expected results in advance of test execution.

We conclude with a look at re-testing and regression testing; why it is so important to go that extra mile and run a suite of tests again, just when you thought it was safe to hit the release date.

1.2 OBJECTIVES

After completing this module you will:

» Understand basic testing terminology.

» Understand why testing is necessary.

» Be able to define error, fault and failure.

» Appreciate why errors occur and how costly they can be.

» Understand that you cannot test everything and that testing is therefore a risk management process.

» Understand the fundamental test process.

» Understand that developers and testers have different mindsets.

» Learn how to communicate effectively with both developers and testers.

» Find out why you cannot test your own work.

» Understand the need for regression testing.

» Understand the importance of specifying your expected results in advance.

» Understand how and why tests should be prioritized.

1.3 TESTING TERMINOLOGY

There is no generally accepted set of testing definitions used by the worldwide testing community. The British Standard for Software Component Testing published in August 1998 provides a new source of testing definitions. When involved in a testing project it is important to ensure everyone understands terminology adopted; you may find different terminology is used in your organization.

Exercise

The British Standard for Software Component Testing is known as BS___________

A useful glossary of terms used in software testing is called BS___________

Although it is a British Standard published by the BSI (British Standards Institute), the Specialist Interest Group in Software Testing (SIGIST) developed it over a period of several years.

ANSWERS: 7925-PART2, 7925-PART1

1.4 WHY IS TESTING NECESSARY?

This section explains why testing is necessary and closely at the cost and consequences of errors in computer software.

1.4.1 Definitions Errors, Faults and Failures

An error is a human action that produces an incorrect result. A fault is a manifestation of an error in software. Faults are also known colloquially as defaults or bugs. A fault, if encountered, may cause a fault, which is a deviation of the software from its existing delivery or service.

We can illustrate these points with the true story Mercury spacecraft. The computer program aboa spacecraft contained the following statement wri1 the FORTRAN programming language.

DO 100 i = 1.10

The programmer's intention was to execute a succeeding statements up to line 100 ten times then creating a loop where the integer variable I was using the loop counter, starting 1 and ending at 10.

Unfortunately, what this code actually does is writing variable i do to decimal value 1.1 and it does that once only. Therefore remaining code is executed once and not 10 times within the loop. As a result spacecraft went off course and mission was abort considerable cost!

The correct syntax for what the programmer intended is

DO 100 i =1,10

Exercise

What do you think was the error, fault and failure in this example?

The error is __________

The fault is ___________

The failure is __________

1.4.2 Reliability

Reliability is the probability that software will not cause the failure of a system for a specified time under specified conditions. Measures of reliability include MTBF (mean time between failure), MTTF (mean time to failure) as well as service level agreements and other mechanisms.

1.4.3 Errors and how they occur

Why do we make errors that cause faults in computer software leading to potential failure of our systems? Well, firstly we are all prone to making simple human errors. This is an unavoidable fact of life. However, this is compounded by the fact that we all operate under real world pressures such as tight deadlines, budget restrictions, conflicting priorities and so on.

1.4.4 Cost of errors

The cost of an error can vary from nothing at all to large amounts of money and even loss of life. The aborted Mercury mission was obviously very costly but surely this is just an isolated example. Or is it? There are hundreds of stories about failures of computer systems that have been attributed to errors in the software. A few examples are shown below:

A nuclear reactor was shut down because a single line of code was coded as X = Y instead of X=ABS (Y) i.e. the absolute value of Y irrespective of whether Y was positive or negative.

Blue Cross of Wisconsin installed a new