It’s all fun and games until you realize your database is wrong bc there are no tests—Kirsten Garrison
Did You Know?
Unit testing may save your life, and it can even be fun. Accountants “test” with double-entry accounting. Accountants that don’t do double-entry don’t last long in the profession. In fact, such accountants may not even exist. The programming profession is not like this, but it should be.
Kinds of Testing
First things first! Unit testing is not the only kind of testing. In fact the whole idea of testing has several dimensions.
How much of a system are you testing?
Unit Testing helps ensure individual components function as advertised. Must be extremely fast and never interact with filesystem, databases, networks, and so on—all external systems must be mocked.
Integration Testing helps ensure components can communicate with each other properly. Okay to interact with filesystem, databases, and networks.
System Testing (sometimes called Acceptance Testing) helps ensure a fully-built system has no defects from a user’s point of view. Usually done on a production-like environment. If automated, it will script or mock the GUI.
What are you testing?
Performance Testing checks to see if the system runs within specified temporal and spatial constraints.
Regression Testing checks to see if the latest fixes and enhancements broke something that used to work.
Stress Testing sees how the system holds up under excessive load, or when run in an environment without reasonable resources.
Do you care about implementation or just the functional results?
Black Box Testing looks only at inputs and outputs and makes sure the outputs are expected, caring nothing at all for what goes on in between.
Whitebox Testing “exercises” an implementation attempting to get complete “code coverage” (access to source code is needed for this).
Unit testing is crucial. It is done by developers, not QA. In many organizations, unit tests are required: programmers are not allowed to check-in code or build a system unless their unit tests pass. Please read:
The classic Test Infected paper (focuses completely on JUnit, a Java-based framework, but you should still be able to follow it if you don’t know Java)
How NOT to Unit Test
Sometimes one wonders if the world is divided into (1) practitioners that know how to unit test and (2) textbook writers that don’t know squat about it.
Some textbook writers (I won’t name them here) show you pathetic “test drivers” that go through a the code and print out results for humans to check.
This Ain’t It, Chief
Like someone’s gonna sit there are run the program a gazillion times during debugging and read the output off the screen?!?
The Right Way To Unit Test
Unit tests should be non-interactive. They are made up of a huge number of statements that exercise some code and compare the computed result with the expected result. (See: the computer checks the result, not a slow human that doesn't have the patience to type in all of tests each time a change is made to the code.) A test driver can report the number of successes and number of failures when it’s finished. The many, many, many reasons for unit testing and the benefits it provides can be found all over the web, but one advantage stands out in particular:
A COMPREHENSIVE UNIT TEST SUITE ALLOWS YOU TO REFACTOR WITH CONFIDENCE
Go ahead and make that change to improve your code! Your tests will protect you!
Unit Tests F.I.R.S.T.
Unit tests differ from other kinds of tests in that they must be F.I.R.S.T.
Fast: You may have thousands of these and you will run them constantly, so each test should take milliseconds at the most. No network connections! No database connections! Stay in memory. Mock external components.
Isolatable/Independent: Tests should never depend on each other. You must be able to run them in any order whatsoever.
Repeatable: Tests should run the same in any environment. They must not depend on external factors.
Self-Describing: Each test just returns a boolean result for pass or fail. You must never have to sift through output to find out whether it worked or not.
Timely: Write them before the code. No one likes to write tests for code that is already written: that is just boring and totally unsatisfying work.
When you do unit testing properly, you get some great advantages:
Confidence in refactoring
Totally awesome documentation!
Always in sync with the code (provided the tests are passing)
Feedback during design of the code
Read those advantages again.
It is hard to overstate how important each of those items really and truly is.
For more, read Uncle Bob’s Clean Code or watch the testing episodes (especially Episode 6) of Clean Coders.
What Do Tests Look Like?
Tests contain assertions about the behavior of your software.
For example, suppose we were asked to write a function to determine whether a value was a prime number. We want it to return true or false if the value is an integer in the range 2..1000000000000 inclusive, and throw an error message for all other inputs. Our test suite would make the following assertions:
2 should be prime
20 should not be prime
47 should be prime
933 should not be prime
1000000000000 should not be prime
Should throw on non-integer (e.g., 2.5)
Should throw on integers less than 2 (e.g., 1)
Should throw on negatives (e.g., -14)
Should throw if too large (e.g., 1000000000001)
For another example, let’s say we want to build a type for money bags. A money bag holds a certain amount of money in a variety of different currencies. Let’s specify the money bag behavior in language that is very, very close to code:
A newly created bag has nothing in it.
Calling the to-string method on a newly created bag produces [ ].
After inserting a currency with amount 0 to an empty bag, the bag is still empty.
Getting the amount of an arbitrary currencies on an empty bag return exactly zero.
After inserting a non-zero amount of a given currency to an empty bag,
there is only one currency type in the bag
to-string works as expected
Getting the amount of the currency type just added returns the value just added.
After adding an amount of a currency already present, the number of currency types is unchanged and getting that currency returns the new amount.
After adding an amount that makes an existing currency type become zero, the number of currency types decreases by one.
Specifying a non-numeric value for a value raises the desired exception.
Specifying an unknown currency type raises the desired exception.
This is not a complete list of behaviors. These are just examples of what your tests may look like.
Exercise: Write some more things to test.
Unit Testing Frameworks
It is possible to write a little application of your own which executes each of the tests and counts the number of successes and failures. But since the work of setting up and tearing down tests, counting successes and failures, and reporting and so on is the same for every test, it makes a lot of sense to use an existing test framework. What are some popular frameworks?
Writing good tests takes experience; you need to be an accomplished developer to figure out how to get good coverage
and develop an eye for boundary conditions. These are the “extremes” of the input domains, such as zero, empty, full, one more than full, minimum, maximum, just below minimum, just below maximum. Related are atypical inputs, Ridiculous large or small values, and values of the wrong type.
Large-scale applications are developed with test suites with hundreds or thousands of tests. The test suites are run periodically, often overnight.
Tests should be written before a unit is coded. The buzzwords are:
Test-driven development
Test-first programming
Test-infection
In some environments, developers are not allowed to check code into a revision control system until unit tests pass.
Exercise: What’s a boundary condition anyway? Do some research. Write a three-page paper about them. Give a bunch of examples.
Powerful stuff! This requires discipline, but it pays off. See what he is saying:
You are not allowed to write any production code unless it is to make a failing
unit test pass.
So start by writing tests.
Whenever you want to write production code, write a test for it first.
(This helps you understand what you are going to write.)
You are not allowed to write any more of a unit test than is sufficient
to fail; and compilation failures are failures.
So don’t get ahead of yourself with the tests.
You aren’t allowed to write super big tests. Just one little bit
of test behavior at a time. When you write a failing test and you see
it fail, switch to writing production code to make it pass.
You are not allowed to write any more production code than is sufficient
to pass the one failing unit test.
Again, don’t get ahead of the tests, ever.
In practice, you can do TDD with Red-Green-Refactor. Watch the
little clip here, but consider paying the 10 bucks to see the whole
episode: