Testing by Measuring


At Zemanta we strive to have our code 100% covered by unittests. While unittests are important for achieving the desired quality of service, they are even more important as a safety net for programmers. Incrementally developing a complex service, such as Zemanta's, requires jumping between parts of the system and doing piecemeal code changes. Quite often you are required to change code not written by you and therefore it would be illusional to require from the developer to fully understand the system that he's trying to bugfix or augment with new functionality. The main role of unittests is therefore to alert the programmer that the changes he's implementing have side effects outside of his immediate focus. While unittests are great for testing small units of functionalities (hence their name), they are not suitable for testing whether the overall system functions as expectedly. A perfect unittests has all its dependencies to outside systems mocked, but no mock is equal to the real system. We are currently dealing with the question how to effectively test our systems as a whole, that is, how to do functional tests. One way in which we have tried to approach this problem is to use unittesting framework also for system tests, that is, we have written a separate set of unittests that do not mock external dependencies, but instead connect to production or staging services. The main problem I see with this approach is that functional tests are notoriously brittle and require constant maintenance. If you include constantly failing unittests in your deployment procedure, you're causing more harm than there are benefits.

At Zemanta we are measuring most of the aspects of the performance of our system in order to be alerted if something goes wrong. In recent projects we are specifying what we need to measure along with other aspects of functional specifications. I've noticed that with the correct measurements in place, we are able to detect and fix defects very quickly. This effect made me think, that instead of spending our effort on devising system tests, we should spend more time on instrumentalizing our code with effective measurements and nice charts for our Operations guys. Real-time measures are not only much easier to maintain, but are also testing the operation of the system continuously and not just at fixed points, as is the case with system tests.