Integration Tests are Useless


A complex web service such as Zemanta's is composed of many components that are connected in intricate ways. When service is still small and composed of just a few components, it feels most natural to set up the whole system locally and then test changes in your development environment. As system grows, become more complex, and involves increasingly more developers setting local copy of complete system becomes a non-trivial and time-consuming endeavor. It is at this moment in time that people start thinking about setting up staging servers to which developers could connect their local components. Programmers are almost by definition control freaks and for some time a staging environment gives a good illusion of control. But as things mature and grow even more complex it takes increasingly more effort to keep staging environment in sync with production environment. At some point people just give up and they start deleting integration tests, finally realizing that integration tests are utterly useless for web applications. Instead of wasting time with integration tests I'd suggest everybody to implement monitoring from day one and use monitoring to validate the proper function of the system. In many regards monitoring can be though of as continuous integration testing where testing happens on live production servers. The deployment procedure thus skips staging environment altogether and installs new version of software directly to a subset of production servers, followed by a full roll-out if all the monitoring parameters are within normal limits.

Most of successful start-ups have such systems in place, but to the best of my knowledge these are all in-house solutions. I wonder if there exists some ready-made framework that would handle monitoring and staged roll-outs for us. Let me know in the comments, if you know of such solution yourself.

Enhanced by Zemanta