Minimum Measurable Change

Out of the three steps of the lean approach towards product development (build-measure-learn) the most difficult in my experience is the measure step. While selecting suitable metrics and instrumenting your product/service is not easy, it is the size of the change that is the most difficult to define. If the change is too big it requires lots of resources to be implemented which makes learning costly. On the other hand, if the change isn't big enough the effect of the change won't be observable due to all other factors influencing behavior of users using your product/service. For example, we have updated the banner yesterday for our Related Posts plugin in WordPress directory. This is the old version

and this is the new version

While the difference between the two versions is obvious, I will be very surprised to see a noticeable difference in number of downloads, which is the metric of our choice in this case. Namely, the number of downloads depends on a lot of different factors with most of them beyond our control.

If we would have more control over WordPress directory we might have found a metric that would be sensitive enough to detect this change. But with the only available metric available being number of downloads, we decided to do a change of banner based on our gut feeling alone. The situation like this happens awfully a lot of times to us. I'm leaning more and more towards the opinion, that while we should definitely measure influence of the major changes in the direction of our product, we should do smaller changes based on our instinct and experience without banging our heads against the wall trying to measure what's immeasurable.

Maybe we should stop using the term Minimum Viable Product and start using Minimum Measurable Change instead.

Enhanced by Zemanta