I've had a conversation recently with a friend who is considering trading coziness of an employment at a large public utility company with the thrills and frills of starting his own enterprise. He has a bunch of business ideas that he'd like to pursue and in order to evaluate them, he has asked me for the opinion about the trends that will define information technology in this decade. My answer to him was, that out of the three big trends present at the moment (mobile, cloud, big data), the big data is the safest bet of them all. We got interrupted at this point, so I didn't have a chance to elaborate on my opinion, so I'll justify my reasoning in this post instead. What is interesting in big data is not that you have gazillion gigabytes of data stored, but that a large amount of data enables you to make smart predictions, as testified by the last fifty years of research in artificial intelligence that has shown, that the best strategy how to do smart predictions is not devising smart algorithms but listening to data. In my opinion data-driven decision making enabled by the big data might become the main source of competitive advantage for companies in this and the next decade. Unfortunately, almost no established company is in position at the moment to benefit from the big data opportunity, since they don't have neither expertise nor processes in place that would enable them exploit big data to make smart predictions. In my opinion, there lies a tremendous opportunity at the moment for independent vendors to help established companies ride the big data trend.
In both working with and thinking about machine learning and statistics I am always amazed at the differences in perspective and view between these two fields. In caricature it boils down to: machine learning initiates expect to get rich and statistical initiates expect to get yelled at.
- Big Data Facing a Big Talent Shortage (semanticweb.com)