[This article is written by our Guest Editor Dr. Wenzhe Zhou, Co-founder of Brainpage, which is a big data processing & analysis start-up, focused on cloud-based engine for time series data and sensors. The company’s Hadoop-based service provides developers and industry partners easy-to-use, scalable and flexible database and analysis solutions.]

The world’s information is growing at an incredibly rate. In 2011, 1.8 Zettabytes of data was created – enough to fill trillions of CDs that would stretch beyond the moon. But that’s just the start. Fueled by increasingly diverse sources of rich data such as social networks and “tweets”, mobile devices, a plethora of public and privates sensors, the amount of data collected is expected to grow 50% per year.

More data from more sources opens up more opportunities. You can use the 12TB of tweets generated each day to market products more precisely; analyze thousands of billions of smart meter readings to predict energy use and increase saving; or look into your business performance data to understand failure and increase margin.

The irony is this: while the field is now called “Big Data”, the real value is actually in extracting the micro or individual. Before, a store could only target their promotions en-mass; now they can do it individually. Before, an energy company could only tune their network regionally; now they can do it on a per-circuit level. Before, doctors could only analyze diseases across a population; now they can do it on a per individual genome level. And so on.

In this article we are going to focus on this macro-to-micro value proposition as it relates to businesses understanding their customers better. This is both one of the better-understood use cases of Big Data as well as one that many enterprises are failing to take advantage of. According to Gartner’s analysis, by 2015 enterprises who take advantage of new data will outperform their peers by 20%.

So what does “understanding customers better” really mean? For businesses, gaining customer insight can potentially help in three ways: building existing customer loyalty, gaining new customers, and create new products or business models.

Customer Retention

Earlier this year, a man stormed into his local Target store (Target is a major “big box” retailer, like WalMart, in the U.S.).

“How dare you!”, the man yelled at the store manager, “You sent coupons for baby diapers and strollers to my 17 year old daughter. She’s only 17!” The store manager apologized and, unaware of the company’s big data systems, explained it must have been a mistake. A month later the angry father called back to apologize. He was wrong and Target was right. His daughter was pregnant.

We start with this story – a true example of (attempted) customer retention and up-sell. Customer acquisition often receives more attention and investment than customer retention; however, the 80/20 rule cannot be ignored: 80% of a company’s future profit actually comes from 20% of existing customers. By mining existing customers’ usage patterns (i.e. when someone uses their clubcard, or when someone uses their bank card), smart retailers are maximizing their marketing campaign, supply chain, and upsell efficiency to huge results.

Using advanced statistical methods, smart retailers are creating models that predict what a customer will buy from what they previously bought. By knowing what individual customers will want to buy in the future, these companies can entice consumers to buy from them as opposed to their competitors with targeted coupons and promotions. In our teen-pregnancy example, Target had created a correlation between the change in purchase behavior of the women and when they become pregnant. Indeed, Target claims that if a new parent begins buying baby products from them, they have captured that customer for years. A small amount of analysis and targetting can mean years of customer loyalty.

Tesco is another example. As the world’s second-largest retailer by profit (right after Wal-Marts), the British supermarket has had obtained great results through segmenting their customers and watching their shopping patterns. Using data from the Tesco Clubcard, Tesco will understand what “type” of consumer someone is (“fast-food junkie, family with school age kids, etc.). Segmentation allows marketing efficiencies. Mail or Email promotions can target what a consumer likes, in-store promotions can be tuned to who is in each individual retail location at a specific time and regional advertising can be more targeted. Indeed, this shopping partner recognition has saved Tesco £350 million a year on expensive blanket marketing campaigns since it’s introduced.

Of course, there is also Amazon. Perhaps no company is more famous for their uses of data than this online retailing giant. Anyone who has bought from Amazon is familiar with its “Customers who Bought X…also bought Y” feature. This simple feature is not only useful, but it is also immensely profitable; it is also extremely complex. Amazon doesn’t just look at simple buying correlations, but rather tracks everything a customer does on their site – how long they spend on a page, whether they look at the reviews and so on. They understand there is a simple win-win: if the recommendations are useful, the buyer is happy and Amazon makes more money.

It is this ability of turning data into gold that drove Amazon beyond its traditional shopping model. Amazon mobile apps, which offer users blended online/offline shopping experience, allow Amazon to better understand what each individual consumer likes. The Kindle Fire bring more user preference data onto Amazon’s servers as does their new Silk browser. To a smart company like Amazon, more data equals more sales.

Customer Acquisition

Companies are not just using data to retain and upsell customers, but also to acquire them. “Social” has opened a whole new field for finding customers and big data technologies are transforming how digital advertising works. Here are a few examples:

  • Social Feed Mining – Smart banks and airlines ingest the twitter tweets to recognize a potential customer who might be on the lookout for a new bank or a flight. These companies use natural language processing to look for tweets like, “Can anyone recommend a good home-loan?” or, “Cheapest flight to New York?”. By sending a friendly and targeted response they can achieve huge conversion rates and happy customers.
  • Ad Retargetting – A good sales person will tell you that persistence, or “following leads” is key. Yet advertising has traditionally been hit or miss: either blanket websites, TV or billboards with your ads or hope that one or two targeted ads (i.e. “Search Engine Marketing”) are enough to do the trick. A look at modern advertising metrics will tell you that neither method has particularly good conversion ratios.

However, using new data technologies, companies like Chango in the U.S. and Uniqlick in China are changing this in the digital domain. By understanding which internet users are most likely to respond to an advertiser’s value proposition – either by what that user has searched for or the sites they’ve visited – they can deliver targeted ads, continuously, to just the people who are most likely to respond. It’s like Google AdWords (search ads), except across the entire internet instead of just the search results page. The result is more useful banner and non-search advertisements, higher conversion rates for advertisers and more value created by the websites that provide the ad inventory space.

Create New Products and Business Models

Data is not just creating massive opportunities for optimization, but also opening entirely new business practices. In the energy sector, Opower, a Software-as-a-Service company, is using data to improve power efficiency. Partnering with utility providers, the company analyzes families’ power bills and provides a report that compares each family’s usage to their neighbors along with data-based recommendations for energy savings.. Opower’s service has covered millions of homes in the US and is estimated to help American energy consumer save $500USD each year.

In the healthcare sector, San Francisco-based SeeChange sees a better way of designing health insurance plans through understanding clients’ personal health records. The company analyzes a substantial amount of data obtained from personal health records, health claims databases and pharmacy data to identify patients with chronic illness who would benefit from a customized compliance program. In addition, it also offers cash incentive plans to help patients complete specific health action plans to meet clinical goals, with its data analytics engine monitoring the process. SeeChange now provides services for SMEs with employs between 2 and 200 people.

In the retails sector, a customer retention marketing platform Retention Science has launched to provide e-Commerce businesses with actionable data analytics tools to cost-effectively prevent customer churn and encourage reengagement. Its Customer Profiling Engine is a learning engine that uses algorithms and statistical modeling to build custom retention optimization strategies. Analytics are performed in real-time to ensure up-to-date customer behavior predictions, and at the same time, dynamically create offers based on customer unique characteristics. The company has received $1.3 million seed round comes from multiple sources including Baroda Ventures, Mohr Davidow Ventures, Double M Partners and several prominent angels.

Whether it’s optimizing existing opportunities or finding entirely new opportunities, big data and new data technologies opens up an unprecedented opportunity to create experiences targeted at the individual. With near ubiquitous information technology being a part of every business in the 21st century, every business is generating data and should focus on collecting every piece of that they can. The difference between the winners and the losers will be those that can effectively store and mine their data for gold.