Despite decades talking about ‘big data’ strategies, just 1 in 7 companies is excelling at delivering on those strategies, according to new research that also found many struggle to embrace machine learning (ML) because they can’t manage the data it requires.

High-achieving companies are delivering measurable business results thanks to their investment in “the foundations of sound data management and architecture”, an MIT Technology Review Insights-Databricks survey of 351 technology executives found.

Half of executives confessed to looking at completely new data-management platforms to keep up, while 43 per cent said their new data management platforms will be based in the cloud.

Cloud platforms were delivering at least half of data services in 74 per cent of the highest-achieving companies, which were embracing open-source standards and open data formats to build extensible data platforms.

Others spoke of the importance of “democratising” data to maximise the value of ML across the organisation.

Data proved critical in helping Coles stay ahead of rapidly-changing customer demand throughout the COVID-19 pandemic, chief marketing officer Lisa Ronson told a recent Melbourne Business School analytics conference.

After years of investing in data collection and analytics platforms to analyse customer behaviour that used to change “over the course of a month, quarter or year,” Ronson said, rapid pandemic-era changes meant customer needs were “changing from morning to night, and day to day”.

Executives “had to be really focused on all of our different data sets” and were often meeting several times per day to stay ahead of the changing data, she added, noting that the involvement of business leaders was critical to ensure the company could avoid decision paralysis.

“My research and insights teams had a lot of data at their disposal during this time,” Ronson explained, “but what was absolutely critical – when you’re swimming in data in an uncertain time and making decisions so quickly – was putting insights into themes or buckets that really matter, that we could then action our decision-making on.”

Data is only the beginning

Just 13 per cent of MIT study participants said they were excelling at delivering on data strategies marked by reduced data duplication, easy access to relevant data, the ability to process large amounts of data at high speeds, and improved data quality.

Cloud platforms were helping rework data models for ML – which was the top priority for 53 per cent of high achievers but remains “exceedingly complex for many organisations”.

Fully 39 per cent of respondents confessed to struggling to get enough ML skills, with a similar percentage reporting problems when data analyses were handed off from data-science teams to production teams.

Such interruptions “suggest severe difficulties in making collaboration between ML, data, and business-user teams a reality,” the study found.

A more consistent, open data infrastructure, its authors noted, requires “a strong data culture…. [where] the right users have access to the right data to quickly generate insights that drive business value.”

A robust data culture has helped contextualise the raw outputs of analytics systems at massive data consumers like eBay, where hundreds of millions of daily customer interactions generate oceans of data.

“Having all the data in the world is great but customer intuition, and business intuition, helps you thread through and navigate where to go looking in the data,” eBay senior vice president and general manager for the Americas market, Jordan Sweetnam, explained.

Yet while analysing customer feedback data may help a company add a few net promoter score (NPS) points, he warned, dramatically increasing NPS scores requires new ways of thinking.

Data can help a company “understand your flow, optimise, and take out friction,” Sweetnam explained, “but you’re not going to be able to optimise yourself to a winning value proposition.”

“That requires leaps of faith and transformative vision [and] a bit of leaning into the void, where the data isn’t going to show you how to get there – but once you’ve built it, the data can validate your instinct.”

eBay analysts regularly partner with business and technology teams for “a lot of exploratory analytics,” Sweetnam explained, “to take things that might be unexplained by standard dashboards, then go figure out why something happened.”

Ultimately, he added, business agility is key: “Leaders need to give teams permission to fail in a truly risk-free environment where they’re not just worried about making the next 1 per cent [NPS] uplift.”

“Eight things they test may be completely irrelevant, but then you will discover that magic that really changes the outcome. Let them swing for the fences and have some fun.”