Companies are currently living in continuous competition, under the pressure of a cyber war. They are subjected daily to a constant adaptation, for example: the cloud, mobile devices, machine learning and the Internet of Things (IoT), among many other innovations.

To move quickly, companies are struggling to implement innovation within their workforce. They have made great strides in their agile transformation, they have begun their journey to the cloud, they have implemented new DevOps practices, they have hired the best and brightest people to take advantage of the latest technology and processes. However, they remain disappointed by the results. In fact, 84% of companies still fail to achieve digital transformation, in part because they do not address one of the most critical aspects of the new digital economy: data.

And every company is now a software company, this market change, where products are more complex and require more features to meet the demand for personalized experiences, demands a sophisticated data strategy. Data and access to it are a competitive advantage: those who can leverage data to drive innovation will win.

The data, current situation

The main problem is not the data itself, but the concept known as data-friction, which occurs when data restrictions prevent people from meeting the growing demands of the business.

On the one hand, there is an explosion in data needs, users, and environments that requires data to be whereit it is needed, for anyone who needs it, in the way that is best suited to the task at hand.

On the other hand, data has grown exponentially in size, complexity, and cost with growing security and privacy concerns, meaning IT experts must limit and protect data access and availability.

As a result, companies are in volved in a battle between data-friction and people, processes and technology. For a while, businesses have advanced using the cloud and DevOps. The delivery of IT environments has gone from weeks to minutes, with an automated, elastic and on-demand infrastructure. But the data doesn’t look like calculations. Data handling is, expensive to maintain, full of sensitive information, difficult to copy, difficult to track over time, and slow to deliver to computers that need it.

Although the cloud and DevOps have ultimately helped are insufficient. As DevOps and the cloud broke down barriers between people and infrastructure, the emergence of more environments, more automation, and more speed meant increased data demand. IT experts are still struggling to manage, secure, and deliver the data environments demanded by the business. And users still have difficulty accessing, manipulating, and sharing the information they need.

According to research “Strength in Numbers: How Does Data-Driven Decision making Affect Firm Performance?” conducted by Erik Brynjolfsson and Heekyung Kim of MIT, along with Lorin M. Hitt of the University of Pennsylvania, companies that called themselves based on data were a 5% more productive and a 6% more profitable than their competitors.

DataOps

When data-friction becomes the blocker of innovation, customers leave, competitors win, and companies spend more time reacting rather than leading.

But that doesn’t have to happen, companies can win. IT can outweigh the cost, complexity, and risk to become a business enabler. Users can get the data they need to unlock their capacity for innovation. And everyone can work as a single team to generate massive results for the business. Addressing all of this requires a new approach, one that does for data what DevOps did for infrastructure, DataOps.

According to Gartner, DataOps is the center for data collection and distribution, with a mandate to provide controlled access to customer registration systems and marketing performance data, while protecting privacy, usage restrictions, and data integrity. It aims to improve results by bringing together those who need data with those who provide it, eliminating data-friction throughout the data lifecycle.

How does it work?

Mastering DataOps requires overcoming the organizational and cultural barriers that separate people from data. Start by joining two key audiences as one team:

  • Data operators: responsible for infrastructure, security and maintenance. Includes DBA, security and compliance, system administrators, and more.
  • Data consumers– Responsible for using data to drive new projects and innovation. Includes developers, testmen, data scientists, analysts and more.

But all the cultural transformation in the world will not help if your infrastructure cannot withstand the new demands that are put upon it. DataOps also requires a comprehensive technology approach that eliminates key points of friction in:

  • Governance:security, quality and data integrity, including audit and access controls.
  • Operation: Scalability, availability, monitoring, recovery and reliability of data systems.
  • Delivery: Distribution and provisioning of data environments.
  • Transformation– Modifying data, including platform masking and migration.
  • Versioncontrol– Capture data as it changes over time, with the ability to access, publish, and share states between users and environments.

To make data work, executive management mandate is required for democratized data access, a centralized data infrastructure, data analysts/scientists, and data equipment.

DataOps enables data-driven enterprises

Ashish Thusoo, offered a more pragmatic definition:

“DataOps is a new way to manage data that promotes communication and integration of previously isolated data, teams, and systems, leverages process change, organizational realignment, and technology to facilitate relationships between all those who handle the data, whether it’s developers, data engineers, data scientists, analysts and/or business users. DataOps closely connects the people who collect and prepare the data, those who analyze the data, and those who use the findings of those analyses for commercial good use.”

Thusoo’s approach to data and a data-driven culture consists of a team that publishes data and manages the infrastructure used to publish this information, and those in charge of making business decisions that typically have scientists analysts on their teams, to back up.

In the Thusoo model, data scientists or data analysts are integrated into business units such as finance, sales, marketing, etc. They work with business leaders to identify questions, identify the datasets to be analyzed, and then translate them into SQL (structured query language) or a more sophisticated language. The work is then delivered to the data team.

Other DataOps models, such as the one offered by Ellen Friedman and Ted Dunning, revolve around “organizing teams around data-related goals for faster time.” They suggest that DataOps team members may come from product operations, software engineering, architecture and planning, data science, data engineering, and product management.

Unlike Thusoo, Dunning and Friedman noted that the infrastructure capabilities around the platform and data network (needs affecting all projects) tend to be supported by DataOps teams by dataOps organizations Support.

Possible failures

The result that is sought after adopting DataOps is the decrease in information silos within an organization. These silos are caused when analysts rely on self-service solutions, but they don’t communicate with other departments that may have similar needs.

This lack of communication can lead to data errors. Because a group or department can examine the data and interpret it differently from another, based on different criteria. According to research conducted by MIT, these communication errors can be reflected between 15 and 25% in a company’s revenue.

Data analytics specialists and marketing professionals should be aware that both DevOps and DataOps developments include audits, ensuring that the content and associated tasks meet the level Required.

Version control improves quality

Platforms like GitHub ensure teams use the right version of DevOps, helping to maintain quality. These measures have spread to data science, and that is that professionals must learn to share quality projects. Because working with the best version, it offers better conditions to raise data quality and business performance.

Data management through version control also helps improve the security of the algorithm. According to a study by Forrester, devOps and DataOps tools are expected to proliferate across devices and industries throughout 2018.

Examples of such tools for DataOps would be: MapR provide useful business solutions, rather than computer science projects. Tamr calls its DataOps strategy “Enterprise Data Unification”. Delphix talks about its “Dynamic Data Platform”. Switchboard Software offers its own DataOps platform.