What Is Data Management?

Data management is the practice of collecting, keeping, and using data securely, efficiently, and cost-effectively. The goal of data management is to help people, organizations, and connected things optimize the use of data within the bounds of policy and regulation so that they can make decisions and take actions that maximize the benefit to the organization. A robust data management strategy is becoming more important than ever as organizations increasingly rely on intangible assets to create value.

Managing digital data in an organization involves a broad range of tasks, policies, procedures, and practices. The work of data management has a wide scope, covering factors such as how to

• Create, access, and update data across a diverse data tier
• Store data across multiple clouds and on premises
• Provide high availability and disaster recovery
• Use data in a growing variety of apps, analytics, and algorithms
• Ensure data privacy and security
• Archive and destroy data in accordance with retention schedules and compliance requirements

A formal data management strategy addresses the activity of users and administrators, the capabilities of data management technologies, the demands of regulatory requirements, and the needs of the organization to obtain value from its data.

Data Management Systems Today

Today’s organizations need a data management solution that provides an efficient way to manage data across a diverse but unified data tier. Data management systems are built on data management platforms and can include databases, data lakes and warehouses, big data management systems, data analytics, and more.

All these components work together as a “data utility” to deliver the data management capabilities an organization needs for its apps, and the analytics and algorithms that use the data originated by those apps. Although current tools help database administrators (DBAs) automate many of the traditional management tasks, manual intervention is still often required because of the size and complexity of most database deployments. Whenever manual intervention is required, the chance for errors increases. Reducing the need for manual data management is a key objective of a new data management technology, the autonomous database.

Big Data Management Systems

In some ways, big data is just what it sounds like—lots and lots of data. But big data also comes in a wider variety of forms than traditional data, and it’s collected at a high rate of speed. Think of all the data that comes in every day, or every minute, from a social media source such as Facebook. The amount, variety, and speed of that data are what make it so valuable to businesses, but they also make it very complex to manage.

As more and more data is collected from sources as disparate as video cameras, social media, audio recordings, and Internet of Things (IoT) devices, big data management systems have emerged. These systems specialize in three general areas.

• Big data integration brings in different types of data—from batch to streaming—and transforms it so that it can be consumed.
• Big data management stores and processes data in a data lake or data warehouse efficiently, securely, and reliably, often by using object storage.
• Big data analysis uncovers new insights with analytics and uses machine learning and AI visualization to build models.

Companies are using big data to improve and accelerate product development, predictive maintenance, the customer experience, security, operational efficiency, and much more. As big data gets bigger, so will the opportunities.

Data Management Challenges

Most of the challenges in data management today stem from the faster pace of business and the increasing proliferation of data. The ever-expanding variety, velocity, and volume of data available to organizations is pushing them to seek more-effective management tools to keep up. Some of the top challenges organizations face include the following:

• They don’t know what data they have. Data from an increasing number and variety of sources such as sensors, smart devices, social media, and video cameras is being collected and stored. But none of that data is useful if the organization doesn’t know what data it has, where it is, and how to use it.
• They must maintain performance levels as the data tier expands. Organizations are capturing, storing, and using more data all the time. To maintain peak response times across this expanding tier, organizations need to continuously monitor the type of questions the database is answering and change the indexes as the queries change—without affecting performance.
• They must meet constantly changing compliance requirements. Compliance regulations are complex and multijurisdictional, and they change constantly. Organizations need to be able to easily review their data and identify anything that falls under new or modified requirements. In particular, personally identifiable information (PII) must be detected, tracked, and monitored for compliance with increasingly strict global privacy regulations.
• They aren’t sure how to repurpose data to put it to new uses. Collecting and identifying the data itself doesn’t provide any value—the organization needs to process it. If it takes a lot of time and effort to convert the data into what they need for analysis, that analysis won’t happen. As a result, the potential value of that data is lost.
• They must keep up with changes in data storage. In the new world of data management, organizations store data in multiple systems, including data warehouses and unstructured data lakes that store any data in any format in a single repository. An organization’s data scientists need a way to quickly and easily transform data from its original format into the shape, format, or model they need it to be in for a wide array of analyses.

Data Management Best Practices

Addressing data management challenges requires a comprehensive, well-thought-out set of best practices. Although specific best practices vary depending on the type of data involved and the industry, the following best practices address the major data management challenges organizations face today:

• Create a discovery layer to identify your data. A discovery layer on top of your organization’s data tier allows analysts and data scientists to search and browse for datasets to make your data useable.
• Develop a data science environment to efficiently repurpose your data. A data science environment automates as much of the data transformation work as possible, streamlining the creation and evaluation of data models. A set of tools that eliminates the need for the manual transformation of data can expedite the hypothesizing and testing of new models.
• Use autonomous technology to maintain performance levels across your expanding data tier. Autonomous data capabilities use AI and machine learning to continuously monitor database queries and optimize indexes as the queries change. This allows the database to maintain rapid response times and frees DBAs and data scientists from time-consuming manual tasks.
• Use discovery to stay on top of compliance requirements. New tools use data discovery to review data and identify the chains of connection that need to be detected, tracked, and monitored for multijurisdictional compliance. As compliance demands increase globally, this capability is going to be increasingly important to risk and security officers.
• Use a common query layer to manage multiple and diverse forms of data storage. New technologies are enabling data management repositories to work together, making the differences between them disappear. A common query layer that spans the many kinds of data storage enables data scientists, analysts, and applications to access data without needing to know where it is stored and without needing to manually transform it into a usable format.

Data Management Evolves

With data’s new role as business capital, organizations are discovering what digital startups and disruptors already know: Data is a valuable asset for identifying trends, making decisions, and taking action before competitors. The new position of data in the value chain is leading organizations to actively seek better ways to derive value from this new capital.

Within companies, the data management responsibilities of the DBA are also evolving, reducing the number of mundane tasks so that DBAs can concentrate on more strategic issues and provide critical data management support in cloud environments (PDF) involving key initiatives such as data modeling and data security.