Data Management comprises all disciplines related to managing data as a valuable resource
Data management is the process of ingesting, storing, organizing and maintaining the Data created and collected by an organization. Effective data management is a crucial piece of deploying the IT systems that run business applications and provide analytical information to help drive operational decision-making and strategic planning by corporate executives, business managers and other end users.
The data management process includes a combination of different functions that collectively aim to make sure that the data in corporate systems is accurate, available and accessible. Most of the required work is done by IT and data management teams, but business users typically also participate in some parts of the process to ensure that the data meets their needs and to get them on board with policies governing its use
This comprehensive guide to data management further explains what it is and provides insight on the individual disciplines it includes, best practices for managing data, challenges that organizations face and the business benefits of a successful data management strategy. You’ll also find an overview of data management tools and techniques. Click through the hyperlinks on the page to read about data management trends and get expert advice on managing corporate data.
Types of data management functions
The separate disciplines that are part of the overall data management process cover a series of steps, from data processing and storage to governance of how data is formatted and used in operational and analytical systems. Development of a data architecture is often the first step, particularly in large organizations with lots of data to manage. An architecture provides a blueprint for the databases and other data platforms that will be deployed, including specific technologies to fit individual applications.
Databases are the most common platform used to hold corporate data; they contain a collection of data that’s organized so it can be accessed, updated and managed. They’re used in both transaction processing systems that create operational data, such as customer records and sales orders, and data warehouses which store consolidated data sets from business systems for BI and analytics.
Database administration is a core data management function. Once databases have been set up, performance monitoring and tuning must be done to maintain acceptable response times on database queries that users run to get information from the data stored in them. Other administrative tasks include database design, configuration, installation and updates; data security; database backup and recovery; and application of software upgrades and security patches.
The primary technology used to deploy and administer databases is a database management system DBMS which is software that acts as an interface between the databases it controls and the database administrators, end users and applications that access them. Alternative data platforms to databases include file systems and cloud object storage services; they store data in less structured ways than mainstream databases do, which offers more flexibility on the types of data that can be stored and how it’s formatted. As a result, though, they aren’t a good fit for transactional applications.
Other fundamental data management disciplines include data modelling, which diagrams the relationships between data elements and how data flows through systems; data integration, which combines data from different data sources for operational and analytical uses; data governance, which sets policies and procedures to ensure data is consistent throughout an organization; and data quality management, which aims to fix data errors and inconsistencies. Another is master data management (MDM), which creates a common set of reference data on things like customers and products.
Data Management Skills
Five Data Management Skills that are important for successfully managing and using information.
Looking at and Analyzing Data. The ability to use data effectively to improve your programs, including looking at lists and summaries, looking for patterns, analyzing results, and making presentations to others. Includes familiarity with the data available to you; knowledge of the goals of your program; understanding the needs of your clients/audiences; and willingness to explore creative approaches to using data.
Navigating Database Software. Knowing how to use database software to find records, sort, review, edit, print, and other functions. Knowing how to use built-in forms and reports in a database. Exploring the software and learning various functions and features. Writing queries and reports using available tools; copying data into Excel or other formats for further analysis.
Data Integrity. Understanding definitions, program guidelines, and sources of data. Developing clear channels of communication. Reviewing data and working with colleagues to make sure that data is accurate. Being aware of potential weaknesses in the data when analyzing and using the data.
Managing Accounts and Files. Keeping track of online accounts and helping others to keep track of their accounts, usernames and passwords. Knowing how to organize files and folders on your computer or network. Knowing how to copy, move, upload or download files and photos; understanding how to use email to send attachments.
Database Design and Planning. Understanding database design concepts, including “relational database design” concepts (table structure; one-to-many relationships). Understanding the benefits and limits of various types of databases, including PC and online databases. Ability to participate in short-term and long-term planning about database projects and to decide how to efficiently store and analyze various types of data.
Importance of data management
Data increasingly is seen as a corporate asset that can be used to make more-informed business decisions, improve marketing campaigns, optimize business operations and reduce costs, all with the goal of increasing revenue and profits. But a lack of proper data management can saddle organizations with incompatible data silos, inconsistent data sets and data quality problems that limit their ability to run business intelligence (BI) and analytics applications — or, worse, lead to faulty findings.
Data management has also grown in importance as businesses are subjected to an increasing number of regulatory compliance requirements, including data privacy and protection laws such as GDPR and the California Consumer Privacy Act. In addition, companies are capturing ever-larger volumes of data and a wider variety of data types, both hallmarks of the big-data systems many have deployed. Without good data management, such environments can become unwieldy and hard to navigate.
WHAT IS DATA MANAGEMENT GATEWAY?
What is the Data Management Gateway?
There are 3 components involved as far as data mgt gateway is concerned, all of which are set up by the Power BI administrator:
- Data Management Gateway (DMG). This is a client agent installed on the on-premises server. The DMG handles encryption, compression, and transmission of data using a service bus. Communicates with O365 using a key provided by the Power BI Gateway.
- Power BI Gateway. This is set up in the Power BI Admin Center (in O365). Handles communication with an on-premises server.
- Power BI Data Source. This is set up in the Power BI Admin Center (in O365). Connects to a database via the Gateway.
When the administrator creates a Data Source in the Power BI Admin Center, you have 2 choices: To allow cloud access and/or to create an OData feed. If cloud access is allowed, that will permit users of the Power BI app to schedule data refreshes that point to that database. If you create an OData feed, that will expose tables and views to organizational search using Power Query.
When is the Data Management Gateway Required?
SQL Server Database On-Premises: Yes
Oracle Database On-Premises: Yes
Windows Azure SQL Database (WASD): No*
SQL Server Database in an Azure VM: No*
Public OData Sources: No
Each of the * items are cloud services associated with Azure. For those types of data sources, Power BI permits the scheduled data refresh to occur using stored credentials in the workbook, or using the secure store service, without requiring the 3 components listed above to be set up by the administrator. As of now (March 2014), it’s currently just the on-premises data sources which require the 3 components listed above.
The advantage of not having to set up the DMG is that it’s less steps to set up. If you’re using a cloud service as a way to quickly and cheaply set up a data source for temporary use or a one-time analysis, the simplicity is nice.
One disadvantage of not having to set up the DMG is that the Power BI system administrator isn’t seeing the data refresh activity occur in the Power BI Admin Center. As of this release of Power BI, I’m not seeing where data refreshes from cloud sources are being logged. In a managed self-service BI environment, this visibility to what’s really happening in the system is very important.
Another disadvantage is that it’s a different user experience. Let’s say User Emily upload a new workbook which has a data source from SQL Server on-premises. Her scheduled data refresh will fail unless the Power BI administrator has set up the 3 components listed above. However, if User Emily uploads a new workbook which has a data source from an Azure SQL Database, her scheduled refresh will be successful without these extra steps. Since users aren’t always aware of where data sources are, this can lead to an inconsistent experience where they’re not exactly sure when it “just works” versus when to contact the Power BI administrator for additional help.
Having said all that, I’m sure we all expect that much of how things work will evolve as Power BI matures and grows. It is, after all, in its first release. Am just passing along how I’ve observed that it’s working currently so administrators and users are aware of the behavior.
How are gateways used in data management?
The Data management gateway is a client agent that you must install in your on-premises environment to copy data between cloud and on-premises data stores. The on-premises data stores supported by Data Factory are listed in the Supported data sources section.
How To Setup the Data Management Gateway Client
After you download and run the Client Setup EXE, you will go through the wizard to set it up.
Clicking Finish will close the setup wizard and launch the Data Management Gateway Configuration Manager.
Configure the Data Management Gateway Client
After we get the Gateway setup, we will need to configure it. Remember that Gateway Key we got from the web site? We’ll need that here. You’ll want to copy the Gateway key into the dialog and click register.
You’ll then need to configure the certificate to be used to encrypt the credentials that you will supply for the Data Source. In a production environment, you’ll want to use a real certificate, but for the purpose of this walkthrough, you can just select Use service generated certificate (self-signed cert). Then click next.
You’ll then need to provide a password for that certificate. Then click next.
It will then prompt you to save the PFX package for the self-signed certificate. Be sure to put that into a location you will remember for safe keeping. Then that will bring us to the endpoint access configuration. Here we can choose to use HTTP or HTTPS (SSL) for OData access. For data sources that you define that will use this gateway, you can expose them as OData feeds. This selection is about how that data will be exposed. Again, in a production environment, you should use HTTPS. However for our purposes, go ahead and choose HTTP. Then click next.
You have successfully create a Gateway. Now you can go back to the Power BI Admin Center and create a data source that will use this gateway to access a server on the machine you installed the Gateway on. Click the Finish button.
You should then be presented with a status screen within the Configuration Manager. From here you can change the Gateway key, or stop/start the Gateway Windows Service. You can also review the settings for the endpoint.
If we go back to the Power BI Admin Center, we can also see the Gateway listed and its status.
