Data Monitoring

What is Data Monitoring?

Data monitoring is the process of reviewing the information entered into the database for accuracy and completeness. The data can also be evaluated to ensure the entered information will accomplish the goals of the registry or clinical study. Data monitoring is performed proactively, either continuously or on a set schedule. The goal of data monitoring is to ensure high-quality data. Data monitoring can catch data that are inconsistent with other participants and therefore may need verification. It can also detect trends or patterns that differ from information collected in the past. Interventional clinical trials are most often monitored by a Data Monitoring Committee (DMC).

The Need for Data Monitoring

Data monitoring allows an organization to proactively maintain a high, consistent standard of data quality. By checking data routinely as it is stored within applications, organizations can avoid the resource-intensive pre-processing of data before it is moved. With data monitoring, data is quality checked at creation time rather than before a move.

Attributes of Data Monitoring

Simply put, monitoring data is the act of having procedures, technologies, and benchmarks in place for tracking the quality and usefulness of data.

The first step to monitoring data is establishing data quality metrics or criteria that are tied to specific business objectives. After establishing the groundwork, you will compare the results over time, allowing for improvement and a deeper understanding of how your data can best be used.

Some critical attributes of data quality that are frequently monitored by organizations include:

• Completeness
• Uniformity
• Accuracy
• Uniqueness

Data quality monitoring helps you reveal problem areas where the most inaccuracies are observed, track unusual or abnormal behaviors, and identify where you should focus your data quality initiatives.

Why is data monitoring important? Data monitoring is important because it helps preserve data integrity and usefulness. By ensuring high-quality data, a business can avoid multiplying ongoing problems with future data interpretation and analysis. Examples of potential problems with data include duplicated or missing data points, vague or indistinguishable data, inconsistent data and data drift.

Data monitoring can help a business determine the source of data problems and identify solutions. Depending on how the organization uses its data, monitoring can help assure regulatory compliance, reduce costs or increase profitability, so regularly monitoring and correcting data deficiencies is essential.

Elements of an Automated Data Monitoring System

Given the importance of data, data monitoring capabilities in companies are generally below expectations. While large tech companies like Netflix and Uber have built sophisticated data-monitoring platforms and processes, many small and even large companies still can’t easily monitor their data. Enterprises often rely on their legacy data quality software that doesn’t map to their current data stack. Many startups scrape by with a combination of homegrown scripts and tools like Grafana and Prometheus.

In response to this data monitoring gap, there is an emergent class of companies building data monitoring tools.

While there are differences in these tools’ approaches, they typically consist of 3 core parts:
(1) a data collector that connects with the user’s data store,
(2) specific health checks (such as volume, freshness, etc) that run on the connected data, and
(3) a dashboard/alerting system to let users observe and act on the overall health of their data.

Methods of Data Monitoring

• Speech Analytics: This technique uses artificial intelligence (AI) to monitor calls in real-time to identify tone and sentiment, gauge customer emotion and satisfaction, and even use algorithms to study the skills of agents.
• Text (Interaction) Analytics: Contact centers using chat, email, or social media to interact with customers have many data quality monitoring tools at their disposal. These can scan the text to identify and analyze data. Note: interaction analytics may include speech analytics as well.
• Predictive Analytics: This approach analyzes previous data to identify and predict customer behavior, and finds the most effective method of interaction.
• Performance Analytics: Using custom dashboards, managers can keep track of agent performance and coaching programs for individual agents.

Data Monitoring Techniques===

• Data-Driven Approach: this is when the process of analyzing the data and identifying areas that need improvement is done manually by human beings. This approach is time-consuming and it has high costs as well.
• Data Analytics: This approach collects, analyzes, and reports on the performance of a business in real time. This approach has low cost but it can only be used for certain types of businesses such as manufacturing or retail companies.
• Metrics-Based Approach: This approach uses metrics for assessing the performance of a business without having to do any manual analysis or reporting on the data. This approach is efficient and inexpensive.

Effective Data Monitoring

For a data monitoring system to be useful, it must be:

• Granular: it must indicate specifically where an issue is occurring, and with what code.
• Persistent: you must monitor things in a time series, otherwise you can’t understand where data sets or errors began (lineage).
• Automatic: the more freedom you have to set thresholds and use machine learning and anomaly detection, the less active attention it requires.
• Ubiquitous: you can’t measure just one part of the pipeline.
• Timely: because what good are late alerts?

Benefits of Data Monitoring

• A holistic view of the data ecosystem is created and maintained, providing increased visibility.
• Agility and speed are increased across an organization because data can be used immediately.
• Areas, where inaccuracies are most commonly found, can be identified and monitored to find the causes of the issues.
• Challenges associated with the propagation of erroneous or inconsistent data are eliminated by checking data at the time of its creation.
• Closely monitored data enables completeness, consistency, and accuracy.
• Connections can be established between data from disparate sources across an organization.
• Data can more easily be standardized.
• Established data quality metrics can be tracked and reported on to provide insights into adherence to standards and goals.
• New data quality parameters can be added to data monitoring processes to keep pace with changing priorities or concerns within an organization.
• Proactive checking of data against rules helps maintain high-quality, consistent data.
• Time and money are saved by eliminating manual data checks.
• Time spent preparing data for use is minimized, because its data monitoring facilitates ongoing data quality.

See Also

Data Compatibility
Data Access
Data Analysis
Data Analytics
Data Architecture
Data Asset Framework (DAF)
Data Buffer
Data Center
Data Center Infrastructure
Data Center Infrastructure Management (DCIM)
Data Cleansing
Big Data
Big Data Integration
Big Data Maturity Model (BDMM)
Metadata
Data Collection
Data Compatibility
Data Consolidation
Data Deduplication
Data Delivery Platform (DDP)
Data Description (Definition) Language (DDL)
Data Dictionary
Data Discovery
Data Driven Organization
Data Element
Data Enrichment
Data Entry
Data Federation
Data Flow Diagram
Data Governance
Data Health Check
Data Hierarchy
Data Independence
Data Integration
Data Integration Framework (DIF)
Data Integrity
Data Island
Data Item
Data Lake
Data Life Cycle
Data Lineage
Data Loss Prevention (DLP)
Data Management
Data Migration
Data Minimization
Data Mining
Data Model
Data Modeling
Data
Data Munging
Data Portability
Data Preparation
Data Presentation Architecture
Data Processing
Data Profiling
Data Proliferation
Data Propagation
Data Protection Act
Data Prototyping
Data Quality
Data Quality Assessment (DQA)
Data Quality Dimension
Data Quality Standard
Data Reconciliation
Data Reference Model (DRM)
Data Science
Data Security
Data Stewardship
Data Structure
Data Structure Diagram
Data Suppression
Data Transformation
Data Validation
Data Value Chain
Data Vault Modeling
Data Virtualization
Data Visualization
Data Warehouse
Data Wrangling
Data and Information Reference Model (DRM)
Data as a Service (DaaS)
Database (DB)
Database Design
Database Design Methodology
Database Management System (DBMS)
Database Marketing
Database Schema
Database System