Actions

Difference between revisions of "Data Monitoring"

Line 1: Line 1:
===What is Data Monitoring?<ref>[https://toolkit.ncats.nih.gov/glossary/data-monitoring/ What is Data Monitoring?]</ref>
+
===What is Data Monitoring?<ref>[https://toolkit.ncats.nih.gov/glossary/data-monitoring/ What is Data Monitoring?]</ref>===
 
'''Data monitoring''' is the process of reviewing the information entered into the database for accuracy and completeness. The data can also be evaluated to ensure the entered information will accomplish the goals of the registry or clinical study. Data monitoring is performed proactively, either continuously or on a set schedule. The goal of data monitoring is to ensure high-quality data. Data monitoring can catch data that are inconsistent with other participants and therefore may need verification. It can also detect trends or patterns that differ from information collected in the past. Interventional clinical trials are most often monitored by a Data Monitoring Committee (DMC).
 
'''Data monitoring''' is the process of reviewing the information entered into the database for accuracy and completeness. The data can also be evaluated to ensure the entered information will accomplish the goals of the registry or clinical study. Data monitoring is performed proactively, either continuously or on a set schedule. The goal of data monitoring is to ensure high-quality data. Data monitoring can catch data that are inconsistent with other participants and therefore may need verification. It can also detect trends or patterns that differ from information collected in the past. Interventional clinical trials are most often monitored by a Data Monitoring Committee (DMC).
  
Line 8: Line 8:
  
 
===Attributes of Data Monitoring'''<ref>[https://www.edq.com/glossary/data-monitoring/ What is data monitoring software?]</ref>===
 
===Attributes of Data Monitoring'''<ref>[https://www.edq.com/glossary/data-monitoring/ What is data monitoring software?]</ref>===
 +
Simply put, monitoring data is the act of having procedures, technologies, and benchmarks in place for tracking the quality and usefulness of data.
  
Simply put, monitoring data is the act of having procedures, technologies and benchmarks in place for tracking the quality and usefulness of data.
+
The first step to monitoring data is establishing data quality metrics or criteria that are tied to specific business objectives. After establishing the groundwork, you will compare the results over time, allowing for improvement and a deeper understanding of how your data can best be used.
 
 
The first step to monitoring data is establishing data quality metrics or criteria that are tied to specific business objectives. After establishing the groundwork, you will compare the results over time, allowing for improvement and deeper understanding of how your data can best be used.
 
  
 
Some critical attributes of data quality that are frequently monitored by organizations include:
 
Some critical attributes of data quality that are frequently monitored by organizations include:
Line 19: Line 18:
 
*Uniqueness
 
*Uniqueness
 
Data quality monitoring helps you reveal problem areas where the most inaccuracies are observed, track unusual or abnormal behaviors, and identify where you should focus your data quality initiatives.
 
Data quality monitoring helps you reveal problem areas where the most inaccuracies are observed, track unusual or abnormal behaviors, and identify where you should focus your data quality initiatives.
 +
 +
 +
===Elements of an Automated Data Monitoring System<ref>[https://medium.com/@vikati/the-rise-of-data-monitoring-7221eef63e7f Elements of an Automated Data Monitoring System]</ref>===
 +
Given the importance of data, data monitoring capabilities in companies are generally below expectations. While large tech companies like Netflix and Uber have built sophisticated data-monitoring platforms and processes, many small and even large companies still can’t easily monitor their data. Enterprises often rely on their legacy data quality software that doesn’t map to their current data stack. Many startups scrape by with a combination of homegrown scripts and tools like Grafana and Prometheus.
 +
 +
In response to this data monitoring gap, there is an emergent class of companies building data monitoring tools.
 +
 +
While there are differences in these tools’ approaches, they typically consist of 3 core parts:<br />
 +
(1) a data collector that connects with the user’s data store,<br />
 +
(2) specific health checks (such as volume, freshness, etc) that run on the connected data, and<br />
 +
(3) a dashboard/alerting system to let users observe and act on the overall health of their data.
 +
 +
 +
[[File:Data Monitoring Elements.png|500px|Elements of Data Monitoring]]<br />
  
  
Line 37: Line 50:
  
 
=== See Also ===
 
=== See Also ===
 +
== See Also ==
 +
<div style="column-count:2;-moz-column-count:4;-webkit-column-count:4">
 +
[[Data Compatibility]]<br />
 +
[[Data Access]]<br />
 +
[[Data Analysis]]<br />
 +
[[Data Analytics]]<br />
 +
[[Data Architecture]]<br />
 +
[[Data Asset Framework (DAF)]]<br />
 +
[[Data Buffer]]<br />
 +
[[Data Center]]<br />
 +
[[Data Center Infrastructure]]<br />
 +
[[Data Center Infrastructure Management (DCIM)]]<br />
 +
[[Data Cleansing]]<br />
 +
[[Big Data]]<br />
 +
[[Big Data Integration]]<br />
 +
[[Big Data Maturity Model (BDMM)]]<br />
 +
[[Metadata]]<br />
 +
[[Data Collection]]<br />
 +
[[Data Compatibility]]<br />
 +
[[Data Consolidation]]<br />
 +
[[Data Deduplication]]<br />
 +
[[Data Delivery Platform (DDP)]]<br />
 +
[[Data Description (Definition) Language (DDL)]]<br />
 +
[[Data Dictionary]]<br />
 +
[[Data Discovery]]<br />
 +
[[Data Driven Organization]]<br />
 +
[[Data Element]]<br />
 +
[[Data Enrichment]]<br />
 +
[[Data Entry]]<br />
 +
[[Data Federation]]<br />
 +
[[Data Flow Diagram]]<br />
 +
[[Data Governance]]<br />
 +
[[Data Health Check]]<br />
 +
[[Data Hierarchy]]<br />
 +
[[Data Independence]]<br />
 +
[[Data Integration]]<br />
 +
[[Data Integration Framework (DIF)]]<br />
 +
[[Data Integrity]]<br />
 +
[[Data Island]]<br />
 +
[[Data Item]]<br />
 +
[[Data Lake]]<br />
 +
[[Data Life Cycle]]<br />
 +
[[Data Lineage]]<br />
 +
[[Data Loss Prevention (DLP)]]<br />
 +
[[Data Management]]<br />
 +
[[Data Migration]]<br />
 +
[[Data Minimization]]<br />
 +
[[Data Mining]]<br />
 +
[[Data Model]]<br />
 +
[[Data Modeling]]<br />
 +
[[Data]]<br />
 +
[[Data Munging]]<br />
 +
[[Data Portability]]<br />
 +
[[Data Preparation]]<br />
 +
[[Data Presentation Architecture]]<br />
 +
[[Data Processing]]<br />
 +
[[Data Profiling]]<br />
 +
[[Data Proliferation]]<br />
 +
[[Data Propagation]]<br />
 +
[[Data Protection Act]]<br />
 +
[[Data Prototyping]]<br />
 +
[[Data Quality]]<br />
 +
[[Data Quality Assessment (DQA)]]<br />
 +
[[Data Quality Dimension]]<br />
 +
[[Data Quality Standard]]<br />
 +
[[Data Reconciliation]]<br />
 +
[[Data Reference Model (DRM)]]<br />
 +
[[Data Science]]<br />
 +
[[Data Security]]<br />
 +
[[Data Stewardship]]<br />
 +
[[Data Structure]]<br />
 +
[[Data Structure Diagram]]<br />
 +
[[Data Suppression]]<br />
 +
[[Data Transformation]]<br />
 +
[[Data Validation]]<br />
 +
[[Data Value Chain]]<br />
 +
[[Data Vault Modeling]]<br />
 +
[[Data Virtualization]]<br />
 +
[[Data Visualization]]<br />
 +
[[Data Warehouse]]<br />
 +
[[Data Wrangling]]<br />
 +
[[Data and Information Reference Model (DRM)]]<br />
 +
[[Data as a Service (DaaS)]]<br />
 +
[[Database (DB)]]<br />
 +
[[Database Design]]<br />
 +
[[Database Design Methodology]]<br />
 +
[[Database Management System (DBMS)]]<br />
 +
[[Database Marketing]]<br />
 +
[[Database Schema]]<br />
 +
[[Database System]]<br />
 +
</div>
  
  

Revision as of 13:21, 17 October 2022

What is Data Monitoring?[1]

Data monitoring is the process of reviewing the information entered into the database for accuracy and completeness. The data can also be evaluated to ensure the entered information will accomplish the goals of the registry or clinical study. Data monitoring is performed proactively, either continuously or on a set schedule. The goal of data monitoring is to ensure high-quality data. Data monitoring can catch data that are inconsistent with other participants and therefore may need verification. It can also detect trends or patterns that differ from information collected in the past. Interventional clinical trials are most often monitored by a Data Monitoring Committee (DMC).


The Need for Data Monitoring[2]

Data monitoring allows an organization to proactively maintain a high, consistent standard of data quality. By checking data routinely as it is stored within applications, organizations can avoid the resource-intensive pre-processing of data before it is moved. With data monitoring, data is quality checked at creation time rather than before a move.


Attributes of Data Monitoring[3]

Simply put, monitoring data is the act of having procedures, technologies, and benchmarks in place for tracking the quality and usefulness of data.

The first step to monitoring data is establishing data quality metrics or criteria that are tied to specific business objectives. After establishing the groundwork, you will compare the results over time, allowing for improvement and a deeper understanding of how your data can best be used.

Some critical attributes of data quality that are frequently monitored by organizations include:

  • Completeness
  • Uniformity
  • Accuracy
  • Uniqueness

Data quality monitoring helps you reveal problem areas where the most inaccuracies are observed, track unusual or abnormal behaviors, and identify where you should focus your data quality initiatives.


Elements of an Automated Data Monitoring System[4]

Given the importance of data, data monitoring capabilities in companies are generally below expectations. While large tech companies like Netflix and Uber have built sophisticated data-monitoring platforms and processes, many small and even large companies still can’t easily monitor their data. Enterprises often rely on their legacy data quality software that doesn’t map to their current data stack. Many startups scrape by with a combination of homegrown scripts and tools like Grafana and Prometheus.

In response to this data monitoring gap, there is an emergent class of companies building data monitoring tools.

While there are differences in these tools’ approaches, they typically consist of 3 core parts:
(1) a data collector that connects with the user’s data store,
(2) specific health checks (such as volume, freshness, etc) that run on the connected data, and
(3) a dashboard/alerting system to let users observe and act on the overall health of their data.


Elements of Data Monitoring


Benefits of Data Monitoring[5]

  • A holistic view of the data ecosystem is created and maintained, providing increased visibility.
  • Agility and speed are increased across an organization because data can be used immediately.
  • Areas, where inaccuracies are most commonly found, can be identified and monitored to find the causes of the issues.
  • Challenges associated with the propagation of erroneous or inconsistent data are eliminated by checking data at the time of its creation.
  • Closely monitored data enables completeness, consistency, and accuracy.
  • Connections can be established between data from disparate sources across an organization.
  • Data can more easily be standardized.
  • Established data quality metrics can be tracked and reported on to provide insights into adherence to standards and goals.
  • New data quality parameters can be added to data monitoring processes to keep pace with changing priorities or concerns within an organization.
  • Proactive checking of data against rules helps maintain high-quality, consistent data.
  • Time and money are saved by eliminating manual data checks.
  • Time spent preparing data for use is minimized, because its data monitoring facilitates ongoing data quality.


See Also

See Also

Data Compatibility
Data Access
Data Analysis
Data Analytics
Data Architecture
Data Asset Framework (DAF)
Data Buffer
Data Center
Data Center Infrastructure
Data Center Infrastructure Management (DCIM)
Data Cleansing
Big Data
Big Data Integration
Big Data Maturity Model (BDMM)
Metadata
Data Collection
Data Compatibility
Data Consolidation
Data Deduplication
Data Delivery Platform (DDP)
Data Description (Definition) Language (DDL)
Data Dictionary
Data Discovery
Data Driven Organization
Data Element
Data Enrichment
Data Entry
Data Federation
Data Flow Diagram
Data Governance
Data Health Check
Data Hierarchy
Data Independence
Data Integration
Data Integration Framework (DIF)
Data Integrity
Data Island
Data Item
Data Lake
Data Life Cycle
Data Lineage
Data Loss Prevention (DLP)
Data Management
Data Migration
Data Minimization
Data Mining
Data Model
Data Modeling
Data
Data Munging
Data Portability
Data Preparation
Data Presentation Architecture
Data Processing
Data Profiling
Data Proliferation
Data Propagation
Data Protection Act
Data Prototyping
Data Quality
Data Quality Assessment (DQA)
Data Quality Dimension
Data Quality Standard
Data Reconciliation
Data Reference Model (DRM)
Data Science
Data Security
Data Stewardship
Data Structure
Data Structure Diagram
Data Suppression
Data Transformation
Data Validation
Data Value Chain
Data Vault Modeling
Data Virtualization
Data Visualization
Data Warehouse
Data Wrangling
Data and Information Reference Model (DRM)
Data as a Service (DaaS)
Database (DB)
Database Design
Database Design Methodology
Database Management System (DBMS)
Database Marketing
Database Schema
Database System


References