Data Architecture - Revision history

User at 23:40, 22 March 2024

2024-03-22T23:40:55Z

User at 23:40, 22 March 2024

2024-03-22T23:40:32Z

User at 12:05, 24 February 2024

2024-02-24T12:05:52Z

User at 15:48, 14 March 2023

2023-03-14T15:48:12Z

User at 17:49, 12 January 2023

2023-01-12T17:49:17Z

User at 14:56, 21 November 2022

2022-11-21T14:56:36Z

User at 14:56, 21 November 2022

2022-11-21T14:56:03Z

User at 14:54, 21 November 2022

2022-11-21T14:54:20Z

User at 14:44, 21 November 2022

2022-11-21T14:44:18Z

User at 14:13, 21 November 2022

2022-11-21T14:13:58Z

← Older revision		Revision as of 23:40, 22 March 2024
Line 87:		Line 87:
	*[[Master Data Management (MDM)]]: Discussing the process of creating and maintaining a consistent and accurate set of master data across an organization. MDM is a critical component of data architecture for ensuring data integrity and consistency.		*[[Master Data Management (MDM)]]: Discussing the process of creating and maintaining a consistent and accurate set of master data across an organization. MDM is a critical component of data architecture for ensuring data integrity and consistency.
	*Data Quality Management: Covering the processes and technologies used to measure, manage, and improve the quality of data. High-quality data is crucial for effective decision-making and operational efficiency.		*Data Quality Management: Covering the processes and technologies used to measure, manage, and improve the quality of data. High-quality data is crucial for effective decision-making and operational efficiency.
−	*[[Architecture]]
−	*[[Enterprise Architecture]]

@@ Line 73: / Line 73: @@
 == See Also ==
 *[[Architecture]]
 *[[Enterprise Architecture]]

@@ Line 73: / Line 73: @@
 == See Also ==
 *[[Architecture]]
 *[[Enterprise Architecture]]

@@ Line 1: / Line 1: @@
-'''Data Architecture''' is a set of rules, policies, standards, and models that govern and define the type of data collected and how it is used, stored, managed, and integrated within an organization and its database systems. It provides a formal approach to creating and managing the flow of data and how it is processed across an organization’s IT systems and applications.<ref>[https://www.techopedia.com/definition/6730/data-architecture Defining Data Architecture]</ref>
+==What is Data Architecture? ==
-'''Overview of Data Architecture<ref>[https://en.wikipedia.org/wiki/Data_architecture An Overview of Data Architecture]</ref>'''<br />
+== Overview of Data Architecture<ref>[https://en.wikipedia.org/wiki/Data_architecture An Overview of Data Architecture]</ref> ==
 A data architecture should set data standards for all its data systems as a vision or a model of the eventual interactions between those data systems. [[Data Integration|Data integration]], for example, should be dependent upon data architecture standards since data integration requires data interactions between two or more data systems. A data architecture, in part, describes the data structures used by a business and its computer applications software. Data architectures address data in storage, data in use, and data in motion; descriptions of data stores, data groups, and data items; and mappings of those data artifacts to data qualities, applications, locations, etc.
@@ Line 16: / Line 22: @@
 Data architecture should be defined in the planning phase of the design of a new data processing and storage system. The major types and sources of data necessary to support an enterprise should be identified in a manner that is complete, consistent, and understandable. The primary requirement at this stage is to define all of the relevant data entities, not to specify computer hardware items. A data entity is any real or abstracted thing about which an organization or individual wishes to store data.
-'''Components in Data Architecture'''<ref>[https://towardsdatascience.com/fundamentals-of-data-architecture-to-help-data-scientists-understand-architectural-diagrams-better-7bd26de41c66 Three Components in Data Architecture]</ref>'''<br />
 “Data Lake”, “Data Warehouse”, and “Data Mart” are typical components in the architecture of a data platform. In this order, data produced in the business is processed and set to create another data implication.
@@ Line 28: / Line 35: @@
 *Data Mart: holds a subpart and/or aggregated data set for the use of a particular business function, e.g. specific business unit or specific geographical area. A typical example is when we prepare the summary of KPIs for a specific business line followed by visualization in BI tool. Especially, preparing this kind of separate and independent component after the warehouse is worthwhile when the user wants the data mart regularly and frequently updated. On contrary, this portion can be skipped in cases the user only wants some set of data for ad hoc analysis done only once.
-Roughly speaking, data engineers cover from data extraction produced in business to the data lake and data model building in the data warehouse as well as establishing ETL pipeline; while data scientists cover from data extraction out of the data warehouse, building data mart, and to lead to further business application and value creation.
+Roughly speaking, data engineers cover from data extraction produced in business to the data lake and data model building in the data warehouse as well as establishing the ETL pipeline; while data scientists cover from data extraction out of the data warehouse, building data mart, and to lead to further business application and value creation.
 Of course, this role assignment between data engineers and data scientists is somewhat ideal and many companies do not hire both just to fit this definition. Actually, their job descriptions tend to overlap.
-===See Also===
+== Data Architecture Principles ==
-<div style="column-count:2;-moz-column-count:4;-webkit-column-count:4">
+According to Joshua Klahr, vice president of product management, and core products, at Splunk, and former vice president of product management at AtScale, six principles form the foundation of modern data architecture:
-</div>
+== See Also ==
-===References===
+== References ==
 <references/>

@@ Line 35: / Line 35: @@
 ===See Also===
 <div style="column-count:2;-moz-column-count:4;-webkit-column-count:4">
-*[[Data]]
 </div>

← Older revision		Revision as of 14:56, 21 November 2022
Line 1:		Line 1:
−	'''Data Architecture'' is a set of rules, policies, standards, and models that govern and define the type of data collected and how it is used, stored, managed, and integrated within an organization and its database systems. It provides a formal approach to creating and managing the flow of data and how it is processed across an organization’s IT systems and applications.<ref>[https://www.techopedia.com/definition/6730/data-architecture Defining Data Architecture]</ref>	+	'''Data Architecture''' is a set of rules, policies, standards, and models that govern and define the type of data collected and how it is used, stored, managed, and integrated within an organization and its database systems. It provides a formal approach to creating and managing the flow of data and how it is processed across an organization’s IT systems and applications.<ref>[https://www.techopedia.com/definition/6730/data-architecture Defining Data Architecture]</ref>

← Older revision		Revision as of 14:56, 21 November 2022
Line 27:		Line 27:
	*Data Warehouse: holds data processed and structured by a managed data model, reflecting the global (not specific) direction of the final use of the data. In many cases, the data is in tabular format.		*Data Warehouse: holds data processed and structured by a managed data model, reflecting the global (not specific) direction of the final use of the data. In many cases, the data is in tabular format.
	*Data Mart: holds a subpart and/or aggregated data set for the use of a particular business function, e.g. specific business unit or specific geographical area. A typical example is when we prepare the summary of KPIs for a specific business line followed by visualization in BI tool. Especially, preparing this kind of separate and independent component after the warehouse is worthwhile when the user wants the data mart regularly and frequently updated. On contrary, this portion can be skipped in cases the user only wants some set of data for ad hoc analysis done only once.		*Data Mart: holds a subpart and/or aggregated data set for the use of a particular business function, e.g. specific business unit or specific geographical area. A typical example is when we prepare the summary of KPIs for a specific business line followed by visualization in BI tool. Especially, preparing this kind of separate and independent component after the warehouse is worthwhile when the user wants the data mart regularly and frequently updated. On contrary, this portion can be skipped in cases the user only wants some set of data for ad hoc analysis done only once.
		+
		+	Roughly speaking, data engineers cover from data extraction produced in business to the data lake and data model building in the data warehouse as well as establishing ETL pipeline; while data scientists cover from data extraction out of the data warehouse, building data mart, and to lead to further business application and value creation.
		+
		+	Of course, this role assignment between data engineers and data scientists is somewhat ideal and many companies do not hire both just to fit this definition. Actually, their job descriptions tend to overlap.

@@ Line 3: / Line 3: @@
 '''Overview of Data Architecture<ref>An Overview of Data Architecture [https://en.wikipedia.org/wiki/Data_architecture Wikipedia]</ref>'''<br />
-A data architecture should set data standards for all its data systems as a vision or a model of the eventual interactions between those data systems. [[Data Integration|Data integration]], for example, should be dependent upon data architecture standards since data integration requires data interactions between two or more data systems. A data architecture, in part, describes the [[Data Structure|data structures]] used by a business and its computer applications software. Data architectures address data in storage, data in use, and data in motion; descriptions of data stores, data groups, and data items; and mappings of those data artifacts to data qualities, applications, locations, etc.
+A data architecture should set data standards for all its data systems as a vision or a model of the eventual interactions between those data systems. [[Data Integration|Data integration]], for example, should be dependent upon data architecture standards since data integration requires data interactions between two or more data systems. A data architecture, in part, describes the data structures used by a business and its computer applications software. Data architectures address data in storage, data in use, and data in motion; descriptions of data stores, data groups, and data items; and mappings of those data artifacts to data qualities, applications, locations, etc.
 Essential to realizing the target state, Data Architecture describes how data is processed, stored, and utilized in an information system. It provides criteria for data processing operations so as to make it possible to design data flows and also control the flow of data in the system.

@@ Line 1: / Line 1: @@
-'''Data Architecture''' is a set of rules, policies, standards and models that govern and define the type of [[Data|data]] collected and how it is used, stored, managed and integrated within an [[Organization|organization]] and its database systems. It provides a formal approach to creating and managing the flow of data and how it is processed across an organization’s IT systems and applications.<ref>Defining Data Architecture [https://www.techopedia.com/definition/6730/data-architecture Techopedia]</ref>
+'''Data Architecture''' is a set of rules, policies, standards, and models that govern and define the type of data collected and how it is used, stored, managed, and integrated within an organization and its database systems. It provides a formal approach to creating and managing the flow of data and how it is processed across an organization’s IT systems and applications.<ref>[https://www.techopedia.com/definition/6730/data-architecture Defining Data Architecture]</ref>
 '''Overview of Data Architecture<ref>An Overview of Data Architecture [https://en.wikipedia.org/wiki/Data_architecture Wikipedia]</ref>'''<br />
-A data architecture should set data standards for all its data systems as a [[vision]] or a [[model]] of the eventual interactions between those data systems. [[Data Integration|Data integration]], for example, should be dependent upon data architecture standards since data integration requires data interactions between two or more data systems. A data architecture, in part, describes the [[Data Structure|data structures]] used by a [[business]] and its [[computer]] applications [[software]]. Data architectures address data in storage, data in use and data in motion; descriptions of data stores, data groups and data items; and mappings of those data artifacts to data qualities, applications, locations etc.
+A data architecture should set data standards for all its data systems as a vision or a model of the eventual interactions between those data systems. [[Data Integration|Data integration]], for example, should be dependent upon data architecture standards since data integration requires data interactions between two or more data systems. A data architecture, in part, describes the [[Data Structure|data structures]] used by a business and its computer applications software. Data architectures address data in storage, data in use, and data in motion; descriptions of data stores, data groups, and data items; and mappings of those data artifacts to data qualities, applications, locations, etc.
-Essential to realizing the [[target]] state, Data Architecture describes how data is processed, stored, and utilized in an information [[system]]. It provides criteria for data processing operations so as to make it possible to [[design]] data flows and also [[control]] the flow of data in the system.
+Essential to realizing the target state, Data Architecture describes how data is processed, stored, and utilized in an information system. It provides criteria for data processing operations so as to make it possible to design data flows and also control the flow of data in the system.
-The data architect is typically responsible for defining the target state, aligning during development and then following up to ensure enhancements are done in the spirit of the original [[blueprint]].
+The data architect is typically responsible for defining the target state, aligning during development, and then following up to ensure enhancements are done in the spirit of the original blueprint.
 During the definition of the target state, the Data Architecture breaks a subject down to the atomic level and then builds it back up to the desired form. The data architect breaks the subject down by going through 3 traditional architectural processes:
@@ Line 14: / Line 14: @@
 *Physical - the realization of the data mechanisms for a specific type of functionality.
-Data architecture should be defined in the planning phase of the design of a new data processing and storage system. The major types and sources of data necessary to support an enterprise should be identified in a manner that is complete, consistent, and understandable. The primary [[requirement]] at this stage is to define all of the relevant data entities, not to specify computer [[hardware]] items. A data entity is any real or abstracted thing about which an [[organization]] or individual wishes to store data.
+Data architecture should be defined in the planning phase of the design of a new data processing and storage system. The major types and sources of data necessary to support an enterprise should be identified in a manner that is complete, consistent, and understandable. The primary requirement at this stage is to define all of the relevant data entities, not to specify computer hardware items. A data entity is any real or abstracted thing about which an organization or individual wishes to store data.
@@ Line 31: / Line 31: @@
 *[[Data Compatibility]]
 *[[Data Governance]]
 *[[Data Management]]
 *[[Data Mining]]