Enterprise Information Integration (EII)
Enterprise information integration (EII) is software that combines enterprise data and information into a unique data monitoring interface where data is expressed via uniform representation. EII consolidates a large group of distinct data sources into one user and system resource. EII uses data integration in business applications. Enterprise data may be saved as multiple file formats, including relational databases, text, Extensible Markup Language (XML), Excel and many storage systems with proprietary indexing and data access schemes. EII theories and opinions vary, but there is universal concern about data integration feasibility as a stand-alone product. This is because EII is based on speed and practicality rather than correctness and manageability.
EII refers to software systems that can take data in different formats from a variety of internal and external sources and treat them as a single data source.
Figure 1. source: SOA Blueprint
These capabilities should be provided by EII:
- Data modeling across multiple sources
- Query (read and write) development to extract information from multiple data source.
- Support for multiple data sources such as database, file, application adapter, LDAP, and web services
- Data transformation
- Data validation
- Exposure of data services to client applications using RMI or web services.
- Adherence to standards such as SQL, XQuery, XML, web services, JDBC, and J2EE.
Enterprise Information Integration (EII) applies data integration commercially. Despite the theoretical problems described above, the private sector shows more concern with the problems of data integration as a viable product. EII emphasizes neither on correctness nor tractability, but speed and simplicity. An EII industry has emerged, but many professionals believe it does not perform to its full potential. Practitioners cite the following major issues which EII must address for the industry to become mature:
- Combining disparate data sets : Each data source is disparate and as such is not designed to support EII. Therefore, data virtualization as well as data federation depends upon accidental data commonality to support combining data and information from disparate data sets. Because of this lack of data value commonality across data sources, the return set may be inaccurate, incomplete, and impossible to validate. One solution is to recast disparate databases to integrate these databases without the need for ETL. The recast databases support commonality constraints where referential integrity may be enforced between databases. The recast databases provide designed data access paths with data value commonality across databases.
- Simplicity of understanding: Answering queries with views arouses interest from a theoretical standpoint, but difficulties in understanding how to incorporate it as an "enterprise solution". Some developers believe it should be merged with EAI. Others believe it should be incorporated with ETL systems, citing customers' confusion over the differences between the two services.
- Simplicity of deployment: Even if recognized as a solution to a problem, EII as of 2009 currently takes time to apply and offers complexities in deployment. People have proposed a variety of schema-less solutions such as "Lean Middleware", but ease-of-use and speed of employment appear inversely proportional to the generality of such systems. Others cite the need for standard data interfaces to speed and simplify the integration process in practice.
- Handling higher-order information : Analysts experience difficulty — even with a functioning information integration system — in determining whether the sources in the database will satisfy a given application. Answering these kinds of questions about a set of repositories requires semantic information like metadata and/or ontologies. The few commercial tools that leverage this information remain in their infancy.
Enterprise Information Integration: The Need and Benefits
“Enterprise Information Integration is an approach to integration that has arisen out of the need for organizations to identify and correlate related, but separate data,” according to JP Morganthal, author of the seminar book on EII entitled Enterprise Information Integration: A Pragmatic Approach. “Enterprise Information Integration allows users to derive new data structures and information models without having to understand the nuances of underlying data structures, data locations, data types, etc. In essence, EII solutions provide access to the data without the hindrances of the underlying technology.” Both the business and IT benefit from Enterprise Information Integration. “For business users, Enterprise Information Integration removes the barriers to accessing enterprise data and provides a common infrastructure to transform that data into usable information, usually on demand”, according to Morganthal. “For the technologist, Enterprise Information Integration simplifies the longstanding data integration problem by applying a ‘semantic veneer’ over the complex physical data layer.”
Typical Uses of EII Technology
EII’s distinguishing characteristic is its ability to access and integrate current data from heterogeneous sources. How? A virtualized data federation layer delivers data to applications in real time from original sources. Various vertical segments increasingly use EII for business decisions:
- Banking: Banks provide multiple lines of service: checking, home equity loans, brokerage services, etc. Banks want a complete view of customer activities across lines of business from multiple divisions and applications. Most banking data is in silos, making EII a natural choice for integration.
- Securities Trading: As hedge funds trade across asset classes, traders need a complete view that incorporates up-to-the-minute data. Typically, different applications generate trading data, and different databases store it. EII allows integration from different sources, providing complete trading information across asset classes.
- Single Customer View: As organizations roll out services, customer service representatives need complete views across all lines of business. For example, insurance companies are adding services besides life, automobile, and homeowners insurance. Service representatives must access all relevant data. EII technology is ideal for this application.
- Federal Government: Certain laws and regulations make sharing information difficult. However, Homeland Security applications must integrate the data government agencies collect. Aggregating and copying data into a central repository won’t work. Agencies need to expose particular data elements, stipulate uses, and make specific data available to other agencies. EII meets this need.
Enterprise Information Integration (EII) Characteristics
- Support for a variety of data sources, including relational database management systems (DBMSs), non-relational DBMSs, files, XML documents, and others
- SQL - based API
- Ability to join, union, aggregate, and otherwise correlate data from multiple sources in a single query
- Ability to create individual views or virtual data objects based on data integrated from multiple sources
- Location Transparency
- Automatic data type conversion services
- Real-time programming model
Generic Enterprise Information Integration Architecture (Gantz, 2004)
Figure 2. source: UC Berkeley
When Should Enterprise Information Integration (EII) Be Used
The following are situations when EII makes sense for data integration requirements.
- Connecting structured (as in data in a data warehouse) data in particular with unstructured data takes advantage of EII’s strength of leaving data in place that could dramatically increase overall storage requirements if duplicated
- When immediate data change in response to the data view is desired (changing a copy of the data would not suffice)
- When data transformation is relatively light or nonexistent and just getting the data together for integrated query is the biggest challenge
- When the relatively worse query performance of EII query is acceptable (versus the obvious advantages of physically cohabitating all data for the query)
- Some operational and regulatory reporting where the data needed is not completely integrated in one place
- Integration of Performance Management software with multiple underlying line of business BI systems to allow a company to see performance management at the enterprise level (across line of business BI systems) using LOB metrics to calculate enterprise KPIs
Enterprise Information Integration (EII): Strengths and Challenges
EII Major Strengths
- Relational access to non-relational sources
- Ability to explore data before a formal data model and metadata are created
- Quicker deployment
- Can be reused by ETL and/or EAI further developments
- Access in place data, meaning it avoids unnecessary movement of data.
- Optimized for global access to remote sources
- Event publishing technology provides a non-intrusive means to “listen” for particular changes (insert, update or deletes) that defined as being of interest.
EII Main Challenges
- Need Matching keys across sources
- Data types mismatch
- Data reconciliation
- Possibly high resource utilization on the source system
- Limited to hundreds of thousands of rows for remote result sets
- Performance degradation when query pushdown is not used
- Limited transformation – bounded by SQL capability and system capacity
- May consume network bandwidth during peak hours
- Multi-site updates require transactional control
The Future of EII
The future of EII will be determined to a large degree by the continued convergence of unstructured and structured data. With Microsoft, Oracle and IBM all morphing their database platforms into repositories for all types of data, it is only natural to expect that basic integration capabilities will increasingly be included. As this occurs the main challenges to integration will be less technical and more informational. That is, information integration will still be very difficult, but the center of effort will shift towards information architecture, including structure and metadata management. Web services are also especially relevant to the future of EII because they provide a way to meet fundamental needs associated with both the application and information integration requirements. Web services are part of the solution to each half of the problem because they: • simplify the application integration to largely a one-sided problem • allow for the scaling necessary for the large volumes of information • get around some of the information complexity by allowing for the information to reside in environments where it is understood – where it can be manage by the humans and applications that understand it.
Information integration will continue to be a challenge. Like many content technologies, consensus on technical and architectural approaches to EII won’t come close to being reached until a majority of IT and developers understand more about unstructured and semi-structured information, and such understanding won’t happen until there has been a critical mass of industry experience building integrated applications. Twenty-something developers are faced with this integration challenge, and while they may not fully grasp the complexity of managing unstructured and semi-structured content, neither are they handicapped by data and API-centric views of the previous generations. Not all integration issues are equal – the number of applications that need to be integrated is far smaller than the number of documents, or content elements, that need to be integrated. There are a lot of similar requirements across industries and among companies who need to connect information piping between, e.g., an ERP system and a catalog internally, and even when they need to connect with partner systems. But the diversity of the content and related contextual processing once the information has gotten from one system to the other is infinite by comparison. This means that there will continue to be a need for a wide variety of EII technologies and approaches for the foreseeable future. Even though many content management functions are becoming commoditized, the value added by those who understand the content will ensure the market for multiple content technologies will remain healthy. The same is true for EII. You can argue that EII technology needs to be everywhere, and in particular, included in platforms, but the need for expertise on how to effectively use it guarantees that integration specialists and product companies with domain expertise will be around for awhile.
Enterprise Application Integration (EAI)
Enterprise Asset Management (EAM)
Information System (IS)
Information Management (IM)
Enterprise Information Management (EIM)
- What is Enterprise Information Integration (EII)? Techopedia
- Capabilities provided by Enterprise Information Integration (EII) SOABlueprint
- Overview of Enterprise Information Integration (EII) Wikipedia
- Enterprise Information Integration: The Need and Benefits TIBC
- What are the Typical Uses of EII Technology? TDWI
- Enterprise Information Integration (EII) Characteristics Sharbani Bhattacharya
- When to Use Enterprise Information Integration Mike Ferguson, William McKnight
- Strengths and Challenges of Enterprise Information Integration (EII) IBM
- What is the Future of EII Gilbane Report
- Enterprise information integration (EII) delivers information on demand Wayne Kernochan
- Enterprise Information Integration: Successes, Challenges and Controversies Alon Y. Halevy, Naveen Ashish et al.
- Enterprise Information Integration Services Thomas C. Hammergren
- EII, EAI, and ETL in Laymans Terms Daniel Linstedt
- The Myth and the Magic of Enterprise Information Integration Information Week