View source for Enterprise Service Bus (ESB)

An '''Enterprise Service Bus (ESB)''' implements a communication system between mutually interacting software applications in a [[Service Oriented Architecture (SOA)]]. As it implements a [[Software_Architecture|software architecture]] for [[Distributed_Computing|distributed computing]], it implements a special variant of the more general [[Client Server Architecture|client-server architecture]] also. Whereas in general any application using ESB can behave as server or client in turns. ESB promotes [[Agility|agility]] and flexibility with regard to high protocol-level communication between applications. The primary goal of the high protocol-level communication is [[Enterprise_Application_Integration_(EAI)|enterprise application integration (EAI)]] of heterogeneous and complex landscapes.<ref>Definition - What is Enterprise Service Bus [https://en.wikipedia.org/wiki/Enterprise_service_bus Wikipedia]</ref>

An Enterprise Service Bus (ESB) facilitates the transfer of data and instructions among services, processes, applications, existing internal systems, data warehouses, analytical systems, and so on. The term bus is borrowed from computer architecture, since a computer bus similarly transfers data and instructions among components in a computer. There are no standard features for an ESB. Nevertheless, all can serve as a router and all need to provide the functionality of adapters. This allows a designer to delegate routing, protocol conversion, and message transformation to the ESB.<ref>Explaining Enterprise Service Bus [http://www.service-architecture.com/articles/web-services/enterprise_service_bus_esb.html|Service Architecture]</ref>


'''Characteristics of an ESB'''<ref>Characteristics of an ESB [http://shop.oreilly.com/product/9780596006754.do David A Chappell]</ref><br />
Due to the fast flurry of vendors trying to gain attention in the growing ESB product category, combined with the number of industry analysts and journalists reporting with their opinions on it, understandably there is much confusion as to what an ESB actually is. This section serves to outline the main characteristics of an ESB.
*Pervasiveness: The ESB can form the core of a pervasive grid. It is capable of spanning an extended enterprise and beyond, having a global reach across departmental organizations, business units, and trading partners. An ESB is also well suited for localized integration projects, and provides flexible underpinnings enabling it to adapt to any type of integration environment.
*Standards-Based Integration: Standards-based integration is a fundamental concept of an ESB. For connectivity, an ESB can utilize J2EE components such as the Java Message Service (JMS) for MOM connectivity, and J2EE Connector Architecture (JCA or J2CA) for connecting to application adapters. An ESB also integrates nicely with applications built with .NET, COM, C#, and C/C++. In addition, an ESB can easily integrate with anything that supports SOAP and web-services APIs, which includes de facto standard web-services toolkit implementations such as Apache Axis. For dealing with data manipulation, an ESB can use XML standards such as XSLT, XPath, and XQuery to provide data transformation, intelligent routing, and querying of "inflight" data as it flows through the bus. For dealing with SOA and business process routing, an ESB can use the Web Services Description Language (WSDL) to describe abstract service interfaces, and Business Process Execution Language for Web Services (BPEL4WS), WS-Choreography, or some other XML based vocabulary such as ebXML, BPSS, to describe abstract business processes. These standards-based interfaces and components are put together in a meaningful way that comprises an open-ended pluggable architecture. An ESB provides an infrastructure that supports both industry-standard integration components as well as proprietary elements through the use of standardized interfaces.
*Highly Distributed Integration and Selective Deployment: The ESB draws from traditional enterprise application integration (EAI) broker functionality in that it provides integration services such as [[Business_Process_Orchestration|business process orchestration]] and routing of data, [[Data_Transformation|data transformation]], and adapters to applications. However, integration brokers are usually highly centralized and monolithic in nature. The ESB provides these integration capabilities as individual services that can work together in a highly distributed fashion, and can be scaled independently from one another.
*Distributed Data Transformation: Data transformation is inherently part of the bus in an ESB deployment. Transformation services that are specialized for the needs of individual applications plugged into the bus can be located anywhere and accessible from anywhere on the bus. Because the data transformation is such an integral part of the ESB itself, an ESB can be thought of as solving the impedance mismatch between applications.
*Extensibility Through Layered Services: An ESB gives you all the required core capabilities for virtually any integration project, and can be augmented with layered technology to handle more specific uses. For example, specialized capabilities such as Business Process Management (BPM) software can process [[workflow]]-related business processes, and collaboration servers can provide specialized services for managing business partners. Specialized third-party translators can provide data conversion from external data formats such as [[Electronic_Data_Interchange_(EDI)|EDI]] into a target [[Enterprise_Resource_Planning_(ERP)|Enterprise Resource Planning (ERP)]] system or onto the general bus as an internal canonical [[XML_(Extensible_Markup_Language)|XML]] representation.
*Event-Driven SOA:In an ESB-enabled, event-driven SOA, applications and services are treated as abstract service endpoints, which can readily respond to asynchronous events. The SOA provides an abstraction away from the details of the underlying connectivity and plumbing. The implementations of the services do not need to understand protocols. Services do not need to understand how messages are routed to other services. They simply receive a message from the ESB as an event, and process that message. The ESB gets the message to anywhere else it needs to go. In an ESB SOA, custom integration services may be created, extended, and reused as ESB functionality. Application endpoints, which are exposed as services, can be constructed together with specialized integration enablers to form composite services and process flows that can be recombined and reused for various purposes, with the goal of automating business functions in a real-time enterprise.
*Process Flow: An ESB's process flow capabilities range from simple sequences of finite steps to sophisticated business process orchestration with parallel process execution paths using conditional splits and joins. The process flow capabilities of the ESB make it possible to define business processes that are local to an individual department or business unit, and that can also coexist within a larger integration [[network]]. This is something a hub-and-spoke integration broker or a BPM tool can't do very well on its own. Process flow in an ESB can also involve specialized integration services that perform intelligent routing of messages based on content. Because the process flow is built on top of the distributed SOA, it is also capable of spanning highly distributed deployment topologies (even spanning oceans at times) without the need to be painfully aware of the physical network boundaries or multiple protocol hops between applications and services on the bus.
*Security and Reliability: The connections between nodes on the ESB are firewall-capable. The security between applications and the ESB, and even between the ESB nodes themselves, is capable of establishing and maintaining the most stringent authentication, credential-management, and access control. Reliability is achieved by having an enterprise-capable MOM at the core of the ESB. The MOM core provides asynchronous communications, reliable delivery of business data, and transactional integrity. As you will learn in Chapter 5, this is not your traditional MOM technology from a decade ago. Requirements have evolved and matured since then, and a MOM core of an ESB must be capable of meeting today's requirements.
*Autonomous but Federated Environment: Traditional hub-and-spoke enterprise application integration (EAI) broker approaches tend to have organizational boundary problems, which are sometimes caused by the physical limitations of the enterprise application integration (EAI)  broker's incapability of easily spanning firewalls and network domains. More importantly, even if a hub-and-spoke architecture is capable of being stretched out across organizational boundaries, it still does not allow the local autonomy that individual business units need to operate semi-independently of one another. One of the biggest problems associated with extending the reach of integration beyond the departmental level is the issue of local autonomy versus centralized control. Local business units and departments need to have control over their own local IT resources, such as the applications running at their site. The integration infrastructure should support deployment topologies to support that business model with practicality. The ESB provides this deployment model, allowing for local message traffic, integration components, and adapters to be locally installed, configured, secured, and managed, while still being able to plug together the local integration domains into a larger federated integration network with an integrated security model.
*Remote Configuration and Management: In some business models it doesn't make sense to have local IT staff at each remote location, although there is still a need for a loosely coupled, autonomous yet federated integration network. To illustrate this point, let's examine a simple case study for an ESB deployment in the retail industry. A video rental chain can have hundreds or thousands of remote locations that all contain the same set of applications, and all operate in the same fashion with regard to inventory management, accounting, and reporting. Using an ESB, an integration blueprint can be established to handle the local communication between the applications at a remote store. This includes the interface definitions to in-store applications, the routing of message traffic and message channel management, and security permissions. It may also include integration components such as an application adapter, protocol adapter, or data transformation. This integration blueprint, or template, can be deployed and customized at each site and act independently of all the other stores.
*XML as the "Native" Datatype of the ESB: XML is an ideal foundation for representing data as it flows between applications across the ESB. The data that is produced and consumed by a vast array of applications can exist in a variety of formats and packaging schemes. While it is certainly possible for the ESB to carry data using any form of packaging or enveloping scheme that you like, there are tremendous benefits to representing in-flight data as XML, including the ability to use specialized ESB services that combine data from different sources to create new views of data, and to enrich and retarget messages for advanced data sharing between applications. 
*Real-Time Throughput of Business Data: An ESB eliminates latency problems by providing real-time throughput into in-flight data as it travels between applications across the ESB. Currently, one of the most popular integration methods is nightly batch processing. However, bulk batch-processing integration strategies, nightly or otherwise, are prone to high margins for error and can cause delays in information retrieval. The resulting high latency in getting up-to-date information can be costly. 
*Operational Awareness: Operational awareness refers to the ability of a business analyst to gain insight into the state and health of business operations. An infrastructure that allows the timely tracking and reporting of data as it flows across an organization in the form of business messages in a business process is an invaluable tool in helping to achieve operational awareness. A separate category of products known as Business Activity Monitoring (BAM) has emerged to address the many issues of operational awareness. Auditing and tracking capabilities that are a base part of an ESB allow you to monitor and track the health of your business processes and the flow of messages throughout your SOA. Value-added services such as data caching, data collection and aggregation, and visual rendition of XML data can create a substrate for generating operational alerts, notifications, and reporting that can provide real-time insight into the state of your business as the data traverses your enterprise.
*Incremental Adoption: One of the primary differentiating qualities of an ESB is its ability to allow incremental adoption, as opposed to being an all-or-nothing proposition. In the post-Y2K spending slowdown, budgets for multimillion-dollar IT projects are just not what they used to be. There is some indication that IT funding is getting freed up to solve the short-term tactical integration needs, but budgets are still being highly scrutinized at an executive level. At the same time, however, there is still a desire to implement larger corporate-wide strategic initiatives—all of which rely heavily on integration and reuse of existing IT assets.


'''When Should an ESB be Used?'''<ref>Defining the ESB [http://www.oracle.com/technetwork/articles/soa/ind-soa-esb-1967705.html Oracle]</ref><br />
Are there best practices to determine when the implementation of an ESB is worthwhile? The use of an ESB is worth considering when three or more applications or services need to be integrated. A simple point-to-point integration is significantly easier and much more cost-effective when connecting two applications. An ESB can also be worthwhile if services are going to be incorporated from external service providers over which the company has no control. The ESB can then be used to monitor the service level agreements that the external provider guarantees. The impact of the adjustments to service contracts can further be kept to a minimum, as the ESB continues to provide a stable interface while making the necessary changes to the messages. If many different protocols, such as [[Hypertext_Transfer_Protocol_(HTTP)|HTTP]], SOAP, and [[File_Transfer_Protocol_(FTP)|FTP]], are to be used and standardized to one protocol like [[Simple_Object_Access_Protocol_(SOAP)|SOAP]], the ESB can perform the necessary protocol transformation. If services are to be consistently incorporated into an architecture to be able to receive, process, and produce messages, then the use of ESB is also suitable. This is also applicable if a collection of pre-defined components and adapters need to be accessed, which allows various protocols and legacy applications to be integrated in a standardized fashion. If messages need to be reliably and securely processed, the ESB can simplify the implementation of transactional message flows between two heterogeneous transactional data sources. Using an ESB can become problematic if large volumes of data need to be sent via the bus as a large number of individual messages. ESBs should never replace traditional data integration like [[Extract,_Transform,_Load_(ETL)|ETL]] tools. Data replication from one database to another can be resolved more efficiently using data integration, as it would only burden the ESB unnecessarily. An ESB should support stateless message flows if long-term business processes are to be implemented. Long-running business processes are stateful and can best be implemented using [[Business_Process_Execution_Language_(BPEL)|BPEL]] and/or [[Business_Process_Modeling_Notation_(BPMN)|BPMN]]. These are not usually available via the ESB, but rather via a [[Business_Process_Management_(BPM)|business process management system (BPMS)]].


'''Implementation of ESB'''<ref>Enteprise Service Bus - Implementation [https://www.mulesoft.com/resources/esb/what-esb Mulesoft]</ref><br />
The ESB architecture has some key principles that allow for business agility and scale. The key focus is to decouple systems from each other while allowing them to communicate in a consistent and manageable way.
*The "bus" concept decouples applications from each other. This is usually acheived using a messaging server like JMS or AMQP.
*The data that travels on the bus is a canonical format and is almost always XML
*There is an "adapter" between the application and the bus that marshals data between the two parties.
*The adapter is responsible for talking to the backend application and transforming data from the application format to the bus format. The adapter can also perform a host of other activities such as message routing transaction management, security, monitoring, error handling, etc.
*ESBs are generally stateless; the state is embedded in the messages passing through the bus.
*The canonical message format is the contract between systems. The canonical format means that there is one consistent message format traveling on the bus and that every application on the bus can communicate with each other


[[File:ESB.png|400px|Enterprise Service Bus]]<br />
source: [http://www.service-architecture.com/articles/web-services/enterprise_service_bus_esb.html Barry & Associates]


'''Challenges in ESB Implementations'''<ref>Challenges in ESB Implementations [http://techno-consulting.blogspot.com/2013/01/challenges-in-esb-implementations.html Techno-Consulting]</ref><br />
Often large enterprises have committed and invested heavily in buying vendor products. There are many cases of adoption that have either failed to give the benefits and even have created problems in implementations. Some of the causes and experiences are:
*Undefined Usage Patterns : Many enterprises buy vendor products with vendor documents that define product patterns for adoption and starts believing that the product patterns are the golden goose to chase. It is often short-sighted view showing no vision in adoption within the enterprise. With this gap in where & how to apply ESB within enteprises, Designers and Architects make un-informed decision for the sake of project timelines and other pressures. Many places where ESB are used where they are wrongly applied and some place they are left un-used loosing the opportunity for adoption. These decision aggravate and becomes monster, then some-one recognizes the problem but does not find the business support to fund the adoption or change. Hence, the ESB vision & strategy for the enterprise need to be documented and shared to teams to give clear design & usage patterns. In the absence of it, I see lengthier debates, prolonged decisions, and political pressures building in the enterprise, causing costlier adoption and scraping of initiatives.
*Noisy Design : There are different opinions about what kind of activities to be implemented in ESB design. Many vendor based ESB's provide higher level lanugage & constructs to create mediation flows. Design of mediation flows are usually corrupted to include business logic and un-justifiable dependencies. This makes the design Noisy and brittle.
*Adhoc decision : In many occasion project teams take adhoc decisions to meet timelines and also to accomodate team pressures. Avoid tactical decisions or keep the scope for re-factoring at later iterations if you are following agile methods.
*Product rigidity : Products are built on proprietary frameworks structured to meet common scenario. Any customization on it is like walking blind. Also there are heavy components as they come with built in functionality when you install. Performance demands are high on ESB's, where most proprietary products have challenges. Applying multithreaded processing is difficult to apply due to its inherent rigidity.  ESB's need to be lightweight and pluggable to meet highly distributed nature of its usage. Integrating ESB's in Continuous integration environment is also not well supported by vendor products, and the basic feature for unit test features are also difficult. This is the gap in which open source implementation score more and gives fleixibility in usage.  ESB's need to be lightweight and pluggable to meet highly distributed nature of its usage.
*Distributed Design challenges: As the ESB inherently is distributed in nature, one needs to be clear in how this distribution is applied and the techniques used to ensure reliability, perfromance and maintainability. ESB's are built on messaging backbone, the usage of messaging mechanism, to distribute & consolidate capabilities needs to come out from the strategic thinking. Often there are lack of understanding in teams on how this mechanism helps in distribution, posses big challenge. Associated with distributed challenge is end-to-end visibility of processing. Many problems are faced by teams who use message queuing systems, they complain issue of visibility and randomly adhoc behavior. I have seen in enterprises, support teams still tamper with messages in queues, deleting them in worst cases. Clearly these challenges need clear strategy, with more discipline and right processes supported by custom utilities will help the teams manage the challenges.
*ESB Governance : This has often been the most neglected area in ESB adoption. The challenge will be on how to bring all the key systems on to the ESB in a phased and incremental manner. This needs real implementation strategy. On the one hand there will be many projects wanting to use ESB, the other end there will be limited team capacity to build, there will be trade-offs to do. There are some cases of design corrections to do through re-factorings, support new interfaces etc. Enterprise IT teams are struggling to keep the momentum to meet business pressure. One option is to form an ESB centre-of-excellence that consist of Integration Analyst, Integration developers and designers, and Platform architects. Along with them the project manager coordinates activities with project teams and plans for implementation.
*Skill & Competency: This area often becomes the weak-link in the process. It is common saying in the enterprise that the platform skills and analysis skills fall short, and they become the core reasons for project delays, low quality deliverable, and higher spending. So it is important to identify and build compentency, retain skills, and encourage re-factoring. It also depends on which platforms you use as an ESB. Appserver, WMB, Opensource Mule ESB etc. All have there own programming models, deployment options and management tools. Hence the skills varies on platforms. Important roles that are a worth having are:
**Integration Analyst - The person who analyses details of the integrations, identifies data elements, transformations and rules, completeness of integration flows, dependency analyses. Important deliverable for this role will be Integration Spec.
**Integration Developer - The person is expert in platform tools and techniques, experienced in integration design patterns and have hands-on for implementations. He should be experienced in Webservices, XML, EAI patterns, messaging systems etc. He plays role of designer and developer, responsible for quality of deliverables and meeting timelines.
**Platform Architect - The person is experienced as System and Integration architect, responsible for overall design decisions meeting NFR requirements, ensure de-coupling and mediation are performed at right levels and Services are granular meeting enterprise standards. He is one who identifies design risks, and defined design requirements. The deliverable will be Integration design spec, articulating design decisions, key dependency interfaces, translation mechanism. He defines the integration topology, data & interface standards to follow and platform dependent mechanism to follow.


'''Benefits of ESB'''<ref>The Benefits of Enterprise Service Bus (ESB) [http://www.soabloke.com/2008/01/31/the-benfits-of-an-esb/ SOA bloke]</ref><br />
The key advantages of using an ESB are less about features and functions and more about how you use it.
*Standardization: One of the primary advantages of an ESB is that it gives you a standardized platform for integration. When everyone is using the same tools you can develop enterprise-wide frameworks, patterns and best practices for building re-usable services. Without a unifying platform, you get a divergence of integration methods which leads to inconsistency and higher cost of management and change. So an ESB platform helps with design-time governance. Note that this is not the same as standardization in the sense of using web-services standards. The important thing is that you use the ESB to support your own enterprise standards. These may be based on external standards – but that may be of secondary importance.
*Loose Coupling: The bus architecture of an ESB encourages you toward a loosely coupled architecture. Physically decoupled by making use of message passing mechanisms (e.g. JMS) versus direct network connections (e.g. HTTP). Semantically decoupled by use of message transformation so that services are exposed in a system-neutral format which reduces application lock-in and reduces the cost of change.
*Scalability and Reliability: Physical loose-coupling provides scalability advantages such as high-availability, fault-tolerance and load balancing. The messaging layer in the ESB directs messages between service endpoints to the appropriate instance of the endpoint. For example, in the event of a service provider failure, messages will be redirected to a backup provider – thus supporting high availability. In the case of load balancing, messages are distributed between redundant providers (or consumers) to handle high volumes of message traffic. You could say that physical loose-coupling supports change at the “micro” level where short term changes in the system topology can be compensated for via real-time message redirection.
*Routing and mediation: Message routing supports scalability and fault tolerance. An ESB can also be used to support business-level routing and mediation. For example content-based routing allows services to be invoked based on the content of a service request. A business example would be routing of a customer enquiry to the branch where that customer account is located. A technical example would be the routing of a service request based on the version of the service being invoked.
*Complex message exchange patterns: Traditional HTTP-based services support only one-to-one request-reply MEPs. An ESB supports more complex MEPs such as asynchronous one-way messaging and to multiple subscribers using topic-based messaging. Asynchronous publish and subscribe mechanisms support new ways of intermediating service consumers and subscribers – such as auditing, service monitoring – which are extremely useful for runtime management and governance of your services. Beyond mere governance, higher level business functions such as complex event processing (CEP) and [[Business_Activity_Monitoring_(BAM)|business activity monitoring (BAM)]] are supported by this ability to “listen in” to service traffic on the ESB.

The benefits of an ESB stem largely from the architecture of an ESB and in particular from the use of a message bus as the primary underlying transport. But it is important to understand that these benefits don’t automatically come “out of the box”. Your solution architecture (and your architects) must recognize and utilize the architectural principles underlying the ESB.


===References===
<references />


===Further Reading===
*Choosing the Right ESB for Your Integration Needs [https://www.infoq.com/articles/ESB-Integration InfoQ]
*ESB is an architectural style, a software product, or a group of software products? [http://www.consultoriajava.com/articulos/esb_arquitecture_software_product.jsp Consultoria Java]
*Rethinking the ESB: Building a simple, secure, scalable Service Bus with an SOA Gateway [http://www.computerworld.com/article/2510364/infrastructure-management/rethinking-the-esb--building-a-simple--secure--scalable-service-bus-with-a.html Computerworld]
*The Enterprise Service Bus, re-examined [https://www.ibm.com/developerworks/websphere/techjournal/1105_flurry/1105_flurry.html IBM]
*Microservices - Death of the Enterprise Service Bus (ESB)? [https://www.slideshare.net/KaiWaehner/microservices-death-of-the-enterprise-service-bus-esb-update-2016 Kai Waehner]