Data - SOA's Last Mile

According to Wikipedia, “The last mile (or last kilometer) is the final leg of delivering connectivity from a communications provider to a customer. Usually referred to by the telecommunications and cable television industries, it is typically seen as an expensive challenge because ‘fanning out’ wires and cables is a considerable physical undertaking.” The significant complexity, cost, and risk associated with this situation delayed many broadband rollouts until new technology became available that could realize the “last mile.”

This powerful and real-world analogy from the world of telecommunications and cable TV succinctly summarizes the current state of Service Oriented Architecture (SOA) in the enterprise. It is widely agreed in concept that SOA has the potential to break down barriers between application silos and enable the reuse of business or application-level services across an enterprise. Hence, SOA has held great promise as the enabler of a new generation of more agile and cost-effective IT solutions. Unfortunately, the last mile has been the hardest for SOA, causing it to fail to live up to people’s high expectations. And that’s because, in their zeal to realize the benefits promised by early SOA hype, customers and technology providers have overlooked a crucial component for the success of any SOA initiative – they’ve ignored the data!

To illustrate this fact, let’s discuss the case of a large financial services provider that was looking to enhance customer service and enable up-sell and cross-sell opportunities across all their lines of business, such as banking, insurance, and mortgage. The company planned to achieve this goal by offering its customers the convenience of Web-based self-service banking that would offer significant competitive differentiation and increase customer satisfaction, which in turn would drive greater revenue and market share. To deliver on the company’s goals for increased agility, the IT organization built out an SOA that included a Web-based self-service portal to provide access to all customer accounts across all the lines of business in one place.

However, as the various SOA infrastructure components such as an SOA Registry, Enterprise Service Bus (ESB), etc., were being rolled out to the multiple lines of business, it was soon observed that enabling easy, secure, reliable, and timely access to customer account information across all the lines of business, in one place, posed huge data-centric challenges such as:

Thus holistic, accurate, and timely data is the most strategic enterprise asset. Although the applications and business processes that consume data maybe architected with modern approaches like SOA they are only as efficient as the data that is exchanged through them.

Throughout its highly complex lifecycle, data goes through a series of sophisticated treatments as it gets created, processed, and consumed by different applications and business processes for various operational and analytic purposes. As data is exchanged, SOA creates interdependencies between applications and business processes demanding high-quality, consistent, and timely data, suggesting that the full benefits of SOA can’t be realized if IT projects don’t incorporate a well-planned Enterprise Information Management (EIM) strategy to address the widespread and expensive problem of data inconsistency and inaccuracy.

While traditional approaches to SOA, based on a simple Web Services paradigm, address high-level application integration and business process orchestration needs, they tend to minimize, or worse yet ignore, the complexities of heterogeneous, inconsistent, dirty data that lies fragmented throughout the enterprise. Furthermore, technologies such as Enterprise Application Integration (EAI) and Enterprise Information Integration (EII), frequently used as the underpinnings of SOA, can’t effectively deal with varying data volume and data latency needs.

In this article, we’ll discuss the last mile issue in SOA, namely, overcoming the hidden data-centric pitfalls that prevent SOA from delivering on its promise to ensure business agility. We will also showcase how IT organizations can enhance SOA’s inherent capabilities for flexibility, responsiveness, and reuse with a scalable and sophisticated data integration technology that makes data available as a service, or data services. With a data services technology in the mix, IT organizations can look to maximize the business value of their extensive investments in SOA by ensuring the availability of holistic and accurate information at the speed of business.

Right-Time Availability of Enterprise Data Is Key to Agility
If we look around, we can see that powerful market forces such as globalization, mergers and acquisitions, regulatory compliance, fierce competition, tight operational budgets, and increased demand for improved customer service are pushing businesses to become more agile. To deliver increased business agility, enterprise IT organizations have been consistently trying to build infrastructures capable of flexibility, responsiveness, and reuse. However, the real question to ask is whether the infrastructure is supplying the data the business needs, and when, where, and how it needs it to help the company stay agile and competitive. Can the IT infrastructure handle all the complexity of enterprise data?

Holistic, accurate, and timely business-critical information is the lifeblood of any enterprise and the key driver for any business looking to gain, maintain, and grow the customer’s business and trust. To drive competitive advantage in today’s dynamic environments, businesses are increasingly trying to leverage all their information to support mission-critical applications such as consolidation of customer data to support a call center and delivery of forecasts for supply chain operation optimizations. Additionally, businesses are adopting industry standards like SWIFT in the financial services industry, ACORD in insurance, and HL7 in healthcare to exchange information with their partners. These scenarios will require all forms of enterprise data – structured unstructured, or semi-structured – be constantly accessed, manipulated, and used by more users through more applications, exactly when needed, be it batch, near real-time or real-time.

Thus, businesses need to be able to leverage all their enterprise data, holistically, accurately, and at the right time to gain business advantage. So IT organizations are looking for a scalable and flexible data integration technology that can complement its existing infrastructure investment and seamlessly handle any form of data, eliminate the complexity of the data integration “hairball,” and deliver timely data for efficiently responding to changing business demands. With a sophisticated and flexible enterprise information management strategy that treats data as a strategic asset and effectively exploits all the information contained in various silos spread across the enterprise, businesses can significantly increase their agility.

The Current SOA Landscape
As we’ve seen in the previous section, the ability to use up-to-date information in reducing time-to-market and increasing the speed of rolling out new and differentiated functionality for competitive advantage is what the modern enterprise is all about. However, developing and maintaining the right infrastructure to support these capabilities is crucial to enabling enterprises to achieve business agility.

As the first step towards agility, businesses need to leverage existing and new applications, business processes, and data in their mission-critical functions such as finance, supply chain, and customer management. In many organizations, well over 50% of the IT budget is devoted to building and maintaining points of integration among these systems. CIOs and IT managers are constantly confronted with the following challenges:

Integrating fragmented applications is a big challenge facing IT and business, with tangible business implications. Business agility increasingly depends on a global view of customers, suppliers, products, and partners — an ideal not achievable without integration.

SOA has emerged as the leading technology for enabling a new generation of more flexible and cost-effective IT solutions. SOA offers a highly flexible layer of abstraction for delivering the following compelling business benefits:

SOA provides an elegant solution for IT to deal with the complexity of application integration in the enterprise. As shown in Figure 1 the current SOA landscape consists of delivering an effective abstraction layer for dealing with application integration using technologies such as Enterprise Application Integration (EAI), Business Process Management (BPM), and Enterprise Service Buses (ESB).

However, to enable agility in the enterprise efficiently, the underlying IT infrastructure must also support the right-time provisioning of holistic and accurate data that lies fragmented all across the enterprise. Implementers that embark on an SOA path quickly realize that SOA presents the following data-centric challenges that can’t be handled by their SOA platform providers:

Information being the key to business agility, it’s critical that the application and business process layers can access holistic and accurate data when it’s desired, wherever it resides, in whatever form is required, consistently and accurately.

Thus, the payoff for SOA as the enabler of business agility in the enterprise will be great, if implemented with upfront emphasis on right-time data integration.

The Need for a Data Delivered as a Service to SOA
Applications and business processes interconnected as services for flexibility aren’t worth anything if the data in them is inaccurate, inconsistent, or not available exactly when needed. If the data stuck inside siloed applications is bad, imagine the calamity when, through an application and business process integration technology, the silos disappear and data from many different applications is commingled.

Without a complementary, standards-based, and flexible technology that delivers holistic, accurate, and timely data as a service to an SOA, typical end-user challenges may include:

Therefore, far from making data issues irrelevant, the underlying IT infrastructure poses an actual risk to enterprises that lack a right-time data integration strategy. SOA provides a flexible framework by breaking down barriers between silos of applications and enabling the orchestration of business processes using flexible and reusable application-level or business services. If SOA’s inherent capabilities for flexibility, responsiveness, and reuse are enhanced with the holistic, accurate, and timely information, business agility can definitely be achieved.

What’s needed is a scalable data services technology built on a sophisticated data integration platform that can provide a standards-based data abstraction layer to an SOA, and deliver holistic, accurate, and timely data as a service (see Figure 2). A data service is a modular, reusable, well-defined, business-relevant service that enables the access, integration, and right-time delivery of enterprise data throughout the enterprise and across corporate firewalls. However, unlike application-level or business services, data services are specialized services that are more granular, enable loose-coupling with data sources, are data-centric, and are purpose-built to enable right-time data integration in an SOA.

The data services technology should also be capable of orchestrating or sequencing atomic data services to enable flexible combinations and reuse of sophisticated data integration tasks across projects. As shown in Figure 3, atomic data services must enable sophisticated data integration tasks such as access, profiling, cleansing, transformation, and delivery of data that can be sequenced in various combinations to support the entire data integration lifecycle in an enterprise.

Ideally, a data services technology needs to deliver these sophisticated capabilities to an SOA:

Finally, as performance and scalability are at the heart of any mission-critical operation, the data services technology should also be able to handle high concurrency and large data volumes that are typical requirements of such demanding environments.

SOA & Integration Competency Centers
An Integration Competency Center or ICC is an infrastructure of people, technology, policies, best practices, and processes focused on rapid, repeatable, and cost-effective deployment of data integration projects critical to meeting organizational objectives. Organizations have found that there’s a direct connection between the caliber of their ICC and their ability to respond quickly to dynamically changing business models, intense competition, and demanding customers. In short, the ICC is the infrastructure responsible to deliver trustworthy data flexibly and at the speed of the business.

Data services technology encourages the creation and management of ICCs by standardizing development processes on common technology standards for greater reuse, supporting a shared services environment, supporting a centralized services environment, and enabling the governance processes of data services and data architecture so that definitions, semantics, and SLAs are maintained by the appropriate parties.

An ICC enabled by a data services technology is thus the perfect enabler for data integration in an SOA since it provides the adaptive architecture and technology foundation required to access and deliver holistic, accurate, and timely data throughout the enterprise (see Figure 4). It also delivers the collaboration tools necessary to align cross-functional teams through a single technology, increasing responsiveness to the changing needs of the business. Finally, it provides a secure, scalable, and responsive platform to guarantee access to mission-critical information to support various projects across the enterprise.

Conclusion
As we’ve seen in this article, a sophisticated and enterprise-grade data services technology can address SOA’s last mile by providing the cornerstone in an SOA for ensuring the availability of holistic and accurate business-critical information at the speed of business.

Resources

© 2008 SYS-CON Media