Canonical Data Models, ESBs, and a Reuse Trap

The original uphill battle

In “Reduce, Reuse, Recycle”, I discussed how reuse introduces additional complexity and cost as the scope expands (application to family of applications to extra-organization) and in “Coping with change using the Canonical Data Model”, I illustrated how that technique could be used to manage API changes without breaking existing clients. Last week on LinkedIn, an interesting question was posted that combined both reuse and the Canonical Data Model pattern – how do you create a service-oriented architecture around a canonical data model when the teams for the component applications are unaware of the benefits of reuse?

Initially, the benefit seems to be self-evident. If you have ten applications publishing messages to each other, this can involve up to 90 mappings per service method [n(n – 1)], 180 if the services are request-response. If, however, all ten applications use the same set of messages, you have one format per call (two for request-response). Things get sticky when you realize that you must now coordinate ten development teams to code an interface to your service bus (we’re not even considering the case of a third-party product). Assuming you manage to achieve that feat, things get stickier still when the inevitable change is needed to the canonical format. Do you violate the one true message layout rule or do you try to get ten application teams to deploy an update simultaneously?

Using the canonical data model pattern inside the service bus (the incoming native message is transformed to the canonical format on the way in and then to the appropriate outgoing native format on the way out) allows you to vary the external formats while reducing the number of mappings to a maximum 20 [2n] (40 if using request-response). Any logic inside the bus (e.g. orchestrations) retains the same benefit of operating on the canonical format without the constraint of having all clients constantly synchronized. Obviously, nothing prevents multiple applications from using the same format, so this method can reap the benefits of reuse without constraining change.

A successful service-oriented architecture will need to balance two potentially contradictory forces: stable service contracts for existing consumers and the ability to quickly accommodate new consumers. Using a canonical data model internally while allowing for a variety of external formats can allow you to meet both objectives with minimal complexity. Attempting to enforce a canonical model externally pushes changes onto all consumer applications regardless of need and will slow the pace of change.

Advertisements

8 thoughts on “Canonical Data Models, ESBs, and a Reuse Trap

  1. Great post, as usual, Gene. The canonical format issue is one I explored in my first master’s thesis, in an entirely different context. I was working on ISO 12620 at the time–a canonical format for terminology exchange. We made exactly the same claim that you do–that N-way interchanges are just too overwhelming to tackle. It’s ironic that I’ve forgotten this principle a few times in my career. The application to ESBs is very astute.

    Another way to reduce the complexity of version mgmt on messages on an ESB is something that I explored in detail in a series of blog posts a few years back. Have a look at http://codecraft.co/2008/07/29/decoupling-interfaces-as-versions-evolve-part-1/ and see what you think.

    Like

    • Thanks, Daniel.

      Service and messaging versioning is definitely an area where lots of work remains to be done. I like your proposal – sort of a networked IDispatch (your fault for mentioning COM in the series of posts 😉 ). Strict versioning provides very good performance at the cost of extra governance, but I think there are some definite scenarios where the mechanism you described could be very useful and worth the overhead.

      Like

  2. Excellent conclusion Gene that’s what i looking for when i started the thread in linkedin, i have a lot of work and i need to sell it, i see that you wrote about that in “Selling SOA” thanks in advance.

    Like

  3. Pingback: Dependency Management is Risk Management | Form Follows Function

  4. Pingback: Dependency Management is Risk Management | Iasa Global

  5. I discovered your blog site on google and test a number of of your early posts. Proceed to keep up the excellent operate. I simply extra up your RSS feed to my MSN Information Reader. Looking for forward to reading extra from you in a while! efbekgeekcee

    Like

  6. Pingback: links | friedesb

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s