Leverage XSD annotations to build richer XML schema

In the service-oriented enterprise IT landscape, XML schema is the default language used to describe information models. Products and IT departments today care less about the domain-model, and view the systems as a collection of services, orchestrated to perform business functions. And how do you describe and document your services? Unlike domain-models which can be expressed in very well in-terms of UML class diagrams, ER diagrams there is not a popular formal notation to design or describe your service portfolio.

This gap is partly addressed by the several SOA governance tools which govern the lifecycle of service definitions and implementation. You can use these tools to capture information such as the status, usages, dependencies, SLAs, security aspects associated with a service along with a broad description of the service itself.

While this is extremely important and needed, it does not solve the entire problem. Somewhat neglected in this solution is the description the service request and response objects. It is a common assumption that since services are described using WSDL and therefore XML Schema and eventual XML – the messages are self-describing. But more often than not the XML Schema definition of the XML message does not convey enough information about the message.

How do you use XML schema to convey conditional composition of a message, what each specific element means, how it maps to existing terms/concepts in the business model, business validation logic applicable on elements etc?  For example consider the following XML schema

<xsd:element name=”Address”>
<xsd:complexType>
<xsd:element name=”Line1″ type=”xsd:string”/>
<xsd:element name=”Line2″ type=”xsd:string”/>
<xsd:element name=”Line3″ type=”xsd:string”/>
<xsd:element name=”City” type=”xsd:string”/>
<xsd:element name=”District” type=”xsd:string”/>
<xsd:element name=”State” type=”xsd:string”/>
<xsd:element name=”Region” type=”xsd:string”/>
<xsd:element name=”Country” type=”xsd:string”/>
<xsd:element name=”Zipcode” type=”xsd:string”/>
<xsd:element name=”pincode” type=”xsd:string”/>
<xsd:element name=”landmark” type=”xsd:string”/>
<xsd:element name=”latitude” type=”xsd:numeric”/>
<xsd:element name=”longitude” type=”xsd:numeric”/>
</xsd:complexType>
</xsd:element>

The above is a fairly decent  description of a “Address” element which can be applied to most geographies. Now let us say a CreateCustomer service uses this address element as part of a NewCustomer message.

<xsd:element name=”Customer”>
<xsd:complexType>
<xsd:element name=”name” type=”xsd:string”/>
<xsd:element name=”age” type=”xsd:integer”/>
<xsd:element ref=”Address”/>
</xsd:element>

Great. At this point I seem to know what I should do to create a customer. Use a request with the customer name, age and address details. A simple as that.  But now when we take a look at the address schema – which of the Address elements need to be populated to correctly create a customer? A US address would need a ZIPCode, Indian addresses need a PIN code, but yet it is optional if you enter City, District and Line3 (which should be Street information btw for Indian addresses.). How do I inform the service consumer all this? A PDF user-manual would be needed here. WSDL + XML Schema does not tell the poor programmer everything he needs to get the job done well enough.

One approach to make your schema’s more powerful – is using comments and annotations. A XML schema document with comments

<xsd:element name=”Address”>
<xsd:complexType>
<xsd:element name=”Line1″ type=”xsd:string”/>
<xsd:element name=”Line2″ type=”xsd:string”/>
<!– should contain “Street name” in India –>
<xsd:element name=”Line3″ type=”xsd:string”/>
<xsd:element name=”City” type=”xsd:string”/>
<xsd:element name=”District” type=”xsd:string”/>
<!– only valid 2 letter codes for US address, full state name for indian –>
<xsd:element name=”State” type=”xsd:string”/>
<xsd:element name=”Region” type=”xsd:string”/>
<xsd:element name=”Country” type=”xsd:string”/>
<!– must for US, UK, Britain –>
<xsd:element name=”Zipcode” type=”xsd:string”/>
<!– needed for indian addresses if city/state/description are left empty –>
<xsd:element name=”pincode” type=”xsd:string”/>
<xsd:element name=”landmark” type=”xsd:string”/>
<xsd:element name=”latitude” type=”xsd:numeric”/>
<xsd:element name=”longitude” type=”xsd:numeric”/>
</xsd:complexType>
</xsd:element>

While the above comments are very helpful, it is a very flaky solution.

A better approach would be to leverage XSD annotations which can be used to specify additional meta-data about the schema elements. For example,

<xsd:element name=”Line3″ type=”xsd:string”>
<xsd:annotation>
<xsd:documentation>3rd address line. Do not use this element if more specific element e.g. State, Country, City is already available. Use only for free-text identification. </xsd:documentation>
</xsd:annotation>
</xsd:element>

However, while this seems more structured, it still degenerates to being equivalent of xml comments in the schema file. It is left to the schema author to describe the rules well enough, and since the documentation is embedded in the schema itself, it becomes very difficult to keep the business rules current – considering the schema is going to be copied and used in multiple places.

A better approach is to use the xsd:appInfo element within the xsd:annotation. The appInfo element can contain a well-defined XML and can be used to provide a more structure meta-data to the XML schema user. In addition the contents of the appInfo element can be made available to the application which is processing  XML instances based on the schema – and this can be very useful.

You can refer for example to business rules defined in say an external rules repository

<xsd:element name=”Line3″ type=”xsd:string”>
<xsd:annotation>
<xsd:appInfo>
<cust:RuleSet xmlns:cust=”http://myenterprise/it9/rule-engine”&gt;
<cust:RuleID>rules:crm:address:7</cust:RuleID>
<cust:RuleID>rules:crm:address:8</cust:RuleID>
<cust:RuleID>rules:crm:address:15</cust:RuleID>
</cust:RuleSet>
</xsd:appInfo>
</xsd:annotation>
</xsd:element>

Alternatively, you can use a language such as schematron to define additional constraint / validation logic.

<xsd:element name=”Line1″>
<xsd:annotation>
<xsd:appinfo>
<sch:pattern name=”validateLine1″>
<sch:rule context=”ns1:Address”>
<sch:assert test=”/ns1:Line1″>
Line1 is must
</sch:assert>
</sch:rule>
</sch:pattern>

Along with validation, business rule logic – the xsd:appInfo element can be extremely useful to capture mapping information in XML Schema. For example,

<xsd:element name=”Line3″ type=”xsd:string”>
<xsd:annotation>
<xsd:appInfo>
<cust:Mappings xmlns:cust=”http://myenterprise/mapping/v10.4″&gt;
<cust:Map cust:mappingContext=”SiebelCRM”>
<cust:MapTarget>ADDRESS_TBL.LINE_THREE</cust:MapTarget>
</cust:Map>
<cust:Map cust:mappingContext=”Salesforce_Address”>
<cust:MapTarget>/sf:addressdetails/sf:addresslinethree</cust:MapTarget>
</cust:Map>
</cust:Mappings>
</xsd:appInfo>
</xsd:annotation>

Having made this case, it needs to be mentioned that there is limited support to the xsd:annotation element in most schema design tools. For example none of the schema authoring tools provide a feature to associate pre-defined templates for annotations.  Without such support and lack of ability to verify that the appInfo elements are correctly defined for a project’s requirement – there are challenges in using this approach.

Few links/references on xsd:annotations and xsd:appinfo

http://www.w3schools.com/schema/el_appinfo.asp
http://www.w3schools.com/schema/el_annotation.asp

| Tagged , , | 2 Comments

Large XML processing

Was reading a bit about options available for processing large XML document. To constrain the use-case: What is the best way to process a large XML document (over 1 GB in size) which basically is a XML representation of a collection of records.

If the “process” involves reading each record, doing something with it and then discarding it.  SAX should be a good option. The SAX parser fires events for every XML node being processed and the event handler code can then construct every record as  it is read and hand it over for processing.

If the “process” involves reading each record, doing something with it and then say modifying some content in the file (say a flag):   Stax would be my API of choice since SAX can’t write and DOM loads the whole document into memory.

Stax also differs from Sax in the sense that it basically is a pull-based model – where the application code loops through the document looking for nodes which match certain criteria. (Whereas in SAX, the parser takes over control of the program flow and calls the application handlers).

If the “process” requires multiple searches on the document based on various search criteria, occasional modifications to parts of the document

Stax does not work very well – because the entire document needs to be scanned for every search and update.

Option 1 : DOM.  But this causes a memory foot-print 5 times the document size. Also if the document updates need to be saved – the entire DOM object has to be written to disk. A part of the XML file cannot be modified using DOM.

Option 2 : VTD-XML. http://vtd-xml.sourceforge.net/userGuide/0.html A relatively new API (well it’s apparently been around since 2004) which allows a indexed read-write access to the XML document. It uses a non-extractive style of parsing the document : which means  that while parsing the document instead of loading the entire content into memory, it loads the offset+size of each XML element it encounters into a index. A good article on non-extractive parsing is here – http://www.xml.com/pub/a/2004/05/19/parsing.html

Once parsed, this index is ready to traversed, either using a top-down descent, or using Xpath, or using a index-lookup.   The index itself can be persisted to a file and can be used for subsequent processing of the document without need for parsing the document.

Also writes are also efficient since the only the contents of the modified element can be written-to without the need for re-parsing the entire document.

A note on how VTD-XML compares with other XML APIs http://vtd-xml.sourceforge.net/userGuide/5.html

[ i have not yet written any code using VTD-XML, planning to get my hands dirty sometime this week. will upload the snippets]

| Tagged , , , , | Leave a comment

Logging in Java – which one to use?

Recently I got into a situation where I needed to choose a logging API or library to be used in a Java/J2ee based enterprise s/w platform layer.

There are a number of choices available for this, all of which pretty much do the same thing.

  • Apache Commons Logging (ACL)
  • Java Util Logging (JUL)
  • Simple Logging Facade for Java (SL4J)
  • Log4J

Firstly these all logging APIs basically provide

  • Ability to define multiple loggers for an application identifed by some namespace
  • Ability to define different logging levels for different loggers from configuration
  • Ability to change the logging destination without effecting code.

Here is a little detail on why there are so many logging options, and when to chose which.

Log4J was the first widely adopted logging API. It became very popular and was widely used. http://logging.apache.org/log4j/1.2/

Java Util Logging then came along and thus logging became a core Java feature. This kind of made Log4J unnecessary. However the API is slightly different in terms of the log levels, and input arguments – so a lot of people continue to use Log4J. A lot of new applications use Java Util Logging since it does not need any additional libraries etc. Neat. http://docs.oracle.com/javase/1.4.2/docs/api/java/util/logging/package-summary.html

Apache Commons Logging provided a abstract logging interface over the logging APIs. So applications can write to ACL APIs which in turn could be configured to use either Java Util or Log4J at deployment/run time. Applications (such as libraries) which cared about the need for such flexibility chose Apache Commons Logging. However ACL relied on some rather difficult understand classloader hacks which caused some applications to crash for hard to debug reasons! http://commons.apache.org/logging/

SL4J then came along and seems to fix some of these issues. It provides a lightweight facade over the logging APIs in a straight-forward manner (without any classpath magic). It also offers bridges to route logs written by components directly to other logging APIs into the SL4J logs. http://www.slf4j.org/

I feel moving forward applications should chose between Java Util Logging and SL4J. JUL is good enough for most use cases. If you are writing libraries etc and really care a lot about keeping your code abstractes from implementation APIs – then you may want to use SL4J.

 

 

| Tagged | Leave a comment

Follow me on Twitter – @SaratBuddhiraju

https://twitter.com/#!/SaratBuddhiraju

Posted in Uncategorized | Tagged | 1 Comment

AMQP – Advanced Message Queueing Protocol

AMQP is language neutral application-layer wire-protocol for asynchronous messaging. The protocol supports flow-control, guaranteed message delivery, transactions, routing, security. Rabbit MQ is a popular implementation of this protocol.

but I already have JMS?

One key difference between AMQP and JMS – is that the JMS is a API spec. whereas AMQP is a wire spec. Thus AMQP allows for greater inter-operability. i.e. AMQP client developed by vendor X can send messages to another AMQP client developed by vendor Y, without need for any bridge to translated one vendor’s wire-message to another.

Quote from wikipedia : http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol

QUOTE “AMQP mandates the behaviour of the messaging provider and client to the extent that implementations from different vendors are truly interoperable, in the same way as SMTP, HTTP, FTP, etc. have created interoperable systems. Previous attempts to standardize middleware have happened at the API level (e.g. JMS) and this did not create interoperability[2]. Unlike JMS, which merely defines an API, AMQP is a wire-level protocol. A wire-level protocol is a description of the format of the data that is sent across the network as a stream of octets. Consequently any tool that can create and interpret messages that conform to this data format can interoperate with any other compliant tool irrespective of implementation language.” UNQOUOTE

Also AMQP offers much richer functionality than JMS in terms of routing/addressing etc.

 

What next?

If you are building a distributed application which needs messaging (read pub-sub / guaranteed delivery etc)  – consider and evaluate using AQMP as the messaging protocol.

Also, instead of just pushing XML in and out of these brand new queues – do you want to think about JSON / Google Protocol Buffers for serializing data to/from a language neutral message.

 

| Tagged , , , | 2 Comments

WS-BusinessActivity and BPEL

I was reading a bit of details about transactions, locking, distributed transaction etc till I stumbled upon a specification named WS-BusinessActivity / WS-BA. (Refer specification at this link)

On first read it seemed to have a big functional overlap with BPEL in terms of ability to support long running transactions(LRT) using activities and compensation.

But dig a little deeper, and it is interesting to note that BPEL and WS-BA have to alternate approaches for solving the problem.

While in BPEL the orchestrating service (application flow) is responsible for managing the ensuring the consistency of the LRT by performing compensation activities. The BPEL and thus the application needs to know the semantics of both the business activities and  compensation/cancel activities of any remote services it is invoking.

On  the other hand in WS-BA all  participants share the responsibility of ensuring the LRT consistency by registering with a central transaction coordinator and performing a compensation when the coordinator directs them to do so.  Thus none of the parties involved in the global transaction need to be aware of the semantics of transaction specific complete, compensation/cancel operations of other parties.

In fact, by leveraging WS-BA as an underlying protocol,  BPEL flows with long running transactions can be designed with no significant compensation flows involving external services.

| Tagged , , , , | Leave a comment

Challenges with the canonical schema design pattern

The canonical XML schema design pattern is a popular approach to solved application integration requirements.

http://en.wikipedia.org/wiki/Canonical_schema_pattern

The basic idea being that XML schemas/DTD are used to define a common data-model of objects and messages which are used to defined application-API independent service contracts.

While it is a great idea to use canonical XML schema, a few things need to be considered.

  1. Development Cost :- Defining and maintaining the common  data-model is a significant SOA Governance exercise.  Existing applications, future needs, industry trends need to analysed
  2. Versioning :- Change is inevitable. The canonical schema definition needs to undergo changes. There is a need to implement a versioning strategy for the canonical.
  3. Complexity :- Given two or more application data-models, coming up with a canonical data-model is a non trivial task. The problem lies in the fact that a data-model design (canonical or not) implicitly defines certain business rules and constraints. Thus when a canonical data-model needs to be created, along with the vocabulary / terms in the two data-models, the implicit business rules need to be reconciled. This requires extensive business process analysis.
  4. Usability :- When XML Schema is used to express the canonical data-model, a significant amount of the business process analysis that has gone into defining the data-model is typically lost.  For example, it is not straight-forward to express in XML schema something like “While a SalesOrder can have multiple Order-lines, System A can support only SalesOrders with a single Order-line, System B can support SalesOrders with upto 100 Order-lines but only 1 priority order-line per SalesOrder”. The canonical XML schema by definition needs to support the more generic use-case, SalesOrders with multiple Order-lines.  So when both System A and System B try to expose their service-contract for a CreateSalesOrderService using the common canonical SalesOrder XML Schema, the canonical does not express their respective service interfaces accurately.
  5. Run-time cost :- When a canonical XML schema is involved a request needs to be transformed first to a canonical format and then to the target provider’s format before being processed. Considering the performance overhead inherent in XML based processing – an additional transformation hop may make the canonical unviable for several high performance systems.
All these costs and issues with using the canonical approach need to be considered before using a canonical XML schema for integration. There would be situations where you are better-off to directly expose the service providers to consumers without putting a canonical XML schema based service contract layer on top of them.
| Tagged , , | Leave a comment

Hello World!

“Welcome to WordPress.com. This is your first post. Edit or delete it and start blogging!”

WordPress creates this post by default.  Just to quench my curiosity – a brief history of “Hello, World” http://en.wikipedia.org/wiki/Hello_world_program

 

Posted in Uncategorized | Tagged | 1 Comment