Mason Draft 2 ready for use

I am happy to announce that Mason (the JSON + hypermedia format) is now ready for use in Draft 2. See https://github.com/JornWildt/Mason/blob/master/Documentation/Mason-draft-2.md.

For those that haven't heard about Mason before - please take a look at https://github.com/JornWildt/Mason. Mason is a JSON based format with conventions for representing API data and hypermedia control elements.

This version combines links, actions and link templates into one single @controls object. As the name indicates this object contains all the hypermedia elements that control the application.

This move to @controls makes it possible to combine link templates with POST data - and should make it easier to parse and represent hypermedia elements in code.

There are currently no further ideas in the pipeline that could change how things are represented in Mason. Future versions may though add new features.

Feedback is as always appreciated :-)

Representing an issue tracker with Mason

Two weeks ago I introduced Mason - a media type for representing data with embedded hypermedia elements. Now I would like to go through an example implementation of a fictive issue tracker which uses Mason to represent its data.

At its core the issue tracker has, not surprisingly, "issues" that represent issues that needs to be solved. Issues are organized in projects and may have one or more file attachments associated with them. Issues are represented by their title, a description and a severity level (from 1 to 5).

It would be natural to include comments on issues too but I want to keep the domain as simple as possible while still being able to illustrate all the features of Mason - and comments would not add anything but clutter as they are very similar to attachments.

Please note that the issue tracker will be defined without reference to any existing implementation thus making it a "real" REST service totally independent of any specific implementation. I do have a reference implementation available but that is only used as a proof of concept - it is not part of the issue tracker definition itself.

The issue tracker is defined in terms of data types, links, link templates and actions as described in the next sections. That is not exactly interesting reading so you might want to skip it and jump to the later sections with examples of how to actually use the issue tracker.

You should try the demo issue tracker yourself with the generic Mason browser in order to see how actions and URL templates are supposed to work.

Data types, links and actions

CURIE definitions

In the following the CURIE name "is" should be expanded to "http://soabits.dk/mason/issue-tracker/reltypes.html#". So for instance "is:add-issue" becomes "http://soabits.dk/mason/issue-tracker/reltypes.html#add-issue" (you can GET that).

These URL relationship types ensures that we have no name collisions with other relationship types. In addition to this it is actually possible to dereference the URLs and GET some documentation about them.

Data types

Issue

A single issue consists of the following properties:

  Id: int
  Title: string
  Description: string
  Severity: integer

Expected links:

• self
• up: link to parent project
• is:attachments: link to collection of attachments for issue
• is:common: link to common resource data

Example resource: http://mason-issue-tracker.cbrain.net/issues/1

Issue collection

A collection of issues has one top level property:

  Issues: array of
  - Id: int
  - Title: string

Expected links:

• self
• up: link to parent project
• is:common: link to common resource data

Example resource: http://mason-issue-tracker.cbrain.net/projects/1/issues

Project

A single project consists of the following properties:

  Id: int
  Code: string
  Title: string
  Description: string

Expected links:

• self
• is:issues: link to collection of issues for project
• is:common: link to common resource data

Expected actions:

• is:project-update
• is:add-issue
• is:project-delete

Example resource: http://mason-issue-tracker.cbrain.net/projects/1

Project collection

A collection of projects has one top level property:

  Projects: array of
  - Id: int
  - Code: string (a short code or abbreviation of the project name)
  - Title: string

Expected links:

• self
• is:common: link to common resource data

Example resource: http://mason-issue-tracker.cbrain.net/projects

Common resource data

A set of properties and links which are common to all resources:

  Title: string (title of the whole issue tracker)
  Description: string (description of the whole issue tracker)

Expected links:

• self
• is:contact: link to contact information
• is:logo: link to issue tracker logo
• is:projects: link to collection of all projects
• is:common: link to common resource data

Expected link templates:

• is:issue-query

Expected actions:

• is:project-create

Example resource: http://mason-issue-tracker.cbrain.net/resource-common

Contact information

Contact information (related to the owner of the issue tracker) consists of:

  Name: string
  Address1: string
  Address2: string
  PostalCode: string
  City: string
  EMail: string
  Phone: string
  Country: string

Expected links:

• self
• alternate (alternate address representations in other formats like for instance text/vcard)
• is:common

Example resource: http://mason-issue-tracker.cbrain.net/contact (take a close look at the "alternate" link as it shows how to have multiple targets for one link).

Link relations

is:projects

Link to collection of all projects.

is:issues

Link to collection of issues for a given project.

is:attachments

Link to collection of attachments for a given issue.

is:contact

Link to contact information.

is:logo

Link to issue tracker logo.

is:common

Link to data common for all resources in the issue tracker.

Link templates

 

is:issue-query

A link template for querying issues. Parameters:

  text: any text to look for in issues.
  severity: severity level (1-5)
  pid: project ID.

Example usage: http://mason-issue-tracker.cbrain.net/resource-common

Actions

is:project-create

Action for creating a new project. Arguments:

  Code: string
  Title: string
  Description: string

Example usage: http://mason-issue-tracker.cbrain.net/resource-common

is:project-update

Action for updating a single project. Arguments:

  Code: string
  Title: string
  Description: string

Example usage: http://mason-issue-tracker.cbrain.net/projects/1

is:project-delete

Action for deleting a project. Has no arguments.

Example usage: http://mason-issue-tracker.cbrain.net/projects/1

is:add-issue

Action for adding a new issue to a project. Arguments:

  Title: string
  Description: string
  Severity: int
  Attachment: object of
  - Title: string
  - Description: string

In addition to this it is possible to pass a file as an attachment to the issue. The file name is "attachment".

The "Attachment" object contains additional information about the attached file.

Example usage: http://mason-issue-tracker.cbrain.net/projects/1

is:issue-update

Action for updating a single issue. Arguments:

  Title: string
  Description: string
  Severity: int

Example usage: http://mason-issue-tracker.cbrain.net/issues/1

is:issue-delete

Action for deleting a single issue. Has no arguments.

Example usage: http://mason-issue-tracker.cbrain.net/issues/1

is:add-attachment

Action for adding an attachment to an issue. Arguments:

  Title: string
  Description: string

In addition to this it is possible to pass a file as the actual attachment. The file name is "attachment".

Example usage: http://mason-issue-tracker.cbrain.net/issues/1

Examples

Getting started

The first thing a client must do in order to work with the issue tracker is to GET the "common" resource that contains useful links, templates and actions for the issue tracker. The common resource can be thought of as the "home page" or "landing page" of the issue tracker.

Try it yourself: GET http://mason-issue-tracker.cbrain.net/resource-common

Creating a new project

Once we have a copy of the "common" resource we can look for the action "is:project-create". That action will tell us how to encode the project data and how to submit it to the server.

Here is an example:

"@actions": {
  "is:project-create": {
    "type": "json",
    "href": "http://mason-issue-tracker.cbrain.net/projects",
    "title": "Create new project",
    "schemaUrl": "http://mason-issue-tracker.cbrain.net/schemas/create-project"
  }
}

The most important parts are the "type" and "href" properties which tells us how to encode the data and where to send it. Mason actions also have a "method" property for identifying the HTTP method to use but it defaults to POST so its not always needed.

The type "json" tells us to encode the action arguments in plain JSON. The net result is a request like this:

POST /projects HTTP/1.1
Accept: application/vnd.mason+json
Content-Type: application/json

{
  "Code":    "SHOP",
  "Title":    "Webshop",
  "Description":    "Project for issues related to the webshop"
}

The response is a redirect to the created project:

HTTP/1.1 201 Created
Location: http://mason-issue-tracker.cbrain.net/projects/2

Adding a new issue

Now that we have a project we can start adding issues to it (with optional attachments). Each project representation contains an "is:add-issue" action for this purpose as can be seen here:

"@actions": {
  "is:add-issue": {
    "type": "json+files",
    "href": "http://mason-issue-tracker.cbrain.net/projects/2/issues",
    "title": "Add new issue to project",
    "schemaUrl": "http://mason-issue-tracker.cbrain.net/schemas/create-issue",
    "jsonFile": "args",
    "files": [
        {
            "name": "attachment",
            "description": "Attachment for issue"
        }
    ]
  }
}

This action tells us the following:

• The type is "json+files" which means we must send the JSON data together with some files wrapped in the media type multipart/form-data.
• The target URL is "http://mason-issue-tracker.cbrain.net/projects/2/issues".
• The JSON data must conform to the schema definition at "http://mason-issue-tracker.cbrain.net/schemas/create-issue".
• The JSON data must be contained in a multipart element named "args".
• The attached file must be contained in a multipart element named "attachment".

The net result is a request like this:

POST /projects/2/issues HTTP/1.1
Accept: application/vnd.mason+json
Content-Type: multipart/form-data; boundary=d636dfda-b79f-4f29-aaf6-4b6687baebeb

--d636dfda-b79f-4f29-aaf6-4b6687baebeb
Content-Disposition: form-data; name="attachment"; filename="hogweed.jpg"
... binary data for attached image ...

--d636dfda-b79f-4f29-aaf6-4b6687baebeb
Content-Disposition: form-data; name="args"; filename="args"
Content-Type: application/json

{
  "Title":    "Hogweeds on the plaza",
  "Description":    "Could you please remove the hogweeds growing at the plaza?",
  "Severity":    5,
  "Attachment":
    {
      "Title":        "Hogweed",
      "Description":        "Photo of the hogweeds."
    }
}

Updating project details

Each project has a "is:project-update" action for updating the project details:

"@actions": {
  "is:project-update": {
    "type": "json",
    "href": "http://mason-issue-tracker.cbrain.net/projects/1",
    "title": "Update project details",
    "template": {
      "Code": "SHOP",
      "Title": "Webshop",
      "Description": "All issues related to the webshop."
    }
  }
}

This action tells us to encode project arguments in JSON and POST it to "http://mason-issue-tracker.cbrain.net/projects/1". The JSON data should be build from the JSON template in the action.

Request:

POST /projects/1 HTTP/1.1
User-Agent: API Explorer
Accept: application/vnd.mason+json
Content-Type: application/json

{
  "Code": "SHOP",
  "Title": "Web shop",
  "Description": "All issues related to the new web shop."
}

Deleting a project

Each project has a "is:project-delete" action for deleting the project and its related issues:

"@actions": {
  "is:project-delete": {
    "type": "void",
    "href": "http://mason-issue-tracker.cbrain.net/projects/1",
    "method": "DELETE",
    "title": "Delete project"
}

Request:

DELETE /projects/1 HTTP/1.1
Accept: application/vnd.mason+json

Response:

HTTP/1.1 204 No Content

Searching for issues

The "common" resource has a link template for issue queries:

"@link-templates": {
  "is:issue-query": {
    "template": "http://mason-issue-tracker.cbrain.net//issues-query?text={text}&severity={severity}&project={pid}",
    "title": "Search for issues",
    "description": "This is a simple search that do not check attachments.",
    "parameters": [
      {
        "name": "text",
        "description": "Substring search for text in title and description"
      },
      {
        "name": "severity",
        "description": "Issue severity (exact value, 1..5)"
      },
      {
        "name": "pid",
        "description": "Project ID"
      }
    ]
  }
}

This templates tells us to replace the parameters "text", "severity" and "pid" into the URL template "http://mason-issue-tracker.cbrain.net//issues-query?text={text}&severity={severity}&project={pid}".

Should we for instance want to query for issues of severity 5 in project 1 then we would get this request:

GET /issues-query?text=&severity=5&project=1 HTTP/1.1
Accept: application/vnd.mason+json

The result is a collection of issues.

Implementing hypermedia APIs and REST services with Mason

I am happy to announce that I have taken all the lessons learned during the last few years and stuffed it into a new JSON based mediatype for hypermedia APIs and REST services. The media type is application/vnd.mason+json or simply "Mason". There is an IANA registration for it pending.

With Mason you get hypermedia elements for linking and modifying data, features for communicating to client developers and standardized error handling. Mason is built on JSON, reads JSON, writes JSON and generally fits well into a JSON based eco-system.

Here is a simple example illustrating how a single issue from a fictive issue tracker could be represented in Mason. It contains the basic API data like issue Title, Description and Severity and then it adds hypermedia elements for linking to other related resources and actions for writing stuff back to the issue tracker.

{
  // Classic API data
  "ID": 1,
  "Title": "Program crashes when pressing ctrl-p",
  "Description": "I pressed ctrl-p and, boom, it crashed.",
  "Severity": 5,
  "Attachments": [
    {
      "Id": 1,
      "Title": "Error report",
      // Hypermedia linking to attachment
      "@links": {
        "self": {
          "href": "http://issue-tracker.org/attachments/1"
        }
      }
    }
  ],

  // Additional hypermedia links
  "@links": {
    // Hypermedia linking to self
    "self": {
      "href": "http://issue-tracker.org/issues/1"
    },
    // Hypermedia linking to containing project
    "up": {
      "href": "http://issue-tracker.org/projects/1",
      "title": "Containing project"
    },
  },

  // Hypermedia "action" element for creating a new project
  "@actions": {
    "is:project-create": {
      "type": "json",
      "href": "http://issue-tracker.org/mason-demo/projects",
      "title": "Create new project",
      "schemaUrl": "http://issue-tracker.org/mason-demo/schemas/create-project"
    }
  }
}

Those that are familiar with HAL may recognize some parts of the format. That is expected as Mason builds on the ideas from HAL. HAL was never intended to have hypermedia elements for writing stuff so I decided to go for it and design a new format based on HAL.

The Mason specification, online example and stand-alone API explorer are available from https://github.com/JornWildt/Mason.

Design goals

My design goals with Mason are:

1. It should be easy to adopt in existing JSON based solutions and have a low barrier of entry for new developers.

2. It should contain hypermedia elements sufficient for both reading and writing data without any out-of-band information.

3. It should contain elements for information directed to client developers for the purpose of improving "API developer experience".

4. It should contain error elements sufficient for most kinds of applications.

5. It should work with JSON when both reading and writing.

Let me dig into each of those design goals one by one.

1. Easy to adopt

A classic JSON payload makes the raw API data directly accessible as JSON object properties. I believe it should be so too when working with hypermedia enabled APIs. So Mason merges hypermedia elements into existing JSON structures. To avoid name collisions Mason property names are prefixed with a '@'.

Mason can be adopted gradually:

Step 1: Change content type to application/vnd.mason+json instead of application/json.

Step 2: Add a @meta property with additional information targeted at client developers.

Step 3: Use links to remove client knowledge of server defined URLs.

Step 4: Use Mason's error format.

Step 5: Use actions to truly decouple client and server implementations.

2. Hypermedia for both reading and writing

Hypermedia has a lot of benefits as I wrote in http://soabits.blogspot.dk/2013/12/selling-benefits-of-hypermedia.html. Among these is the ability to remove a client's dependency on server URL structures using links.

But links are only good for, well, linking resources together - they don't say anything about how to change and modify API data. So Mason adds "actions" for writing API data.

An action defines both target URL, HTTP method and action type (payload encoding). With this information being discoverable at runtime it is no longer necessary to hard code clients with information about HTTP method and how to encode the payload. This means client and server only have to agree on WHICH data to send - not HOW to send it.

3. Information targeted at client developers

One of the great things about hypermedia enabled APIs is the ability to explore the API using a browser of some kind. As I wrote in http://soabits.blogspot.dk/2013/12/selling-benefits-of-hypermedia.html; Do not underestimate the power of an explorable API. The ability to browse around the data makes it a lot easier for the client developers to build a mental model of the API and its data structures.

And if client developers are browsing the API why not also be able to communicate directly with them? Mason adds a few meta data elements for sending messages directly to the client developers. An API browser should highlight these such that devs can instantly read some documentation and comments about the resource they are currently looking at.

At the same time Mason defines a technique for removing this client developer information from the payload in production.

4. Error handling

By standardizing error handling Mason makes it possible for clients to interact with unknown services and still be able to communicate error conditions clearly to end users.

I have previously discussed error handling here in http://soabits.blogspot.dk/2013/05/error-handling-considerations-and-best.html and apparently that article hit a nerve somewhere because it keeps attracting a lot of attention (for an amateur blogger like me).

5. JSON read/write

One of things that annoys me about the traditional key/value forms based on application/x-www-form-urlencoded is that there are no standards for encoding complex data structures. Neither does it define any standard for encoding booleans, integers and other basic data types. The consequence is that client and server needs to agree on these things before they can start talking about business data - and different servers are surely going to implement different encoding schemes - all in all making life miserable for developers that just want to get stuff done.

By using JSON Mason ensures interoperability on some of the lower levels. JSON defines more types than simple string based key/value formats and handles structures like objects and arrays.

Restricting implementations of Mason to handle JSON only reduces design choices and variations and thus improving the chances of things working out of the box (compared to simple string based key/value formats).

Transcending from web APIs to REST services

Most web APIs today are defined in terms of a single server implementation around which developers build dedicated clients (think "Twitter" or "Facebook"). In such a world clients have a strong coupling to server URL structures, HTTP methods, error formats and other quirks of the API. These APIs were never designed to be implemented by more than one organization (and are for this reason also called "snowflake APIs").

True REST services on the other hand are defined without reference to any specific server implementation. The best known example of this is the ATOM format which enables clients to interact with any ATOM enabled service on the web - no matter who implemented it, where it is hosted or what URL structures it is implemented with. The enabling factor for this is the ATOM media type specification.

But ATOM is restricted to feed-like data and does not fit well with other applications. So other media types are needed and Mason is an attempt to fill out this space. Mason attempts to facilitate complete decoupling from technical implementation details such that clients can discover HOW to interact with service at runtime.

One of my earlier blog posts discussed this problem in more detail: http://soabits.blogspot.no/2013/05/the-role-of-media-types-in-restful-web.html

Data profiles

Mason itself does not prescribe any business specific details. Clients and servers still have to agree on WHAT data to interchange - but Mason do remove the technical coupling on HOW to interchange the data.

Mason depends on profiles to enable clients to know WHAT data they are looking at. You can find an in-depth discussion about it here: http://soabits.blogspot.no/2013/12/media-types-for-apis.html.

At the time of writing I haven't put profiles into the specification yet.

Further reading

Mason homepage: https://github.com/JornWildt/Mason

Generic Mason browser (API explorer): https://github.com/JornWildt/Mason/wiki/Generic-Mason-browser

Online live example of fictive issue tracker using Mason: https://github.com/JornWildt/Mason/wiki/Example-service%3A-issue-tracker

/Jørn

Media types for APIs

I have previously touched upon the concept of media types (see http://soabits.blogspot.no/2013/05/the-role-of-media-types-in-restful-web.html), but somehow it has always been difficult for me to really nail the concept down in a concise and useful article.

Now the latest discussion about the benefits of hypermedia (see http://soabits.blogspot.no/2013/12/selling-benefits-of-hypermedia.html) got me thinking about media types again - but this time in the perspective of unique service implementations with dedicated clients versus large scale ecosystems of mixed implementations.

As it turns out, media types doesn't mean sh*t on a small scale. That kind of explains why it has been so difficult to get to some kind of consensus about media types for APIs.

Background

When the discussion touches upon media types the arguments usually follow these lines:

• Completely generic media types like JSON and XML should be avoided since they do not include any kind of hypermedia elements.
• One school of thought argues that we should have very few (generic) media types. This is to avoid the need for clients to understand too many media types.
• Another school of thought argues that we should have many different domain specific media types. Otherwise the client wouldn't know what kind of resource it was looking at.

But, as I said, it really doesn't matter. Both schools are right. At least when you look at unique service implementations with dedicated clients - like for instance dedicated Twitter clients.

Let me give you a concrete example from the Twitter API (see https://dev.twitter.com/discussions/5662): the return value from their oauth/request_token "endpoint" is key/value pairs encoded as application/x-www-form-urlencoded - but the server says it is "text/html" which is clearly wrong. Does that break any client implementations? No. Why? Because all clients are dedicated to the Twitter API; they KNOW about this little peculiarity and has been hard coded to work with it.

My point is:

Media types are irrelevant for unique service implementations with dedicated clients. In this world the client always knows exactly what it is doing and what kind of result to expect from the server (and it can safely ignore the media type).

Media types on a large scale

Let us broaden our view and look at the example of "Big corporation buys smaller companies and the result is a big unruly combination of customers, sales orders and other stuff living on different systems" which I introduced in my previous blog post (http://soabits.blogspot.no/2013/12/selling-benefits-of-hypermedia.html).

Now lets assume our fictive client is handed a link/URL to a customer resource in this mess of a heterogeneous mix of different company resources. The client can issue a GET on the URL and in return it will receive a stream of bytes. How does the client interpret those bytes? Obviously it will depend on the media type. But which kind of media type is useful for this purpose?

Let us assume the client understand a generic (hypermedia enabled) media type like HAL. Together with the GET request the client sends an accept header "Accept: application/hal+json". Luckily the server knows how to serve the customer resource as HAL, so the client gets a HAL document in return.

Now what? We have integrated customer resources from three different organizations and each of these have been encoding customer records in HAL - but in different ways.

For instance: Company X has these customer properties:

{
  ID: 1234,
  Name: "John Larsson",
  Address: "Marienborg 1, 2830 Virum, Denmark"
}

while company Y uses these properties:

{
  ID: 1234,
  FirstName: "John",
  LastName: "Larsson",
  Address:
  {
    Address: "Marienborg 1",
    PostalCode: "2830",
    City: "Virum",
    Country: "Denmark"
  }
}

With nothing but this information our client must either give up or do some guessing like "If FirstName is present then assume format of company Y". So apparently we need a bit more information than we already have.

Now we can either choose to add some kind of profile to the representation - either as a header or in the payload - or we can use a domain specific media type.

1) A profile in the payload could be done like this:

{
  ID: 1234,
  profile: "http://company-x.com/profiles/customer-care",
  ... other properties ...
}

2) The profile could also be part of the media type, so we would get "application/hal+json;profile=http://company-x.com/profiles/customer-care".

3) A domain specific media type could be something like "application/company-x.customer-care.hal+json" or similar.

But which method should we choose? Lets take a look at how the client process the server response before we answer that.

Processing a server response

There are three things the client must know in order to process a server response correctly:

1. How to decode the byte stream (generic knowledge).
2. What the data represents (domain specific knowledge).
3. How to locate hypermedia elements in the response (generic knowledge).

The media type is obviously the key to decoding the byte stream - it will tell the client whether it is looking at XML, PDF, HTML, HAL, Sirene and so on.

The media type should also be the key to locating hypermedia elements in the response.

But what about the domain specific knowledge - should we identify what a resource represents with a domain specific media type or with a profile? Both methods work, but there is one more thing to take into account: making the API explorable by client developers (see http://soabits.blogspot.no/2013/12/selling-benefits-of-hypermedia.html).

It is of course possible to implement a browser for any domain specific media type we can think of, but it would obviously be more practical if we could have one single API browser for all kinds of APIs. For this reason we should avoid domain specific media types. The domain knowledge can then be identified by a profile - either in the payload or in a HTTP header.

Wrapping it all up

As with the hypermedia problem: if you stick to unique service implementations with dedicated clients (like a dedicated Twitter client) then media types are utterly irrelevant. The client can safely assume that there will be one, and only one, representation of what ever kind of resource it is looking for.

But if you take broader perspective and venture into a highly heterogeneous, loosely coupled, unorganized, incoherent and fragmented ecology (also called "The internet") - then you need more domain specific information about the resources - either through domain specific media types, or generic media types with profiles.

My recommendation is:

1. Use generic media types that include hypermedia elements.
2. Identify domain specific information through profiles.

The media type will tell the client HOW to decode the byte stream and HOW to interact with the resource. The profile will tell the client WHAT it is looking at.

The role of media types in RESTful web services

One of the never ending discussions in the REST community is that of custom and domain specific media types; should we, or should we not, create new media types - and if we should, for what reasons should it be done?

In this blog post I will discuss the role of media types in web services and illustrate it with an example media type. I will go through the requirements for this media type and from this I will build up the features it needs to support. Together with this I will show some example scenarios and sketch out the processing algorithm for the client side. At last I compare this media type to other similar media types (HAL, Sirene, JSON-API).

My goals for this blog post are:

1. To improve my own understanding of the role of media types in RESTful web services - and share that with others.
2. To define a new media type for what I call systems integration - and show how it facilitates loose coupling between the integration components.

By systems integration I mean the kind of background processing that takes place behind the scenes in almost any IT enabled business today; shuffling data from one system to another in a safe and durable way without any human interaction.

REST seems like a good fit for systems integration. It has a strong focus on loosely coupled systems where servers and clients can evolve independently of each others; if we can leverage that then the whole ecosystem of multiple servers and clients should be a lot easier to maintain and with much less downtime required for upgrading the various components.

There is an ongoing trend to include hyper media controls in never web services; that is a good trend as it removes the clients dependency on specific URL structures. This in turn allows the server to evolve by adding new resources and link to these - and it also facilitates the ability to use multiple servers without the clients ever noticing (since the client do not care about either URL path structures or host names).

But there is still a thing missing in the puzzle. In Roy Fielding's (in)famous rant "REST APIs must be hypertext-driven" he states:

... Any effort spent describing what methods to use on what URIs of interest should be entirely defined within the scope of the processing rules for a media type

... From that point on, all application state transitions must be driven by client selection of server-provided choices that are present in the received representations

Especially the last statement is interesting "all application state transitions must be driven by client selection of server-provided choices". This means the client should not make any requests without first being instructed to do so (and how to do it). The client should not POST a new Tweet, bug report or similar without being instructed, on the fly, by some mechanism embedded in the server responses. Todays use of links in responses is on the right track, but links do not inform the client about what HTTP method to use (it assumes GET) and neither does it say anything about the possible payload.

With this blog post I will try to explain how a media type, with a sufficient number of hyper media controls, together with some intelligent client side code, can enable what Fielding is describing. The downside of this approach is that client implementations become more complex - the upside is that the whole client/server application becomes much more loosely coupled which, in the end, hopefully will help us reach a maintenance Nirvana of loosely coupled systems integration :-)

By the way, I am not comparing REST with SOAP/WSDL and EDA (event driven architectures) - that is not the purpose here even though these are often found in systems integration projects. I would rather just explore what benefits we can get from REST.

Media type requirements and constrains

The primary driver for this new media type is loose coupling where the clients only depends on the media type and some out-of-band business specific data structures and identifiers. This means:

• The client must not make any assumptions about URL structures.
• The client must not make any assumptions about what concrete service implementation it is interacting with.
• The client must not initiate any HTTP request without following instructions embedded in server responses (besides the initial request).
• The client should not be given more than:
  • A root URL from which all other resources must be discovered at runtime.
  • A set of business specific data structures.
  • A set of well known identifiers for locating hyper media controls and business data.

The media type itself must be generic with respect to the business domain; it must not contain references to concepts like medical records, e-commerce and so on.

The media type must be rich enough in terms of hyper media affordances to enable all the operations needed for systems integration.

The media type does not need to included much, if any, in terms of UI elements since it is intended for operations without human interaction. Neither is the media type intended for mobile use where bandwidth and message size is a concern.

The media type will be based on JSON. It could just as well be based on XML but, in my experience, JSON is lot simpler to work with, fits the data needs I have met, and has a simple and easy-to-work-with patch format (application/json-patch) which will come in handy later on.

Armed with these constraints and requirements we are ready to build up our new media type.

Example business domain "BugMe"

Through out this blog post I will use the imaginary open standard "BugMe" for interacting with bug tracking systems through the new media type. BugMe supports adding of new bug reports, attaching documents to reports, adding comments to reports and similar features shown later on.

BugMe is not a part of the media type specification - it is only used to illustrate how the media type facilitates interaction with BugMe servers anywhere on the web.

Neither is BugMe a vendor specific "standard", it is strictly defined in terms of the generic media type and a set of bug reporting specific data structures and identifiers (more on that later on).

Compare this to APIs like Twitter and others; these are always defined in terms of vendor specific resources and explicit URL structures and was never designed to be implemented on servers anywhere else on the web.

To highlight the difference between a standard like BugMe and an actual implementation I will assume that some clever guy named Joe, who studies computer science 101 at Example.edu, has set up a BugMe server for some local study project. He is using an implementation that uses a vocabulary slightly different from  BugMe - it talks about "issues" where BugMe talks about "bug reports". This fact is illustrated through the concrete URLs used in the examples . The root URL is http://example.edu/~joe/track.

Example 1 - Creating a bug report

The first thing we will try is to create a new bug report with BugMe. To do so we must supply our client with a few details about the operation:

• The root URL: http://example.edu/~joe/track/index.
• A "create bug report" identifier (as defined by BugMe): "http://bugme.org/names/create-bug-report".
• Bug reporting data (as defined by BugMe)
  • Title: "Something bad happened",
  • Description: "I pressed ctrl-alt-del and all went black",
  • Severity: 5

We must also have an identifier for the media type. Lets call it it "application/razor+json" for no specific reason.
Now we are ready to set our client loose and make it create the bug report. It will do so in the same manner as a human working with a web based UI: get a resource representation, look for well known identifiers that labels data and hyper media controls, fill out data and activate hyper media controls.

This interaction pattern, getting a resource representation and following instructions on the fly, has a price: it requires more complex client side logic than "normal RPC" patterns with design time binding of methods and it results in higher bandwidth due to the embedded hyper media controls. The upside is a much looser coupling between clients and serves. But all of this is of course already discussed in Fielding's thesis on REST ;-)

GET initial resource

At the very beginning our client has nothing to do but GET the root URL in hope of finding something useful there:

Request
GET /~joe/track/index
Accept: application/razor+json

Response
Content-Type: application/razor+json

{
  curies:
  [
    { prefix: "bug", reference: "http://bugme.org/names/" }
  ],
  controls:
  [
    ...,
    {
      type: "link",
      name: "bug:create-bug-report",
      href: "http://example.edu/~joe/track/add-issue",
      title: "Add issue to issue tracker"
    },
    ...
  ]
}

The returned JSON data contains two top level properties defined by the media type: curies and controls. "curies" define short names for URLs used as identifiers in the other elements (see http://www.w3.org/TR/curie/) and "controls" contains various hyper media controls. The use of curies should be optioinal - but it helps reading the responses in posts like this.

Now the client scans the "controls" element looking for the identifier "bug:create-bug-report". In this case it finds a "link" control which is equivalent to an ATOM link. Since our client understands all the features of the media type it will know that a link should be "followed" by issuing a HTTP GET on the "href" value.

This little "algorithm" is equivalent to what a human would do: open up a webpage, look for instructions on how to perform the task at hand and then follow them.

You may have noticed the dots "..." in the example. Those are there for a reason: they illustrate how the client only cares about stuff that is relevant to its current task. Anything else in the response is ignored. The consequence is that the server is free to evolve the content of the resource over time without breaking any clients - as long as it only adds new stuff. Neither does the client care if the content is supposed to be a "link page", a service index, a medical record or have any other specific "type" - as long as it contains elements that will help the client getting closer to its goal.

Follow link

Here we have the next operation:

Request
GET /~joe/track/add-issue
Accept: application/razor+json

Response
200 Ok
Content-Type: application/razor+json

{
  curies: ...,
  controls:
  [
    {
      type: "poe-factory",
      name: "bug:create-bug-report",
      href: "http://example.edu/~joe/track/add-issue",
      title: "Create new idempotent POE resource"
    }
  ]
}

Bingo! This time the client finds an "poe-factory" control with the right name "bug:create-bug-report" and now its time to create the bug report. The control type "poe-factory" means "Post Once Exactly factory" and is a special action element that enables idempotent POST operations. If you do not know what "idempotent" means then take a look at this page: http://www.infoq.com/news/2013/04/idempotent.

The good thing about idempotent operations is that they can safely be repeated if anything goes wrong on the network. If an operation times out the client can simply retry it again without the risk of creating the same entry multiple times. And since this new media type is for safe and durable "behind the scenes" work I find it rather important to include a mechanism for idempotent POST operations.

The implementation chosen here requires the client to do an empty POST first. This will create a new POE resource (thus the name "poe-factory") and redirect the client to it. The client can then POST to the new resource as many times it needs until the operation succeeds. The server returns "201 Created" first time it completes the operation whereas it returns "303 See Other" on following requests. In either case the server includes a "Location" header pointing to the new POE resource.

Subbu Allamaraju has a nice blog post on post once exactly techniques.

I chose this approach for the following reasons:

• It has the simplest possible client side logic - at the cost of an extra round trip to the server. A similar solution could have required the client to create a GUID (message ID) and include it in the payload somehow, but that would make the protocol slightly more prone to client side errors.
• It requires no special headers.
• It adds no extra information to the payload.
• URLs are opaque and the server gets to choose how the POE/message ID is encoded.

Create POE resource

In order to complete its task the client first issues an empty POST operation to the URL of the "href" attribute:

Request
POST /~joe/track/add-issue
Content-length: 0

Response
201 Created
Location: http://example.edu/~joe/track/add-issue/bd925-ye174h

GET POE resource

It should be rather obvious now that the client has no choice but to follow the response:

Request
GET /~joe/track/add-issue/bd925-ye174h
Accept: application/razor+json

Response
400 Ok
Content-Type: application/razor+json

{
  curies: ...,
  controls:
  [
    {
      type: "poe-action",
      name: "bug:create-bug-report",
      documentation: ... some URL ...,
      method: "POST",
      href: "http://example.edu/~joe/track/add-issue/bd925-ye174h",
      type: "application/json",
      scaffold: ... any JSON object ...,
      title: "Add issue"
    }
  ]
}

Now the client gets a response with a "poe-action" control. This tells the client that it can safely POST as many times it needs to the "href" URL. The actual payload is given by the BugMe specification (Title, Description, Severity).

Some comments on the above response:

1. The payload is encoded in application/json as a trivial JSON object. Other formats may be included in the media type spec later on.
2. This format is NOT intended for automatic creation of UI's and thus it contains no UI related list of field definitions or similar.
3. It is NOT necessary to embed any kind of schema information - that sort of thing is given by the name of the control element.
4. The optional "scaffold" value is the JSON payload equivalent of a URL template: it supplies default values to some properties and adds additional "hidden" properties the client can ignore (as long as they are sent back).
5. POE-actions are not restricted to POST - a PATCH with json/patch would work as well (but then perhaps we need to change the action type name).

Create bug report

Then the client issues a new request:

Request
POST /~joe/track/add-issue/bd925-ye174h
Accept: application/razor+json
Content-Type: application/json

{
  Title: "Something bad happened",
  Description: "I pressed ctrl-alt-del and all went black",
  Severity: 5
}

Response
201 Created
Location: http://example.edu/~joe/track/issues/32

GET created bug report

Now we are done unless we want to see the actual created bug report by following the Location header:

Request
GET /~joe/track/issues/32
Accept: application/razor+json

Response
Content-Type: application/razor+json

{
  curies: ...,
  controls: ...,
  payloads:
  [
    ...,
    {
      name: "bug:bug-report",
      data:
      {
        Id: 32,
        Title: "Something bad happened",
        Description: "I pressed ctrl-alt-del and all went black",
        Severity: 5,
        Created: "2012-04-23T18:25:43Z"
      }
    },
    ...
  ]
}

Now that the client can see the actual bug report it wanted to create it knows that the task is completed. Everyone is smiling and put on their happy face :-)

Other hyper media controls

There are of course more scenarios to cover than this single "Create stuff" scenario and these scenarios will call for other kinds of hyper media controls, for instance URL templates, PATCH actions, binary file upload and more (I should cover these in some future blog posts ...)

Error handling

If the client receives a 4xx or 5xx status code it can inspect the JSON payload and look for a property named "error" together with the other "payloads" and "controls" properties. The "error" property should contain data according to my previous blog post on error handling.

Here is an example:

Request
POST /~joe/track/add-issue/bd925-ye174h
Accept: application/razor+json
Content-Type: application/json

{
  Title: "Something bad happened",
  Description: "I pressed ctrl-alt-del and all went black",
  Severity: 5
}

Response
503 Service Unavailable
Content-Type: application/razor+json

{
  error:
  {
    message: "Could not create new bug report; server is down for maintenance",
    ...
  }
}

In addition to this the client can try to use content negotiation to receive error information in the format of application/api-problem+json.

Client side processing algorithm

Here is a simplified view of how the client should process the content:

1. GET initial root resource.
2. [LOOP:] Look for hyper media controls with appropriate names.
3. Check the type of the found control element:
   1. If it is a "link" then follow that link and restart from [LOOP].
   2. If it is a "poe-factory" then issue an empty POST to the href value and restart from [LOOP].
   3. if it is a "poe-action" then issue a request with the specified method and data encoded according to the "target" media type. Then restart from [LOOP].
4. Look for a payload with the appropriate name: If it exists then the task is complete - otherwise it has failed (actually I don't like this last step, but that is the only kind of "acknowledge" I can see the server responding with).

A consequence of this approach is that the service specification (BugMe in my example) should state nothing about how to find and update data since that is up to the servers actual implementation. The service specification should only consider what kind of data to look for or modify. The "how"-part is contained entirely in the returned hyper media controls.

As the media type evolves and more types of hyper media controls are added the client(s) will grow more and more complex. This is one of the trade offs that has to be accepted in order to keep clients and servers as loosely coupled as possible.

If the media type gets popular one could even expect to see the same scenario we see with todays web browsers: there will be multiple implementations of the client libraries and some will implement more than others of the final specification.

No profile needed

It may be tempting to allow for a "profile" parameter with the media type ID. But typically that would be used to ask for a specific "type" of a resource like for instance "application/razor+json;profile=user". As can be seen in the client side processing algorithm above there is no need for such a thing, so lets not introduce it.

Related work

Quite a few other people are trying to create new media types to reach similar goals, but neither of them include features such as POE semantics. Here is the list of related media types that I am aware of:

• HAL
• Sirene
• JSON API
• Restful Objects
• JSON Home

And then there is Jim Webber's fantastic "How to GET a cup of coffee" which has been a big inspiration for me over the years.

Reasons for creating a new media type

How many media types should we invent? Well, as many as needed, I would say. The media type described here includes some features not found in other media types (POE semantics for instance) and that should be sufficient argument for creating a new one.

I don't see anything wrong by creating many media types - eventually a few of them will be good enough and gain enough traction to become ubiquitous standards. That's called evolution.

Summary

In this blog post I have tried to explain one way of understanding media type's role in RESTful web services and illustrated it by building up (parts of) a media type for systems integration. I have also touched upon the issue of "typed" resources and how to avoid it (by not assuming anything about the resource type and instead look for certain identifiers in the response) ... there could be a blog post more to come on this issue.

So what do you think? Was this useful, understandable, totally overkill, outright naive or simply a pile of, well, rubbish? Feel free to add a comment, Tweet me or send me an e-mail. I would love to get some feedback.

Happy hacking, Jørn

UPDATE 2014-02-24: I have actually put much of this into a media type called Mason. See http://soabits.blogspot.dk/2014/02/implementing-hypermedia-apis-and-rest.html.

Error handling considerations and best practices

A recurring topic in REST and Web API discussions is that of error handling (see for instance https://groups.google.com/d/topic/api-craft/GLz_nNbK-6U/discussion or http://stackoverflow.com/questions/942951/rest-api-error-return-good-practices]; what information should be included in error responses, how should HTTP status codes be used and what media type should the response be encoded in? In this blog post I will try to address these issues and give some guidelines based on my own experience and existing solutions.

Existing solutions

Let us first take a look at some existing solutions to get started:

• The twitter API uses a list of descriptive error messages and error codes. Twitter has both JSON and XML representations with property names: "errors", "error", "code"
  
• The Facebook Graph API has  a single descriptive error message, an error code and even a sub-code. Facebook uses a JSON representation with property names: "error", "message", "type", "code" and "error_subcode".
  
• The Github API has a top level descriptive error message and a optional list of additional error elements. The items in the error list refers to resources, fields and codes. Github uses a JSON representation with property names: "message", "errors", "resource", "field", "code".
  
• The US White House has a set of guidelines for its APIs on GitHub. The error message used here contains the HTTP status code, a developer message, a user message, an error code and links to further information.
  
• Ben Longden has proposed a media type for error reporting. This specification includes an "logref" identifier that some how refers to a log entry on the server side - such a feature can help debugging server errors later on.
  
• Mark Nottingham has introduced "Problem Details for HTTP APIs" as an IETF draft. This proposal makes use of URIs for identifying errors and is as such meant as a general and extensible format for "problem reporting".

All of these response formats share some similar content: one or more descriptive messages, status codes and links to further information. But as can be seen there is a wide variety in the actual implementation and wire format.

Considerations and guidelines

So, what should you do with your web API? Well, here are some considerations and guidelines you can base your error reporting format on ...

Target audience

Remember that your audience includes both the end user, the client developer, the client application and your frontline support (which may just happen to be you). Your error responses should include information that caters for all of these parties:

• The end user needs a short descriptive message.
• The client developer needs as much detailed information as possible to debug the application.
• The client application needs error codes (HTTP status codes) for error recovery actions.
• The frontline support people needs detailed information and/or keywords to look for in their knowledge database.

Use the HTTP status codes correct

The HTTP status codes are standardized all over the web and your clients will know immediately how to handle them. Make sure to use them correct:

• Do NOT just return HTTP status code 200 (OK) regardless of success or failure.
• Use 2xx when a request succeeds.
• Use 4xx when a request fails and the client should be able to fix it by modifying its own request.
• Use 5xx when a request fails due to some internal server error.

Use descriptive error messages

Be descriptive in your error messages and include as much context as possible. Failure to do so will cost you dearly in support later on: if your client developers cannot figure out why their request went wrong, they will look for help - and eventually that will be you who will spend time tracking down client errors instead of coding new and exiting features for your service.

If it is a validation error, be sure to include why it failed, where it failed and what part of it that failed. A message like "Invalid input" is horrible and client developers will bug you for it over and over again, wasting your precious development time. Be descriptive and include context: "Could not place order: the field 'Quantity' should be an integer between 0 and 99 (got 127)".

You may want to include both a short version for end users and a more verbose version for the client developer.

Localization

Error messages for end users should be localized (translated into other languages) if your service is already a multi language service. Personally I don't think developer messages should be localized: it is difficult to translate technical terms correct and it will make it more difficult to search online for more information.

When localization is introduced it may also be necessary to include language codes and maybe even allow for a list of different translations to be returned in the error response.

Allow for more than one message

Make it possible to include more than one message in the error response. Then try to collect all possible errors on the server side and return the complete list in a single response. This is not always possible - and requires some more coding on the server side (compared to simply throwing an exception first time some invalid input is detected).

Additional status codes

If your business domain calls for more detailed information than can be found in the normal HTTP status codes then include a business specific status code in the response. Make sure all of the codes are documented.

You may be tempted to include more technical error codes, but consider who your audience is for that: It won't help your end user. It may help your client application recovering from errors - but probably not in any way that was not already covered by the HTTP status codes. Your client developer may have some need for it - but why make them lookup error codes in online documentation when you can include descriptive error text and links that refers directly to the documentation? It may help your support - but if the client dev have enough information in the error response they won't need to call your support anyway - right?

Use letters for status codes

I often find myself searching for online resources that can help me when I get some error while interacting with third party APIs. Usually I search for a combination of the API name, error messages and codes. If you include additional error codes in your response then you might want to use letters instead of digits: it is simply more likely to get a relevant hit for something like "OAUTH_AUTHSERVER_UNAVAILABLE" than "1625".

Include links to online resources

Include links to online help and other resources that will either clarify what went wrong or in some other way help the client developer to solve the problem.

Support multiple media types

If your have a RESTful service that allows both client applications and developers to explore it then you might want to support a human readable media type for your error responses. HTML is perfect for this as it allows the client developers to view the error information righ in their browsers without installing any additional plugins. A fallback to plain text could also be useful (but probably overkill).

Include a timestamp or log-reference

It can help support and bug hunting if the error report contains a timestamp (server timezone or UTC). This may help locating the right logfile entries later on.

Another possibility is to include some other kind of information that refers back to the logfiles such that server developers and support people can track what happened.

Field-by-field messages

In some cases it makes sense to be explicit about the fields in the input that caused the errors and include field names in separate elements of the error response. For instance something like this JSON response:

{
  message: "One or more inputs were not entered correctly",
  errors:
  [
    { field: "Weight", message: "The value if 'Weight' exceeds 100 - the value should be between 0 and 100" },
    { field: "Height", message: "A value must be entered for 'Height'" }
  ]
}

This would make it possible for the client to highlight those fields in the UI and draw the end users attention to them. It is although difficult to keep clients and servers in sync and requires a lot of coding on both sides to get it to work. Usually field-by-field information is handled by client side validation logic anyway. So a clear error message like "The value of 'Weight' exceeded 100 - the value should be between 0 and 100" should be enough for most applications.

Include the HTTP status code

This may sound a bit odd, but according to people on api-craft there are some client side environments where the application code do not have access to the HTTP headers and status codes. To cater for these clients it may be necessary to include the HTTP status code in the error message payload.

Do not include stack traces

It may be tempting to include a stack trace for easier support when something goes wrong. Don't do it! This kind of information is too valuable for hackers and should be avoided.

Implementation

Now that we have our "requirements" ready we should be able to design a useful solution. Lets first try to define the response without considering an actual wire format:

• message (string): the primary descriptive error message - either in the primary language of the server or translated into a language negotiated via the HTTP header "Accept-Language".
• messages (List of string): an optional list of descriptive error messages (with the same language rules as above).
• details (string): an optional descriptive text targeted at the client developer. This text should always be in the primary language of the expected developer community (that would be English in my case).
• errorCode (string): an optional error code.
• httpStatusCode (integer): an optional copy of the HTTP status code.
• time (date-time): an optional timestamp of when the error occurred.
• additional (any data): a placeholder for any kind of business specific data.
• links (List of <string,string,string>): an optional list of links to other resources that can be helpful for debugging (but should probably not be shown to the end user). Each link consists of <href, rel, title> just like an ATOM link element.

I have ignored the possibility of having multiple translations of the messages. Neither does this implementation include any field-by-field validation since I expect that to be performed by the client. That doesn't mean the server shouldn't do the validation - it just doesn't have to return the detailed field information in a format readable by the client application.

JSON format example

Now it is time to select a wire format for the error information. I will choose JSON since that is a wide spread and well known format that can be handled by just about any piece of infrastructure nowadays. The format is straight forward and is probably best illustrated with a few examples:

Example 1 - the simplest possible instantiation

{
  message: "The field 'StartDate' did not contain a valid date (the value provided was '2013-20-23'). Dates should be formated as YYYY-MM-DD."
}

Example 2 - handling multiple validation errors

{
  message: "There was something wrong with the input (see below)",
  messages:
  [
    "The field 'StartDate' did not contain a valid date (the value provided was '2013-20-23'). Dates should be formated as YYYY-MM-DD.",
    "The field 'Title' must have a value."
  ]
}

Example 3 - using most of the features

{
  message: "Could not authorize user due to an internal problem - please try again later.",
  details: "The OAuth2 service is down for maintenance.",
  errorCode: "O2SERUNAV",
  httpStatusCode: 503,
  time: "2013-04-30T10:27:12",
  links:
  [
    { 
      href: "http://example.com/oauth2status.html", 
      rel: "help", 
      title: "Service status information" 
    }
  ]
}

Client implementation and media types - a matter of perspective

The client implementation should, at a suitable high level, be straight forward:

1. Client makes an HTTP request.
2. Request fails for some reason, server returns HTTP status code 4xx or 5xx and includes error information in the HTTP body.
3. Client checks HTTP status code, sees that it is 4xx or 5xx and decodes the error information.
4. Client tries to recover from error - either showing the error message to the end user, write the error to a log, give up or maybe retry the request - all depending on the error and the client's own capabilities.

But, hey, wait a minute ... how does the client know how to to decode the payload? I mean, perhaps the client asked for a resource representation containing medical records, but then it got a HTTP status code 400 - how is it supposed to know the format of the error information?

If the client is working with a vendor specific service, like Twitter and GitHub, then chances are that the client is hard wired to extract the error information based on the vendor specific service documentation. My guess is that this is how most clients are implemented.

But what if the client is working with a more, shall we say, RESTful service? That is; the client doesn't know what actual implementation it is interacting with. This could for instance be the case of clients consuming an ATOM feed (application/atom+xml). How would the client know how to decode the error response payload? Actually this seems like an unanswered question for ATOM since the spec is rather vague about this point (see for instance http://stackoverflow.com/questions/9874319/how-to-represent-error-messages-in-atom-feeds)

A RESTful service specification may call for a media type dedicated to error reporting; lets call such a media type "application/error+json". When the client receives a 4xx or 5xx HTTP status it can then look at the content-type header: if it matches "application/error+json" then the client would know exactly what to look for in the HTTP body.

It could also be that the base media type included detailed specification about error payloads.

I would prefer one of the two last options: either specify error handling in the base media type of the service - or use an existing standard media type. The last option is actually what Mark Nottingham has done with https://tools.ietf.org/html/draft-nottingham-http-problem-03.

So it is a matter of perspective: vendor specific "one-of-a-kind" services tend to invent their own error formats whereas RESTful services (like ATOM) should standardize error reporting via media types for everyone to reuse all over the web.

Have fun, Jørn

Ramone: Media types and codecs

One of the basic building blocks of REST is the concept of a media type - the file format used to represent a resource on the web. Media types comes in many different flavours - images, PDF, vCard, XML, JSON, spreadsheets and so on, each of them having their own specific formats and capabilities. If you haven't done it already then take a look at my previous post where I go deeper into details about media types.

Media types are considered first class citizens of Ramone, my C# library for consuming web APIs and RESTful services on the web - just like the uniform interface (GET/POST/PUT/...) and resource identifiers (URLs) - and in this post I will show how to work with different kinds of media types.

Codecs

A codec is a class that translates to and from the file format on the wire and some kind of internal representation in C#. To do so it must first implement either IMediaTypeWriter, IMediaTypeReader or both and then register with the current codec manager such that Ramone will be able to find it.

The codec interfaces are rather simple:

  public interface IMediaTypeCodec
  {
    object CodecArgument { get; set; }
  }

  public interface IMediaTypeWriter : IMediaTypeCodec
  {
    void WriteTo(WriterContext context);
  }

  public interface IMediaTypeReader : IMediaTypeCodec
  {
    object ReadFrom(ReaderContext context);
  }

The context parameter contains references to the current session, the data stream, HTTPRequest, HTTPResponse and others that are available for the codec.

Decoding an HTML micro format

One example of a codec is the BlogCodec from Ramone's test library. This codec demonstrates how to decode a (non-standard) micro format from an HTML page that shows a blog listing (see https://gist.github.com/2305777 for the actual HTML).

  public class BaseCodec_Html : TextCodecBase<Resources.Blog>
  {
    // This method is from TextCodecBase which has wrapped the binary input stream in a TextReader
    // using the charset encoding stated by the client's request headers.
    protected override Resources.Blog ReadFrom(TextReader reader, ReaderContext context)
    {
      // Using HtmlDocument from HtmlAgilityPack
      HtmlDocument doc = new HtmlDocument();
      doc.Load(reader);
      return ReadFromHtml(doc, context);
    }

    protected Resources.Blog ReadFromHtml(HtmlDocument html, ReaderContext context)
    {
      HtmlNode doc = html.DocumentNode;

      List<Resources.Blog.Post> posts = new List<Resources.Blog.Post>();

      // Scan through HTML and look for "class" attributes identifying values
      foreach (HtmlNode postNode in doc.SelectNodes(@"//div[@class=""post""]"))
      {
        HtmlNode title = postNode.SelectNodes(@".//*[@class=""post-title""]").First();
        HtmlNode content = postNode.SelectNodes(@".//*[@class=""post-content""]").First();
        List<Anchor> links = new List<Anchor>(postNode.Anchors(context.Response.ResponseUri));

        posts.Add(new Resources.Blog.Post
        {
          Title = title.InnerText,
          Text = content.InnerText,
          Links = links
        });
      }

      // Extract all HTML anchors together with <head> links and store them as ILink instances
      List<ILink> blogLinks = new List<ILink>(doc.Anchors(context.Response.ResponseUri).Cast<ILink>().Union(doc.Links(context.Response.ResponseUri)));

      // Create and return an object that represents the data extracted from the HTML
      Resources.Blog blog = new Resources.Blog()
      {
        Title = doc.SelectNodes(@".//*[@class=""blog-title""]").First().InnerText,
        Posts = posts,
        Links = blogLinks
      };

      return blog;
    }

    // This method is also from TextCodecBase, but is not used (since we do not write HTML)
    protected override void WriteTo(T item, System.IO.TextWriter writer, WriterContext context)
    {
      throw new NotImplementedException();
    }
  }

You can read a bit more about using hyper media links in another of my earlier posts.

Other codec examples

Other examples of codecs could be:

• Decoding cooking recipe data from XML or JSON.
• Decoding binary image data.
• Decoding CSV into tabular data.
• Decoding and writing vCard information.

Built-in generic codecs

All of the previous codecs has been "typed" in the sense that they decode response data into a typed object with the specific properties needed. But Ramone has also built-in support for various generic formats such as XML, JSON and HTML.

Here is an example use of the XML codec which decodes into C#'s XML DOM class XmlDocument:

  Request req = Session.Bind("... some url ...);

  XmlDocument doc = req.Get<XmlDocument>().Body;

The JSON codec can do some nifty stuff with C# dynamics:

  Request req = Session.Bind("... some URL for cat data ...");

  dynamic cat = req.Accept("application/json").Get().Body;

  Assert.IsNotNull(cat);
  Assert.AreEqual("Ramstein", cat.Name);

Advantages of typed codecs versus generic codecs

By working with typed codecs you gain a few advantages over the generic ones:

1. The application code is completely decoupled from the wire format. This gives you the ability to work with different wire formats without changing the application code, e.g., decoding both JSON, XML, and vCard into the same internal representation.
2. It results in more readable application code.
3. It makes the parsing code reusable across difference pieces of application code.

The downside of codecs is that you have to write a few more pieces of boilerplate code to create and register them. You also have to implement the client side representation of the resource as a specific class. But in the end it all pays off, in my opinion, and yields more readable and maintainable code.

Update (20/04/2012): Codec Manager

I forgot to show how codecs can be registered with either the current service (more on services at a later time); Codecs must be registered with Ramone, otherwise they will simply be ignored. To do so you first grab a reference to ICodecManager and then call AddCodec(...):

  ICodecManager cm = MyService.CodecManager;
  cm.AddCodec<MyClass, MyCodec>(MyMediaType);

Here MyClass is the type of object returned or written by the codec, MyCodec is the type of the codec and MyMediaType is the media type id string, e.g., "application/vnd.mytype+xml".


Ramone can be downloaded from https://github.com/JornWildt/Ramone