torsdag, april 19, 2012

Ramone: Media types and codecs

One of the basic building blocks of REST is the concept of a media type - the file format used to represent a resource on the web. Media types comes in many different flavours - images, PDF, vCard, XML, JSON, spreadsheets and so on, each of them having their own specific formats and capabilities. If you haven't done it already then take a look at my previous post where I go deeper into details about media types.

Media types are considered first class citizens of Ramone, my C# library for consuming web APIs and RESTful services on the web - just like the uniform interface (GET/POST/PUT/...) and resource identifiers (URLs) - and in this post I will show how to work with different kinds of media types.

Codecs

A codec is a class that translates to and from the file format on the wire and some kind of internal representation in C#. To do so it must first implement either IMediaTypeWriter, IMediaTypeReader or both and then register with the current codec manager such that Ramone will be able to find it.

The codec interfaces are rather simple:

  public interface IMediaTypeCodec
  {
    object CodecArgument { get; set; }
  }

  public interface IMediaTypeWriter : IMediaTypeCodec
  {
    void WriteTo(WriterContext context);
  }

  public interface IMediaTypeReader : IMediaTypeCodec
  {
    object ReadFrom(ReaderContext context);
  }

The context parameter contains references to the current session, the data stream, HTTPRequest, HTTPResponse and others that are available for the codec.

Decoding an HTML micro format

One example of a codec is the BlogCodec from Ramone's test library. This codec demonstrates how to decode a (non-standard) micro format from an HTML page that shows a blog listing (see https://gist.github.com/2305777 for the actual HTML).

  public class BaseCodec_Html : TextCodecBase<Resources.Blog>
  {
    // This method is from TextCodecBase which has wrapped the binary input stream in a TextReader
    // using the charset encoding stated by the client's request headers.
    protected override Resources.Blog ReadFrom(TextReader reader, ReaderContext context)
    {
      // Using HtmlDocument from HtmlAgilityPack
      HtmlDocument doc = new HtmlDocument();
      doc.Load(reader);
      return ReadFromHtml(doc, context);
    }

    protected Resources.Blog ReadFromHtml(HtmlDocument html, ReaderContext context)
    {
      HtmlNode doc = html.DocumentNode;

      List<Resources.Blog.Post> posts = new List<Resources.Blog.Post>();

      // Scan through HTML and look for "class" attributes identifying values
      foreach (HtmlNode postNode in doc.SelectNodes(@"//div[@class=""post""]"))
      {
        HtmlNode title = postNode.SelectNodes(@".//*[@class=""post-title""]").First();
        HtmlNode content = postNode.SelectNodes(@".//*[@class=""post-content""]").First();
        List<Anchor> links = new List<Anchor>(postNode.Anchors(context.Response.ResponseUri));

        posts.Add(new Resources.Blog.Post
        {
          Title = title.InnerText,
          Text = content.InnerText,
          Links = links
        });
      }

      // Extract all HTML anchors together with <head> links and store them as ILink instances
      List<ILink> blogLinks = new List<ILink>(doc.Anchors(context.Response.ResponseUri).Cast<ILink>().Union(doc.Links(context.Response.ResponseUri)));

      // Create and return an object that represents the data extracted from the HTML
      Resources.Blog blog = new Resources.Blog()
      {
        Title = doc.SelectNodes(@".//*[@class=""blog-title""]").First().InnerText,
        Posts = posts,
        Links = blogLinks
      };

      return blog;
    }

    // This method is also from TextCodecBase, but is not used (since we do not write HTML)
    protected override void WriteTo(T item, System.IO.TextWriter writer, WriterContext context)
    {
      throw new NotImplementedException();
    }
  }

You can read a bit more about using hyper media links in another of my earlier posts.

Other codec examples

Other examples of codecs could be:
  • Decoding cooking recipe data from XML or JSON.
  • Decoding binary image data.
  • Decoding CSV into tabular data.
  • Decoding and writing vCard information.

Built-in generic codecs

All of the previous codecs has been "typed" in the sense that they decode response data into a typed object with the specific properties needed. But Ramone has also built-in support for various generic formats such as XML, JSON and HTML.

Here is an example use of the XML codec which decodes into C#'s XML DOM class XmlDocument:

  Request req = Session.Bind("... some url ...);

  XmlDocument doc = req.Get<XmlDocument>().Body;

The JSON codec can do some nifty stuff with C# dynamics:

  Request req = Session.Bind("... some URL for cat data ...");

  dynamic cat = req.Accept("application/json").Get().Body;

  Assert.IsNotNull(cat);
  Assert.AreEqual("Ramstein", cat.Name);


Advantages of typed codecs versus generic codecs

By working with typed codecs you gain a few advantages over the generic ones:
  1. The application code is completely decoupled from the wire format. This gives you the ability to work with different wire formats without changing the application code, e.g., decoding both JSON, XML, and vCard into the same internal representation.
  2. It results in more readable application code.
  3. It makes the parsing code reusable across difference pieces of application code.
The downside of codecs is that you have to write a few more pieces of boilerplate code to create and register them. You also have to implement the client side representation of the resource as a specific class. But in the end it all pays off, in my opinion, and yields more readable and maintainable code.

Update (20/04/2012): Codec Manager

I forgot to show how codecs can be registered with either the current service (more on services at a later time); Codecs must be registered with Ramone, otherwise they will simply be ignored. To do so you first grab a reference to ICodecManager and then call AddCodec(...):

  ICodecManager cm = MyService.CodecManager;
  cm.AddCodec<MyClass, MyCodec>(MyMediaType);

Here MyClass is the type of object returned or written by the codec, MyCodec is the type of the codec and MyMediaType is the media type id string, e.g., "application/vnd.mytype+xml".


Ramone can be downloaded from https://github.com/JornWildt/Ramone


onsdag, april 18, 2012

RESTful resources are not typed

I have always been puzzled by this quote from Roy Fielding [1]:

    A REST API should never have “typed” resources that are significant to the client

How on earth is a client supposed to be able to do anything with a resource if it is not allowed to know the type of it?

The context here is a REST API that exposes business data from a certain business domain. This could be e-procurement, biochemistry or what ever else you could think of. But lets go with the e-procurement example and imagine a web shop that exposes catalogues, sales orders, customers and so on.

A machine-driven client may be interacting with this API in order to automatically buy something, lookup customers, gather sales statistics and what not. The point is - at a given time the client has a specific task to solve. Lets go with an example like this; "Get information about a certain sales order and print a letter to its customer".

Now, how can the client get the address of the customer unless it knows it is, well, of type "Customer" - and thus having a well known set of properties like "Name", "Birthdate", "Customer number", "Billing address" and so on?

Down the rabbit hole we go ...

Typed resources is a misconception

It is extremely easy to fall into the trap of thinking of resources as having specific types. A hyper link with the link relation type "customer" would surely point to a "Person" or perhaps a more generic "Contact" resource - right? Just like the resource identified by http://example.com/people/Pete must certainly be a "Person" too - right?

But what if the "customer" hyper link referred to http://www.cph.dk, or if you were presented with the URL http://example.com/fgx27-yw - what could you then conclude about types? Nothing. At the URL www.cph.dk we find a homepage from Copenhagen airport - that is not something which has a business domain type in traditional understanding - and fgx27-yw should be meaningless to anyone.

From a client's point of view, there is no such thing as a typed resource.

Resource views? No. Facets? No. Capabilities? No. Ah, yes, Media types!

Instead of talking about "types" we can reframe our thinking and talk about various views or facets of a resource - or perhaps the capabilities it offers to a client interacting with it. Just make sure you quit thinking of what the resource "is" and think of what it can do for you instead.

Example; a client that has been given the task of printing a letter to the customer at some {Customer} URL can ask the resource at that URL to return a VCard such that the client can get a mailing address - and it couldn't care less if the resource was a cat, city, or movie actor, as long as it it was able to return a VCard when asked for it.

Such a view of a resource is called a representation and the representation's format is the media type and asking for a specific media type is called content negotiation.

Media types

Lets take a look at some other media types and see how they can be applied without knowing the type of a given resource.
  1. A client following a link relation "news" may ask for application/atom+xml for a news feed.
  2. A client following a link "destination" might ask for application/gml+xml to get a geographical model of the destination.
  3. A client following a link "contact" might ask for text/vcard to get in contact with, well, the contact. It could also ask for application/vcard+xml in addition if it supported both formats. The server would then return one of them, depending on what it supported.
Again, there is absolutely no mentioning of the type of the resource. As an experiment, try to imagine the resource at www.cph.dk as the target of the above links ... works quite well, right? All the client need is an understanding of how the next resource in the application flow is related to the current one - it does not need to know what that resource is as long as it can ask for a representation that solves the given task of the client.

Domain specific media types

Now back to the original task of getting business domain specific details from a resource - without knowing the type of it. How can we get Name, Birthdate, Order No. and so on from the resource? What is needed is a domain specific media type - a media type which is specialized to a specific business domain.

Example: suppose a client needs the details of a sales order for a specific customer. All it needs to do is
  1. follow a link to the resource that someone else says is the sales order for the customer, and 
  2. ask for a media type that covers the sales order details needed.

This of course assumes that someone has defined a suitable domain specific media type that can be used this scenario. If no such thing exists then go ahead and design your own. Just make sure you register it!

Media type standardization

A media type must be registered with some authority - otherwise there is nothing for the client and server to agree on when communicating. On the big world wide web this is the job of IANA [2]. But it might as well be some central authority inside a big enterprise corporation - as long as (all) the client(s) and the server can agree on it.

Some business domains can be rather difficult to standardize and in these cases IANA may not be the place to go. Luckily there is nothing inherently wrong about creating private media types - as long as they stay inside the enterprise.

More capabilities

A media type is not restricted to only define the "wire format" or "serialization format" of a resource. It also defines link relations like "customer", "orders" or "owner" as well as how to interact with the resource. The typical interaction is navigation by following a URL - but media types can also define how to write data back and update resources aka "forms" handling. In this way media types goes way beyond simple understanding of the type of a resource.

Media type design is a whole art in itself that Mike Amundsen has written quite a lot about [3].

Links

[1] REST APIs must be hypertext-driven - http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven

[2] MIME Media Types - http://www.iana.org/assignments/media-types/index.html

[3] Hypermedia affordances - http://amundsen.com/blog/archives/1109


onsdag, april 04, 2012

Ramone: Consuming Hyper-Media REST Services in C#

Hyper-media controls are one of the core features of the World Wide Web as we know it today - without them there would be no way to link web pages and no way to interact with them through web forms. One of the REST constraints is the famous HATEOAS - "Hyper-media As The Engine Of Application State" and in order to consume RESTful web services we should make hyper-media controls first class citizens of the client interface we work with.

Hyper-media controls can be classified in different ways (see for instance Mike Amundsen's classification), but in Ramone we distinguish between the two well-known hyper-media elements; links and key/value forms as we know them from HTML, ATOM and more. These are represented with the following C# interface definitions:

  public interface ILink
  {
    Uri HRef { get; }
    string Title { get; }
    IEnumerable<string> RelationTypes { get; }
    MediaType MediaType { get; }
  }

  public interface IKeyValueForm
  {
    IKeyValueForm Value(string key, object value);
    IKeyValueForm Value(object value);
    Request Bind(string button = null);
  }

The idea is to use some C# extension method to get an instance of one of the above interface from the resource representation in the current response. This means in HTML there are methods for extracting <a> elements, <link> elements and <form> elements and return them as instances of the hyper-media interfaces. As more media-types are added to Ramone so will more similar extension methods be added - this can be done as add-ons without touching the core code.

The following example is from Ramone's Blog test collection (please take a closer look at the Github code since it includes some more explanations). This example shows how to follow an "author" link from an HTML page that shows a list of blog entries. The HTML contains (non-standard) micro formats for identifying various elements of the blog (post, author, title).

The actual HTML can be found here: https://gist.github.com/2305777#file_ramone_blog_list_html.html (and the HTML for an author can be found here: https://gist.github.com/2305988#file_gistfile1.html)

Here we go:

  
  // Create request by binding well known path to session's base URL
  Request blogRequest = Session.Bind(BlogRootPath);

  // GET blog HTML represented as an HtmlDocument (from Html Agility Pack)
  Response<HtmlDocument> blog = blogRequest.Get<HtmlDocument>();

  // Select first HTML anchor node with rel="author" as a anchor link.
  // This uses HtmlDocument specific extension methods to convert from anchor to ILink
  ILink authorLink = blog.Body.DocumentNode.SelectNodes(@"//a[@rel=""author""]").First().Anchor(blog);

  // Follow author link and get HTML document representing the author
  HtmlDocument author = authorLink.Follow(Session).Get<htmldocument>().Body;

  // Check e-mail of author
  HtmlNode email = author.DocumentNode.SelectNodes(@"//a[@rel=""email""]").First();
  Assert.AreEqual("pp@ramonerest.dk", email.Attributes["href"].Value);

The two important pieces of code in this example are the .Anchor(...) and .Follow(...) parts that respectively instantiates a link and follows it.

Forms can be loaded and submitted in a similar fashion.

Hopefully this gives an understandable introduction to working with hyper-media in Ramone - and, even more important, I hope you find it relatively intuitive and easy to work with.

tirsdag, april 03, 2012

Introducing the Ramone C# Library for Web API Clients

Ramone is a C# library for client applications that want to access some of the many web APIs available today.

What Ramone does is to wrap a lot of the bread and butter code into a library for easy reuse (see this previous post for my motivation). Out of the box it offers XML, JSON, ATOM and HTML formatters, URL templating and parameter binding, hyper-media traversal, form handling, and has HTTPs "Uniform Interface" as first class citizen in the interface.

With Ramone you won't see methods like GetOrders(...) or SaveUserData(...). You will instead be working with URLs that points to resources and use HTTPs uniform interface to interact with these. This means for instance that GetOrders(...) may become OrderRequest.Get() and SaveUserData(...) becomes UserRequest.Put(...).

The smallest example I can think of is this little four-liner that fetches the timeline of a specific Twitter account:

// All interaction with Ramone goes through a session (which defines a service base URL)
ISession session = RamoneConfiguration.NewSession(new Uri("https://api.twitter.com"));

// Creating a Ramone request by binding variables to a URI template
Request request = session.Bind(UserTimeLineTemplate, new { screen_name = ScreenName, count = 2 });

// GET response from Twitter
Response response = request.Get();

// Extract payload as a C# dynamic created from the JSON response
dynamic timeline = response.Body;

What happens here is:
  1. A session is created. This carries information about the base URL for template binding and keeps track of client state (cookies among other things).
  2. A request is created by binding a URL template with parameters. Input is a relative URL template path ("/1/statuses/user_timeline.json?screen_name={screen_name}&count={count}") and an anonymous parameter object. Output is a request for an absolute URI.
  3. One of the operations from HTTPs Uniform Interface is called (GET) and the response is recorded.
  4. Ramone checks the returned media-type, sees it is application/json, and converts it to a dynamic C# object with late binding of the returned values.
The code for iterating through the timeline and printing status updates is plain C#:


Console.WriteLine("This is the timeline for {0}:", ScreenName);
Console.WriteLine();

foreach (dynamic tweet in timeline)
{
  Console.WriteLine("* {0}.", tweet.text);
  Console.WriteLine();
} 

A few things worth noticing:
  1. Twitter may not be the best REST example out there, but it is well understood by many people and is a web API which is quite easy to work with.
  2. Ramone does not only handle JSON. It works with a repository of parsers and formatters (called codecs) which are responsible for handling the various formats Ramone may encounter.

You can find a complete Twitter demo at Github: https://github.com/JornWildt/Ramone/tree/master/TwitterDemo/TwitterDemo. This demo shows both how to fetch a timeline as well as how to post status updates with OAuth1 authentication.

All source code is available on Github: https://github.com/JornWildt/Ramone. At the time of writing there are no official binaries released so you will have to download the code from Github and compile it yourself.

Later on I will write about how Ramone helps you consuming more RESTful hyper-media based services by making links, link relations and forms first class citizens of the interface. For now you can check the blog example at Github: https://github.com/JornWildt/Ramone/tree/master/Ramone.Tests/Blog

Have fun!

søndag, april 01, 2012

Consuming Web APIs in C# with Ramone

The World Wide Web is a fascinating place with its enormous amount of information right at your hand. It is even more fascinating how relatively simple the upper application layer is: we have resources, URLs - Uniform Resource Locators which we can dereference, a small set of methods to operate on said resources - GET/POST/PUT/DELETE (and then some), and media-types for identifying how those resources are represented.

Roy T. Fielding wrapped it all up in his dissertation "Architectural Styles and the Design of Network-based Software Architectures" where he coined the term REST - "Representational State Transfer". Somehow this term caught on and developers, who was trying to find something more simple than SOAP (over HTTP), started looking at REST for machine-to-machine APIs on the web. Soon we got REST APIs - and plenty of hype around them. Whether or not these APIs are actually RESTful is an ongoing debate, but it should be safe to call them "Web APIs" no matter how they present themself, so I will stick to that term for now.

Today there are thousands of public web APIs available for anyone to access - just take a look at http://www.programmableweb.com/ - at the time of writing it boasts some 5529 APIs. And those are just the public APIs - in addition to this there's probably many many more in closed enterprises and partnerships.

When I first started working with web APIs in C# I soon got a bit tired of the rituals you have to go through in order to work with HTTP in .NET. Usually I have a URL to some resource, the name of a method I want to invoke, maybe a payload (a C# object instance), and then a returned resource representation (also an object instance). So I should be able to write something like this:

  Uri url = new Uri("...");
  Payload p = new Payload(...);
  Resource r = url.<Method>(p);

You can do this in a few lines of code with .NET's WebRequest class ... except that you get absolutely no help when it comes to serializing Payload and Resource data. And what format are those classes actually serialized as? Is it XML, JSON, HTML or something else?

The next problem with low level HTTP access using WebRequest is handling of authorization using either HTTP Basic, OAuth or something similar - these are standardized authorization mechanisms, but you need to add them on top of HTTP yourself.

So there is a lot of bread and butter code for you to write before you can get up and running with your first web API. Luckily it is all based on the uniform interface, URLs and media-types - something that can be wrapped into a framework and lower the entry barrier for consuming web APIs - whether they are RESTful or not.

That is what Ramone is all about - making it easy to consume web APIs in C#. Interacting with the web should be as easy going as, well, hmm, Ramone from Pixar's movie Cars, from which I got the name.

Hopy you enjoy it! More posts will follow, showing how to get started with Ramone.

All source code is available on Github: https://github.com/JornWildt/Ramone (distributed under the MIT license).

Happy hacking, Jørn Wildt