torsdag, april 19, 2012

Ramone: Media types and codecs

One of the basic building blocks of REST is the concept of a media type - the file format used to represent a resource on the web. Media types comes in many different flavours - images, PDF, vCard, XML, JSON, spreadsheets and so on, each of them having their own specific formats and capabilities. If you haven't done it already then take a look at my previous post where I go deeper into details about media types.

Media types are considered first class citizens of Ramone, my C# library for consuming web APIs and RESTful services on the web - just like the uniform interface (GET/POST/PUT/...) and resource identifiers (URLs) - and in this post I will show how to work with different kinds of media types.

Codecs

A codec is a class that translates to and from the file format on the wire and some kind of internal representation in C#. To do so it must first implement either IMediaTypeWriter, IMediaTypeReader or both and then register with the current codec manager such that Ramone will be able to find it.

The codec interfaces are rather simple:

  public interface IMediaTypeCodec
  {
    object CodecArgument { get; set; }
  }

  public interface IMediaTypeWriter : IMediaTypeCodec
  {
    void WriteTo(WriterContext context);
  }

  public interface IMediaTypeReader : IMediaTypeCodec
  {
    object ReadFrom(ReaderContext context);
  }

The context parameter contains references to the current session, the data stream, HTTPRequest, HTTPResponse and others that are available for the codec.

Decoding an HTML micro format

One example of a codec is the BlogCodec from Ramone's test library. This codec demonstrates how to decode a (non-standard) micro format from an HTML page that shows a blog listing (see https://gist.github.com/2305777 for the actual HTML).

  public class BaseCodec_Html : TextCodecBase<Resources.Blog>
  {
    // This method is from TextCodecBase which has wrapped the binary input stream in a TextReader
    // using the charset encoding stated by the client's request headers.
    protected override Resources.Blog ReadFrom(TextReader reader, ReaderContext context)
    {
      // Using HtmlDocument from HtmlAgilityPack
      HtmlDocument doc = new HtmlDocument();
      doc.Load(reader);
      return ReadFromHtml(doc, context);
    }

    protected Resources.Blog ReadFromHtml(HtmlDocument html, ReaderContext context)
    {
      HtmlNode doc = html.DocumentNode;

      List<Resources.Blog.Post> posts = new List<Resources.Blog.Post>();

      // Scan through HTML and look for "class" attributes identifying values
      foreach (HtmlNode postNode in doc.SelectNodes(@"//div[@class=""post""]"))
      {
        HtmlNode title = postNode.SelectNodes(@".//*[@class=""post-title""]").First();
        HtmlNode content = postNode.SelectNodes(@".//*[@class=""post-content""]").First();
        List<Anchor> links = new List<Anchor>(postNode.Anchors(context.Response.ResponseUri));

        posts.Add(new Resources.Blog.Post
        {
          Title = title.InnerText,
          Text = content.InnerText,
          Links = links
        });
      }

      // Extract all HTML anchors together with <head> links and store them as ILink instances
      List<ILink> blogLinks = new List<ILink>(doc.Anchors(context.Response.ResponseUri).Cast<ILink>().Union(doc.Links(context.Response.ResponseUri)));

      // Create and return an object that represents the data extracted from the HTML
      Resources.Blog blog = new Resources.Blog()
      {
        Title = doc.SelectNodes(@".//*[@class=""blog-title""]").First().InnerText,
        Posts = posts,
        Links = blogLinks
      };

      return blog;
    }

    // This method is also from TextCodecBase, but is not used (since we do not write HTML)
    protected override void WriteTo(T item, System.IO.TextWriter writer, WriterContext context)
    {
      throw new NotImplementedException();
    }
  }

You can read a bit more about using hyper media links in another of my earlier posts.

Other codec examples

Other examples of codecs could be:
  • Decoding cooking recipe data from XML or JSON.
  • Decoding binary image data.
  • Decoding CSV into tabular data.
  • Decoding and writing vCard information.

Built-in generic codecs

All of the previous codecs has been "typed" in the sense that they decode response data into a typed object with the specific properties needed. But Ramone has also built-in support for various generic formats such as XML, JSON and HTML.

Here is an example use of the XML codec which decodes into C#'s XML DOM class XmlDocument:

  Request req = Session.Bind("... some url ...);

  XmlDocument doc = req.Get<XmlDocument>().Body;

The JSON codec can do some nifty stuff with C# dynamics:

  Request req = Session.Bind("... some URL for cat data ...");

  dynamic cat = req.Accept("application/json").Get().Body;

  Assert.IsNotNull(cat);
  Assert.AreEqual("Ramstein", cat.Name);


Advantages of typed codecs versus generic codecs

By working with typed codecs you gain a few advantages over the generic ones:
  1. The application code is completely decoupled from the wire format. This gives you the ability to work with different wire formats without changing the application code, e.g., decoding both JSON, XML, and vCard into the same internal representation.
  2. It results in more readable application code.
  3. It makes the parsing code reusable across difference pieces of application code.
The downside of codecs is that you have to write a few more pieces of boilerplate code to create and register them. You also have to implement the client side representation of the resource as a specific class. But in the end it all pays off, in my opinion, and yields more readable and maintainable code.

Update (20/04/2012): Codec Manager

I forgot to show how codecs can be registered with either the current service (more on services at a later time); Codecs must be registered with Ramone, otherwise they will simply be ignored. To do so you first grab a reference to ICodecManager and then call AddCodec(...):

  ICodecManager cm = MyService.CodecManager;
  cm.AddCodec<MyClass, MyCodec>(MyMediaType);

Here MyClass is the type of object returned or written by the codec, MyCodec is the type of the codec and MyMediaType is the media type id string, e.g., "application/vnd.mytype+xml".


Ramone can be downloaded from https://github.com/JornWildt/Ramone


onsdag, april 18, 2012

RESTful resources are not typed

I have always been puzzled by this quote from Roy Fielding [1]:

    A REST API should never have “typed” resources that are significant to the client

How on earth is a client supposed to be able to do anything with a resource if it is not allowed to know the type of it?

The context here is a REST API that exposes business data from a certain business domain. This could be e-procurement, biochemistry or what ever else you could think of. But lets go with the e-procurement example and imagine a web shop that exposes catalogues, sales orders, customers and so on.

A machine-driven client may be interacting with this API in order to automatically buy something, lookup customers, gather sales statistics and what not. The point is - at a given time the client has a specific task to solve. Lets go with an example like this; "Get information about a certain sales order and print a letter to its customer".

Now, how can the client get the address of the customer unless it knows it is, well, of type "Customer" - and thus having a well known set of properties like "Name", "Birthdate", "Customer number", "Billing address" and so on?

Down the rabbit hole we go ...

Typed resources is a misconception

It is extremely easy to fall into the trap of thinking of resources as having specific types. A hyper link with the link relation type "customer" would surely point to a "Person" or perhaps a more generic "Contact" resource - right? Just like the resource identified by http://example.com/people/Pete must certainly be a "Person" too - right?

But what if the "customer" hyper link referred to http://www.cph.dk, or if you were presented with the URL http://example.com/fgx27-yw - what could you then conclude about types? Nothing. At the URL www.cph.dk we find a homepage from Copenhagen airport - that is not something which has a business domain type in traditional understanding - and fgx27-yw should be meaningless to anyone.

From a client's point of view, there is no such thing as a typed resource.

Resource views? No. Facets? No. Capabilities? No. Ah, yes, Media types!

Instead of talking about "types" we can reframe our thinking and talk about various views or facets of a resource - or perhaps the capabilities it offers to a client interacting with it. Just make sure you quit thinking of what the resource "is" and think of what it can do for you instead.

Example; a client that has been given the task of printing a letter to the customer at some {Customer} URL can ask the resource at that URL to return a VCard such that the client can get a mailing address - and it couldn't care less if the resource was a cat, city, or movie actor, as long as it it was able to return a VCard when asked for it.

Such a view of a resource is called a representation and the representation's format is the media type and asking for a specific media type is called content negotiation.

Media types

Lets take a look at some other media types and see how they can be applied without knowing the type of a given resource.
  1. A client following a link relation "news" may ask for application/atom+xml for a news feed.
  2. A client following a link "destination" might ask for application/gml+xml to get a geographical model of the destination.
  3. A client following a link "contact" might ask for text/vcard to get in contact with, well, the contact. It could also ask for application/vcard+xml in addition if it supported both formats. The server would then return one of them, depending on what it supported.
Again, there is absolutely no mentioning of the type of the resource. As an experiment, try to imagine the resource at www.cph.dk as the target of the above links ... works quite well, right? All the client need is an understanding of how the next resource in the application flow is related to the current one - it does not need to know what that resource is as long as it can ask for a representation that solves the given task of the client.

Domain specific media types

Now back to the original task of getting business domain specific details from a resource - without knowing the type of it. How can we get Name, Birthdate, Order No. and so on from the resource? What is needed is a domain specific media type - a media type which is specialized to a specific business domain.

Example: suppose a client needs the details of a sales order for a specific customer. All it needs to do is
  1. follow a link to the resource that someone else says is the sales order for the customer, and 
  2. ask for a media type that covers the sales order details needed.

This of course assumes that someone has defined a suitable domain specific media type that can be used this scenario. If no such thing exists then go ahead and design your own. Just make sure you register it!

Media type standardization

A media type must be registered with some authority - otherwise there is nothing for the client and server to agree on when communicating. On the big world wide web this is the job of IANA [2]. But it might as well be some central authority inside a big enterprise corporation - as long as (all) the client(s) and the server can agree on it.

Some business domains can be rather difficult to standardize and in these cases IANA may not be the place to go. Luckily there is nothing inherently wrong about creating private media types - as long as they stay inside the enterprise.

More capabilities

A media type is not restricted to only define the "wire format" or "serialization format" of a resource. It also defines link relations like "customer", "orders" or "owner" as well as how to interact with the resource. The typical interaction is navigation by following a URL - but media types can also define how to write data back and update resources aka "forms" handling. In this way media types goes way beyond simple understanding of the type of a resource.

Media type design is a whole art in itself that Mike Amundsen has written quite a lot about [3].

Links

[1] REST APIs must be hypertext-driven - http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven

[2] MIME Media Types - http://www.iana.org/assignments/media-types/index.html

[3] Hypermedia affordances - http://amundsen.com/blog/archives/1109


onsdag, april 04, 2012

Ramone: Consuming Hyper-Media REST Services in C#

Hyper-media controls are one of the core features of the World Wide Web as we know it today - without them there would be no way to link web pages and no way to interact with them through web forms. One of the REST constraints is the famous HATEOAS - "Hyper-media As The Engine Of Application State" and in order to consume RESTful web services we should make hyper-media controls first class citizens of the client interface we work with.

Hyper-media controls can be classified in different ways (see for instance Mike Amundsen's classification), but in Ramone we distinguish between the two well-known hyper-media elements; links and key/value forms as we know them from HTML, ATOM and more. These are represented with the following C# interface definitions:

  public interface ILink
  {
    Uri HRef { get; }
    string Title { get; }
    IEnumerable<string> RelationTypes { get; }
    MediaType MediaType { get; }
  }

  public interface IKeyValueForm
  {
    IKeyValueForm Value(string key, object value);
    IKeyValueForm Value(object value);
    Request Bind(string button = null);
  }

The idea is to use some C# extension method to get an instance of one of the above interface from the resource representation in the current response. This means in HTML there are methods for extracting <a> elements, <link> elements and <form> elements and return them as instances of the hyper-media interfaces. As more media-types are added to Ramone so will more similar extension methods be added - this can be done as add-ons without touching the core code.

The following example is from Ramone's Blog test collection (please take a closer look at the Github code since it includes some more explanations). This example shows how to follow an "author" link from an HTML page that shows a list of blog entries. The HTML contains (non-standard) micro formats for identifying various elements of the blog (post, author, title).

The actual HTML can be found here: https://gist.github.com/2305777#file_ramone_blog_list_html.html (and the HTML for an author can be found here: https://gist.github.com/2305988#file_gistfile1.html)

Here we go:

  
  // Create request by binding well known path to session's base URL
  Request blogRequest = Session.Bind(BlogRootPath);

  // GET blog HTML represented as an HtmlDocument (from Html Agility Pack)
  Response<HtmlDocument> blog = blogRequest.Get<HtmlDocument>();

  // Select first HTML anchor node with rel="author" as a anchor link.
  // This uses HtmlDocument specific extension methods to convert from anchor to ILink
  ILink authorLink = blog.Body.DocumentNode.SelectNodes(@"//a[@rel=""author""]").First().Anchor(blog);

  // Follow author link and get HTML document representing the author
  HtmlDocument author = authorLink.Follow(Session).Get<htmldocument>().Body;

  // Check e-mail of author
  HtmlNode email = author.DocumentNode.SelectNodes(@"//a[@rel=""email""]").First();
  Assert.AreEqual("pp@ramonerest.dk", email.Attributes["href"].Value);

The two important pieces of code in this example are the .Anchor(...) and .Follow(...) parts that respectively instantiates a link and follows it.

Forms can be loaded and submitted in a similar fashion.

Hopefully this gives an understandable introduction to working with hyper-media in Ramone - and, even more important, I hope you find it relatively intuitive and easy to work with.

tirsdag, april 03, 2012

Introducing the Ramone C# Library for Web API Clients

Ramone is a C# library for client applications that want to access some of the many web APIs available today.

What Ramone does is to wrap a lot of the bread and butter code into a library for easy reuse (see this previous post for my motivation). Out of the box it offers XML, JSON, ATOM and HTML formatters, URL templating and parameter binding, hyper-media traversal, form handling, and has HTTPs "Uniform Interface" as first class citizen in the interface.

With Ramone you won't see methods like GetOrders(...) or SaveUserData(...). You will instead be working with URLs that points to resources and use HTTPs uniform interface to interact with these. This means for instance that GetOrders(...) may become OrderRequest.Get() and SaveUserData(...) becomes UserRequest.Put(...).

The smallest example I can think of is this little four-liner that fetches the timeline of a specific Twitter account:

// All interaction with Ramone goes through a session (which defines a service base URL)
ISession session = RamoneConfiguration.NewSession(new Uri("https://api.twitter.com"));

// Creating a Ramone request by binding variables to a URI template
Request request = session.Bind(UserTimeLineTemplate, new { screen_name = ScreenName, count = 2 });

// GET response from Twitter
Response response = request.Get();

// Extract payload as a C# dynamic created from the JSON response
dynamic timeline = response.Body;

What happens here is:
  1. A session is created. This carries information about the base URL for template binding and keeps track of client state (cookies among other things).
  2. A request is created by binding a URL template with parameters. Input is a relative URL template path ("/1/statuses/user_timeline.json?screen_name={screen_name}&count={count}") and an anonymous parameter object. Output is a request for an absolute URI.
  3. One of the operations from HTTPs Uniform Interface is called (GET) and the response is recorded.
  4. Ramone checks the returned media-type, sees it is application/json, and converts it to a dynamic C# object with late binding of the returned values.
The code for iterating through the timeline and printing status updates is plain C#:


Console.WriteLine("This is the timeline for {0}:", ScreenName);
Console.WriteLine();

foreach (dynamic tweet in timeline)
{
  Console.WriteLine("* {0}.", tweet.text);
  Console.WriteLine();
} 

A few things worth noticing:
  1. Twitter may not be the best REST example out there, but it is well understood by many people and is a web API which is quite easy to work with.
  2. Ramone does not only handle JSON. It works with a repository of parsers and formatters (called codecs) which are responsible for handling the various formats Ramone may encounter.

You can find a complete Twitter demo at Github: https://github.com/JornWildt/Ramone/tree/master/TwitterDemo/TwitterDemo. This demo shows both how to fetch a timeline as well as how to post status updates with OAuth1 authentication.

All source code is available on Github: https://github.com/JornWildt/Ramone. At the time of writing there are no official binaries released so you will have to download the code from Github and compile it yourself.

Later on I will write about how Ramone helps you consuming more RESTful hyper-media based services by making links, link relations and forms first class citizens of the interface. For now you can check the blog example at Github: https://github.com/JornWildt/Ramone/tree/master/Ramone.Tests/Blog

Have fun!

søndag, april 01, 2012

Consuming Web APIs in C# with Ramone

The World Wide Web is a fascinating place with its enormous amount of information right at your hand. It is even more fascinating how relatively simple the upper application layer is: we have resources, URLs - Uniform Resource Locators which we can dereference, a small set of methods to operate on said resources - GET/POST/PUT/DELETE (and then some), and media-types for identifying how those resources are represented.

Roy T. Fielding wrapped it all up in his dissertation "Architectural Styles and the Design of Network-based Software Architectures" where he coined the term REST - "Representational State Transfer". Somehow this term caught on and developers, who was trying to find something more simple than SOAP (over HTTP), started looking at REST for machine-to-machine APIs on the web. Soon we got REST APIs - and plenty of hype around them. Whether or not these APIs are actually RESTful is an ongoing debate, but it should be safe to call them "Web APIs" no matter how they present themself, so I will stick to that term for now.

Today there are thousands of public web APIs available for anyone to access - just take a look at http://www.programmableweb.com/ - at the time of writing it boasts some 5529 APIs. And those are just the public APIs - in addition to this there's probably many many more in closed enterprises and partnerships.

When I first started working with web APIs in C# I soon got a bit tired of the rituals you have to go through in order to work with HTTP in .NET. Usually I have a URL to some resource, the name of a method I want to invoke, maybe a payload (a C# object instance), and then a returned resource representation (also an object instance). So I should be able to write something like this:

  Uri url = new Uri("...");
  Payload p = new Payload(...);
  Resource r = url.<Method>(p);

You can do this in a few lines of code with .NET's WebRequest class ... except that you get absolutely no help when it comes to serializing Payload and Resource data. And what format are those classes actually serialized as? Is it XML, JSON, HTML or something else?

The next problem with low level HTTP access using WebRequest is handling of authorization using either HTTP Basic, OAuth or something similar - these are standardized authorization mechanisms, but you need to add them on top of HTTP yourself.

So there is a lot of bread and butter code for you to write before you can get up and running with your first web API. Luckily it is all based on the uniform interface, URLs and media-types - something that can be wrapped into a framework and lower the entry barrier for consuming web APIs - whether they are RESTful or not.

That is what Ramone is all about - making it easy to consume web APIs in C#. Interacting with the web should be as easy going as, well, hmm, Ramone from Pixar's movie Cars, from which I got the name.

Hopy you enjoy it! More posts will follow, showing how to get started with Ramone.

All source code is available on Github: https://github.com/JornWildt/Ramone (distributed under the MIT license).

Happy hacking, Jørn Wildt

mandag, september 26, 2011

A few answers to CQRS questions

I have tried to pick a few common CQRS questions and supply my own answers:

1) I'm really missing some definition of WHEN to use CQRS. Collaboration, I know, but 90% is collaboration in my book. A Customer table for example. And Orders, Invoices and shipping tables. Can those be accessed and edited by more than one user? Yes - well there's collaboration!

The point here is that, yes, they can be edited by more than one user, but are they really? How high is the probability of having more than one user updating the same Customer simultaneously? I would say close to zero. The same goes for the other entities.

Sure, all of the users are working together to keep the customer table in sync with reality, that is collaboration. But it is not collaboration in the way of working on the same shared resource!

But we still need to handle that 0.001% change of simultaneous updates! Sure - so add a simple version check (first update wins, second update gets a notification). That is not CQRS One Way Commands - that's plain old synchronous and locking database updates.

So when are we collaborating on shared resources and expecting lots of simultaneous work? Good question (and I do not have a good answer). The fact that so few examples exists indicates that CQRS is something which is not normally required.

2) What about the fact that many implementors feel that things are getting much more simple just by having read and write separation. Isn't that a reason in itself to use CQRS?

You can separate reading and writing without CQRS and eventual consistency. All you need is two different code bases working on the same database table. Very low tech. Maybe throw in multiple database views to support more or less complex queries.

3) But having different read and write databases helps scaling out.

Yes. So does plain single-write-master/multiple-read-slave database replication. If scaling out the query part is your problem then use master/slave on the database level. If scaling out writing is your problem then look into CQRS for that particular use case which requires it!

Scaling out writing is required if you have lots of issues with database deadlocks and timeouts due to intensive locking of the database. This won't happen if all your users do, is to work on different entities all the time.

4) Adding some repository and an event store might be hard the first time, but after that it's really simple. Right?

In my little experience - No. But I may certainly be wrong and have used the wrong tools. In theory it is simple, yes, in practice, no. It adds complexity - not that much, but enough add friction to your project. If your are reading this for some advice on CQRS - well, you might just have experienced that friction and started wondering why.

CQRS adds extra time spent on infrastructure for simple problems, when you should be spending time on complex problems, while keeping simple stuff, well, simple. See also my previous post: http://soabits.blogspot.com/2011/09/why-cqrs-may-not-be-answer-you-are.html.


Please, go ahead and use CQRS! I am not saying "no, do not" (who am I to do that?) - I am just sharing my experience.

/Jørn

Why CQRS may not be the answer you are looking for

With CQRS I mean Command Query Responsibility Segregation. If you have never heard of this before then you are probably better off reading about CQRS elsewhere on the web.

If you know CQRS already then, please, read on.

Have you ever tried to implement some kind of CQRS and had that feeling of friction - not really getting it to work and then wondering why? I mean - everybody else is using it, right? Like this quote from the NServiceBus forum states it:

> I think CQRS has been getting a lot of hype recently and one frustration for a beginner (like me) is that if all these people are using these concepts/technologies, rather than just telling conventional crud/n-tier folks that CQRS is the "future", let's have some good examples and documentation that helps us learn why.

I have been there myself - trying to do CQRS exercises for training and ultimately giving up. Lately I have been digging more into it and now I would like to share some thoughts based on my own experiences and learnings from Udi Dahan's Advanced Distributed Systems Design Course. To be honest, lot's of it is just rephrasing of stuff from the course.

First of all, CQRS is one of those annoying technologies that looks very simple in theory, but turns out to cost a bit to build in practice.

Lets try with an example from the NCQRS project (I do not hold anything against that project, they just have a good on-line example) - creating a Tweet in a Twitter-like system: http://ncqrs.org/getting-started/getting-started-with-ncqrs/.

Now compare the CQRS solution with a Ruby on Rails solution: http://www.noupe.com/ajax/create-a-simple-twitter-app.html. Which one is the simplest? I hope you agree with me that the Ruby on Rails solution is simpler - it is after all only a matter of writing a tweet to a database. Right?

And that is the most important point: it is after all only a matter of writing a tweet to a database! Right? None of the two examples has addressed the more difficult problems of making sure the right people gets the right tweets, re-tweeting and so on.

So why use CQRS when it can be done in a much simpler way? Here are some arguments:
  1. Its never going to scale!
  2. Of course RoR is easier, CQRS is for complex/collaboration scenarios, you won't see any gain in such a simple example!
  3. Okay, the RoR solution is simpler, but(!) when we add complex business rules then it will be a mess!
Regarding (1): the code we are seeing so far will scale - it is simple CRUD operations and it can easily be scaled out on a database level with a one-master/multiple-slaves setup.

Regarding (2): that it is the whole point - CQRS is not for simple scenarios. You can do it, but it may hurt and work against you all the time.

Regarding (3): you may be right! So lets take out the complex business rules and handle them elsewhere - asynchronously using messaging.

Our message handlers can parse the tweet, looking for #tag and @name, and distribute tweets to other users. All of it the same (eventually consistent) query database!

Duh! You probably say. Only Command handlers modify command state and send out events that the query context listens for. But is any command state involved here? No - all we have done is the simple RoR style direct update of the query model, and some background processing also working directly on the query model.

How can we do this? Where is the command handling? Well, it is not here ... as I started out saying, CQRS may not be the answer you are looking for. CQRS is not to be applied to any and all problems - it carries a big overhead (even if it is simple in theory).

What is left is then to explain where CQRS really applies - but that is beyond the scope of this blog post (and also beyond my current knowledge of CQRS).

So who am I to judge what CQRS is and is not? Well, not any better than so many other people out there. I am no expert. I build my opinion on my own experiences and some valuable input from Udi. Read this as one advice out of many and use it to make your own decision.

/Jørn