API authentication considerations and best practices

I have been answering a few security questions on Stackoverflow and going through some APIs on programmableweb.com - and it keeps amazing me how often people gets HTTP authorization wrong.

The typical questions I have noticed on SO are something like "I want to secure my API, have control of clients and require users to login - but I don't want to use OAuth". Duh? Why not? Perhaps because OAuth is conceived as being too difficult to work with? Well, OAuth2  is definitively not difficult (OAuth1 is another story though).

But lets take a look at some of the existing practices:

API keys in URLs: one example is the Finish National Gallery API (which I somehow stumbled upon). Here you are required to obtain an API key and put it into the URL in every request made to the API:

  http://kokoelmat.fng.fi/api/v2?apikey=********&q=A+III+2172


API keys in custom headers: one example is from DocuSign. They require you to encode both your (developer) user name, password and API key in a JSON object and then send that in a custom header X-DocuSign-Authentication:

  GET /some-url
  X-DocuSign-Authentication: { "Username": "...", "Password": "...", "IntegratorKey": "..." }
  ... more ...

Signed URL/body parameters: one example is last.fm which requires you to use your API key to obtain a session token and use that token later on to sign requests, either using URL parameters or in the body of the HTTP request.

HTTP Basic authentication: GitHub supports authentication via the standard HTTP basic authentication mechanism where you supply username and password BASE64 encoded in the "Authorization" header.

OAuth1 and Oauth2: PhotoBucket uses OAuth1 and BaseCamp uses Oauth2 (in addition to HTTP basic authentication).

I did set out with the expectation of finding many more strange authorization schemes, but it turned out that at least most of the big players use OAuth1 or OAuth2. That's great! Now we just need to get the word out to all the "dark matter developers" out there (as Scott Hanselman calls them).

Things to be aware of

1) API keys in URLs are very easily accidentally exposed to third parties. If you take a copy of the URL and mail it to somebody you will end up sending your private key to someone else. For the same reason API keys may end up in log files here and there. Not good!

2) API keys in URLs changes your resource URL such that it depends on who is using it. Instead of everybody referring to /some-items/1234 it will be /some-items/1234?key=abc for one user and /some-items/1234?key=qwe for another - and these two are completely different URLs even though they "only" differ on the query parameters. It is semantically the same as encoding the key in the path segment as for instance /api/abc/some-items/1234 and /api/qwe/some-items/1234 which I don't expect any one to think of as a good idea - right?

3) API keys in headers are better than API keys in URLs since it will keep the URL stable for all users and it doesn't expose API keys when sending links to others. The problem though is that you are still sending your secret credentials (the API key) over the wire and it may be wiretapped by third parties. Another problem is that a client may accidentally connect to the wrong server (by misconfiguration or otherwise) and expose its credentials there.

4) Signed requests (done right!) should be preferred over sending the API key directly. The signature should not be part of the URL for the reasons stated above - and the same goes somehow for sending the signature in the body since different users will see different bodies which is not necessary when you have a standard HTTP header for the purpose.

5) Proprietary methods for signing requests are prone to design errors as it is easy to get the signature mechanism wrong - and thus accidentally making the signature technique easy to circumvent.

6) Proprietary methods for signing requests requires client developer to understand yet another signature mechanism. It also makes it less likely to find existing client libraries to handle the crypto stuff. Both of these issues makes client adoption of your API less likely. And you know what, dear server developer? Your client developers will call for support and it will end up on YOUR table, reducing your ability to focus on coding the next great thing! So stick to well documented standards - it will be less annoying for everybody including yourself.

7) HTTP basic authentication is great for debugging and getting started scenarios but should not be used in production. The problem is that client credentials are send in clear text and are thus susceptible to accidental exposure as mentioned before. Your API should support basic authentication as it will make it possible to explore the API using a standard web browser - but it should be disabled in production.

8) It should be possible to revoke API keys in case they are compromised in some way.

What kind of technique would solve all of these issues? Well, Oauth2 is one standard solution; it uses the HTTP "Authorization" header (thus avoiding authentication stuff in URL and body), it is a standard and used correctly it protects clients from exposing their API keys to the server and over the wire.

Another solution is to use SSL/TLS with client certificates. With the proper HTTP client libraries this can be as easy as loading a certificate (one line of code) and assigning it to the HTTP request object (second line of code). This can although not authorize a combination of both client credentials and end user credentials.

OAuth2

OAuth2 roles and terms

Before jumping into OAuth2 I better explain some of the terms used when talking about OAuth2 (mostly copied from the RFC at http://tools.ietf.org/html/rfc6749#section-1.1):

- Protected resources: the data you want to protect.

- Resource owner: An entity capable of granting access to a protected resource. The "entity" is often a person - the end user.

- Resource server: The server hosting the protected resources.

- Client: An application making protected resource requests on behalf of the resource owner. This can for instance be a mobile application, a website or a background integration process impersonating an existing end user.

- Client credentials: a pair of client ID and client secret. This could be your developer or application ID and API key.

- Resource owner password credentials: the typical user-name/password combination issued to an end user.

OAuth2 flows

OAuth2 has four different modes of operating (called flows) - going from the relatively complex 3-legged authorization flow to the very simple "client credentials" authorization flow. At least one of the flows should fit into just about any public API ecosystem out there - and if not then it is possible to add new extension flows (as for instance Google does with the JWS implementation).

At its core OAuth2 has only two high level steps:

  1) Swap a set of user and/or client credentials for an access token (authorization step), and

  2) Use the access token to access the API resources.

That's it. It can be very simple. The difficult part of OAuth2 is the many ways a client can obtain an access token.

Here I will focus on a scenario with a trusted client that accepts user credentials and acts on behalf of them. This can be very useful for many internal integration patterns where no humans are involved. I won't cover the scenario where an untrusted client (third party website) needs to access the end user's resources using the 3-legged authorization flow.

Acces tokens and bearer tokens

At a suitable high level OAuth2 only has two steps as mentioned before: 1) authorization, 2) resource access. The output of a successful authorization step is always an access token which must be used in subsequent requests to the resource server (the API).

In most cases the access token is a so called "bearer token" - a token which the bearer can present to gain access to resources. Such a token has no built-in semantics and should be considered as nothing but a plain text string.

The bearer token is included in the HTTP Authorization header like this:

     GET /resource HTTP/1.1
     Host: server.example.com
     Authorization: Bearer SOME-TOKEN

Authorizing with client credentials only

This is just about the simplest flow possible (see http://tools.ietf.org/html/rfc6749#section-4.4): all the client has to do is to send it's client credentials (ID and secret) as if it was using HTTP basic authorization. In return the client receives an access token it can use in subsequent requests for the protected resources. Here is an example from the RFC:

     POST /token HTTP/1.1
     Host: server.example.com
     Authorization: Basic czZCaGRSa3F0MzpnWDFmQmF0M2JW
     Content-Type: application/x-www-form-urlencoded

     grant_type=client_credentials

The response could be:

     HTTP/1.1 200 OK
     Content-Type: application/json;charset=UTF-8
     Cache-Control: no-store
     Pragma: no-cache

     {
       "access_token":"2YotnFZFEjr1zCsicMWpAA",
       "token_type":"bearer"
     }

This flow allows the client to act on behalf of itself but it does not include any end user information.

Authorizing with both client credentials (API key) and user credentials (password)

If a client needs to act on behalf of the end user (the resource owner) then it can use the "Resource Owner Password Credentials Grant". This flow is almost as simple is the previous flow - all the client has to do is to add the client credentials to the authorization request. Here is an example from the RFC:

     POST /token HTTP/1.1
     Host: server.example.com
     Authorization: Basic czZCaGRSa3F0MzpnWDFmQmF0M2JW
     Content-Type: application/x-www-form-urlencoded

     grant_type=password&username=johndoe&password=A3ddj3w

The response could be:

     HTTP/1.1 200 OK
     Content-Type: application/json;charset=UTF-8
     Cache-Control: no-store
     Pragma: no-cache

     {
       "access_token":"2YotnFZFEjr1zCsicMWpAA",
       "token_type":"bearer"
     }

This flow allows the client to act on behalf of the end user. The client application must be trusted as the end user will have to pass his/her credentials to it.

Protecting credentials in transit

There is one big problem with the two simple flows mentioned above; both of them pass the client and end user credentials in clear text. For this reason OAuth2 makes it mandatory to use TLS/SSL to encrypt all of the requests. Neither does these flows cover scenarios with untrusted clients (such as a third party website that wants to access an end users resources on a different server) - but that scenario is covered in the "Authorization Code Grant" scenario which I will not discuss here.

The use of TLS/SSL protects the credentials from eavesdropping and man-in-the-middle attacks but it does not protect against sending credentials to the wrong server - either by misconfiguration or because the server has been compromised.

This lack of protection is perhaps one of the biggest difference between OAuth1 and OAuth2; OAuth1 protects credentials by signing requests instead of sending the raw credentials. This requires a rather complex signing algorithm which has caused many headaches over the time. OAuth2 trades the protection of credentials for simplicity which makes it a lot easier to use - but susceptible to leaking credentials to rogue servers.

It is possible to protect credentials from this problem though but it requires clients to use some complex crypto stuff to sign requests instead of sending the raw credentials (just like OAuth1 does). This is what Google does when it requires requests to be signed using JSON Web Signatures (JWS).

If you are using .NET then my Ramone HTTP library has support for OAuth2 with JWS as can be seen in this example: http://soabits.blogspot.com/2013/03/using-ramone-for-oauth2-authorization.html.

Protecting credentials in clients

One more word of caution - websites, mobile apps, desktop apps and similar with public clients cannot protect their client credentials. Other application may be able to do it (server to server integrations for instance) but public clients, be it JavaScript, byte code or assembler, can always be downloaded, decompiled and scrutinized by various forensic tools and in the end it will be impossible to keep client secrets from being stolen. This is a fact we have to live with.

Client credentials can be useful for statistics. They also give the ability to revoke credentials in order to block rogue applications - but they cannot be trusted in general.

Further reading

Eran Hammer has a discussion about the drawbacks of OAuth2 at http://hueniverse.com/2012/07/26/oauth-2-0-and-the-road-to-hell/ and here is another discussion of the drawbacks of bearer tokens http://hueniverse.com/2010/09/29/oauth-bearer-tokens-are-a-terrible-idea/

Ramone 1.2 released

This new version of Ramone adds a few utility methods to the existing library:

• Introducing cache headers on Request object:
    Request.IfModifiedSince()
    Request.IfUnmodifiedSince()
    Request.IfMatch()
    Request.IfNoneMatch()
• Introducing .NET cache policy on Session and Service:
    Session.CachePolicy
    Service.CachePolicy
• Adding Request.OnHeadersReady() for working with the underlying HttpWebRequest object.
• Adding Request.AddQueryParameters() as an alternative to binding with predefined URL templates.
• Adding XML settings in XmlConfiguration.XmlReaderSettings. Used when deserializing XML documents. Default is to allow DTD processing.

Ramone is a C# client side library for easy consuming of web APIs and REST services. It is available on GitHub: https://github.com/JornWildt/Ramone and as a NuGet package https://nuget.org/packages/Ramone/1.2.056.

Have fun, Jørn

Asynchronous HTTP requests using Ramone

Ramone (my C# web API and REST client) now supports asynchronous HTTP request :-) This has taken quite a while for me to implement due to a large amount of automatic tests for the the async request handling. But now its there and I hope some of you will find it useful in your applications.

Asynchronous requests can be prepared via a call to Async() on the Request object. After this you can do the actual request using any of the GET, POST, etc. methods where a callback delegate is passed as on of the arguments. When the request completes it will call back to the delegate, passing in the response from the request.

Here is one example:

// Define resource type
public class Cat
{
  public string Name { get; set; }
  public DateTime DateOfBirth { get; set; }
}

// URL template (relative to service root)
const string CatUrlTemplate = "/cat/{name}";

public static void GetAsync()
{
  // Create session pointing to service root
  ISession Session = RamoneConfiguration.NewSession(...));

  // Setup HTTP request
  Request req = Session.Bind(CatUrlTemplate, new { name = "Mike" });

  Console.WriteLine("Waiting for request to finish ...");

  // Initiate asynchronous request
  req.AcceptJson().Async()
     .Get<Cat>(response =>
       {
         Console.WriteLine("Cat: {0}.", response.Body.Name);
         Console.Write("Press Enter to complete: ");
       });

  Console.ReadLine();
}

It is also possible to register a callback delegate for error handling; if anything goes bad with the request (or if the operation callback delegate raises an exception) then this error delegate will be called.

In the same way it is possible to register a callback delegate to be called after the request has been completed. This delegate will always be called no matter what the outcome of the operation is.

Building on the previous example, we can now include the two OnError and OnComplete handlers:

// Initiate asynchronous request with additional handlers
req.AcceptJson().Async()
   .OnError(e => Console.WriteLine("Failed: {0}", e.Exception.Message))
   .OnComplete(() => Console.Write("Press Enter to complete: "))
   .Get<Cat>(response =>
     {
       Console.WriteLine("Cat: {0}.", response.Body.Name);
     });

POST, PUT and other operations usually includes a body in the request. This can easily be added as a parameter in the asynchronous operation methods:

// Initiate asynchronous POST including a request body
var body = ... any data object to POST ...
req.AcceptJson().Async()
   .Post<Cat>(body, response =>
     {
       Console.WriteLine("Cat: {0}.", response.Body.Name);
     });

Ramone will also handle redirects, authentication and all the other standard synchronous stuff when going asynchronous.

Have fun :-)

Using Ramone for OAuth2 authorization with Google APIs

I have blogged a couple of times about Ramone which is a C# client side library for interacting with RESTful web APIs (see https://github.com/JornWildt/Ramone). This time I will show how to use Ramone to authorize with Google's web services using OAuth2.

The OAuth2 flow to use is a variation of the "Client Credentials Grant" flow from OAuth2 (see http://tools.ietf.org/html/rfc6749#section-4.4) - but instead of sending the client credentials in clear text over the wire it uses a signed JSON Web Token (JWT) assertion instead (see http://tools.ietf.org/html/draft-ietf-oauth-json-web-token-06).

The concept itself is relatively simple: create a digitally signed piece of data (an assertion) that proves who the client is and then present that assertion to Google. It does although require quite a few steps to get it right.

Before we start making HTTP requests we must first acquire a set of client credentials from Google. This comes in the form of a X509 certificate which contains the private signing key for the client. To obtain this certificate we use Googles API console at https://code.google.com/apis/console. Login and go to "API access" where you click on the "Create client ID" button and select "Service account":

When your client credentials has been created you are asked to save them somewhere secure. Save them in a place where your program can load it from later.

The next step is to get Google's Token Endpoint URL. At the time of writing it is https://accounts.google.com/o/oauth2/token but this may of course change in the future.

Now we are ready to rock! First we initialize Ramone with the Google stuff:

using System;
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;
using Ramone;
using Ramone.OAuth2;
using Ramone.Utility.JsonWebToken;

const string GoogleAPIBaseUrl = "https://www.googleapis.com/oauth2/v1";
const string TokenEndpointUrl = "https://accounts.google.com/o/oauth2/token";

ISession Session = RamoneConfiguration.NewSession(new Uri(GoogleAPIBaseUrl));

OAuth2Settings settings = new OAuth2Settings
{
  TokenEndpoint = new Uri(TokenEndpointUrl),
  ClientAuthenticationMethod = OAuth2Settings.DefaultClientAuthenticationMethods.Other
};

Session.OAuth2_Configure(settings);

Then we load the certificate we got from Google. That is albeit not as easy as it sounds since the default X509 certificate doesn't support the RSA256 signing required by the standard. So we have to load the certificate first and then create a new RSACryptoServiceProvider from that:

private static RSACryptoServiceProvider GetCryptoServiceProvider()
{
  const string CertificatePath = "path-to-your-certificate-file";

  X509Certificate2 certificate = new X509Certificate2(CertificatePath, "notasecret", X509KeyStorageFlags.Exportable);

  using (RSACryptoServiceProvider cp = (RSACryptoServiceProvider)certificate.PrivateKey)
  {
    // Create new crypto service provider that supports SHA256 (and don't ask me why the first one doesn't)
    CspParameters cspParam = new CspParameters
    {
      KeyContainerName = cp.CspKeyContainerInfo.KeyContainerName,
      KeyNumber = cp.CspKeyContainerInfo.KeyNumber == KeyNumber.Exchange ? 1 : 2
    };

    return new RSACryptoServiceProvider(cspParam) { PersistKeyInCsp = false };
  }
}

At last we can authenticate with Google using the client credentials we loaded from the certificate. The "Issuer" value below is the e-mail address of your "Service account" client as stated in Google's API console:

const string Issuer = "your-client-issuer-email-address-from-google";
const string Scope = "https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/userinfo.profile";

using (RSACryptoServiceProvider csp = GetCryptoServiceProvider())
{
  AssertionArgs args = new AssertionArgs
  {
    Audience = TokenEndpointUrl,
    Issuer = Issuer,
    Scope = Scope
  };

  Session.OAuth2_GetAccessTokenFromJWT_RSASHA256(csp, args);
}

The last statement makes the actual request to Google with the signed JWT assertion, reads the returned OAuth2 access token and stores it in the session for use in the following requests.

If this works you are done. Ramone has authorized with Google using your client credentials without any intervention from a user.

To prove that it works we can now fetch the user name and e-mail of the current user (your client) identified by the access token returned from Google:

Console.WriteLine("Reading user information from Google");
using (var response = Session.Bind("userinfo").AcceptJson().Get<dynamic>())
{
  var body = response.Body;
  Console.WriteLine("\nRESULT:");
  Console.WriteLine("User name: " + body.name);
  Console.WriteLine("E-mail: " + body.email);
}

Have fun :-)

JSON-Patch support in Ramone

Ramone (my C# web API and REST client) now supports JSON patch documents as defined in http://tools.ietf.org/html/draft-ietf-appsawg-json-patch-08. This is yet a draft standard so Ramone's implementation may need some tweaking later on to work.

A patch document represents a set of changes to a JSON resource. This can be “add” new value, “remove” value or other similar operations.

In Ramone we use the class JsonPatchDocument to build a patch document. For instance like this:

JsonPatchDocument patch = new JsonPatchDocument<myresourcetype>();

// Supply path as a string
patch.Replace("/Title", "My new title");

// Supply path as a typed referenc
patch.Replace(d => d.Title, "Another title");

The above example corresponds to this json-patch document (which you can get by calling ToString() or Write() on the patch variable):

[
  { op: "replace", path: "/Title", value: "My new title" },
  { op: "replace", path: "/Title", value: "Another title" }
]

The use of lambdas like d => d.Title is a simple and type safe way to supply a patch path with respect to a given resource class. Using lambdas like this ensure paths are updated when refactoring variable names in Visual Studio.

Patch documents can be sent to a server like this:

JsonPatchDocument patch = ... build patch document ...

Request request = Session.Bind(ResourceUrl);

using (var response = request.Patch(patch))
{
  ... do stuff with response ...
}

Ramone also supports reading and applying patch documents (using the “visitor” pattern). You do although have to implement the actual operations yourself—Ramone cannot do that for you. Here is one example:

using (TextReader r = ... get reader from some JSON input ...)
{
  // Read patch document from input
  JsonPatchDocument patch = JsonPatchDocument.Read(r);

  // Create instance of patching logic
  MyPatchVisitor visitor = new MyPatchVisitor();

  // Apply patch document
  patch.Apply(visitor);
}


// Implements "visitor" pattern where each patch operation in 
// the patch document will be translated to a call to Add(...),
// Replace(...) and so on.
class MyPatchVisitor : JsonPatchDocumentVisitor
{
  public override void Add(string path, object value)
  {
    ... implement "add" logic for target data ...
  }
}

There is also an optional typed version of the JsonPatchDocumentVisitor which will simplify your operations slightly:

class MyTypedPatchVisitor : JsonPatchDocumentVisitor<myresourcetype>
{
  public override void Add(string path, object value)
  {
    // Test path against delegate and cast value to int if
    // they match. Then execute the action delegate.
    IfMatch<int>(r => r.Id, path, value,
      v => ... do stuff with integer "v"...);
  }
}

HTTP PUT, PATCH or POST - Partial updates or full replacement?

I have recently been working on the write side of a REST service for managing case files. During this work I have gone through a lot of discussions about partial updates versus full updates of resources in a RESTful service - and whether to perform these updates using the HTTP verbs PUT, PATCH or POST.

In the end we settled on using PATCH for partial updates (with the media type application/json-patch) and PUT for complete updates. That is in itself not so surprising, it is how we got to these decisions that I would like to share.

Example resource model

For demonstration purposes I will use the representation of a bug report from some fictive bug reporting system. A GET of such a bug report would return the following XML which should be rather self explaining:

<BugReport>
  <Id>15</Id>
  <Title>Program crashes when hitting ctrl-P</Title>
  <Status>Open</Status>
  <Responsible id="22">
    <Name>John Smith</Name>
    <Link rel="self" href="{url-to-person}" title="John Smith"/>
  </Responsible>
  <Link rel="self" href="{url-to-bug-report}" title="This bug report"/>
</BugReport>

Complete updates with PUT or POST

At first we wanted to use PUT with URL encoded key-value pairs for complete updates. We wanted to make it clear that such updates were idempotent and thus PUT seemed to be a perfect match - it signals full idempotent replacement of the target resource (see for instance http://www.emergentone.com/blog/http-methods-and-idempotence/ for an explanation of "idempotent").

There is although no reason to PUT all the server-generated stuff like the links and bug report ID when performing an update. But that meant we were not doing a complete update any more - and so we entered the foggy zone of unclear semantics for PUT: is it, or is it not, allowed by the HTTP specification to do such a full-but-somehow-also-partial update using PUT? Some people say yes, other people say no ...

Due to this uncertainty we gave up using PUT and turned to POST instead. This would not signal idempotency as we wanted, but it would certainly be a legal and standard use of POST.

Partial updates with POST

In the end it turned out that we could have saved us the trouble of discussing these "quite-but-not-entirely-unlike-partial" updates using PUT because it soon became apparent that "real" partial updates was a much better fit for our use case. The business case for implementing updates in the system was an integration service that transferred changes from one external system into our system - with heavy focus on only transferring changes from A to B.

We chose the intuitive solution of interpreting missing values (or, rather, null values) as "do not change this property". That would allow the client to POST an update to the Title property only as:

  POST /bug-report-url
  Content-Type: application/x-www-form-urlencoded

  Title=A%20new%20title

That worked well for the Title property, but how about the Responsible property? It had to be possible to either 1) ignore the Responsible, 2) set it to something, or 3) clear it ... But if a null value meant "ignore" what should then be used for "clear"? We decided to add a new (boolean) field "NoResponsible", so we could clear the responsible with a POST like this:

  POST /bug-report-url
  Content-Type: application/x-www-form-urlencoded

  NoResponsible=true

Somehow that seemed quite a bit hacked - what other sorts of artificial key-value fields would we need to implement the full solution?

The move to PATCH

This was surely not going in the right direction, so we started researching partial updates again and looked at the HTTP verb PATCH with the media type application/json-patch (see http://tools.ietf.org/html/draft-ietf-appsawg-json-patch-08). This turned out to be a perfect match for our use case where the integration component could build up a patch document, based on the changes that needs to be transferred, and then apply that to the bug report resource.

A json-patch document is basically a list of operations to be applied to the target resource. Here is an example of how such a patch document can be used to update the title of our fictive bug report and remove the responsible person at the same time:

  PATCH /bug-report-url
  Content-Type: application/json-patch

  [
    { op: "replace", path: "/Title", value: "New title" },
    { op: "remove", path: "/Responsible" }
  ]

Back to complete updates using PUT

PATCH turned out to be perfect for our integration project. So far so good. But another relevant use case could soon be to support some kind of human facing application that works like this:

1) User opens an existing bug report.

2) User modifies the bug report.

3) User saves the new version of the bug report.

In this case tracking changes for a patch document may not be desirable, so instead we should use the well known pattern of GET resource - modify local representation - PUT result back, with the relevant use of ETag headers etc. to avoid lost updates (see for instance http://www.w3.org/1999/04/Editing/).

My point here is that even though we discarded PUT for partial updates earlier on, doesn't mean that it cannot be done - but you need to do it right with GET+PUT and ETag headers - and that is quite a bit more complex than using PATCH as described above.

json-patch with Ramone

Since we use Ramone internally for our client applications, it was natural to add support for json-patch here - which, not surprisingly, is the topic for my next blog post.

Other approaches (which we found less useful)

Doing partial updates using PUT with sub-resources

Another common way of doing idempotent partial updates using PUT is to represent certain properties as sub-resources that contains a subset of the main resource. We could for instance move the Responsible property to a resource of it's own with only that value and then link to it in the main resource.

Such a solution would work, but 1) it would require lots of boiler plate code to implement, and, 2) more importantly, it would not support atomic and transactional updates to multiple properties in one request.

Encoding modified property names in the URL

Some people has suggested putting the names of the properties to modify in the URL of the request, thus resulting in a new URL which can be updated using a complete PUT. For instance, to update only the Title in our example we could make a PUT like this:

  PUT /bug-report-url;Title
  Content-Type: application/x-www-form-urlencoded

  Title=New%20title

But this requires the client to build URLs with the assumption of a given URL structure - something to be avoided in hyper media based APIs (since this would couple the client with the server's URL structure).

Further more, I cannot foresee if it will work for more complex structures, so why not use a standardized approach like json-patch?

Using the XML patch framework

We might as well have used the XML patch framework (see http://tools.ietf.org/html/rfc5261) instead of json-patch for the patch document. That would certainly be more in line with the XML representation we use already. The main reason for not doing this is very simple and pragmatic - we discovered the XML framework too late.

Should I although choose between the two patch formats today, I would probably choose json-patch again for the simple reason that it matches our data better - XML is only a thin wrapper around the public data model and JSON might in fact have been a better choice. The XML patch framework is quite over-engineered for the kind of data we serve.

UPDATE (January 3, 2013): Is this RPC?

Someone on Stackoverflow commented that this "sending commands" (via patch) is more RPC style than REST. Let me try to cover that issue ...

If you define RPC as sending commands to a server then any and all HTTP operations are RPC calls by definition - whether you GET a resource, PUT a new representation or DELETE it again - each of them consist of a sending a command (verb) GET/PUT/DELETE etc. and a optional payload. It just happens that the HTTP working group (or who ever it is) has introduced a new verb PATCH which allows clients to do partial updates to a resource.

If anything else than sending the complete representation to the server is considered RPC style, then, by definition, partial updates cannot be RESTful. One can choose to have this point of view, but the people behind the web infrastructure says differently - and has thus defined a new verb for this purpose.

RPC is more about tunneling method calls through HTTP in a way that is invisible to intermediaries on the web - for instance using SOAP to wrap method names and parameters. These operations are "invisible" since there are no standards defining the methods and parameters inside the payload.

Compare this to PATCH with the media type application/json-patch - the intention of the operation is clearly visible to any intermediary on the web since the verb PATCH has a well defined meaning and the payload is encoded in another well defined public available format owned by common authority on the web (IETF). The net result is full visibility for everybody and no application specific secret semantics.

REST is also about "serendipitous reuse" which is exactly what PATCH with application/json-patch is - reusing an existing standard instead of inventing application specific protocols that do more or less the same.

Ramone: Media types and codecs

One of the basic building blocks of REST is the concept of a media type - the file format used to represent a resource on the web. Media types comes in many different flavours - images, PDF, vCard, XML, JSON, spreadsheets and so on, each of them having their own specific formats and capabilities. If you haven't done it already then take a look at my previous post where I go deeper into details about media types.

Media types are considered first class citizens of Ramone, my C# library for consuming web APIs and RESTful services on the web - just like the uniform interface (GET/POST/PUT/...) and resource identifiers (URLs) - and in this post I will show how to work with different kinds of media types.

Codecs

A codec is a class that translates to and from the file format on the wire and some kind of internal representation in C#. To do so it must first implement either IMediaTypeWriter, IMediaTypeReader or both and then register with the current codec manager such that Ramone will be able to find it.

The codec interfaces are rather simple:

  public interface IMediaTypeCodec
  {
    object CodecArgument { get; set; }
  }

  public interface IMediaTypeWriter : IMediaTypeCodec
  {
    void WriteTo(WriterContext context);
  }

  public interface IMediaTypeReader : IMediaTypeCodec
  {
    object ReadFrom(ReaderContext context);
  }

The context parameter contains references to the current session, the data stream, HTTPRequest, HTTPResponse and others that are available for the codec.

Decoding an HTML micro format

One example of a codec is the BlogCodec from Ramone's test library. This codec demonstrates how to decode a (non-standard) micro format from an HTML page that shows a blog listing (see https://gist.github.com/2305777 for the actual HTML).

  public class BaseCodec_Html : TextCodecBase<Resources.Blog>
  {
    // This method is from TextCodecBase which has wrapped the binary input stream in a TextReader
    // using the charset encoding stated by the client's request headers.
    protected override Resources.Blog ReadFrom(TextReader reader, ReaderContext context)
    {
      // Using HtmlDocument from HtmlAgilityPack
      HtmlDocument doc = new HtmlDocument();
      doc.Load(reader);
      return ReadFromHtml(doc, context);
    }

    protected Resources.Blog ReadFromHtml(HtmlDocument html, ReaderContext context)
    {
      HtmlNode doc = html.DocumentNode;

      List<Resources.Blog.Post> posts = new List<Resources.Blog.Post>();

      // Scan through HTML and look for "class" attributes identifying values
      foreach (HtmlNode postNode in doc.SelectNodes(@"//div[@class=""post""]"))
      {
        HtmlNode title = postNode.SelectNodes(@".//*[@class=""post-title""]").First();
        HtmlNode content = postNode.SelectNodes(@".//*[@class=""post-content""]").First();
        List<Anchor> links = new List<Anchor>(postNode.Anchors(context.Response.ResponseUri));

        posts.Add(new Resources.Blog.Post
        {
          Title = title.InnerText,
          Text = content.InnerText,
          Links = links
        });
      }

      // Extract all HTML anchors together with <head> links and store them as ILink instances
      List<ILink> blogLinks = new List<ILink>(doc.Anchors(context.Response.ResponseUri).Cast<ILink>().Union(doc.Links(context.Response.ResponseUri)));

      // Create and return an object that represents the data extracted from the HTML
      Resources.Blog blog = new Resources.Blog()
      {
        Title = doc.SelectNodes(@".//*[@class=""blog-title""]").First().InnerText,
        Posts = posts,
        Links = blogLinks
      };

      return blog;
    }

    // This method is also from TextCodecBase, but is not used (since we do not write HTML)
    protected override void WriteTo(T item, System.IO.TextWriter writer, WriterContext context)
    {
      throw new NotImplementedException();
    }
  }

You can read a bit more about using hyper media links in another of my earlier posts.

Other codec examples

Other examples of codecs could be:

• Decoding cooking recipe data from XML or JSON.
• Decoding binary image data.
• Decoding CSV into tabular data.
• Decoding and writing vCard information.

Built-in generic codecs

All of the previous codecs has been "typed" in the sense that they decode response data into a typed object with the specific properties needed. But Ramone has also built-in support for various generic formats such as XML, JSON and HTML.

Here is an example use of the XML codec which decodes into C#'s XML DOM class XmlDocument:

  Request req = Session.Bind("... some url ...);

  XmlDocument doc = req.Get<XmlDocument>().Body;

The JSON codec can do some nifty stuff with C# dynamics:

  Request req = Session.Bind("... some URL for cat data ...");

  dynamic cat = req.Accept("application/json").Get().Body;

  Assert.IsNotNull(cat);
  Assert.AreEqual("Ramstein", cat.Name);

Advantages of typed codecs versus generic codecs

By working with typed codecs you gain a few advantages over the generic ones:

1. The application code is completely decoupled from the wire format. This gives you the ability to work with different wire formats without changing the application code, e.g., decoding both JSON, XML, and vCard into the same internal representation.
2. It results in more readable application code.
3. It makes the parsing code reusable across difference pieces of application code.

The downside of codecs is that you have to write a few more pieces of boilerplate code to create and register them. You also have to implement the client side representation of the resource as a specific class. But in the end it all pays off, in my opinion, and yields more readable and maintainable code.

Update (20/04/2012): Codec Manager

I forgot to show how codecs can be registered with either the current service (more on services at a later time); Codecs must be registered with Ramone, otherwise they will simply be ignored. To do so you first grab a reference to ICodecManager and then call AddCodec(...):

  ICodecManager cm = MyService.CodecManager;
  cm.AddCodec<MyClass, MyCodec>(MyMediaType);

Here MyClass is the type of object returned or written by the codec, MyCodec is the type of the codec and MyMediaType is the media type id string, e.g., "application/vnd.mytype+xml".


Ramone can be downloaded from https://github.com/JornWildt/Ramone

Ramone: Consuming Hyper-Media REST Services in C#

Hyper-media controls are one of the core features of the World Wide Web as we know it today - without them there would be no way to link web pages and no way to interact with them through web forms. One of the REST constraints is the famous HATEOAS - "Hyper-media As The Engine Of Application State" and in order to consume RESTful web services we should make hyper-media controls first class citizens of the client interface we work with.

Hyper-media controls can be classified in different ways (see for instance Mike Amundsen's classification), but in Ramone we distinguish between the two well-known hyper-media elements; links and key/value forms as we know them from HTML, ATOM and more. These are represented with the following C# interface definitions:

  public interface ILink
  {
    Uri HRef { get; }
    string Title { get; }
    IEnumerable<string> RelationTypes { get; }
    MediaType MediaType { get; }
  }

  public interface IKeyValueForm
  {
    IKeyValueForm Value(string key, object value);
    IKeyValueForm Value(object value);
    Request Bind(string button = null);
  }

The idea is to use some C# extension method to get an instance of one of the above interface from the resource representation in the current response. This means in HTML there are methods for extracting <a> elements, <link> elements and <form> elements and return them as instances of the hyper-media interfaces. As more media-types are added to Ramone so will more similar extension methods be added - this can be done as add-ons without touching the core code.

The following example is from Ramone's Blog test collection (please take a closer look at the Github code since it includes some more explanations). This example shows how to follow an "author" link from an HTML page that shows a list of blog entries. The HTML contains (non-standard) micro formats for identifying various elements of the blog (post, author, title).

The actual HTML can be found here: https://gist.github.com/2305777#file_ramone_blog_list_html.html (and the HTML for an author can be found here: https://gist.github.com/2305988#file_gistfile1.html)

Here we go:

  
  // Create request by binding well known path to session's base URL
  Request blogRequest = Session.Bind(BlogRootPath);

  // GET blog HTML represented as an HtmlDocument (from Html Agility Pack)
  Response<HtmlDocument> blog = blogRequest.Get<HtmlDocument>();

  // Select first HTML anchor node with rel="author" as a anchor link.
  // This uses HtmlDocument specific extension methods to convert from anchor to ILink
  ILink authorLink = blog.Body.DocumentNode.SelectNodes(@"//a[@rel=""author""]").First().Anchor(blog);

  // Follow author link and get HTML document representing the author
  HtmlDocument author = authorLink.Follow(Session).Get<htmldocument>().Body;

  // Check e-mail of author
  HtmlNode email = author.DocumentNode.SelectNodes(@"//a[@rel=""email""]").First();
  Assert.AreEqual("pp@ramonerest.dk", email.Attributes["href"].Value);

The two important pieces of code in this example are the .Anchor(...) and .Follow(...) parts that respectively instantiates a link and follows it.

Forms can be loaded and submitted in a similar fashion.

Hopefully this gives an understandable introduction to working with hyper-media in Ramone - and, even more important, I hope you find it relatively intuitive and easy to work with.

Introducing the Ramone C# Library for Web API Clients

Ramone is a C# library for client applications that want to access some of the many web APIs available today.

What Ramone does is to wrap a lot of the bread and butter code into a library for easy reuse (see this previous post for my motivation). Out of the box it offers XML, JSON, ATOM and HTML formatters, URL templating and parameter binding, hyper-media traversal, form handling, and has HTTPs "Uniform Interface" as first class citizen in the interface.

With Ramone you won't see methods like GetOrders(...) or SaveUserData(...). You will instead be working with URLs that points to resources and use HTTPs uniform interface to interact with these. This means for instance that GetOrders(...) may become OrderRequest.Get() and SaveUserData(...) becomes UserRequest.Put(...).

The smallest example I can think of is this little four-liner that fetches the timeline of a specific Twitter account:

// All interaction with Ramone goes through a session (which defines a service base URL)
ISession session = RamoneConfiguration.NewSession(new Uri("https://api.twitter.com"));

// Creating a Ramone request by binding variables to a URI template
Request request = session.Bind(UserTimeLineTemplate, new { screen_name = ScreenName, count = 2 });

// GET response from Twitter
Response response = request.Get();

// Extract payload as a C# dynamic created from the JSON response
dynamic timeline = response.Body;

What happens here is:

1. A session is created. This carries information about the base URL for template binding and keeps track of client state (cookies among other things).
2. A request is created by binding a URL template with parameters. Input is a relative URL template path ("/1/statuses/user_timeline.json?screen_name={screen_name}&count={count}") and an anonymous parameter object. Output is a request for an absolute URI.
3. One of the operations from HTTPs Uniform Interface is called (GET) and the response is recorded.
4. Ramone checks the returned media-type, sees it is application/json, and converts it to a dynamic C# object with late binding of the returned values.

The code for iterating through the timeline and printing status updates is plain C#:

Console.WriteLine("This is the timeline for {0}:", ScreenName);
Console.WriteLine();

foreach (dynamic tweet in timeline)
{
  Console.WriteLine("* {0}.", tweet.text);
  Console.WriteLine();
} 

A few things worth noticing:

1. Twitter may not be the best REST example out there, but it is well understood by many people and is a web API which is quite easy to work with.
2. Ramone does not only handle JSON. It works with a repository of parsers and formatters (called codecs) which are responsible for handling the various formats Ramone may encounter.

You can find a complete Twitter demo at Github: https://github.com/JornWildt/Ramone/tree/master/TwitterDemo/TwitterDemo. This demo shows both how to fetch a timeline as well as how to post status updates with OAuth1 authentication.

All source code is available on Github: https://github.com/JornWildt/Ramone. At the time of writing there are no official binaries released so you will have to download the code from Github and compile it yourself.

Later on I will write about how Ramone helps you consuming more RESTful hyper-media based services by making links, link relations and forms first class citizens of the interface. For now you can check the blog example at Github: https://github.com/JornWildt/Ramone/tree/master/Ramone.Tests/Blog

Have fun!

Consuming Web APIs in C# with Ramone

The World Wide Web is a fascinating place with its enormous amount of information right at your hand. It is even more fascinating how relatively simple the upper application layer is: we have resources, URLs - Uniform Resource Locators which we can dereference, a small set of methods to operate on said resources - GET/POST/PUT/DELETE (and then some), and media-types for identifying how those resources are represented.

Roy T. Fielding wrapped it all up in his dissertation "Architectural Styles and the Design of Network-based Software Architectures" where he coined the term REST - "Representational State Transfer". Somehow this term caught on and developers, who was trying to find something more simple than SOAP (over HTTP), started looking at REST for machine-to-machine APIs on the web. Soon we got REST APIs - and plenty of hype around them. Whether or not these APIs are actually RESTful is an ongoing debate, but it should be safe to call them "Web APIs" no matter how they present themself, so I will stick to that term for now.

Today there are thousands of public web APIs available for anyone to access - just take a look at http://www.programmableweb.com/ - at the time of writing it boasts some 5529 APIs. And those are just the public APIs - in addition to this there's probably many many more in closed enterprises and partnerships.

When I first started working with web APIs in C# I soon got a bit tired of the rituals you have to go through in order to work with HTTP in .NET. Usually I have a URL to some resource, the name of a method I want to invoke, maybe a payload (a C# object instance), and then a returned resource representation (also an object instance). So I should be able to write something like this:

  Uri url = new Uri("...");
  Payload p = new Payload(...);
  Resource r = url.<Method>(p);

You can do this in a few lines of code with .NET's WebRequest class ... except that you get absolutely no help when it comes to serializing Payload and Resource data. And what format are those classes actually serialized as? Is it XML, JSON, HTML or something else?

The next problem with low level HTTP access using WebRequest is handling of authorization using either HTTP Basic, OAuth or something similar - these are standardized authorization mechanisms, but you need to add them on top of HTTP yourself.

So there is a lot of bread and butter code for you to write before you can get up and running with your first web API. Luckily it is all based on the uniform interface, URLs and media-types - something that can be wrapped into a framework and lower the entry barrier for consuming web APIs - whether they are RESTful or not.

That is what Ramone is all about - making it easy to consume web APIs in C#. Interacting with the web should be as easy going as, well, hmm, Ramone from Pixar's movie Cars, from which I got the name.

Hopy you enjoy it! More posts will follow, showing how to get started with Ramone.

All source code is available on Github: https://github.com/JornWildt/Ramone (distributed under the MIT license).

Happy hacking, Jørn Wildt