tag:blogger.com,1999:blog-73951723573685534382024-02-19T17:36:40.343+01:00SOA Bits and RamblingsVarious bits and pieces from my SOA/ROA/Whatever learnings.Elfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.comBlogger28125tag:blogger.com,1999:blog-7395172357368553438.post-50421171398696411162015-07-01T09:58:00.001+02:002015-07-01T09:58:33.385+02:00Introducing the Mason Cook BookI have started an on-line book of recipes for building hypermedia APIs with the Mason (hyper)media type. It is very much work in progress but hopefully I can add one or two recipes a week on my daily train commute to and from work.<br />
<br />
The book source is hosted on <a href="https://github.com/JornWildt/MasonCookBook">GitHub</a> and is made available on-line at <a href="http://jornwildt.gitbooks.io/mason-cook-book/content/">GitBook</a> in various formats. Later on I may host it another server depending on how GitBook turns out to work with - but so far it has been a nice experience.<br />
<br />
You are most welcome to add issues, fork it and send me pull requests for improvements.Elfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com0tag:blogger.com,1999:blog-7395172357368553438.post-70330202101936271912015-06-16T17:23:00.004+02:002015-06-16T17:23:58.111+02:00Mason Draft 2 ready for useI am happy to announce that Mason (the JSON + hypermedia format) is now ready for use in Draft 2. See <a href="https://github.com/JornWildt/Mason/blob/master/Documentation/Mason-draft-2.md">https://github.com/JornWildt/Mason/blob/master/Documentation/Mason-draft-2.md</a>.<br />
<br />
For those that haven't heard about Mason before - please take a look at <a href="https://github.com/JornWildt/Mason">https://github.com/JornWildt/Mason</a>. Mason is a JSON based format with conventions for representing API data and hypermedia control elements.<br />
<br />
This version combines links, actions and link templates into one single @controls object. As the name indicates this object contains all the hypermedia elements that control the application.<br /><br />This move to @controls makes it possible to combine link templates with POST data - and should make it easier to parse and represent hypermedia elements in code.<br /><br />There are currently no further ideas in the pipeline that could change how things are represented in Mason. Future versions may though add new features.<br />
<br />
Feedback is as always appreciated :-)<br />
<br />
<br />
<br />Elfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com0tag:blogger.com,1999:blog-7395172357368553438.post-47939205244518136612014-12-19T08:19:00.001+01:002014-12-19T08:19:23.653+01:00Announcing Macaroons.Net - a C# implementation of macaroons authorization credentialsIn my <a href="http://soabits.blogspot.dk/2014/12/macaroons-authorization-credentials.html" target="_blank">previous blog post</a> I introduced "macaroons" - a new technique for creating authorization credentials.<br /><br />But I even went a bit further and created a C# implementation of macaroons called Macaroons.Net. You can find it on GitHub at <a href="https://github.com/JornWildt/Macaroons.Net">https://github.com/JornWildt/Macaroons.Net</a> together with examples and documentation.<br /><br />Have fun.<br />Elfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com0tag:blogger.com,1999:blog-7395172357368553438.post-19189067739337065802014-12-19T08:14:00.001+01:002014-12-19T08:14:39.214+01:00Macaroons authorization credentials, better than cookiesThis was the year I first heard of the term "macaroon" used in the context of the web, authorization frameworks and other crypto stuff. I had absolutely no idea of what it was, but "the web" said it was good, so I got curious and did some studying. It turned out to be a technique for creating authorization tokens with some very interesting properties:<br />
<ul>
<li><i>Proof carrying</i>: a macaroon carries its own proof of authorization, cryptographically secured.</li>
<li><i>Delegation</i>: a macaroon can be given to another user who can then act on your behalf.</li>
<li><i>Attenuation</i>: any user can further restrict (attenuate) the authorization before sharing the macaroon with others.</li>
<li><i>Distributed authorization</i>: any user can require authorization by other services before using a macaroon.</li>
</ul>
<br />
Here is a small scenario illustrating the above features:<br />
<ol>
<li>Alice wants to share a set of images on a photo sharing website. Lets call it "Phlocker" for now.</li>
<li>Alice asks Phlocker to create a macaroon which enables access to exactly those images.</li>
<li>Alice sends the macaroon to Bob.</li>
<li>Bob receives the macaroon, most likely as part of a URL to Phlocker, and he uses it to access the images.</li>
<li>Bob decides to share exactly one of the images with Cecilia. So Bob creates a new macaroon from the original macaroon <i>without involving Phlocker at all</i>. This new macaroon "attenuates" the original macaroon and restricts it to the single image.</li>
<li>Bob does not want Cecilia to share the image with others, so he attenuates the macaroon even further by adding the requirement that only Cecilia, as identified by her Twitter account, should be allowed to use the macaroon.</li>
<li>Bob sends the macaroon to Cecilia.</li>
<li>Cecilia receives the macaroon and sees the requirement of being logged into Twitter as Cecilia.</li>
<li>Cecilia interacts with Twitter to prove her identity and Twitter issues a <i>discharge</i> macaroon to Cecilia. This is all done without Twitter knowing why Cecilia needs the authorization.</li>
<li>Cecilia prepares the discharge macaroon for use at Phlocker and sends it together with the original macaroon she received from Bob.</li>
<li>Phlocker checks Cecilia's macaroon from Bob, recognizes the Twitter identity requirement and verifies it with the discharge macaroon.</li>
<li>Cecilia gets to see the single image without being able to see the other images that Alice originally authorized Bob to access.</li>
</ol>
<br />
The technology for this flow is not yet perfect as there are some interoperability problems regarding how to encode third party requirements like "being authorized as Cecilia @ Twitter". But the underlying crypto stuff is solid and published in the paper "<a href="http://research.google.com/pubs/pub41892.html" target="_blank">Macaroons: Cookies with Contextual Caveats for Decentralized Authorization in the Cloud</a>" by Arnar Birgisson, Joe Gibbs Politz, Úlfar Erlingsson, Ankur Taly, Michael Vrable and Mark Lentczner.<br />
<br />
Personally I think this little piece of technology has a great potential for creating distributed system.Elfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com0tag:blogger.com,1999:blog-7395172357368553438.post-57012930710094209022014-08-02T23:19:00.000+02:002014-08-02T23:21:07.596+02:00Take away from the 2014 API-Craft conference in DetroitLast week I attended the <a href="http://apicraft.org/">2014 API-Craft conference</a> in Detroit - and I had a great time! Met lots of interesting people, had great conversations and learned a lot about what other people make out there.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOdp1z-IAbYFfbn1gPbpw8syB-KIvBwk9crBZVjRWxp1-oKs7gB9wPfykr7nO0SvK27sj6n_6LmCSbFZxxP10k9KMzr6BLyw3V6BVoWtilkCwj8IkGhra8ZJNra8E3JOdjp78QBXlJIA/s1600/Billede+29-07-14+16.05.36.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOdp1z-IAbYFfbn1gPbpw8syB-KIvBwk9crBZVjRWxp1-oKs7gB9wPfykr7nO0SvK27sj6n_6LmCSbFZxxP10k9KMzr6BLyw3V6BVoWtilkCwj8IkGhra8ZJNra8E3JOdjp78QBXlJIA/s1600/Billede+29-07-14+16.05.36.jpg" height="297" width="400" /></a></div>
<br />
<br />
Some of the stuff I saw and listen to in Detroit made a few pieces of puzzle click together in my brain. First there was <a href="https://twitter.com/zdne">Z</a> who reminded me that we still don't have any good description language for documenting resource oriented hypermedia APIs. I've heard that before and been wondering how it would look like.<br />
<br />
Then I talked with Sergey and <a href="https://twitter.com/zeldigas">Dmitri</a> who showed me the resource diagram for their API - this one reminded me so much of UML diagrams that I think we should look into that for inspiration.<br />
<br />
I was also at the <a href="https://github.com/apicraft/detroit2014/wiki/Documenting-Hypermedia-API">"Documenting APIs" session</a> where I suddenly found myself talking about our own API documentation - which is written in Word (and that's okay, kind'a). It made me realize that, at its core, such a documentation consists of resource descriptions, link relation descriptions, action descriptions and of course some kind of overview, introduction and tutorial sections.<br />
<br />
Someone asked us to explain what a hypermedia API is in one sentence. I came up with "<i>That is an API which can be documented without writing one single absolute URL or relative URL path</i>". I don't exactly think that will catch on, especially not outside this community, but I do believe it is true - if you need to explain URL structures then you are missing something. Unfortunately that one sentence doesn't tell you how to actually implement such an API and as such it is really not very useful.<br />
<br />
[I'm pretty sure there is a good Yoda quote to put in here ... like, you now ... "The links with you very strong are here, Luke". Anyone got a better one?]<br />
<br />
<a href="https://twitter.com/davidgoldberg">Dave</a> introduced me to his JSON based description language for APIs and it made me realize that you can do awesome things even in a dead simple format like JSON. Why having to state <twice>everything</twice> when you can do { "single": "everything" }.<br />
<br />
Then there were people talking about resources and state machines (thanks again Z). Personally I don't believe much in that analogy - it kind of confuses the graph of resource relationships with the graph of a way-too-simple-never-to-be-found-in-real-life state machine. Z did try to convince me about this but never succeeded - maybe because I am highly colored by my PhD thesis on automatic verification of obnoxiously great state machines :-)<br />
<br />
We also got a great deal around API explorers/browsers of various kinds. It seems like every media type have such one now a days - and it makes great sense to me. I truly believe in the power it has when it comes to learning a new API. Thanks to <a href="https://twitter.com/klevland">Kristopher Kleva</a> for his story about links, actions and the Sirene browser (you really should blog about that experience!).<br />
<br />
I also met <a href="https://twitter.com/gamache">Pete Gamache</a> who is the guy behind <a href="https://github.com/gamache/hyperresource">HyperResource</a> - unfortunately we never really spoke together. He's hypermedia client for Ruby is an interesting solution for interacting with hypermedia APIs in code. Its not exactly new as the same kind of ideas exists in the older tool <a href="http://restfulie.caelum.com.br/">Restfulie</a> (for Ruby) and my own <a href="https://github.com/JornWildt/Ramone">Ramone</a> library (for C#) - but none the less it shows that there are something worth pursuing here.<br />
<br />
All the talks about tooling, SDKs and object models for API resources made me want to work a bit on Ramone and Mason again. It could be cool to make a C# hypermedia resource representation that would allow the client developer to follow links and execute actions in an easy and intuitive way - while at the same time be able to hide some of the stuff that complicates client code in order to make it long lasting and handle (some types of) API changes.<br />
<br />
So, thanks to all of you - also those of you that I have forgotten to mention here and of course the people at <a href="http://apigee.com/about/">Apigee</a> that made all of this possible!<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4vfoERkfsd8hOMnq-aB6KkTrHGVxQ4aX37Oi85z12ReGcuThumWuEVYY_KyYP7Of71Q2F_ncR0PGqEmQLFJqZ8WhvkgAnR4sFQ36R-68rUHG1iBwx__qAWrWCNlXPZX_dtk-8va7XvA/s1600/Billede+29-07-14+23.55.28.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4vfoERkfsd8hOMnq-aB6KkTrHGVxQ4aX37Oi85z12ReGcuThumWuEVYY_KyYP7Of71Q2F_ncR0PGqEmQLFJqZ8WhvkgAnR4sFQ36R-68rUHG1iBwx__qAWrWCNlXPZX_dtk-8va7XvA/s1600/Billede+29-07-14+23.55.28.jpg" height="298" width="400" /></a></div>
<br />Elfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com2tag:blogger.com,1999:blog-7395172357368553438.post-44512086687646471322014-03-27T07:39:00.001+01:002014-03-27T07:42:46.258+01:00Modelling a Shipment example as a hypermedia service with MasonYesterday I was attending the "RAML" workshop at the <a href="http://www.apistrategyconference.com/2014Amsterdam/index.php">API Strategy konference</a>. In this workshop the speakers introduced a very simple little Web API; It allowed a customer to GET a (shipment) quote and afterwards create (POST) an actual shipment request based on the quote.<br />
<br />
I decided that it could be fun to hypermedia-ize the example and show how it could be represented using the <a href="https://github.com/JornWildt/Mason">media type Mason</a>. So here we go :-)<br />
<br />
The first step is to ask for a quote; the customer has a package of some sort and needs to ship it from A to B, so he access the shipment service and asks for a quote (a price) given the size, weight, origin and destination for the package.<br />
<br />
In the original example you could ask for a quote by issuing a GET request to /quote. But I believe that asking for a quote would result in a concrete quote being created and stored in the system as a separate resource to access later on, either by the customer or by the customer service of the shipment company. So I would rather go for a POST of a quote request followed by a redirect to the newly created quote.<br />
<br />
At this point we could either document how to POST such a quote - or we could tell the client how to do it using hypermedia controls - and obviously I would go for the later. So lets ask the service for instructions and issue a GET /quote request. The output is a Mason document with suitable hypermedia controls embedded in it:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">{<br /> "@namespaces":<br /> {<br /> "myth":<br /> {<br /> "name": "http://mythological-shipment.com/api/rel-types"<br /> }<br /> },<br /> "@actions":<br /> {<br /> "myth:quote":<br /> {<br /> "type": "json",<br /> "method": "POST",<br /> "href": "http://mythological-shipment.com/api/quote",<br /> "title": "Ask for a quote",<br /> "description": "Ask for a quote by posting package details. Weight is in kilograms, volume in cubic decimeters, origin and destination must be known identifiers for airports.",</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> "schemaUrl": "... URL to JSON schema describing the request ..."<br /> }<br /> }<br />}</span><br />
<br />
The client reads this action specification, encodes the package details in JSON and POST it to the URL of the "href" property. As a result the service creates a new quote resource and redirects the client to it:<br />
<br />
Request:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> POST http://mythological-shipment.com/api/quote HTTP/1.1</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> content-type: application/json</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<br />
<span style="font-family: "Courier New",Courier,monospace;"> {</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> "weight": 2.3,</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> "volume": 4,</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> "origin": "CPH",</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> "destination": "AMS"</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> }</span><br />
<br />
Response:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> 201 Created</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> Location: http://mythological-shipment.com/api/quotes/myqo-129-gyh</span><br />
<br />
Now the client can GET the newly created quote to get further instructions of how to accept the quote. The result is again a Mason representation of the quote itself plus hypermedia controls for accepting the quote:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">{<br /> "id": "myqo-129-gyh",<br /> "weight": 2.3,<br /> "volume: 4,<br /> "origin": "CPH",<br /> "destination": "AMS":<br /> "price": 12,<br /> "@links":<br /> {<br /> "self":<br /> {<br /> "href": "http://mythological-shipment.com/api/quotes/myqo-129-gyh"<br /> }<br /> },<br /> "@actions":<br /> {<br /> "myth:accept-quote":<br /> {<br /> "type": "POST",<br /> "href": "http://mythological-shipment.com/api/quotes/myqo-129-gyh/state",<br /> "template":<br /> {<br /> "accepted": "yes"<br /> }<br /> }<br /> }<br />}</span><br />
<br />
As you can see the quote has a "self" link identifying the location of the quote resource. It also have a "accept-quote" action that instructs the client about how to accept the quote. In this case all the client has to do is to POST a predefined JSON value to http://mythological-shipment.com/api/quotes/myqo-129-gyh/state.<br />
<br />
The result of accepting a quote is that it is converted to a sales order (in lack of better domain understanding - there's probably a better word for it). So the accept-quote operation results in a redirect to the newly create sales order which the client can GET:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">{<br /> "id": "myqo-129-gyh",<br /> "weight": 2.3,<br /> "volume: 4,<br /> "origin": "CPH",<br /> "destination": "AMS":<br /> "price": 12,<br /> "@links":<br /> {<br /> "self":<br /> {<br /> "href": "http://mythological-shipment.com/api/orders/myqo-129-gyh"<br /> },<br /> "myth:quote":<br /> {<br /> "href": "http://mythological-shipment.com/api/quotes/myqo-129-gyh"<br /> },<br /> "myth:shipment-label":<br /> {<br /> "href": "http://mythological-shipment.com/api/quotes/myqo-129-gyh/label",<br /> "type": "application/pdf"<br /> }<br /> }<br />}</span><br />
<br />
At last the customer needs a shipment label to print out and stick onto the package. All it has to do is to follow the "myth:shipment-label" link and GET the PDF. Thats it.<br />
<br />
/JørnElfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com0tag:blogger.com,1999:blog-7395172357368553438.post-10936164944477968632014-02-26T07:53:00.002+01:002014-10-29T06:45:22.263+01:00API authentication considerations and best practicesI have been answering a few security questions on <a href="http://stackoverflow.com/questions/21824890/moving-from-session-based-token-mechanism-to-oauth-2-0-mechanism/21826248#21826248">Stackoverflow</a> and going through some APIs on <a href="http://www.programmableweb.com/">programmableweb.com</a> - and it keeps amazing me how often people gets HTTP authorization wrong.<br />
<br />
The typical questions I have noticed on SO are something like "I want to secure my API, have control of clients and require users to login - but I don't want to use OAuth". Duh? Why not? Perhaps because OAuth is conceived as being too difficult to work with? Well, OAuth2 is definitively not difficult (OAuth1 is another story though).<br />
<br />
But lets take a look at some of the existing practices:<br />
<br />
<b>API keys in URLs</b>: one example is the <a href="http://kokoelmat.fng.fi/api/v2support/docs/#/documentation">Finish National Gallery API</a> (which I somehow stumbled upon). Here you are required to obtain an API key and put it into the URL in every request made to the API:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> http://kokoelmat.fng.fi/api/v2?apikey=********&q=A+III+2172</span><br />
<br />
<br />
<b>API keys in custom headers</b>: one example is from <a href="http://www.docusign.com/developer-center/quick-start/first-api-call">DocuSign</a>. They require you to encode both your (developer) user name, password and API key in a JSON object and then send that in a custom header <span style="font-family: "Courier New",Courier,monospace;">X-DocuSign-Authentication</span>:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> GET /some-url<br /> X-DocuSign-Authentication: { "Username": "...", "Password": "...", "IntegratorKey": "..." }<br /> ... more ...</span><br />
<br />
<b>Signed URL/body parameters</b>: one example is <a href="http://www.last.fm/api/webauth">last.fm</a> which requires you to use your API key to obtain a session token and use that token later on to sign requests, either using URL parameters or in the body of the HTTP request.<br />
<br />
<b>HTTP Basic authentication</b>: <a href="http://developer.github.com/v3/#authentication">GitHub</a> supports authentication via the standard HTTP basic authentication mechanism where you supply username and password BASE64 encoded in the "Authorization" header.<br />
<br />
<b>OAuth1 and Oauth2</b>: <a href="http://pic.pbsrc.com/dev_help/WebHelpPublic/PhotobucketPublicHelp.htm">PhotoBucket</a> uses OAuth1 and <a href="https://github.com/basecamp/api/blob/master/sections/authentication.md">BaseCamp</a> uses Oauth2 (in addition to HTTP basic authentication).<br />
<br />
I did set out with the expectation of finding many more strange authorization schemes, but it turned out that at least most of the big players use OAuth1 or OAuth2. That's great! Now we just need to get the word out to all the "dark matter developers" out there (as <a href="http://www.hanselman.com/blog/DarkMatterDevelopersTheUnseen99.aspx">Scott Hanselman</a> calls them).<br />
<br />
<h3>
Things to be aware of</h3>
<br />
1) API keys in URLs are very easily accidentally exposed to third parties. If you take a copy of the URL and mail it to somebody you will end up sending your private key to someone else. For the same reason API keys may end up in log files here and there. Not good!<br />
<br />
2) API keys in URLs changes your resource URL such that it depends on who is using it. Instead of everybody referring to /some-items/1234 it will be /some-items/1234?key=abc for one user and /some-items/1234?key=qwe for another - and these two are completely different URLs even though they "only" differ on the query parameters. It is semantically the same as encoding the key in the path segment as for instance /api/abc/some-items/1234 and /api/qwe/some-items/1234 which I don't expect any one to think of as a good idea - right?<br />
<br />
3) API keys in headers are better than API keys in URLs since it will keep the URL stable for all users and it doesn't expose API keys when sending links to others. The problem though is that you are still sending your secret credentials (the API key) over the wire and it may be wiretapped by third parties. Another problem is that a client may accidentally connect to the wrong server (by misconfiguration or otherwise) and expose its credentials there.<br />
<br />
4) Signed requests (done right!) should be preferred over sending the API key directly. The signature should not be part of the URL for the reasons stated above - and the same goes somehow for sending the signature in the body since different users will see different bodies which is not necessary when you have a standard HTTP header for the purpose.<br />
<br />
5) Proprietary methods for signing requests are prone to design errors as it is easy to get the signature mechanism wrong - and thus accidentally making the signature technique easy to circumvent.<br />
<br />
6) Proprietary methods for signing requests requires client developer to understand yet another signature mechanism. It also makes it less likely to find existing client libraries to handle the crypto stuff. Both of these issues makes client adoption of your API less likely. And you know what, dear server developer? Your client developers will call for support and it will end up on YOUR table, reducing your ability to focus on coding the next great thing! So stick to well documented standards - it will be less annoying for everybody including yourself.<br />
<br />
7) HTTP basic authentication is great for debugging and getting started scenarios but should not be used in production. The problem is that client credentials are send in clear text and are thus susceptible to accidental exposure as mentioned before. Your API should support basic authentication as it will make it possible to explore the API using a standard web browser - but it should be disabled in production.<br />
<br />
8) It should be possible to revoke API keys in case they are compromised in some way.<br />
<br />
What kind of technique would solve all of these issues? Well, Oauth2 is one standard solution; it uses the HTTP "Authorization" header (thus avoiding authentication stuff in URL and body), it is a standard and used correctly it protects clients from exposing their API keys to the server and over the wire.<br />
<br />
Another solution is to use SSL/TLS with client certificates. With the proper HTTP client libraries this can be as easy as loading a certificate (one line of code) and assigning it to the HTTP request object (second line of code). This can although not authorize a combination of both client credentials and end user credentials.<br />
<br />
<h3>
OAuth2</h3>
<br />
<h4>
OAuth2 roles and terms</h4>
<br />
Before jumping into OAuth2 I
better explain some of the terms used when talking about OAuth2 (mostly
copied from the RFC at <a href="http://tools.ietf.org/html/rfc6749#section-1.1">http://tools.ietf.org/html/rfc6749#section-1.1</a>):<br />
<br />
- <b>Protected resources</b>: the data you want to protect.<br />
<br />
- <b>Resource owner</b>: An entity capable of granting access to a protected resource. The "entity" is often a person - the end user.<br />
<br />
- <b>Resource server</b>: The server hosting the protected resources.<br />
<br />
- <b>Client</b>:
An application making protected resource requests on behalf of the
resource owner. This can for instance be a mobile application, a website
or a background integration process impersonating an existing end user.<br />
<br />
- <b>Client credentials</b>: a pair of client ID and client secret. This could be your developer or application ID and API key.<br />
<br />
- <b>Resource owner password credentials</b>: the typical user-name/password combination issued to an end user.<br />
<br />
<h4>
</h4>
<h4>
OAuth2 flows</h4>
<br />
OAuth2 has four different modes of operating (called flows) - going from the relatively complex 3-legged authorization flow to the very simple "client credentials" authorization flow. At least one of the flows should fit into just about any public API ecosystem out there - and if not then it is possible to add new extension flows (as for instance Google does with the JWS implementation).<br />
<br />
At its core OAuth2 has only two high level steps:<br />
<br />
1) Swap a set of user and/or client credentials for an access token (authorization step), and <br />
<br />
2) Use the access token to access the API resources.<br />
<br />
That's it. It <i>can</i> be very simple. The difficult part of OAuth2 is the many ways a client can obtain an access token.<br />
<br />
Here I will focus on a scenario with a trusted client that accepts user credentials and acts on behalf of them. This can be very useful for many internal integration patterns where no humans are involved. I won't cover the scenario where an untrusted client (third party website) needs to access the end user's resources using the 3-legged authorization flow.<br />
<br />
<h4>
Acces tokens and bearer tokens</h4>
<br />
At a suitable high level OAuth2 only has two steps as mentioned before: 1) authorization, 2) resource access. The output of a successful authorization step is always an access token which must be used in subsequent requests to the resource server (the API).<br />
<br />
In most cases the access token is a so called "bearer token" - a token which the bearer can present to gain access to resources. Such a token has no built-in semantics and should be considered as nothing but a plain text string.<br />
<br />
The bearer token is included in the HTTP Authorization header like this:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> GET /resource HTTP/1.1<br /> Host: server.example.com<br /> Authorization: Bearer SOME-TOKEN</span><br />
<br />
<h4>
Authorizing with client credentials only</h4>
<br />
This is just about the simplest flow possible (see <a href="http://tools.ietf.org/html/rfc6749#section-4.4">http://tools.ietf.org/html/rfc6749#section-4.4</a>): all the client has to do is to send it's client credentials (ID and secret) as if it was using HTTP basic authorization. In return the client receives an access token it can use in subsequent requests for the protected resources. Here is an example from the RFC:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> POST /token HTTP/1.1<br /> Host: server.example.com<br /> Authorization: Basic czZCaGRSa3F0MzpnWDFmQmF0M2JW<br /> Content-Type: application/x-www-form-urlencoded<br /><br /> grant_type=client_credentials</span><br />
<br />
The response could be:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> HTTP/1.1 200 OK<br /> Content-Type: application/json;charset=UTF-8<br /> Cache-Control: no-store<br /> Pragma: no-cache<br /><br /> {<br /> "access_token":"2YotnFZFEjr1zCsicMWpAA",<br /> "token_type":"bearer"<br /> }</span><br />
<br />
This flow allows the client to act on behalf of itself but it does not include any end user information.<br />
<br />
<br />
<h4>
Authorizing with both client credentials (API key) and user credentials (password)</h4>
<br />
If a client needs to act on behalf of the end user (the resource owner) then it can use the "<a href="http://tools.ietf.org/html/rfc6749#section-4.3">Resource Owner Password Credentials Grant</a>". This flow is almost as simple is the previous flow - all the client has to do is to add the client credentials to the authorization request. Here is an example from the RFC:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> POST /token HTTP/1.1<br /> Host: server.example.com<br /> Authorization: Basic czZCaGRSa3F0MzpnWDFmQmF0M2JW<br /> Content-Type: application/x-www-form-urlencoded<br /><br /> grant_type=password&username=johndoe&password=A3ddj3w</span><br />
<br />
The response could be:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> HTTP/1.1 200 OK<br /> Content-Type: application/json;charset=UTF-8<br /> Cache-Control: no-store<br /> Pragma: no-cache<br /><br /> {<br /> "access_token":"2YotnFZFEjr1zCsicMWpAA",<br /> "token_type":"bearer"<br /> }</span><br />
<br />
This flow allows the client to act on behalf of the end user. The client application must be trusted as the end user will have to pass his/her credentials to it.<br />
<br />
<br />
<h4>
Protecting credentials in transit</h4>
<br />
There is one big problem with the two simple flows mentioned above; both of them pass the client and end user credentials in clear text. For this reason OAuth2 makes it mandatory to use TLS/SSL to encrypt <i>all</i> of the requests. Neither does these flows cover scenarios with untrusted clients (such as a third party website that wants to access an end users resources on a different server) - but that scenario is covered in the "<a href="http://tools.ietf.org/html/rfc6749#section-4.1">Authorization Code Grant</a>" scenario which I will not discuss here.<br />
<br />
The use of TLS/SSL protects the credentials from eavesdropping and man-in-the-middle attacks but it does not protect against sending credentials to the wrong server - either by misconfiguration or because the server has been compromised. <br />
<br />
This lack of protection is perhaps one of the biggest difference between OAuth1 and OAuth2; OAuth1 protects credentials by signing requests instead of sending the raw credentials. This requires a rather complex signing algorithm which has caused many headaches over the time. OAuth2 trades the protection of credentials for simplicity which makes it a lot easier to use - but susceptible to leaking credentials to rogue servers.<br />
<br />
It <i>is</i> possible to protect credentials from this problem though but it requires clients to use some complex crypto stuff to sign requests instead of sending the raw credentials (just like OAuth1 does). This is what <a href="https://developers.google.com/accounts/docs/OAuth2ServiceAccount">Google</a> does when it requires requests to be signed using <a href="http://tools.ietf.org/html/draft-ietf-jose-json-web-signature-21">JSON Web Signatures</a> (JWS).<br />
<br />
If you are using .NET then my Ramone HTTP library has support for OAuth2 with JWS as can be seen in this example: <a href="http://soabits.blogspot.com/2013/03/using-ramone-for-oauth2-authorization.html">http://soabits.blogspot.com/2013/03/using-ramone-for-oauth2-authorization.html</a>. <br />
<br />
<br />
<h4>
Protecting credentials in clients</h4>
<br />
One more word of caution - websites, mobile apps, desktop apps and similar with public clients <i>cannot</i> protect their client credentials. Other application may be able to do it (server to server integrations for instance) but public clients, be it JavaScript, byte code or assembler, can always be downloaded, decompiled and scrutinized by various forensic tools and in the end it will be impossible to keep client secrets from being stolen. This is a fact we have to live with.<br />
<br />
Client credentials can be useful for statistics. They also give the ability to revoke credentials in order to block rogue applications - but they cannot be trusted in general.<br />
<br />
<h4>
Further reading</h4>
Eran Hammer has a discussion about the drawbacks of OAuth2 at <a href="http://hueniverse.com/2012/07/26/oauth-2-0-and-the-road-to-hell/">http://hueniverse.com/2012/07/26/oauth-2-0-and-the-road-to-hell/</a> and here is another discussion of the drawbacks of bearer tokens <a href="http://hueniverse.com/2010/09/29/oauth-bearer-tokens-are-a-terrible-idea/">http://hueniverse.com/2010/09/29/oauth-bearer-tokens-are-a-terrible-idea/</a> <br />
<br />Elfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com2tag:blogger.com,1999:blog-7395172357368553438.post-2859479525132889742014-02-20T14:58:00.001+01:002014-02-20T14:58:22.925+01:00Representing an issue tracker with MasonTwo weeks ago I introduced <a href="https://github.com/JornWildt/Mason">Mason</a> - a media type for representing data with embedded hypermedia elements. Now I would like to go through an example implementation of a fictive issue tracker which uses Mason to represent its data.<br />
<br />
At its core the issue tracker has, not surprisingly, "issues" that represent issues that needs to be solved. Issues are organized in projects and may have one or more file attachments associated with them. Issues are represented by their title, a description and a severity level (from 1 to 5).<br />
<br />
It would be natural to include comments on issues too but I want to keep the domain as simple as possible while still being able to illustrate all the features of Mason - and comments would not add anything but clutter as they are very similar to attachments.<br />
<br />
Please note that the issue tracker will be defined without reference to any existing implementation thus making it a "real" REST service totally independent of any specific implementation. I do have a reference implementation available but that is only used as a proof of concept - it is not part of the issue tracker definition itself.<br />
<br />
The issue tracker is defined in terms of data types, links, link templates and actions as described in the next sections. That is not exactly interesting reading so you might want to skip it and jump to the later sections with examples of how to actually use the issue tracker.<br />
<br />
You should try the demo issue tracker yourself with the generic <a href="https://github.com/JornWildt/Mason/wiki/Generic-Mason-browser">Mason browser</a> in order to see how actions and URL templates are supposed to work.<br />
<h2>
Data types, links and actions</h2>
<h3>
CURIE definitions</h3>
In the following the CURIE name "is" should be expanded to "http://soabits.dk/mason/issue-tracker/reltypes.html#". So for instance "is:add-issue" becomes "<a href="http://soabits.dk/mason/issue-tracker/reltypes.html#add-issue">http://soabits.dk/mason/issue-tracker/reltypes.html#add-issue</a>" (you can GET that).<br />
<br />
These URL relationship types ensures that we have no name collisions with other relationship types. In addition to this it is actually possible to dereference the URLs and GET some documentation about them.<br />
<br />
<h3>
Data types</h3>
<br />
<h4>
Issue</h4>
A single issue consists of the following properties:<br />
<br />
Id: <i>int</i><br />
Title: <i>string</i><br />
Description: <i>string</i><br />
Severity: <i>integer</i><br />
<br />
Expected links:<br />
<ul>
<li>self</li>
<li>up: link to parent project</li>
<li>is:attachments: link to collection of attachments for issue</li>
<li>is:common: link to common resource data</li>
</ul>
<br />
Example resource: <a href="http://mason-issue-tracker.cbrain.net/issues/1">http://mason-issue-tracker.cbrain.net/issues/1</a> <br />
<br />
<h4>
Issue collection</h4>
A collection of issues has one top level property:<br />
<br />
Issues: <i>array of</i><br />
- Id: <i>int</i><br />
- Title: <i>string</i><br />
<br />
Expected links:<br />
<ul>
<li>self</li>
<li>up: link to parent project</li>
<li>is:common: link to common resource data</li>
</ul>
Example resource: <a href="http://mason-issue-tracker.cbrain.net/projects/1/issues">http://mason-issue-tracker.cbrain.net/projects/1/issues</a> <br />
<br />
<h4>
Project</h4>
A single project consists of the following properties:<br />
<br />
Id: <i>int</i><br />
Code: <i>string</i><br />
Title: <i>string</i><br />
Description: <i>string</i><br />
<br />
Expected links:<br />
<ul>
<li>self</li>
<li>is:issues: link to collection of issues for project</li>
<li>is:common: link to common resource data</li>
</ul>
<br />
Expected actions:<br />
<ul>
<li>is:project-update</li>
<li>is:add-issue</li>
<li>is:project-delete</li>
</ul>
Example resource: <a href="http://mason-issue-tracker.cbrain.net/projects/1">http://mason-issue-tracker.cbrain.net/projects/1</a> <br />
<br />
<h4>
Project collection</h4>
A collection of projects has one top level property:<br />
<br />
Projects: <i>array of </i><br />
- Id: <i>int</i><br />
- Code: <i>string </i>(a short code or abbreviation of the project name)<br />
- Title: <i>string</i><br />
<br />
Expected links:<br />
<ul>
<li>self</li>
<li>is:common: link to common resource data</li>
</ul>
<br />
Example resource: <a href="http://mason-issue-tracker.cbrain.net/projects">http://mason-issue-tracker.cbrain.net/projects</a><br />
<br />
<h4>
Common resource data</h4>
A set of properties and links which are common to all resources:<br />
<br />
Title: <i>string</i> (title of the whole issue tracker)<br />
Description: <i>string</i> (description of the whole issue tracker)<br />
<br />
Expected links:<br />
<ul>
<li>self</li>
<li>is:contact: link to contact information</li>
<li>is:logo: link to issue tracker logo</li>
<li>is:projects: link to collection of all projects</li>
<li>is:common: link to common resource data</li>
</ul>
<br />
Expected link templates:<br />
<ul>
<li>is:issue-query</li>
</ul>
<br />
Expected actions:<br />
<ul>
<li>is:project-create</li>
</ul>
<br />
Example resource: <a href="http://mason-issue-tracker.cbrain.net/resource-common">http://mason-issue-tracker.cbrain.net/resource-common</a> <br />
<br />
<h4>
Contact information</h4>
Contact information (related to the owner of the issue tracker) consists of:<br />
<br />
Name: <i>string</i><br />
Address1: <i>string</i><br />
Address2: <i>string</i><br />
PostalCode: <i>string</i><br />
City: <i>string</i><br />
EMail: <i>string</i><br />
Phone: <i>string</i><br />
Country: <i>string</i><br />
<br />
Expected links:<br />
<ul>
<li>self</li>
<li>alternate (alternate address representations in other formats like for instance text/vcard)</li>
<li>is:common</li>
</ul>
Example resource: <a href="http://mason-issue-tracker.cbrain.net/contact">http://mason-issue-tracker.cbrain.net/contact</a> (take a close look at the "alternate" link as it shows how to have multiple targets for one link).<br />
<br />
<h3>
Link relations</h3>
<br />
<h4>
is:projects</h4>
Link to collection of all projects.<br />
<br />
<h4>
is:issues</h4>
Link to collection of issues for a given project.<br />
<br />
<h4>
is:attachments</h4>
Link to collection of attachments for a given issue.<br />
<br />
<h4>
is:contact</h4>
Link to contact information.<br />
<br />
<h4>
is:logo</h4>
Link to issue tracker logo.<br />
<br />
<h4>
is:common</h4>
Link to data common for all resources in the issue tracker.<br />
<br />
<h3>
Link templates</h3>
<h3>
</h3>
<h4>
is:issue-query</h4>
A link template for querying issues. Parameters:<br />
<br />
text: any text to look for in issues.<br />
severity: severity level (1-5)<br />
pid: project ID.<br />
<br />
Example usage: <a href="http://mason-issue-tracker.cbrain.net/resource-common">http://mason-issue-tracker.cbrain.net/resource-common</a> <br />
<br />
<h3>
Actions</h3>
<br />
<h4>
is:project-create</h4>
Action for creating a new project. Arguments:<br />
<br />
Code: <i>string</i><br />
Title: <i>string</i><br />
Description: <i>string</i><br />
<br />
Example usage: <a href="http://mason-issue-tracker.cbrain.net/resource-common">http://mason-issue-tracker.cbrain.net/resource-common</a> <br />
<br />
<h4>
is:project-update</h4>
Action for updating a single project. Arguments:<br />
<br />
Code: <i>string</i><br />
Title: <i>string</i><br />
Description: <i>string</i><br />
<br />
Example usage: <a href="http://mason-issue-tracker.cbrain.net/projects/1">http://mason-issue-tracker.cbrain.net/projects/1</a> <br />
<br />
<h4>
is:project-delete</h4>
Action for deleting a project. Has no arguments.<br />
<br />
Example usage: <a href="http://mason-issue-tracker.cbrain.net/projects/1">http://mason-issue-tracker.cbrain.net/projects/1</a> <br />
<br />
<h4>
is:add-issue</h4>
Action for adding a new issue to a project. Arguments:<br />
<br />
Title: <i>string</i><br />
Description: <i>string</i><br />
Severity: <i>int</i><br />
Attachment: <i>object of </i><br />
- Title: <i>string</i><br />
- Description: <i>string</i><br />
<br />
In addition to this it is possible to pass a file as an attachment to the issue. The file name is "attachment".<br />
<br />
The "Attachment" object contains additional information about the attached file.<br />
<br />
Example usage: <a href="http://mason-issue-tracker.cbrain.net/projects/1">http://mason-issue-tracker.cbrain.net/projects/1</a> <br />
<br />
<h4>
is:issue-update</h4>
Action for updating a single issue. Arguments:<br />
<br />
Title: <i>string</i><br />
Description: <i>string</i><br />
Severity: <i>int</i><br />
<br />
Example usage: <a href="http://mason-issue-tracker.cbrain.net/issues/1">http://mason-issue-tracker.cbrain.net/issues/1</a> <br />
<br />
<h4>
is:issue-delete</h4>
Action for deleting a single issue. Has no arguments.<br />
<br />
Example usage: <a href="http://mason-issue-tracker.cbrain.net/issues/1">http://mason-issue-tracker.cbrain.net/issues/1</a> <br />
<br />
<h4>
is:add-attachment</h4>
Action for adding an attachment to an issue. Arguments:<br />
<br />
Title: <i>string</i><br />
Description: <i>string</i><br />
<br />
In addition to this it is possible to pass a file as the actual attachment. The file name is "attachment".<br />
<br />
Example usage: <a href="http://mason-issue-tracker.cbrain.net/issues/1">http://mason-issue-tracker.cbrain.net/issues/1</a> <br />
<br />
<h2>
Examples</h2>
<br />
<h3>
Getting started</h3>
The first thing a client must do in order to work with the issue tracker is to GET the "common" resource that contains useful links, templates and actions for the issue tracker. The common resource can be thought of as the "home page" or "landing page" of the issue tracker.<br />
<br />
Try it yourself: GET <a href="http://mason-issue-tracker.cbrain.net/resource-common">http://mason-issue-tracker.cbrain.net/resource-common</a> <br />
<br />
<h3>
Creating a new project</h3>
Once we have a copy of the "common" resource we can look for the action "is:project-create". That action will tell us how to encode the project data and how to submit it to the server.<br />
<br />
Here is an example:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">"@actions": {<br /> "is:project-create": {<br /> "type": "json",<br /> "href": "http://mason-issue-tracker.cbrain.net/projects",<br /> "title": "Create new project",<br /> "schemaUrl": "http://mason-issue-tracker.cbrain.net/schemas/create-project"<br /> }<br />}</span><br />
<br />
The most important parts are the "type" and "href" properties which tells us how to encode the data and where to send it. Mason actions also have a "method" property for identifying the HTTP method to use but it defaults to POST so its not always needed.<br />
<br />
The type "json" tells us to encode the action arguments in plain JSON. The net result is a request like this:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">POST /projects HTTP/1.1<br />Accept: application/vnd.mason+json<br />Content-Type: application/json<br /><br />{<br /> "Code": "SHOP",<br /> "Title": "Webshop",<br /> "Description": "Project for issues related to the webshop"<br />}</span><br />
<br />
The response is a redirect to the created project:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">HTTP/1.1 201 Created<br />Location: http://mason-issue-tracker.cbrain.net/projects/2</span><br />
<br />
<h3>
Adding a new issue</h3>
Now that we have a project we can start adding issues to it (with optional attachments). Each project representation contains an "is:add-issue" action for this purpose as can be seen here:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">"@actions": {<br /> "is:add-issue": {<br /> "type": "json+files",<br /> "href": "http://mason-issue-tracker.cbrain.net/projects/2/issues",<br /> "title": "Add new issue to project",<br /> "schemaUrl": "http://mason-issue-tracker.cbrain.net/schemas/create-issue",<br /> "jsonFile": "args",<br /> "files": [<br /> {<br /> "name": "attachment",<br /> "description": "Attachment for issue"<br /> }<br /> ]<br /> }<br />}</span><br />
<br />
This action tells us the following:<br />
<ul>
<li>The type is "json+files" which means we must send the JSON data together with some files wrapped in the media type multipart/form-data.</li>
<li>The target URL is "http://mason-issue-tracker.cbrain.net/projects/2/issues".</li>
<li>The JSON data must conform to the schema definition at "http://mason-issue-tracker.cbrain.net/schemas/create-issue".</li>
<li>The JSON data must be contained in a multipart element named "args".</li>
<li>The attached file must be contained in a multipart element named "attachment".</li>
</ul>
The net result is a request like this:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">POST /projects/2/issues HTTP/1.1<br />Accept: application/vnd.mason+json<br />Content-Type: multipart/form-data; boundary=d636dfda-b79f-4f29-aaf6-4b6687baebeb<br /><br />--d636dfda-b79f-4f29-aaf6-4b6687baebeb<br />Content-Disposition: form-data; name="attachment"; filename="hogweed.jpg"<br />... binary data for attached image ...<br /><br />--d636dfda-b79f-4f29-aaf6-4b6687baebeb<br />Content-Disposition: form-data; name="args"; filename="args"<br />Content-Type: application/json<br /><br />{<br /> "Title": "Hogweeds on the plaza",<br /> "Description": "Could you please remove the hogweeds growing at the plaza?",<br /> "Severity": 5,<br /> "Attachment":<br /> {<br /> "Title": "Hogweed",<br /> "Description": "Photo of the hogweeds."<br /> }<br />}</span><br />
<br />
<h3>
Updating project details</h3>
Each project has a "is:project-update" action for updating the project details:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">"@actions": {<br /> "is:project-update": {<br /> "type": "json",<br /> "href": "http://mason-issue-tracker.cbrain.net/projects/1",<br /> "title": "Update project details",<br /> "template": {<br /> "Code": "SHOP",<br /> "Title": "Webshop",<br /> "Description": "All issues related to the webshop."<br /> }<br /> }<br />}</span><br />
<br />
This action tells us to encode project arguments in JSON and POST it to "http://mason-issue-tracker.cbrain.net/projects/1". The JSON data should be build from the JSON template in the action.<br />
<br />
Request:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">POST /projects/1 HTTP/1.1<br />User-Agent: API Explorer<br />Accept: application/vnd.mason+json<br />Content-Type: application/json<br /><br />{<br /> "Code": "SHOP",<br /> "Title": "Web shop",<br /> "Description": "All issues related to the new web shop."<br />}</span><br />
<br />
<h3>
Deleting a project</h3>
Each project has a "is:project-delete" action for deleting the project and its related issues:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">"@actions": {<br /> "is:project-delete": {<br /> "type": "void",<br /> "href": "http://mason-issue-tracker.cbrain.net/projects/1",<br /> "method": "DELETE",<br /> "title": "Delete project"<br />}</span><br />
<br />
Request:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">DELETE /projects/1 HTTP/1.1<br />Accept: application/vnd.mason+json</span><br />
<br />
Response:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">HTTP/1.1 204 No Content</span><br />
<br />
<h3>
Searching for issues</h3>
The "common" resource has a link template for issue queries:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">"@link-templates": {<br /> "is:issue-query": {<br /> "template": "http://mason-issue-tracker.cbrain.net//issues-query?text={text}&severity={severity}&project={pid}",<br /> "title": "Search for issues",<br /> "description": "This is a simple search that do not check attachments.",<br /> "parameters": [<br /> {<br /> "name": "text",<br /> "description": "Substring search for text in title and description"<br /> },<br /> {<br /> "name": "severity",<br /> "description": "Issue severity (exact value, 1..5)"<br /> },<br /> {<br /> "name": "pid",<br /> "description": "Project ID"<br /> }<br /> ]<br /> }<br />}</span><br />
<br />
This templates tells us to replace the parameters "text", "severity" and "pid" into the URL template "http://mason-issue-tracker.cbrain.net//issues-query?text={text}&severity={severity}&project={pid}".<br />
<br />
Should we for instance want to query for issues of severity 5 in project 1 then we would get this request:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">GET /issues-query?text=&severity=5&project=1 HTTP/1.1<br />Accept: application/vnd.mason+json</span><br />
<br />
The result is a collection of issues.<br />
<br />
<br />Elfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com0tag:blogger.com,1999:blog-7395172357368553438.post-81967172137147873042014-02-06T22:25:00.000+01:002014-02-07T09:25:25.965+01:00Implementing hypermedia APIs and REST services with MasonI am happy to announce that I have taken all the lessons learned during the last few years and stuffed it into a new JSON based mediatype for hypermedia APIs and REST services. The media type is <span style="font-family: Courier New, Courier, monospace;">application/vnd.mason+json</span> or simply "Mason". There is an IANA registration for it pending.<br />
<br />
With Mason you get hypermedia elements for linking and modifying data, features for communicating to client developers and standardized error handling. Mason is built on JSON, reads JSON, writes JSON and generally fits well into a JSON based eco-system.<br />
<br />
Here is a simple example illustrating how a single issue from a fictive issue tracker could be represented in Mason. It contains the basic API data like issue Title, Description and Severity and then it adds hypermedia elements for linking to other related resources and actions for writing stuff back to the issue tracker.<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">{</span><br />
<span style="font-family: Courier New, Courier, monospace;"> // Classic API data</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "ID": 1,</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "Title": "Program crashes when pressing ctrl-p",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "Description": "I pressed ctrl-p and, boom, it crashed.",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "Severity": 5,</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "Attachments": [</span><br />
<span style="font-family: Courier New, Courier, monospace;"> {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "Id": 1,</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "Title": "Error report",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> // Hypermedia linking to attachment</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "@links": {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "self": {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "href": "http://issue-tracker.org/attachments/1"</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> ],</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> // Additional hypermedia links</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "@links": {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> // Hypermedia linking to self</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "self": {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "href": "http://issue-tracker.org/issues/1"</span><br />
<span style="font-family: Courier New, Courier, monospace;"> },</span><br />
<span style="font-family: Courier New, Courier, monospace;"> // Hypermedia linking to containing project</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "up": {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "href": "http://issue-tracker.org/projects/1",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "title": "Containing project"</span><br />
<span style="font-family: Courier New, Courier, monospace;"> },</span><br />
<span style="font-family: Courier New, Courier, monospace;"> },</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> // Hypermedia "action" element for creating a new project</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "@actions": {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "is:project-create": {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "type": "json",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "href": "http://issue-tracker.org/mason-demo/projects",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "title": "Create new project",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "schemaUrl": "http://issue-tracker.org/mason-demo/schemas/create-project"</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<br />
Those that are familiar with <a href="http://tools.ietf.org/html/draft-kelly-json-hal-06">HAL</a> may recognize some parts of the format. That is expected as Mason builds on the ideas from HAL. HAL was never intended to have hypermedia elements for writing stuff so I decided to go for it and design a new format based on HAL.<br />
<br />
The Mason specification, online example and stand-alone API explorer are available from <a href="https://github.com/JornWildt/Mason">https://github.com/JornWildt/Mason</a>.<br />
<br />
<br />
<h3>
Design goals</h3>
<br />
My design goals with Mason are:<br />
<br />
1. It should be easy to adopt in existing JSON based solutions and have a low barrier of entry for new developers.<br />
<br />
2. It should contain hypermedia elements sufficient for both reading and writing data without any out-of-band information.<br />
<br />
3. It should contain elements for information directed to client developers for the purpose of improving "API developer experience".<br />
<br />
4. It should contain error elements sufficient for most kinds of applications.<br />
<br />
5. It should work with JSON when both reading and writing.<br />
<br />
Let me dig into each of those design goals one by one.<br />
<br />
<h4>
1. Easy to adopt</h4>
<br />
A classic JSON payload makes the raw API data directly accessible as JSON object properties. I believe it should be so too when working with hypermedia enabled APIs. So Mason merges hypermedia elements into existing JSON structures. To avoid name collisions Mason property names are prefixed with a '@'.<br />
<br />
Mason can be adopted gradually:<br />
<br />
Step 1: Change content type to <span style="font-family: Courier New, Courier, monospace;">application/vnd.mason+json</span> instead of <span style="font-family: Courier New, Courier, monospace;">application/json</span>.<br />
<br />
Step 2: Add a <span style="font-family: Courier New, Courier, monospace;">@meta</span> property with additional information targeted at client developers.<br />
<br />
Step 3: Use links to remove client knowledge of server defined URLs.<br />
<br />
Step 4: Use Mason's error format.<br />
<br />
Step 5: Use actions to truly decouple client and server implementations.<br />
<br />
<h4>
2. Hypermedia for both reading and writing</h4>
<br />
Hypermedia has a lot of benefits as I wrote in <a href="http://soabits.blogspot.dk/2013/12/selling-benefits-of-hypermedia.html">http://soabits.blogspot.dk/2013/12/selling-benefits-of-hypermedia.html</a>. Among these is the ability to remove a client's dependency on server URL structures using links.<br />
<br />
But links are only good for, well, linking resources together - they don't say anything about how to change and modify API data. So Mason adds "actions" for writing API data.<br />
<br />
An action defines both target URL, HTTP method and action type (payload encoding). With this information being discoverable at runtime it is no longer necessary to hard code clients with information about HTTP method and how to encode the payload. This means client and server only have to agree on WHICH data to send - not HOW to send it.<br />
<br />
<h4>
3. Information targeted at client developers</h4>
<br />
One of the great things about hypermedia enabled APIs is the ability to explore the API using a browser of some kind. As I wrote in <a href="http://soabits.blogspot.dk/2013/12/selling-benefits-of-hypermedia.html">http://soabits.blogspot.dk/2013/12/selling-benefits-of-hypermedia.html</a>; Do not underestimate the power of an explorable API. The ability to browse around the data makes it a lot easier for the client developers to build a mental model of the API and its data structures.<br />
<br />
And if client developers are browsing the API why not also be able to communicate directly with them? Mason adds a few meta data elements for sending messages directly to the client developers. An API browser should highlight these such that devs can instantly read some documentation and comments about the resource they are currently looking at.<br />
<br />
At the same time Mason defines a technique for removing this client developer information from the payload in production.<br />
<br />
<h4>
4. Error handling</h4>
<br />
By standardizing error handling Mason makes it possible for clients to interact with unknown services and still be able to communicate error conditions clearly to end users.<br />
<br />
I have previously discussed error handling here in <a href="http://soabits.blogspot.dk/2013/05/error-handling-considerations-and-best.html">http://soabits.blogspot.dk/2013/05/error-handling-considerations-and-best.html</a> and apparently that article hit a nerve somewhere because it keeps attracting a lot of attention (for an amateur blogger like me).<br />
<br />
<h4>
5. JSON read/write</h4>
<br />
One of things that annoys me about the traditional key/value forms based on application/x-www-form-urlencoded is that there are no standards for encoding complex data structures. Neither does it define any standard for encoding booleans, integers and other basic data types. The consequence is that client and server needs to agree on these things before they can start talking about business data - and different servers are surely going to implement different encoding schemes - all in all making life miserable for developers that just want to get stuff done.<br />
<br />
By using JSON Mason ensures interoperability on some of the lower levels. JSON defines more types than simple string based key/value formats and handles structures like objects and arrays.<br />
<br />
Restricting implementations of Mason to handle JSON only reduces design choices and variations and thus improving the chances of things working out of the box (compared to simple string based key/value formats).<br />
<br />
<h3>
Transcending from web APIs to REST services</h3>
<br />
Most web APIs today are defined in terms of a single server implementation around which developers build dedicated clients (think "Twitter" or "Facebook"). In such a world clients have a strong coupling to server URL structures, HTTP methods, error formats and other quirks of the API. These APIs were never designed to be implemented by more than one organization (and are for this reason also called "snowflake APIs").<br />
<br />
True REST services on the other hand are defined without reference to any specific server implementation. The best known example of this is the ATOM format which enables clients to interact with any ATOM enabled service on the web - no matter who implemented it, where it is hosted or what URL structures it is implemented with. The enabling factor for this is the ATOM media type specification.<br />
<br />
But ATOM is restricted to feed-like data and does not fit well with other applications. So other media types are needed and Mason is an attempt to fill out this space. Mason attempts to facilitate complete decoupling from technical implementation details such that clients can discover HOW to interact with service at runtime.<br />
<br />
One of my earlier blog posts discussed this problem in more detail: <a href="http://soabits.blogspot.no/2013/05/the-role-of-media-types-in-restful-web.html">http://soabits.blogspot.no/2013/05/the-role-of-media-types-in-restful-web.html</a><br />
<br />
<h3>
Data profiles</h3>
<br />
Mason itself does not prescribe any business specific details. Clients and servers still have to agree on WHAT data to interchange - but Mason do remove the technical coupling on HOW to interchange the data.<br />
<br />
Mason depends on profiles to enable clients to know WHAT data they are looking at. You can find an in-depth discussion about it here: <a href="http://soabits.blogspot.no/2013/12/media-types-for-apis.html">http://soabits.blogspot.no/2013/12/media-types-for-apis.html</a>.<br />
<br />
At the time of writing I haven't put profiles into the specification yet.<br />
<br />
<h3>
Further reading</h3>
Mason homepage: <a href="https://github.com/JornWildt/Mason">https://github.com/JornWildt/Mason</a><br />
<br />
Generic Mason browser (API explorer): <a href="https://github.com/JornWildt/Mason/wiki/Generic-Mason-browser">https://github.com/JornWildt/Mason/wiki/Generic-Mason-browser</a><br />
<br />
Online live example of fictive issue tracker using Mason: <a href="https://github.com/JornWildt/Mason/wiki/Example-service%3A-issue-tracker">https://github.com/JornWildt/Mason/wiki/Example-service%3A-issue-tracker</a><br />
<br />
/Jørn<br />
<div>
<br /></div>
Elfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com0tag:blogger.com,1999:blog-7395172357368553438.post-57112087137809343472013-12-08T08:36:00.001+01:002013-12-08T11:00:44.507+01:00Media types for APIsI have previously touched upon the concept of media types (see <a href="http://soabits.blogspot.no/2013/05/the-role-of-media-types-in-restful-web.html">http://soabits.blogspot.no/2013/05/the-role-of-media-types-in-restful-web.html</a>), but somehow it has always been difficult for me to really nail the concept down in a concise and useful article.<br />
<br />
Now the latest discussion about the benefits of hypermedia (see <a href="http://soabits.blogspot.no/2013/12/selling-benefits-of-hypermedia.html">http://soabits.blogspot.no/2013/12/selling-benefits-of-hypermedia.html</a>) got me thinking about media types again - but this time in the perspective of unique service implementations with dedicated clients versus large scale ecosystems of mixed implementations.<br />
<br />
As it turns out, media types doesn't mean sh*t on a small scale. That kind of explains why it has been so difficult to get to some kind of consensus about media types for APIs.<br />
<br />
<h3>
Background</h3>
<br />
When the discussion touches upon media types the arguments usually follow these lines:<br />
<br />
<ul>
<li>Completely generic media types like JSON and XML should be avoided since they do not include any kind of hypermedia elements.</li>
<li>One school of thought argues that we should have very few (generic) media types. This is to avoid the need for clients to understand too many media types.</li>
<li>Another school of thought argues that we should have many different domain specific media types. Otherwise the client wouldn't know what kind of resource it was looking at.</li>
</ul>
<br />
But, as I said, it really doesn't matter. Both schools are right. At least when you look at unique service implementations with dedicated clients - like for instance dedicated Twitter clients.<br />
<br />
Let me give you a concrete example from the Twitter API (see <a href="https://dev.twitter.com/discussions/5662">https://dev.twitter.com/discussions/5662</a>): the return value from their oauth/request_token "endpoint" is key/value pairs encoded as application/x-www-form-urlencoded - but the server says it is "text/html" which is clearly wrong. Does that break any client implementations? No. Why? Because all clients are dedicated to the Twitter API; they KNOW about this little peculiarity and has been hard coded to work with it.<br />
<br />
My point is:<br />
<blockquote class="tr_bq">
<i>Media types are irrelevant for unique service implementations with dedicated clients. In this world the client always knows exactly what it is doing and what kind of result to expect from the server (and it can safely ignore the media type).</i></blockquote>
<br />
<h3>
Media types on a large scale</h3>
<br />
Let us broaden our view and look at the example of "Big corporation buys smaller companies and the result is a big unruly combination of customers, sales orders and other stuff living on different systems" which I introduced in my previous blog post (<a href="http://soabits.blogspot.no/2013/12/selling-benefits-of-hypermedia.html">http://soabits.blogspot.no/2013/12/selling-benefits-of-hypermedia.html</a>).<br />
<br />
Now lets assume our fictive client is handed a link/URL to a customer resource in this mess of a heterogeneous mix of different company resources. The client can issue a GET on the URL and in return it will receive a stream of bytes. How does the client interpret those bytes? Obviously it will depend on the media type. But which kind of media type is useful for this purpose?<br />
<br />
Let us assume the client understand a generic (hypermedia enabled) media type like HAL. Together with the GET request the client sends an accept header "Accept: application/hal+json". Luckily the server knows how to serve the customer resource as HAL, so the client gets a HAL document in return.<br />
<br />
Now what? We have integrated customer resources from three different organizations and each of these have been encoding customer records in HAL - but in different ways.<br />
<br />
For instance: Company X has these customer properties:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">{</span><br />
<span style="font-family: Courier New, Courier, monospace;"> ID: 1234,</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Name: "John Larsson",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Address: "Marienborg 1, 2830 Virum, Denmark"</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<br />
while company Y uses these properties:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">{</span><br />
<span style="font-family: Courier New, Courier, monospace;"> ID: 1234,</span><br />
<span style="font-family: Courier New, Courier, monospace;"> FirstName: "John",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> LastName: "Larsson",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Address:</span><br />
<span style="font-family: Courier New, Courier, monospace;"> {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Address: "Marienborg 1",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> PostalCode: "2830",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> City: "Virum",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Country: "Denmark"</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<br />
With nothing but this information our client must either give up or do some guessing like "If <span style="font-family: Courier New, Courier, monospace;">FirstName</span> is present then assume format of company Y". So apparently we need a bit more information than we already have.<br />
<br />
Now we can either choose to add some kind of profile to the representation - either as a header or in the payload - or we can use a domain specific media type.<br />
<br />
1) A profile in the payload could be done like this:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">{</span><br />
<span style="font-family: Courier New, Courier, monospace;"> ID: 1234,</span><br />
<span style="font-family: Courier New, Courier, monospace;"> profile: "http://company-x.com/profiles/customer-care",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> ... other properties ...</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<br />
2) The profile could also be part of the media type, so we would get "application/hal+json;profile=http://company-x.com/profiles/customer-care".<br />
<br />
3) A domain specific media type could be something like "application/company-x.customer-care.hal+json" or similar.<br />
<br />
But which method should we choose? Lets take a look at how the client process the server response before we answer that.<br />
<br />
<h3>
Processing a server response</h3>
<br />
There are three things the client must know in order to process a server response correctly:<br />
<br />
<ol>
<li>How to decode the byte stream (generic knowledge).</li>
<li>What the data represents (domain specific knowledge).</li>
<li>How to locate hypermedia elements in the response (generic knowledge).</li>
</ol>
<br />
The media type is obviously the key to decoding the byte stream - it will tell the client whether it is looking at XML, PDF, HTML, HAL, Sirene and so on.<br />
<br />
The media type should also be the key to locating hypermedia elements in the response.<br />
<br />
But what about the domain specific knowledge - should we identify what a resource represents with a domain specific media type or with a profile? Both methods work, but there is one more thing to take into account: making the API explorable by client developers (see <a href="http://soabits.blogspot.no/2013/12/selling-benefits-of-hypermedia.html">http://soabits.blogspot.no/2013/12/selling-benefits-of-hypermedia.html</a>).<br />
<br />
It is of course possible to implement a browser for any domain specific media type we can think of, but it would obviously be more practical if we could have one single API browser for all kinds of APIs. For this reason we should avoid domain specific media types. The domain knowledge can then be identified by a profile - either in the payload or in a HTTP header.<br />
<br />
<h3>
Wrapping it all up</h3>
<br />
As with the hypermedia problem: if you stick to unique service implementations with dedicated clients (like a dedicated Twitter client) then media types are utterly irrelevant. The client can safely assume that there will be one, and only one, representation of what ever kind of resource it is looking for.<br />
<br />
But if you take broader perspective and venture into a highly heterogeneous, loosely coupled, unorganized, incoherent and fragmented ecology (also called "The internet") - then you need more domain specific information about the resources - either through domain specific media types, or generic media types with profiles.<br />
<br />
My recommendation is:<br />
<br />
<ol>
<li>Use generic media types that include hypermedia elements.</li>
<li>Identify domain specific information through profiles.</li>
</ol>
<br />
The media type will tell the client HOW to decode the byte stream and HOW to interact with the resource. The profile will tell the client WHAT it is looking at.Elfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com0tag:blogger.com,1999:blog-7395172357368553438.post-4703989993898597142013-12-06T17:19:00.001+01:002013-12-08T08:39:33.303+01:00Selling the benefits of hypermedia in APIsOnce more I have found myself deeply engaged in a discussion about REST on the api-craft mailing list (<a href="https://groups.google.com/forum/#!topic/api-craft/ZxnLD6q6w7w">https://groups.google.com/forum/#!topic/api-craft/ZxnLD6q6w7w</a>). This time it started with the question "How do I sell the benefits of hypermedia". It turned out to be harder to answer than one would expect, but after some time we came up with the list below. But before we get into that I better explain "hypermedia" in a few sentences.<br />
<br />
The most common use of hypermedia is embedding of links in representations returned from some service on the web. As an example we can look at the representation of a customer record containing a customer ID, customer name, customer contact information and related sales orders. Encoded in JSON we can get something like this:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">// Customer record</span><br />
<span style="font-family: Courier New, Courier, monospace;">// URL template: http://company-x.com/customers/{customer-id}</span><br />
<span style="font-family: Courier New, Courier, monospace;">{</span><br />
<span style="font-family: Courier New, Courier, monospace;"> ID: 1234,</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Name: "John Larsson",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Address: "Marienborg 1, 2830 Virum, Denmark",</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">// List of sales order</span><br />
<span style="font-family: Courier New, Courier, monospace;">// URL template: http://company-x.com/customers/{customer-id}/sales-orders</span><br />
<span style="font-family: Courier New, Courier, monospace;">{</span><br />
<span style="font-family: Courier New, Courier, monospace;"> CustomerId: 1234,</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Orders:</span><br />
<span style="font-family: Courier New, Courier, monospace;"> [</span><br />
<span style="font-family: Courier New, Courier, monospace;"> {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> ID: 10,</span><br />
<span style="font-family: Courier New, Courier, monospace;"> ItemNumber: 15,</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Quantity: 4</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> ]</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<br />
These two resources can be found by expanding the customer ID into the URL templates. This requires the client to be hard coded with 1) the URL templates and 2) the knowledge of which values to use as parameters.<br />
<br />
Now, if we embed links in the responses then we can remove the hard coded knowledge of at least the sales orders URL template:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">// Customer record</span><br />
<span style="font-family: Courier New, Courier, monospace;">// URL template: http://company-x.com/customers/{customer-id}</span><br />
<span style="font-family: Courier New, Courier, monospace;">{</span><br />
<span style="font-family: Courier New, Courier, monospace;"> ID: 1234,</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Name: "John Larsson",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Address: "Marienborg 1, 2830 Virum, Denmark",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> _links:</span><br />
<span style="font-family: Courier New, Courier, monospace;"> [</span><br />
<span style="font-family: Courier New, Courier, monospace;"> {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> rel: "http://linkrels.company-x.com/sales-orders",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> href: "{link-to-sales-orders}",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> title = "Sales orders"</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> ]</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">// List of sales order</span><br />
<span style="font-family: Courier New, Courier, monospace;">// URL template unpublished</span><br />
<span style="font-family: Courier New, Courier, monospace;">{</span><br />
<span style="font-family: Courier New, Courier, monospace;"> CustomerId: 1234,</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Orders:</span><br />
<span style="font-family: Courier New, Courier, monospace;"> [</span><br />
<span style="font-family: Courier New, Courier, monospace;"> {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> ID: 10,</span><br />
<span style="font-family: Courier New, Courier, monospace;"> ItemNumber: 15,</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Quantity: 4,</span><br />
<span style="font-family: Courier New, Courier, monospace;"> _links:</span><br />
<span style="font-family: Courier New, Courier, monospace;"> [</span><br />
<span style="font-family: Courier New, Courier, monospace;"> {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> rel: "http://linkrels.company-x.com/order-details",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> href: "{link-to-sales-order-details}",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> title = "Sales order details"</span><br />
<span style="font-family: Courier New, Courier, monospace;"> },</span><br />
<span style="font-family: Courier New, Courier, monospace;"> {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> rel: "http://linkrels.company-x.com/item-details",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> href: "{link-to-item-details}",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> title = "Item order details (catalog)"</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> ]</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> ]</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<br />
Notice how links are encoded:<br />
<br />
<ul>
<li>Links are always found in collections named _links</li>
<li>A single link consists of a link relation identifier "rel", the hypermedia reference "href" and a human readable description "title".</li>
<li>Link relations are identified by URLs.</li>
</ul>
<br />
The question is know, what is gained by adding such links?<br />
<br />
<h3>
Short term effects</h3>
<div>
<br /></div>
<h4>
1. Explorable API</h4>
It may sound trivial but do not underestimate the power of an explorable API. The ability to browse around the data makes it a lot easier for the client developers to build a mental model of the API and its data structures.<br />
<br />
Think of it like this; traditionally, as a client developer, you would have to read through a pile of documentation before you sit down and write some test programs more or less blindfolded. After that you run your test program to see how the API behaves. Then you have to go back to the documentation and read some more - and then back to coding again. This exercise has three different mental context switches going back and forth between reading documentation, programming and trying out test programs.<br />
<br />
With an explorable API you can simply try out the API and test your understanding of it without any programming. Any mental "what-if" hypothesis testing of the API can be carried out right there without any additional tools or programming. The data as well as the interaction tools is right there in front of you, reducing the mental hoops you have to go through to understand the API.<br />
<br />
The immediate benefits of an explorable API is perhaps more social than technical. But mind you - a lower barrier of entry means happier client developers, higher API adoption rates and less support, which in the end means fewer annoying support calls to bug YOU at the most annoying times of your work.<br />
<br />
<h4>
2. Inline documentation</h4>
Did you notice how link relations are identified by URLs? These URLs can point to online documentation where the API elements can be explained.<br />
<br />
The immediate benefits of this are also social just like the explorability of the API. It will lower the barrier of entry to understanding the API and improve API adoption by client developers.<br />
<br />
<h4>
3. Simple client logic</h4>
A client that simply follows URLs instead of constructing them itself, should be easier to implement and maintain. It won't need logic to figure out which values to substitute into what URL templates. All it has to do is to identify links in the payload and extract the hypermedia reference URL.<br />
<br />
<h3>
Long term effects</h3>
<div>
<br /></div>
<h4>
4. The server takes ownership of URL structures</h4>
The use of hypermedia removes the client's hard coded knowledge of the URL structures used by the server. This means the server is free to change its URL structures over time when the API evolves without any need to upgrade the clients.<br />
<br />
The benefits of this is obviously less coupling between the server and the client, removing the need to upgrade all clients in lock step with the server.<br />
<br />
But, you may ask, why should the server change its URL structures? Once the server developers has decided that the URL is /customers/{customer-id} why should they then suddenly decide to change it? Well, I cannot tell you what will change in your API, but here are two examples:<br />
<br />
- A resource grows too big. Over time it has been necessary to add more and more features to a single resource and one day it simply becomes too big to handle. So it is decided to split it into multiple sub-resources with new URL structures.<br />
<br />
- It turns out that some resources requires bits and pieces of information from other resources when the client access them. It can for instance be an access token of some kind that need to be generated in one place and passed to another resource. With a traditional API the client has to be upgraded with this kind of business logic. With a hypermedia API the client can ignore this complexity and leave it to the server to add the desired parameters to the links it generates.<br />
<br />
<h4>
5. Off loading content to other services</h4>
Consider how APIs evolve: after some time you figure out that some of the content should be off-loaded to a Content Delivery Network (CDN). This means new URLs that points to completely different hosts all over the internet. The actual URL cannot be hard coded into the client since it may change over time or contain random pieces of server generated information for the CDN (like for instance some kind of access token). Now the server HAS to embed the URLs in the responses and the client HAS to follow them.<br />
<br />
<h4>
6. Versioning with links</h4>
With a hypermedia API it becomes trivial to implement new versions of the API resources without breaking existing clients: old clients will follow existing link relations to old-style resources whereas new clients will know how to follow new link relations to new resources - as long as the server response includes both the old as well as the new links.<br />
<br />
Hypermedia also allows the server to re-implement an existing resource with a completely different technology stack, on a completely different server, without the client ever noticing it - given, of course, that the new implementation doesn't make any breaking changes.<br />
<br />
If you want to read more about versioning then take a look at Mark Nottingham's "Web API versioning smackdown" at <a href="http://www.mnot.net/blog/2011/10/25/web_api_versioning_smackdown">http://www.mnot.net/blog/2011/10/25/web_api_versioning_smackdown</a><br />
<br />
<h3>
Large scale effects</h3>
<div>
<br /></div>
<h4>
7. Multiple implementations of the same service</h4>
So far we have only looked at clients dedicated to a unique implementation of a single service. That could for instance be something kike a dedicated Twitter client. But where hypermedia really excels is when we start to work with multiple independent implementations of the same service.<br />
<br />
Let us try to broaden the scene: think of a big corporation that has engulfed and bought up a lot of smaller companies. All of these smaller companies have their own server setup with lists of inventory, sales orders, customers and so on ...<br />
<br />
Now I give you the ID 4328 of customer John Burton ... how will you be able to find the resource that represent the contact information of this customer, when it can live on any one of a dozen servers?<br />
<br />
<b>Solution 1</b>: We need a central indexing service that allows the client to search for customer with ID 4328. But how can the indexing service tell the client where the resulting customer record resides? The answer is simple; the response has to contain a link to the customer record resource.<br />
<br />
<b>Solution 2</b>: Don't use IDs like 4328 at all. Always refer to resources with their full URLs.<br />
<br />
Either way, the client won't know anything about the URL it gets in return - all it has to do is to trust the search result and follow the link.<br />
<br />
And now that we have some opaque, meaningless, URL to our customer record information, how do we get to the sales orders placed by said customer? We could take the customer ID again and throw it into some other indexing service and get a new URL out of it - or we could follow a "sales-orders" link-relation embedded in the customer information.<br />
<br />
The point is:<br />
<blockquote class="tr_bq">
<i>When you transcend from unique one-off service implementations with dedicated clients to multiple independent service implementations with a variety of clients then you simply have to use hypermedia elements.</i></blockquote>
This also means that it can be difficult to sell hypermedia to startup APIs since hypermedia won't add much benefit to one single API living on a isolated island without any requirements of being able to co-exists and co-work seamlessly with other similar services.<br />
<br />
<h3>
Other large scale effects</h3>
Hypermedia solves some of the problems related to large scale service implementations as I have just argued. But there are a few more issues to be solved in order to decouple clients completely from specific server implementations; one is related to how the client understands the result (media types) and one is related to error handling.<br />
<br />
I have already written about error handling in <a href="http://soabits.blogspot.no/2013/05/error-handling-considerations-and-best.html">http://soabits.blogspot.no/2013/05/error-handling-considerations-and-best.html</a> where the last section discuss error handling on a larger scale.<br />
<br />
Unfortunately I have yet to write an article explaining my current view on media types - until then you can either check this post in api-craft <a href="https://groups.google.com/d/msg/api-craft/5N5SS0JMAJw/b0diFRzopY0J">https://groups.google.com/d/msg/api-craft/5N5SS0JMAJw/b0diFRzopY0J</a> or read my ramblings about media types and type systems <a href="http://soabits.blogspot.no/2013/05/the-role-of-media-types-in-restful-web.html">http://soabits.blogspot.no/2013/05/the-role-of-media-types-in-restful-web.html</a>.<br />
<br />
UPDATE (December 8th 2013): I have just added a blog post about media types: <a href="http://soabits.blogspot.no/2013/12/media-types-for-apis.html">http://soabits.blogspot.no/2013/12/media-types-for-apis.html</a><br />
<br />
<h3>
Acknowledgments</h3>
Thanks to Mike Kelly for his initial blogpost on this tema: <a href="http://blog.stateless.co/post/68259564511/the-case-for-hyperlinks-in-apis">http://blog.stateless.co/post/68259564511/the-case-for-hyperlinks-in-apis</a> - and his work on hypermedia linking in JSON with HAL (<a href="http://stateless.co/hal_specification.html">http://stateless.co/hal_specification.html</a>).<br />
<br />
<br />Elfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com4tag:blogger.com,1999:blog-7395172357368553438.post-77732087711873248922013-10-02T22:56:00.001+02:002013-10-02T22:56:51.451+02:00URL structures and hyper media for Web APIs and RESTful servicesA recurring theme on various mailing lists is that of choosing the "right" URL structure for a specific kind of web API. In this post I will present my view on this issue, based on various input from for instance "API-craft" (<a href="https://groups.google.com/forum/#!forum/api-craft">https://groups.google.com/forum/#!forum/api-craft</a>).<br /><br />First of all, let me hammer it in: URL structuring has absolutely nothing to do with REST. Period. REST is not concerned about URL structures - in REST a URL is an opaque string of characters with no meaning beyond the fact that it is both an identifier and a resource locator. An on-line web service doesn't become a RESTful service just because it has a nice pretty looking URL structure. There is simply no such thing as "A RESTful URL".<br /><br />What this means is that a URL like http://geo.com/countries/usa/states/nevada is just as "RESTful" (or non-RESTful) as http://geo.com/states/nevada, http://geo.com/states/321, http://geo.com/states?id=321 and http://geo.com/foo-bar-U7q. The URL structure simply doesn't matter in REST.<br /><br />But from a human point of view it helps understanding if the API has some kind of meaningful URL structure. Computers may easily ignore URL structures but as humans we tend to look at URLs and try to infer meaning from that. Thus, having pretty and well structured URLs helps us understand what is going on - not only as client developers but certainly also as server developers who often have to navigate from URLs to source code - and having a well defined URL structure helps us with that process.<br /><br />In order to discuss URL structures we need a domain to model. I think geographical information with countries, states and cities should be easily understood by most, so lets try that. I will ignore the fact that many countries doesn't have states ;-)<br />
<br />
<h3>
URLs as identifiers for entities</h3>
Our geographical domain easily lends itself to some kind of hierarchical structure of countries / states / cities. So intuitively we reach out for a hierarchical URL structure as shown below (for the sake of clarity I will ignore the host name and only show the path element of the URL). Let us try to build the URL for the city of Las Vegas in Nevada, USA:<br /><br /> <i>/countries/USA/states/Nevada/cities/Las+Vegas</i><br /><br />That would work, but think a bit about it; what if the state of Nevada had more than one city called Las Vegas? How would we be able to distinguish between the two cities? The problem here is that we confuse searching for a city named Las vegas in Nevada, USA with the concept of identifying a specific city.<br /><br />I believe that it is fair to assume that most geographical systems will have some kind of backend that assigns unique identifies to all of its entities. This may be integers, GUIDs or strings with composite keys - but in the end it boils down to a sequence of characters that uniquely identifies the entity in the system.<br /><br />So let us assume that the well known city of Las Vegas is identified by the integer 82137 which is a unique city number. It may happen to be the same number which is used for a country or a state, but in the context of cities it is unique.<br /><br />The same goes for countries and states: USA has the ID 54 and Nevada is identified by 7334. Now we get the URL:<br /><br /> <i>/countries/54/states/7334/cities/82137</i><br /><br />But what happens if some client decided to lookup this URL with mismatching IDs:<br /><br /> <i>/countries/54/states/8112/cities/82137</i><br /><br />Well, that should be considered a non existing resource and the server should return HTTP code 404 Not Found.<br /><br />But why bother at all with the overhead of checking both state, country and city IDs when the city ID uniquely identifies the city? It would be easier for all parties if only the city ID was needed in the URL:<br /><br /> <i>/cities/82137</i><br /><br />Now the server can do one single lookup by the ID to see if the referenced city exists. No need for any additional checking for matching state and country.<br /><br />The same logic can be applied to states (and countries is trivial), so we end up with the following canonical URL structures for countries, states and cities:<br /><br /> <i>/countries/{country-id}</i><br /> <i>/states/{state-id}</i><br /> <i>/cities/{city-id}</i><br /><br />Should it happen that the server doesn't assign unique IDs to cities (or states), and really needs the state reference for a city, because two cities in different states may have the same (non-unique) ID, then we must include both in the URL:<br /><br /> <i>/states/123/cities/77</i> => Rome in Italy (assuming some state in Italy is identified by 123)<br /> <i>/states/432/cities/77</i> => Rome in the state of New York<br /><br />In the rest of this post I will assume that all cities and states has "globally" unique IDs.<br /><br />
<h3>
Finding the right ID with UI dropdowns</h3>
But how does the client know what ID to use, you may ask? This depends on the application, but lets take the scenario where an end user needs to get information about the city of Las Vegas (while still assuming that Nevada may have two Las Vegas).<br /><br />The UI could be structured by three dropdowns: one for countries, one for states in the selected country and one for cities in the selected state. To present such a UI for the end user we first need to be able to get the list of all countries. The obvious choice for this resource is /countries. Then, for the selected country we need the list of states. The obvious choice here is <i>/countries/{country-id}/states</i>.<br /><br />But what about the list of all cities for a specific state in a specific country? Let us avoid the trap of a hierarchical URL with multiple IDs and use the short <i>/states/{state-id}/cities</i>.<br /><br />So now we have the following resources representing lists of geographical items:<br /><br /> <i>/countries</i><br /> <i>/countries/{country-id}/states</i><br /> <i>/states/{state-id}/cities</i><br /><br />Each of these resources returns a JSON list as shown below and from this list the UI can easily build a dropdown element for selecting a city:<br /><br /><span style="font-family: "Courier New",Courier,monospace;">[<br /> { Name: "Item name A", ID: xxx },<br /> { Name: "Item name B", ID: yyy }<br />]</span><br /><br />In this way the client gets the unique city ID by letting the end user select a city and its corresponding ID.<br /><br />
<h3>
Query by text search</h3>
Another approach could be to use textual searching where the end user enters a query text like "Las Vegas, USA" (which is how Google maps work). This would require a new query resource:<br /><br /> <i>/cities?query=Las+Vegas,+USA</i><br /><br />The result would be a list of matching cities:<br /><br /><span style="font-family: "Courier New",Courier,monospace;">[<br /> { Title: "Las Vegas, County A, Nevada, USA", ID: 16352 },<br /> { Title: "Las Vegas, County B, Nevada, USA", ID: 82137 }</span><br />
<span style="font-family: "Courier New",Courier,monospace;">]</span><br />
<br />
Now the end user can select one of the results and thus get the ID of the city.<br /><br />
<h3>
Adding hyper media, getting closer to REST</h3>
The previously mentioned approaches requires the client to create URLs by combining URL templates with IDs. This means the client has to be hard coded with the URL templates - and the consequence is a tight coupling to the URL structure of the web API.<br /><br />But it is very easy to avoid this kind of URL coupling by using hyper media elements in the returned representations. Take for instance the list of cities matching the text "Las Vegas, USA"; here we can include the actual city URLs in the response instead of requiring the clients to construct the URLs itself:<br /><br /><span style="font-family: "Courier New",Courier,monospace;">[<br /> { <br /> Title: "Las Vegas, County A, Nevada, USA", <br /> ID: 16352,<br /> CityLink: "http://.../cities/16352"<br /> },<br /> { <br /> Title: "Las Vegas, County B, Nevada, USA", <br /> ID: 82137,<br /> CityLink: "http://.../cities/82137"<br /> }<br />]</span><br /><br />Now we can start talking about a RESTful service instead of a static web API: by including hyper media elements we allow the server to include links to other hosts that might be better to represent cities:<br /><br /><span style="font-family: "Courier New",Courier,monospace;">[<br /> { <br /> Title: "Las Vegas, County A, Nevada, USA", <br /> CityLink: "http://other-geo-service/jump.aspx?type=city&ID=16352"<br /> },<br /> { <br /> Title: "Las Vegas, County B, Nevada, USA", <br /> CityLink: "http://geo.com/cities/82137"<br /> }<br />]</span><br /><br />By including links we have stopped worrying about URL structures and has come one step closer to a RESTful service.<br /><br />The upside is looser coupling to server URL structures, simpler client logic and enabling the use of different services on different servers. The downside is a larger payload with bigger URLs than simple IDs.<br /><br />
<h3>
Filtering</h3>
So far we have looked at hierarchical data with some obvious URL structures. But what if we need to get the list of cities with a population of more than 200000 citizens? And what if we only want cities from the state of Massachusetts?<br /><br />There are many different ways to do this depending on the complexity of the filtering. But it may be fine to start out with simple queries like "All cities in (Massachusetts or New York)"; first we need to use state IDs and thus we get "All cities in states (2321, 2981)". Such simple integer IDs can be separated with commas, so one possible URL structure could be:<br /><br /> <i>/cities?states=2321,2981</i><br /><br />It is also possible to encode an SQL like query language in one single parameter:<br /><br /> <i>/cities?where=state+in+(2321,2981)+and+population+greater-than+200000</i><br /><br />The possibilities are endless, but it usually consists of a path like /cities that identifies the type of query together with some set of URL parameters encoding the query specification.<br /><br />A common solution is to interpret "&" as AND and "," as OR when possible. So for instance <i>/cities?states=2321,2981&size=large,huge</i> would mean "All cities where state is either (Massachusetts OR New Your) AND size is either (large OR huge)".<br /><br />And no discussion about filtering without mentioning OData's URL conventions: <a href="http://www.odata.org/documentation/odata-v3-documentation/url-conventions/">http://www.odata.org/documentation/odata-v3-documentation/url-conventions/</a><br /><br />
<h3>
Handling large input filters</h3>
URLs for filtering may become rather large, so another recurring question is "How do I handle filter strings too large for a URL"? The recommended solution is to POST the filter to a query resource, for instance like this:<br /><br /><span style="font-family: "Courier New",Courier,monospace;"> POST /city-filters<br /> Content-Type: application/x-www-form-urlencoded<br /><br /> where=state+in+(2321,2981)+and+population+greater-than+200000</span><br /><br />The server then creates a temporary resource for this query and returns a redirect to it:<br /><br /><span style="font-family: "Courier New",Courier,monospace;"> 201 Created<br /> Location: /city-filters/9638</span><br /><br />The client can then GET <i>/city-filters/9638</i> to get the result of the query.<br /><br />A nice side effect of this is that the created filter resource can be cached to avoid re-calculating the potentially very slow query on the server.<br /><br />
<h3>
Natural keys, surrogate keys, URL aliases and resource duplication</h3>
A common question relates to the use of natural keys versus surrogate keys in URL construction. It is more or less the same discussion as we see with databases (see for instance <a href="http://www.agiledata.org/essays/keys.html">http://www.agiledata.org/essays/keys.html</a>). Examples of natural keys could be order numbers, e-mails, postal codes, social security numbers and phone numbers.<br /><br />When choosing between natural keys versus surrogate keys you should consider the lifespan of the key; URLs are supposed to be stable over a very long period of time, so do not choose keys that vary over time. For instance, do not use phone numbers and e-mails to identify people since people tend to change these during their life.<br /><br />You should also beware of natural keys which can be used by more than one entity. It is for instance (still) common for some members of a family to share a common e-mail, so e-mails are not good candidates for identifying persons. Even social security numbers may sometimes change. In Denmark for instance a person may get a new social security number if they change gender.<br /><br />A valid natural key could be a sales order number since these are supposed to be both unique and stable.<br /><br />But if we introduce natural keys, should we then only use natural keys? What if an entity has both a natural key and an internal surrogate key? You can use both but you should decide on one being the canonical ID and avoid duplicating resources by using HTTP redirects for the secondary keys.<br />
<br />
Take for instance a sales order with the order number SK324-1 and internal surrogate key 887766 - if we consider the order number as the canonical ID then we can use these URL structures:<br /><br /> <i>/orders/SK324-1</i> => returns order representation<br /> <i>/orders/id/887766</i> => redirects to /orders/SK324-1<br /><br />Redirects should be done using the HTTP status code 303 See Other with a Location header containing the canonical URL.<br /><br />As stated earlier on: do not confuse searching with identity. You may want to search for a person with a specific e-mail, but the result should include the canonical URL of the found person.<br /><br />See also <a href="http://www.w3.org/TR/webarch/#uri-aliases">http://www.w3.org/TR/webarch/#uri-aliases</a> for a discussion of URL aliases and duplication.<br /><br />
<h3>
Relations and back-references</h3>
What if we want back-references and other relations to other resources, does that influence the URL structure? For instance, now that we have links to states in a country, we might also want links to the country in which a state belongs. That might lead to something like this:<br /><br /> <i>/states/{state-id}/country</i><br /><br />But, wait a minute, we already have links to countries, right? The canonical version is although <i>/countries/{country-id}</i> so how do we get from <i>/states/{state-id}/country</i> to <i>/countries/{country-id}</i>? The obvious answer is to consider the <i>/states/{state-id}/country</i> URL as an alias for some country and use HTTP redirects to get to the canonical country URL.<br /><br />But lets step back and take a broader look at relations in general; a back-reference is just one kind of relation from one resource to another - but we could have many other kinds of relations, like "neighbor states", "the country of a city", "statistical information about a state" and so on. The general solution to this concept is to include links in the payloads instead of creating a myriade of small alias resources that only redirects to canonical URLs.<br /><br />So, instead of using <i>/states/{state-id}/country</i> for the country of a certain state, we include the canonical country link in the representation of the state:<br /><br /><span style="font-family: "Courier New",Courier,monospace;"> GET /states/4321</span><br />
<br /> returns =><br /><br /><span style="font-family: "Courier New",Courier,monospace;"> {<br /> Name: "State X",<br /> CountryLink: "http://.../countries/1234",<br /> NeighborStatesLink: "http://.../states/4321/neighbors"<br /> }</span><br /><br />
<h3>
Static data, volatile data and caching</h3>
Some times we end up with some sort of "hotspot" resource with tons of requests and a very volatile content making it impossible to cache the result and improve performance in that way. A solution to this may be to split the resource into two (or more) different sub-resources; a cacheable resource and a volatile non-cacheable resource.<br /><br />Take for instance our state resource at <i>/states/{state-id}</i> - it may contain some very static data like the name of the state, its area and such like plus some volatile data like for instance the number of Tweets tweeted from that state the last ten minutes. The static information could easily be cached, but we have no way to do it since the complete resource also contains the number of Tweets.<br /><br />The solution is straight forward: split the resource into two different resources:<br /><br /> <i>/states/{state-id}</i> => static state information<br /> <i>/states/{state-id}/tweet-stats</i> => volatile tweet information<br /><br />I'll admit that the above example is rather contrived, so lets try a more realistic example: a streaming music distribution network publishes information about its songs through an online web API. Each song has its own resource representation with details about the song. The title, lyrics, artist and such like won't change much (if ever), but the company also publishes the number of current listeners which changes all the time. To improve caching characteristics the song data is split into (at least) two different resources:<br /><br /> <i>/songs/{song-id}</i> => static song details (cacheable)<br /> <i>/songs/{song-id}/usage</i> => volatile usage information (non cacheable)<br /><br />But who says the song usage is published by the same API? Some time after the initial release of the web API the company off-load some of the streaming to another content delivery network which will also deliver the usage statistics. Now suddenly not only the URL structure changes but even the host name changes:<br /><br /> <i>/songs/{song-id}</i> => static song details (cacheable)<br /> <i>http://cdn.com/acme/file-usage/{song-id}</i> => volatile usage information (non cacheable)<br /><br />This is a breaking change and all clients must now be upgraded. Had the API instead contained hyper links then the change would have been transparent to all clients.<br /><br />Classic song representation:<br /><br /><span style="font-family: "Courier New",Courier,monospace;">{<br /> Id: 1234,<br /> Name: "My song"<br />}</span><br /><br />Hyper media improved representation:<br /><br /><span style="font-family: "Courier New",Courier,monospace;">{<br /> Id: 1234,<br /> Name: "My song",<br /> UsageLink: "http://cdn.com/acme/file-usage/1234"<br />}</span><br /><br />Once again we see how unimportant the actual URL structure is when we start using hyper media elements in the responses.<br /><br />
<h3>
Formats and content types</h3>
If the same resource can be found in different formats (encoded with different media types) then we can ask ourself, should URLs end on .json .xml or similar extensions? On one side it makes it easy to explore the different representations using a standard web browser - on the other side it introduces different URL aliases for the same resource.<br /><br />My recommendation is to implement the extensions as a convenience for the client developers, but avoid using them when interacting with the API "for real". If for instance our geographical API can return both JSON as well as XML and HTML then I would use these URLs for states:<br /><br /> <i>/states/{state-id}</i> => canonical URL used in all returned hyper media elements<br /> <i>/states/{state-id}.json</i> => JSON representation of state<br /> <i>/states/{state-id}.xml</i> => XML representation of state<br /> <i>/states/{state-id}.html</i> => HTML representation of state<br /><br />The canonical URL would also support standard HTTP content negotiation for JSON, XML and HTML representations of the exact same resource. The framework I use, <a href="http://openrasta.org/">OpenRasta</a>, supports this dual type of "content negotiation" right out of the box with no implementation overhead.<br /><br />If our resources have different variations then we can add them as "sub resources" of the primary resource (not that such a thing really exists since URLs are opaque strings). Where I work we have resources for documents in a case management system. These resources contains meta data about the document (title, owner and so on) - and then we have various other (sub) resources for the documents themselves - the raw binary document (image, power point, pdf etc.), a PDF replica of the document and a PDF replica with an added front page containing the document meta data. Thus we get these URLs:<br /><br /> <i>/documents/{doc-id}</i> => canonical document meta data URL<br /> <i>/documents/{doc-id}/pdf</i> => PDF replica<br /> <i>/documents/{doc-id}/meta-pdf</i> => PDF replica with meta data frontpage<br /><br />
<h3>
Use NOUNS not VERBS</h3>
I think most people get this right nowadays: URLs should be NOUNS not VERBS. Avoid URLs like <i>/getOrders</i> and <i>/updateCountry</i> - use all of the HTTP verbs instead when interacting with the resources and use something like <i>/orders/{order-id}</i> and <i>/countries/{country-id}</i> for the URLs. If you run out of HTTP verbs then invent new resources.<br /><br />In this way you will avoid the trap of doing something horrible like this which I would expect to delete order number 1234 when you GET the resource:<br /><br /><span style="font-family: "Courier New",Courier,monospace;"> GET /orders/1234/delete</span><br /><br />You also get the ability to identify all your resources and add caching, which is not possible with this sort of old school SOAP'ish look-up mechanism:<br /><br /><span style="font-family: "Courier New",Courier,monospace;"> POST /orders<br /> Body => { OrderId: 1234, Operation: "read" }</span><br /><br />And you get a nice explorable and consistent API that you developers will love to use :-)<br /><br />
<h3>
Versioning</h3>
Where should API version numbers go in the URL? Should it be /api/v1/countries, /countries-1 or maybe in the host name http://v1.api.geo.com/countries?<br />
<br />
Well, API and URL versioning is a whole story in itself so I suggest you take a look at Mark Nottingham's excellent "API versioning smackdown" (<a href="http://www.mnot.net/blog/2011/10/25/web_api_versioning_smackdown">http://www.mnot.net/blog/2011/10/25/web_api_versioning_smackdown</a>) for a good discussion on this subject.<br /><br /><br />
Have fun, hack some code and create beautiful APIs out there :-)<br />
<br />
/JørnElfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com2tag:blogger.com,1999:blog-7395172357368553438.post-52936080756247287712013-06-05T22:36:00.000+02:002013-06-05T22:41:08.534+02:00Ramone 1.2 releasedThis new version of Ramone adds a few utility methods to the existing library:<br />
<ul>
<li>Introducing cache headers on Request object:<br /> Request.IfModifiedSince()<br /> Request.IfUnmodifiedSince()<br /> Request.IfMatch()<br /> Request.IfNoneMatch()</li>
<li>Introducing .NET cache policy on Session and Service:<br /> Session.CachePolicy<br /> Service.CachePolicy</li>
<li>Adding Request.OnHeadersReady() for working with the underlying HttpWebRequest object.</li>
<li>Adding Request.AddQueryParameters() as an alternative to binding with predefined URL templates.</li>
<li>Adding XML settings in XmlConfiguration.XmlReaderSettings. Used when deserializing XML documents. Default is to allow DTD processing.</li>
</ul>
Ramone is a C# client side library for easy consuming of web APIs and REST services. It is available on GitHub: <a href="https://github.com/JornWildt/Ramone">https://github.com/JornWildt/Ramone</a> and as a NuGet package <a href="https://nuget.org/packages/Ramone/1.2.056">https://nuget.org/packages/Ramone/1.2.056</a>.<br />
<br />
Have fun, Jørn Elfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com0tag:blogger.com,1999:blog-7395172357368553438.post-22216743492364020422013-05-17T22:19:00.000+02:002014-02-24T21:47:45.996+01:00The role of media types in RESTful web servicesOne of the never ending discussions in the REST community is that of custom and domain specific media types; should we, or should we not, create new media types - and if we should, for what reasons should it be done?<br />
<br />
In this blog post I will discuss the role of media types in web services and illustrate it with an example media type. I will go through the requirements for this media type and from this I will build up the features it needs to support. Together with this I will show some example scenarios and sketch out the processing algorithm for the client side. At last I compare this media type to other similar media types (HAL, Sirene, JSON-API).<br />
<br />
My goals for this blog post are:<br />
<ol>
<li>To improve my own understanding of the role of media types in RESTful web services - and share that with others.</li>
<li>To define a new media type for what I call <i>systems integration</i> - and show how it facilitates loose coupling between the integration components.</li>
</ol>
<br />
By <i>systems integration</i> I mean the kind of background processing that takes place behind the scenes in almost any IT enabled business today; shuffling data from one system to another in a safe and durable way without any human interaction.<br />
<br />
REST seems like a good fit for systems integration. It has a strong focus on loosely coupled systems where servers and clients can evolve independently of each others; if we can leverage that then the whole ecosystem of multiple servers and clients should be a lot easier to maintain and with much less downtime required for upgrading the various components.<br />
<br />
There is an ongoing trend to include hyper media controls in never web services; that is a good trend as it removes the clients dependency on specific URL structures. This in turn allows the server to evolve by adding new resources and link to these - and it also facilitates the ability to use multiple servers without the clients ever noticing (since the client do not care about either URL path structures or host names).<br />
<br />
But there is still a thing missing in the puzzle. In Roy Fielding's (in)famous rant "<a href="http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven">REST APIs must be hypertext-driven</a>" he states:<br />
<br />
<blockquote class="tr_bq">
<i>... Any effort spent describing what methods to use on what URIs of interest should be entirely defined within the scope of the processing rules for a media type</i><br />
<i><br /></i>
<i>... From that point on, all application state transitions must be driven by client selection of server-provided choices that are present in the received representations</i></blockquote>
<br />
Especially the last statement is interesting "all application state transitions must be driven by client selection of server-provided choices". This means the client should not make any requests without first being instructed to do so (and how to do it). The client should not POST a new Tweet, bug report or similar without being instructed, on the fly, by some mechanism embedded in the server responses. Todays use of links in responses is on the right track, but links do not inform the client about what HTTP method to use (it assumes GET) and neither does it say anything about the possible payload.<br />
<br />
With this blog post I will try to explain how a media type, with a sufficient number of hyper media controls, together with some intelligent client side code, can enable what Fielding is describing. The downside of this approach is that client implementations become more complex - the upside is that the whole client/server application becomes much more loosely coupled which, in the end, hopefully will help us reach a maintenance Nirvana of loosely coupled systems integration :-)<br />
<br />
By the way, I am not comparing REST with SOAP/WSDL and EDA (event driven architectures) - that is not the purpose here even though these are often found in systems integration projects. I would rather just explore what benefits we can get from REST.<br />
<br />
<h3>
Media type requirements and constrains</h3>
The primary driver for this new media type is loose coupling where the clients only depends on the media type and some out-of-band business specific data structures and identifiers. This means:<br />
<br />
<ul>
<li>The client must not make any assumptions about URL structures.</li>
<li>The client must not make any assumptions about what concrete service implementation it is interacting with.</li>
<li>The client must not initiate any HTTP request without following instructions embedded in server responses (besides the initial request).</li>
<li>The client should not be given more than:
<ul>
<li>A root URL from which all other resources must be discovered at runtime.</li>
<li>A set of business specific data structures.</li>
<li>A set of well known identifiers for locating hyper media controls and business data.</li>
</ul>
</li>
</ul>
The media type itself must be generic with respect to the business domain; it must not contain references to concepts like medical records, e-commerce and so on.<br />
<br />
The media type must be rich enough in terms of <a href="http://www.amundsen.com/blog/archives/1109">hyper media affordances</a> to enable all the operations needed for systems integration.<br />
<br />
The media type does not need to included much, if any, in terms of UI elements since it is intended for operations without human interaction. Neither is the media type intended for mobile use where bandwidth and message size is a concern.<br />
<br />
The media type will be based on JSON. It could just as well be based on XML but, in my experience, JSON is lot simpler to work with, fits the data needs I have met, and has a simple and easy-to-work-with patch format (application/json-patch) which will come in handy later on.<br />
<br />
Armed with these constraints and requirements we are ready to build up our new media type.<br />
<br />
<h3>
Example business domain "BugMe"</h3>
Through out this blog post I will use the imaginary open standard "BugMe" for interacting with bug tracking systems through the new media type. BugMe supports adding of new bug reports, attaching documents to reports, adding comments to reports and similar features shown later on.<br />
<br />
BugMe is <i>not</i> a part of the media type specification - it is only used to illustrate how the media type facilitates interaction with BugMe servers anywhere on the web.<br />
<br />
Neither is BugMe a vendor specific "standard", it is strictly defined in terms of the generic media type and a set of bug reporting specific data structures and identifiers (more on that later on).<br />
<br />
Compare this to APIs like Twitter and others; these are always defined in terms of vendor specific resources and explicit URL structures and was never designed to be implemented on servers anywhere else on the web.<br />
<br />
To highlight the difference between a standard like BugMe and an actual implementation I will assume that some clever guy named Joe, who studies computer science 101 at Example.edu, has set up a BugMe server for some local study project. He is using an implementation that uses a vocabulary slightly different from BugMe - it talks about "issues" where BugMe talks about "bug reports". This fact is illustrated through the concrete URLs used in the examples . The root URL is http://example.edu/~joe/track.<br />
<br />
<h3>
Example 1 - Creating a bug report</h3>
The first thing we will try is to create a new bug report with BugMe. To do so we must supply our client with a few details about the operation:<br />
<ul>
<li>The root URL: http://example.edu/~joe/track/index.</li>
<li>A "create bug report" identifier (as defined by BugMe): "http://bugme.org/names/create-bug-report".</li>
<li>Bug reporting data (as defined by BugMe)
<ul>
<li>Title: "Something bad happened",</li>
<li>Description: "I pressed ctrl-alt-del and all went black",</li>
<li>Severity: 5</li>
</ul>
</li>
</ul>
We must also have an identifier for the media type. Lets call it it "application/razor+json" for no specific reason.<br />
Now we are ready to set our client loose and make it create the bug report. It will do so in the same manner as a human working with a web based UI: get a resource representation, look for well known identifiers that labels data and hyper media controls, fill out data and activate hyper media controls.<br />
<br />
This interaction pattern, getting a resource representation and following instructions on the fly, has a price: it requires more complex client side logic than "normal RPC" patterns with design time binding of methods and it results in higher bandwidth due to the embedded hyper media controls. The upside is a much looser coupling between clients and serves. But all of this is of course already discussed in Fielding's thesis on REST ;-)<br />
<br />
<h4>
GET initial resource</h4>
At the very beginning our client has nothing to do but GET the root URL in hope of finding something useful there:<br />
<br />
<b><span style="font-family: "Courier New",Courier,monospace;">Request</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">GET /~joe/track/index</span><br />
<span style="font-family: "Courier New",Courier,monospace;">Accept: application/razor+json</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<b><span style="font-family: "Courier New",Courier,monospace;">Response</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">Content-Type: application/razor+json</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<span style="font-family: "Courier New",Courier,monospace;">{</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> curies:</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> [</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> { prefix: "bug", reference: "http://bugme.org/names/" }</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> ],</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> controls:</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> [</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> ...,</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> {</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> type: "link",</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> name: "bug:create-bug-report",</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> href: "http://example.edu/~joe/track/add-issue",</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> title: "Add issue to issue tracker"</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> },</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> ...</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> ]</span><br />
<span style="font-family: "Courier New",Courier,monospace;">}</span><br />
<br />
The returned JSON data contains two top level properties defined by the media type: <i>curies</i> and <i>controls</i>. "curies" define short names for URLs used as identifiers in the other elements (see <a href="http://www.w3.org/TR/curie/">http://www.w3.org/TR/curie/</a>) and "controls" contains various hyper media controls. The use of curies should be optioinal - but it helps reading the responses in posts like this.<br />
<br />
Now the client scans the "controls" element looking for the identifier "bug:create-bug-report". In this case it finds a "link" control which is equivalent to an <a href="http://tools.ietf.org/html/rfc4287#section-4.2.7">ATOM link</a>. Since our client understands all the features of the media type it will know that a link should be "followed" by issuing a HTTP GET on the "href" value.<br />
<br />
This little "algorithm" is equivalent to what a human would do: open up a webpage, look for instructions on how to perform the task at hand and then follow them.<br />
<br />
You may have noticed the dots "..." in the example. Those are there for a reason: they illustrate how the client only cares about stuff that is relevant to its current task. Anything else in the response is ignored. The consequence is that the server is free to evolve the content of the resource over time without breaking any clients - as long as it only adds new stuff. Neither does the client care if the content is supposed to be a "link page", a service index, a medical record or have any other specific "type" - as long as it contains elements that will help the client getting closer to its goal.<br />
<br />
<h4>
Follow link </h4>
Here we have the next operation:<br />
<br />
<b><span style="font-family: "Courier New",Courier,monospace;">Request</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">GET /~joe/track/add-issue</span><br />
<span style="font-family: "Courier New",Courier,monospace;">Accept: application/razor+json</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<b><span style="font-family: "Courier New",Courier,monospace;">Response</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">200 Ok</span><br />
<span style="font-family: "Courier New",Courier,monospace;">Content-Type: application/razor+json</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<span style="font-family: "Courier New",Courier,monospace;">{</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> curies: ...,</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> controls:</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> [</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> {</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> type: "poe-factory",</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> name: "bug:create-bug-report",</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> href: "http://example.edu/~joe/track/add-issue",</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> title: "Create new idempotent POE resource"</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> }</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> ]</span><br />
<span style="font-family: "Courier New",Courier,monospace;">}</span><br />
<br />
Bingo! This time the client finds an "poe-factory" control with the right name "bug:create-bug-report" and now its time to create the bug report. The control type "poe-factory" means "Post Once Exactly factory" and is a special action element that enables idempotent POST operations. If you do not know what "idempotent" means then take a look at this page: <a href="http://www.infoq.com/news/2013/04/idempotent">http://www.infoq.com/news/2013/04/idempotent</a>.<br />
<br />
The good thing about idempotent operations is that they can safely be repeated if anything goes wrong on the network. If an operation times out the client can simply retry it again without the risk of creating the same entry multiple times. And since this new media type is for safe and durable "behind the scenes" work I find it rather important to include a mechanism for idempotent POST operations.<br />
<br />
The implementation chosen here requires the client to do an empty POST first. This will create a new POE resource (thus the name "poe-factory") and redirect the client to it. The client can then POST to the new resource as many times it needs until the operation succeeds. The server returns "201 Created" first time it completes the operation whereas it returns "303 See Other" on following requests. In either case the server includes a "Location" header pointing to the new POE resource.<br />
<br />
Subbu Allamaraju has a nice <a href="http://www.subbu.org/blog/2008/10/double-post-and-poe">blog post on post once exactly techniques</a>.<br />
<br />
I chose this approach for the following reasons:<br />
<ul>
<li>It has the simplest possible client side logic - at the cost of an extra round trip to the server. A similar solution could have required the client to create a GUID (message ID) and include it in the payload somehow, but that would make the protocol slightly more prone to client side errors.</li>
<li>It requires no special headers.</li>
<li>It adds no extra information to the payload.</li>
<li>URLs are opaque and the server gets to choose how the POE/message ID is encoded.</li>
</ul>
<br />
<h4>
Create POE resource </h4>
In order to complete its task the client first issues an empty POST operation to the URL of the "href" attribute:<br />
<br />
<b><span style="font-family: "Courier New",Courier,monospace;">Request</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">POST /~joe/track/add-issue</span><br />
<span style="font-family: "Courier New",Courier,monospace;">Content-length: 0</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<b><span style="font-family: "Courier New",Courier,monospace;">Response</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">201 Created</span><br />
<span style="font-family: "Courier New",Courier,monospace;">Location: http://example.edu/~joe/track/add-issue/bd925-ye174h</span><br />
<br />
<h4>
GET POE resource </h4>
It should be rather obvious now that the client has no choice but to follow the response:<br />
<br />
<b><span style="font-family: "Courier New",Courier,monospace;">Request</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">GET /~joe/track/add-issue/bd925-ye174h</span><br />
<span style="font-family: "Courier New",Courier,monospace;">Accept: application/razor+json</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<b><span style="font-family: "Courier New",Courier,monospace;">Response</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">400 Ok</span><br />
<span style="font-family: "Courier New",Courier,monospace;">Content-Type: application/razor+json</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<span style="font-family: "Courier New",Courier,monospace;">{</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> curies: ...,</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> controls:</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> [</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> {</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> type: "poe-action",</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> name: "bug:create-bug-report",</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> documentation: ... some URL ...,</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> method: "POST",</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> href: "http://example.edu/~joe/track/add-issue/bd925-ye174h",</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> type: "application/json",</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> scaffold: ... any JSON object ...,</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> title: "Add issue"</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> }</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> ]</span><br />
<span style="font-family: "Courier New",Courier,monospace;">}</span><br />
<br />
Now the client gets a response with a "poe-action" control. This tells the client that it can safely POST as many times it needs to the "href" URL. The actual payload is given by the BugMe specification (Title, Description, Severity).<br />
<br />
Some comments on the above response:<br />
<ol>
<li>The payload is encoded in application/json as a trivial JSON object. Other formats may be included in the media type spec later on.</li>
<li>This format is NOT intended for automatic creation of UI's and thus it contains no UI related list of field definitions or similar.</li>
<li>It is NOT necessary to embed any kind of schema information - that sort of thing is given by the name of the control element.</li>
<li>The optional "scaffold" value is the JSON payload equivalent of a URL template: it supplies default values to some properties and adds additional "hidden" properties the client can ignore (as long as they are sent back).</li>
<li>POE-actions are not restricted to POST - a PATCH with json/patch would work as well (but then perhaps we need to change the action type name).</li>
</ol>
<h4>
Create bug report </h4>
Then the client issues a new request:<br />
<br />
<b><span style="font-family: "Courier New",Courier,monospace;">Request</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">POST /~joe/track/add-issue/bd925-ye174h</span><br />
<span style="font-family: "Courier New",Courier,monospace;">Accept: application/razor+json</span><br />
<span style="font-family: "Courier New",Courier,monospace;">Content-Type: application/json</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<span style="font-family: "Courier New",Courier,monospace;">{</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> Title: "Something bad happened",</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> Description: "I pressed ctrl-alt-del and all went black",</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> Severity: 5</span><br />
<span style="font-family: "Courier New",Courier,monospace;">}</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<b><span style="font-family: "Courier New",Courier,monospace;">Response</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">201 Created</span><br />
<span style="font-family: "Courier New",Courier,monospace;">Location: http://example.edu/~joe/track/issues/32</span><br />
<br />
<h4>
GET created bug report </h4>
Now we are done unless we want to see the actual created bug report by following the Location header:<br />
<br />
<b><span style="font-family: "Courier New",Courier,monospace;">Request</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">GET /~joe/track/issues/32</span><br />
<span style="font-family: "Courier New",Courier,monospace;">Accept: application/razor+json</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<b><span style="font-family: "Courier New",Courier,monospace;">Response</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">Content-Type: application/razor+json</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<span style="font-family: "Courier New",Courier,monospace;">{</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> curies: ...,</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> controls: ...,</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> payloads:</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> [</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> ...,</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> {</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> name: "bug:bug-report",</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> data:</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> {</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> Id: 32,</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> Title: "Something bad happened",</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> Description: "I pressed ctrl-alt-del and all went black",</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> Severity: 5,</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> Created: "2012-04-23T18:25:43Z"</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> }</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> },</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> ...</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> ]</span><br />
<span style="font-family: "Courier New",Courier,monospace;">}</span><br />
<br />
Now that the client can see the actual bug report it wanted to create it knows that the task is completed. Everyone is smiling and put on their happy face :-)<br />
<br />
<h3>
Other hyper media controls</h3>
There are of course more scenarios to cover than this single "Create stuff" scenario and these scenarios will call for other kinds of hyper media controls, for instance URL templates, PATCH actions, binary file upload and more (I should cover these in some future blog posts ...)<br />
<br />
<h3>
Error handling</h3>
If the client receives a 4xx or 5xx status code it can inspect the JSON payload and look for a property named "error" together with the other "payloads" and "controls" properties. The "error" property should contain data according to <a href="http://soabits.blogspot.dk/2013/05/error-handling-considerations-and-best.html">my previous blog post on error handling</a>.<br />
<br />
Here is an example:<br />
<br />
<b><span style="font-family: "Courier New",Courier,monospace;">Request</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">POST /~joe/track/add-issue/bd925-ye174h</span><br />
<span style="font-family: "Courier New",Courier,monospace;">Accept: application/razor+json</span><br />
<span style="font-family: "Courier New",Courier,monospace;">Content-Type: application/json</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<span style="font-family: "Courier New",Courier,monospace;">{</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> Title: "Something bad happened",</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> Description: "I pressed ctrl-alt-del and all went black",</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> Severity: 5</span><br />
<span style="font-family: "Courier New",Courier,monospace;">}</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<b><span style="font-family: "Courier New",Courier,monospace;">Response</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">503 Service Unavailable</span><br />
<span style="font-family: "Courier New",Courier,monospace;">Content-Type: application/razor+json</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<span style="font-family: "Courier New",Courier,monospace;">{</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> error:</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> {</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> message: "Could not create new bug report; server is down for maintenance",</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> ...</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> }</span><br />
<span style="font-family: "Courier New",Courier,monospace;">}</span><br />
<br />
In addition to this the client can try to use content negotiation to receive error information in the format of <a href="https://tools.ietf.org/html/draft-nottingham-http-problem-03">application/api-problem+json</a>.<br />
<br />
<h3>
Client side processing algorithm</h3>
Here is a simplified view of how the client should process the content:<br />
<ol>
<li>GET initial root resource.</li>
<li>[LOOP:] Look for hyper media controls with appropriate names.</li>
<li>Check the type of the found control element:
<ol>
<li>If it is a "link" then follow that link and restart from [LOOP].</li>
<li>If it is a "poe-factory" then issue an empty POST to the href value and restart from [LOOP].</li>
<li>if it is a "poe-action" then issue a request with the specified method and data encoded according to the "target" media type. Then restart from [LOOP].</li>
</ol>
</li>
<li>Look for a payload with the appropriate name: If it exists then the task is complete - otherwise it has failed (actually I don't like this last step, but that is the only kind of "acknowledge" I can see the server responding with).</li>
</ol>
A consequence of this approach is that the service specification (BugMe in my example) should state nothing about <i>how</i> to find and update data since that is up to the servers actual implementation. The service specification should only consider <i>what</i> kind of data to look for or modify. The "how"-part is contained entirely in the returned hyper media controls.<br />
<br />
As the media type evolves and more types of hyper media controls are added the client(s) will grow more and more complex. This is one of the trade offs that has to be accepted in order to keep clients and servers as loosely coupled as possible.<br />
<br />
If the media type gets popular one could even expect to see the same scenario we see with todays web browsers: there will be multiple implementations of the client libraries and some will implement more than others of the final specification.<br />
<br />
<h3>
No profile needed</h3>
It may be tempting to allow for a "profile" parameter with the media type ID. But typically that would be used to ask for a specific "type" of a resource like for instance "application/razor+json;profile=user". As can be seen in the client side processing algorithm above there is no need for such a thing, so lets not introduce it.<br />
<br />
<h3>
Related work</h3>
Quite a few other people are trying to create new media types to reach similar goals, but neither of them include features such as POE semantics. Here is the list of related media types that I am aware of:<br />
<ul>
<li><a href="http://stateless.co/hal_specification.html">HAL</a></li>
<li><a href="https://github.com/kevinswiber/siren">Sirene</a> </li>
<li><a href="http://jsonapi.org/">JSON API</a> </li>
<li><a href="http://restfulobjects.org/">Restful Objects</a> </li>
<li><a href="http://tools.ietf.org/html/draft-nottingham-json-home-03">JSON Home</a> </li>
</ul>
And then there is Jim Webber's fantastic "<a href="http://www.infoq.com/articles/webber-rest-workflow">How to GET a cup of coffee</a>" which has been a big inspiration for me over the years.<br />
<br />
<h3>
Reasons for creating a new media type</h3>
How many media types should we invent? Well, as many as needed, I would say. The media type described here includes some features not found in other media types (POE semantics for instance) and that should be sufficient argument for creating a new one.<br />
<br />
I don't see anything wrong by creating many media types - eventually a few of them will be good enough and gain enough traction to become ubiquitous standards. That's called evolution.<br />
<br />
<h3>
Summary</h3>
In this blog post I have tried to explain one way of understanding media type's role in RESTful web services and illustrated it by building up (parts of) a media type for systems integration. I have also touched upon the issue of "typed" resources and how to avoid it (by not assuming anything about the resource type and instead look for certain identifiers in the response) ... there could be a blog post more to come on this issue.<br />
<br />
So what do you think? Was this useful, understandable, totally overkill, outright naive or simply a pile of, well, rubbish? Feel free to add a comment, Tweet me or send me an e-mail. I would love to get some feedback.<br />
<br />
Happy hacking, Jørn<br />
<br />
UPDATE 2014-02-24: I have actually put much of this into a media type called Mason. See <a href="http://soabits.blogspot.dk/2014/02/implementing-hypermedia-apis-and-rest.html">http://soabits.blogspot.dk/2014/02/implementing-hypermedia-apis-and-rest.html</a>.Elfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com4tag:blogger.com,1999:blog-7395172357368553438.post-54543456754073287582013-05-15T21:42:00.000+02:002013-05-15T21:42:44.382+02:00Error handling considerations and best practicesA recurring topic in REST and Web API discussions is that of error handling (see for instance <a href="https://groups.google.com/d/topic/api-craft/GLz_nNbK-6U/discussion">https://groups.google.com/d/topic/api-craft/GLz_nNbK-6U/discussion</a> or <a href="http://stackoverflow.com/questions/942951/rest-api-error-return-good-practices">http://stackoverflow.com/questions/942951/rest-api-error-return-good-practices</a>]; what information should be included in error responses, how should HTTP status codes be used and what media type should the response be encoded in? In this blog post I will try to address these issues and give some guidelines based on my own experience and existing solutions.<br />
<h2>
Existing solutions</h2>
Let us first take a look at some existing solutions to get started:<br />
<ul>
<li>The <a href="https://dev.twitter.com/docs/error-codes-responses">twitter API</a> uses a list of descriptive error messages and error codes. Twitter has both JSON and XML representations with property names: "errors", "error", "code"<br /></li>
<li>The <a href="https://developers.facebook.com/docs/reference/api/errors/">Facebook Graph API</a> has a single descriptive error message, an error code and even a sub-code. Facebook uses a JSON representation with property names: "error", "message", "type", "code" and "error_subcode".<br /></li>
<li>The <a href="http://developer.github.com/v3/">Github API</a> has a top level descriptive error message and a optional list of additional error elements. The items in the error list refers to resources, fields and codes. Github uses a JSON representation with property names: "message", "errors", "resource", "field", "code".<br /></li>
<li>The <a href="https://github.com/WhiteHouse/api-standards">US White House</a> has a set of guidelines for its APIs on GitHub. The error message used here contains the HTTP status code, a developer message, a user message, an error code and links to further information.<br /></li>
<li><a href="https://github.com/blongden/vnd.error">Ben Longden</a> has proposed a media type for error reporting. This specification includes an "logref" identifier that some how refers to a log entry on the server side - such a feature can help debugging server errors later on.<br /></li>
<li><a href="https://tools.ietf.org/html/draft-nottingham-http-problem-03">Mark Nottingham</a> has introduced "Problem Details for HTTP APIs" as an IETF draft. This proposal makes use of URIs for identifying errors and is as such meant as a general and extensible format for "problem reporting".</li>
</ul>
All of these response formats share some similar content: one or more descriptive messages, status codes and links to further information. But as can be seen there is a wide variety in the actual implementation and wire format.<br />
<h2>
Considerations and guidelines</h2>
So, what should you do with your web API? Well, here are some considerations and guidelines you can base your error reporting format on ...<br /><br />
<h3>
Target audience</h3>
Remember that your audience includes both the end user, the client developer, the client application and your frontline support (which may just happen to be <i>you</i>). Your error responses should include information that caters for all of these parties:<br />
<ul>
<li>The end user needs a short descriptive message.</li>
<li>The client developer needs as much detailed information as possible to debug the application.</li>
<li>The client application needs error codes (HTTP status codes) for error recovery actions.</li>
<li>The frontline support people needs detailed information and/or keywords to look for in their knowledge database.</li>
</ul>
<h3>
Use the HTTP status codes correct</h3>
The HTTP status codes are standardized all over the web and your clients will know immediately how to handle them. Make sure to use them correct:<br />
<ul>
<li>Do NOT just return HTTP status code 200 (OK) regardless of success or failure.</li>
<li>Use 2xx when a request succeeds.</li>
<li>Use 4xx when a request fails and the client should be able to fix it by modifying its own request.</li>
<li>Use 5xx when a request fails due to some internal server error.</li>
</ul>
<h3>
Use descriptive error messages</h3>
Be descriptive in your error messages and include as much context as possible. Failure to do so will cost you dearly in support later on: if your client developers cannot figure out why their request went wrong, they will look for help - and eventually that will be you who will spend time tracking down client errors instead of coding new and exiting features for your service.<br /><br />If it is a validation error, be sure to include why it failed, where it failed and what part of it that failed. A message like "Invalid input" is horrible and client developers will bug you for it over and over again, wasting your precious development time. Be descriptive and include context: "Could not place order: the field 'Quantity' should be an integer between 0 and 99 (got 127)".<br /><br />You may want to include both a short version for end users and a more verbose version for the client developer.<br />
<br />
<h3>
Localization</h3>
Error messages for end users should be localized (translated into other languages) if your service is already a multi language service. Personally I don't think developer messages should be localized: it is difficult to translate technical terms correct and it will make it more difficult to search online for more information.<br /><br />When localization is introduced it may also be necessary to include language codes and maybe even allow for a list of different translations to be returned in the error response.<br />
<br />
<h3>
Allow for more than one message</h3>
Make it possible to include more than one message in the error response. Then try to collect all possible errors on the server side and return the complete list in a single response. This is not always possible - and requires some more coding on the server side (compared to simply throwing an exception first time some invalid input is detected).<br />
<br />
<h3>
Additional status codes</h3>
If your business domain calls for more detailed information than can be found in the normal HTTP status codes then include a business specific status code in the response. Make sure all of the codes are documented.<br /><br />You may be tempted to include more technical error codes, but consider who your audience is for that: It won't help your end user. It may help your client application recovering from errors - but probably not in any way that was not already covered by the HTTP status codes. Your client developer may have some need for it - but why make them lookup error codes in online documentation when you can include descriptive error text and links that refers directly to the documentation? It may help your support - but if the client dev have enough information in the error response they won't need to call your support anyway - right?<br />
<br />
<h3>
Use letters for status codes</h3>
I often find myself searching for online resources that can help me when I get some error while interacting with third party APIs. Usually I search for a combination of the API name, error messages and codes. If you include additional error codes in your response then you might want to use letters instead of digits: it is simply more likely to get a relevant hit for something like "OAUTH_AUTHSERVER_UNAVAILABLE" than "1625".<br />
<br />
<h3>
Include links to online resources</h3>
Include links to online help and other resources that will either clarify what went wrong or in some other way help the client developer to solve the problem.<br />
<br />
<h3>
Support multiple media types</h3>
If your have a RESTful service that allows both client applications and developers to explore it then you might want to support a human readable media type for your error responses. HTML is perfect for this as it allows the client developers to view the error information righ in their browsers without installing any additional plugins. A fallback to plain text could also be useful (but probably overkill).<br /><br />
<h3>
Include a timestamp or log-reference</h3>
It can help support and bug hunting if the error report contains a timestamp (server timezone or UTC). This may help locating the right logfile entries later on.<br /><br />Another possibility is to include some other kind of information that refers back to the logfiles such that server developers and support people can track what happened.<br />
<br />
<h3>
Field-by-field messages</h3>
In some cases it makes sense to be explicit about the fields in the input that caused the errors and include field names in separate elements of the error response. For instance something like this JSON response:<br /><br /><span style="font-family: "Courier New",Courier,monospace;">{<br /> message: "One or more inputs were not entered correctly",<br /> errors:<br /> [<br /> { field: "Weight", message: "The value if 'Weight' exceeds 100 - the value should be between 0 and 100" },<br /> { field: "Height", message: "A value must be entered for 'Height'" }<br /> ]<br />}</span><br /><br />This would make it possible for the client to highlight those fields in the UI and draw the end users attention to them. It is although difficult to keep clients and servers in sync and requires a lot of coding on both sides to get it to work. Usually field-by-field information is handled by client side validation logic anyway. So a clear error message like "The value of 'Weight' exceeded 100 - the value should be between 0 and 100" should be enough for most applications.<br /><br />
<h3>
Include the HTTP status code</h3>
This may sound a bit odd, but according to people on api-craft there are some client side environments where the application code do not have access to the HTTP headers and status codes. To cater for these clients it may be necessary to include the HTTP status code in the error message payload.<br /><br />
<h3>
Do not include stack traces</h3>
It may be tempting to include a stack trace for easier support when something goes wrong. Don't do it! This kind of information is too valuable for hackers and should be avoided.<br />
<br />
<h2>
Implementation</h2>
Now that we have our "requirements" ready we should be able to design a useful solution. Lets first try to define the response without considering an actual wire format:<br />
<br />
<ul>
<li><span style="font-family: "Courier New",Courier,monospace;">message</span> (string): the primary descriptive error message - either in the primary language of the server or translated into a language negotiated via the HTTP header "Accept-Language".</li>
<li><span style="font-family: "Courier New",Courier,monospace;">messages</span> (List of string): an optional list of descriptive error messages (with the same language rules as above).</li>
<li><span style="font-family: "Courier New",Courier,monospace;">details</span> (string): an optional descriptive text targeted at the client developer. This text should always be in the primary language of the expected developer community (that would be English in my case).</li>
<li><span style="font-family: "Courier New",Courier,monospace;">errorCode</span> (string): an optional error code.</li>
<li><span style="font-family: "Courier New",Courier,monospace;">httpStatusCode</span> (integer): an optional copy of the HTTP status code.</li>
<li><span style="font-family: "Courier New",Courier,monospace;">time</span> (date-time): an optional timestamp of when the error occurred.</li>
<li><span style="font-family: "Courier New",Courier,monospace;">additional</span> (any data): a placeholder for any kind of business specific data.</li>
<li><span style="font-family: "Courier New",Courier,monospace;">links</span> (List of <string,string,string>): an optional list of links to other resources that can be helpful for debugging (but should probably not be shown to the end user). Each link consists of <href, rel, title> just like an <a href="http://tools.ietf.org/html/rfc4287#section-4.2.7">ATOM link</a> element.</li>
</ul>
I have ignored the possibility of having multiple translations of the messages. Neither does this implementation include any field-by-field validation since I expect that to be performed by the client. That doesn't mean the server shouldn't do the validation - it just doesn't have to return the detailed field information in a format readable by the client application.<br />
<br />
<h2>
JSON format example</h2>
Now it is time to select a wire format for the error information. I will choose JSON since that is a wide spread and well known format that can be handled by just about any piece of infrastructure nowadays. The format is straight forward and is probably best illustrated with a few examples:<br /><br /><b>Example 1 - the simplest possible instantiation</b><br /><br /><span style="font-family: "Courier New",Courier,monospace;">{<br /> message: "The field 'StartDate' did not contain a valid date (the value provided was '2013-20-23'). Dates should be formated as YYYY-MM-DD."<br />}</span><br /><br /><b>Example 2 - handling multiple validation errors</b><br /><br /><span style="font-family: "Courier New",Courier,monospace;">{<br /> message: "There was something wrong with the input (see below)",<br /> messages:<br /> [<br /> "The field 'StartDate' did not contain a valid date (the value provided was '2013-20-23'). Dates should be formated as YYYY-MM-DD.",<br /> "The field 'Title' must have a value."<br /> ]<br />}</span><br /><br /><b>Example 3 - using most of the features</b><br /><br />
<span style="font-family: "Courier New",Courier,monospace;">{<br /> message: "Could not authorize user due to an internal problem - please try again later.",<br /> details: "The OAuth2 service is down for maintenance.",<br /> errorCode: "O2SERUNAV",<br /> httpStatusCode: 503,<br /> time: "2013-04-30T10:27:12",<br /> links:<br /> [<br /> { </span><br />
<span style="font-family: "Courier New",Courier,monospace;"> href: "http://example.com/oauth2status.html", </span><br />
<span style="font-family: "Courier New",Courier,monospace;"> rel: "help", </span><br />
<span style="font-family: "Courier New",Courier,monospace;"> title: "Service status information" </span><br />
<span style="font-family: "Courier New",Courier,monospace;"> }<br /> ]<br />}</span><br />
<h2>
Client implementation and media types - a matter of perspective</h2>
The client implementation should, at a suitable high level, be straight forward:<br />
<ol>
<li>Client makes an HTTP request.</li>
<li>Request fails for some reason, server returns HTTP status code 4xx or 5xx and includes error information in the HTTP body.</li>
<li>Client checks HTTP status code, sees that it is 4xx or 5xx and decodes the error information.</li>
<li>Client tries to recover from error - either showing the error message to the end user, write the error to a log, give up or maybe retry the request - all depending on the error and the client's own capabilities.</li>
</ol>
But, hey, wait a minute ... how does the client know how to to decode the payload? I mean, perhaps the client asked for a resource representation containing medical records, but then it got a HTTP status code 400 - how is it supposed to know the format of the error information?<br /><br />If the client is working with a vendor specific service, like Twitter and GitHub, then chances are that the client is hard wired to extract the error information based on the vendor specific service documentation. My guess is that this is how most clients are implemented.<br /><br />But what if the client is working with a more, shall we say, RESTful service? That is; the client doesn't know what actual implementation it is interacting with. This could for instance be the case of clients consuming an ATOM feed (application/atom+xml). How would the client know how to decode the error response payload? Actually this seems like an unanswered question for ATOM since the spec is rather vague about this point (see for instance <a href="http://stackoverflow.com/questions/9874319/how-to-represent-error-messages-in-atom-feeds">http://stackoverflow.com/questions/9874319/how-to-represent-error-messages-in-atom-feeds</a>)<br /><br />A RESTful service specification may call for a media type dedicated to error reporting; lets call such a media type "application/error+json". When the client receives a 4xx or 5xx HTTP status it can then look at the content-type header: if it matches "application/error+json" then the client would know exactly what to look for in the HTTP body.<br /><br />It could also be that the base media type included detailed specification about error payloads.<br /><br />I would prefer one of the two last options: either specify error handling in the base media type of the service - or use an existing standard media type. The last option is actually what Mark Nottingham has done with <a href="https://tools.ietf.org/html/draft-nottingham-http-problem-03">https://tools.ietf.org/html/draft-nottingham-http-problem-03</a>.<br /><br />So it is a matter of perspective: vendor specific "one-of-a-kind" services tend to invent their own error formats whereas RESTful services (like ATOM) should standardize error reporting via media types for everyone to reuse all over the web.<br /><br />Have fun, JørnElfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com7tag:blogger.com,1999:blog-7395172357368553438.post-36383542613018523352013-04-11T22:08:00.001+02:002013-04-12T04:59:22.537+02:00Asynchronous HTTP requests using Ramone<a href="https://github.com/JornWildt/Ramone">Ramone</a> (my C# web API and REST client) now supports asynchronous HTTP request :-) This has taken quite a while for me to implement due to a large amount of automatic tests for the the async request handling. But now its there and I hope some of you will find it useful in your applications.<br />
<br />
Asynchronous requests can be prepared via a call to <span style="font-family: "Courier New",Courier,monospace;">Async()</span> on the <span style="font-family: "Courier New",Courier,monospace;">Request</span> object. After this you can do the actual request using any of the GET, POST, etc. methods where a callback delegate is passed as on of the arguments. When the request completes it will call back to the delegate, passing in the response from the request.<br />
<br />
Here is one example:<br />
<pre class="brush: csharp">// Define resource type
public class Cat
{
public string Name { get; set; }
public DateTime DateOfBirth { get; set; }
}
// URL template (relative to service root)
const string CatUrlTemplate = "/cat/{name}";
public static void GetAsync()
{
// Create session pointing to service root
ISession Session = RamoneConfiguration.NewSession(...));
// Setup HTTP request
Request req = Session.Bind(CatUrlTemplate, new { name = "Mike" });
Console.WriteLine("Waiting for request to finish ...");
// Initiate asynchronous request
req.AcceptJson().Async()
.Get<Cat>(response =>
{
Console.WriteLine("Cat: {0}.", response.Body.Name);
Console.Write("Press Enter to complete: ");
});
Console.ReadLine();
}
</pre>
It is also possible to register a callback delegate for error handling; if anything goes bad with the request (or if the operation callback delegate raises an exception) then this error delegate will be called.<br />
<br />
In the same way it is possible to register a callback delegate to be called after the request has been completed. This delegate will always be called no matter what the outcome of the operation is.<br />
<br />
Building on the previous example, we can now include the two OnError and OnComplete handlers:<br />
<pre class="brush: csharp">// Initiate asynchronous request with additional handlers
req.AcceptJson().Async()
.OnError(e => Console.WriteLine("Failed: {0}", e.Exception.Message))
.OnComplete(() => Console.Write("Press Enter to complete: "))
.Get<Cat>(response =>
{
Console.WriteLine("Cat: {0}.", response.Body.Name);
});
</pre>
POST, PUT and other operations usually includes a body in the request. This can easily be added as a parameter in the asynchronous operation methods:<br />
<br />
<pre class="brush: csharp">// Initiate asynchronous POST including a request body
var body = ... any data object to POST ...
req.AcceptJson().Async()
.Post<Cat>(body, response =>
{
Console.WriteLine("Cat: {0}.", response.Body.Name);
});
</pre>
Ramone will also handle redirects, authentication and all the other standard synchronous stuff when going asynchronous.<br />
<br />
Have fun :-) <br />
<script type="text/javascript">
SyntaxHighlighter.all()
</script>
Elfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com0tag:blogger.com,1999:blog-7395172357368553438.post-64564057762565337522013-03-08T22:09:00.001+01:002013-03-08T22:12:56.610+01:00Using Ramone for OAuth2 authorization with Google APIsI have blogged a couple of times about Ramone which is a C# client side library for interacting with RESTful web APIs (see <a href="https://github.com/JornWildt/Ramone">https://github.com/JornWildt/Ramone</a>). This time I will show how to use Ramone to authorize with Google's web services using OAuth2.<br />
<br />
The OAuth2 flow to use is a variation of the "Client Credentials Grant" flow from OAuth2 (see <a href="http://tools.ietf.org/html/rfc6749#section-4.4">http://tools.ietf.org/html/rfc6749#section-4.4</a>) - but instead of sending the client credentials in clear text over the wire it uses a signed JSON Web Token (JWT) assertion instead (see <a href="http://tools.ietf.org/html/draft-ietf-oauth-json-web-token-06">http://tools.ietf.org/html/draft-ietf-oauth-json-web-token-06</a>).<br />
<br />
The concept itself is relatively simple: create a digitally signed piece of data (an assertion) that proves who the client is and then present that assertion to Google. It does although require quite a few steps to get it right.<br />
<br />
Before we start making HTTP requests we must first acquire a set of client credentials from Google. This comes in the form of a X509 certificate which contains the private signing key for the client. To obtain this certificate we use Googles API console at <a href="https://code.google.com/apis/console">https://code.google.com/apis/console</a>. Login and go to "API access" where you click on the "Create client ID" button and select "Service account":<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8g8ji9j5JihW-J__qNjA-doOxv5tvM7VZKRoa5vnScgi0Vb97lIGDO32Ld5TUcf-AGfg97urvJCx_yn5g6gbNAAUhO9L1Jnsrdi6UIXBE-auKGa97kAPpl5zoWLPIPkUASWtwv1WG_A/s1600/GoogleApi-CreateClientId.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="220" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8g8ji9j5JihW-J__qNjA-doOxv5tvM7VZKRoa5vnScgi0Vb97lIGDO32Ld5TUcf-AGfg97urvJCx_yn5g6gbNAAUhO9L1Jnsrdi6UIXBE-auKGa97kAPpl5zoWLPIPkUASWtwv1WG_A/s400/GoogleApi-CreateClientId.png" width="400" /></a></div>
<br />
<br />
When your client credentials has been created you are asked to save them somewhere secure. Save them in a place where your program can load it from later.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUa8hmB57ygbeupEGJBRGRF8nYgDj6ePbbGfF7RuG7dSpqrKkyPhzTGEZauycY84UY68gsrlw1TKM3ZeMBxxemG48slNQVpINEFNSgyDpVpV0Zn5GX3rJacNeL_2pgDN8dFuZg8-98Lw/s1600/GoogleApi-SavePrivateKey.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="153" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUa8hmB57ygbeupEGJBRGRF8nYgDj6ePbbGfF7RuG7dSpqrKkyPhzTGEZauycY84UY68gsrlw1TKM3ZeMBxxemG48slNQVpINEFNSgyDpVpV0Zn5GX3rJacNeL_2pgDN8dFuZg8-98Lw/s400/GoogleApi-SavePrivateKey.png" width="400" /></a></div>
<br />
The next step is to get Google's Token Endpoint URL. At the time of writing it is https://accounts.google.com/o/oauth2/token but this may of course change in the future.<br />
<br />
Now we are ready to rock! First we initialize Ramone with the Google stuff:<br />
<br />
<pre class="brush: csharp">using System;
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;
using Ramone;
using Ramone.OAuth2;
using Ramone.Utility.JsonWebToken;
const string GoogleAPIBaseUrl = "https://www.googleapis.com/oauth2/v1";
const string TokenEndpointUrl = "https://accounts.google.com/o/oauth2/token";
ISession Session = RamoneConfiguration.NewSession(new Uri(GoogleAPIBaseUrl));
OAuth2Settings settings = new OAuth2Settings
{
TokenEndpoint = new Uri(TokenEndpointUrl),
ClientAuthenticationMethod = OAuth2Settings.DefaultClientAuthenticationMethods.Other
};
Session.OAuth2_Configure(settings);
</pre>
Then we load the certificate we got from Google. That is albeit not as easy as it sounds since the default X509 certificate doesn't support the RSA256 signing required by the standard. So we have to load the certificate first and then create a new RSACryptoServiceProvider from that:<br />
<br />
<pre class="brush: csharp">private static RSACryptoServiceProvider GetCryptoServiceProvider()
{
const string CertificatePath = "path-to-your-certificate-file";
X509Certificate2 certificate = new X509Certificate2(CertificatePath, "notasecret", X509KeyStorageFlags.Exportable);
using (RSACryptoServiceProvider cp = (RSACryptoServiceProvider)certificate.PrivateKey)
{
// Create new crypto service provider that supports SHA256 (and don't ask me why the first one doesn't)
CspParameters cspParam = new CspParameters
{
KeyContainerName = cp.CspKeyContainerInfo.KeyContainerName,
KeyNumber = cp.CspKeyContainerInfo.KeyNumber == KeyNumber.Exchange ? 1 : 2
};
return new RSACryptoServiceProvider(cspParam) { PersistKeyInCsp = false };
}
}
</pre>
At last we can authenticate with Google using the client credentials we loaded from the certificate. The "Issuer" value below is the e-mail address of your "Service account" client as stated in Google's API console:<br />
<br />
<pre class="brush: csharp">const string Issuer = "your-client-issuer-email-address-from-google";
const string Scope = "https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/userinfo.profile";
using (RSACryptoServiceProvider csp = GetCryptoServiceProvider())
{
AssertionArgs args = new AssertionArgs
{
Audience = TokenEndpointUrl,
Issuer = Issuer,
Scope = Scope
};
Session.OAuth2_GetAccessTokenFromJWT_RSASHA256(csp, args);
}
</pre>
The last statement makes the actual request to Google with the signed JWT assertion, reads the returned OAuth2 access token and stores it in the session for use in the following requests.<br />
<br />
If this works you are done. Ramone has authorized with Google using your client credentials without any intervention from a user.<br />
<br />
To prove that it works we can now fetch the user name and e-mail of the current user (your client) identified by the access token returned from Google:<br />
<br />
<pre class="brush: csharp">Console.WriteLine("Reading user information from Google");
using (var response = Session.Bind("userinfo").AcceptJson().Get<dynamic>())
{
var body = response.Body;
Console.WriteLine("\nRESULT:");
Console.WriteLine("User name: " + body.name);
Console.WriteLine("E-mail: " + body.email);
}
</pre>
Have fun :-)
<script type="text/javascript">
SyntaxHighlighter.all()
</script>Elfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com0tag:blogger.com,1999:blog-7395172357368553438.post-13376061816833947522013-02-18T09:35:00.001+01:002013-03-08T22:10:41.447+01:00Visualizing adventure race data using animated heat maps on Google mapsGoogle maps is an amazing technology and I have spent countless hours zooming in and out with Google Earth - looking down on places I know or places I am going to visit. It is also amazing how accessible and simple the API is for mashing up some data and showing it on a map.<br />
<br />
This week I have been busy trying to visualize the race data available from the Danish scout adventure race "Alligatorløbet" 2013. This year they made the race tracking data available online as an Excel file with check-in and check-out times for each team on each checkpoint. What I have done is to animate that information as a little interactive movie that displays team-at-checkpoint intensity as heat maps on Google map.<br />
<br />
It is rather neat, if you ask me, but check it out yourself: <a href="http://www.elfisk.dk/allitrack/CheckpointAnimation.html">http://www.elfisk.dk/allitrack/CheckpointAnimation.html.</a><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<img alt="Heat maps overlay on Google maps" border="0" height="207" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdN5vP6rt3nQWr3_tEw_6G3X3Cr1JuYQn9Ya958eG8Q5I17VFM32yAtWGIrkZa6fjaQSn1BGv5n98Hzbx9qUikMvhyO64mN7IYbj99NTL_eh6Ur1b7J8H857MnkjP-X1VkP3KCe3-aqg/s400/allitrack.png" title="Heat maps overlay on Google maps" width="400" /></div>
<br />
It is implemented like this:<br />
<ol>
<li>All data is normalized to CSV files with one record for each team/checkpoint/arrival-time/departure-time registration.</li>
<li>The CSV files are read by a C# program that transforms it to a Javascript file to be included on the page. This file contains the checkpoint intensity for each time frame sample (5 minutes) along with team names and team scores.</li>
<li>The HTML page loads Google maps and overlays it with a KML file that shows the checkpoints. This KML file was created in Google Earth.</li>
<li>Then the Javascript animation kicks in and updates the heat maps every half second. It also constantly updates the position of the two teams you want to follow during the race.</li>
<li>Add to this some event handlers for interacting with the buttons and you are done :-)</li>
</ol>
You can find the code on <a href="https://github.com/JornWildt/racetrack">GitHub</a> if it has any interest.<br />
<br />
Have fun. Elfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com0tag:blogger.com,1999:blog-7395172357368553438.post-91277450815042976132013-01-04T10:13:00.000+01:002013-03-08T22:10:03.778+01:00JSON-Patch support in Ramone<a href="https://github.com/JornWildt/Ramone">Ramone</a> (my C# web API and REST client) now supports JSON patch documents as defined in <a href="http://tools.ietf.org/html/draft-ietf-appsawg-json-patch-08">http://tools.ietf.org/html/draft-ietf-appsawg-json-patch-08</a>. This is yet a draft standard so Ramone's implementation may need some tweaking later on to work.<br />
<br />
A patch document represents a set of changes to a JSON resource. This can be “add” new value, “remove” value or other similar operations.<br />
<br />
In Ramone we use the class <span style="font-family: "Courier New",Courier,monospace;">JsonPatchDocument</span> to build a patch document. For instance like this:<br />
<br />
<pre class="brush: csharp">JsonPatchDocument patch = new JsonPatchDocument<myresourcetype>();
// Supply path as a string
patch.Replace("/Title", "My new title");
// Supply path as a typed referenc
patch.Replace(d => d.Title, "Another title");
</pre>
<br />
The above example corresponds to this json-patch document (which you can get by calling <span style="font-family: "Courier New",Courier,monospace;">ToString()</span> or <span style="font-family: "Courier New",Courier,monospace;">Write()</span> on the <span style="font-family: "Courier New",Courier,monospace;">patch</span> variable):<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">[</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> { op: "replace", path: "/Title", value: "My new title" },</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> { op: "replace", path: "/Title", value: "Another title" } </span><br />
<span style="font-family: "Courier New",Courier,monospace;">]</span><br />
<br />
The use of lambdas like <span style="font-family: "Courier New",Courier,monospace;">d => d.Title</span> is a simple and type safe way to supply a patch path with respect to a given resource class. Using lambdas like this ensure paths are updated when refactoring variable names in Visual Studio.<br />
<br />
Patch documents can be sent to a server like this:<br />
<br />
<pre class="brush: csharp">JsonPatchDocument patch = ... build patch document ...
Request request = Session.Bind(ResourceUrl);
using (var response = request.Patch(patch))
{
... do stuff with response ...
}</pre>
<br />
Ramone also supports reading and applying patch documents (using the “visitor” pattern). You do although have to implement the actual operations yourself—Ramone cannot do that for you. Here is one example:<br />
<br />
<pre class="brush: csharp">using (TextReader r = ... get reader from some JSON input ...)
{
// Read patch document from input
JsonPatchDocument patch = JsonPatchDocument.Read(r);
// Create instance of patching logic
MyPatchVisitor visitor = new MyPatchVisitor();
// Apply patch document
patch.Apply(visitor);
}
// Implements "visitor" pattern where each patch operation in
// the patch document will be translated to a call to Add(...),
// Replace(...) and so on.
class MyPatchVisitor : JsonPatchDocumentVisitor
{
public override void Add(string path, object value)
{
... implement "add" logic for target data ...
}
}
</pre>
<br />
There is also an optional typed version of the JsonPatchDocumentVisitor which will simplify your operations slightly:<br />
<br />
<pre class="brush: csharp">class MyTypedPatchVisitor : JsonPatchDocumentVisitor<myresourcetype>
{
public override void Add(string path, object value)
{
// Test path against delegate and cast value to int if
// they match. Then execute the action delegate.
IfMatch<int>(r => r.Id, path, value,
v => ... do stuff with integer "v"...);
}
}
</pre>
<br />
<script type="text/javascript">
SyntaxHighlighter.all()
</script>Elfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com6tag:blogger.com,1999:blog-7395172357368553438.post-32696116392957337862013-01-02T22:58:00.002+01:002013-03-08T22:11:16.180+01:00HTTP PUT, PATCH or POST - Partial updates or full replacement?I have recently been working on the write side of a REST service for managing case files. During this work I have gone through a lot of discussions about partial updates versus full updates of resources in a RESTful service - and whether to perform these updates using the HTTP verbs PUT, PATCH or POST.<br />
<br />
In the end we settled on using PATCH for partial updates (with the media type application/json-patch) and PUT for complete updates. That is in itself not so surprising, it is how we got to these decisions that I would like to share.<br />
<br />
<h2>
Example resource model</h2>
For demonstration purposes I will use the representation of a bug report from some fictive bug reporting system. A GET of such a bug report would return the following XML which should be rather self explaining:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"><BugReport><br /> <Id>15</Id><br /> <Title>Program crashes when hitting ctrl-P</Title><br /> <Status>Open</Status><br /> <Responsible id="22"><br /> <Name>John Smith</Name><br /> <Link rel="self" href="{url-to-person}" title="John Smith"/><br /> </Responsible><br /> <Link rel="self" href="{url-to-bug-report}" title="This bug report"/><br /></BugReport></span><br />
<br />
<h2>
Complete updates with PUT or POST</h2>
At first we wanted to use PUT with URL encoded key-value pairs for complete updates. We wanted to make it clear that such updates were <i>idempotent</i> and thus PUT seemed to be a perfect match - it signals full idempotent replacement of the target resource (see for instance <a href="http://www.emergentone.com/blog/http-methods-and-idempotence/">http://www.emergentone.com/blog/http-methods-and-idempotence/</a> for an explanation of "idempotent").<br />
<br />
There is although no reason to PUT all the server-generated stuff like the links and bug report ID when performing an update. But that meant we were not doing a complete update any more - and so we entered the foggy zone of unclear semantics for PUT: is it, or is it not, allowed by the HTTP specification to do such a full-but-somehow-also-partial update using PUT? Some people say yes, other people say no ...<br />
<br />
Due to this uncertainty we gave up using PUT and turned to POST instead. This would not signal idempotency as we wanted, but it would certainly be a legal and standard use of POST.<br />
<br />
<h2>
Partial updates with POST</h2>
In the end it turned out that we could have saved us the trouble of discussing these "quite-but-not-entirely-unlike-partial" updates using PUT because it soon became apparent that "real" partial updates was a much better fit for our use case. The business case for implementing updates in the system was an integration service that transferred changes from one external system into our system - with heavy focus on only transferring <i>changes</i> from A to B.<br />
<br />
We chose the intuitive solution of interpreting missing values (or, rather, null values) as "do not change this property". That would allow the client to POST an update to the Title property only as:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> POST /bug-report-url<br /> Content-Type: application/x-www-form-urlencoded<br /><br /> Title=A%20new%20title</span><br />
<br />
That worked well for the Title property, but how about the Responsible property? It had to be possible to either 1) ignore the Responsible, 2) set it to something, or 3) clear it ... But if a null value meant "ignore" what should then be used for "clear"? We decided to add a new (boolean) field "NoResponsible", so we could clear the responsible with a POST like this:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> POST /bug-report-url<br /> Content-Type: application/x-www-form-urlencoded<br /><br /> NoResponsible=true</span><br />
<br />
Somehow that seemed quite a bit hacked - what other sorts of artificial key-value fields would we need to implement the full solution?<br />
<br />
<h2>
The move to PATCH</h2>
This was surely not going in the right direction, so we started researching partial updates again and looked at the HTTP verb PATCH with the media type application/json-patch (see <a href="http://tools.ietf.org/html/draft-ietf-appsawg-json-patch-08">http://tools.ietf.org/html/draft-ietf-appsawg-json-patch-08</a>). This turned out to be a perfect match for our use case where the integration component could build up a patch document, based on the changes that needs to be transferred, and then apply that to the bug report resource.<br />
<br />
A json-patch document is basically a list of operations to be applied to the target resource. Here is an example of how such a patch document can be used to update the title of our fictive bug report and remove the responsible person at the same time:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> PATCH /bug-report-url<br /> Content-Type: application/json-patch<br /><br /> [<br /> { op: "replace", path: "/Title", value: "New title" },<br /> { op: "remove", path: "/Responsible" }<br /> ]</span><br />
<br />
<h2>
</h2>
<h2>
Back to complete updates using PUT</h2>
PATCH turned out to be perfect for our integration project. So far so good. But another relevant use case could soon be to support some kind of human facing application that works like this:<br />
<br />
1) User opens an existing bug report. <br />
<br />
2) User modifies the bug report.<br />
<br />
3) User saves the new version of the bug report.<br />
<br />
In this case tracking changes for a patch document may not be desirable, so instead we should use the well known pattern of GET resource - modify local representation - PUT result back, with the relevant use of ETag headers etc. to avoid lost updates (see for instance <a href="http://www.w3.org/1999/04/Editing/">http://www.w3.org/1999/04/Editing/</a>).<br />
<br />
My point here is that even though we discarded PUT for partial updates earlier on, doesn't mean that it cannot be done - but you need to do it right with GET+PUT and ETag headers - and that is quite a bit more complex than using PATCH as described above.<br />
<br />
<h2>
json-patch with Ramone</h2>
Since we use <a href="http://soabits.blogspot.dk/2012/04/introducing-ramone-c-library-for-web.html">Ramone</a> internally for our client applications, it was natural to add support for json-patch here - which, not surprisingly, is the topic for my next blog post.<br />
<br />
<h2>
Other approaches (which we found less useful)</h2>
<br />
<h3>
Doing partial updates using PUT with sub-resources</h3>
Another
common way of doing idempotent partial updates using PUT is to represent
certain properties as sub-resources that contains a subset of the main
resource. We could for instance move the Responsible property to a
resource of it's own with only that value and then link to it in the
main resource.<br />
<br />
Such a solution would work, but 1) it would require
lots of boiler plate code to implement, and, 2) more importantly, it
would not support atomic and transactional updates to multiple
properties in one request.<br />
<br />
<h3>
Encoding modified property names in the URL</h3>
Some people has suggested putting the names of the properties to modify in the URL of the request, thus resulting in a new URL which can be updated using a complete PUT. For instance, to update only the Title in our example we could make a PUT like this:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> PUT /bug-report-url;Title</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> Content-Type: application/x-www-form-urlencoded</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<span style="font-family: "Courier New",Courier,monospace;"> Title=New%20title</span><br />
<br />
But this requires the client to build URLs with the assumption of a given URL structure - something to be avoided in hyper media based APIs (since this would couple the client with the server's URL structure).<br />
<br />
Further more, I cannot foresee if it will work for more complex structures, so why not use a standardized approach like json-patch?<br />
<br />
<h3>
Using the XML patch framework</h3>
We might as well have used the XML patch framework (see <a href="http://tools.ietf.org/html/rfc5261">http://tools.ietf.org/html/rfc5261</a>)
instead of json-patch for the patch document. That would certainly be
more in line with the XML representation we use already. The main reason for not
doing this is very simple and pragmatic - we discovered the XML framework too late.<br />
<br />
Should I although choose between the two patch formats today, I would probably choose json-patch again for the simple reason that it matches our data better - XML is only a thin wrapper around the public data model and JSON might in fact have been a better choice. The XML patch framework is quite over-engineered for the kind of data we serve.<br />
<br />
<h2>
UPDATE (January 3, 2013): Is this RPC?</h2>
Someone on Stackoverflow commented that this "sending commands" (via patch) is more RPC style than REST. Let me try to cover that issue ...<br />
<br />
If you define RPC as sending commands to a server then any and all HTTP operations are RPC calls by definition - whether you GET a resource, PUT a new representation or DELETE it again - each of them consist of a sending a command (verb) GET/PUT/DELETE etc. and a optional payload. It just happens that the HTTP working group (or who ever it is) has introduced a new verb PATCH which allows clients to do partial updates to a resource.<br />
<br />
If anything else than sending the complete representation to the server is considered RPC style, then, by definition, partial updates cannot be RESTful. One can choose to have this point of view, but the people behind the web infrastructure says differently - and has thus defined a new verb for this purpose.<br />
<br />
RPC is more about tunneling method calls through HTTP in a way that is invisible to intermediaries on the web - for instance using SOAP to wrap method names and parameters. These operations are "invisible" since there are no standards defining the methods and parameters inside the payload.<br />
<br />
Compare this to PATCH with the media type application/json-patch - the intention of the operation is clearly visible to any intermediary on the web since the verb PATCH has a well defined meaning and the payload is encoded in another well defined public available format owned by common authority on the web (IETF). The net result is full visibility for everybody and no application specific secret semantics.<br />
<br />
REST is also about "serendipitous reuse" which is exactly what PATCH with application/json-patch is - reusing an existing standard instead of inventing application specific protocols that do more or less the same.Elfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com1tag:blogger.com,1999:blog-7395172357368553438.post-52166085269888351052012-04-19T22:55:00.000+02:002012-04-20T22:05:29.860+02:00Ramone: Media types and codecsOne of the basic building blocks of REST is the concept of a media type - the file format used to represent a resource on the web. Media types comes in many different flavours - images, PDF, vCard, XML, JSON, spreadsheets and so on, each of them having their own specific formats and capabilities. If you haven't done it already then take a look at <a href="http://soabits.blogspot.com/2012/04/restful-resources-are-not-typed.html">my previous post</a> where I go deeper into details about media types.<br />
<br />
Media types are considered first class citizens of <a href="https://github.com/JornWildt/Ramone">Ramone</a>, my C# library for consuming web APIs and RESTful services on the web - just like the uniform interface (GET/POST/PUT/...) and resource identifiers (URLs) - and in this post I will show how to work with different kinds of media types.<br />
<br />
<h3>
Codecs</h3>
A codec is a class that translates to and from the file format on the wire and some kind of internal representation in C#. To do so it must first implement either <span style="font-family: "Courier New",Courier,monospace;">IMediaTypeWriter</span>, <span style="font-family: "Courier New",Courier,monospace;">IMediaTypeReader</span> or both and then register with the current codec manager such that Ramone will be able to find it.<br />
<br />
The codec interfaces are rather simple:<br />
<br />
<pre class="brush: csharp"> public interface IMediaTypeCodec
{
object CodecArgument { get; set; }
}
public interface IMediaTypeWriter : IMediaTypeCodec
{
void WriteTo(WriterContext context);
}
public interface IMediaTypeReader : IMediaTypeCodec
{
object ReadFrom(ReaderContext context);
}
</pre>
<br />
The context parameter contains references to the current session, the data stream, <span style="font-family: "Courier New",Courier,monospace;">HTTPRequest</span>, <span style="font-family: "Courier New",Courier,monospace;">HTTPResponse </span>and others that are available for the codec.<br />
<br />
<h3>
Decoding an HTML micro format</h3>
One example of a codec is the BlogCodec from <a href="https://github.com/JornWildt/Ramone/blob/master/Ramone.Tests/Blog/Codecs/Html/BlogCodec_Html.cs">Ramone's test library</a>. This codec demonstrates how to decode a (non-standard) micro format from an HTML page that shows a blog listing (see <a href="https://gist.github.com/2305777">https://gist.github.com/2305777</a> for the actual HTML).<br />
<br />
<pre class="brush: csharp"> public class BaseCodec_Html : TextCodecBase<Resources.Blog>
{
// This method is from TextCodecBase which has wrapped the binary input stream in a TextReader
// using the charset encoding stated by the client's request headers.
protected override Resources.Blog ReadFrom(TextReader reader, ReaderContext context)
{
// Using HtmlDocument from HtmlAgilityPack
HtmlDocument doc = new HtmlDocument();
doc.Load(reader);
return ReadFromHtml(doc, context);
}
protected Resources.Blog ReadFromHtml(HtmlDocument html, ReaderContext context)
{
HtmlNode doc = html.DocumentNode;
List<Resources.Blog.Post> posts = new List<Resources.Blog.Post>();
// Scan through HTML and look for "class" attributes identifying values
foreach (HtmlNode postNode in doc.SelectNodes(@"//div[@class=""post""]"))
{
HtmlNode title = postNode.SelectNodes(@".//*[@class=""post-title""]").First();
HtmlNode content = postNode.SelectNodes(@".//*[@class=""post-content""]").First();
List<Anchor> links = new List<Anchor>(postNode.Anchors(context.Response.ResponseUri));
posts.Add(new Resources.Blog.Post
{
Title = title.InnerText,
Text = content.InnerText,
Links = links
});
}
// Extract all HTML anchors together with <head> links and store them as ILink instances
List<ILink> blogLinks = new List<ILink>(doc.Anchors(context.Response.ResponseUri).Cast<ILink>().Union(doc.Links(context.Response.ResponseUri)));
// Create and return an object that represents the data extracted from the HTML
Resources.Blog blog = new Resources.Blog()
{
Title = doc.SelectNodes(@".//*[@class=""blog-title""]").First().InnerText,
Posts = posts,
Links = blogLinks
};
return blog;
}
// This method is also from TextCodecBase, but is not used (since we do not write HTML)
protected override void WriteTo(T item, System.IO.TextWriter writer, WriterContext context)
{
throw new NotImplementedException();
}
}
</pre>
<br />
You can read a bit more about using hyper media links in <a href="http://soabits.blogspot.com/2012/04/ramone-consuming-hyper-media-rest.html">another of my earlier posts</a>.<br />
<br />
<h3>
Other codec examples</h3>
Other examples of codecs could be:<br />
<ul>
<li>Decoding cooking recipe data from XML or JSON.</li>
<li>Decoding binary image data.</li>
<li>Decoding CSV into tabular data.</li>
<li>Decoding and writing vCard information.</li>
</ul>
<br />
<h3>
Built-in generic codecs</h3>
All of the previous codecs has been "typed" in the sense that they decode response data into a typed object with the specific properties needed. But Ramone has also built-in support for various generic formats such as XML, JSON and HTML.<br />
<br />
Here is an example use of the XML codec which decodes into C#'s XML DOM class XmlDocument:<br />
<br />
<pre class="brush: csharp"> Request req = Session.Bind("... some url ...);
XmlDocument doc = req.Get<XmlDocument>().Body;
</pre>
<br />
The JSON codec can do some nifty stuff with C# dynamics:<br />
<br />
<pre class="brush: csharp"> Request req = Session.Bind("... some URL for cat data ...");
dynamic cat = req.Accept("application/json").Get().Body;
Assert.IsNotNull(cat);
Assert.AreEqual("Ramstein", cat.Name);
</pre>
<br />
<br />
<h3>
Advantages of typed codecs versus generic codecs</h3>
By working with typed codecs you gain a few advantages over the generic ones:<br />
<ol>
<li>The application code is completely decoupled from the wire format. This gives you the ability to work with different wire formats without changing the application code, e.g., decoding both JSON, XML, and vCard into the same internal representation.</li>
<li>It results in more readable application code.</li>
<li>It makes the parsing code reusable across difference pieces of application code.</li>
</ol>
The downside of codecs is that you have to write a few more pieces of boilerplate code to create and register them. You also have to implement the client side representation of the resource as a specific class. But in the end it all pays off, in my opinion, and yields more readable and maintainable code.<br />
<br />
<h3>
Update (20/04/2012): Codec Manager </h3>
I forgot to show how codecs can be registered with either the current service (more on services at a later time); Codecs must be registered with Ramone, otherwise they will simply be ignored. To do so you first grab a reference to ICodecManager and then call AddCodec(...):<br />
<br />
<pre class="brush: csharp"> ICodecManager cm = MyService.CodecManager;
cm.AddCodec<MyClass, MyCodec>(MyMediaType);
</pre>
<br />
Here <span style="font-family: "Courier New",Courier,monospace;">MyClass</span> is the type of object returned or written by the codec, MyCodec is the type of the codec and MyMediaType is the media type id string, e.g., "application/vnd.mytype+xml".<br />
<br />
<br />
Ramone can be downloaded from <a href="https://github.com/JornWildt/Ramone">https://github.com/JornWildt/Ramone</a><br />
<br />
<br />
<script type="text/javascript">
SyntaxHighlighter.all()
</script>Elfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com0tag:blogger.com,1999:blog-7395172357368553438.post-15797951722523317322012-04-18T08:44:00.001+02:002012-04-19T22:56:24.064+02:00RESTful resources are not typedI have always been puzzled by this quote from Roy Fielding [1]:<br />
<br />
<i> A REST API should never have “typed” resources that are significant to the client</i><br />
<br />
How on earth is a client supposed to be able to do anything with a resource if it is not allowed to know the type of it?<br />
<br />
The context here is a REST API that exposes business data from a certain business domain. This could be e-procurement, biochemistry or what ever else you could think of. But lets go with the e-procurement example and imagine a web shop that exposes catalogues, sales orders, customers and so on.<br />
<br />
A machine-driven client may be interacting with this API in order to automatically buy something, lookup customers, gather sales statistics and what not. The point is - at a given time the client has a specific task to solve. Lets go with an example like this; "Get information about a certain sales order and print a letter to its customer".<br />
<br />
Now, how can the client get the address of the customer unless it knows it is, well, of type "Customer" - and thus having a well known set of properties like "Name", "Birthdate", "Customer number", "Billing address" and so on?<br />
<br />
Down the rabbit hole we go ...<br />
<br />
<h3>
Typed resources is a misconception</h3>
It is extremely easy to fall into the trap of thinking of resources as having specific types. A hyper link with the link relation type "customer" would surely point to a "Person" or perhaps a more generic "Contact" resource - right? Just like the resource identified by http://example.com/people/Pete must certainly be a "Person" too - right?<br />
<br />
But what if the "customer" hyper link referred to <a href="http://www.cph.dk/">http://www.cph.dk</a>, or if you were presented with the URL http://example.com/fgx27-yw - what could you then conclude about types? Nothing. At the URL www.cph.dk we find a homepage from Copenhagen airport - that is not something which has a business domain type in traditional understanding - and fgx27-yw should be meaningless to anyone.<br />
<br />
From a client's point of view, there is no such thing as a typed resource.<br />
<br />
<h3>
Resource views? No. Facets? No. Capabilities? No. Ah, yes, Media types!</h3>
Instead of talking about "types" we can reframe our thinking and talk about various views or facets of a resource - or perhaps the capabilities it offers to a client interacting with it. Just make sure you quit thinking of what the resource "is" and think of what it can do for you instead.<br />
<br />
Example; a client that has been given the task of printing a letter to the customer at some {Customer} URL can ask the resource at that URL to return a VCard such that the client can get a mailing address - and it couldn't care less if the resource was a cat, city, or movie actor, as long as it it was able to return a VCard when asked for it.<br />
<br />
Such a view of a resource is called a <i>representation</i> and the representation's format is the <i>media type</i> and asking for a specific media type is called <i>content negotiation</i>.<br />
<br />
<h3>
Media types</h3>
Lets take a look at some other media types and see how they can be applied without knowing the type of a given resource.<br />
<ol>
<li>A client following a link relation "news" may ask for application/atom+xml for a news feed.</li>
<li>A client following a link "destination" might ask for application/gml+xml to get a geographical model of the destination.</li>
<li>A client following a link "contact" might ask for text/vcard to get in contact with, well, the contact. It could also ask for application/vcard+xml in addition if it supported both formats. The server would then return one of them, depending on what it supported.</li>
</ol>
Again, there is absolutely no mentioning of the type of the resource. As an experiment, try to imagine the resource at www.cph.dk as the target of the above links ... works quite well, right? All the client need is an understanding of how the next resource in the application flow is related to the current one - it does not need to know what that resource is as long as it can ask for a representation that solves the given task of the client.<br />
<br />
<h3>
Domain specific media types</h3>
Now back to the original task of getting business domain specific details from a resource - without knowing the type of it. How can we get Name, Birthdate, Order No. and so on from the resource? What is needed is a domain specific media type - a media type which is specialized to a specific business domain.<br />
<br />
Example: suppose a client needs the details of a sales order for a specific customer. All it needs to do is<br />
<ol>
<li>follow a link to the resource that someone else says is the sales order for the customer, and </li>
<li>ask for a media type that covers the sales order details needed.</li>
</ol>
<br />
This of course assumes that someone has defined a suitable domain specific media type that can be used this scenario. If no such thing exists then go ahead and design your own. Just make sure you register it!<br />
<br />
<h3>
Media type standardization</h3>
A media type must be registered with some authority - otherwise there is nothing for the client and server to agree on when communicating. On the big world wide web this is the job of IANA [2]. But it might as well be some central authority inside a big enterprise corporation - as long as (all) the client(s) and the server can agree on it.<br />
<br />
Some business domains can be rather difficult to standardize and in these cases IANA may not be the place to go. Luckily there is nothing inherently wrong about creating private media types - as long as they stay inside the enterprise.<br />
<br />
<h3>
More capabilities</h3>
A media type is not restricted to only define the "wire format" or "serialization format" of a resource. It also defines link relations like "customer", "orders" or "owner" as well as how to interact with the resource. The typical interaction is navigation by following a URL - but media types can also define how to write data back and update resources aka "forms" handling. In this way media types goes way beyond simple understanding of the type of a resource.<br />
<br />
Media type design is a whole art in itself that Mike Amundsen has written quite a lot about [3].<br />
<br />
<h3>
Links</h3>
[1] REST APIs must be hypertext-driven - <a href="http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven">http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven</a><br />
<br />
[2] MIME Media Types - <a href="http://www.iana.org/assignments/media-types/index.html">http://www.iana.org/assignments/media-types/index.html</a><br />
<br />
[3] Hypermedia affordances - <a href="http://amundsen.com/blog/archives/1109">http://amundsen.com/blog/archives/1109</a><br />
<br />
<br />Elfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com4tag:blogger.com,1999:blog-7395172357368553438.post-51046964219140354842012-04-04T23:59:00.000+02:002012-05-21T21:00:31.641+02:00Ramone: Consuming Hyper-Media REST Services in C#Hyper-media controls are one of the core features of the World Wide Web as we know it today - without them there would be no way to link web pages and no way to interact with them through web forms. One of the REST constraints is the famous HATEOAS - "Hyper-media As The Engine Of Application State" and in order to consume RESTful web services we should make hyper-media controls first class citizens of the client interface we work with.<br />
<br />
Hyper-media controls can be classified in different ways (see for instance <a href="http://amundsen.com/hypermedia/hfactor/">Mike Amundsen's classification</a>), but in Ramone we distinguish between the two well-known hyper-media elements; links and key/value forms as we know them from HTML, ATOM and more. These are represented with the following C# interface definitions:<br />
<br />
<pre class="brush: csharp"> public interface ILink
{
Uri HRef { get; }
string Title { get; }
IEnumerable<string> RelationTypes { get; }
MediaType MediaType { get; }
}
public interface IKeyValueForm
{
IKeyValueForm Value(string key, object value);
IKeyValueForm Value(object value);
Request Bind(string button = null);
}
</pre>
<br />
The idea is to use some C# extension method to get an instance of one of the above interface from the resource representation in the current response. This means in HTML there are methods for extracting <a> elements, <link> elements and <form> elements and return them as instances of the hyper-media interfaces. As more media-types are added to Ramone so will more similar extension methods be added - this can be done as add-ons without touching the core code.<br />
<br />
The following example is from <a href="https://github.com/JornWildt/Ramone/tree/master/Ramone.Tests/Blog">Ramone's Blog test collection</a> (please take a closer look at the Github code since it includes some more explanations). This example shows how to follow an "author" link from an HTML page that shows a list of blog entries. The HTML contains (non-standard) micro formats for identifying various elements of the blog (post, author, title).<br />
<br />
The actual HTML can be found here: <a href="https://gist.github.com/2305777#file_ramone_blog_list_html.html">https://gist.github.com/2305777#file_ramone_blog_list_html.html</a> (and the HTML for an author can be found here: <a href="https://gist.github.com/2305988#file_gistfile1.html">https://gist.github.com/2305988#file_gistfile1.html</a>)<br />
<br />
Here we go:<br />
<br />
<pre class="brush: csharp">
// Create request by binding well known path to session's base URL
Request blogRequest = Session.Bind(BlogRootPath);
// GET blog HTML represented as an HtmlDocument (from Html Agility Pack)
Response<HtmlDocument> blog = blogRequest.Get<HtmlDocument>();
// Select first HTML anchor node with rel="author" as a anchor link.
// This uses HtmlDocument specific extension methods to convert from anchor to ILink
ILink authorLink = blog.Body.DocumentNode.SelectNodes(@"//a[@rel=""author""]").First().Anchor(blog);
// Follow author link and get HTML document representing the author
HtmlDocument author = authorLink.Follow(Session).Get<htmldocument>().Body;
// Check e-mail of author
HtmlNode email = author.DocumentNode.SelectNodes(@"//a[@rel=""email""]").First();
Assert.AreEqual("pp@ramonerest.dk", email.Attributes["href"].Value);
</pre>
<br />
The two important pieces of code in this example are the <span style="font-family: "Courier New",Courier,monospace;">.Anchor(...)</span> and <span style="font-family: "Courier New",Courier,monospace;">.Follow(...)</span> parts that respectively instantiates a link and follows it.<br />
<br />
Forms can be loaded and submitted in a similar fashion.<br />
<br />
Hopefully this gives an understandable introduction to working with hyper-media in Ramone - and, even more important, I hope you find it relatively intuitive and easy to work with.<br />
<script type="text/javascript">
SyntaxHighlighter.all()
</script>Elfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com0tag:blogger.com,1999:blog-7395172357368553438.post-15597483949106848972012-04-03T00:08:00.001+02:002012-04-04T22:56:18.470+02:00Introducing the Ramone C# Library for Web API ClientsRamone is a C# library for client applications that want to access some of the many web APIs available today.<br />
<br />
What Ramone does is to wrap a lot of the bread and butter code into a library for easy reuse (see <a href="http://soabits.blogspot.com/2012/04/consuming-web-apis-in-c-with-ramone.html">this previous post</a> for my motivation). Out of the box it offers XML, JSON, ATOM and HTML formatters, URL templating and parameter binding, hyper-media traversal, form handling, and has HTTPs "Uniform Interface" as first class citizen in the interface.<br />
<br />
With Ramone you won't see methods like GetOrders(...) or SaveUserData(...). You will instead be working with URLs that points to resources and use HTTPs uniform interface to interact with these. This means for instance that GetOrders(...) may become OrderRequest.Get() and SaveUserData(...) becomes UserRequest.Put(...).<br />
<br />
The smallest example I can think of is this little four-liner that fetches the timeline of a specific Twitter account:<br />
<br />
<pre class="brush: csharp">// All interaction with Ramone goes through a session (which defines a service base URL)
ISession session = RamoneConfiguration.NewSession(new Uri("https://api.twitter.com"));
// Creating a Ramone request by binding variables to a URI template
Request request = session.Bind(UserTimeLineTemplate, new { screen_name = ScreenName, count = 2 });
// GET response from Twitter
Response response = request.Get();
// Extract payload as a C# dynamic created from the JSON response
dynamic timeline = response.Body;</pre>
<br />
What happens here is:<br />
<ol>
<li>A session is created. This carries information about the base URL for template binding and keeps track of client state (cookies among other things).</li>
<li>A request is created by binding a URL template with parameters. Input is a relative URL template path ("/1/statuses/user_timeline.json?screen_name={screen_name}&count={count}") and an anonymous parameter object. Output is a request for an absolute URI.</li>
<li>One of the operations from HTTPs Uniform Interface is called (GET) and the response is recorded.</li>
<li>Ramone checks the returned media-type, sees it is application/json, and converts it to a dynamic C# object with late binding of the returned values.</li>
</ol>
The code for iterating through the timeline and printing status updates is plain C#:<br />
<br />
<br />
<pre class="brush: csharp">Console.WriteLine("This is the timeline for {0}:", ScreenName);
Console.WriteLine();
foreach (dynamic tweet in timeline)
{
Console.WriteLine("* {0}.", tweet.text);
Console.WriteLine();
} </pre>
<br />
A few things worth noticing:<br />
<ol>
<li>Twitter may not be the best REST example out there, but it is well understood by many people and is a web API which is quite easy to work with.</li>
<li>Ramone does not only handle JSON. It works with a repository of parsers and formatters (called codecs) which are responsible for handling the various formats Ramone may encounter.</li>
</ol>
<br />
You can find a complete Twitter demo at Github: <a href="https://github.com/JornWildt/Ramone/tree/master/TwitterDemo/TwitterDemo">https://github.com/JornWildt/Ramone/tree/master/TwitterDemo/TwitterDemo</a>. This demo shows both how to fetch a timeline as well as how to post status updates with OAuth1 authentication.<br />
<br />
All source code is available on Github: <a href="https://github.com/JornWildt/Ramone">https://github.com/JornWildt/Ramone</a>. At the time of writing there are no official binaries released so you will have to download the code from Github and compile it yourself.<br />
<br />
Later on I will write about how Ramone helps you consuming more RESTful hyper-media based services by making links, link relations and forms first class citizens of the interface. For now you can check the blog example at Github: <a href="https://github.com/JornWildt/Ramone/tree/master/Ramone.Tests/Blog">https://github.com/JornWildt/Ramone/tree/master/Ramone.Tests/Blog</a><br />
<br />
Have fun!<br />
<script type="text/javascript">
SyntaxHighlighter.all()
</script>Elfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com0tag:blogger.com,1999:blog-7395172357368553438.post-84476467168342598392012-04-01T22:07:00.000+02:002012-04-05T00:23:52.086+02:00Consuming Web APIs in C# with RamoneThe World Wide Web is a fascinating place with its enormous amount of information right at your hand. It is even more fascinating how relatively simple the upper application layer is: we have resources, URLs - Uniform Resource Locators which we can dereference, a small set of methods to operate on said resources - GET/POST/PUT/DELETE (and then some), and media-types for identifying how those resources are represented. <br />
<br />
Roy T. Fielding wrapped it all up in his dissertation "Architectural Styles and the Design of Network-based Software Architectures" where he coined the term REST - "Representational State Transfer". Somehow this term caught on and developers, who was trying to find something more simple than SOAP (over HTTP), started looking at REST for machine-to-machine APIs on the web. Soon we got REST APIs - and plenty of hype around them. Whether or not these APIs are actually RESTful is an ongoing debate, but it should be safe to call them "Web APIs" no matter how they present themself, so I will stick to that term for now.<br />
<br />
Today there are thousands of public web APIs available for anyone to access - just take a look at <a href="http://www.programmableweb.com/">http://www.programmableweb.com/</a> - at the time of writing it boasts some 5529 APIs. And those are just the public APIs - in addition to this there's probably many many more in closed enterprises and partnerships.<br />
<br />
When I first started working with web APIs in C# I soon got a bit tired of the rituals you have to go through in order to work with HTTP in .NET. Usually I have a URL to some resource, the name of a method I want to invoke, maybe a payload (a C# object instance), and then a returned resource representation (also an object instance). So I should be able to write something like this:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> Uri url = new Uri("...");</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> Payload p = new Payload(...);</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> Resource r = url.<Method>(p);</span><br />
<br />
You can do this in a few lines of code with .NET's WebRequest class ... except that you get absolutely no help when it comes to serializing Payload and Resource data. And what format are those classes actually serialized as? Is it XML, JSON, HTML or something else?<br />
<br />
The next problem with low level HTTP access using WebRequest is handling of authorization using either HTTP Basic, OAuth or something similar - these are standardized authorization mechanisms, but you need to add them on top of HTTP yourself.<br />
<br />
So there is a lot of bread and butter code for you to write before you can get up and running with your first web API. Luckily it is all based on the uniform interface, URLs and media-types - something that can be wrapped into a framework and lower the entry barrier for consuming web APIs - whether they are RESTful or not.<br />
<br />
That is what Ramone is all about - making it easy to consume web APIs in C#. Interacting with the web should be as easy going as, well, hmm, Ramone from Pixar's movie <a href="http://en.wikipedia.org/wiki/Cars_%28film%29">Cars</a>, from which I got the name.<br />
<br />
Hopy you enjoy it! More posts will follow, showing how to get started with Ramone.<br />
<br />
All source code is available on Github: <a href="https://github.com/JornWildt/Ramone">https://github.com/JornWildt/Ramone</a> (distributed under the MIT license).<br />
<br />
Happy hacking, Jørn WildtElfiskhttp://www.blogger.com/profile/01091018516358987653noreply@blogger.com1