In my previous blog post I introduced "macaroons" - a new technique for creating authorization credentials.
But I even went a bit further and created a C# implementation of macaroons called Macaroons.Net. You can find it on GitHub at https://github.com/JornWildt/Macaroons.Net together with examples and documentation.
Have fun.
fredag, december 19, 2014
Macaroons authorization credentials, better than cookies
This was the year I first heard of the term "macaroon" used in the context of the web, authorization frameworks and other crypto stuff. I had absolutely no idea of what it was, but "the web" said it was good, so I got curious and did some studying. It turned out to be a technique for creating authorization tokens with some very interesting properties:
Here is a small scenario illustrating the above features:
The technology for this flow is not yet perfect as there are some interoperability problems regarding how to encode third party requirements like "being authorized as Cecilia @ Twitter". But the underlying crypto stuff is solid and published in the paper "Macaroons: Cookies with Contextual Caveats for Decentralized Authorization in the Cloud" by Arnar Birgisson, Joe Gibbs Politz, Úlfar Erlingsson, Ankur Taly, Michael Vrable and Mark Lentczner.
Personally I think this little piece of technology has a great potential for creating distributed system.
- Proof carrying: a macaroon carries its own proof of authorization, cryptographically secured.
- Delegation: a macaroon can be given to another user who can then act on your behalf.
- Attenuation: any user can further restrict (attenuate) the authorization before sharing the macaroon with others.
- Distributed authorization: any user can require authorization by other services before using a macaroon.
Here is a small scenario illustrating the above features:
- Alice wants to share a set of images on a photo sharing website. Lets call it "Phlocker" for now.
- Alice asks Phlocker to create a macaroon which enables access to exactly those images.
- Alice sends the macaroon to Bob.
- Bob receives the macaroon, most likely as part of a URL to Phlocker, and he uses it to access the images.
- Bob decides to share exactly one of the images with Cecilia. So Bob creates a new macaroon from the original macaroon without involving Phlocker at all. This new macaroon "attenuates" the original macaroon and restricts it to the single image.
- Bob does not want Cecilia to share the image with others, so he attenuates the macaroon even further by adding the requirement that only Cecilia, as identified by her Twitter account, should be allowed to use the macaroon.
- Bob sends the macaroon to Cecilia.
- Cecilia receives the macaroon and sees the requirement of being logged into Twitter as Cecilia.
- Cecilia interacts with Twitter to prove her identity and Twitter issues a discharge macaroon to Cecilia. This is all done without Twitter knowing why Cecilia needs the authorization.
- Cecilia prepares the discharge macaroon for use at Phlocker and sends it together with the original macaroon she received from Bob.
- Phlocker checks Cecilia's macaroon from Bob, recognizes the Twitter identity requirement and verifies it with the discharge macaroon.
- Cecilia gets to see the single image without being able to see the other images that Alice originally authorized Bob to access.
The technology for this flow is not yet perfect as there are some interoperability problems regarding how to encode third party requirements like "being authorized as Cecilia @ Twitter". But the underlying crypto stuff is solid and published in the paper "Macaroons: Cookies with Contextual Caveats for Decentralized Authorization in the Cloud" by Arnar Birgisson, Joe Gibbs Politz, Úlfar Erlingsson, Ankur Taly, Michael Vrable and Mark Lentczner.
Personally I think this little piece of technology has a great potential for creating distributed system.
lørdag, august 02, 2014
Take away from the 2014 API-Craft conference in Detroit
Last week I attended the 2014 API-Craft conference in Detroit - and I had a great time! Met lots of interesting people, had great conversations and learned a lot about what other people make out there.
Some of the stuff I saw and listen to in Detroit made a few pieces of puzzle click together in my brain. First there was Z who reminded me that we still don't have any good description language for documenting resource oriented hypermedia APIs. I've heard that before and been wondering how it would look like.
Then I talked with Sergey and Dmitri who showed me the resource diagram for their API - this one reminded me so much of UML diagrams that I think we should look into that for inspiration.
I was also at the "Documenting APIs" session where I suddenly found myself talking about our own API documentation - which is written in Word (and that's okay, kind'a). It made me realize that, at its core, such a documentation consists of resource descriptions, link relation descriptions, action descriptions and of course some kind of overview, introduction and tutorial sections.
Someone asked us to explain what a hypermedia API is in one sentence. I came up with "That is an API which can be documented without writing one single absolute URL or relative URL path". I don't exactly think that will catch on, especially not outside this community, but I do believe it is true - if you need to explain URL structures then you are missing something. Unfortunately that one sentence doesn't tell you how to actually implement such an API and as such it is really not very useful.
[I'm pretty sure there is a good Yoda quote to put in here ... like, you now ... "The links with you very strong are here, Luke". Anyone got a better one?]
Dave introduced me to his JSON based description language for APIs and it made me realize that you can do awesome things even in a dead simple format like JSON. Why having to state <twice>everything</twice> when you can do { "single": "everything" }.
Then there were people talking about resources and state machines (thanks again Z). Personally I don't believe much in that analogy - it kind of confuses the graph of resource relationships with the graph of a way-too-simple-never-to-be-found-in-real-life state machine. Z did try to convince me about this but never succeeded - maybe because I am highly colored by my PhD thesis on automatic verification of obnoxiously great state machines :-)
We also got a great deal around API explorers/browsers of various kinds. It seems like every media type have such one now a days - and it makes great sense to me. I truly believe in the power it has when it comes to learning a new API. Thanks to Kristopher Kleva for his story about links, actions and the Sirene browser (you really should blog about that experience!).
I also met Pete Gamache who is the guy behind HyperResource - unfortunately we never really spoke together. He's hypermedia client for Ruby is an interesting solution for interacting with hypermedia APIs in code. Its not exactly new as the same kind of ideas exists in the older tool Restfulie (for Ruby) and my own Ramone library (for C#) - but none the less it shows that there are something worth pursuing here.
All the talks about tooling, SDKs and object models for API resources made me want to work a bit on Ramone and Mason again. It could be cool to make a C# hypermedia resource representation that would allow the client developer to follow links and execute actions in an easy and intuitive way - while at the same time be able to hide some of the stuff that complicates client code in order to make it long lasting and handle (some types of) API changes.
So, thanks to all of you - also those of you that I have forgotten to mention here and of course the people at Apigee that made all of this possible!
Some of the stuff I saw and listen to in Detroit made a few pieces of puzzle click together in my brain. First there was Z who reminded me that we still don't have any good description language for documenting resource oriented hypermedia APIs. I've heard that before and been wondering how it would look like.
Then I talked with Sergey and Dmitri who showed me the resource diagram for their API - this one reminded me so much of UML diagrams that I think we should look into that for inspiration.
I was also at the "Documenting APIs" session where I suddenly found myself talking about our own API documentation - which is written in Word (and that's okay, kind'a). It made me realize that, at its core, such a documentation consists of resource descriptions, link relation descriptions, action descriptions and of course some kind of overview, introduction and tutorial sections.
Someone asked us to explain what a hypermedia API is in one sentence. I came up with "That is an API which can be documented without writing one single absolute URL or relative URL path". I don't exactly think that will catch on, especially not outside this community, but I do believe it is true - if you need to explain URL structures then you are missing something. Unfortunately that one sentence doesn't tell you how to actually implement such an API and as such it is really not very useful.
[I'm pretty sure there is a good Yoda quote to put in here ... like, you now ... "The links with you very strong are here, Luke". Anyone got a better one?]
Dave introduced me to his JSON based description language for APIs and it made me realize that you can do awesome things even in a dead simple format like JSON. Why having to state <twice>everything</twice> when you can do { "single": "everything" }.
Then there were people talking about resources and state machines (thanks again Z). Personally I don't believe much in that analogy - it kind of confuses the graph of resource relationships with the graph of a way-too-simple-never-to-be-found-in-real-life state machine. Z did try to convince me about this but never succeeded - maybe because I am highly colored by my PhD thesis on automatic verification of obnoxiously great state machines :-)
We also got a great deal around API explorers/browsers of various kinds. It seems like every media type have such one now a days - and it makes great sense to me. I truly believe in the power it has when it comes to learning a new API. Thanks to Kristopher Kleva for his story about links, actions and the Sirene browser (you really should blog about that experience!).
I also met Pete Gamache who is the guy behind HyperResource - unfortunately we never really spoke together. He's hypermedia client for Ruby is an interesting solution for interacting with hypermedia APIs in code. Its not exactly new as the same kind of ideas exists in the older tool Restfulie (for Ruby) and my own Ramone library (for C#) - but none the less it shows that there are something worth pursuing here.
All the talks about tooling, SDKs and object models for API resources made me want to work a bit on Ramone and Mason again. It could be cool to make a C# hypermedia resource representation that would allow the client developer to follow links and execute actions in an easy and intuitive way - while at the same time be able to hide some of the stuff that complicates client code in order to make it long lasting and handle (some types of) API changes.
So, thanks to all of you - also those of you that I have forgotten to mention here and of course the people at Apigee that made all of this possible!
torsdag, marts 27, 2014
Modelling a Shipment example as a hypermedia service with Mason
Yesterday I was attending the "RAML" workshop at the API Strategy konference. In this workshop the speakers introduced a very simple little Web API; It allowed a customer to GET a (shipment) quote and afterwards create (POST) an actual shipment request based on the quote.
I decided that it could be fun to hypermedia-ize the example and show how it could be represented using the media type Mason. So here we go :-)
The first step is to ask for a quote; the customer has a package of some sort and needs to ship it from A to B, so he access the shipment service and asks for a quote (a price) given the size, weight, origin and destination for the package.
In the original example you could ask for a quote by issuing a GET request to /quote. But I believe that asking for a quote would result in a concrete quote being created and stored in the system as a separate resource to access later on, either by the customer or by the customer service of the shipment company. So I would rather go for a POST of a quote request followed by a redirect to the newly created quote.
At this point we could either document how to POST such a quote - or we could tell the client how to do it using hypermedia controls - and obviously I would go for the later. So lets ask the service for instructions and issue a GET /quote request. The output is a Mason document with suitable hypermedia controls embedded in it:
{
"@namespaces":
{
"myth":
{
"name": "http://mythological-shipment.com/api/rel-types"
}
},
"@actions":
{
"myth:quote":
{
"type": "json",
"method": "POST",
"href": "http://mythological-shipment.com/api/quote",
"title": "Ask for a quote",
"description": "Ask for a quote by posting package details. Weight is in kilograms, volume in cubic decimeters, origin and destination must be known identifiers for airports.",
"schemaUrl": "... URL to JSON schema describing the request ..."
}
}
}
The client reads this action specification, encodes the package details in JSON and POST it to the URL of the "href" property. As a result the service creates a new quote resource and redirects the client to it:
Request:
POST http://mythological-shipment.com/api/quote HTTP/1.1
content-type: application/json
{
"weight": 2.3,
"volume": 4,
"origin": "CPH",
"destination": "AMS"
}
Response:
201 Created
Location: http://mythological-shipment.com/api/quotes/myqo-129-gyh
Now the client can GET the newly created quote to get further instructions of how to accept the quote. The result is again a Mason representation of the quote itself plus hypermedia controls for accepting the quote:
{
"id": "myqo-129-gyh",
"weight": 2.3,
"volume: 4,
"origin": "CPH",
"destination": "AMS":
"price": 12,
"@links":
{
"self":
{
"href": "http://mythological-shipment.com/api/quotes/myqo-129-gyh"
}
},
"@actions":
{
"myth:accept-quote":
{
"type": "POST",
"href": "http://mythological-shipment.com/api/quotes/myqo-129-gyh/state",
"template":
{
"accepted": "yes"
}
}
}
}
As you can see the quote has a "self" link identifying the location of the quote resource. It also have a "accept-quote" action that instructs the client about how to accept the quote. In this case all the client has to do is to POST a predefined JSON value to http://mythological-shipment.com/api/quotes/myqo-129-gyh/state.
The result of accepting a quote is that it is converted to a sales order (in lack of better domain understanding - there's probably a better word for it). So the accept-quote operation results in a redirect to the newly create sales order which the client can GET:
{
"id": "myqo-129-gyh",
"weight": 2.3,
"volume: 4,
"origin": "CPH",
"destination": "AMS":
"price": 12,
"@links":
{
"self":
{
"href": "http://mythological-shipment.com/api/orders/myqo-129-gyh"
},
"myth:quote":
{
"href": "http://mythological-shipment.com/api/quotes/myqo-129-gyh"
},
"myth:shipment-label":
{
"href": "http://mythological-shipment.com/api/quotes/myqo-129-gyh/label",
"type": "application/pdf"
}
}
}
At last the customer needs a shipment label to print out and stick onto the package. All it has to do is to follow the "myth:shipment-label" link and GET the PDF. Thats it.
/Jørn
I decided that it could be fun to hypermedia-ize the example and show how it could be represented using the media type Mason. So here we go :-)
The first step is to ask for a quote; the customer has a package of some sort and needs to ship it from A to B, so he access the shipment service and asks for a quote (a price) given the size, weight, origin and destination for the package.
In the original example you could ask for a quote by issuing a GET request to /quote. But I believe that asking for a quote would result in a concrete quote being created and stored in the system as a separate resource to access later on, either by the customer or by the customer service of the shipment company. So I would rather go for a POST of a quote request followed by a redirect to the newly created quote.
At this point we could either document how to POST such a quote - or we could tell the client how to do it using hypermedia controls - and obviously I would go for the later. So lets ask the service for instructions and issue a GET /quote request. The output is a Mason document with suitable hypermedia controls embedded in it:
{
"@namespaces":
{
"myth":
{
"name": "http://mythological-shipment.com/api/rel-types"
}
},
"@actions":
{
"myth:quote":
{
"type": "json",
"method": "POST",
"href": "http://mythological-shipment.com/api/quote",
"title": "Ask for a quote",
"description": "Ask for a quote by posting package details. Weight is in kilograms, volume in cubic decimeters, origin and destination must be known identifiers for airports.",
"schemaUrl": "... URL to JSON schema describing the request ..."
}
}
}
The client reads this action specification, encodes the package details in JSON and POST it to the URL of the "href" property. As a result the service creates a new quote resource and redirects the client to it:
Request:
POST http://mythological-shipment.com/api/quote HTTP/1.1
content-type: application/json
{
"weight": 2.3,
"volume": 4,
"origin": "CPH",
"destination": "AMS"
}
Response:
201 Created
Location: http://mythological-shipment.com/api/quotes/myqo-129-gyh
Now the client can GET the newly created quote to get further instructions of how to accept the quote. The result is again a Mason representation of the quote itself plus hypermedia controls for accepting the quote:
{
"id": "myqo-129-gyh",
"weight": 2.3,
"volume: 4,
"origin": "CPH",
"destination": "AMS":
"price": 12,
"@links":
{
"self":
{
"href": "http://mythological-shipment.com/api/quotes/myqo-129-gyh"
}
},
"@actions":
{
"myth:accept-quote":
{
"type": "POST",
"href": "http://mythological-shipment.com/api/quotes/myqo-129-gyh/state",
"template":
{
"accepted": "yes"
}
}
}
}
As you can see the quote has a "self" link identifying the location of the quote resource. It also have a "accept-quote" action that instructs the client about how to accept the quote. In this case all the client has to do is to POST a predefined JSON value to http://mythological-shipment.com/api/quotes/myqo-129-gyh/state.
The result of accepting a quote is that it is converted to a sales order (in lack of better domain understanding - there's probably a better word for it). So the accept-quote operation results in a redirect to the newly create sales order which the client can GET:
{
"id": "myqo-129-gyh",
"weight": 2.3,
"volume: 4,
"origin": "CPH",
"destination": "AMS":
"price": 12,
"@links":
{
"self":
{
"href": "http://mythological-shipment.com/api/orders/myqo-129-gyh"
},
"myth:quote":
{
"href": "http://mythological-shipment.com/api/quotes/myqo-129-gyh"
},
"myth:shipment-label":
{
"href": "http://mythological-shipment.com/api/quotes/myqo-129-gyh/label",
"type": "application/pdf"
}
}
}
At last the customer needs a shipment label to print out and stick onto the package. All it has to do is to follow the "myth:shipment-label" link and GET the PDF. Thats it.
/Jørn
onsdag, februar 26, 2014
API authentication considerations and best practices
I have been answering a few security questions on Stackoverflow and going through some APIs on programmableweb.com - and it keeps amazing me how often people gets HTTP authorization wrong.
The typical questions I have noticed on SO are something like "I want to secure my API, have control of clients and require users to login - but I don't want to use OAuth". Duh? Why not? Perhaps because OAuth is conceived as being too difficult to work with? Well, OAuth2 is definitively not difficult (OAuth1 is another story though).
But lets take a look at some of the existing practices:
API keys in URLs: one example is the Finish National Gallery API (which I somehow stumbled upon). Here you are required to obtain an API key and put it into the URL in every request made to the API:
http://kokoelmat.fng.fi/api/v2?apikey=********&q=A+III+2172
API keys in custom headers: one example is from DocuSign. They require you to encode both your (developer) user name, password and API key in a JSON object and then send that in a custom header X-DocuSign-Authentication:
GET /some-url
X-DocuSign-Authentication: { "Username": "...", "Password": "...", "IntegratorKey": "..." }
... more ...
Signed URL/body parameters: one example is last.fm which requires you to use your API key to obtain a session token and use that token later on to sign requests, either using URL parameters or in the body of the HTTP request.
HTTP Basic authentication: GitHub supports authentication via the standard HTTP basic authentication mechanism where you supply username and password BASE64 encoded in the "Authorization" header.
OAuth1 and Oauth2: PhotoBucket uses OAuth1 and BaseCamp uses Oauth2 (in addition to HTTP basic authentication).
I did set out with the expectation of finding many more strange authorization schemes, but it turned out that at least most of the big players use OAuth1 or OAuth2. That's great! Now we just need to get the word out to all the "dark matter developers" out there (as Scott Hanselman calls them).
1) API keys in URLs are very easily accidentally exposed to third parties. If you take a copy of the URL and mail it to somebody you will end up sending your private key to someone else. For the same reason API keys may end up in log files here and there. Not good!
2) API keys in URLs changes your resource URL such that it depends on who is using it. Instead of everybody referring to /some-items/1234 it will be /some-items/1234?key=abc for one user and /some-items/1234?key=qwe for another - and these two are completely different URLs even though they "only" differ on the query parameters. It is semantically the same as encoding the key in the path segment as for instance /api/abc/some-items/1234 and /api/qwe/some-items/1234 which I don't expect any one to think of as a good idea - right?
3) API keys in headers are better than API keys in URLs since it will keep the URL stable for all users and it doesn't expose API keys when sending links to others. The problem though is that you are still sending your secret credentials (the API key) over the wire and it may be wiretapped by third parties. Another problem is that a client may accidentally connect to the wrong server (by misconfiguration or otherwise) and expose its credentials there.
4) Signed requests (done right!) should be preferred over sending the API key directly. The signature should not be part of the URL for the reasons stated above - and the same goes somehow for sending the signature in the body since different users will see different bodies which is not necessary when you have a standard HTTP header for the purpose.
5) Proprietary methods for signing requests are prone to design errors as it is easy to get the signature mechanism wrong - and thus accidentally making the signature technique easy to circumvent.
6) Proprietary methods for signing requests requires client developer to understand yet another signature mechanism. It also makes it less likely to find existing client libraries to handle the crypto stuff. Both of these issues makes client adoption of your API less likely. And you know what, dear server developer? Your client developers will call for support and it will end up on YOUR table, reducing your ability to focus on coding the next great thing! So stick to well documented standards - it will be less annoying for everybody including yourself.
7) HTTP basic authentication is great for debugging and getting started scenarios but should not be used in production. The problem is that client credentials are send in clear text and are thus susceptible to accidental exposure as mentioned before. Your API should support basic authentication as it will make it possible to explore the API using a standard web browser - but it should be disabled in production.
8) It should be possible to revoke API keys in case they are compromised in some way.
What kind of technique would solve all of these issues? Well, Oauth2 is one standard solution; it uses the HTTP "Authorization" header (thus avoiding authentication stuff in URL and body), it is a standard and used correctly it protects clients from exposing their API keys to the server and over the wire.
Another solution is to use SSL/TLS with client certificates. With the proper HTTP client libraries this can be as easy as loading a certificate (one line of code) and assigning it to the HTTP request object (second line of code). This can although not authorize a combination of both client credentials and end user credentials.
Before jumping into OAuth2 I better explain some of the terms used when talking about OAuth2 (mostly copied from the RFC at http://tools.ietf.org/html/rfc6749#section-1.1):
- Protected resources: the data you want to protect.
- Resource owner: An entity capable of granting access to a protected resource. The "entity" is often a person - the end user.
- Resource server: The server hosting the protected resources.
- Client: An application making protected resource requests on behalf of the resource owner. This can for instance be a mobile application, a website or a background integration process impersonating an existing end user.
- Client credentials: a pair of client ID and client secret. This could be your developer or application ID and API key.
- Resource owner password credentials: the typical user-name/password combination issued to an end user.
OAuth2 has four different modes of operating (called flows) - going from the relatively complex 3-legged authorization flow to the very simple "client credentials" authorization flow. At least one of the flows should fit into just about any public API ecosystem out there - and if not then it is possible to add new extension flows (as for instance Google does with the JWS implementation).
At its core OAuth2 has only two high level steps:
1) Swap a set of user and/or client credentials for an access token (authorization step), and
2) Use the access token to access the API resources.
That's it. It can be very simple. The difficult part of OAuth2 is the many ways a client can obtain an access token.
Here I will focus on a scenario with a trusted client that accepts user credentials and acts on behalf of them. This can be very useful for many internal integration patterns where no humans are involved. I won't cover the scenario where an untrusted client (third party website) needs to access the end user's resources using the 3-legged authorization flow.
At a suitable high level OAuth2 only has two steps as mentioned before: 1) authorization, 2) resource access. The output of a successful authorization step is always an access token which must be used in subsequent requests to the resource server (the API).
In most cases the access token is a so called "bearer token" - a token which the bearer can present to gain access to resources. Such a token has no built-in semantics and should be considered as nothing but a plain text string.
The bearer token is included in the HTTP Authorization header like this:
GET /resource HTTP/1.1
Host: server.example.com
Authorization: Bearer SOME-TOKEN
This is just about the simplest flow possible (see http://tools.ietf.org/html/rfc6749#section-4.4): all the client has to do is to send it's client credentials (ID and secret) as if it was using HTTP basic authorization. In return the client receives an access token it can use in subsequent requests for the protected resources. Here is an example from the RFC:
POST /token HTTP/1.1
Host: server.example.com
Authorization: Basic czZCaGRSa3F0MzpnWDFmQmF0M2JW
Content-Type: application/x-www-form-urlencoded
grant_type=client_credentials
The response could be:
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
Cache-Control: no-store
Pragma: no-cache
{
"access_token":"2YotnFZFEjr1zCsicMWpAA",
"token_type":"bearer"
}
This flow allows the client to act on behalf of itself but it does not include any end user information.
If a client needs to act on behalf of the end user (the resource owner) then it can use the "Resource Owner Password Credentials Grant". This flow is almost as simple is the previous flow - all the client has to do is to add the client credentials to the authorization request. Here is an example from the RFC:
POST /token HTTP/1.1
Host: server.example.com
Authorization: Basic czZCaGRSa3F0MzpnWDFmQmF0M2JW
Content-Type: application/x-www-form-urlencoded
grant_type=password&username=johndoe&password=A3ddj3w
The response could be:
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
Cache-Control: no-store
Pragma: no-cache
{
"access_token":"2YotnFZFEjr1zCsicMWpAA",
"token_type":"bearer"
}
This flow allows the client to act on behalf of the end user. The client application must be trusted as the end user will have to pass his/her credentials to it.
There is one big problem with the two simple flows mentioned above; both of them pass the client and end user credentials in clear text. For this reason OAuth2 makes it mandatory to use TLS/SSL to encrypt all of the requests. Neither does these flows cover scenarios with untrusted clients (such as a third party website that wants to access an end users resources on a different server) - but that scenario is covered in the "Authorization Code Grant" scenario which I will not discuss here.
The use of TLS/SSL protects the credentials from eavesdropping and man-in-the-middle attacks but it does not protect against sending credentials to the wrong server - either by misconfiguration or because the server has been compromised.
This lack of protection is perhaps one of the biggest difference between OAuth1 and OAuth2; OAuth1 protects credentials by signing requests instead of sending the raw credentials. This requires a rather complex signing algorithm which has caused many headaches over the time. OAuth2 trades the protection of credentials for simplicity which makes it a lot easier to use - but susceptible to leaking credentials to rogue servers.
It is possible to protect credentials from this problem though but it requires clients to use some complex crypto stuff to sign requests instead of sending the raw credentials (just like OAuth1 does). This is what Google does when it requires requests to be signed using JSON Web Signatures (JWS).
If you are using .NET then my Ramone HTTP library has support for OAuth2 with JWS as can be seen in this example: http://soabits.blogspot.com/2013/03/using-ramone-for-oauth2-authorization.html.
One more word of caution - websites, mobile apps, desktop apps and similar with public clients cannot protect their client credentials. Other application may be able to do it (server to server integrations for instance) but public clients, be it JavaScript, byte code or assembler, can always be downloaded, decompiled and scrutinized by various forensic tools and in the end it will be impossible to keep client secrets from being stolen. This is a fact we have to live with.
Client credentials can be useful for statistics. They also give the ability to revoke credentials in order to block rogue applications - but they cannot be trusted in general.
The typical questions I have noticed on SO are something like "I want to secure my API, have control of clients and require users to login - but I don't want to use OAuth". Duh? Why not? Perhaps because OAuth is conceived as being too difficult to work with? Well, OAuth2 is definitively not difficult (OAuth1 is another story though).
But lets take a look at some of the existing practices:
API keys in URLs: one example is the Finish National Gallery API (which I somehow stumbled upon). Here you are required to obtain an API key and put it into the URL in every request made to the API:
http://kokoelmat.fng.fi/api/v2?apikey=********&q=A+III+2172
API keys in custom headers: one example is from DocuSign. They require you to encode both your (developer) user name, password and API key in a JSON object and then send that in a custom header X-DocuSign-Authentication:
GET /some-url
X-DocuSign-Authentication: { "Username": "...", "Password": "...", "IntegratorKey": "..." }
... more ...
Signed URL/body parameters: one example is last.fm which requires you to use your API key to obtain a session token and use that token later on to sign requests, either using URL parameters or in the body of the HTTP request.
HTTP Basic authentication: GitHub supports authentication via the standard HTTP basic authentication mechanism where you supply username and password BASE64 encoded in the "Authorization" header.
OAuth1 and Oauth2: PhotoBucket uses OAuth1 and BaseCamp uses Oauth2 (in addition to HTTP basic authentication).
I did set out with the expectation of finding many more strange authorization schemes, but it turned out that at least most of the big players use OAuth1 or OAuth2. That's great! Now we just need to get the word out to all the "dark matter developers" out there (as Scott Hanselman calls them).
Things to be aware of
1) API keys in URLs are very easily accidentally exposed to third parties. If you take a copy of the URL and mail it to somebody you will end up sending your private key to someone else. For the same reason API keys may end up in log files here and there. Not good!
2) API keys in URLs changes your resource URL such that it depends on who is using it. Instead of everybody referring to /some-items/1234 it will be /some-items/1234?key=abc for one user and /some-items/1234?key=qwe for another - and these two are completely different URLs even though they "only" differ on the query parameters. It is semantically the same as encoding the key in the path segment as for instance /api/abc/some-items/1234 and /api/qwe/some-items/1234 which I don't expect any one to think of as a good idea - right?
3) API keys in headers are better than API keys in URLs since it will keep the URL stable for all users and it doesn't expose API keys when sending links to others. The problem though is that you are still sending your secret credentials (the API key) over the wire and it may be wiretapped by third parties. Another problem is that a client may accidentally connect to the wrong server (by misconfiguration or otherwise) and expose its credentials there.
4) Signed requests (done right!) should be preferred over sending the API key directly. The signature should not be part of the URL for the reasons stated above - and the same goes somehow for sending the signature in the body since different users will see different bodies which is not necessary when you have a standard HTTP header for the purpose.
5) Proprietary methods for signing requests are prone to design errors as it is easy to get the signature mechanism wrong - and thus accidentally making the signature technique easy to circumvent.
6) Proprietary methods for signing requests requires client developer to understand yet another signature mechanism. It also makes it less likely to find existing client libraries to handle the crypto stuff. Both of these issues makes client adoption of your API less likely. And you know what, dear server developer? Your client developers will call for support and it will end up on YOUR table, reducing your ability to focus on coding the next great thing! So stick to well documented standards - it will be less annoying for everybody including yourself.
7) HTTP basic authentication is great for debugging and getting started scenarios but should not be used in production. The problem is that client credentials are send in clear text and are thus susceptible to accidental exposure as mentioned before. Your API should support basic authentication as it will make it possible to explore the API using a standard web browser - but it should be disabled in production.
8) It should be possible to revoke API keys in case they are compromised in some way.
What kind of technique would solve all of these issues? Well, Oauth2 is one standard solution; it uses the HTTP "Authorization" header (thus avoiding authentication stuff in URL and body), it is a standard and used correctly it protects clients from exposing their API keys to the server and over the wire.
Another solution is to use SSL/TLS with client certificates. With the proper HTTP client libraries this can be as easy as loading a certificate (one line of code) and assigning it to the HTTP request object (second line of code). This can although not authorize a combination of both client credentials and end user credentials.
OAuth2
OAuth2 roles and terms
Before jumping into OAuth2 I better explain some of the terms used when talking about OAuth2 (mostly copied from the RFC at http://tools.ietf.org/html/rfc6749#section-1.1):
- Protected resources: the data you want to protect.
- Resource owner: An entity capable of granting access to a protected resource. The "entity" is often a person - the end user.
- Resource server: The server hosting the protected resources.
- Client: An application making protected resource requests on behalf of the resource owner. This can for instance be a mobile application, a website or a background integration process impersonating an existing end user.
- Client credentials: a pair of client ID and client secret. This could be your developer or application ID and API key.
- Resource owner password credentials: the typical user-name/password combination issued to an end user.
OAuth2 flows
OAuth2 has four different modes of operating (called flows) - going from the relatively complex 3-legged authorization flow to the very simple "client credentials" authorization flow. At least one of the flows should fit into just about any public API ecosystem out there - and if not then it is possible to add new extension flows (as for instance Google does with the JWS implementation).
At its core OAuth2 has only two high level steps:
1) Swap a set of user and/or client credentials for an access token (authorization step), and
2) Use the access token to access the API resources.
That's it. It can be very simple. The difficult part of OAuth2 is the many ways a client can obtain an access token.
Here I will focus on a scenario with a trusted client that accepts user credentials and acts on behalf of them. This can be very useful for many internal integration patterns where no humans are involved. I won't cover the scenario where an untrusted client (third party website) needs to access the end user's resources using the 3-legged authorization flow.
Acces tokens and bearer tokens
At a suitable high level OAuth2 only has two steps as mentioned before: 1) authorization, 2) resource access. The output of a successful authorization step is always an access token which must be used in subsequent requests to the resource server (the API).
In most cases the access token is a so called "bearer token" - a token which the bearer can present to gain access to resources. Such a token has no built-in semantics and should be considered as nothing but a plain text string.
The bearer token is included in the HTTP Authorization header like this:
GET /resource HTTP/1.1
Host: server.example.com
Authorization: Bearer SOME-TOKEN
Authorizing with client credentials only
This is just about the simplest flow possible (see http://tools.ietf.org/html/rfc6749#section-4.4): all the client has to do is to send it's client credentials (ID and secret) as if it was using HTTP basic authorization. In return the client receives an access token it can use in subsequent requests for the protected resources. Here is an example from the RFC:
POST /token HTTP/1.1
Host: server.example.com
Authorization: Basic czZCaGRSa3F0MzpnWDFmQmF0M2JW
Content-Type: application/x-www-form-urlencoded
grant_type=client_credentials
The response could be:
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
Cache-Control: no-store
Pragma: no-cache
{
"access_token":"2YotnFZFEjr1zCsicMWpAA",
"token_type":"bearer"
}
This flow allows the client to act on behalf of itself but it does not include any end user information.
Authorizing with both client credentials (API key) and user credentials (password)
If a client needs to act on behalf of the end user (the resource owner) then it can use the "Resource Owner Password Credentials Grant". This flow is almost as simple is the previous flow - all the client has to do is to add the client credentials to the authorization request. Here is an example from the RFC:
POST /token HTTP/1.1
Host: server.example.com
Authorization: Basic czZCaGRSa3F0MzpnWDFmQmF0M2JW
Content-Type: application/x-www-form-urlencoded
grant_type=password&username=johndoe&password=A3ddj3w
The response could be:
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
Cache-Control: no-store
Pragma: no-cache
{
"access_token":"2YotnFZFEjr1zCsicMWpAA",
"token_type":"bearer"
}
This flow allows the client to act on behalf of the end user. The client application must be trusted as the end user will have to pass his/her credentials to it.
Protecting credentials in transit
There is one big problem with the two simple flows mentioned above; both of them pass the client and end user credentials in clear text. For this reason OAuth2 makes it mandatory to use TLS/SSL to encrypt all of the requests. Neither does these flows cover scenarios with untrusted clients (such as a third party website that wants to access an end users resources on a different server) - but that scenario is covered in the "Authorization Code Grant" scenario which I will not discuss here.
The use of TLS/SSL protects the credentials from eavesdropping and man-in-the-middle attacks but it does not protect against sending credentials to the wrong server - either by misconfiguration or because the server has been compromised.
This lack of protection is perhaps one of the biggest difference between OAuth1 and OAuth2; OAuth1 protects credentials by signing requests instead of sending the raw credentials. This requires a rather complex signing algorithm which has caused many headaches over the time. OAuth2 trades the protection of credentials for simplicity which makes it a lot easier to use - but susceptible to leaking credentials to rogue servers.
It is possible to protect credentials from this problem though but it requires clients to use some complex crypto stuff to sign requests instead of sending the raw credentials (just like OAuth1 does). This is what Google does when it requires requests to be signed using JSON Web Signatures (JWS).
If you are using .NET then my Ramone HTTP library has support for OAuth2 with JWS as can be seen in this example: http://soabits.blogspot.com/2013/03/using-ramone-for-oauth2-authorization.html.
Protecting credentials in clients
One more word of caution - websites, mobile apps, desktop apps and similar with public clients cannot protect their client credentials. Other application may be able to do it (server to server integrations for instance) but public clients, be it JavaScript, byte code or assembler, can always be downloaded, decompiled and scrutinized by various forensic tools and in the end it will be impossible to keep client secrets from being stolen. This is a fact we have to live with.
Client credentials can be useful for statistics. They also give the ability to revoke credentials in order to block rogue applications - but they cannot be trusted in general.
Further reading
Eran Hammer has a discussion about the drawbacks of OAuth2 at http://hueniverse.com/2012/07/26/oauth-2-0-and-the-road-to-hell/ and here is another discussion of the drawbacks of bearer tokens http://hueniverse.com/2010/09/29/oauth-bearer-tokens-are-a-terrible-idea/torsdag, februar 20, 2014
Representing an issue tracker with Mason
Two weeks ago I introduced Mason - a media type for representing data with embedded hypermedia elements. Now I would like to go through an example implementation of a fictive issue tracker which uses Mason to represent its data.
At its core the issue tracker has, not surprisingly, "issues" that represent issues that needs to be solved. Issues are organized in projects and may have one or more file attachments associated with them. Issues are represented by their title, a description and a severity level (from 1 to 5).
It would be natural to include comments on issues too but I want to keep the domain as simple as possible while still being able to illustrate all the features of Mason - and comments would not add anything but clutter as they are very similar to attachments.
Please note that the issue tracker will be defined without reference to any existing implementation thus making it a "real" REST service totally independent of any specific implementation. I do have a reference implementation available but that is only used as a proof of concept - it is not part of the issue tracker definition itself.
The issue tracker is defined in terms of data types, links, link templates and actions as described in the next sections. That is not exactly interesting reading so you might want to skip it and jump to the later sections with examples of how to actually use the issue tracker.
You should try the demo issue tracker yourself with the generic Mason browser in order to see how actions and URL templates are supposed to work.
These URL relationship types ensures that we have no name collisions with other relationship types. In addition to this it is actually possible to dereference the URLs and GET some documentation about them.
Id: int
Title: string
Description: string
Severity: integer
Expected links:
Example resource: http://mason-issue-tracker.cbrain.net/issues/1
Issues: array of
- Id: int
- Title: string
Expected links:
Id: int
Code: string
Title: string
Description: string
Expected links:
Expected actions:
Projects: array of
- Id: int
- Code: string (a short code or abbreviation of the project name)
- Title: string
Expected links:
Example resource: http://mason-issue-tracker.cbrain.net/projects
Title: string (title of the whole issue tracker)
Description: string (description of the whole issue tracker)
Expected links:
Expected link templates:
Expected actions:
Example resource: http://mason-issue-tracker.cbrain.net/resource-common
Name: string
Address1: string
Address2: string
PostalCode: string
City: string
EMail: string
Phone: string
Country: string
Expected links:
text: any text to look for in issues.
severity: severity level (1-5)
pid: project ID.
Example usage: http://mason-issue-tracker.cbrain.net/resource-common
Code: string
Title: string
Description: string
Example usage: http://mason-issue-tracker.cbrain.net/resource-common
Code: string
Title: string
Description: string
Example usage: http://mason-issue-tracker.cbrain.net/projects/1
Example usage: http://mason-issue-tracker.cbrain.net/projects/1
Title: string
Description: string
Severity: int
Attachment: object of
- Title: string
- Description: string
In addition to this it is possible to pass a file as an attachment to the issue. The file name is "attachment".
The "Attachment" object contains additional information about the attached file.
Example usage: http://mason-issue-tracker.cbrain.net/projects/1
Title: string
Description: string
Severity: int
Example usage: http://mason-issue-tracker.cbrain.net/issues/1
Example usage: http://mason-issue-tracker.cbrain.net/issues/1
Title: string
Description: string
In addition to this it is possible to pass a file as the actual attachment. The file name is "attachment".
Example usage: http://mason-issue-tracker.cbrain.net/issues/1
Try it yourself: GET http://mason-issue-tracker.cbrain.net/resource-common
Here is an example:
"@actions": {
"is:project-create": {
"type": "json",
"href": "http://mason-issue-tracker.cbrain.net/projects",
"title": "Create new project",
"schemaUrl": "http://mason-issue-tracker.cbrain.net/schemas/create-project"
}
}
The most important parts are the "type" and "href" properties which tells us how to encode the data and where to send it. Mason actions also have a "method" property for identifying the HTTP method to use but it defaults to POST so its not always needed.
The type "json" tells us to encode the action arguments in plain JSON. The net result is a request like this:
POST /projects HTTP/1.1
Accept: application/vnd.mason+json
Content-Type: application/json
{
"Code": "SHOP",
"Title": "Webshop",
"Description": "Project for issues related to the webshop"
}
The response is a redirect to the created project:
HTTP/1.1 201 Created
Location: http://mason-issue-tracker.cbrain.net/projects/2
"@actions": {
"is:add-issue": {
"type": "json+files",
"href": "http://mason-issue-tracker.cbrain.net/projects/2/issues",
"title": "Add new issue to project",
"schemaUrl": "http://mason-issue-tracker.cbrain.net/schemas/create-issue",
"jsonFile": "args",
"files": [
{
"name": "attachment",
"description": "Attachment for issue"
}
]
}
}
This action tells us the following:
POST /projects/2/issues HTTP/1.1
Accept: application/vnd.mason+json
Content-Type: multipart/form-data; boundary=d636dfda-b79f-4f29-aaf6-4b6687baebeb
--d636dfda-b79f-4f29-aaf6-4b6687baebeb
Content-Disposition: form-data; name="attachment"; filename="hogweed.jpg"
... binary data for attached image ...
--d636dfda-b79f-4f29-aaf6-4b6687baebeb
Content-Disposition: form-data; name="args"; filename="args"
Content-Type: application/json
{
"Title": "Hogweeds on the plaza",
"Description": "Could you please remove the hogweeds growing at the plaza?",
"Severity": 5,
"Attachment":
{
"Title": "Hogweed",
"Description": "Photo of the hogweeds."
}
}
"@actions": {
"is:project-update": {
"type": "json",
"href": "http://mason-issue-tracker.cbrain.net/projects/1",
"title": "Update project details",
"template": {
"Code": "SHOP",
"Title": "Webshop",
"Description": "All issues related to the webshop."
}
}
}
This action tells us to encode project arguments in JSON and POST it to "http://mason-issue-tracker.cbrain.net/projects/1". The JSON data should be build from the JSON template in the action.
Request:
POST /projects/1 HTTP/1.1
User-Agent: API Explorer
Accept: application/vnd.mason+json
Content-Type: application/json
{
"Code": "SHOP",
"Title": "Web shop",
"Description": "All issues related to the new web shop."
}
"@actions": {
"is:project-delete": {
"type": "void",
"href": "http://mason-issue-tracker.cbrain.net/projects/1",
"method": "DELETE",
"title": "Delete project"
}
Request:
DELETE /projects/1 HTTP/1.1
Accept: application/vnd.mason+json
Response:
HTTP/1.1 204 No Content
"@link-templates": {
"is:issue-query": {
"template": "http://mason-issue-tracker.cbrain.net//issues-query?text={text}&severity={severity}&project={pid}",
"title": "Search for issues",
"description": "This is a simple search that do not check attachments.",
"parameters": [
{
"name": "text",
"description": "Substring search for text in title and description"
},
{
"name": "severity",
"description": "Issue severity (exact value, 1..5)"
},
{
"name": "pid",
"description": "Project ID"
}
]
}
}
This templates tells us to replace the parameters "text", "severity" and "pid" into the URL template "http://mason-issue-tracker.cbrain.net//issues-query?text={text}&severity={severity}&project={pid}".
Should we for instance want to query for issues of severity 5 in project 1 then we would get this request:
GET /issues-query?text=&severity=5&project=1 HTTP/1.1
Accept: application/vnd.mason+json
The result is a collection of issues.
At its core the issue tracker has, not surprisingly, "issues" that represent issues that needs to be solved. Issues are organized in projects and may have one or more file attachments associated with them. Issues are represented by their title, a description and a severity level (from 1 to 5).
It would be natural to include comments on issues too but I want to keep the domain as simple as possible while still being able to illustrate all the features of Mason - and comments would not add anything but clutter as they are very similar to attachments.
Please note that the issue tracker will be defined without reference to any existing implementation thus making it a "real" REST service totally independent of any specific implementation. I do have a reference implementation available but that is only used as a proof of concept - it is not part of the issue tracker definition itself.
The issue tracker is defined in terms of data types, links, link templates and actions as described in the next sections. That is not exactly interesting reading so you might want to skip it and jump to the later sections with examples of how to actually use the issue tracker.
You should try the demo issue tracker yourself with the generic Mason browser in order to see how actions and URL templates are supposed to work.
Data types, links and actions
CURIE definitions
In the following the CURIE name "is" should be expanded to "http://soabits.dk/mason/issue-tracker/reltypes.html#". So for instance "is:add-issue" becomes "http://soabits.dk/mason/issue-tracker/reltypes.html#add-issue" (you can GET that).These URL relationship types ensures that we have no name collisions with other relationship types. In addition to this it is actually possible to dereference the URLs and GET some documentation about them.
Data types
Issue
A single issue consists of the following properties:Id: int
Title: string
Description: string
Severity: integer
Expected links:
- self
- up: link to parent project
- is:attachments: link to collection of attachments for issue
- is:common: link to common resource data
Example resource: http://mason-issue-tracker.cbrain.net/issues/1
Issue collection
A collection of issues has one top level property:Issues: array of
- Id: int
- Title: string
Expected links:
- self
- up: link to parent project
- is:common: link to common resource data
Project
A single project consists of the following properties:Id: int
Code: string
Title: string
Description: string
Expected links:
- self
- is:issues: link to collection of issues for project
- is:common: link to common resource data
Expected actions:
- is:project-update
- is:add-issue
- is:project-delete
Project collection
A collection of projects has one top level property:Projects: array of
- Id: int
- Code: string (a short code or abbreviation of the project name)
- Title: string
Expected links:
- self
- is:common: link to common resource data
Example resource: http://mason-issue-tracker.cbrain.net/projects
Common resource data
A set of properties and links which are common to all resources:Title: string (title of the whole issue tracker)
Description: string (description of the whole issue tracker)
Expected links:
- self
- is:contact: link to contact information
- is:logo: link to issue tracker logo
- is:projects: link to collection of all projects
- is:common: link to common resource data
Expected link templates:
- is:issue-query
Expected actions:
- is:project-create
Example resource: http://mason-issue-tracker.cbrain.net/resource-common
Contact information
Contact information (related to the owner of the issue tracker) consists of:Name: string
Address1: string
Address2: string
PostalCode: string
City: string
EMail: string
Phone: string
Country: string
Expected links:
- self
- alternate (alternate address representations in other formats like for instance text/vcard)
- is:common
Link relations
is:projects
Link to collection of all projects.is:issues
Link to collection of issues for a given project.is:attachments
Link to collection of attachments for a given issue.is:contact
Link to contact information.is:logo
Link to issue tracker logo.is:common
Link to data common for all resources in the issue tracker.Link templates
is:issue-query
A link template for querying issues. Parameters:text: any text to look for in issues.
severity: severity level (1-5)
pid: project ID.
Example usage: http://mason-issue-tracker.cbrain.net/resource-common
Actions
is:project-create
Action for creating a new project. Arguments:Code: string
Title: string
Description: string
Example usage: http://mason-issue-tracker.cbrain.net/resource-common
is:project-update
Action for updating a single project. Arguments:Code: string
Title: string
Description: string
Example usage: http://mason-issue-tracker.cbrain.net/projects/1
is:project-delete
Action for deleting a project. Has no arguments.Example usage: http://mason-issue-tracker.cbrain.net/projects/1
is:add-issue
Action for adding a new issue to a project. Arguments:Title: string
Description: string
Severity: int
Attachment: object of
- Title: string
- Description: string
In addition to this it is possible to pass a file as an attachment to the issue. The file name is "attachment".
The "Attachment" object contains additional information about the attached file.
Example usage: http://mason-issue-tracker.cbrain.net/projects/1
is:issue-update
Action for updating a single issue. Arguments:Title: string
Description: string
Severity: int
Example usage: http://mason-issue-tracker.cbrain.net/issues/1
is:issue-delete
Action for deleting a single issue. Has no arguments.Example usage: http://mason-issue-tracker.cbrain.net/issues/1
is:add-attachment
Action for adding an attachment to an issue. Arguments:Title: string
Description: string
In addition to this it is possible to pass a file as the actual attachment. The file name is "attachment".
Example usage: http://mason-issue-tracker.cbrain.net/issues/1
Examples
Getting started
The first thing a client must do in order to work with the issue tracker is to GET the "common" resource that contains useful links, templates and actions for the issue tracker. The common resource can be thought of as the "home page" or "landing page" of the issue tracker.Try it yourself: GET http://mason-issue-tracker.cbrain.net/resource-common
Creating a new project
Once we have a copy of the "common" resource we can look for the action "is:project-create". That action will tell us how to encode the project data and how to submit it to the server.Here is an example:
"@actions": {
"is:project-create": {
"type": "json",
"href": "http://mason-issue-tracker.cbrain.net/projects",
"title": "Create new project",
"schemaUrl": "http://mason-issue-tracker.cbrain.net/schemas/create-project"
}
}
The most important parts are the "type" and "href" properties which tells us how to encode the data and where to send it. Mason actions also have a "method" property for identifying the HTTP method to use but it defaults to POST so its not always needed.
The type "json" tells us to encode the action arguments in plain JSON. The net result is a request like this:
POST /projects HTTP/1.1
Accept: application/vnd.mason+json
Content-Type: application/json
{
"Code": "SHOP",
"Title": "Webshop",
"Description": "Project for issues related to the webshop"
}
The response is a redirect to the created project:
HTTP/1.1 201 Created
Location: http://mason-issue-tracker.cbrain.net/projects/2
Adding a new issue
Now that we have a project we can start adding issues to it (with optional attachments). Each project representation contains an "is:add-issue" action for this purpose as can be seen here:"@actions": {
"is:add-issue": {
"type": "json+files",
"href": "http://mason-issue-tracker.cbrain.net/projects/2/issues",
"title": "Add new issue to project",
"schemaUrl": "http://mason-issue-tracker.cbrain.net/schemas/create-issue",
"jsonFile": "args",
"files": [
{
"name": "attachment",
"description": "Attachment for issue"
}
]
}
}
This action tells us the following:
- The type is "json+files" which means we must send the JSON data together with some files wrapped in the media type multipart/form-data.
- The target URL is "http://mason-issue-tracker.cbrain.net/projects/2/issues".
- The JSON data must conform to the schema definition at "http://mason-issue-tracker.cbrain.net/schemas/create-issue".
- The JSON data must be contained in a multipart element named "args".
- The attached file must be contained in a multipart element named "attachment".
POST /projects/2/issues HTTP/1.1
Accept: application/vnd.mason+json
Content-Type: multipart/form-data; boundary=d636dfda-b79f-4f29-aaf6-4b6687baebeb
--d636dfda-b79f-4f29-aaf6-4b6687baebeb
Content-Disposition: form-data; name="attachment"; filename="hogweed.jpg"
... binary data for attached image ...
--d636dfda-b79f-4f29-aaf6-4b6687baebeb
Content-Disposition: form-data; name="args"; filename="args"
Content-Type: application/json
{
"Title": "Hogweeds on the plaza",
"Description": "Could you please remove the hogweeds growing at the plaza?",
"Severity": 5,
"Attachment":
{
"Title": "Hogweed",
"Description": "Photo of the hogweeds."
}
}
Updating project details
Each project has a "is:project-update" action for updating the project details:"@actions": {
"is:project-update": {
"type": "json",
"href": "http://mason-issue-tracker.cbrain.net/projects/1",
"title": "Update project details",
"template": {
"Code": "SHOP",
"Title": "Webshop",
"Description": "All issues related to the webshop."
}
}
}
This action tells us to encode project arguments in JSON and POST it to "http://mason-issue-tracker.cbrain.net/projects/1". The JSON data should be build from the JSON template in the action.
Request:
POST /projects/1 HTTP/1.1
User-Agent: API Explorer
Accept: application/vnd.mason+json
Content-Type: application/json
{
"Code": "SHOP",
"Title": "Web shop",
"Description": "All issues related to the new web shop."
}
Deleting a project
Each project has a "is:project-delete" action for deleting the project and its related issues:"@actions": {
"is:project-delete": {
"type": "void",
"href": "http://mason-issue-tracker.cbrain.net/projects/1",
"method": "DELETE",
"title": "Delete project"
}
Request:
DELETE /projects/1 HTTP/1.1
Accept: application/vnd.mason+json
Response:
HTTP/1.1 204 No Content
Searching for issues
The "common" resource has a link template for issue queries:"@link-templates": {
"is:issue-query": {
"template": "http://mason-issue-tracker.cbrain.net//issues-query?text={text}&severity={severity}&project={pid}",
"title": "Search for issues",
"description": "This is a simple search that do not check attachments.",
"parameters": [
{
"name": "text",
"description": "Substring search for text in title and description"
},
{
"name": "severity",
"description": "Issue severity (exact value, 1..5)"
},
{
"name": "pid",
"description": "Project ID"
}
]
}
}
This templates tells us to replace the parameters "text", "severity" and "pid" into the URL template "http://mason-issue-tracker.cbrain.net//issues-query?text={text}&severity={severity}&project={pid}".
Should we for instance want to query for issues of severity 5 in project 1 then we would get this request:
GET /issues-query?text=&severity=5&project=1 HTTP/1.1
Accept: application/vnd.mason+json
The result is a collection of issues.
torsdag, februar 06, 2014
Implementing hypermedia APIs and REST services with Mason
I am happy to announce that I have taken all the lessons learned during the last few years and stuffed it into a new JSON based mediatype for hypermedia APIs and REST services. The media type is application/vnd.mason+json or simply "Mason". There is an IANA registration for it pending.
With Mason you get hypermedia elements for linking and modifying data, features for communicating to client developers and standardized error handling. Mason is built on JSON, reads JSON, writes JSON and generally fits well into a JSON based eco-system.
Here is a simple example illustrating how a single issue from a fictive issue tracker could be represented in Mason. It contains the basic API data like issue Title, Description and Severity and then it adds hypermedia elements for linking to other related resources and actions for writing stuff back to the issue tracker.
{
// Classic API data
"ID": 1,
"Title": "Program crashes when pressing ctrl-p",
"Description": "I pressed ctrl-p and, boom, it crashed.",
"Severity": 5,
"Attachments": [
{
"Id": 1,
"Title": "Error report",
// Hypermedia linking to attachment
"@links": {
"self": {
"href": "http://issue-tracker.org/attachments/1"
}
}
}
],
// Additional hypermedia links
"@links": {
// Hypermedia linking to self
"self": {
"href": "http://issue-tracker.org/issues/1"
},
// Hypermedia linking to containing project
"up": {
"href": "http://issue-tracker.org/projects/1",
"title": "Containing project"
},
},
// Hypermedia "action" element for creating a new project
"@actions": {
"is:project-create": {
"type": "json",
"href": "http://issue-tracker.org/mason-demo/projects",
"title": "Create new project",
"schemaUrl": "http://issue-tracker.org/mason-demo/schemas/create-project"
}
}
}
Those that are familiar with HAL may recognize some parts of the format. That is expected as Mason builds on the ideas from HAL. HAL was never intended to have hypermedia elements for writing stuff so I decided to go for it and design a new format based on HAL.
The Mason specification, online example and stand-alone API explorer are available from https://github.com/JornWildt/Mason.
My design goals with Mason are:
1. It should be easy to adopt in existing JSON based solutions and have a low barrier of entry for new developers.
2. It should contain hypermedia elements sufficient for both reading and writing data without any out-of-band information.
3. It should contain elements for information directed to client developers for the purpose of improving "API developer experience".
4. It should contain error elements sufficient for most kinds of applications.
5. It should work with JSON when both reading and writing.
Let me dig into each of those design goals one by one.
A classic JSON payload makes the raw API data directly accessible as JSON object properties. I believe it should be so too when working with hypermedia enabled APIs. So Mason merges hypermedia elements into existing JSON structures. To avoid name collisions Mason property names are prefixed with a '@'.
Mason can be adopted gradually:
Step 1: Change content type to application/vnd.mason+json instead of application/json.
Step 2: Add a @meta property with additional information targeted at client developers.
Step 3: Use links to remove client knowledge of server defined URLs.
Step 4: Use Mason's error format.
Step 5: Use actions to truly decouple client and server implementations.
Hypermedia has a lot of benefits as I wrote in http://soabits.blogspot.dk/2013/12/selling-benefits-of-hypermedia.html. Among these is the ability to remove a client's dependency on server URL structures using links.
But links are only good for, well, linking resources together - they don't say anything about how to change and modify API data. So Mason adds "actions" for writing API data.
An action defines both target URL, HTTP method and action type (payload encoding). With this information being discoverable at runtime it is no longer necessary to hard code clients with information about HTTP method and how to encode the payload. This means client and server only have to agree on WHICH data to send - not HOW to send it.
One of the great things about hypermedia enabled APIs is the ability to explore the API using a browser of some kind. As I wrote in http://soabits.blogspot.dk/2013/12/selling-benefits-of-hypermedia.html; Do not underestimate the power of an explorable API. The ability to browse around the data makes it a lot easier for the client developers to build a mental model of the API and its data structures.
And if client developers are browsing the API why not also be able to communicate directly with them? Mason adds a few meta data elements for sending messages directly to the client developers. An API browser should highlight these such that devs can instantly read some documentation and comments about the resource they are currently looking at.
At the same time Mason defines a technique for removing this client developer information from the payload in production.
By standardizing error handling Mason makes it possible for clients to interact with unknown services and still be able to communicate error conditions clearly to end users.
I have previously discussed error handling here in http://soabits.blogspot.dk/2013/05/error-handling-considerations-and-best.html and apparently that article hit a nerve somewhere because it keeps attracting a lot of attention (for an amateur blogger like me).
One of things that annoys me about the traditional key/value forms based on application/x-www-form-urlencoded is that there are no standards for encoding complex data structures. Neither does it define any standard for encoding booleans, integers and other basic data types. The consequence is that client and server needs to agree on these things before they can start talking about business data - and different servers are surely going to implement different encoding schemes - all in all making life miserable for developers that just want to get stuff done.
By using JSON Mason ensures interoperability on some of the lower levels. JSON defines more types than simple string based key/value formats and handles structures like objects and arrays.
Restricting implementations of Mason to handle JSON only reduces design choices and variations and thus improving the chances of things working out of the box (compared to simple string based key/value formats).
Most web APIs today are defined in terms of a single server implementation around which developers build dedicated clients (think "Twitter" or "Facebook"). In such a world clients have a strong coupling to server URL structures, HTTP methods, error formats and other quirks of the API. These APIs were never designed to be implemented by more than one organization (and are for this reason also called "snowflake APIs").
True REST services on the other hand are defined without reference to any specific server implementation. The best known example of this is the ATOM format which enables clients to interact with any ATOM enabled service on the web - no matter who implemented it, where it is hosted or what URL structures it is implemented with. The enabling factor for this is the ATOM media type specification.
But ATOM is restricted to feed-like data and does not fit well with other applications. So other media types are needed and Mason is an attempt to fill out this space. Mason attempts to facilitate complete decoupling from technical implementation details such that clients can discover HOW to interact with service at runtime.
One of my earlier blog posts discussed this problem in more detail: http://soabits.blogspot.no/2013/05/the-role-of-media-types-in-restful-web.html
Mason itself does not prescribe any business specific details. Clients and servers still have to agree on WHAT data to interchange - but Mason do remove the technical coupling on HOW to interchange the data.
Mason depends on profiles to enable clients to know WHAT data they are looking at. You can find an in-depth discussion about it here: http://soabits.blogspot.no/2013/12/media-types-for-apis.html.
At the time of writing I haven't put profiles into the specification yet.
Generic Mason browser (API explorer): https://github.com/JornWildt/Mason/wiki/Generic-Mason-browser
Online live example of fictive issue tracker using Mason: https://github.com/JornWildt/Mason/wiki/Example-service%3A-issue-tracker
/Jørn
With Mason you get hypermedia elements for linking and modifying data, features for communicating to client developers and standardized error handling. Mason is built on JSON, reads JSON, writes JSON and generally fits well into a JSON based eco-system.
Here is a simple example illustrating how a single issue from a fictive issue tracker could be represented in Mason. It contains the basic API data like issue Title, Description and Severity and then it adds hypermedia elements for linking to other related resources and actions for writing stuff back to the issue tracker.
{
// Classic API data
"ID": 1,
"Title": "Program crashes when pressing ctrl-p",
"Description": "I pressed ctrl-p and, boom, it crashed.",
"Severity": 5,
"Attachments": [
{
"Id": 1,
"Title": "Error report",
// Hypermedia linking to attachment
"@links": {
"self": {
"href": "http://issue-tracker.org/attachments/1"
}
}
}
],
// Additional hypermedia links
"@links": {
// Hypermedia linking to self
"self": {
"href": "http://issue-tracker.org/issues/1"
},
// Hypermedia linking to containing project
"up": {
"href": "http://issue-tracker.org/projects/1",
"title": "Containing project"
},
},
// Hypermedia "action" element for creating a new project
"@actions": {
"is:project-create": {
"type": "json",
"href": "http://issue-tracker.org/mason-demo/projects",
"title": "Create new project",
"schemaUrl": "http://issue-tracker.org/mason-demo/schemas/create-project"
}
}
}
Those that are familiar with HAL may recognize some parts of the format. That is expected as Mason builds on the ideas from HAL. HAL was never intended to have hypermedia elements for writing stuff so I decided to go for it and design a new format based on HAL.
The Mason specification, online example and stand-alone API explorer are available from https://github.com/JornWildt/Mason.
Design goals
My design goals with Mason are:
1. It should be easy to adopt in existing JSON based solutions and have a low barrier of entry for new developers.
2. It should contain hypermedia elements sufficient for both reading and writing data without any out-of-band information.
3. It should contain elements for information directed to client developers for the purpose of improving "API developer experience".
4. It should contain error elements sufficient for most kinds of applications.
5. It should work with JSON when both reading and writing.
Let me dig into each of those design goals one by one.
1. Easy to adopt
A classic JSON payload makes the raw API data directly accessible as JSON object properties. I believe it should be so too when working with hypermedia enabled APIs. So Mason merges hypermedia elements into existing JSON structures. To avoid name collisions Mason property names are prefixed with a '@'.
Mason can be adopted gradually:
Step 1: Change content type to application/vnd.mason+json instead of application/json.
Step 2: Add a @meta property with additional information targeted at client developers.
Step 3: Use links to remove client knowledge of server defined URLs.
Step 4: Use Mason's error format.
Step 5: Use actions to truly decouple client and server implementations.
2. Hypermedia for both reading and writing
Hypermedia has a lot of benefits as I wrote in http://soabits.blogspot.dk/2013/12/selling-benefits-of-hypermedia.html. Among these is the ability to remove a client's dependency on server URL structures using links.
But links are only good for, well, linking resources together - they don't say anything about how to change and modify API data. So Mason adds "actions" for writing API data.
An action defines both target URL, HTTP method and action type (payload encoding). With this information being discoverable at runtime it is no longer necessary to hard code clients with information about HTTP method and how to encode the payload. This means client and server only have to agree on WHICH data to send - not HOW to send it.
3. Information targeted at client developers
One of the great things about hypermedia enabled APIs is the ability to explore the API using a browser of some kind. As I wrote in http://soabits.blogspot.dk/2013/12/selling-benefits-of-hypermedia.html; Do not underestimate the power of an explorable API. The ability to browse around the data makes it a lot easier for the client developers to build a mental model of the API and its data structures.
And if client developers are browsing the API why not also be able to communicate directly with them? Mason adds a few meta data elements for sending messages directly to the client developers. An API browser should highlight these such that devs can instantly read some documentation and comments about the resource they are currently looking at.
At the same time Mason defines a technique for removing this client developer information from the payload in production.
4. Error handling
By standardizing error handling Mason makes it possible for clients to interact with unknown services and still be able to communicate error conditions clearly to end users.
I have previously discussed error handling here in http://soabits.blogspot.dk/2013/05/error-handling-considerations-and-best.html and apparently that article hit a nerve somewhere because it keeps attracting a lot of attention (for an amateur blogger like me).
5. JSON read/write
One of things that annoys me about the traditional key/value forms based on application/x-www-form-urlencoded is that there are no standards for encoding complex data structures. Neither does it define any standard for encoding booleans, integers and other basic data types. The consequence is that client and server needs to agree on these things before they can start talking about business data - and different servers are surely going to implement different encoding schemes - all in all making life miserable for developers that just want to get stuff done.
By using JSON Mason ensures interoperability on some of the lower levels. JSON defines more types than simple string based key/value formats and handles structures like objects and arrays.
Restricting implementations of Mason to handle JSON only reduces design choices and variations and thus improving the chances of things working out of the box (compared to simple string based key/value formats).
Transcending from web APIs to REST services
Most web APIs today are defined in terms of a single server implementation around which developers build dedicated clients (think "Twitter" or "Facebook"). In such a world clients have a strong coupling to server URL structures, HTTP methods, error formats and other quirks of the API. These APIs were never designed to be implemented by more than one organization (and are for this reason also called "snowflake APIs").
True REST services on the other hand are defined without reference to any specific server implementation. The best known example of this is the ATOM format which enables clients to interact with any ATOM enabled service on the web - no matter who implemented it, where it is hosted or what URL structures it is implemented with. The enabling factor for this is the ATOM media type specification.
But ATOM is restricted to feed-like data and does not fit well with other applications. So other media types are needed and Mason is an attempt to fill out this space. Mason attempts to facilitate complete decoupling from technical implementation details such that clients can discover HOW to interact with service at runtime.
One of my earlier blog posts discussed this problem in more detail: http://soabits.blogspot.no/2013/05/the-role-of-media-types-in-restful-web.html
Data profiles
Mason itself does not prescribe any business specific details. Clients and servers still have to agree on WHAT data to interchange - but Mason do remove the technical coupling on HOW to interchange the data.
Mason depends on profiles to enable clients to know WHAT data they are looking at. You can find an in-depth discussion about it here: http://soabits.blogspot.no/2013/12/media-types-for-apis.html.
At the time of writing I haven't put profiles into the specification yet.
Further reading
Mason homepage: https://github.com/JornWildt/MasonGeneric Mason browser (API explorer): https://github.com/JornWildt/Mason/wiki/Generic-Mason-browser
Online live example of fictive issue tracker using Mason: https://github.com/JornWildt/Mason/wiki/Example-service%3A-issue-tracker
/Jørn
Abonner på:
Opslag (Atom)