It's thus surprising to see little materials helping develop API clients. We all know the "best effort" nature of HTTP, services timing out, network hiccups, latencies, etc. and yet we rarely account for any of those when (quickly) coding an API client.
We have all the building blocks though: for most programming languages, there is an Http Client component, there are many message formats implemented along with parsers/(un)marshallers (XML, JSON, ProtoBuf, etc.), and most if not all of them are open source so we can customize at will.
I provide here some thoughts (and more) on developing API clients.
Inspiration
This is heavily inspired by the AWS Java SDK. At a high level, it's a beautiful API Client. When you dig deeper though, things get a little hairy. If I may be so blunt, I'd say a C/C++ developer coded it. It's very efficient, precise and straight to the functionality but it's C/C++ hairy.
I ran static analysis on it (not needed, just for fun) and oh boy!...packages depending on each other, no tangible layers, very little separation of concerns, deep hierarchy, etc.
It's not broken cause it works really well so I'm not here to fix or change anything in AWS Java SDK. I'm here to leverage some of the good parts and from there build what I think makes sense and yield to something flexible and hopefully useful.
Antiope Project
Along the (thought process) way, I decided to go ahead and code whatever I thought useful to help build API clients. Being inspired by AWS Java SDK, I called this project Antiope, Amazon woman, the daughter of Ares in Greek Mythology.
Antiope follows all the guidelines presented here. It provides a Java implementation and nothing more.
From time to time, I might explain how certain concepts have been implemented in Antiope.
Architecture
 |
| Fig. 1. API Client Layer Architecture |
The idea is to develop API clients for end developers. Which means there's a layer for them. This is the End-developer Domain Layer.
This represents the classes end-developers deal with when using AWS Java SDK for example: DescribeInstancesRequest, DescribeInstancesResponse, Instance, EC2SecurityGroup , ... and of course the clients, AmazonEC2, AmazonS3, etc.
All those classes must be straightforward and simple. Just data containers and well documented methods in the client interfaces. All the complexity is then be scoped within the implementation of those client interfaces.
At the very bottom is the framework and/or libraries helping deal with marshalling, unmarshalling, making http requests, handling http responses, tweaking headers, user agents, collecting metrics, setting up proxy, etc. This is the Framework Layer.
Because of what we're leveraging from the Framework layer, we would always have this "bridge" layer: implementing our domain layer using classes from the Framework layer. This is the Implementation Layer.
This architecture reduces the fragility of an API Client:
- The Domain Layer abstracts the inner workings of how the client interacts with the service. It provides a simple and loose-coupled view of the service at a client level. This makes this layer robust and not easily affected by changes.
- Service-level, engineering or simply implementation changes will only affect the Implementation Layer. This is the behind-the-scene code, the back-end code of the API client. End-developers would not directly depend on those classes and therefore should not be affected by the volatility of the implementation.
Both aspects are what makes an API client adaptable and robust, or for short, anti-fragile[4].
Example
Here's succinct architecture of an imaginary Polite API Client:
 | |
|
Antiope is one implementation of that Framework Layer. Basic and advanced version of an API client are provided to help developers writes their own. It provides the structure and process of going from the Domain Layer, to the endpoint of the web service, and back to creating results in the Domain Layer, among other things.
Workflow
An API client is composed of methods that leads to an exchange over the wire. For each of those methods, the workflow is more or less as follows:
- Marshall domain object into Request.
- Send Request and get Response
- Unmarshall Response (actual result or error) into domain object
- Returns domain object
Execution Context
To help provide information at each step of the process an Execution Context is used. It basically holds the domain class (e.g. DescribeInstancesRequest), but it can contain credentials information as well as metrics.
Metrics
Metrics are bits of measures. They hold information on how long it took to execute the HTTP request, how long it took to unmarshall the response, etc. This is just a data container, filled by classes executed during the process.
Metrics Collector
At the end of the workflow, right before returning a result (domain object), a client uses a Metrics Collector to do something of the metrics defined so far. It might log those metrics or send them to a (remote) service (e.g. AWS CloudWatch).
Such information can then be used to monitor performance of client and service, fine tune settings and even help troubleshoot problems.
A Client can be configured with a certain metrics collector but each individual Web Request can have its own Metrics Collector as well, overriding the client-level metrics collector for the duration of the workflow.
Marshaller & Unmarshaller
Those form the bridge between the Domain Layer and what goes over the wire and how.
For example, properties from a domain object might be marshalled into query string params. A JSON response would be unmarshalled into domain object.
In Antiope, some helpers are provided to unmarshal XML (Stax) and JSON.
But one is free to use anything else (JAXB, Dozer, JDOM, etc.).
Exceptions
There are 2 kind of exceptions in an API Client: client-side exceptions and a server-side exceptions.
The former is when something went wrong with the client (missing parameters, marshalling/unmarshalling errors, etc.).
The latter is a translation of a response representing an error message from the web service. This means there were no errors in the code of the client itself, but the request might have been invalid or incomplete or the service itself might have been busy, overloaded or down.
Advanced logic can use such information to correct or fail fast depending on the situation.
For example, retry logic could be implemented based on certain types of server-side exceptions (e.g. busy, overloaded, etc.).
Sample Application: Yahoo! Weather Client
This is a very basic application that consumes a single Yahoo Web Service[3]: the Weather RSS Feed.
This web service takes 2 parameters:
- w, being the Station Id
- u, being actually the Unit System to use (F for Farenheight/US system and C for Celsius/Metric system)
Domain Layer
This simple web service provides information about the current conditions, a 3-5 days forecast, atmospheric, astronomic and wind information using RSS format.
 |
| Fig. 3. Domain Layer of Yahoo! Client |
Implementation Layer
The client implementation is simple using Antiope:
- Marshall WeatherRequest into Request parameters (w and u mentioned above)
- Unmarshall response as WeatherResponse. This unmarshaller will actually (re)use other marshallers corresponding to each distinct piece of information: LocationUnmarshaller, ConditionsUnmarshaller, ForecastUnmarshaller, etc.
- Create a IYahooClient implementation using BaseAPIClient from Antiope.
 |
| Fig. 4. Implementation Layer of Yahoo! Client |
The code of the getWeather() method implementation is as follows:
ExecutionContext oContext = createExecutionContext(pRequest);
IMetrics oMetrics = oContext.getMetrics();
Request<WeatherRequest> request = null;
Response<WeatherResponse> response = null;
oMetrics.startEvent(APIRequestMetrics.ClientExecuteTime);
try {
oMetrics.startEvent(APIRequestMetrics.RequestMarshallTime);
try {
request = new WeatherRequestMarshaller().marshall(pRequest);
request.setMetrics(oMetrics);
} finally {
oMetrics.endEvent(APIRequestMetrics.RequestMarshallTime);
}
response = invoke(request, new WeatherResponseUnmarshaller(), oContext);
return response.getAPIResponse();
} catch (Exception e) {
throw new APIClientException(e);
} finally {
endClientExecution(oMetrics, request, response);
}
Most of the code above is boilerplate code (e.g. metrics, using try {} finally for metrics again, etc.) so the core logic of the client-side implementation of the weather service is really focused on marshalling the request and unmarhsalling the response.
With this architecture, it is easy to improve the client (e.g. adding fail-fast, retry & protect logic) without having to change a single line of code for the end-developer.
Considerations
Using such framework for something as simple as the Yahoo Weather API Client, might sound a little overkill. I actually disagree, for a couple of reasons:
- It may not be the first API Client you'll be coding. How many different framework/implementation are you comfortable maintaining over time? Starting by deciding to stick to one framework/methodology (be it this one or another one) up front usually pays off. It's easier to translate existing code following the same rules/guidelines.
- Good RSS libraries in Java are a rarity. This is very specific to this sample application (consuming an RSS feed), but it does show that you could use such framework for anything. As mentioned above, it might help maintaining code if the same approach/methodology is used throughout the application(s).
- The framework can actually be entirely discarded and swapped with something else. It goes beyond just helper classes: the End-developer Domain Layer will stay and only the Implementation Layer will need to be re-coded or replaced with something else. Since there isn't much work to get started with this framework, why not use it first and see later? Unless u know HttpClient like the back of your hand and are comfortable dealing with it upfront.
- If things get more sophisticated (e.g. new web services), the framework provides flexibility to grow and customize when/where needed.
What's important is to try to stick to a simple design and architecture that will help grow one's application with smooth evolution(s) and no revolution.
Applying concepts like anti-fragility will go a long way, especially if done upfront and not as an after-thought.
Resources
[1] Antiope, Luc Pezet, 2014
[2] AWS Java SDK, Amazon Web Services
[3] Yahoo Weather RSS Feed, Yahoo Developer Network
[4] Antifragile Software Design: Abstraction and the Barbell Strategy, Hayim Makabee, May 25th 2014
No comments:
Post a Comment