Back to clever-cloud.com

Well, if you need some logos, we provide these:

svg ·  png
Red Clever Cloud logo
svg ·  png
svg ·  png
White Clever Cloud logo
svg · png
White Clever Cloud logo
svg · png
Posted at March 24, 2015 — Engineering

Building an extensible HTTP client for GitHub in Java

At Clever Cloud, we use a lot of Java. A large part of our API is written using JEE (aka Java Enterprise Edition). Here is how we implemented the http client we use for our Github integration.

The standard HTTP implementation in JEE is Jersey, so that's what we'll use here.

A bit of background first

Our API is designed in a "micro-services" way. There's a central part and various external components.

Here, we'll be dealing with the central part, which itself is internally designed in a sort-of "micro-services" way. JEE offers a powerful dependency injection mechanism called CDI, which we use to build a lot of small tools we can then use just as we need.

Serialization

When you're building APIs and clients, one of the problematics is about serialization. We made a tool to do just that which you'll see in the code as MapperHelper, namely mh. It's basically a wrapper around jackson's ObjectMapper.

Let's get to work

We'll be working on an internal tool called GithubAPIHelper. We'll not get through the whole of it, but you'll quickly get the concept.

First, some standard JEE boilerplate injecting required tools using CDI.

@Stateless
public class GithubAPIHelper {

   @Inject
   private MapperHelper mh;
   @Inject
   private LogsHelper lh;

   private final static String GITHUB_URL = "https://api.github.com";
}

Now, we want to build an HTTP client using jersey-client, this is pretty straightforward. Btw, we create a method to make the client directly point at Github.

private Client getClient() {
   if (client == null) {
      client = ClientBuilder.newClient(new ClientConfig().property(ClientProperties.FOLLOW_REDIRECTS, true));
   }
   return client;
}

private WebTarget getTarget() {
   return getClient().target(GITHUB_URL);
}

What about authentication? Let's create a new method to get an authenticated request to Github:

private Invocation.Builder getBuilder(WebTarget target, String token) {
   return target.request().header("Authorization", "token " + token);
}

Ok, now that we have those basic pieces in place, we can start requesting for real.

Let's get the basic user profile, for starters

First, we create a basic data structure that we'll use to deserialize the payload from Github. I'll skip most of the available fields here as they're not needed.

public class GithubOwner implements Serializable {

   public int id;
   public String name;
   public String login;
   public String email;
   public String avatar_url;
}

Now, you should all be yelling at me about those public fields. Let me remember you that this data structure is only used for deserializing purpose, it won't ever be used for anything else, so what's the point in complicating things?

Now let's get back to business, we have a way to construct authenticated requests, and a model in which to store the data. So let's do it!

First, let's get to the right endpoint: /user

   getTarget().path("/user");

Now, let's actually issue the request, and get a response

   Response r = getBuilder(getTarget().path("/user"), token).get();

Let's deserialize the data from github

   GithubOwner owner = mh.readValue(r.readEntity(String.class), GithubOwner.class);

And here we are, with some integrity checks added:

   public GithubOwner getSelf(String token) {
      Response r = getBuilder(getTarget().path("/user"), token).get();
      if (r.getStatus() < 300) {
         try {
            GithubOwner owner = mh.readValue(r.readEntity(String.class), GithubOwner.class);
            return owner;
         } catch (IOException e) {
            lh.log(GithubAPIHelper.class.getName(), Level.SEVERE, null, e);
         }
      }
      return null;
   }

Dealing with pagination

Getting the user's profile was quite easy and straightforward, but everything isn't that smooth.

Let's say we now want all the repositories, instinctively, here is what we'd do:

public class GithubRepository implements Serializable {

   public long id;
   public String name;
   public String description;
   public GithubOwner owner;
   @JsonProperty(value = "private")
   public boolean isPrivate;
   public String ssh_url;
   public String git_url;
}
   private TypeSafeList<GithubRepository> getSelfRepositories(String token) {
      TypeSafeList<GithubRepository> repos = new TypeSafeArrayList<>();
      Response r = getBuilder(getTarget().path("/user/repos"), token).get();
      if (r.getStatus() < 300) {
         try {
            List<GithubRepository> ms = mh.readValue(r.readEntity(String.class), new TypeReference<List<GithubRepository>>() {
            });
            repos.addAll(new TypeSafeArrayList<>(ms));
         } catch (IOException e) {
            lh.log(GithubAPIHelper.class.getName(), Level.SEVERE, null, e);
         }
      }
      return repos;
   }

That will actually work, but you won't get all of them, only the first 20 or so. If you want all of them, you'll have to follow the pagination.

First, we want to get the link marked as rel=next in the Link http header.

   private String getNextLink(Response r) {
      String link = r.getHeaderString("Link");   /* <http://foobar>; rel="next", <http://blah/>; rel=last */
      if (link != null) {
         String[] links = link.split(",");
         for (String l : links) {               /* <http://foobar>; rel="next" */
            if (l.contains("rel=\"next\"")) {
               String[] tmp1 = l.split("<", 2);
               if (tmp1.length == 2) {
                  return tmp1[1].split(">", 2)[0];  /* http://foobar */
               }
            }
         }
      }
      return null;
   }

Next, we want to request this url and get the next data to our initial request

   private Response getNextData(Response r, String token) {
      String link = getNextLink(r);
      if (link == null)
         return null;
      return getBuilder(getClient().target(link), token).get();
   }

Now let's put all the pieces together, addind a loop to retrieve all the data

   private TypeSafeList<GithubRepository> getSelfRepositories(String token) {
      TypeSafeList<GithubRepository> repos = new TypeSafeArrayList<>();
      Response r = getBuilder(getTarget().path("/user/repos"), token).get();
      while (r != null) {
         if (r.getStatus() < 300) {
            try {
               List<GithubRepository> ms = mh.readValue(r.readEntity(String.class), newhTypeReference<List<GithubRepository>>() {
               });
               repos.addAll(new TypeSafeArrayList<>(ms));
            } catch (IOException e) {
               lh.log(GithubAPIHelper.class.getName(), Level.SEVERE, null, e);
            }
         }
         r = getNextData(r, token);
      }
      return repos;
   }

And voila, you now have a working Github HTTP client. Once you got these, the other endpoints are really easy.

Tags:

Check out our Blog

We write about programming, security and Clever Cloud products

Join thousands of developers already using Clever Cloud. And try it for free!

Create your account now to start your apps and collaborate within a few minutes. For free with 20€ offered.

By signing up, your agree to our terms of use
Or
Sign in with Github