Using an Async Hyper Client

Lately I've been revamping my GitHub
API Library
to be both more
ergonomic and to use the upcoming 0.11 release of
Hyper which is asynchronous
using Futures and
Tokio under the hood. Mainly this has been due to my
experiences using my library in my GitHub bot
Thearesia. I figured if I'm already
going to be redoing how my library works might as well upgrade to the new
version of Hyper as well and provide some explanations to those wishing
to upgrade their own libraries. I'll be using Hyper at
this commit
for today's example. The docs are good enough for now if you want to dig
into it, but you might need to fish around for what you need. Good news
is this seems to be the last issue
open
before release!

Before we begin I'm assuming you have a cursory knowledge of Futures and
Tokio. If you need an introduction to it I'd highly recommend reading
Andrew Hobden's post over on Asquera's blog called The Future with
Futures
.
It's an informative read and should cover enough of what we need to know
for this example!

Today, we'll go through making a request to the GitHub API asking for ourselves
as a user (in this case I'll be making a request using Thearesia's token
but follow along using your own). This means we'll need HTTPS support so
we'll be importing the hyper-tls library as well. This is a more
involved example than what is in the Hyper repo currently and should
help cover a good few use cases for people.

Let's get started by creating a new project:

cargo new --bin ghub

Then open up our new Cargo.toml file and add these lines:

hyper = { git = "https://github.com/hyperium/hyper" }
hyper-tls = { git = "https://github.com/hyperium/hyper-tls" }
tokio-core = "0.1"
futures = "0.1"

This will give use the newest version of hyper and hyper-tls since it'll
be using the git dependency (if you're from the future and following
along try 0.11 as the version to use instead if it's out and the
examples below are failing). Your Cargo.toml should look something like this
now:

[package]
name = "ghub"
version = "0.1.0"
authors = ["Michael Gattozzi <mgattozzi@gmail.com>"]

[dependencies]
hyper = { git = "https://github.com/hyperium/hyper" }
hyper-tls = { git = "https://github.com/hyperium/hyper-tls" }
tokio-core = "0.1"
futures = "0.1"

Cool we've specified all the dependencies we'll actually need! Now let's setup
the imports in our program. Open up your main.rs file and add the following
lines at the top:

extern crate hyper_tls;
extern crate futures;
extern crate tokio_core;

use tokio_core::reactor::Core;
use futures::{Future, Stream};
use futures::future;

use hyper::{Url, Method, Error};
use hyper::client::{Client, Request};
use hyper::header::{Authorization, Accept, UserAgent, qitem};
use hyper::mime::Mime;
use hyper_tls::HttpsConnector;

Seems like a lot right? I thought so too, but Hyper is a low level HTTP library
and we need this level of granularity to make sure our requests are setup right
for the GitHub API. The good news is that it's not scary! Let's start
building up our request so you can see where all these imports fit in.

First up let's begin crafting the request we'll need. In order to do
this we'll need
a Url
to point the request at:

fn main() {
    let url = Url::parse("https://api.github.com/user").unwrap();

Pretty self explanatory, pass it in a string and that becomes the Url
struct. This functionality and all of it's methods was just like before in
Hyper so you can easily extend the url dynamically or get other
information from it. What, we care about here is that we have it
pointing at the end point we want to use to get data on
ourselves
.

Sweet! Now let's use that to make
a Request
struct. This is what we'll use to set the headers to what we want for the API

    let mut req = Request::new(Method::Get, url);

It takes a Method, an enum representing all the different types of
requests you can make like GET, POST, PUT, DELETE, PATCH, etc., and a Url.
You have access to the handle and headers from this struct via function
calls so that you can change aspects of it that you want to
change. Alright let's get our
Mime value and authorization
token setup for the headers

    let mime: Mime = "application/vnd.github.v3+json".parse().unwrap();
    let token = String::from("token {Your_Token_Here}");

Why is this media type (Mime) needed? Well if you look at the
GitHub docs you can set what
you want to receive back from the API. Usually we would want JSON, so we
ask for that but we also set which version of the API to use with the
vnd.github.v3+ part. We're telling GitHub to use version 3 of the API
because we don't want anything to break if all of a sudden they switch
to version 4 for some reason.

We also need our token to be in the header. From trial and error when
I first used Hyper in the library I realized that GitHub is expecting
input of the form token {Your_Token_Here} for their Authorization
header. It's a bit weird when I first tried to figure it out. Originally
I thought I was supposed to use Hyper's Bearer struct since it had a token
value inside of it but that was not the case apparently.

Let's change the headers of our Request now:

    req.headers_mut().set(UserAgent(String::from("ghub-example")));
    req.headers_mut().set(Accept(vec![qitem(mime)]));
    req.headers_mut().set(Authorization(token));

I'm doing this with the headers_mut().set() way due to some borrowing
errors I ran into and moved values. Meaning I couldn't do:

let mut headers = req.headers_mut();
headers.set()

And then using req later, as req didn't exist anymore. Not sure
if this was a rust or a Hyper issue but this works just fine. If you
figure out a more ergonomic way to do it let me know!

First up we need
a UserAgent
in our headers. Why? According to the
docs GitHub will reject
any request without it! You'll get a 403 when you try to make the
request.

Next up we are going to change our
Accept header
to utilize that Media type we had made earlier. We pass it to
qitem
which wraps it in
a QualityItem
type that Accept is expecting and then we put it in a Vec since
Accept might hold multiple QualityItem values in the header of
a request. We don't have multiple values here but it does need to be in a
Vec.

Lastly we set our
Authorization
with our token by just passing it in to an Authorization struct. Boom
we've setup all of our headers and crafted the request we need. Now
let's start dealing with Futures.

    let mut event_loop = Core::new().unwrap();
    let handle = event_loop.handle();

First up we need to setup an event loop
(Core)
that will handle processing our Future when we need it. We'll also need a
Handle
to that event loop so that our
Client and
HttpsConnector know which event loop to be processed on.

Alright let's set the Client up so we can make connections:

    let client = Client::configure()
        .connector(HttpsConnector::new(4,&handle))
        .build(&handle);

Since we're not using the default version of the client which only does
HTTP we call the configure() function so that we can change the
connector. In this case we're using HttpsConnector from the hyper-tls
library, but presumably anything that implements the Connect trait
should work. This might allow for requests by other protocols if I'm not
mistaken. You might be wondering what that number four is for, well I had
to look at the source code originally since there were no online docs
for it yet. Here's what the relevant comment said, "Takes number of DNS
worker threads." Four is what had been in an older example in the Hyper
repo so I just went with that. You can change that to the number of your
liking. We then tell it to build itself and we now have a Client with HTTPS
support! We're almost done. Let's actually make our Future:

    let work = client.request(req)
        .and_then(|res| {
            println!("Response: {}", res.status());
            println!("Headers: \n{}", res.headers());

            res.body().fold(Vec::new(), |mut v, chunk| {
                v.extend(&chunk[..]);
                future::ok::<_, Error>(v)
            }).and_then(|chunks| {
                let s = String::from_utf8(chunks).unwrap();
                future::ok::<_, Error>(s)
            })
        });

The thing with futures is that it's always expecting some kind of future
to pass on to the next function call chained to it and eventually it
will pass on a value where it's completed when you run the future. So you can
have futures in futures. If you look at the above code it's exactly what we did.
First we tell our client to make a request and pass it our Request struct from
earlier. This gives us a
FutureResponse
which resolves to
a Response if
it works out. When we call and_then() we're saying once you get the
response back do this. In this case we're saying print out the status
(did we get a 200, 403, 404 or something else?) and the headers from the
response. We then call res.body() which creates another Future
called a Body,
which is a stream of
Chunks,
where a Chunk is basically a vector of bytes (Vec<u8>). If you look at the
first part after body() we're getting each Chunk and folding the values
into a single vector and putting it into a future ok so we can chain
another computation. After that we want it to take that vector and turn it into
a String and return that value in a Future! When run it'll return
either an error or the JSON String from the call.

All right, let's run the future to completion and print out the result.

    let user = event_loop.run(work).unwrap();
    println!("We've made it outside the request! \
              We got back the following from our \
              request:\n");
    println!("{}", user);
}

We pass in the future to the event loop and get back the value from it,
in this case a String and then print it out. Sweet. Let's see it in
action then!

Save the file then do:

cargo run

You'll get output similar to this:

Response: 200 OK
Headers: 
Server: GitHub.com
Date: Thu, 09 Mar 2017 18:59:58 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 1450
Status: 200 OK
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 4999
X-RateLimit-Reset: 1489089598
Cache-Control: private, max-age=60, s-maxage=60
Vary: Accept, Authorization, Cookie, X-GitHub-OTP
Vary: Accept-Encoding
ETag: "a6fbebdd7e3ea78f873e2531b6af2562"
Last-Modified: Wed, 15 Feb 2017 16:43:54 GMT
X-OAuth-Scopes: admin:gpg_key, admin:org, admin:org_hook, admin:public_key, admin:repo_hook, delete_repo, gist, notifications, repo, user
X-Accepted-OAuth-Scopes: 
X-GitHub-Media-Type: github.v3; format=json
Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
Content-Security-Policy: default-src 'none'
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-XSS-Protection: 1; mode=block
X-Served-By: 02ea60dfed58b2a09106fafd6ca0c108
X-GitHub-Request-Id: 8572:356D:62252BA:747B25E:58C1A62E

We've made it outside the request! We got back the following from our request:

{"login":"thearesia","id":25337282,"avatar_url":"https://avatars1.githubusercontent.com/u/25337282?v=3","gravatar_id":"","url":"https://api.github.com/users/thearesia","html_url":"https://github.com/thearesia","followers_url":"https://api.github.com/users/thearesia/followers","following_url":"https://api.github.com/users/thearesia/following{/other_user}","gists_url":"https://api.github.com/users/thearesia/gists{/gist_id}","starred_url":"https://api.github.com/users/thearesia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thearesia/subscriptions","organizations_url":"https://api.github.com/users/thearesia/orgs","repos_url":"https://api.github.com/users/thearesia/repos","events_url":"https://api.github.com/users/thearesia/events{/privacy}","received_events_url":"https://api.github.com/users/thearesia/received_events","type":"User","site_admin":false,"name":"Thearesia \"Sword Saint\" van Astrea","company":null,"blog":"https://github.com/mgattozzi/thearesia","location":"Kingdom of Lugnica","email":null,"hireable":null,"bio":"I'm a Github bot maintained by @mgattozzi","public_repos":0,"public_gists":0,"followers":1,"following":0,"created_at":"2017-01-25T03:25:48Z","updated_at":"2017-02-15T16:43:54Z","private_gists":0,"total_private_repos":0,"owned_private_repos":0,"disk_usage":0,"collaborators":0,"two_factor_authentication":true,"plan":{"name":"free","space":976562499,"collaborators":0,"private_repos":0}}

Awesome it all worked out perfectly! Here's what the code looks like all
together:

extern crate hyper;
extern crate hyper_tls;
extern crate futures;
extern crate tokio_core;

use tokio_core::reactor::Core;
use futures::{Future, Stream};
use futures::future;

use hyper::{Url, Method, Error};
use hyper::client::{Client, Request};
use hyper::header::{Authorization, Accept, UserAgent, qitem};
use hyper::mime::Mime;
use hyper_tls::HttpsConnector;

fn main() {
    let url = Url::parse("https://api.github.com/user").unwrap();
    let mut req = Request::new(Method::Get, url);
    let mime: Mime = "application/vnd.github.v3+json".parse().unwrap();
    let token = String::from("token {Your_Token_Here}");
    req.headers_mut().set(UserAgent(String::from("github-rs")));
    req.headers_mut().set(Accept(vec![qitem(mime)]));
    req.headers_mut().set(Authorization(token));

    let mut event_loop = Core::new().unwrap();
    let handle = event_loop.handle();
    let client = Client::configure()
        .connector(HttpsConnector::new(4,&handle))
        .build(&handle);
    let work = client.request(req)
        .and_then(|res| {
            println!("Response: {}", res.status());
            println!("Headers: \n{}", res.headers());

            res.body().fold(Vec::new(), |mut v, chunk| {
                v.extend(&chunk[..]);
                future::ok::<_, Error>(v)
            }).and_then(|chunks| {
                let s = String::from_utf8(chunks).unwrap();
                future::ok::<_, Error>(s)
            })
        });
    let user = event_loop.run(work).unwrap();
    println!("We've made it outside the request! \
              We got back the following from our \
              request:\n");
    println!("{}", user);
}

Conclusion

Future's is changing the game in the Rust world and Hyper is stepping up
to the plate. Once I wrapped my head around it worked it became really
easy to work with. It really helps if you understand how futures work
and if you plan on upgrading to this I'd recommend having a solid understanding
how tokio and futures work together here with Hyper. Hopefully you've
gotten a better understanding how to use the library and come up with
some even more cool or complex things beyond this. I encourage you to
try it out and start prepping your projects for the eventual upgrade.
I've also posted the code on GitHub for
you if you want to just clone the repo. It won't work at all till you
add your token though, so don't try to run it as is!