HTTP Programming

Unit Goals

Overview

Previously, we created our very own application layer protocols. The date and capitalization application protocols were rather trivial, and the our Tic-Tac-Toe protocol (TTTP) was a bit more interesting. Our chat application also had its very own protocol.

But there’s an existing, well-known protocol called HTTP (read about it at Wikipedia) that we can layer applications on top of!

We’ll study HTTP in more detail later, but for now, let’s use it to make an application where clients simply send a message to the server asking for the current datetime. HTTP is a stateless, request-response protocol. For our app, the client will send an HTTP request like:

GET /date

and the HTTP response will be something like:

2019-03-28T04:49:47.952Z

A Simple HTTP Server

The first HTTP server we are going to write will run on port 59999 and feature two endpoints:

• GET / for returning an HTML “page” for the application
• GET /date for returning the server’s current datetime as plain text.

const http = require('http');

http.createServer((request, response) => {
  if (request.method === 'GET' && request.url === '/date') {
    response.writeHead(200, { 'Content-Type': 'text/plain' });
    response.end(new Date().toISOString(), 'utf-8');
  } else if (request.method === 'GET' && request.url === '/') {
    response.writeHead(200, { 'Content-Type': 'text/html' });
    response.end('<h1>The Date Client</h1><a href="/date">Get date from server</a>');
  } else {
    response.writeHead(404, { 'Content-Type': 'text/plain' });
    response.end('Sorry, that’s not there');
  }
}).listen(59999);

console.log('Date Server running at port 59999');

Discussion:

• Because HTTP is so common, Node.js has a built in module called http.
• HTTP requests begin with a method (such as GET, PUT, DELETE) and a URI (which is an identifier for the resource being created, retrieved, updated, deleted, or otherwise manipulated.
• HTTP responses have a response code, such as 200 for OK, 201 for CREATED, 404 for NOT FOUND, 400 for BAD REQUEST, and dozens more.
• Both requests and responses have metadata in their headers. One of the most common header is Content-type, which we used here to distinguish our plain text responses from our HTML ones.
• We use writeHead() to write headers, and write() and/or end() to write the response body.

Run the server:

$ node datewebserver.js
Date Server running at port 59999

HTTP Clients

There are at least four ways to use this server. For illustration, we’ll run the server on localhost. For classwork, we’ll put the clients and servers on different machines.

nc

First, we can just use nc (and if you are taking a networking class, you should do this). With the server running on your local box:

$ nc localhost 59999
GET /date

HTTP/1.1 200 OK
Content-Type: text/plain
Date: Thu, 28 Mar 2019 04:49:47 GMT
Connection: close

2019-03-28T04:49:47.952Z

You have to hit the Enter key TWICE after entering the GET line. That is mandated by the protocol, as we’ll soon see.

That response was pretty large.

That’s right, an HTTP response contains a lot of metadata, which we’ll cover later. And yes, part of the response metadata is the server date, which might make you wonder why anyone would ever write a date server....

What about the other endpoint?

$ nc localhost 59999
GET /

HTTP/1.1 200 OK
Content-Type: text/html
Date: Thu, 28 Mar 2019 05:05:11 GMT
Connection: close

<h1>The Date Client</h1><a href="/date">Get date from server</a>

And what about that 404 thing?

$ nc localhost 59999
GET /whatever

HTTP/1.1 404 Not Found
Content-Type: text/plain
Date: Thu, 28 Mar 2019 05:06:58 GMT
Connection: close

Sorry, that’s not there

By the way, Node’s http module does some pretty good parsing of requests and handles a good deal of the protocol for you. These examples will just give you an idea of what’s in HTTP:

$ nc localhost 59999
bzusdfyiwuef
HTTP/1.1 400 Bad Request

cURL

The second way is to use curl. Use the -i option to see the whole response:

$ curl -i -X GET localhost:59999/date
HTTP/1.1 200 OK
Content-Type: text/plain
Date: Thu, 28 Mar 2019 05:08:42 GMT
Connection: keep-alive
Transfer-Encoding: chunked

2019-03-28T05:08:42.217Z

Without -i you just get the response body:

$ curl -X GET localhost:59999/date
2019-03-28T05:10:15.931Z

Web Browsers

Okay, yes, the third way is to use a web browser, which speaks HTTP already. With your server running on your local machine, enter http://localhost:59999 into the browser’s address bar, hit Enter, and see the page. Click on the link to hit the other endpoint.

Also, enter the URI of the date endpoint directly: http://localhost:59999/date. That worked great, didn’t it?

Programmatically, from a Server or Command Line

Your favorite programming language should have an http library for making requests. If you are making requests programmatically, you’ll have to examine the response in some detail and take one of several actions depending on the status code. You’ll also have to check and process all the response headers, and read in the response body. The code can get clunky. Here’s how it looks in JavaScript using the raw http library:

const http = require('http');

http.get('http://localhost:59999/date', (res) => {
  if (res.statusCode !== 200) {
    console.log(`Received status code ${res.statusCode}`);
    res.resume();
    return;
  }
  let dateString = '';
  res.on('data', (chunk) => { dateString += chunk; });
  res.on('end', () => console.log(dateString));
}).on('error', (e) => {
  console.error(`Request failed: ${e.message}`);
});

DON’T PANIC!!! If you npm install request then it’s much, much easier:

const request = require('request');

request('http://localhost:59999/date', (error, response, body) => {
  if (error) {
    console.error(error);
  } else if (response.statusCode !== 200) {
    console.log(`Received status code ${response.statusCode}`);
  } else {
    console.log(body);
  }
});

Oh wait, aren’t promises much more awesome than those silly callbacks?

$ npm install request request-promise --save

const rp = require('request-promise');

rp('http://localhost:59999/date').then((body) => {
  console.log(body);
}).catch((error) => {
  console.log(error.message || 'Error');
});

Interesting.... request-promise treats both non-sucess HTTP responses and failures to connect as reasons to reject a promise.

Programmatically, from a Client-Side Script?

Let’s review the different ways we’ve hit the server so far:

• Running nc and typing in the HTTP request by hand.
• Typing the URIs into a web browser address bar, or (equivalently) clicking on a link in an HTML document. This only worked because we were making GET requests and we did not have any special headers to set.
• Running request-containing scripts from the command line, or other server (but not inside of a browser).

Now what if wanted to write our own web client, and not use the web page delivered by that web server? In other words: can we write a web app that uses the date web server just for its data, and not use its HTML?

This is what fetchis for. Let’s try:

<html>
  <head>
    <meta charset="utf-8">
    <title>Date Web Client</title>
  </head>
  <body>
    <p>IP Address of Server: <input id="ip" type="text"></p>
    <p><button>Get Current Date</button></p>
    <p id="response"></p>
    <script>
      document.querySelector('button').addEventListener('click', () => {
        const host = document.querySelector('#ip').value;
        const messageArea = document.querySelector('#response');
        fetch(`http://${host}:59999/date`).then((res) => {
          messageArea.textContent = `${res.status} `;
          return res.text();
        }).then((data) => {
          messageArea.textContent += data;
        }).catch((error) => {
          messageArea.textContent = error;
        })
      });
    </script>
  </body>
</html>

Now if we load this file into the browser, enter localhost into the IP box, and hit the button, we get:

Access to fetch at 'http://localhost:59999/date' from origin 'null' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

CORS? That’s Cross Origin Resource Sharing. Basically web browsers stop you from fetching resources from sites (host+port) other than that which served you the JavaScript you are running. How can you get around this? If you really were writing a web service, one that just delivers data and not web pages, then you can return a response header Access-Control-Allow-Origin, allowing access to certain web clients or to everyone (with the value *).

Classwork: Get into groups of three. One person wll run the server. The other two will use the HTML client and try to hit the server. Both should see CORS errors. Then the person writing the server will add

    'Access-control-allow-origin': 'http://ip_address_of_one_of_the_clients'

into the argument to writeHead for the /date endpoint. Now only that one client will be able to hit the server. Next, the server programmer will change that header to

    'Access-control-allow-origin': '*'

and the group should verify that both clients can now hit the server.

The other way to “get around” CORS is to just make these kinds of calls from other servers. It’s only the call from the browser that is impacted by the same-origin policy.

HTTP Requests and Responses

Let’s look at HTTP, the protocol, in a little more detail.

Formats

HTTP Request format:

Example:

PUT https://ehr.example.com/types/Lab.Diagnosis HTTP/1.1
authorization: Bearer hbGciOiJIUz8I1NiJ9JzdWIiOiJh
content-type: application/json
Origin: https://e1ak8fcq37hprs.cloudfront.net
Referer: https://e1ak8fcq37hprs.cloudfront.net/
User-Agent: Mozilla/5.0 AppleWebKit/537.36Chrome/72.0.3626.96

{"display":"Lab Diagnosis","schema":{"code":"string"}}

HTTP Response format:

Example:

HTTP/1.1 200 OK
access-control-allow-headers: Content-Type,Authorization
access-control-allow-methods: GET,PUT,POST,DELETE,PATCH,HEAD,OPTIONS
access-control-allow-origin: *
content-encoding: gzip
content-length: 94
content-type: application/json
date: Thu, 28 Mar 2019 16:10:20 GMT
status: 200
via: 1.1 9f6b9465776576cba700d600678836e.cloudfront.net (CloudFront)
x-amz-apigw-id: XQq8-Edb8625FbxA=
x-amzn-remapped-content-length: 92
x-amzn-requestid: fc22c7c7-5173-11e9-b44d-b393bfa59355
x-cache: Miss from cloudfront

{
  "post_type": {
    "display": "Lab Diagnosis",
    "name": "Lab.Diagnosis",
    "schema": {
      "code":"string"
    }
  }
}

More details at MDN. Also you should check out the RFCs.

Request Methods

HTTP was designed around the idea of an infinite number of types of resources (nouns) that are manipulated with a very small number of methods (verbs). The methods are:

Headers

Both requests and responses can be packed with metadata which are called headers. Here are some of the common ones:

Check out the list of registered headers at IANA. Note that you can always add your own headers for your own organization or application as long as it starts with x-.

Media Types

Information is generated, transmitted, and stored as bit sequences. A media type describes the way in which bits are to be interpreted. Use media types as the values of the Content-type and other headers. Examples:

• text/html
• image/png
• audio/mp4
• video/H264

More notes here.

Response Status Codes

Every HTTP response begins with a three-digit status code. Most are listed below. For a nice summary, see Wikipedia; for the official documentation, see RFC 7231, Section 6.

HTTP in Practice

Did you notice that our simple webserver had two endpoints, one returning an HTML page and the other just returning raw data? That’s great. HTTP is designed to accept and return resources, which can be raw data, text, HTML, images, videos, whatever.

Some HTTP-based servers are centered around delivering HTML and related content. These are often called webapps. Some are all about exchanging pure data, usually in JSON (or XML, which is still around). These are often called web services or web APIs. If you follow certain principles in your web API architecture, you might get to call your API a REST API.

HTTP Frameworks

In practice, you’d never write an application using the http module directly. You would use a framework written by someone else. That someone else probably built their framework using http directly so you don’t have to.

In no particular order, here are some frameworks that I happen to know about. They are offered without any endorsement:

Later in the course we’ll learn how to build some interesting web applications, using one or more of these frameworks. Remember that HTTP is just a request-response protocol; to build a complete web app there’s tons to learn about HTML, CSS, and JavaScript (for the front-end), and data stores and similar things (for the back-end). Much more to come.

Limitations of HTTP

HTTP is great for requests and responses. But what about applications like interactive games or chats, where the server has to notify the client at any time (not just as a response)? Here the overhead of the protocol is inefficient.

For games and notifications, WebSockets are almost always better!

Appendix: Node’s http Module

Because HTTP is so massively popular, Node.js comes with a built-in module called http to help you write HTTP servers! How convenient! Here are some of the module highlights:

Summary