Invoking Web APIs
What is an API?
Here’s what Wikipedia says:
An application programming interface (API) is an interface or communication protocol between different parts of a computer program intended to simplify the implementation and maintenance of software. An API may be for a web-based system, operating system, database system, computer hardware, or software library. An API specification can take many forms, but often includes specifications for routines, data structures, object classes, variables, or remote calls.
Many programming languages provide “APIs” for their standard libraries or third-party libraries. These APIs are generally documented with detailed descriptions of the functions or classes they provide. These are not Web APIs, though.
What is a Web API?
A Web API is one that provides (resources and) services via HTTP or HTTPS. Yep, that’s all it is.
HTTP
If you are unfamiliar with HTTP or HTTPS, they are (at the most basic-level) request-response protocols. HTTPS is just HTTP with security at the transport layer. I have some brief notes on HTTP or see Wikipedia.
In many circles, the term “API” refers to a Web API.
Someone who knows as much as anyone about APIs is Kin Lane.
There are thousands of Web APIs out there. Some are free, some cost money. Some require authentication, some do not. Where can you find API? Some nice folks have made nice lists of APIs for you! Check out:
- The Big List of Free and Open Public APIs
- API List
- Public APIs (this one is really good)
These lists nicely organize APIs by category, such as: Animals, Art & Design, Books, Business, Calendar, Cloud Storage & File Sharing, Continuous Integration, Cryptocurrency, Development, Environment, Events, Finance, Food & Drink, Games & Comics, Geocoding, Government, Health, Jobs, Machine Learning, Music, News, Open Data, Open Source Projects, Patent, Personality, and many more.
Basic Concepts
An API is accessed via a collection of endpoints, which are combine HTTP verbs (such as GET, PUT, POST, DELETE) with a URL. Here are some example endpoints from the Jikan API. Traditionally, query parameters are shown in braces.
GET /anime?q=mob
GET /anime/{id}
GET /anime/{id}/episodes
GET /anime/{id}/episodes/{page}
GET /anime/{id}/reviews
GET /random/manga
GET /people/{id}/pictures
Most APIs you will encounter with free public information will use GET requests. If you use a service for which you have a personal account, you’ll see POSTs and PUTs for creating and updating information. Examples:
GitHub POST /repos/{owner}/{repo}/issues (create an issue)
GitHub PUT /repos/{owner}/{repo} (update repo details)
Slack POST /chat.postMessage (send a message)
Stripe POST /v1/customers (create a customer)
Stripe PUT /v1/customers/{id} (update a customer)
Twillio POST /Messages.json (send a message)
To invoke an API endpoint, you inject the API’s base url and any necessary headers, query parameters, and request body to make a complete HTTP request. Then an HTTP response comes back, complete with a status code, headers, and content. If none of that makes sense right now, don’t worry—we’ll cover HTTP later. And all you need to know for now will be in the examples below.
Example: For the Jikan API, the base url is https://api.jikan.moe/v4, so to get episodes for Shingeki no Kyojin, you send a request to the URL https://api.jikan.moe/v4/anime/16498/episodes.
Example: For the Studio Ghibli API, the base url is https://ghibliapi.vercel.app, so to get information about the film My Neighbor Totoro, you send a request to the URL https://ghibliapi.vercel.app/films/58611129-2dbc-4a81-a72f-77ddfc1b1b49.
Time for details!
Invocation
There are many ways to invoke a Web API.
curl
curl is a super easy-to-use command line utility. We just have time for a quick example:
$ curl -X GET -i 'https://api.jikan.moe/v4/anime?q=Death+Note&page=1&limit=1'
HTTP/1.1 200 OK
Server: nginx/1.24.0
Date: Thu, 01 Aug 2024 01:20:50 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: must-revalidate, private
pragma: no-cache
expires: -1
access-control-allow-origin: *
Vary: Accept-Encoding
X-Cache-Status: MISS
X-Powered-By: the-power-of-friendship
{"pagination":{"last_visible_page":36,"has_next_page":true,"current_page":1,"items":{"count":1,"total":36,"per_page":1}},"data":[{"mal_id":1535,"url":"https:\/\/myanimelist.net\/anime\/1535\/Death_Note","images":{"jpg":{"image_url":"https:\/\/cdn.myanimelist.net\/images\/anime\/1079\/138100.jpg","small_image_url":"https:\/\/cdn.myanimelist.net\/images\/anime\/1079\/138100t.jpg","large_image_url":"https:\/\/cdn.myanimelist.net\/images\/anime\/1079\/138100l.jpg"},"webp":{"image_url":"https:\/\/cdn.myanimelist.net\/images\/anime\/1079\/138100.webp","small_image_url":"https:\/\/cdn.myanimelist.net\/images\/anime\/1079\/138100t.webp","large_image_url":"https:\/\/cdn.myanimelist.net\/images\/anime\/1079\/138100l.webp"}},"trailer":{"youtube_id":"Vt_3c8BgxV4","url":"https:\/\/www.youtube.com\/watch?v=Vt_3c8BgxV4","embed_url":"https:\/\/www.youtube.com\/embed\/Vt_3c8BgxV4?enablejsapi=1&wmode=opaque&autoplay=1","images":{"image_url":"https:\/\/img.youtube.com\/vi\/Vt_3c8BgxV4\/default.jpg","small_image_url":"https:\/\/img.youtube.com\/vi\/Vt_3c8BgxV4\/sddefault.jpg","medium_image_url":"https:\/\/img.youtube.com\/vi\/Vt_3c8BgxV4\/mqdefault.jpg","large_image_url":"https:\/\/img.youtube.com\/vi\/Vt_3c8BgxV4\/hqdefault.jpg","maximum_image_url":"https:\/\/img.youtube.com\/vi\/Vt_3c8BgxV4\/maxresdefault.jpg"}},"approved":true,"titles":[{"type":"Default","title":"Death Note"},{"type":"Synonym","title":"DN"},{"type":"Japanese","title":"\u30c7\u30b9\u30ce\u30fc\u30c8"},{"type":"English","title":"Death Note"}],"title":"Death Note","title_english":"Death Note","title_japanese":"\u30c7\u30b9\u30ce\u30fc\u30c8","title_synonyms":["DN"],"type":"TV","source":"Manga","episodes":37,"status":"Finished Airing","airing":false,"aired":{"from":"2006-10-04T00:00:00+00:00","to":"2007-06-27T00:00:00+00:00","prop":{"from":{"day":4,"month":10,"year":2006},"to":{"day":27,"month":6,"year":2007}},"string":"Oct 4, 2006 to Jun 27, 2007"},"duration":"23 min per ep","rating":"R - 17+ (violence & profanity)","score":8.62,"scored_by":2777603,"rank":84,"popularity":2,"members":3949348,"favorites":175439,"synopsis":"Brutal murders, petty thefts, and senseless violence pollute the human world. In contrast, the realm of death gods is a humdrum, unchanging gambling den. The ingenious 17-year-old Japanese student Light Yagami and sadistic god of death Ryuk share one belief: their worlds are rotten.\n\nFor his own amusement, Ryuk drops his Death Note into the human world. Light stumbles upon it, deeming the first of its rules ridiculous: the human whose name is written in this note shall die. However, the temptation is too great, and Light experiments by writing a felon's name, which disturbingly enacts his first murder.\n\nAware of the terrifying godlike power that has fallen into his hands, Light\u2014under the alias Kira\u2014follows his wicked sense of justice with the ultimate goal of cleansing the world of all evil-doers. The meticulous mastermind detective L is already on his trail, but as Light's brilliance rivals L's, the grand chase for Kira turns into an intense battle of wits that can only end when one of them is dead.\n\n[Written by MAL Rewrite]","background":"","season":"fall","year":2006,"broadcast":{"day":"Wednesdays","time":"00:56","timezone":"Asia\/Tokyo","string":"Wednesdays at 00:56 (JST)"},"producers":[{"mal_id":29,"type":"anime","name":"VAP","url":"https:\/\/myanimelist.net\/anime\/producer\/29\/VAP"},{"mal_id":1003,"type":"anime","name":"Nippon Television Network","url":"https:\/\/myanimelist.net\/anime\/producer\/1003\/Nippon_Television_Network"},{"mal_id":1365,"type":"anime","name":"Shueisha","url":"https:\/\/myanimelist.net\/anime\/producer\/1365\/Shueisha"},{"mal_id":1791,"type":"anime","name":"D.N. Dream Partners","url":"https:\/\/myanimelist.net\/anime\/producer\/1791\/DN_Dream_Partners"}],"licensors":[{"mal_id":119,"type":"anime","name":"VIZ Media","url":"https:\/\/myanimelist.net\/anime\/producer\/119\/VIZ_Media"}],"studios":[{"mal_id":11,"type":"anime","name":"Madhouse","url":"https:\/\/myanimelist.net\/anime\/producer\/11\/Madhouse"}],"genres":[{"mal_id":37,"type":"anime","name":"Supernatural","url":"https:\/\/myanimelist.net\/anime\/genre\/37\/Supernatural"},{"mal_id":41,"type":"anime","name":"Suspense","url":"https:\/\/myanimelist.net\/anime\/genre\/41\/Suspense"}],"explicit_genres":[],"themes":[{"mal_id":40,"type":"anime","name":"Psychological","url":"https:\/\/myanimelist.net\/anime\/genre\/40\/Psychological"}],"demographics":[{"mal_id":27,"type":"anime","name":"Shounen","url":"https:\/\/myanimelist.net\/anime\/genre\/27\/Shounen"}]}]}
That example used two options:
-Xto specify the HTTP verb GET. Actually GET is the default so we didn’t have to include it.-ito display the status line and headers. Without the-iyou see only the response.
HTTP ResponsesWe look at HTTP in more detail later but for now, notice that a response contains:
- First, a status line with the protocol version, a response code, and the description of the response code
- Zero or more response headers describing the response
- a blank line
- The response body
Details can be found in the HTTP documentation, but all you need for now is to understand the overall structure. Learning about the various response codes, is of course, always fun.
If you have the program jq installed on your machine, you can pipe JSON responses to them for some good-looking output. Try:
$ curl 'https://api.jikan.moe/v4/anime?q=Death+Note&page=1&limit=1' | jq
From the browser, using fetch
Every modern web browser implements the fetch function. After practicing a bit below, read Mozilla’s documentation on fetch for further deatails. It’s pretty good.
From Vanilla JS
Here is a complete example of a simple HTML page that hits the Jikan API on load to list the id numbers and titles of the episodes of the (first season) of the anime Shingeki No Kyojin:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>進撃の巨人</title>
<style>table {border-collapse: collapse} th, td {border: 1px solid black}</style>
</head>
<body>
<h1>Shingeki no Kyojin Season 1 Episodes</h1>
<table>
<tr><th>Episode</th><th>Title</th></tr>
<tbody id="episodes"></tbody>
</table>
<script>
addEventListener('load', () => {
const url = 'https://api.jikan.moe/v4/anime/16498/episodes';
fetch(url).then(r => r.json()).then(response => {
response.data.forEach(episode => {
const row = document.createElement("tr");
const idCell = document.createElement("td");
idCell.textContent = episode.mal_id;
const titleCell = document.createElement("td");
titleCell.textContent = episode.title;
row.appendChild(idCell);
row.appendChild(titleCell);
document.querySelector("#episodes").appendChild(row);
})
})
})
</script>
</body>
</html>When loading this (or any other) HTML page in a browser that makes JavaScript fetch calls, you can open up your browser tools and look at the network tab to see complete details both the HTTP requests and the HTTP responses for every request made on the page. A sample view from the page above might be:
From React
You know, unless your app is trivial, using a framework like React is a bit nicer. Here’s the previous example in React. Note that API invocations go into effects:
Server Side
When we say “server side” we mean either from a literal server or from a script you run off the command line. We’ll look at simple command line scripts in different programming languages.
From Node
Here is a command line JavaScript app that does the same as the previous example. It uses console.table to make a beautiful output. Node is pretty neat.
const response = await fetch("https://api.jikan.moe/v4/anime/16498/episodes")
const body = await response.json()
console.table(body.data.map((episode) => ({ id: episode.mal_id, title: episode.title })))
$ node aot.js ┌─────────┬────┬────────────────────────────────────────────────────────────────────┐ │ (index) │ id │ title │ ├─────────┼────┼────────────────────────────────────────────────────────────────────┤ │ 0 │ 1 │ 'To You, 2,000 Years in the Future: The Fall of Shiganshina (1)' │ │ 1 │ 2 │ 'That Day: The Fall of Shiganshina (2)' │ │ 2 │ 3 │ 'A Dim Light in the Darkness of Despair: Humanity Rises Again (1)' │ │ 3 │ 4 │ 'Night of the Graduation Ceremony: Humanity Rises Again (2)' │ │ 4 │ 5 │ 'First Battle: Battle of Trost (1)' │ │ 5 │ 6 │ 'The World the Girl Saw: Battle for Trost (2)' │ │ 6 │ 7 │ 'The Small Blade: The Battle for Trost (3)' │ │ 7 │ 8 │ 'Hearing the Heartbeat: The Battle for Trost (4)' │ │ 8 │ 9 │ 'Where the Left Arm Went: The Battle for Trost (5)' │ │ 9 │ 10 │ 'Response: The Battle for Trost (6)' │ │ 10 │ 11 │ 'Icon: The Battle for Trost (7)' │ │ 11 │ 12 │ 'Wound: The Battle for Trost (8)' │ │ 12 │ 13 │ 'Primal Desires: The Battle for Trost (9)' │ │ 13 │ 14 │ "Still Can't See: Night Before the Counteroffensive (1)" │ │ 14 │ 15 │ 'Special Ops Squad: Night Before the Counteroffensive (2)' │ │ 15 │ 16 │ 'What Should Be Done: Night Before the Counteroffensive (3)' │ │ 16 │ 17 │ 'Female Titan: 57th Expedition Beyond the Walls (1)' │ │ 17 │ 18 │ 'Forest of Giant Trees: 57th Expedition Beyond the Walls (2)' │ │ 18 │ 19 │ 'Bite: 57th Expedition Beyond the Walls (3)' │ │ 19 │ 20 │ 'Erwin Smith: 57th Expedition Beyond the Walls (4)' │ │ 20 │ 21 │ 'Crushing Blow: 57th Expedition Beyond the Walls (5)' │ │ 21 │ 22 │ 'The Defeated: 57th Expedition Beyond the Walls (6)' │ │ 22 │ 23 │ 'Smile: Raid on Stohess District (1)' │ │ 23 │ 24 │ 'Mercy: Raid on Stohess District (2)' │ │ 24 │ 25 │ 'The Wall: Raid on Stohess District (3)' │ └─────────┴────┴────────────────────────────────────────────────────────────────────┘
From Python
From Python, use the famous requests library. Here is our recurring example:
import requests
body = requests.get('https://api.jikan.moe/v4/anime/16498/episodes', timeout=5)
for episode in body.json()['data']:
print(episode['mal_id'], episode['title'])
$ python aot.py 1 To You, 2,000 Years in the Future: The Fall of Shiganshina (1) 2 That Day: The Fall of Shiganshina (2) 3 A Dim Light in the Darkness of Despair: Humanity Rises Again (1) 4 Night of the Graduation Ceremony: Humanity Rises Again (2) 5 First Battle: Battle of Trost (1) 6 The World the Girl Saw: Battle for Trost (2) 7 The Small Blade: The Battle for Trost (3) 8 Hearing the Heartbeat: The Battle for Trost (4) 9 Where the Left Arm Went: The Battle for Trost (5) 10 Response: The Battle for Trost (6) 11 Icon: The Battle for Trost (7) 12 Wound: The Battle for Trost (8) 13 Primal Desires: The Battle for Trost (9) 14 Still Can't See: Night Before the Counteroffensive (1) 15 Special Ops Squad: Night Before the Counteroffensive (2) 16 What Should Be Done: Night Before the Counteroffensive (3) 17 Female Titan: 57th Expedition Beyond the Walls (1) 18 Forest of Giant Trees: 57th Expedition Beyond the Walls (2) 19 Bite: 57th Expedition Beyond the Walls (3) 20 Erwin Smith: 57th Expedition Beyond the Walls (4) 21 Crushing Blow: 57th Expedition Beyond the Walls (5) 22 The Defeated: 57th Expedition Beyond the Walls (6) 23 Smile: Raid on Stohess District (1) 24 Mercy: Raid on Stohess District (2) 25 The Wall: Raid on Stohess District (3)
Postman
Postman is a pretty cool app where you can hit API endpoints from a very convenient graphical interface. It’s incredibly fully-featured. Here’s a view of it using the endpoint from our recurring example:
Wrappers
Many times an API provider will provide you with a library (sometimes called a “wrapper” or an “SDK”) containing a set of functions that internally make the API call (via fetch).
CORS
Try the endpoint https://loripsum.net/api in your browser’s address bar. Nice, right? Now try it from curl:
$ curl 'https://loripsum.net/api'
And try it from POSTMAN. And now run this JavaScript program from the command line:
fetch('https://loripsum.net/api')
.then(response => response.json())
.then(data => console.log(data))
Four for four! Now run it from your browser by loading this HTML document directly:
<body>
<script>
fetch('https://loripsum.net/api')
.then(response => response.json())
.then(data => console.log(data))
</script>
</body>
A blank screen! Check your console:
A “CORS error”! Whatever is a CORS error? Why do we only get them in the browser? And why did we not get them from previous APIs we looked at, even when we used fetch?
Well CORS is a huge topic, but here’s what’s important to know right now: If a server does not send back the response header Access-Control-Allow-Origin with a value including the url the JavaScript was delivered from, the browser will not give you the fetched data. Instead, it just gives you a CORS error. Many public APIs are happy to provide Access-Control-Allow-Origin: * which means fetches from anywhere are allowed. The Poké, Jikan, and Studio Ghibli APIs we saw earlier all did that. So do a lot of others. But loremipsum does not.
This header does not affect API calls from the command line, servers, or Postman. It’s just a browser thing.
For more on CORS, including why it is just a browser thing and how it aids security, see Mozilla’s CORS documentation and Wikipedia’s CORS article.
If you need to use an API from fetch() that gives you CORS errors, you’ll need to stand up a proxy server.
We’ll make a proxy server in class.
Understanding Authentication
Many APIs require that at least some, and maybe all, of their endpoints, be authenticated. That means that some information in the request must be present that identifies the requestor as a valid “user.” This information could be:
- Credentials, such as a username and password
- A token
- One or more “API keys”
In a well-designed API, this information cannot be easily guessed or stolen (via, say, MitM attacks). Some of the big ideas in authentication are:
- You shouldn’t pass username and password on every request.
- You can pass the username and password in a “login” request, then the server gives you back a token; then that token is used for subsequent requests.
- Tokens should be relatively short-lived.
- There needs to be a way to invalidate tokens (in case they are compromised or may have been compromised).
- There needs to always be transport-layer security (TLS) in place.
- Some requests can be digitally signed
- Some requests can have unique tokens so they cannot be replayed
- Often authentication services can be delegated to third parties (e.g. “login in Google”)
Now, did you find a cool API and asks you to pass an API key on each request? And are you hitting that API directly from a browser? Uh-oh. Anyone using your script will see your key. That’s horrible. Keys and other credentials have to be hidden.
Let’s talk about ways to hide credentials in a browser-based app. Guess what? We’ll use a proxy server.
Choosing an API for a Project
If you are getting started with APIs and would like to find some to use in a hobby or class project, look for favorable answers to these three questions:
Is it free?
Because you just want to learn
Can it be used without authen-
tication?
Is it CORS-enabled?
Because you don’t want to bother with proxies
There’s So Much More
All we covered is how to hit an endpoint of an API with an HTTP GET request and deal with a JSON response that we assumed would come back without any errors.
There is way more to APIs that just GETs, and seriously, you are going to have errors. A lot of errors. Our code needs to deal with them.
The remaining 99% of how to invoke APIs are coming up in future notes and can be found in your self-study.
Recall Practice
Here are some questions useful for your spaced repetition learning. Many of the answers are not found on this page. Some will have popped up in lecture. Others will require you to do your own research.
- What is a Web API? A Web API is one that provides (resources and) services via HTTP or HTTPS.
- Who is the API Evangelist? Kin Lane
- What is an endpoint? An endpoint is a combination of an HTTP verb and a URL.
- What is a query parameter? A query parameter is a parameter in a URL that is used to filter or sort the results.
- What is a base url? A base url is the common part of all the endpoints of an API.
- What is curl? curl is a command line utility for making HTTP requests.
- What is fetch? fetch is a JavaScript function for making HTTP requests.
- What is Postman? Postman is a graphical tool for making HTTP requests.
- What is an SDK, in the context of invoking APIs? An SDK is a library that provides functions for invoking an API.
- What does CORS stand for? Cross-Origin Resource Sharing
- What response header should you look for to see if an endpoint “supports” CORS? Access-Control-Allow-Origin
- What is a CORS error? A CORS error is an error that occurs when a browser fetches data from an API that does not have the Access-Control-Allow-Origin header set to allow the fetch.
- What are different methods that APIs support use to support authentication? Credentials, tokens, API keys
- When learning about APIs, you will be searching for APIs to try out. What three questions might you ask about candidate APIs? Is it free? Does it require authentication? Is it CORS-enabled?
Summary
We’ve covered:
- The difference between APIs in general and Web APIs
- Basic terminology and concepts
- Invoking APIs with curl
- Invoking APIs with fetch in the browser
- Invoking APIs with fetch from Node
- Invoking APIs with the Python requests library
- Invoking APIs with Postman
- What an SDK is
- Understanding CORS
- Understanding authentication
- Choosing an API for a project