Using Airtable with Eleventy

A list of tags for this post.

While planning out an update to an existing project I figured out how to use Airtable with Eleventy. Naturally this was after a couple of weeks of working on a proof concept to migrate from Airtable to markdown.

On the bright side this gives me an apples to apples comparison of using remote data via JavaScript data files and markdown, and I’ll outline those differences as I go. But before we started…

A caveat about my dev skills permalink

I’m not very good at JavaScript. There will no doubt be some choices I made that someone with more JavaScript skills would’ve done differently. And in some cases I gave it a try using articles and examples, I’ll be sure to point those out along the way.

But for now, this represents a vast improvement on the current implementation of the project I’m updating. And I’m actually a bit proud of myself for getting this far! Hopefully in the near future I’ll be able to make further improvements.

This article assumes some familiarity with Airtable. Zapier has a good overview of Airtable if you’re not familar.

You can jump right to any of the sections below if you’re not interested in context of the project.

The project permalink

The current project, Horse Racing Datasets (project page), is backed by Airtable with a monstrous implementation of too many API calls, and un-purged Tailwind CSS. It’s one of those “hey, at least it works!” situations, but at the time it was quite an accomplishment. Now that I know my way around Eleventy, the goal is improve the data handling and performance (and rewrite that CSS).

There are currently 64 datasets listed in Airtable. In the last year I added seven datasets and don’t anticipate that pace picking up too much.

The requirements permalink

  • List all datasets with pagination
  • List by tag or category
  • List by recently added with a limit on number displayed (e.g., list the four most recently added datasets)
  • Display a single random dataset
  • Datasets do NOT need individual pages

API call and listing records permalink

In the current version of the project, as well as other projects, I’ve used Axios to access to Airtable. Those projects used Vue and the call is within the Vue app. Previously I had tried to get a version of that call to work with Eleventy, but couldn’t figure out to make it work on its own outside of the structure of the Vue app.

I decided to take a look around GitHub to see if I could find any examples, and as I mentioned here I found and forked this repository, which uses Airtable.js, as a test.

I was quickly was able to get it to work with my data and thanks to this comment in this Github issue I was able to refine it a bit more. I wish I could explain everything that’s going on in there, but I can’t. At least not yet!

First you’ll need to install Airtable.js.

npm install airtable

And I’m using dotenv here to hide my key. If you’re not familiar with how to use it the first two minutes of this video gave me all the information I needed to install, create the .env file and call it in the script.

npm install dotenv
<!-- .env file -->
KEY='your key here'
// src/_data/all.js
require('dotenv').config();
const Airtable = require('airtable');
let base = new Airtable({ apiKey: process.env.KEY }).base('appMh38AX1IpV3vIR');

module.exports = () => {
return new Promise((resolve, reject) => {
let allDatasets = []; // change 'allDatasets' to something more relevant to your project
base('New') // change 'New' to your base name
.select({ view: 'All' }) // change 'All' to your view name
.eachPage(
function page(records, fetchNextPage) {
records.forEach((record) => {
allDatasets.push({
"id" : record._rawJson.id,
...record._rawJson.fields
});
});
fetchNextPage();
},
function done(err) {
if (err) {
reject(err)
} else {
resolve(allDatasets);
}
}
);
});
};

If you’re not using an .env file to hide your key you could use a second Airtable account to share a read-only version of your base. Then you can use the key from the base in the read-only account. You’ll want to do one or the other to keep your key private.

To modify this for your project, add your base and key information in the second line. I’ve commented the two other places where your Airtable information will need to be swapped in. You’ll probably also want to change the variable name to something more relevant to your project (e.g., I’m using allDatasets). The variable is used in three places.

This code lives in a file named all.js in the _data directory and is a JavaScript data file. The data is fetched at build time and available to templates in the same way that data in global data files is available.

{% for item in all %}
<article>
<h2>{{ item.title }}</h2>
<p>{{ item.description }}</p>
</article>
{% endfor %}

This example creates a listing of all the records retrieved in the API call and displays the title and description for each.

Pagination permalink

Fortunately handling pagination for data files is similar to collections. In my project I’ve created a markdown page to list all the datasets. The pagination is set in front matter…

<!-- all.md -->
---
title: 'All Datasets'
layout: 'layouts/feed.html'
pagination:
data: all
size: 10
---

In the data field is the name of the data file, in this case all.js, Size specifies how many items to list per page. If you were using collections you’d specify your collection in the data field, for example collections.posts.

In the template there’s a Nunjucks variable using set that picks up the pagination data from the markdown file using pagination.items.

<!-- feed.html -->
{% extends "layouts/base.html" %}

{% set datasetList = pagination.items %}

{% block content %}
<h1>{{ title }}</h1>
{{ content | safe }}

{%- for dataset in datasetList -%}
{% include "partials/listing-items.html" %}
{%- endfor -%}

{% include "partials/pagination.html" %}
{% endblock %}

This grabs the paginated data and creates pages based on how many items are specified to be listed on each page. There’s a handy section that explains how this works in detail in the Learn Eleventy from Scratch course. The include for pagination.html includes the ‘Next’ and ‘Previous’ links.

Listing by Tag permalink

Tag is a bit misleading here, because it’s not in reference to tags in collections, but it’s what I’ve called the data element in my Airtable base. The requirement is to be able to view a listing of datasets by tag, for example all datasets for the Kentucky Derby.

This is one area where being more skilled in JavaScript is probably an advantage. When the data is available from collections it’s simple to create a single page to handle tag listing pages for individual tags that doesn’t require any maintenance when adding or removing tags. Without collections I created individual tag pages and passed the tag name into the template in order to render only items with the tag for the page.

The tags for this project are fairly fixed, so the maintenance part of needing to manually add, edit or delete a page isn’t a big drawback. But perhaps someone with more skill could’ve done it another way.

In the individual tag pages there’s a variable called “filter” that has the name of tag as it’s referenced in Airtable.

<!-- kentucky-derby.md -->
---
title: 'Kentucky Derby Datasets'
layout: 'layouts/feed-tags.html'
filter: 'Kentucky Derby'
permalink: '/kentucky-derby/index.html'
---

Here’s an example of some of the tags I have my in my Airtable base. Each row contains tags for a single record.

Some of the tags in my Airtable base
In my Airtable base I have a multi-select field named "tags", I use these values in the "filter" field in front matter in the individual tag page to pass the tag name into the template.

Then in the layout for tags, there’s a Nunjucks variable using set to pick up the value in “filter”.

<!-- tags.html -->
{% extends "layouts/base.html" %}

{% set datasetCategory = filter %}

{% block content %}
<h1>{{ title }}</h1>
<p>{{ content | safe }}</p>

{% for dataset in all %}
{% if datasetCategory in dataset.tags %}
{% include "partials/listing-items.html" %}
{% endif %}
{% endfor %}
{% endblock %}

Within the for loop that calls records from all.js I’m using an if statement to pass the tag name that the Nunjucks variable is picking up from the individual tag page front matter. Continuing the example of the Kentucky Derby tag page, the if statement is saying “if the value of ‘Kentucky Derby’ is found in the ‘tags’ field, then display the record”. This creates a listing of items tagged with ‘Kentucky Derby’.

Here’s an illustration of the data flow, starting at Airtable and ending in a tag page. I’ve only included some of the fields to illustrate the records.

The data flow from Eleventy to individual tag pages
Data at Airtable is called via a JavaScript data file from Eleventy. Each tag has a corresponding markdown file, for example kentucky-derby.md. The name of tag that's used in Airtable is set in the 'filter' key in front matter and passed into the template. In the template the 'filter' value is used in the for loop to display only those records that include the tag. Created using Excalidraw.

I tried a few other things before I got this to work. One of the best things was this article by Bryan Robinson on using JavaScript data files in Eleventy. He uses the Meetup API as an example and provides a helpful video and repository of the code.

He has an example where he creates a filter to limit the data to a specific location. I got the same endblock error as he did, but my filter was saved. Even when I was just calling the array without doing any filtering I got the error. One reason I couldn’t get it to work, aside from not being very good at JavaScript, could be that he’s using Liquid and I’m using Nunjucks.

Regardless of my inability to get his approach to work, I highly recommend the article, and especially appreciated the video and repository.

Recently added permalink

In the current site there’s a page that lists new datasets, and this is something I’ll keeping in the revamped version.

This is another one where having a better command of JavaScript would’ve been helpful. In the initial API call in all.js the data comes back sorted alphabetically by title and then by date added. I tried to to create a filter to sort by the date added field with no luck, but I think someone with more skill could make this work.

I ended up creating a new view in Airtable named ‘New’ that sorts the data by date entered and limits the amount of rows to those added in the last 365 days. This gives me a good time period to work with and doesn’t return every dataset.

I created another data file named new.js that calls that specific view. The only difference between the API call in new.js and all.js is the .select calls the view for ‘New’.

.select({ view: 'New' }) // original call has view: 'All'

The markdown page is simple as I’m not using pagination.

<!-- new.md -->
---
title: 'Recently Added Datasets'
layout: 'layouts/feed.html'
---

I’m using the same template that’s being used for the “All Datasets” page but I’ve added two if statements. The first is to change what’s passed in to the for loop and the second is to display some text if there are no recently added datasets.

<!-- feed.html -->
{% extends "layouts/base.html" %}

{% set datasetList = pagination.items %}

{% if '/new/' in page.url %}
{% set datasetList = new | limit(4) %}
{% endif %}

{% block content %}
<h1>{{ title }}</h1>
{{ content | safe }}

{% for dataset in datasetList %}
{% include "partials/listing-items.html" %}
{% endfor %}

{% if datasetList | isEmpty %}
<p>We haven't added any datasets recently, sorry!</p>
{% endif %}

{% include "partials/pagination.html" %}
{% endblock %}

Since pagination isn’t being used I had to create a way to pass in the data source for the ‘New’ page. The first if statement looks at the url, if it’s the ‘New’ page it sets the same variable of datasetList to pass in the file name that makes the API call (new.js) and limits the amount of items displayed to four. The handy limit filter is from 11ty Rocks. I’m also using it here at this site on the homepage!

The second if statement checks to see if the data source, set in datasetList, is empty. If it is empty then it displays the conditional text. I’ll think more about that text when I start designing. It’s fine for the scenario where there are no recently added datasets, but since this template is used for both ‘All Datasets’ and ‘New’ there’s a chance the conditional text could display on the ‘All Datasets’ page if call fails. And if that were the case the text would be misleading.

That handy isEmpty filter is from Mike Riethmuller’s Eleventy Plugin for Array Filters. He’s creating a bunch of Eleventy plugins this month as an Eleventy Advent thing, so be sure to keep an eye on the Jamshop GitHub account.

Admittedly I couldn’t get the array plugin to work, but I looked the code for the isEmpty filter and added it directly as a filter in the eleventy.js config file and it worked.

// eleventy.js
config.addFilter('isEmpty', (value) => {
return Array.isArray(value) && value.length === 0;
});

Displaying a random dataset permalink

This will be a new addition to the site. Currently on the homepage I have some featured datasets listed. What’s nice about that is that I can swap out datasets relevant to the racing calendar, but it’s also a bit of work for such a low traffic site. For the new site I’m going to replace the featured datasets with a ‘Random Dataset of the Day’.

I was able to use the data from the initial API call in all.js and another handy filter from 11ty Rocks that grabs a random item out the array at build time.

{% for dataset in all | randomItem  %}
{% include "partials/listing-items.html" %}
{% endfor %}

I’ll create a partial and include it on the homepage for the “Random Dataset of the Day”. I’ll be using Netlify for hosting and plan on setting up a daily build. In addition to picking up any additions to Airtable it will display a new random dataset.

Maintenance permalink

Ease of maintenance tips the scales in favor of Airtable over individual markdown files, and especially for this project since it already exists in Airtable. I have an Airtable form to add new datasets, and as I mentioned above I’ll set up a daily build at Netlify using either IFTTT or Zapier to pick up any added datasets.

Prior to figuring out how to do this with Airtable I had converted all the records to markdown files and had figured out a fairly easy to way to still keep the workflow starting at Airtable by using Zapier to email me the record formatted for markdown. Given that datasets don’t get added very often this was an OK solution. I would’ve also had to manually deploy it (or figure out how to automate that too!), but still workable and I’d get all the benefits and ease of use of collections.

Wrapping up permalink

When determining the approach for your own projects, the “best” approach will come down to the specifics of the project. How often will things be added? Do you need individual pages for each item or is it only listings? How much would you benefit from the power of collections? Are you decent at JavaScript? The list probably goes on.

I will be converting Pile of hrefs(project page) to this set-up once I have the Horse Racing Datasets redesign complete since the specifics and project needs are very similar. And The Pile could definitely use some pagination!

Below is some additional information for quick comparison or reference. Happy remote data-ing, or collection-ing!

Collections versus remote data comparison permalink

Here’s a quick comparison between the two approaches.

FeatureCollectionRemote Data
TagsNative part of Eleventy collections, easy to set-up a zero maintenance tag page to handle individual tag pages.Have to create a markdown file for each individual tag page and filter for each tag in the template.
PaginationCreate a pagination object for your collection and use it in a template.Create a pagination object for your data and use it in a template. The only difference in the code is the source of data in the pagination keys.
Creating filtersPlenty of easy to follow examples between the documentation, starter projects and in articles.Helpful to have some solid JavaScript skills to transfer collection examples to arrays and data.
MaintenanceFile based - create new file or edit existing file and then deploy. Can probably automate deploys with webhooks or do it manually depending how frequently you add or update entries.Enter data in Airtable (or your remote data source) and set up a daily deploy at Netlify or your host.
Creating individual pagesIndividual pages are automatically created when you create a markdown file and you can use the power of collections.You can create individual pages from data files but it doesn’t use collections.

Useful Resources permalink

Here are some resources I found helpful. I’m currently not caching the data requests but have that bookmarked for future reference. Also, Airtable has a 100 record per call limit, I currently have less than 100 records but have also bookmarked the Stack Overflow link for future reference.

A list of tags for this post.