Hakyll Pt. 2 – Generating a Sitemap XML File

2018-11-17 [ photo credit theworldwithabackpack ]

Hakyll Pt. 2 – Generating a Sitemap XML File

2018-11-17

This is part 2 of a multipart series where we will look at getting a website / blog set up with hakyll and customized a fair bit.

Pt. 1 – Setup & Initial Customization
Pt. 2 – Generating a Sitemap XML File
(wip) Pt. 3 – Generating RSS and Atom XML Feeds
(wip) Pt. 4 – Customizing Markdown Compiler Options
(wip) Pt. 5 – Generating Custom Post Filenames From a Title Slug

Adding a Sitemap Template

A sitemap.xml template, just like the templates in the last post, receives context fields to work with (variables, essentially), and outputs the result of applying said context to the template. Here is what our sitemap template will look like today in our project’s templates/sitemap.xml:

<?xml version="1.0" encoding="UTF-8"?>
<urlset
  xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
  xmlns:news="http://www.google.com/schemas/sitemap-news/0.9"
  xmlns:xhtml="http://www.w3.org/1999/xhtml"
  xmlns:mobile="http://www.google.com/schemas/sitemap-mobile/1.0"
  xmlns:image="http://www.google.com/schemas/sitemap-image/1.1"
  xmlns:video="http://www.google.com/schemas/sitemap-video/1.1"
>
  <url>
    <loc>$root$</loc>
    <changefreq>daily</changefreq>
    <priority>1.0</priority>
  </url>
$for(pages)$
  <url>
    <loc>$root$$url$</loc>
    <lastmod>$if(updated)$$updated$$else$$if(date)$$date$$endif$$endif$</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.8</priority>
  </url>
$endfor$
</urlset>

Apart from the normal sitemap boilerplate, you can see root, pages, url, date and updated context fields. While date and updated would come from your metadata fields defined for a post, and the url is built from hakyll’s defaultContext, the root and pages fields are custom defined in what will be our very own sitemapCtx context. In the next section, we’ll use this template to generate our sitemap.xml file.

Generating the Sitemap XML File

If you create a hakyll project from scratch, you will start out with a few files that we can add to our sitemap:

index.html
about.rst
contact.markdown
posts/2015-08-12-spqr.html
posts/2015-10-07-rosa-rosa-rosam.html
posts/2015-11-28-carpe-diem.html
posts/2015-12-07-tu-quoque.html

You should note that your site.hs file also has the following:

main :: IO ()
main = hakyllWith config $ do
    -- ...

    match (fromList ["about.rst", "contact.markdown"]) $ do
        route   $ setExtension "html"
        compile $ pandocCompiler
            >>= loadAndApplyTemplate "templates/default.html" defaultContext


    match "posts/*" $ do
        route $ setExtension "html"
        compile $ pandocCompiler
            >>= loadAndApplyTemplate "templates/post.html"    postCtx
            >>= loadAndApplyTemplate "templates/default.html" postCtx

It’s important that you understand that any files you want to be loaded and sent to templates/sitemap.xml must first be matched and compiled before the sitemap can be built. If you don’t do this, you’ll pull your hair out wondering why the file (or folder) you’re trying to include in the sitemap never shows up.

Now, there is something that we are going to emulate to make this sitemap a reality (this should already be in site.hs):

main :: IO ()
main = hakyllWith config $ do
    -- ...

    create ["archive.html"] $ do
        route idRoute
        compile $ do
            posts <- recentFirst =<< loadAll "posts/*"
            let archiveCtx =
                    listField "posts" postCtx (return posts) `mappend`
                    constField "title" "Archives"            `mappend`
                    defaultContext

            makeItem ""
                >>= loadAndApplyTemplate "templates/archive.html" archiveCtx
                >>= loadAndApplyTemplate "templates/default.html" archiveCtx

Reading the code above, this essentially says

here’s a file we want to create that does not yet exist (how create differs from match)
when you create the route, keep the filename (what idRoute does)
when you compile, load all the posts, specify what the context to send to each template will be, then make the item (the "" is an identifier… see the source for more), then pass the context to the archive template and pass that on to the default template, ultimately building up a full webpage from the inside-out

Let’s change this 3-step rule to suit our needs before we wrangle the code. We want our rules to say:

here’s a file we want to create that does not yet exist (sitemap.xml)
when you create the route, keep the filename (what idRoute does)
when you compile, load all the posts, load all the other pages, specify what the context to send to each template will be, then make the item, then pass the context to the sitemap template, ultimately building up an XML file

This is almost the same! Let’s write it:

main :: IO ()
main = hakyllWith config $ do
    -- ...

    create ["sitemap.xml"] $ do
        route idRoute
        compile $ do
            -- load and sort the posts
            posts <- recentFirst =<< loadAll "posts/*"

            -- load individual pages from a list (globs DO NOT work here)
            singlePages <- loadAll (fromList ["about.rst", "contact.markdown"])

                           -- mappend the posts and singlePages together
            let pages = posts <> singlePages

                           -- create the `pages` field with the postCtx
                           -- and return the `pages` value for it
                sitemapCtx = listField "pages" postCtx (return pages)

            -- make the item and apply our sitemap template
            makeItem ""
                >>= loadAndApplyTemplate "templates/sitemap.xml" sitemapCtx

This is starting to look good! But what’s wrong here? Remember the root context bits? We’re going to need to define what that is, and the best way that I’ve found right now is simply as a String; if you want to do something fancy with configuration or reading it in dynamically, then go nuts.

root :: String
root = "https://ourblog.com"

With that defined, we can add it to our contexts:

main :: IO ()
main = hakyllWith config $ do
    -- ...

    create ["sitemap.xml"] $ do
        route idRoute
        compile $ do
            posts <- recentFirst =<< loadAll "posts/*"
            singlePages <- loadAll (fromList ["about.rst", "contact.markdown"])
            let pages = posts <> singlePages
                sitemapCtx =
                    constField "root" root <> -- here
                    listField "pages" postCtx (return pages)
            makeItem ""
                >>= loadAndApplyTemplate "templates/sitemap.xml" sitemapCtx


-- ...


postCtx :: Context String
postCtx =
    constField "root" root      <> -- here
    dateField "date" "%Y-%m-%d" <>
    defaultContext

Hint: if the <> is throwing you for a loop, it’s defined as the same as thing as mappend.

See how we defined constField "root" root in two places? We’re talking about two different contexts here: the sitemap context and the post context. While you could have the postCtx be combined with the sitemapCtx, thus giving the pages field access to the root field, you probably want to use root (and perhaps other constants) wherever you work with posts, so adding them to postCtx for use everywhere seems like the right thing to do.

Once you’ve got all this, run the following to build (or rebuild) your docs/sitemap.xml file:

λ stack build
λ stack exec site clean
λ stack exec site build

Your docs/sitemap.xml should now have all your pages defined in it!

Adding Other Pages and Directories

We’ve done some epic traveling in New Zealand and now want to include a bunch of pages we’ve written in the sitemap. Those pages are:

new-zealand/index.md
new-zealand/otago/index.md
new-zealand/otago/dunedin-area.md
new-zealand/otago/queenstown-area.md
new-zealand/otago/wanaka-area.md

First, we make sure that our pages get compiled (we’ll use postCtx for them):

main :: IO ()
main = hakyllWith config $ do
    -- ...

    match "new-zealand/**" $ do
        route $ setExtension "html"
        compile $ pandocCompiler
            >>= loadAndApplyTemplate "templates/post.html"    postCtx
            >>= loadAndApplyTemplate "templates/default.html" postCtx

And then we want to make sure we add them to our create function:

main :: IO ()
main = hakyllWith config $ do
    -- ... match code up here

    create ["sitemap.xml"] $ do
        route idRoute
        compile $ do
            posts <- recentFirst =<< loadAll "posts/*"
            singlePages <- loadAll (fromList ["about.rst", "contact.markdown"])
            nzPages <- loadAll "new-zealand/**" -- here
            let pages = posts <> singlePages <> nzPages -- here
                sitemapCtx =
                    constField "root" root <>
                    listField "pages" postCtx (return pages)
            makeItem ""
                >>= loadAndApplyTemplate "templates/sitemap.xml" sitemapCtx

I could not figure out how to mix globs (new-zealand/**) in with individual file paths (included in fromList), so I had to load them separately; if you figure out how, let me know!

Once you’ve got all this, run the following to rebuild your docs/sitemap.xml file:

λ stack build
λ stack exec site rebuild

Wrapping Up

In this lesson we learned how to dynamically generate a sitemap.xml file using hakyll. Next time, we’ll use these same skills to generate our own RSS and Atom XML feeds.

Next up:

(wip) Pt. 3 – Generating RSS and Atom XML Feeds
(wip) Pt. 4 – Customizing Markdown Compiler Options
(wip) Pt. 5 – Generating Custom Post Filenames From a Title Slug

Thank you for reading!
Robert