roht.no: on.thor

Projects, articles, notes and other technical musings from a DevOps focused developer

Partial Under-the-Hood Migration With Netlify

Thor K. Høgås

⏱ 11 minutes read. 📅 Published . ✎ Last updated .

For a while I have been the “proud” owner of the domain broken.engineer. As with many other of my domain ventures, this has thus far been yet another fruitless one. That has just, ever so slightly, imperceptibly changed.

My original plan looked sort of like this:

Evidently I had more questions than surefire approaches.

Fast-forward till the completion of my Master’s thesis, whereupon I was suddenly able to devote time to more than one thing in my life. My goal was as follows: I want to move my section of scribbles1 to broken.engineer, where I will divide my posts into a section for that which I think could be useful some day; that which is fragmented; and that which I spent a lot of time on, and I have no idea whether it’ll ever be useful but where it’s not going to stop me from putting it online anyway.

However, I’d like to link to this content in a dynamic and seemingly integrated way from this site roht.no. My current sketches have something akin to a special blob or call-to-action on my homepage that links to the latest and greatest. There is the potential for a “last three posts I Googled my way to a solution” sort of section instead, or a whole long list of “my other five”, perhaps “featured garbage” or “most viewed”2.

Regardless of which one I choose, I’d like to integrate in a manner which fits with a static site. The integration should exist at build time, thus no JavaScript or any other “quick” front-end solution. I should be able to do whatever I want with the files, so ideally I shouldn’t have to rely too much on an API as designed at some point: I’m the developer and the consumer. Ignore best practices, ignore design! Give me the raw stuff!

Here we are. I’d like the content for my other site accessible within my current site. Where do we go?

Considerations: They Were Made

After considering the pros and cons, it would not be feasible or clean to attempt to keep two sites in the same repo with Netlify. “That’s crazy!”, you say, “clearly the people over at Nx knows how to do it, why don’t you?” While a monorepo on its own is possible, that is different from keeping two different sites in the same repository. Keep in mind that the differences that come with a monorepo are not on the table here, and focus on a repository, that contains two distinct Hugo sites with different domains in the same language.

The following were my discarded ideas.

There’s The Way

After considering the solution, the way forward seemed alright.

I was not quite happy with the approach, in part because of my drafts. I’m terrible at publishing the notes I make, but I enjoy dumping my notes into a folder without the slightest trace of concern. Plus, what do I do if I update a post? Should I make the change in two repositories?

How about Git Submodules, they’re perfect for the job! Well, unless you take into account that you pin a submodule to a given reference, and I’d like to make sure the content is fresh across the two.

The solution seemed as bright as the day long. Enter Hugo Modules. They’re really just Go modules with some added integration into Hugo, namely the easy ability to overlay contents of a Hugo Module over your own content.

You can use Hugo’s mod init subcommand, or just create the Go module yourself. Whip out a go.mod in your project and make sure you get your module correctly specified. Chuck in your other module with the mystique specifier latest — all the dependency managers will hate you and self-loathing is not unheard of.

module github.com/thor/roht.no

go 1.14

require github.com/thor/broken.engineer latest // indirect

Proceed to smile as you open your Hugo configuration in one of the many variants it exists, such as the config.yml.

# config.yml
module:
  imports:
    - path: "github.com/thor/broken.engineer"
      ignoreConfig: true
      mounts:
        - source: content/posts
          target: content/posts

I — and you if you were looking for this too! — have now created parts of the solution to the problem. Perhaps, if you’re unlike me, it will be the entirety of the solution. Go with go get will get the modules, whether you do it or hugo mod get -u does it, and Hugo will overlay the files in content/posts over content/posts in our current project.

The Taste of Progress

Ah, I feel good. I test it locally, and it wo– no. It fails.

hugo: downloading modules …
go: errors parsing go.mod:
/home/thor/dev/roht.no/go.mod:5: require github.com/thor/broken.engineer: git ls-remote -q origin in /tmp/thor/hugo_cache/modules/filecache/modules/pkg/mod/cache/vcs/00692885174d6dcf453ccebebafa216e1f15d87b99b90e43028d562572ed5e26: exit status 128:
        remote: Repository not found.
        fatal: Authentication failed for 'https://github.com/thor/broken.engineer/'
hugo: collected modules in 11461 ms
go: errors parsing go.mod:
/home/thor/dev/roht.no/go.mod:5: require github.com/thor/broken.engineer: git ls-remote -q origin in /tmp/thor/hugo_cache/modules/filecache/modules/pkg/mod/cache/vcs/00692885174d6dcf453ccebebafa216e1f15d87b99b90e43028d562572ed5e26: exit status 128:
        remote: Repository not found.
        fatal: Authentication failed for 'https://github.com/thor/broken.engineer/'

Go can’t get my repository because it tries to check it out over HTTPS, and it’s private and I do not use any authentication for Git besides SSH keys. Ugh. What now? Should I, where the Git host supports it, add a token for my account into the repository path? That feels filthy. How about getting Go to check out the modules using SSH instead? Shirley, that must be easy.

As with most things in life, our hindsight is 20/20. Here is a quick list of things I recommend you do not try or do:

The Way in the End

There are two parts to the solution, the first of which pertains in particular to Netlify. I believe it is still generally applicable, but I haven’t come across other build pipelines where it’s been a hassle as such.

Loading SSH Keys into Netlify

It’s not ideal, but if you have a one-person team it may be considered acceptable.

  1. Generate a new SSH key.

    $ ssh-keygen -t ed25519 id_actual_deploy_key_ed25519

  2. Encode it in base64 in one line.

    $ cat id_actual_deploy_key_ed25519 | base64 --wrap 0

  3. Add it as an environment variable to your Netlify site, e.g. SSH_KEY5.

  4. Add the following script to “load” your SSH key into your session.

    # load-ssh-keys.sh
#!/bin/sh

# Check if we're running in a Netlify environment
# See https://www.netlify.com/docs/continuous-deployment/#environment-variables
if [ ! -z "${DEPLOY_PRIME_URL}" ]; then

    # Init .ssh dir and expand $SSH_KEY
    mkdir -p ~/.ssh
    echo "${SSH_KEY}" | base64 -d > ~/.ssh/id_actual_deploy_key_ed25519
    chmod og-rwx ~/.ssh/id_actual_deploy_key_ed25519

    # Add host keys as an alternative to specifying -o UserKnownHostsFile=/dev/null
    ssh-keyscan -H github.com >> ~/.ssh/known_hosts

fi

    The snippet is inspired by a comment on a support guide I haphazardly stumbled over on Netlify’s forum. Note that there’s no point in starting an SSH agent. In fact, it’s more hassle: if you have multiple keys and need to start an SSH agent, you will also need to manually terminate it once the build is done in order to avoid your build failing due to a lingering ssh-agent.

  5. Execute load-ssh-keys.sh somewhere in your build process.

    Two primary approaches: either prepend your build command with load-ssh-keys.sh (and a &&) or add it to your equivalent of a package.json.

    // package.json
{
  ...,
  'scripts': {
    'preinstall': 'load-ssh-keys.sh'
  },
  ...
}

You’re now good to go. If you went with the package.json route you already know that Netlify will automatically install your package when building, which will execute load-ssh-keys.sh before building your site.

Rewriting HTTPS to SSH

We’ve already alluded to it: it’s time to change the Git configuration. If you want to go the route of the ever-increasing build command you can—yet again—prepend something to it. Add git config --global url."[email protected]:".insteadOf https://github.com/ and then && to the build command and be on your merry way.

There is another way, and while I rate it 0/5 in terms of code maturity or robustness, it current does the trick. I wanted to do this in a way that didn’t involve editing the build command, and I found the far-too-integrated solution.

Enter the newly created Netlify plugin netlify-plugin-hugo-modules. It does the following two-and-a-half things:

  1. Configures Git to use [email protected]: instead of https://github.com/, resulting in Go checking out your Go modules over SSH.

  2. Executes hugo mod get -u before building your project.

  3. Half points: removes the Git configuration when done.

The last point may seem useless, but remember that you can use the Netlify command line client to build and test locally (without the build image), in which case you want the plugin to not overwrite any preexisting setting and to clean up after itself.

$ yarn add netlify-plugin-hugo-modules
$ cat >> netlify.toml <<EOF
[[plugins]]
package = "netlify-plugin-hugo-modules"
EOF
$ git add .  #  oo, risque of you!

That’s it. Hit deploy. Regret the lack of a Terraform module for Netlify. Curse manual steps. You’re set.

My Path Forward

Now I am here.

I think, when considering the amount of time one can dedicate to something so seemingly simple, that the amount of time it’ll take to check the last three points of this list at least fits with something: gradual migrations may take more work, but they’re more likely to get you where you want in the end.

You need your own critical mass.

Summary

In this post we’ve seen elements of my own foolery, as well as how to load Go modules over SSH in Netlify in order to allow us to perform a partial migration of the underlying structure of my site without sacrifying flexibility and the ability to update content as I see fit without duplicating my work. This enabled us to utilise Hugo modules to add content from another Hugo repo without requiring us to use submodules or manually checkout a repository, which could’ve been even easier. We’ve also seen some minor frustrations with Netlify, and how we can work around those provided the security circumstances are acceptable, by utilising some good ol’ scripting and—in my own case—yarn.

  1. Possibly known as a blog. ↩︎

  2. Hah! Who are we kidding?! Nonetheless we are in this for the long run. For ourselves. And maybe you. Maybe. Actually, do I even know you? ↩︎

  3. I consider this to be more akin to abusing the functionality than using it. ↩︎

  4. Clone the repository. Filter out all the junk that’s not important. Whoops, is git filter-branch meant to remove everything? ↩︎

  5. I am devoid of imagination when it comes to naming variables. All that matters is semantics and understanding. ↩︎