gsc — Git service configuration
-------------------------------
2014-12-19-1306 madduck@madduck.net

The following describes a method for the configuration of services, using Git
as a backend. A service in this context is e.g. a mail server like Postfix or
a DNS server like NSD, which runs on a Unix system and is configured using
files usually situated in the /etc directory. When those files are changed,
usually some steps have to be taken to fix up permissions, compile/merge
files, and reload the service.

These steps — editing configuration files and reloading services — are also
the domain of so-called configuration management systems, such as CFengine and
Puppet. While these are indispensable tools for the general management of
machines, it is often the case that the fine-grained configuration of
a complex mailserver requires more work to express in terms of the
configuration management syntax than there are benefits.

gsc is an alternative way of doing things. In a nutshell:

  - configuration is maintained in Git and something like gitolite can be used
    to manage fine-grained access control;

  - upon push, a repo-side hook runs, reads from the repository a number of
    hosts to poke and proceeds to do just that using SSH forced commands;

  - a dedicated user on each target machine receives the poke, updates the
    local Git repository and executes an in-tree Makefile, which is
    responsible for the integration of the changes into the running service,
    possibly using in-tree host-specific parameters.

A word for the security-conscious
'''''''''''''''''''''''''''''''''
It is true that one might question the use of in-tree configuration files and
the Makefile. After all, it allows anyone with write-access to the Git repo to
impersonate the dedicated poke user, though not without leaving a trace.

In the end it's a matter of trust and if you're ready to assume that someone
who's supposed to be able to configure a given service on a machine can't
really do any harm by impersonating the user designed to configure the given
service, then you're on the safe side.

The benefit of this approach is that it avoids operation as root whenever
possible, defering to sudo for just those steps that require root privileges.

Set-up by example: nsd
----------------------

Let's walk through a simple example and configure the nsd DNS server to be
configurable/manageable with gsc.

  0. We'll start with the assumption that the contents of /etc/nsd is already
     tracked in a central Git repository and that trusted DNS admins can push
     there.

     We'll also assume that changes to the nsd config should be pushed to two
     DNS servers, grimble.example.org and grumble.example.org.

  1. Create a new, password-less (!) SSH keyfile using ssh-keygen on your
     local machine (which needs a clone of the above Git repo), store the key
     into .gsc/poke-key[.pub] inside that repo, and edit the public keyfile by
     prepending the following to the first (and only) line (broken here into
     two lines for readability):

       command="make -sC target git_pull update",no-agent-forwarding,
       no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding

     You can also add a from="" limit as per the sshd(8) manpage, although
     there are benefits in not doing that, namely the ability for anyone with
     access to the repo to poke the servers.

     Commit the two files .gsc/poke-key and .gsc/poke-key.pub to the
     repository.

  2. On the two "poke" hosts, make sure you have Git, make and sudo installed.
     Then add a new system user, e.g. nsd_update (with group nsd_update,
     disabled password, home directory /var/local/nsd_update, and shell
     /bin/sh).

  3. Copy .gsc/poke-key.pub to ~nsd_update/.ssh/authorized_keys and make sure
     to run `chown -R nsd_update ~nsd_update/.ssh` or similar.

  4. Clone the Git repo with --shared=group to /etc/nsd (or /etc/nsd3) and
     similarly chown it, so that the poke user can maintain it without root
     rights. Consequentially, the user "nsd" needs to be able to read
     (possibly even write) the files in this repository, so you need to either
     add nsd to the group nsd_update and ensure g+rwX permissions (using the
     Makefile, see below), or employ ACLs.

     Make sure that the nsd_update user can pull from the repo, either by
     making it publicly available, or by setting up a new password-less SSH
     key for nsd_update on each host, which is then given access to the
     central repo.

  5. Create a symlink ~nsd_update/target to /etc/nsd (or whatever directory
     you chose). This was done to avoid having to make /etc/nsd the
     home directory of the update user and not hardcoding the location of the
     configuration.

  6. Now you should be able to poke a host from your workstation:

       ssh -Ti .gsc/poke-key -o UserKnownHostFile=.gsc/ssh_known_hosts \
         nsd_update@grimble.example.org

     which should print something like

       make: *** No rule to make target 'git_pull'.  Stop.

     Do this for all hosts in pokehosts to populate .gsc/ssh_known_hosts
     (verifying the fingerprints, of course), and then commit this file as
     well.

     As we are using forced commands, it makes no difference what you put
     after the user@host argument in the above. However, we may need this in
     the future, so let's reserve it. Please refrain from using it and/or let
     me know if you have any ideas.

  7. Install the gsc-post-receive hook to hooks/post-receive in the central
     git repo. When using gitolite, I suggest the use of per-repository hooks
     (see further down).

  8. Commit a file .gsc/pokehosts to the repository, listing each target
     server one-per-line in user@host style, e.g.

       nsd_update@grimble.example.org
       nsd_update@grumble.example.org

     The rest of each line is unused for now, but reserved for future use, so
     don't put anything else there.

  9. Now push your commits to the central repo, which should cause the hook to
     poke both hosts, yielding the make error.

 10. Now all that's left to do is write /etc/nsd/Makefile with at least the
     targets git_pull and update, which should run git-pull (possibly with
     --rebase) and then cause the appropriate next steps. Use sudo sparingly
     for maximum security benefit.

     Please refer to the examples/ directory in the gsc repository for
     a simple example that works with nsd, and which also handles per-host
     difference by parametrising e.g. IP addresses or commands needed to
     reload/restart the service.

TODO:

  - per-host branches
  - how to recover from faulty pushes, e.g. if the server was taken down
  - gitolite notes