Build Artefacts Should Be Version Controlled

• Building dhall files with the results in VC has been wonderful

• Not only because it saves build time but also you can see exactly what happened

• Also because you can truly refactor fearlessly

• The DHH post about farming CI out to dev computers

• Unisonlang stores the fact that tests passed and probably build artefacts

• CI takes ages but people have already built stuff

• Developers should be aware of the code they are actually running

• We should be able to sign build artefacts and test results etc (these should be the same thing actually) with the source code.

• Things like “renaming functions shouldn’t affect the code” should be visible in the git diff.

• If someone builds something, nobody should ever have to build that again. Unison kind of does this but requires wacky tooling. Git is fine.

• BUILDING OFFLINE SHOULD ALWAYS BE POSSIBLE AAAA

• IF you want to dive to a lower level language, that should be totally possible by just integrating into the lower level of the build system.

• If my new langauge generates zig code, you can see how your code changes affect the compiled output. You can also just write some zig code to use it.

• For that matter, it could be that a higher-level language performs more complex checks along the way. A high level functional language (like a dependently typed language) should generate a lower-level functional language (like simply typed lambda calculus) which should be human readable. The final zig code should also be human readable and link back to the original source.

• But I don’t only want to generate zig code. The language should be interpretable with many types of output.

• A primary use case is command line functions (e.g. like JQ). In that case, stdin should be parsed into some format (specific to the tool being used, e.g. json) then passed into the expression for evaluation and then output. This process should not involve compiling to another language, and cannot in fact because dependent typing requires compile-time evalutation. But the general pattern should be possible.

• Perhaps the first step is a build system that can sign build artefacts against all build inputs. This can be used for compiled source code from my language, but also for times when my language has been used to process something else (e.g. this json transformer).

• Make would suffice except that it doesn’t guarantee that artefacts were built from their inputs.

• Is there scope for a tool that can sign any output from its inputs? Or a make extension that does this? Or some other build tool?

  • Can bazel do it?
  • Ninja?
  • Someting else?
  • Zig build?
  • Is there a need for a new build tool?

• What is needed for this?

  • some way to fully specify all inputs to a build artefact, including all sources, libraries, and full specification of the tool being used to build, and any environmental stuff. Nix almost gives you that but it has an unfortunate amount of side effects. This gets easier if you don’t include platform-specific stuff - e.g. storing generated zig code but not machine code.

Here’s an idea

• A command line tool that will either build something or not depending on some version-controllable datastore
• There is a directory structure shadowing the whole repository. Each file contains a hash of the original file, and if it is a derived file then it also contains hashes of all of its inputs, including tool versions and environment etc.
• The tool is responsible for generating a set of files. You say which files you want to generate and what the inputs are (including tools etc). For the files that are source files, if the source file is newer than its shadow then the shadow is regenerated (it is just a hash of the original file). If all prerequisite hashes are the same as the listed hashes, then nothing happens. Otherwise the tool is rebuilt and the hashes are copied over.
• Because final build outputs (e.g. executable binaries) are also hashed and include hashes of all of their prerequisites you can check if a version controlled binary has been generated from the source by recursively checking if all the hashes match in the shadow tree.

You still have to build things in the right order, and you still have to calculate what all the inputs are for some particular process.

Apparently this is related to merkle DAGs or merkle trees. Jack says diff-o-scope. It could also use nix.

I think Jack was onto something. I was initially put off by the “it’s just Nix” idea because I want all artefacts to be visible within the development tree, but this can be remedied by building stuff and then symlinking a link farm into the dev tree. Also the nix language is not essential to nix - you can also generate drv files. I wonder if we could build haskell this way by generating a bunch of drv files from a cabal file? haskell.nix doesn’t really do this.

Here’s an idea

• A dhall build tool that uses nix directly.
• package.dhall is generated into the nix store and symlinked (or hardlinked) into all the places it’s needed
• Generated files also go into the nix store and are symlinked (or hardlinked) back into the source tree.
• Hardlinks instead of symlinks allows a github pr to show diffs, if I understand how they work.
• If these are content addressed then users can push the generated files to nix cache without signing.

Here’s an idea

• An extension to git that synchronises nix build artefacts into a git repository.
• This is for the use cases where you do want to see the contents of generated files.
• I am torn on this because you shouldn’t usually need to see the contents of generated files.
• But it seems beneficial to be able to see them if you need to work with them, and if files are scattered over the nix store they become hard to work with.
• Ideally you could have read-only files that are derived when you look at them, but that’s not how unix works.
  • Though it could be if you had named pipes?
  • But this couldn’t update the set of files in a directory. (adding a new one when needed)
• The next best thing may be to have symlinks to the nix store that are automatically updated as necessary using a daemon, or by some manual invocation.
• So the idea is just a tool that builds a linkfarm in the nix store and a tool to copy or symlink it over the git repo.
• And maybe to update the .gitattributes or .gitignore to mark those files as generated or ignored.

For this to work:

• There needs to be a tool that can apply a shadow to a directory. It can work by making copies (or hardlinks) or symlinks.
• Files can either be ignored or managed with git. Probably if you want to ignore them then you’d choose symlinks, and if you wanted to track them you’d choose hardlinks.
• The tool would not understand git though.
• A separate tool would manage the git integration, updating the ignores file if needed.
• If the files were to be tracked, a pre-commit hook would update them.
• A direnv extension might update them each time it runs.
• A daemon might update them whenever anything changes in the directory.

This is an ecosystem of tools. It would have to be easy to make them all work together. A flake part might be the best way. I wonder if there is already such a thing.

Not by the looks of flake.parts website.

We need a bulletproof way to manage these files:

• Don’t delete data
  • Only replace a file if the tool definitely made the last version of the file
  • Don’t write a file that already existed
• Don’t put stuff in there that shouldn’t be therre
• Don’t miss stuff that should be there

Summarised:

• Always remember all files that are being tracked
• Only write a file if it didn’t already exist, or if it hasn’t changed since the last tool invocation.

What if the tool fails before it updates its tracking list?

• Don’t fail if a file has been changed outside the knowledge of the tool into the state it should be.

How do we ensure the generated files aren’t considered source files for the purpose of generating files?

Given the current state of a directory, you can know what files should be there. But how do you know which to remove? We need a file to remember the state of the directory when files were generated. That file could be a complete tree listing with file hashes. It could also be a complete copy of the tree - this may be simpler since the tree will already have to be copied into nix store for this to work. So the tool just needs to know how to find the old tree in the nix store. We can store its path in a file in the tree.