The Build System Is Dead. Long Live The Build System!

Have you ever been working on a really large product?

If you have ever worked with C++, C#, or Java programming languages you've definitely been involved in writing build scripts for make, make, maven, babel, or other, possibly even custom-developed build systems.

And that's... That's a pain in the ass...

Really, more is needed to code the project, you are also coding how to build the project. These scripts tend to grow in size to the extent that it does not look like a simple collection of source file path sets but rather a framework with its own rules.

If the application is cross-platform things become much worse: rules are complicated, workarounds increase drastically, and eventually, your build system becomes a risky component of the whole product itself.

If you do not understand the pain, just check out an Unreal Engine build system which is a custom build system written in C#, or check an O3DE cmake build scripts, or, for example, Mozilla Build System (I like how they start writing in their documentation: "The Mozilla build system is a complicated collection of scripts and tools..". This sentence sounds to me like a red flag itself, that something must be done. Non-surprising that Rust programming language was born inside the Mozilla company.

Currently, I'm working on a Shatterline game project, and it is comparable in its complexity to those examples, so, yes, while these examples are extreme, they appear more often in real life. We use WAF for building both engine and the game written in C++, and, yeah..., it hurts.

☝ The problem here is covered in the fact that the project structure is not related to the programming design in any way.

And this thought is IMHO the core of all evil.

What Build System MUST do Except Delivering Extra Complexity To My Everyday Job?

So what are the responsibilities of building systems, or, in other words, what are the stages of making a build? Here they are (one may discuss how much the list is comprehensive and/or granular, but I see these four as conceptually prevailing ones):

1. Compilation and Linking.
   Compiling single modules/units, building single modules/compilation units into a whole component (libraries, executables).
2. Component-level dependency management.
   Providing your code base with third-party components (like Qt, boost, Spring, etc..)
3. Build variants.
   Providing different kinds of builds: for different platforms, debugging/release, etc.
4. Post-building facilities.
   Deployment, Installation, etc, are often very customized with outer supporting scripts.

Not that much, until you dig deeper.

Well, the problem is actually partly solved

The designers of modern programming languages are much more careful with the fact that project structuring and source code management must not be such a complex task. They tend to put the logic of source code layout into the compiler design producing such beloved things as Rust and Nim.

These languages employ modularity with a notion of the module rather than compilation unit, the one you pass when invoking the compiler, like:

$ rustc main.rs

Even if your program has more modules, all of them are automatically resolved, and you don't have to pass them all like it is done in C++.

That's how the first two responsibilities are handled. That is, without any build system.

☝ The problem solved here is about not manually managing lists of thousands of source code files, especially if some components share sources and headers. That's the hell.

Let's go deeper talking about module package dependencies management. This is usually done with accompanying tools like Cargo in Rust, or Nimble in Nim. These are not parts of the language itself but are shipped along with the compiler, and allow to specify and download code dependencies.

But employing a module-based approach solves only the first two responsibilities, and the third one is as harsh as the previous ones.

Build Variants. The Tricky One

Why is that? Simple C++:

#ifdef WIN32
// Do windows stuff
#elif _linux_
// Do Linux stuff
#elif __macos__
// Do Mac OS stuff
#endif

Not that simple? Quit ugly stuff, I know. But it is usually even worse in C++:

#ifdef WIN32
    // Do windows stuff
    #if defined(NDEBUG)
         // Do Release Stuff
    #endif
#elif _linux_
    // Do Linux stuff
    #if defined(clang)
         // Something that does not work like in GCC
    #endif
#elif __macos__
    // Do Mac OS stuff
#endif

This can become hundreds of lines long. Reading and writing such code is annoying if not said worse. This has to be obviously done elsewhere, in a build system I suppose.

Let's go further. The building is not usually done in a single compile-link-exe step. It also requires some additional steps, which may contain building and running tests, building, and deployment of accompanying documentation, and building deployment-ready content assets (a common task in game development).

All this custom functionality is the reason why generalistic build systems like cmake grow with a new functionality making learning it harder and harder by introducing basically another programming language into your workflow.

So it would be cool to write this stuff in the same programming language as the project is written.

And this approach is successfully employed by, for example, Nim and Zig programming languages.

Thus building variants are supported within a build toolchain and are written in the same programming language.

This actually looks cool to me. Here is an example of a build scenario written in Zig:

const Builder = @import("std").build.Builder;

pub fn build(b: *Builder) void {
    // Standard target options allows the person running `zig build` to choose
    // what target to build for. Here we do not override the defaults, which
    // means any target is allowed, and the default is native. Other options
    // for restricting supported target set are available.
    const target = b.standardTargetOptions(.{});

    // Standard release options allow the person running `zig build` to select
    // between Debug, ReleaseSafe, ReleaseFast, and ReleaseSmall.
    const mode = b.standardReleaseOptions();

    const exe = b.addExecutable("init-exe", "src/main.zig");
    exe.setTarget(target);
    exe.setBuildMode(mode);
    exe.install();

    const run_cmd = exe.run();
    run_cmd.step.dependOn(b.getInstallStep());

    const run_step = b.step("run", "Run the app");
    run_step.dependOn(&run_cmd.step);
}

Looks like a part of the project itself. When you're doing build tasks in the same language, you do not have to switch your cognitive context between coding what to build and how to build, it becomes a part of the same flow.

At the same time, writing build scripts in such a way does not give you a feeling of mixing concerns. This script is run in a different time and space, so the result is generally satisfying.

Having the problem of source code file lists management eliminated, coding build steps is a much less notorious and error-prone task.

Thus we have received the power to implement build systems responsibility number 4. But still, what about building variants? I've got a good answer for that question either: metaprogramming to the rescue!

The Power of Metaprogramming

Metaprogramming is about dealing with code itself at compile-time programmatically. This means a lot of things, from simple C macros as in the example below to full-featured compile-time code execution and event AST manipulations (Nim macros). The problem with C macros is that they are simple text processors and do not reside on the language level itself, thus cannot be semantically verified.

Thus the simple ifdef code in Nim would look much slicker:

when defined(windows):
    // Do windows stuff
    when (debug):
        // Print something
elif defined(linux):
    // Do linux stuff
elif defined (macos):
    // Do macos staff

No? Not that good? Still, there's a significant difference and it resides in the implementation. Here windows, linux, and macos are constants, being a part of the language itself.

Thus metaprogramming facilities allow us to encapsulate such constructs in a more meaningful program design by injecting abstractions and appropriate design patterns.

Do it this way!

Hail to the language designers that think about build management. IMHO, smearing responsibilities of the build systems between programming language and its build toolchain via modularity and metaprogramming capabilities gives a lot of power and solves a lot of issues dramatically decreasing the scope of monkey jobs around the program. We have too many good examples to ignore this fact and continue to improve build systems and build new ones again and again. This is a dead end.

☝ Build process is better managed in a more cost-effective manner if it is supported by programming language design and implemented via a programming language build toolchain.

I Hope C++ 23 will get to the point where we have STL modularized, and others leave the baggage of the compilation unit concept in favor of the module paradigm.

Thanks for reading.

Tell me what you think.