Skip to content

Write a build backend

A build backend builds the components of a package. The build block of a component names it:

[components.native.build]
backend = "cargo"
module = "fast"

The core of tt has no backend of its own. Every backend comes from a tt module, a Go module compiled into a distribution of tt. tt's own backends - make, shell, c, lua-c, cargo, cmake, go and zig - are the standard module build, in modules/build of the repository of tt. They are written against the same contract as the backend of any other module, github.com/tarantool/tt/sdk/build, and they are examples of every kind of backend.

This page writes a backend from scratch, template. It renders a template of a component with envsubst into a data file of the package. The page then tests it, builds it into tt and builds a package with it. How a module is put together - its constructor, its commands, its distribution - is in Write a tt module.

The SDK is alpha: its API may change in any release of tt.

Set up the Go module

mkdir tt-template && cd tt-template
go mod init example.com/tt-template
go get github.com/tarantool/tt/sdk@<commit> \
    github.com/tarantool/tt/sdk/build@<commit> \
    github.com/tarantool/tt/sdk/connect@<commit>
go get github.com/stretchr/testify

Require the SDK modules at one commit of tt, as Write a tt module explains. sdk/build is not in a release yet: v3.0.0-alpha.2 does not have it, so take a later commit of the core. Give sdk/connect too, even though the module does not import it: the go.mod of sdk requires it at a placeholder version that only tt's own repository resolves.

To build the module on a checkout of tt instead, with changes of the SDK that are not pushed yet, run tt binary workspace in the directory of the distribution below, whose [core].path names the checkout: it writes a go.work that builds the module on the SDK of the checkout whatever its go.mod requires, and tt binary workspace --remove deletes it once the changes are pushed. Develop together with the core shows the cycle and how to require an SDK module no pushed commit has yet. go mod tidy ignores go.work, so it needs the requirements of the go get above. The distribution build ignores go.work too and builds the module on the SDK of its core, whatever the go.mod of the module requires.

Write the backend

backend.go:

// Package template is a tt module with one build backend, template: it
// renders a template of a component with envsubst into a data file of the
// package.
package template

import (
    "bytes"
    "context"
    "crypto/sha256"
    "encoding/hex"
    "fmt"
    "os"
    "path/filepath"
    "strings"

    "github.com/tarantool/tt/sdk/build"
)

// Name is the name of the backend: backend = "template" in a build block.
const Name = "template"

// tool is the program the backend runs.
const tool = "envsubst"

// Config are the fields of a template build block.
type Config struct {
    // Template is the template, relative to the working directory of the
    // build. Its name ends with .in. Required.
    Template string `toml:"template"`
    // Vars are the variables the template is rendered with.
    Vars map[string]string `toml:"vars"`
}

// Backend returns the template backend.
func Backend() build.Backend {
    return build.Backend{
        Name:        Name,
        Produces:    []build.Result{build.Data},
        Config:      func() any { return new(Config) },
        New:         build.Typed(newBuild),
        Incremental: true,
        OutputFrom:  build.OutputNone,
    }
}

// newBuild checks a template build block.
func newBuild(config *Config, _ build.Component, _ build.Common) (build.Build, error) {
    switch {
    case config.Template == "":
        return nil, build.FieldErrorf("", "template is required for backend %q", Name)
    case !strings.HasSuffix(config.Template, ".in"):
        return nil, build.FieldErrorf("template", "%q does not end with .in", config.Template)
    }

    for name := range config.Vars {
        if strings.HasPrefix(name, "TT_") {
            return nil, build.FieldErrorf("vars."+name, "the TT_* variables are tt's")
        }
    }

    return templateBuild{config: *config}, nil
}

// templateBuild is a checked template build block.
type templateBuild struct {
    config Config
}

// The core finds these interfaces of a build by type assertions.
var (
    _ build.Preflighter = templateBuild{}
    _ build.InputHasher = templateBuild{}
)

// Preflight checks that envsubst is in the PATH of the build.
func (b templateBuild) Preflight(_ context.Context, c *build.Context) error {
    _, err := c.LookPath(tool)
    if err != nil {
        return fmt.Errorf("%w: the %s backend runs %s, which is not in PATH",
            build.ErrNoTool, Name, tool)
    }

    return nil
}

// Inputs are what an incremental build reads besides the build block and
// the files of the component: the version of envsubst, and the template,
// which is outside the component when cwd is.
func (b templateBuild) Inputs(ctx context.Context, c *build.Context) ([]string, error) {
    version, err := c.Output(c.Command(ctx, tool, "--version"))
    if err != nil {
        return nil, err
    }

    firstLine, _, _ := bytes.Cut(version, []byte("\n"))

    data, err := os.ReadFile(filepath.Join(c.Dirs.Cwd, b.config.Template))
    if err != nil {
        return nil, err
    }

    sum := sha256.Sum256(data)

    return []string{
        string(firstLine),
        b.config.Template + " sha256:" + hex.EncodeToString(sum[:]),
    }, nil
}

// Run renders the template into the share directory of the component.
func (b templateBuild) Run(ctx context.Context, c *build.Context) error {
    source, err := os.Open(filepath.Join(c.Dirs.Cwd, b.config.Template))
    if err != nil {
        return err
    }

    defer source.Close()

    cmd := c.Command(ctx, tool)

    cmd.Stdin = source
    for name, value := range b.config.Vars {
        cmd.Env = append(cmd.Env, name+"="+value)
    }

    out, err := c.Output(cmd)
    if err != nil {
        return err
    }

    name := strings.TrimSuffix(filepath.Base(b.config.Template), ".in")

    return os.WriteFile(filepath.Join(c.Dirs.Share, name), out, 0o644)
}

The rest of this page explains each part.

What the backend says of itself

A build.Backend is a value. Its fields tell the core what the backend is:

Field
Name the value of backend that selects it: a lowercase letter, then lowercase letters, digits and dashes; no two backends of a tt share one
Produces the types of what its builds make, each once: build.Module, build.Data, build.Program, build.Link (see Results)
Config returns a fresh, empty configuration: a pointer to a struct of the fields of the block
New checks a block and returns the build; build.Typed(newBuild) makes it of a function that takes the configuration as its own type, *Config
Incremental allows incremental = true in the block
OutputFrom whether the block has output, and who copies what it lists
Exclude gitignore patterns of files the backend's tool leaves in the component, which are never laid out: /target/ of cargo
NoSourceLayout switches off the default include of the component, the Lua sources and the prebuilt modules: for a backend whose sources are no files of the package, as go
Hook lets a lifecycle hook, [hooks.pre_build], [hooks.post_build] or [hooks.clean], name the backend
Cross lets its builds be for another platform than the machine tt runs on, tt package build --target (see Other platforms)

The core calls Config and New whenever it checks a manifest: every command that reads the manifest does, tt package add included, and it may call them more than once for one block. Neither may have side effects, and both are to be cheap. New looks at the block only, and is given no directory of the project; a tool or a file the build needs is looked for later, in Preflight.

The fields of the block

The core decodes the fields of the block into the struct Config returns. A field's key is its toml tag, or the Go name of the field when it has no tag; toml:"-" leaves a field out, and the fields of an embedded struct are fields of the block. A field is a string, a boolean, a number, a slice, a map with string keys, a struct of such fields, a pointer to one of these, or any. A pointer tells a field that is not set from its zero value.

The fields every block has - backend, cwd, env, output, incremental and static - are the core's. They are not in Config: the core gives them to New as build.Common.

The decoding is strict. A block of the template backend with a mistake fails every command that reads the manifest, with exit code 1:

⨯ invalid manifest: components.config.build.vars.PORT: must be a string, not an integer
⨯ invalid manifest: components.config.build.templat: unknown field in manifest version "0.1"
⨯ invalid manifest: components.config.build.make_target: is a field of backend make, not of "template"

The first is vars = { PORT = 3301 }. The second is a key no backend has. The third is a key another backend has: the error names the backends that have it.

Check the block

New gets the decoded configuration, the component and the common fields. It returns the build, or an error. The field New of build.Backend takes the configuration as any, since backends of different configurations sit in one list; build.Typed hands it to newBuild as the *Config it is.

An error of a field is a build.FieldError, made by build.FieldErrorf. Its Field is the path of the field from the block, as the block writes it: template, vars.TT_PACKAGE, platforms.linux.cflags, sources[1]. An empty Field is the block itself. The core puts the path of the block in front:

⨯ invalid manifest: components.config.build: template is required for backend "template"
⨯ invalid manifest: components.config.build.template: "defaults.lua" does not end with .in
⨯ invalid manifest: components.config.build.vars.TT_PACKAGE: the TT_* variables are tt's

Any other error of New is reported for the block. There are no tags for required fields or defaults: New checks them.

build.Component is the component: its Name, its Path relative to the project, its Namespace and its Package. Wants are the types of results the manifest asks the component for besides its build block, nil when it asks for none, as a manifest is to. The core checks the build against them; a backend that picks what to build when the block does not say may follow them, as cargo and zig build the program named after the component when Wants is build.Program alone. The one form of a manifest that asks so far is deprecated, so a new backend may leave them to the core. build.Common has Cwd, Env, Output and Static of the block.

Run the build

Run builds the component in a build.Context. The core makes one for each component and build.

Field
Component the component
Target the platform, as GOOS and GOARCH name it: the machine tt runs on, build.HostTarget(), or for a backend with Cross the platform of --target (see Other platforms); ModuleExt is .so, or .dylib on darwin, String is <os>-<arch> and Cross tells another platform than the machine
Dirs the directories of the build, see below
Tarantool the Tarantool the component is built against: Prefix, IncludeDir (its C headers, module.h and lua.h), Version, Executable (the tarantool tt found, to check what the build made, a module that loads say, never needed to build), each empty when tt does not know it
Env the env of the block and the TT_* variables of the build
Fresh set by tt package build --rebuild, and for a component whose block changed since its last build: drop what Work keeps and build from scratch
Locked set by --locked: a tool of the build must not move a version either; LockedVersion returns the version the lock pins a rock at
Static static of the block
Args the arguments a command of the module passed through to this build, for its tool (sdk.BuildOptions.Args: tt wasm build calc -- -opt=2); none in a build of tt package build. They are an input of an incremental build
Jobs the most jobs the build runs at once, set by -j, --jobs (sdk.BuildOptions.Jobs); 0 leaves it to the backend, one a CPU. Hand it to the tool only when it is set, cargo -j, so that the tool's own default stays otherwise; JobsOrCPUs is Jobs, or the number of CPUs when it is 0, as TT_BUILD_JOBS has it. It is not an input of an incremental build: a build with other Jobs is skipped all the same
Hook set for the build of a lifecycle hook
Log, Warn the logger of the build, tt's with backend=<name>, whose debug records tt -V shows, and a function that shows a warning

The directories, all absolute, for a component of the package:

Directory
Project the root of the project: the package, or a path dependency or a source rock being built
Component the directory of the component
Cwd where the build runs: cwd of the block, relative to the project, or Component
Work _build/component/<name>/, TT_BUILD_DIR: intermediate files, kept between builds, emptied by --rebuild and when the block changed
Modules where modules go: _build/rock/<product>/lib/<namespace>/, TT_OUTPUT_DIR
Share where data files go: _build/rock/<product>/lua/<namespace>/
Programs where programs go: _build/rock/<product>/bin/, shared by the components
State _build/, the state of the build of the project
Tools .tarantool/bin/ of the project
Tree .rocks/, the rocks tree the dependencies are installed into
Link where libraries for the components built after this one to link go: _build/link/<product>/<name>/, emptied before each build of the component, never installed nor packed; Dirs.LinkOf(name) is that of another component

The core creates Work, Modules, Share, Programs and Link before Run.

The Context says which Tarantool the build is for, not how to compile for it: a backend that compiles C takes the headers from Tarantool.IncludeDir (TT_TARANTOOL_INCLUDE_DIR) and links a module as Tarantool loads one, -shared, or -bundle -undefined dynamic_lookup on darwin, with the compiler it chooses. c and lua-c (modules/build/cc) do so with CC and CXX of tt's environment, as LuaRocks builds a builtin rock, an object a source in Work.

Commands

c.Command(ctx, name, args...) returns an *exec.Cmd that runs in Cwd. Its environment is tt's, then the env of the block, then PATH with .tarantool/bin of the project first when it exists, then the TT_* variables, so that the block cannot change them. The backend may append to cmd.Env: what it appends wins. build.LookupEnv reads a variable from cmd.Env. The example appends vars so.

  • c.Run(cmd) runs the command as tt runs the commands of a build: what it prints is shown as it runs with tt -V, and only when it fails otherwise.
  • c.Output(cmd) runs it the same way and returns its stdout, which is not shown: for a tool that answers in JSON, or envsubst, whose stdout is the result.
  • c.LookPath(name) finds a program in the PATH of the build, and only there: a PATH the env of the block sets is the whole PATH of the build, and a program of tt's own PATH is not found then. A program it does not find is an error wrapping exec.ErrNotFound; c.Command of it makes a command that fails to start with that error.

The TT_* variables are TT_OUTPUT_DIR, TT_BUILD_DIR, TT_COMPONENT_NAME, TT_PROJECT_ROOT, TT_PACKAGE, TT_VERSION, TT_PLATFORM_OS, TT_PLATFORM_ARCH, TT_BUILD_JOBS (JobsOrCPUs, never 0), and TT_TARANTOOL_PREFIX and TT_TARANTOOL_INCLUDE_DIR when tt knows them. A template of the example can use them: ${TT_PACKAGE}, ${TT_VERSION}. envsubst replaces every variable it is given, tt's environment included, and a variable that is not set with nothing.

Results

What a build makes has a type. The type says where it goes:

Result The build puts it into It is installed into
build.Module, a compiled module Tarantool loads Dirs.Modules .rocks/lib/tarantool/<namespace>/
build.Data, a file beside the Lua sources, such as .wasm or a generated Lua file Dirs.Share .rocks/share/tarantool/<namespace>/
build.Program, a program the instances run Dirs.Programs .rocks/bin/, where tt instance finds it
build.Link, a library, or a header of one, for the c and lua-c components built after it to link (link of their block) Dirs.Link, a header into include/ of it (Dirs.OutputPath) nowhere

The first three types are the places of a rock, lib, share and bin, which the core lays out, records and cleans, and which to of output names; build.Link is the place of a build that is not in the rock, to = "link", whose files the core does not record as made. The set is the core's: a backend says which of the types it makes and adds none of its own, and Validate refuses a backend that names another. What is no module and no program, a .wasm say, is a data file. A type a later version of the SDK adds comes with a directory of Dirs and an entry of build.Results(), as build.Link came, last; the values of the types, module, data, program and link, never change, so a backend that handles the types it produces is not broken by a new one, and one whose Produces is build.Results() makes the new one at once.

One build may make all three. The core takes as made what Run left in these three directories, each by its type, and removes from them what no build made. A file in the directory of a type the backend does not produce fails the build with exit code 2, build.ErrNotProduced, and so does a program a ProgramNamer names for a backend without build.Program:

⨯ building component "config": the build made what its backend does not produce: backend "template" makes no modules: lib/myapp/defaults.so

What an earlier build left there, of another backend the block named before, is not taken as made by this one, and leaves the package.

The core then checks what was made. Each program has to be an executable ELF or Mach-O program of the target, and a static one when the block says static = true; a component of the deprecated kind has to make what its kind says. It records what each component made in _build/results.json:

{
  "components": {
    "config": {
      "data": [
        "defaults.lua"
      ]
    }
  }
}

The lists are modules, data and programs, each relative to its directory. A build of one component knows by the record what the others made, and tt instance tells by it which component makes a program.

A backend whose tool writes elsewhere copies what it built into these directories with build.CopyFile, each file where Dirs.OutputPath(to, name) says. It writes the copy with no fork of tt in between, so that a program it copies can be started at once: Linux refuses to run a file that is open for writing, and a process tt forks while it writes holds the file open until it runs its own program.

An error of Run fails the build with exit code 2:

⨯ building component "config": open /home/me/myapp/config/missing.lua.in: no such file or directory

Output: lib, share and bin

output of a build block lists files the build makes, for a backend whose tool writes them where it likes rather than into the directories of the Context. An entry is a path, or a table with path and to: to = "lib" is a module, to = "share" a data file, to = "bin" a program, to = "link" a library for another component to link. One entry may be written alone, output = "dist/banner.txt" for output = ["dist/banner.txt"]. An entry with no to is of the one type of a backend whose Produces has one type only - a data file of a backend of data files - and a module of any other backend. The core gives the entries to New in Common.Output, each a build.OutputFile with its Path and its type, To. OutputFrom says whether the backend has the field:

  • build.OutputNone: no output. The build puts what it makes into the directories of the Context itself, as template, cargo and zig do. A block with output is refused: components.config.build.output: is a field of backend make, shell or cmake, not of "template".
  • build.OutputCwd: the core copies each file output lists, relative to Cwd, into the directory of its type under its base name (Dirs.OutputPath), after Run succeeds. make and shell work so (modules/build/basic).
  • build.OutputBackend: the backend reads Common.Output and copies the files itself, from where it built them. cmake copies them from its build tree (modules/build/cmake).

The core refuses an entry whose to is not among Produces, before anything is built.

A backend whose output the core copies

render is template for a tool that writes its file into the directory it runs in: the build renders the template into the file output names, in Dirs.Cwd, and the core copies it into the package. render.go:

package template

import (
    "context"
    "os"
    "path/filepath"

    "github.com/tarantool/tt/sdk/build"
)

// RenderConfig are the fields of a render build block.
type RenderConfig struct {
    // Template is the template, relative to the working directory of the
    // build. Required.
    Template string `toml:"template"`
}

// RenderBackend returns the render backend: it renders a template with
// envsubst into the one file output names, relative to the working
// directory of the build, which the core copies into the package.
func RenderBackend() build.Backend {
    return build.Backend{
        Name:       "render",
        Produces:   []build.Result{build.Data},
        Config:     func() any { return new(RenderConfig) },
        New:        build.Typed(newRender),
        OutputFrom: build.OutputCwd,
    }
}

// newRender checks a render build block: a template, and one file of output.
func newRender(config *RenderConfig, _ build.Component, common build.Common) (
    build.Build, error,
) {
    switch {
    case config.Template == "":
        return nil, build.FieldErrorf("", "template is required for backend %q", "render")
    case len(common.Output) != 1:
        return nil, build.FieldErrorf("output", "names %d files; render makes one",
            len(common.Output))
    }

    return renderBuild{template: config.Template, file: common.Output[0].Path}, nil
}

// renderBuild is a checked render build block.
type renderBuild struct {
    template string
    file     string
}

// Run renders the template into the file of output, in the working
// directory of the build.
func (b renderBuild) Run(ctx context.Context, c *build.Context) error {
    source, err := os.Open(filepath.Join(c.Dirs.Cwd, b.template))
    if err != nil {
        return err
    }

    defer source.Close()

    cmd := c.Command(ctx, tool)

    cmd.Stdin = source

    out, err := c.Output(cmd)
    if err != nil {
        return err
    }

    path := filepath.Join(c.Dirs.Cwd, b.file)

    err = os.MkdirAll(filepath.Dir(path), 0o755)
    if err != nil {
        return err
    }

    return os.WriteFile(path, out, 0o644)
}

New of the module (module.go, below) provides both backends: Backends: []build.Backend{Backend(), RenderBackend()}. Once the module is in tt (Build it into tt), a package uses it; in myapp/app.manifest.toml of Use it in a package:

[components.banner]
path = "banner"

[components.banner.build]
backend = "render"
template = "banner.txt.in"
output = "dist/banner.txt"

myapp/banner/banner.txt.in:

${TT_PACKAGE} ${TT_VERSION}

The block says no to, and render makes data files only, so the file is a data file. The build runs in banner/, the directory of the component, which is the working directory of the build since the block has no cwd: it writes banner/dist/banner.txt, and the core copies it under its base name into share/ of the namespace of the component, the package name, which tt package build installs as .rocks/share/tarantool/myapp/:

$ ../dist/tt package build
$ cat banner/dist/banner.txt
myapp 0.0.0-dev.0
$ cat .rocks/share/tarantool/myapp/banner.txt
myapp 0.0.0-dev.0
$ ../dist/tt run -e "local fio = require('fio') \
    local file = fio.open('.rocks/share/tarantool/myapp/banner.txt') \
    io.write(file:read()) os.exit()"
myapp 0.0.0-dev.0

The Lua code of the package reads it beside itself, as fio.pathjoin(debug.sourcedir(), 'banner.txt') in a module of the namespace myapp; other code reads it from the directory of the application, the working directory of its instances, as above. Neither banner.txt.in nor dist/ is laid out: the default include takes *.lua and *.so. The core refuses what the block asks for and render does not make before anything is built, with exit code 1, as for output = { path = "dist/banner.txt", to = "lib" }:

⨯ invalid manifest: components.banner.build.output[0].to: backend "render" makes no modules

A build whose Run succeeds and leaves no file where output says fails with exit code 2, building component "banner": copy declared output "dist/banner.txt": opening the source: ...: no such file or directory. A path of output is relative to Dirs.Cwd for build.OutputCwd: a block with cwd = "." names output = "banner/dist/banner.txt" and a template relative to the project. The output field of the manifest reference has where every to is installed.

More of the build

A build may implement more interfaces. The core asks for each with a type assertion, so a method of another signature would never be called: the core refuses a build whose Preflight, Produces, Programs, Links, Inputs, Clean or Vendor is not the method of the interface, or is a method of a pointer while New returns a value, when it checks the manifest, with exit code 1:

⨯ invalid manifest: components.config.build: invalid build backend "template": Preflight of its build template.templateBuild is func(*build.Context) error, not func(context.Context, *build.Context) error of build.Preflighter: the core would never call it

buildtest.Validate refuses it too. The var _ build.Preflighter = templateBuild{} lines of the example make the compiler say so first.

A build.Preflighter checks what the build needs before any component is built, so that a missing tool fails the build at once, not after the first long compile. Preflight(ctx, c) runs before the version of the package is known: the directories of the Context may not exist yet, and TT_VERSION is empty. It runs before the lock is resolved, too: c.LockedVersion(rock) answers from app.manifest.lock when the lock is fresh, and says the rock is not pinned when the lock is stale or missing, which the build resolves afterwards; with --locked such a lock fails the build, so the answer is exact there. ctx is that of the build: make the commands Preflight runs with it, so that they stop when the build is cancelled. Wrap build.ErrNoTool for a tool that is not in PATH, so that a missing tool is told from a failed build, and build.ErrNoHeaders for a build that compiles against the headers of Tarantool when Tarantool.IncludeDir is empty: tt then adds the directories it looked into for them, and how to point it at some. A failed Preflight fails the build with exit code 2:

⨯ building component "config": build tool not found: the template backend runs envsubst, which is not in PATH

The backends of tt that have a Preflight call it at the start of Run too. The other interfaces:

  • build.Producer: Produces() says what this block asks for, each type among Backend.Produces. The core checks it against Component.Wants before anything is built. cargo and zig make a module, a program or both, as module and bin say.
  • build.ProgramNamer: Programs() names the programs the build makes, and says whether the list is exhaustive. The core checks with it that no two components make one program, and tt instance finds by it the component that makes a program. When the list is not exhaustive, the core adds what the build wrote into Programs.
  • build.Linker: Links() names the components of the package whose libraries the build links, from Dirs.LinkOf of each. The core checks, with the manifest, that each is a component every product of this one builds before it, and refuses the manifest otherwise, naming both. c and lua-c are linkers by their link.
  • build.InputHasher: see the next section.
  • build.Cleaner: see Clean.
  • build.Vendorer: see Offline builds.

Other platforms

tt package build --target linux/arm64 builds the components for another platform than the machine tt runs on (see Other platforms). A backend says whether its builds can be made for one with Cross: true; without it, the default, a component of the backend fails such a build before a hook or anything else runs, with exit code 1, naming the component, the backend and both platforms:

⨯ building component "config": the build cannot be made for the target platform: the template backend builds for the machine tt runs on alone, linux-amd64, and the build is for linux-arm64

build.Backend.CheckTarget(target) is that check, an error wrapping build.ErrTarget. A build of a backend with Cross makes what it makes for Context.Target, TT_PLATFORM_OS and TT_PLATFORM_ARCH of Env: go sets GOOS and GOARCH, zig passes -Dtarget, cargo builds for the Rust target of the platform, and make and shell leave it to the Makefile or the command. c.Target.Cross() tells a build for another platform from one for the machine. A build that cannot be made for the target it is given - go with cgo = true, whose C compiler is the machine's, a zig build.zig that takes no -Dtarget - refuses it in Preflight with an error wrapping build.ErrTarget, which the core reports with exit code 1, as its own refusal, rather than 2.

template makes text, which runs on any platform, and could say Cross: true. A backend that compiles with the toolchain of the machine, as c, lua-c and cmake do, says nothing: a compiler of another platform is not one tt knows how to find.

What a build for another platform made is checked against it as what one for the machine made is against the machine: a module or a program of another platform fails the build with exit code 2. Such a build has a state of its own, Dirs.State being _build/target/<os>-<arch>/, and with it Work, Link, the stage of Modules, Share and Programs, and what a backend keeps in State, as the target directory of cargo: nothing of a build for one platform is taken for that of another. Tarantool.Executable is empty there: the tarantool of the machine cannot load what the build made. The package is installed into _build/target/<os>-<arch>/tree/ rather than into Dirs.Tree, .rocks/, which stays the tree of the dependencies, built for the machine.

buildtest.Harness.SetTarget(build.Target{OS: "linux", Arch: "arm64"}) sets the target of a test, in Context.Target and in TT_PLATFORM_OS and TT_PLATFORM_ARCH. The harness runs the build for it whether the backend says Cross or not, so that a test checks what a backend does for a platform it has no toolchain of, with a fake compiler; the core refuses it.

Incremental builds

Incremental: true allows incremental = true in a block. The core then skips the build of a component whose inputs are those of its last build, when what that build left is still there. The inputs are the build block and the files of the component, without those its .gitignore files ignore. The hash of the last build is in _build/cache/<component>.json.

A build that reads more is a build.InputHasher. Inputs(ctx, c) returns strings the core adds to the inputs. template returns the version of envsubst and the hash of the template, which is outside the component when cwd is; zig returns its version and the hashes of build.zig and build.zig.zon. Inputs runs on every build, the skipped ones too. A skipped build does not call Run, and --rebuild skips nothing:

$ tt -V package build
   ...
   · running envsubst --version dir=/home/me/myapp/config
   · component is up to date component=config

A backend without Incremental refuses the field: components.native.build.incremental: is a field of backend make, shell, zig or template, not of "cargo".

Clean

tt package clean removes what the builds of a package made. The core removes what is its own: for a clean of the whole package, all of Dirs.State, _build/ - Dirs.Work of every component, the records of the builds, the stage that Dirs.Modules, Dirs.Share and Dirs.Programs are in - but the lock of the project, _build/tt.lock; for tt package clean NAME, Dirs.Work of the component, its records and its files in the stage. Either way the rock of the package leaves Dirs.Tree. A backend that writes only into these directories, as template does, has nothing more to clean.

What a tool leaves anywhere else - the directory of the component, Dirs.Cwd, Dirs.Tools - the core does not remove: it cannot tell a file a build made there from one the project keeps, and a file output lists may be a prebuilt one in git. A build that leaves files there is a build.Cleaner and removes them in Clean(ctx, c). render writes the file of output into Dirs.Cwd; add to render.go, with errors and io/fs imported:

// The core finds these interfaces of a build by type assertions.
var _ build.Cleaner = renderBuild{}

// Clean removes the file the build rendered into its working directory:
// the core removes the copy in the package, not what is beside the sources.
func (b renderBuild) Clean(_ context.Context, c *build.Context) error {
    err := os.Remove(filepath.Join(c.Dirs.Cwd, b.file))
    if errors.Is(err, fs.ErrNotExist) {
        return nil
    }

    return err
}

Clean runs for every component of the backend the clean cleans, before the core removes anything, so that Dirs.Work is still there for a tool that cleans its build tree itself. c is the Context of a build of the component, without Args. Its directories may not exist, for a component that was never built, so a file that is not there is no error. ctx is that of the clean: make the commands Clean runs with it. A failed Clean fails the clean with exit code 2, and nothing is removed. --dry-run runs no Clean, and says which would run:

$ ../dist/tt package build
$ ../dist/tt package clean --dry-run
would run the clean of component banner
would remove _build/cache
would remove _build/component
would remove _build/results.json
would remove _build/rock
would remove myapp 0.0.0.dev.0-1 from .rocks (5 files)
  .rocks/share/tarantool/rocks/myapp/0.0.0.dev.0-1/myapp-0.0.0.dev.0-1.rockspec
  .rocks/share/tarantool/myapp/banner.txt
  .rocks/share/tarantool/myapp/defaults.lua
  .rocks/share/tarantool/myapp/rocks.lua
  .rocks/share/tarantool/myapp/version.lua
$ ../dist/tt package clean
ran the clean of component banner
removed _build/cache
removed _build/component
removed _build/results.json
removed _build/rock
removed myapp 0.0.0.dev.0-1 from .rocks (4 files)
$ ls banner
banner.txt.in  dist
$ ls banner/dist
$ ls -A _build
tt.lock

dist/ stays, empty: Clean removes the one file it knows. In a test, harness.Clean(ctx, bld) runs the Clean of a build as the core does, and nothing after it:

func TestRenderClean(t *testing.T) {
    banner := build.Component{Name: "banner", Path: "banner", Namespace: "app", Package: "app"}
    block := buildtest.Block{
        Common: build.Common{Output: []build.OutputFile{{Path: "dist/banner.txt", To: build.Data}}},
        Fields: map[string]any{"template": "banner.txt.in"},
    }

    bld, err := buildtest.Validate(template.RenderBackend(), banner, block)
    require.NoError(t, err)

    harness := buildtest.New(t, template.RenderBackend(), banner)
    require.NoError(t, os.WriteFile(filepath.Join(harness.Context.Dirs.Cwd, "banner.txt.in"),
        []byte("${TT_PACKAGE}\n"), 0o600))
    require.NoError(t, harness.Run(t.Context(), bld))

    rendered := filepath.Join(harness.Context.Dirs.Cwd, "dist", "banner.txt")
    require.FileExists(t, rendered)

    require.NoError(t, harness.Clean(t.Context(), bld))
    assert.NoFileExists(t, rendered)
    require.NoError(t, harness.Clean(t.Context(), bld), "a file that is not there is no error")
}

What the project itself generates beside the sources, which no backend knows of, is for the [hooks.clean] of its manifest, which a clean of the whole package runs after the backends; see the reference.

Offline builds

tt package download --dir DIR, with no arguments, makes DIR a mirror of what an offline build of the project needs: the rocks of its lock, and what the builds of its components download themselves, as cargo downloads the crates of a package. A backend whose tool downloads is a build.Vendorer: Vendor(ctx, c, dir) downloads into dir, DIR/<backend>, what the build of the component would, as the lock of the sources pins it. The components of the backend share dir, those of other projects downloaded into the same mirror too: name what you put there by its name and version, and keep what is there, since a mirror is extended in place. c is the Context of a build of the component with Locked set, and a failed Vendor fails the download with exit code 2.

A build whose rock servers have the mirror, TT_REGISTRIES=DIR tt package build, is given DIR/<backend> in c.Vendored, of the first server that is a directory and has one: take from it what the tool would download, and download nothing. Without one c.Vendored is empty and the build downloads as ever. cargo vendors the crates of Cargo.lock with cargo vendor, and builds from them with --offline. In a test, harness.Vendor(ctx, bld, dir) runs the Vendor of a build as the download does, and setting harness.Context.Vendored makes a build from a mirror.

The lock of the project

The core builds the components of a package one after another, and holds the lock of the project, _build/tt.lock, through the whole build, the clean, and every other command that changes the project. No other tt changes _build/, .rocks/, the manifest or the lock meanwhile: a second tt package build of the project waits for the first. A backend needs no lock of its own for what it writes into the directories of its Context.

A tt that a build runs on the same project - a shell block whose command is tt package fetch, say - works under the lock of the tt that runs it: tt names the locks it holds in TT_PROJECT_LOCK, which every command of the build inherits, through c.Command or not. A tt that a build starts in the background and does not wait for would work beside the build, so a build waits for what it runs.

Lifecycle hooks

[hooks.pre_build] and [hooks.post_build] run around the builds of all the components, and [hooks.clean] in tt package clean of the whole package, after the Clean of the backends. A hook names a backend with Hook: true, make or shell in tt; another is refused, hooks.pre_build: backend "template" is not supported (want make or shell). The core checks the block of a hook as it checks a build block, New given the zero Component. The build runs in a Context whose Hook is set. It has no component, and Dirs has no Component, Work, Modules, Share and Programs. Env has no TT_OUTPUT_DIR, TT_COMPONENT_NAME and TT_BUILD_DIR, and the files output lists are not copied.

Check the files of a package

A module may check the files of every package tt builds, whatever backend made them. A build.FileCheck names a suffix. Every build gives its Check each file of the package whose name ends with it: the files the components lay out, and the data files their builds make in Share, those of a skipped build included. Modules and programs are not given to it: the core checks them itself.

template may render JSON, and a variable that is not set leaves the JSON broken. module.go holds the constructor of the module and a check of the .json files:

package template

import (
    "context"
    "encoding/json"
    "fmt"
    "os"

    "github.com/tarantool/tt/sdk"
    "github.com/tarantool/tt/sdk/build"
)

// New is the constructor of the module: the template and render backends
// and the check of the .json files of a package, and no command.
func New(sdk.Services) sdk.Module {
    return sdk.Module{
        Backends:   []build.Backend{Backend(), RenderBackend()},
        FileChecks: []build.FileCheck{{Suffix: ".json", Check: checkJSON}},
    }
}

// checkJSON checks a .json file of a package, whatever made it: an empty
// one is a warning, one that is not JSON fails the build.
func checkJSON(_ context.Context, path string) (string, error) {
    data, err := os.ReadFile(path)
    if err != nil {
        return "", err
    }

    switch {
    case len(data) == 0:
        return path + " is empty", nil
    case !json.Valid(data):
        return "", fmt.Errorf("%s is not valid JSON", path)
    default:
        return "", nil
    }
}

Check gets the context of the build, for a tool it runs, and the absolute path of the file in the stage of the package. A warning is shown and the build goes on. An error fails the build with exit code 1, and the warnings of the other files are not shown. Both are shown as they are, so they name the file:

⨯ /home/me/myapp/_build/rock/default/lua/myapp/settings.json is not valid JSON

The standard module wasm checks the .wasm files so (modules/wasm/filecheck.go). It has a check and commands, and no backend.

Test it

sdk/build/buildtest checks a block and runs a build as the core does. backend_test.go:

package template_test

import (
    "os"
    "path/filepath"
    "testing"

    "github.com/stretchr/testify/assert"
    "github.com/stretchr/testify/require"

    template "example.com/tt-template"
    "github.com/tarantool/tt/sdk/build"
    "github.com/tarantool/tt/sdk/build/buildtest"
)

// component is the component the tests build.
var component = build.Component{Name: "config", Path: "config", Namespace: "app", Package: "app"}

func TestBuild(t *testing.T) {
    bld, err := buildtest.Validate(template.Backend(), component, buildtest.Block{
        Fields: map[string]any{
            "template": "defaults.lua.in", "vars": map[string]any{"PORT": "3301"},
        },
    })
    require.NoError(t, err)

    harness := buildtest.New(t, template.Backend(), component)
    require.NoError(t, os.WriteFile(filepath.Join(harness.Context.Dirs.Cwd, "defaults.lua.in"),
        []byte("return { port = ${PORT}, package = '${TT_PACKAGE}' }\n"), 0o600))
    require.NoError(t, harness.Run(t.Context(), bld))

    data, err := os.ReadFile(filepath.Join(harness.Context.Dirs.Share, "defaults.lua"))
    require.NoError(t, err)
    assert.Equal(t, "return { port = 3301, package = 'app' }\n", string(data))
}

func TestBlockErrors(t *testing.T) {
    for _, tc := range []struct {
        fields map[string]any
        want   string
    }{
        {map[string]any{}, `template is required for backend "template"`},
        {map[string]any{"template": "defaults.lua"},
            `template: "defaults.lua" does not end with .in`},
        {map[string]any{"template": "a.in", "var": "x"}, `var: unknown field`},
        {map[string]any{"template": "a.in", "vars": map[string]any{"PORT": int64(3301)}},
            `vars.PORT: must be a string, not an integer`},
        {map[string]any{"template": "a.in", "vars": map[string]any{"TT_PACKAGE": "x"}},
            `vars.TT_PACKAGE: the TT_* variables are tt's`},
    } {
        _, err := buildtest.Validate(template.Backend(), component,
            buildtest.Block{Fields: tc.fields})
        assert.EqualError(t, err, tc.want)
    }
}

func TestNoEnvsubst(t *testing.T) {
    bld, err := buildtest.Validate(template.Backend(), component, buildtest.Block{
        Fields: map[string]any{"template": "defaults.lua.in"},
    })
    require.NoError(t, err)

    harness := buildtest.New(t, template.Backend(), component)
    harness.Context.Env["PATH"] = t.TempDir()

    require.ErrorIs(t, harness.Run(t.Context(), bld), build.ErrNoTool)
}

func TestCheckJSON(t *testing.T) {
    checks := template.New(nil).FileChecks
    require.Len(t, checks, 1)

    dir := t.TempDir()
    good := filepath.Join(dir, "good.json")
    bad := filepath.Join(dir, "bad.json")
    require.NoError(t, os.WriteFile(good, []byte(`{"port": 3301}`), 0o600))
    require.NoError(t, os.WriteFile(bad, []byte(`{"port": }`), 0o600))

    warning, err := checks[0].Check(t.Context(), good)
    require.NoError(t, err)
    assert.Empty(t, warning)

    _, err = checks[0].Check(t.Context(), bad)
    require.EqualError(t, err, bad+" is not valid JSON")
}
$ go mod tidy
$ go test ./...
ok      example.com/tt-template 0.010s
  • buildtest.Block is a build block as a test writes it: Common, Incremental, and Fields, the fields of the backend as a TOML decoder makes them - strings, int64, float64, bool, []any and map[string]any.
  • buildtest.Validate checks the backend as the core checks it before it adds it, with the same function, build.Backend.Validate: what the core refuses (see What the core refuses) fails the test. Then it checks the block in the core's order and returns the build or the first error, a *build.FieldError for a field. The tt of a test has no other backend, so a key the configuration has no place for is an unknown field here, where the core may say that another backend has it.
  • buildtest.New(t, backend, component) makes a Context for the builds of backend on directories of t.TempDir(): <tmp>/project with the component in it, and work, lib, share, bin and state beside it. The target is the machine the test runs on, and no Tarantool is known. Env has the TT_* variables of the core: TT_OUTPUT_DIR, TT_PROJECT_ROOT, TT_PACKAGE, TT_COMPONENT_NAME, TT_VERSION (buildtest.Version, 0.0.0-dev.0), TT_PLATFORM_OS, TT_PLATFORM_ARCH, TT_BUILD_DIR and TT_BUILD_JOBS, the number of CPUs. A test may change the Context before it runs the build: TestNoEnvsubst empties PATH, Env["TT_VERSION"] sets another version, harness.SetTarantool tells the build a Tarantool as the core does, in Tarantool and in TT_TARANTOOL_PREFIX and TT_TARANTOOL_INCLUDE_DIR, harness.SetJobs(n) the jobs of -j n, in Jobs and in TT_BUILD_JOBS, and harness.SetTarget(target) the platform of --target, in Target and in TT_PLATFORM_OS and TT_PLATFORM_ARCH (see Other platforms).
  • harness.Run runs Preflight, when the build has one, with TT_VERSION empty as the core has it there, then Run, and fails, as the core does, when Run left a file in the directory of a type the backend does not produce. It does not copy the files of output, which the core does for build.OutputCwd. Printed returns what the commands printed, Warnings the warnings of the build, and Locked sets the versions LockedVersion answers.
  • buildtest.ValidateHook and buildtest.NewHook(t, backend) do the same for a backend that runs hooks.

Build it into tt

Next to the module and a checkout of tt, dist/tt.manifest.toml:

manifest_version = "0.1"
manifest_kind = "tt-distribution"

[core]
path = "../tt"   # a checkout of tt whose SDK has sdk/build

[dependencies]
template = "../tt-template"
$ cd dist
$ tt binary build tt
   • Resolving the core github.com/tarantool/tt/v3
   •   core github.com/tarantool/tt/v3 (path ../tt)
   •   standard aeon, build, emmylua, wasm (with the core)
   •   template example.com/tt-template (path ../tt-template)
   • Building tt
   • Built /home/me/dist/tt
/home/me/dist/tt
$ ./tt version --modules
Tarantool CLI version 3.0.0-alpha.2, linux/amd64. commit: 2f4ff650 (v3.0.0-alpha.2-187-g2f4ff650)
  aeon      (devel)  github.com/tarantool/tt/modules/aeon
  build     (devel)  github.com/tarantool/tt/modules/build
  builtin   (devel)  github.com/tarantool/tt/v3
  emmylua   (devel)  github.com/tarantool/tt/modules/emmylua
  template  (devel)  example.com/tt-template
  wasm      (devel)  github.com/tarantool/tt/modules/wasm

[core] may name a commit instead, tt = "<commit>". The distribution keeps the standard modules, build among them, so it has tt's backends and template. Every field of the manifest is in tt.manifest.toml.

Use it in a package

myapp/app.manifest.toml:

manifest_version = "0.1"

[package]
name = "myapp"

[platform]
tarantool = ">=3.0.0"
tt = ">=3.0.0"

[components.config]
path = "config"

[components.config.build]
backend = "template"
template = "defaults.lua.in"
vars = { PORT = "3301" }
incremental = true

myapp/config/defaults.lua.in:

return { port = ${PORT}, package = '${TT_PACKAGE}', version = '${TT_VERSION}' }
$ cd myapp
$ ../dist/tt package build
$ cat .rocks/share/tarantool/myapp/defaults.lua
return { port = 3301, package = 'myapp', version = '0.0.0-dev.0' }
$ ../dist/tt run -e "print(require('myapp.defaults').port)"
3301

The template itself is not laid out: the default include of a component takes *.lua and *.so only.

A tt without the module warns of the block for a command that builds nothing, such as tt package resolve, and fails a build, with exit code 1:

⚠ components.config.build: no module of this tt provides backend "template"; its fields are not checked
⨯ building component "config": no module of this tt provides backend "template"

What the core refuses

The core registers the backends and the file checks of every module when tt starts. It refuses those that are not well formed, naming the module:

  • a Name that is not a lowercase letter, then lowercase letters, digits and dashes, and a name another module provides too: the modules are added in the order of their names, and the later one is refused;
  • Produces that is empty, or names a type that is not module, data, program or link, or one twice;
  • an OutputFrom that is none of the three, and an empty pattern in Exclude;
  • no New; no Config, or a Config that does not return a pointer to a struct, or whose struct has a field the decoder cannot fill, two fields of one key, or a field of a common key such as env;
  • a FileCheck with no Suffix or no Check.

build.Backend.Validate and build.FileCheck.Validate are these checks but the one of a name another module provides; the core and buildtest both call them, so a backend whose tests pass is one the core takes.

Every refusal is reported at once, and the binary then fails on every command line. tt binary build tt runs tt version of the binary it built, so such a distribution fails to build:

⨯ the built binary does not start: .../tt version: exit status 1
  ⨯ module "bad": invalid build backend "Tmpl": the name must match [a-z][a-z0-9-]*
       module "bad": invalid build backend "noprod": the Produces must name module, data, program or link, once each
       module "bad": invalid build backend "envfield": env is a field of every build block, not one of a backend
       module "bad": invalid build backend "nonew": no New
       module "build": invalid build backend "make": module "bad" provides one of the name too

Here the module bad provides a backend named make. It comes before build by name, so the make of build is the one refused.

In a manifest, with exit code 1, before anything is built: a field of another backend, a field no backend has, a value of a wrong type, an error of New, a build of an optional method the core would never call, output or incremental of a backend that does not allow them, an output entry or a Producer type the backend does not produce, and a hook that names a backend without Hook. Before anything is built, with exit code 1: a build for another platform of a backend without Cross, and a Preflight that refuses the target, wrapping build.ErrTarget. During the build, with exit code 2: a failed Preflight or Run, a file of a type the backend does not produce, a module or a program that is not one of the target, and a module or a program that is not static under static = true.

The backends of tt

The backends of the standard module build show the rest of the contract:

Package Backends Shows
modules/build/basic make, shell OutputCwd, Hook, a Producer and a ProgramNamer from output, Cross left to the script
modules/build/cc c, lua-c C and C++ compilers driven against the Tarantool of the Context, platforms.<os> fields, commands run at once, a cache of its own in Work that honours Fresh, a Linker reading Dirs.LinkOf, modules beside the module it makes
modules/build/cargo cargo Exclude, Component.Wants, a module, a program or both, a Vendorer and a build from Vendored, Cross with a linker of the target
modules/build/cmake cmake OutputBackend, a build tree in Work, Fresh
modules/build/gobuild go NoSourceLayout, static, Cross with a Preflight that refuses cgo
modules/build/zig zig Incremental with an InputHasher, Cross with -Dtarget

Their fields are in the reference of the application manifest.