Перейти к содержанию

tt restore

Restore a Tarantool cluster from a backup (PITR)

Usage

tt restore
tt restore [flags] <command> [command flags]

Commands

  • tt restore apply - Prepare an instance work directory from backup archives
  • tt restore plan - Resolve a recovery time into a plan and download what it needs

tt restore apply

Rebuild an instance's data directories from a backup chain, so Tarantool can start on a chosen recovery point.

Usage:

tt restore apply <APP>:<INSTANCE> --archives <chain> \
    [--scope <scope>] [--template <name>] [...]
tt restore apply --archives <full,inc1,inc2> --work-dir <path> \
    [--target-point '{"replica_id":N,"lsn":M}'] [--patch-uuid <uuid>]
tt restore apply --archives <chain> -c <cluster config> --instance <name> \
    [--work-dir <launch dir>] [...]
tt restore apply --archives <chain> --snapshot-dir <path> \
    --wal-dir <path> --vinyl-dir <path> [...]

Run once per instance, on every node the replicaset is restored onto, after the orchestrator has copied the archives over and stopped the instance. Those nodes are the members the backup recorded -- 'tt restore plan' lists them in restore_targets[<replicaset_uuid>].instances, the master first, and each entry carries the instance UUID that node has to own afterwards. The configured members the backup never saw are not among them: they are wiped instead and join the replicaset once it is up, so apply is not run on them. Apply does not stop or start Tarantool and does not talk to the backup storage.

The chain is unpacked in the order given, so --archives takes the full backup first and then each increment, exactly as 'tt restore plan' lists them under download_plan[<replicaset_uuid>]. --target-point is that plan's recovery_point.trim_to_by_replicaset[<replicaset_uuid>].

--patch-uuid takes the patch_uuid of the node's own entry in restore_targets[<replicaset_uuid>].instances -- for the master, that is the top-level patch_uuid as well: the instance UUID the restored node owned in the backed-up cluster, which is what its own _cluster records and what the headers must say. Restoring onto the instance the backup was taken on makes it the UUID already in the files, so the patch confirms rather than changes them. Omitting the flag keeps whatever the headers carry, which is right for that same case and wrong as soon as the archives are replayed onto a differently named node.

Before an instance starts on a data directory, Tarantool's configuration looks up the instance's name in the newest snapshot alone: the _cluster row carrying the header's UUID has to name it, or the configuration has to pin database.instance_uuid. The journals after the snapshot do not count, so a member that registered after the backed-up snapshot was taken -- the bootstrap snapshot of a replicaset whose master has taken no snapshot since -- does not start, however surely the replay registers it.

With -c, apply makes the same check after unpacking and trimming, together with the replicaset's UUID and name. A mismatch between the snapshot and the configuration is refused. A name the snapshot lacks and the configuration does not stand in for with a UUID (database.instance_uuid, database.replicaset_uuid) is refused as well, saying whether the journals register the UUID and under which name, and naming the ways out: set TT_DATABASE_INSTANCE_UUID=<uuid> in the instance's own environment (a unit drop-in) and run box.snapshot() on every node once it is up, before the variable is removed; or take the full backup with tt backup start --snapshot. The same variable in tt's own environment, holding the instance's UUID, turns the refusal into a warning: apply cannot see the instance's environment, and Tarantool does not start the instance until that carries it too. Without -c the instance may never be started from a cluster configuration, and the finding is a warning.

Tarantool keeps three kinds of file in three directories it configures separately: snapshots in snapshot.dir, journals in wal.dir, and the vylog together with the vinyl run/index files in vinyl.dir. Apply routes every file of the backup into the directory of its own kind, and there are four ways to say which directories those are.

<APP>:<INSTANCE> names an instance tt runs, found as tt instance start finds it, with --scope and --template: the three directories are the ones tt starts it with -- those its cluster config sets, relative ones taken against the application directory it runs in, and otherwise its data directory under the root of its scope, <root>/var/lib/<app>/<instance> -- and the cluster config is the one of its pair, the variables of its record included. The instance must be stopped. -c, --instance and --work-dir name an instance tt does not run and are refused with a target; the per-directory flags still override.

--work-dir alone puts all three in one directory, which is what an instance that configures none of the keys uses. The directory is the data directory itself: nothing is appended to it, and the files land in it directly.

-c and --instance read the three directories out of a cluster configuration, the same file or URI 'tt restore plan -c' takes. --instance names the instance whose configuration is read -- the node apply runs on, one of the instances 'tt restore plan' lists in restore_targets[<replicaset_uuid>].instances. The keys are resolved the way Tarantool resolves them: a key the configuration does not set defaults to var/lib/{{ instance_name }}, {{ instance_name }} is substituted (and it is the only variable that is; any other one is refused rather than passed through as part of a path), a relative directory is taken against process.work_dir, and a relative or absent process.work_dir against --work-dir. --work-dir means the directory Tarantool is launched from in this mode, not a place to put files, and it is needed only by a directory that is still relative once process.work_dir and the per-directory flags have been taken into account: a configuration whose three directories are absolute needs none, and neither does one whose process.work_dir is absolute.

--snapshot-dir, --wal-dir and --vinyl-dir name one directory each. Each one overrides its own directory and nothing else, whether it came from -c or from --work-dir: '--work-dir /data --wal-dir /wal' keeps snapshots and vinyl data under /data and puts the journal in /wal. A directory named this way is used as it stands: a relative one is taken against the directory tt itself is running in, exactly like --work-dir and --archives, never against --work-dir, and {{ instance_name }} is not substituted into it. An overridden key is never read out of the configuration and never has to resolve, so naming all three leaves -c with nothing to answer -- and leaves --work-dir recording the restore's base directory in the marker and doing nothing else.

Whichever way the directories were named, apply prints where each kind of file is about to go and what decided it -- a flag, or a key of the configuration -- before it writes anything. The report is informational and the run goes straight on from it, so it is where a restore aimed at the wrong directories is visible in the log, rather than later as an instance that comes up on part of its data. A call whose directories have to be checked before anything is written is checked by reading the flags, or by making the same call a rejected one -- an --archives that names a file which does not exist prints the layout and then refuses the run with nothing touched.

A layout that cannot be completed is refused, and nothing is touched. That covers naming nothing at all, -c without --instance, --instance without -c, a configuration file that cannot be read or does not parse, an instance the configuration does not declare, a configuration holding something other than a path where one of the keys belongs, some of the three directories with no --work-dir to fill in the rest, and a directory of the configuration that stays relative with no --work-dir to resolve it against.

Re-running for the same point is idempotent: apply clears what the previous run left in each of the directories and repeats the work. Cleanup removes only the files a restore owns -- snapshots, journals, sort data, vinyl data and interrupted leftovers -- so an instance config or log kept in the same directory survives. It reaches every depth of each of the three directories and prunes the subdirectories it empties, which is what vinyl's <space_id>/<index_id>/ trees need; never name the directory of an instance that is running, or of one whose data is not meant to be replaced.

On success a marker is written beside the snapshot directory, named after it: <absolute snapshot directory>.restore_state.json, so a snapshot directory /srv/memtx/storage-001-a gets /srv/memtx/storage-001-a.restore_state.json. The snapshot directory is the anchor because it belongs to one instance, while the directory --work-dir names can be shared by every instance of an application. The marker records schema_version, work_dir, snapshot_dir, wal_dir, vinyl_dir, target_point, archives and applied_at, plus instance_name, point_name and instance_uuid -- these three only when the run was given them, so a marker written without --instance, --point-name or --patch-uuid leaves them out rather than recording them empty. The three directories are absolute; work_dir is the cleaned value of --work-dir, kept relative when it was given relative, or the snapshot directory when the flag was omitted. Compare point_name, target_point and archives across the restored nodes before starting the cluster: a restore that silently skipped one replicaset brings up shards sitting on different states, each self-consistent and replicating happily, which surfaces much later as diverged buckets.

A marker is removed only from beside the snapshot directory of the run that removes it. Pointing a later run at a different snapshot directory therefore leaves the earlier marker where it was, still claiming its own directory is ready; delete it by hand after moving an instance's data directories.

A sort data file (<signature>.sortdata) is restored beside the snapshot it belongs to and is never stamped with --patch-uuid: it carries the UUID of the instance the backup was taken on, so a restore onto another node leaves Tarantool ignoring it with an error line and rebuilding the indexes from the snapshot. That costs startup time and nothing else.

Exit codes:

0  the directories are ready
1  another operation failed while fetching configuration, unpacking,
   patching or trimming, or the restored data was refused after unpacking:
   --patch-uuid is not a member of the restored replicaset, or the instance
   would not start on the restored snapshot (no marker is written)
2  a system operation failed (for example, permission was denied)
3  an input was rejected: a target that is not one stopped instance of a
   cluster config, an incomplete layout, mismatched --config and
   --instance flags, a malformed configuration, an undeclared instance,
   malformed checksums, recovery point or UUID, a missing archive, or an
   archive entry that cannot be placed. Checksum and archive-chain preflight
   checks precede cleanup; a failure during extraction can leave partial files
4  no xlog covers --target-point

Examples:

tt restore apply myapp:storage-001-a \
    --archives /opt/restore/full.tar.zst,/opt/restore/inc1.tar.zst \
    --target-point '{"replica_id":1,"lsn":1502}'
tt restore apply --archives /opt/restore/full.tar.zst,/opt/restore/inc1.tar.zst
\
    --work-dir /var/lib/tarantool/router-001 \
    --target-point '{"replica_id":1,"lsn":1502}' \
    --patch-uuid 550e8400-e29b-41d4-a716-446655440000
tt restore apply --archives /opt/restore/full.tar.zst \
    -c /etc/tarantool/config.yaml --instance storage-001-a \
    --work-dir /opt/tarantool
tt restore apply --archives /opt/restore/full.tar.zst \
    -c /etc/tarantool/config.yaml --instance storage-001-a
tt restore apply --archives /opt/restore/full.tar.zst \
    --snapshot-dir /data/memtx --wal-dir /ssd/wal --vinyl-dir /data/vinyl

cluster configuration source: a path to the cluster configuration file (cluster.yaml), or a URI of etcd or Tarantool Config Storage. A value that parses as a URI is fetched from the storage; anything else is read as a file.

The URL specifies the etcd or tarantool config storage connection settings in the following format: http(s)://[username:password@]host:port/prefix[?arguments]

  • prefix - a base path to Tarantool configuration in etcd or tarantool config storage.

Possible arguments:

  • key - a target configuration key in the prefix.
  • name - a name of an instance in the cluster configuration.
  • timeout - a request timeout in seconds (default 3.0).
  • ssl_key_file - a path to a private SSL key file.
  • ssl_cert_file - a path to an SSL certificate file.
  • ssl_ca_file - a path to a trusted certificate authorities (CA) file.
  • ssl_ca_path - a path to a trusted certificate authorities (CA) directory.
  • ssl_ciphers - a list of allowed SSL ciphers.
  • verify_host - set off (default true) verification of the certificate’s name against the host.
  • verify_peer - set off (default true) verification of the peer’s SSL certificate.

The command supports the following environment variables:

  • TT_CLI_USERNAME - specifies the Tarantool username;
  • TT_CLI_PASSWORD - specifies the Tarantool password.
  • TT_CLI_ETCD_USERNAME - specifies the etcd username;
  • TT_CLI_ETCD_PASSWORD - specifies the etcd password.

The priority of credentials: environment variables < command flags < URL credentials.

HTTPS connection behavior:

  • The https:// scheme enables TLS for both etcd and Tarantool Config Storage (TCS), even when no ssl_* URL arguments are specified.
  • For etcd, certificate verification is skipped if neither ssl_ca_file nor ssl_ca_path is specified. A configured CA enables verification unless explicitly disabled.
  • TCS keeps the certificate verification behavior of its TLS backend. The default Go backend uses system CAs unless ssl_ca_file is specified.
  • Do not mix http:// and https:// addresses in an etcd endpoint list. If any endpoint uses https://, TLS is enabled for all endpoints.

Usage

tt restore apply [<APP>:<INSTANCE>] [flags]

Flags

  • --archives strings - ordered backup chain: the full backup first, then each increment
  • --checksums strings - sha256 of each archive, in the same order as --archives
  • -c, --config string - cluster configuration to read the data directories of --instance out of (file path or config storage URI)
  • --instance string - instance whose data directories are read from --config: the node this chain is replayed onto
  • --patch-uuid string - new instance UUID to stamp into every snap/xlog header
  • --point-name string - name of the cluster recovery point, recorded in restore_state.json
  • --scope string - scope the application is looked for in: project, user or system
  • --snapshot-dir string - directory for the .snap files and the sort data beside them, overriding snapshot.dir and --work-dir; a relative path is taken against the current directory
  • --target-point string - recovery point position to cut the final xlog at, as '{"replica_id":N,"lsn":M}'
  • --template string - template of the application the instances are described in (default "default")
  • --vinyl-dir string - directory for the .vylog and the vinyl run/index files, overriding vinyl.dir and --work-dir; a relative path is taken against the current directory
  • --wal-dir string - directory for the .xlog files, overriding wal.dir and --work-dir; a relative path is taken against the current directory
  • --work-dir string - directory to put every kind of file in; with --config, the directory the instance is launched from, which its relative paths resolve against; recorded in the marker either way

tt restore plan

Work out how the cluster is brought back to a moment in time, and fetch the backups that would do it.

Usage:

tt restore plan --target-time <T> --backup-storage <config> -d <dir> \
    [--cluster-name <name> --environment <env>] \
    [-c <cluster config>] [--replicasets <names>] [--format table|json]

Run on the manager host, before anything is stopped. The command lists the storage itself -- it is not handed a list of manifests -- walks the chain of every backup back to the full it stands on, stitches the per-shard recovery points into cluster-wide ones, and resolves --target-time into the latest cluster point not later than it.

--target-time is one of these, and a time without a zone is local time, of TZ; tt -V logs the moment in UTC it was read as:

1731592956.818                 Unix seconds, integer or fractional
2024-11-14T14:02:36.818Z       RFC 3339, with a zone: Z, +hh:mm or -hh:mm
2024-11-14 17:02:36.818+03:00  the same with a space for the T
2024-11-14 17:02:36.818        no zone: local time; T for the space too
2024-11-14 17:02               to the minute, local time
2024-11-14                     midnight, local time

A cluster point is one whose label is present on every replicaset, so that every shard comes back to the same moment. Points on either side of a topology change are never stitched into one: a cluster whose composition or membership changed mid-window has no single state to return to.

The archives the chosen point needs are downloaded into --dir and their checksums are verified there, so that a missing or corrupt archive stops the plan while the cluster is still untouched. Distributing the files to the nodes and applying them is the orchestrator's work and 'tt restore apply's. Nothing is deleted, and the cluster being restored is never contacted.

With -c the point's topology is checked against the cluster the restore is aimed at. The composition is read from the configuration, not from the instances: this command's main scenario is a cluster whose nodes are dead or crash-looping. Replicasets are matched by the instance names they share, never by UUID, because the sanctioned target is a freshly deployed cluster where every UUID is new and 'tt restore apply' stamps each restored node with the UUID it owned in the backup. A replicaset that has no counterpart blocks the plan; an instance the config no longer carries is a warning.

--replicasets names the replicasets to restore, by the names the cluster config gives them, and requires -c for that reason. The recovery point is then stitched over those replicasets alone, and so is everything built on it: the downloads, the restore targets and the trim positions. A cluster that keeps a replicaset holding no data -- a stateless vshard router, which writes no recovery point -- has no cluster-wide point at all until the storages are named here. A configured replicaset left out is reported in the warnings and is the operator's to bootstrap fresh; a named replicaset the backup has no counterpart for is code 6, because a restore that was asked for and cannot happen must not pass silently.

Every replicaset's chain is replayed onto each member the backup recorded: the master first -- the instance the backup was taken on, whose headers the archives already fit -- and then the rest by name. restore_targets.instances names them together with the instance UUID each has to own afterwards, what 'tt restore apply --patch-uuid' stamps in per node; the master's own entry is also at the top level, as instance_name and patch_uuid. A manifest that recorded no members beside the master restores him alone, and the plan says so in the warnings: single-target restore. With -c the configured instances the backup has no member for are listed under rejoin instead: they are wiped before the restore and come back by joining the replicaset from nothing, and need no UUID of their own, because Tarantool 3.x identifies an instance by name.

Exit codes:

0  the plan is ready and the archives are downloaded
1  invalid input or another planning or download failure
2  a system operation failed (for example, permission was denied)
3  no cluster recovery point is available around the target time
4  the chain is broken around the target time
5  the target time is outside what the storage covers
6  the point's topology does not match the cluster config
7  the target time falls between two different topologies

A failed plan carries nearest_safe wherever a usable point exists: the recovery times on either side of the target, to offer and retry with. Code 6 also carries the difference that blocked it, manifest against config.

Examples:

tt restore plan --target-time 2026-03-25T10:30:00Z \
    --backup-storage @s3-prod.yaml -d /tmp/restore/
tt restore plan --target-time 1774435800 -c cluster.yaml \
    --backup-storage file:///var/backups -d /tmp/restore/ --format table
TZ=Europe/Moscow tt restore plan --target-time "2026-03-25 13:30" \
    --backup-storage file:///var/backups -d /tmp/restore/
tt restore plan --target-time 2026-03-25T10:30:00Z \
    --backup-storage file:///var/backups -d /tmp/restore/ \
    --cluster-name payments-cluster --environment production
tt restore plan --target-time 2026-03-25T10:30:00Z -c cluster.yaml \
    --replicasets storage-a,storage-b \
    --backup-storage file:///var/backups -d /tmp/restore/

cluster configuration source: a path to the cluster configuration file (cluster.yaml), or a URI of etcd or Tarantool Config Storage. A value that parses as a URI is fetched from the storage; anything else is read as a file.

The URL specifies the etcd or tarantool config storage connection settings in the following format: http(s)://[username:password@]host:port/prefix[?arguments]

  • prefix - a base path to Tarantool configuration in etcd or tarantool config storage.

Possible arguments:

  • key - a target configuration key in the prefix.
  • name - a name of an instance in the cluster configuration.
  • timeout - a request timeout in seconds (default 3.0).
  • ssl_key_file - a path to a private SSL key file.
  • ssl_cert_file - a path to an SSL certificate file.
  • ssl_ca_file - a path to a trusted certificate authorities (CA) file.
  • ssl_ca_path - a path to a trusted certificate authorities (CA) directory.
  • ssl_ciphers - a list of allowed SSL ciphers.
  • verify_host - set off (default true) verification of the certificate’s name against the host.
  • verify_peer - set off (default true) verification of the peer’s SSL certificate.

The command supports the following environment variables:

  • TT_CLI_USERNAME - specifies the Tarantool username;
  • TT_CLI_PASSWORD - specifies the Tarantool password.
  • TT_CLI_ETCD_USERNAME - specifies the etcd username;
  • TT_CLI_ETCD_PASSWORD - specifies the etcd password.

The priority of credentials: environment variables < command flags < URL credentials.

HTTPS connection behavior:

  • The https:// scheme enables TLS for both etcd and Tarantool Config Storage (TCS), even when no ssl_* URL arguments are specified.
  • For etcd, certificate verification is skipped if neither ssl_ca_file nor ssl_ca_path is specified. A configured CA enables verification unless explicitly disabled.
  • TCS keeps the certificate verification behavior of its TLS backend. The default Go backend uses system CAs unless ssl_ca_file is specified.
  • Do not mix http:// and https:// addresses in an etcd endpoint list. If any endpoint uses https://, TLS is enabled for all endpoints.

The --backup-storage flag accepts a URI describing the storage backend, or @<path> naming a YAML config file:

file://<abs_path>?Prefix=<subdir>
  Local filesystem storage. The path must be absolute.
  Query parameters:
    Prefix           - subdirectory within the path (optional).

s3://<bucket>/<prefix>
  S3 storage taken from the standard AWS_* environment variables, so that
  no credential is written in the command line:
    AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY  - credentials (required).
    AWS_SESSION_TOKEN                         - for temporary credentials.
    AWS_REGION, AWS_DEFAULT_REGION            - region.
    AWS_ENDPOINT_URL, AWS_ENDPOINT_URL_S3     - endpoint of an
                                                S3-compatible storage.
    AWS_CA_BUNDLE                             - CA of an endpoint the
                                                system roots do not cover.

s3+https://host/bucket/prefix?AccessKeyID=...&SecretAccessKey=...
s3+http://host/bucket/prefix?AccessKeyID=...&SecretAccessKey=...
  S3-compatible storage with the endpoint and the credentials in the URI.
  Use s3+https for TLS, s3+http for plain TCP. The first path segment
  after the host is the bucket name, the rest is the optional key prefix.
  Query parameters:
    Region           - AWS region (optional).
    AccessKeyID      - access key ID (required).
    SecretAccessKey  - secret access key (required).

@<path>.yaml
  What a URI cannot express: a custom endpoint, a region, explicit
  credentials, a private CA. The backend is chosen by 'type', an unknown
  field is an error, and ${VAR} in a value is read from the environment
  ('$$' is a literal '$'), so a secret stays out of the file:

    type: s3
    endpoint: https://storage.yandexcloud.net # https unless http:// is asked
    for
    region: ru-central1
    bucket: payments-backups
    prefix: tarantool # storage root in the bucket, optional
    access_key_id: ${AWS_ACCESS_KEY_ID}          # optional, AWS_* by default
    secret_access_key: ${AWS_SECRET_ACCESS_KEY}
    ca_cert: /etc/ssl/private-ca.pem             # or skip_verify: true

    type: fs
    root: /mnt/backups/payments        # storage root, must be absolute
    prefix: mycluster                  # subdirectory within the root, optional

--cluster-name and --environment select a subtree of that storage, <storage_root>/<cluster_name>/<environment>/, and every command reading or writing a storage takes them. They compose with the prefix above, so one storage can hold several clusters. Pass the same pair to every command: a backup uploaded with them is invisible to a verify, gc, last or plan run without them, and those commands report an empty storage rather than an error. The exception is upload, which reads the pair out of the plan and needs the flags only to override it.

Usage

tt restore plan [flags]

Flags

  • --backup-storage string - storage URI or @path to a storage configuration file
  • --cluster-name string - cluster name; used as a storage path component
  • -c, --config string - cluster configuration of the cluster being restored, to check the topology of the recovery point against (file path or config storage URI)
  • -d, --dir string - local directory to download the manifests and archives into
  • --environment string - environment tag (production, staging, ...); used as a storage path component, requires --cluster-name
  • --format string - output format: table or json (default json)
  • --replicasets strings - replicasets to restore, by the names the cluster config gives them; the rest are left to bootstrap fresh. Requires -c
  • --target-time string - moment to recover to: Unix seconds, RFC 3339 (2026-03-25T10:30:00Z), or a date, time and zone, such as "2026-03-25 13:30:00+03:00", down to a date; a time without a zone is local
  • --timeout duration - timeout for reading from and downloading out of the storage; 0 means no limit (default 30m0s)