Перейти к содержанию

tt data

Read and replay Tarantool journals - snapshots (.snap), write-ahead logs (.xlog) and vinyl logs (.vylog) - and vinyl runs (.run, .index) without Tarantool.

cat   prints the rows of the files as YAML, JSON Lines or Lua.
play  applies the data changes of the journals to an instance.

Usage

tt data
tt data [flags] <command> [command flags]

Commands

  • tt data cat - Print the rows of snapshots and write-ahead logs
  • tt data play - Apply the data changes of snapshots and write-ahead logs to an instance

tt data cat

Print the rows of snapshots, write-ahead logs, vinyl logs and vinyl runs to stdout. No Tarantool is needed.

The files are read the way Tarantool recovers: the newest snapshot, then the write-ahead logs in the order of their vclocks, without the transactions of its instance the snapshot already contains. A directory gives its .snap and .xlog files (-r: of its subdirectories too); a .vylog file is read when it is named. A write-ahead log an instance is still writing is read as far as it goes.

Bounds select whole transactions: a transaction is taken when any of its rows is within them, so a bound inside a transaction takes all of it. They apply to write-ahead logs; a snapshot is taken whole - the newest one within the upper bounds - and none is taken with a lower bound, which asks for changes, not for a state.

--from N, --to N         the LSNs of one replica, both ends included: the
                         replica --replica names, or the only one the files
                         have rows of. A transaction with no row of it is
                         not taken.
--from-vclock V          the transactions past V: with a row whose LSN is
                         above that of its replica in V.
--to-vclock V            the transactions within V: with a row whose LSN is
                         at most that of its replica in V. A replica V does
                         not name, 0 (local spaces) included, has LSN 0.
--timestamp T            the transactions written before T.

V is {1: 100, 2: 50}, as box.info.vclock prints it, or {"1": 100, "2": 50}. T is one of these, and a time without a zone is local time, of TZ; tt -V logs the moment in UTC it was read as:

1731592956.818                 Unix seconds, integer or fractional
2024-11-14T14:02:36.818Z       RFC 3339, with a zone: Z, +hh:mm or -hh:mm
2024-11-14 17:02:36.818+03:00  the same with a space for the T
2024-11-14 17:02:36.818        no zone: local time; T for the space too
2024-11-14 17:02               to the minute, local time
2024-11-14                     midnight, local time

Of the transactions taken, only the changes of user spaces are kept: the rows of system spaces and those that change no space - RAFT, synchronous replication, NOP - only with --show-system. --space keeps the rows of the spaces it names, system ones included: by id, or by name when the snapshot read defines it. --replica keeps the rows of write-ahead logs the replicas it names wrote.

A snapshot holds no data of vinyl spaces: it is in their runs, the .run files of <vinyl_dir>/<space_id>/<index_id>/, each with a .index file. The runs named are read, and those of a directory (-r: of its subdirectories, as of a vinyl directory), after the journals, by space, index and run. A run holds the statements of one index - INSERT, REPLACE, DELETE, UPSERT - in the order of their keys, the newest first for a key, and its path names the space and the index. The LSN of a statement is the vclock signature - the sum of the LSNs - of its transaction: --from-vclock and --to-vclock bound it by the signature of their vclock, exact for a vclock of the instance that wrote the run; --from and --to by their number, which is the LSN of a replica only on an instance of one replica that changed no local space. --space keeps the runs of a space, --replica does not apply, and --timestamp leaves the runs out: their statements have no time. The row index that ends each page of a run is shown with --show-system; a .index file - the info of the run and of its pages - is shown whole.

-o yaml, the default, prints a YAML document per row; -o json a JSON object per line (JSON Lines). Both have the fields of the xlog module of Tarantool:

HEADER  type (INSERT, REPLACE, UPDATE, DELETE, UPSERT, RAFT, RAFT_PROMOTE,
        RAFT_CONFIRM, NOP, ...), lsn (a row number in a snapshot) and
        replica_id; group_id and timestamp when they are set; tsn, the LSN
        of the first row, on the rows of a transaction of several, and
        commit: true on its last row. Of a vinyl run, the type (a
        statement, ROWINDEX, RUNINFO, PAGEINFO) and the lsn of a statement.
BODY    the body of the row by key: space_id, index_id, key, tuple,
        operations, ... for a data change; replica_id, lsn, term, vclock for
        synchronous replication; term, vote, state, vclock, leader_id for
        RAFT; type and the keys of the record for a vinyl log. None for NOP.
        A statement of a run has the space_id and index_id of its path,
        its tuple (a key, of a secondary index) or key, the operations of
        an UPSERT - a list for each UPSERT squashed into it, the fields
        counted from 0 - and its flags: deferred_delete, skip_read.

In JSON a value with no JSON type is a string: a decimal, a UUID, a datetime in RFC 3339, an interval, a binary value in base64. -o lua prints each data change as a statement of the box API, such as box.space[512]:replace({1, 'a'}), that makes it in Tarantool. Of a run of a primary index it prints the statement that makes what a statement holds - an UPSERT of n lists as n upserts - and of a secondary index nothing: its tuples are keys. Unlike the Lua of a journal, that of runs remakes no space when run: a run is in key order, the newest statement of a key first, and the runs of an index overlap.

Usage

tt data cat <FILE|DIR>... [flags]

Flags

  • -o, --format string - output format: yaml, json or lua (default yaml)
  • --from uint - take the transactions from this LSN of the replica on, inclusive
  • --from-vclock string - take the transactions past this vclock
  • -r, --recursive - read the files of the subdirectories of a directory too
  • --replica uints - keep the rows a replica wrote, by id; may be repeated
  • --show-system - keep the rows of system spaces and those that change no space
  • --skip-corrupt - skip a block of transactions whose checksum does not match instead of failing
  • --space strings - keep the rows of a space, by id or by name; may be repeated
  • --timestamp string - take the transactions written before this time: Unix seconds, RFC 3339, or a date, time and zone, such as "2024-11-14 17:02:36.818+03:00", down to a date; a time without a zone is local
  • --to uint - take the transactions up to this LSN of the replica, inclusive
  • --to-vclock string - take the transactions within this vclock

Examples

# Print a snapshot and the write-ahead logs after it.
$ tt data cat var/lib/myapp/instance-001

# Print the runs of the vinyl space 512, the info of each run and its pages.
$ tt data cat -r var/lib/myapp/instance-001/512

# Print the changes of space 512 from LSN 100 to 200 as JSON Lines.
$ tt data cat 00000000000000000000.xlog --space 512 --from 100 --to 200 -o json

# Print the changes of two replicas up to a vclock as Lua.
$ tt data cat ./wal --to-vclock '{1: 1500, 2: 300}' -o lua

# Print the changes made before a time, the rows of system spaces too.
$ tt data cat -r ./backups --timestamp 2024-11-14T14:02:36Z --show-system

tt data play

Apply the data changes of snapshots and write-ahead logs to an instance, over its binary port, transaction by transaction, in their order. No local Tarantool is needed. The target is an instance of an application, reached at iproto.advertise.client of its config, else at the first TCP address of its iproto.listen, 0.0.0.0 and [::] dialed as 127.0.0.1 and [::1], else at its binary port, with TLS and the TLS files of the iproto.listen entry there unless the flags give them, or a URI: [tcp://][username:password@]host:port or [unix://][username:password@]socketpath, with the TLS settings in the flags or in the query: ?ssl_key_file=...&ssl_cert_file=...&ssl_ca_file=..., and the authentication method in --auth-type or ?auth_type=auto|chap-sha1|pap-sha256.

The files are read the way Tarantool recovers: the newest snapshot, then the write-ahead logs in the order of their vclocks, without the transactions of its instance the snapshot already contains. A directory gives its .snap and .xlog files (-r: of its subdirectories too); a .vylog file is read when it is named. A write-ahead log an instance is still writing is read as far as it goes.

Bounds select whole transactions: a transaction is taken when any of its rows is within them, so a bound inside a transaction takes all of it. They apply to write-ahead logs; a snapshot is taken whole - the newest one within the upper bounds - and none is taken with a lower bound, which asks for changes, not for a state.

--from N, --to N         the LSNs of one replica, both ends included: the
                         replica --replica names, or the only one the files
                         have rows of. A transaction with no row of it is
                         not taken.
--from-vclock V          the transactions past V: with a row whose LSN is
                         above that of its replica in V.
--to-vclock V            the transactions within V: with a row whose LSN is
                         at most that of its replica in V. A replica V does
                         not name, 0 (local spaces) included, has LSN 0.
--timestamp T            the transactions written before T.

V is {1: 100, 2: 50}, as box.info.vclock prints it, or {"1": 100, "2": 50}. T is one of these, and a time without a zone is local time, of TZ; tt -V logs the moment in UTC it was read as:

1731592956.818                 Unix seconds, integer or fractional
2024-11-14T14:02:36.818Z       RFC 3339, with a zone: Z, +hh:mm or -hh:mm
2024-11-14 17:02:36.818+03:00  the same with a space for the T
2024-11-14 17:02:36.818        no zone: local time; T for the space too
2024-11-14 17:02               to the minute, local time
2024-11-14                     midnight, local time

Of the transactions taken, only the changes of user spaces are kept: the rows of system spaces and those that change no space - RAFT, synchronous replication, NOP - only with --show-system. --space keeps the rows of the spaces it names, system ones included: by id, or by name when the snapshot read defines it. --replica keeps the rows of write-ahead logs the replicas it names wrote.

Vinyl runs are not applied, and a .run or .index file named is an error: a run holds the statements of one index of a space in the order of their keys, not the transactions that made them, and the runs of an index overlap. A directory gives no runs. To replay a vinyl space, replay the write-ahead logs that changed it; tt data cat prints a run.

Only data changes are applied: INSERT, REPLACE, UPDATE, DELETE and UPSERT, of system spaces too with --show-system. The requests go in one IPROTO stream, which the instance processes in the order they were sent, --batch of them in flight at a time. --txn-mode says how a transaction of several rows is applied:

auto    in a transaction of the stream, and row by row from the first
        transaction the target cannot take so: a transaction that spans
        requests cannot change a memtx space unless memtx_use_mvcc_engine
        is enabled. The default.
stream  in a transaction of the stream; a target that cannot take it is an
        error.
row     row by row. A reader of the target may see a transaction half
        applied, and a refused row leaves the others of its transaction
        applied.

A transaction of one row is a request of its own in every mode.

--on-error says what to do when the target refuses a row, with the server's error, the LSN and the replica of the row:

stop    stop, the default. The requests already in flight - up to --batch -
        are still applied.
skip    report the transaction and go on.

--dry-run reads and filters the files, checks that the target has the spaces the rows change and applies nothing. A summary of what was applied ends the run. Exit codes: 0 - every transaction was applied; 3 - some were, and the target refused others; 1 - the target refused the first one it was given, or the request is invalid; 2 - the connection failed.

An instance of an application is logged in to with the credentials of --username and --password only. The command supports the TT_CLI_USERNAME and TT_CLI_PASSWORD environment variables for a URI without credentials. Without any credentials, the connection is guest's.

Usage

tt data play (<APP>:<INSTANCE> | <URI>) <FILE|DIR>... [flags]

Flags

  • --auth-type string - authentication method: auto, chap-sha1 or pap-sha256; pap-sha256 needs TLS and Tarantool EE or CE 3.9+ (default auto)
  • --batch int - how many requests are in flight at most; 1 waits for each (default 128)
  • --connect-timeout duration - timeout for establishing the connection (default 1s)
  • --dry-run - check the files and the spaces of the target, and apply nothing
  • --from uint - take the transactions from this LSN of the replica on, inclusive
  • --from-vclock string - take the transactions past this vclock
  • --on-error string - what to do when the target refuses a row: stop, skip (default stop)
  • -p, --password string - password
  • -r, --recursive - read the files of the subdirectories of a directory too
  • --replica uints - keep the rows a replica wrote, by id; may be repeated
  • --scope string - scope the application is looked for in: project, user or system
  • --show-system - keep the rows of system spaces and those that change no space
  • --skip-corrupt - skip a block of transactions whose checksum does not match instead of failing
  • --space strings - keep the rows of a space, by id or by name; may be repeated
  • --sslcafile string - path to a trusted certificate authorities (CA) file
  • --sslcertfile string - path to an SSL certificate file
  • --sslciphers string - colon-separated (:) list of SSL cipher suites the connection can use
  • --sslkeyfile string - path to a private SSL key file
  • --sslpassword string - password for decrypting the private SSL key file
  • --sslpasswordfile string - path to a file containing the password for decrypting the private SSL key file
  • --template string - template of the application the instances are described in (default "default")
  • --timestamp string - take the transactions written before this time: Unix seconds, RFC 3339, or a date, time and zone, such as "2024-11-14 17:02:36.818+03:00", down to a date; a time without a zone is local
  • --to uint - take the transactions up to this LSN of the replica, inclusive
  • --to-vclock string - take the transactions within this vclock
  • --txn-mode string - how a transaction of several rows is applied: auto, stream, row (default auto)
  • -u, --username string - username

Examples

# Replay a snapshot and the write-ahead logs after it into an instance.
$ tt data play myapp:instance-001 ./backup

# Replay the changes of a space past a vclock, skipping the refused ones.
$ tt data play admin:secret@localhost:3301 ./wal --from-vclock '{1: 1500}' \
    --space 512 --on-error skip

# Check what a replay up to a time would change, applying nothing.
$ tt data play localhost:3301 ./wal --timestamp 2024-11-14T14:02:36Z --dry-run