| VTC(7) | Miscellaneous Information Manual | VTC(7) |
VTC - Varnish Test Case Syntax
This document describes the syntax used by Varnish Test Cases files (.vtc). A vtc file describe a scenario with different scripted HTTP-talking entities, and generally one or more Varnish instances to test.
A vtc file will be read word after word, with very little tokenization, meaning a syntax error won't be detected until the test actually reach the relevant action in the test.
A parsing error will most of the time result in an assert being triggered. If this happens, please refer yourself to the related source file and line number. However, this guide should help you avoid the most common mistakes.
The parser splits words by detecting whitespace characters and a string is a word, or a series of words on the same line enclosed by double-quotes ("..."), or, for multi-line strings, enclosed in curly brackets ({...}).
The leading whitespaces of lines are ignored. Empty lines (or ones consisting only of whitespaces) are ignored too, as are the lines starting with "#" that are comments.
Test files take at most one command per line, with the first word of the line being the command and the following ones being its arguments. To continue over to a new line without breaking the argument string, you can escape the newline character (\n) with a backslash (\).
When a string is processed, macro expansion is performed. Macros are in the form ${<name>[,<args>...]}, they have a name followed by an optional comma- or space-separated list of arguments. Leading and trailing spaces are ignored.
The macros ${foo,bar,baz} and ${ foo bar baz } are equivalent. If an argument contains a space or a comma, arguments can be quoted. For example the macro ${foo,"bar,baz"} gives one argument bar,baz to the macro called foo.
Unless documented otherwise, all macros are simple macros that don't take arguments.
NOTE: This command is available everywhere commands are given.
Barriers allows you to synchronize different threads to make sure events occur in the right order. It's even possible to use them in VCL.
First, it's necessary to declare the barrier:
barrier bNAME TYPE NUMBER [-cyclic]
With the arguments being:
Then, to add a sync point:
barrier bNAME sync
This will block the parent thread until the number of sync points for bNAME reaches the NUMBER given in the barrier declaration.
If you wish to synchronize the VCL, you need to declare a "sock" barrier. This will emit a macro definition named "bNAME_sock" that you can use in VCL (after importing the vtc vmod):
vtc.barrier_sync("${bNAME_sock}");
This function returns 0 if everything went well and is the equivalent of barrier bNAME sync at the VTC top-level.
Client and server threads are fake HTTP entities used to test your Varnish and VCL. They take any number of arguments, and the one that are not recognized, assuming they don't start with '-', are treated as specifications, laying out the actions to undertake:
client cNAME [...] server sNAME [...]
Clients and server are identified by a string that's the first argument, clients' names start with 'c' and servers' names start with 's'.
As the client and server commands share a good deal of arguments and specification actions, they are grouped in this single section, specific items will be explicitly marked as such.
However, -dispatch is only allowed for the server name "s0".
To make things easier in the general case, clients will connect by default to a Varnish server called v1. To connect to a different Varnish server, use '-connect ${vNAME_sock}'.
The -vcl+backend switch of the varnish command will add all the declared servers as backends. Be careful though, servers will by default listen to the 127.0.0.1 IP and will pick a random port, and publish 3 macros: sNAME_addr, sNAME_port and sNAME_sock, but only once they are started. For 'varnish -vcl+backend' to create the vcl with the correct values, the server must be started first.
It's a string, either double-quoted "like this", but most of the time enclosed in curly brackets, allowing multilining. Write a command per line in it, empty line are ignored, and long line can be wrapped by using a backslash. For example:
client c1 {
txreq -url /foo \
-hdr "bar: baz"
rxresp
} -run
varnishtest will first try to resolve STRING1 and STRING2 by looking if they have special meanings, in which case, the resolved value is use for the test. Note that this value can be a string representing a number, allowing for tests such as:
expect req.http.x-num > 2
Here's the list of recognized strings, most should be obvious as they either match VCL logic, or the txreq/txresp options:
expect_pattern
Expect as the http body the test pattern generated by chunkedlen ('0'..'7' repeating).
-max : max length of this receive, 0 for all
txreq is client-specific and txresp is server-specific.
The only thing different between a request and a response, apart from who can send them is that the first line (request line vs status line), so all the options are pretty much the same.
These three switches can appear in any order but must come before the following ones.
You can then use the arguments related to the body:
NOTE: This command is available everywhere commands are given.
Sleep for the number of seconds specified in the argument. The number can include a fractional part, e.g. 1.5.
Test that the required feature(s) for a test are available, and skip the test otherwise; or change the interpretation of the test, as documented below. feature takes any number of arguments from this list:
A feature name can be prefixed with an exclamation mark (!) to skip a test if the feature is present.
Be careful with ignore_unknown_macro, because it may cause a test with a misspelled macro to fail silently. You should only need it if you must run a test with strings of the form "${...}".
Write strings to file
The -a flag opens the file in append mode.
Define and interact with haproxy instances.
To define a haproxy server, you'll use this syntax:
haproxy hNAME -conf-OK CONFIG haproxy hNAME -conf-BAD ERROR CONFIG haproxy hNAME [-D] [-W] [-arg STRING] [-conf[+vcl] STRING]
The first haproxy hNAME invocation will start the haproxy master process in the background, waiting for the -start switch to actually start the child.
Arguments:
Executes a vtc fragment:
include FILE [...]
Open a file and execute it as a VTC fragment. This command is available everywhere commands are given.
Reads the VSL and looks for records matching a given specification. It will process records trying to match the first pattern, and when done, will continue processing, trying to match the following pattern. If a pattern isn't matched, the test will fail.
logexpect threads are declared this way:
logexpect lNAME -v <id> [-g <grouping>] [-d 0|1] [-q query] \
[vsl arguments] {
expect <skip> <vxid> <tag> <regex>
expect <skip> <vxid> <tag> <regex>
fail add <vxid> <tag> <regex>
fail clear
abort
...
} [-start|-wait|-run]
And once declared, you can start them, or wait on them:
logexpect lNAME <-start|-wait>
With:
VSL arguments (similar to the varnishlog options):
expect specification:
For skip, vxid and tag, '*' matches anything, '=' expects the value of the previous matched record. The '?' marker is equivalent to zero, expecting a match on the next record. The difference is that '?' can be used when the order of individual consecutive logs is not deterministic. In other words, lines from a block of alternatives marked by '?' can be matched in any order, but all need to match eventually.
fail specification:
add: Add to the fail list
clear: Clear the fail list
Any number of fail specifications can be active during execution of a logexpect. All active fail specifications are matched against every log line and, if any match, the logexpect fails immediately.
For a logexpect to end successfully, there must be no specs on the fail list, so logexpects should always end with
abort specification:
abort(3) varnishtest, intended to help debugging of the VSL client library itself.
This works inside all specification strings
Run a process with stdin+stdout on a pseudo-terminal and stderr on a pipe.
Output from the pseudo-terminal is copied verbatim to ${pNAME_out}, and the -log/-dump/-hexdump flags will also put it in the vtc-log.
The pseudo-terminal is not in ECHO mode, but if the programs run set it to ECHO mode ("stty sane") any input sent to the process will also appear in this stream because of the ECHO.
Output from the stderr-pipe is copied verbatim to ${pNAME_err}, and is always included in the vtc_log.
In most cases, if you just want to start a process and wait for it to finish, you can use the shell command instead. The following commands are equivalent:
shell "do --something" process p1 "do --something" -run
However, you may use the process variant to conveniently collect the standard input and output without dealing with shell redirections yourself. The shell command can also expect an expression from either output, consider using it if you only need to match one.
If you need to use other signal names, you can use the kill(1) command directly:
shell "kill -USR1 ${pNAME_pid}"
Note that SIGHUP usage is discouraged in test cases.
Set or change an environment variable:
setenv FOO "bar baz"
The above will set the environment variable $FOO to the value provided. There is also an -ifunset argument which will only set the value if the environment variable does not already exist:
setenv -ifunset FOO quux
NOTE: This command is available everywhere commands are given.
Pass the string given as argument to a shell. If you have multiple commands to run, you can use curly brackets to describe a multi-lines script, eg:
shell {
echo begin
cat /etc/fstab
echo end
}
By default a zero exit code is expected, otherwise the vtc will fail.
Notice that the commandstring is prefixed with "exec 2>&1;" to combine stderr and stdout back to the test process.
Optional arguments:
(note: this section is at the top-level for easier navigation, but it's part of the client/server specification)
Streams map roughly to a request in HTTP/2, a request is sent on stream N, the response too, then the stream is discarded. The main exception is the first stream, 0, that serves as coordinator.
Stream syntax follow the client/server one:
stream ID [SPEC] [ACTION]
ID is the HTTP/2 stream number, while SPEC describes what will be done in that stream. If ID has the value next, the actual stream number is computed based on the last one.
Note that, when parsing a stream action, if the entity isn't operating in HTTP/2 mode, these spec is ran before:
txpri/rxpri # client/server
stream 0 {
txsettings
rxsettings
txsettings -ack
rxsettings
expect settings.ack == true
} -run
And HTTP/2 mode is then activated before parsing the specification.
The specification of a stream follows the exact same rules as one for a client or a server.
These four commands are about sending headers. txreq and txresp will send HEADER frames; txcont will send CONTINUATION frames; txpush PUSH frames. The only difference between txreq and txresp are the default headers set by each of them.
INT is the index of the header name to use.
The third argument informs about the Huffman encoding: yes (huf) or no (plain).
The last term is the literal value of the header.
The second and third terms tell what the name of the header is and if it should be Huffman-encoded, while the last two do the same regarding the value.
By default, data frames are empty. The receiving end will know the whole body has been delivered thanks to the END_STREAM flag set in the last DATA frame, and txdata automatically set it.
These are two convenience functions to receive headers and body of an incoming request or response. The only difference is that rxreq can only be by a server, and rxresp by a client.
rxhdrs will expect one HEADER frame, then, depending on the arguments, zero or more CONTINUATION frame.
This works like rxhdrs, expecting a PUSH frame and then zero or more CONTINUATION frames.
Receiving data is done using the rxdata keywords and will retrieve one DATA frame, if you wish to receive more, you can use these two convenience arguments:
Receive a frame, any frame.
Push bytes directly on the wire. sendhex takes exactly one argument: a string describing the bytes, in hex notation, with possible whitespaces between them. Here's an example:
sendhex "00 00 08 00 0900 8d"
Receive a GOAWAY frame.
Possible options include:
Same as the gunzip command for HTTP/1.
Receive a PING frame.
Send PING frame.
Receive a PRIORITY frame.
Send a PRIORITY frame
Receive a RST_STREAM frame.
Send a RST_STREAM frame. By default, txrst will send a 0 error code (NO_ERROR).
Receive a SETTINGS frame.
SETTINGS frames must be acknowledge, arguments are as follow (most of them are from rfc7540#6.5.2):
Receive a WINDOW_UPDATE frame.
Transmit a WINDOW_UPDATE frame, increasing the amount of credit of the connection (from stream 0) or of the stream (any other stream).
expect in stream works as it does in client or server, except that the elements compared will be different.
Most of these elements will be frame specific, meaning that the last frame received on that stream must of the correct type.
Here the list of keywords you can look at.
Note: it's possible to inspect a request or response while it is still being construct (in-between two frames for example).
Define and interact with syslog instances (for use with haproxy)
To define a syslog server, you'll use this syntax:
syslog SNAME
Arguments:
The goal of a tunnel is to help control the data transfer between two parties, for example to trigger socket timeouts in the middle of protocol frames, without the need to change how both parties are implemented.
A tunnel accepts a connection and then connects on behalf of the source to the desired destination. Once both connections are established the tunnel will transfer bytes unchanged between the source and destination. Transfer can be interrupted, usually with the help of synchronization methods like barriers. Once the transfer is paused, it is possible to let a specific amount of bytes move in either direction.
Listens by defaults to a local random port.
Connects by default to a varnish instance called v1.
The specification contains a list of tunnel commands that can be combined with barriers and delays. For example:
tunnel t1 {
barrier b1 sync
pause
delay 1
send 42
barrier b2 sync
resume
} -start
If one end of the tunnel is closed before the end of the specification the test case will fail. A specification that ends in a paused state will implicitly resume the tunnel.
The tunnel must be running.
The tunnel must be paused, it remains paused afterwards.
The tunnel must be paused.
The tunnel must be paused, it remains paused afterwards.
Define and interact with varnish instances.
To define a Varnish server, you'll use this syntax:
varnish vNAME [-arg STRING] [-vcl STRING] [-vcl+backend STRING]
[-errvcl STRING STRING] [-jail STRING] [-proto PROXY]
The first varnish vNAME invocation will start the varnishd master process in the background, waiting for the -start switch to actually start the child.
Types used in the description below:
Arguments:
If the ${varnishd_args_prepend} or ${varnishd_args_append} macros are defined, they are expanded and inserted before / appended to the varnishd command line as constructed by varnishtest, before the command line itself is expanded. This enables tweaks to the varnishd command line without editing test cases. This macros can be defined using the -D option for varnishtest.
You can decide to start the Varnish instance and/or wait for several events:
varnish vNAME [-start] [-wait] [-wait-running] [-wait-stopped]
Once successfully started, the following macros are available for the default listen address: ${vNAME_addr}, ${vNAME_port} and ${vNAME_sock}. Additional macros are available, including the listen address name for each address vNAME listens to, like for example: ${vNAME_a0_addr}.
Once Varnish is started, you can talk to it (as you would through varnishadm) with these additional switches:
varnish vNAME [-cli STRING] [-cliok STRING] [-clierr STRING]
[-clijson STRING]
It is also possible to interact with its shared memory (as you would through tools like varnishstat) with additional switches:
varnish v1 -expect SM?.s1.g_space > 1000000 varnish v1 -expect cache_hit >= cache_hit_grace
In the ! form the test fails if a counter matches PATTERN.
The MAIN. namespace can be omitted from PATTERN.
The test takes up to 5 seconds before timing out.
Alternate name for 'vtest', see above.
Interact with the shared memory of a varnish instance.
To define a VSM consumer, use this syntax:
vsm mNAME [-n STRING]
Arguments:
The available flags are:
Expecting a status automatically attaches to the varnish instance if that was not already the case.
This should be the first command in your vtc as it will identify the test case with a short yet descriptive sentence. It takes exactly one argument, a string, eg:
vtest "Check that vtest is actually a valid command"
It will also print that string in the log.
This document has been written by Guillaume Quintard.
This document is licensed under the same licence as Varnish itself. See LICENCE for details.