greater than

expect-lite


Automation for the rest of us



List of Features

Expect-lite has many features which make expect-style script writing easy.

Easy of Use
Variables
  • Static and Dynamic Variables with math functions and pseudo arrays
    • String Math allows manipulation of strings: search/replace, +, - 
  • Constants, passed on the command line or set as environment variables, which override variables in the script. Powerful feature allows any variable to be passed on the command line
  • Leverages Bash shell passing unknown variables to the shell
Programmatic
Advanced
Debugging



Introduction

Automatic Login

expect-lite was born into a multi-host environment, and therefore there was the early requirement to remote login to a host (usually a Linux machine) allocated by a sharing facility such as the commercial package, LSF. Therefore, a remote host must be specified on the command line where the script will wake up and begin executing.

Three methods of remote login are supported:
  • telnet
  • ssh with password
  • ssh with keys (no password)
ssh with keys is the preferred method, since no password is required, however this requires some environment setup before hand.

Even when no remote host is required, it is best to log into the localhost, since the underlying Expect has problems with synchronization (between send and expect strings) when a remote host is not specified. The provided shell script 'setup_local_ssh.sh' will setup the localhost with ssh keys. It is only necessary to run it once:
./setup_local_ssh.sh
Based on the command line arguments passed, expect-lite will do the following:
  • Automatically log into the remote host via telnet or ssh (if ssh keys have been previously setup)
  • Change directory to the passed directory (if included)

In the following example of expect-lite:
./expect-lite remote_host=remote-host-018 cmd_file=basic_ping_test.elt user_dir=/etc
Or the shorter version
./expect-lite r=remote-host-018 c=basic_ping_test.elt d=/etc
Or the shorter version which is more linux like
./expect-lite -r remote-host-018 -c basic_ping_test.elt -d /etc

expect-lite will log into remote-host-018, change directory to /etc/ and begin to send commands from basic_ping_test.elt to the remote-host. The parameters of remote_host and cmd_file are required, as expect-lite must know what host to log into, and what script (aka command file) to run.

Environment Variables

The following environment variables permit the customization of expect-lite without having to edit the expect-lite script itself. It is recommended that commonly used environment variables be placed in the .bashrc or .cshrc file where they will be automatically set upon login.
Env Var
Description
EL_REMOTE_HOST
Name or IP of remote host
EL_CMD_FILE Name of expect-lite script to run
EL_USER_DIR Change to this directory upon login before executing script
EL_CONNECT_METHOD The method expect-lite uses to connect to remote host. Valid options are: none|telnet|ssh|ssh_key
Default is none
EL_CONNECT_USER User name to use for login on remote host (telnet|ssh)
EL_CONNECT_PASS Password to use for login on remote host (telnet|ssh)
EL_LOG_EXT
The log file name extension string, default=".log" (see *LOG)
EL_INFINITE_LOOP=N
Infinite Loop protection value, set N to user's default value (default=5000) (see *INFINITE_LOOP)
EL_DELAY_WAIT_FOR_HOST Delay (in ms) to wait for host in Not Expect, and Dynamic Var Capture. 100 ms is a good value for a local LAN, 200 ms if running across high speed internet
EL_SHELL
Start this shell (default=bash) when using EL_CONNECT_METHOD=none
EL_DYN_VAR_PROMPT
Set the Dynamic Variable capture method to require a prompt (default=1). Disable by setting to 0.
EL_* Any other shell environment variables starting with EL_ will become constants

Special Character Sequences

Although basic expect-lite scripts can be created by simply cutting and pasting text from a terminal window into a script, and adding '>' '<' characters, such as:
>pwd
</home/user
Much more functionality is available. expect-lite interprets punctuation characters at the beginning of each line in the script:
Char
Action
>
send string to the remote host
>>
send string to remote host, without waiting for prompt (see implementation details)
<
string/regex MUST be received from the remote host in the alloted timeout or the script will FAIL! (see test failure)
<< literal string MUST be received (similar to '<' without regex evaluation)
-<
if string/regex IS received from the remote host the script will FAIL! (see Not Expect)
~<
expect an approximate number (see Fuzzy Expect)
#
used to indicate comment lines, and have no effect
;
are also used to indicate comment lines, but are printed to stdout (for logging)
;<colour>
add custom colour comment lines, colour may be blue,  ltblue,  gray,  ltgray,  cyan,  black, pink, purple, red, green, yellow
;;
similar to above, but no extra newlines are printed (useful for printing script help)
;;;
Marks the beginning and end of user defined script help (see script help)
@num
changes the expect timeout to num of seconds
:num
sleeps for num seconds. Mili-seconds also supported e.g.  :1.005 is 1005 mili-seconds (see sleep)
$var=
static variable assignment at script invocation (see variables for details)
+$var=
dynamic variable assignment  (see variables)
+$var increment value of $var by 1 decimal (see repeat loop)
-$var decrement value of $var by 1 decimal
=$var
math functions, perform bitwise and arithmetic operations:  << >> & | ^ * / % + - (see math functions)
=$string
string math functions, maniuplate strings: + - / // (see string math functions)
!
indicates an embedded expect line (see embedded expect)
?
c-style if/then/else in the format ?cond?action::else_action (see conditionals and code blocks)
[ <test>
]
while loop (see code blocks)
[ $var=list
]
foreach loop (see code blocks)
%
label - used for jumping to labels (see conditionals)
~filename var=value
includes a expect-lite automation file, useful for creation of common variable files, or 'subprograms/subroutines' (see include files)

In addition to the above special characters, blank lines or lines starting with any non-special character are allowed to make the script file more readable.

Sometimes control characters must be sent. Currently all control characters are supported (from ^@ to ^\):
  • ^C or break, when it is desired to stop a running program e.g. >^C or >>^C
  • ^] or escape from telnet, e.g. >^] This will bring the script to the telnet prompt, then use: >quit
  • ^<any char> will send a control character to the remote-host, such as ^D to logout
Control characters must be the first two characters on the line (e.g.[^][A-Z]). The sequence of ^C anywhere else in the line will not be interpreted as a control-C, but rather the two characters '^' 'C'.

Lastly, any line beginning with an asterisk '*' will be interpreted as an expect-lite directive. Directives change the behaviour of expect-lite.

Expect-lite Directives
*~filename Include a fail script, which expect-lite runs only if the main script fails (see fail script)
*/prompt/
Set a user defined prompt (see user defined prompt)
*TERM <N>
Immediately terminates script, but returns 0 (pass), or if N is specified returns the value of N (0..255)  (see stopping execution early)
*FAIL
Immediately fails script, and returns 1 (fail) (see stopping execution early)
*NOFAIL Script will run to completion, and returns 1, if failure occurred during execution, or 0, if pass
*NOINTERACT
Will ignore breakpoints (*INTERACT), good for regression testing (see directives)
**SHELL=<shell>
Configuration directive which sets the shell immediately after automatic login to remote host (see setting the shell)
*INTERACT
Sets a breakpoint and places user in interact mode, which pauses the script and turns control of the keyboard over to the user (see Tips & Techniques: Interact)
*FORK <session> Multiple session support. Directs expect-lite to open a new session and spawns a newshell (see multiple sessions)
*SHOW VARS
Debug information, displays all expect-lite variables. Can be used in interact mode.
*EOLS LF
*EOLS CRLF
Controls end of line sequence sent to remote host, either line feed, or carriage return & line feed.
*DVPROMPT
*NODVPROMPT
Enable/disable the Dynamic Variable capture method to require a prompt (default=*DVPROMPT).  (see Dynamic Variable capture method)
*INFINITE_LOOP <N>
Sets/resets the infinite loop protection value, which is a total count of all loops in script. Default is 5000
*NOINCLUDE
Ignore include files (was default behavour in library mode)

Logging Directives
*LOG
*NOLOG
Enable/disable logging to a file. Will create file <script_name>.log (same as $arg0.log) if no path/file_name is supplied.(see managing logging)
*LOG  <file_name>
*LOGAPPEND <file_name>
Enable  logging to a user specified path/file_name. When invoked on the CLI, the specified file_name must end in ".log" to  avoid ambiguity
*INFO
*NOINFO
Enable/disable informational messages
*WARN
*NOWARN
Enable/disable warning messages
*DEBUG
*NODEBUG
Enable/disable debug messages
*NOCOLOR Disables color output, best when logging output
*TIMESTAMP <ISO|YMD|DMY|MDY> Prints Date and Timestamp for each command sent, ISO is default
*NOTIMESTAMP Disables timestamp printing

IPv6 Support

expect-lite is an application, and will use communications protocols such as telnet and ssh to connect to a remote host. If telnet or ssh is IPv6 enabled, then expect-lite can/will use it.

However, it may be important to include IPv6 addresses in expect-lite scripts. expect-lite now fully supports IPv6 addresses including short hand notation with double colons, e.g. 2001:db8::dead

Care has been taken to have expect-lite do the right thing with IPv6 addresses, however, it may be necessary to be more explicit when using conditionals (if/then/else), see conditionals and IPv6.

General Tips for writing scripts

Here are some simple tips when script writing:
  • Use reasonable timeouts, if 30 seconds is needed to get a response, set the timeout at 45 or 60 seconds, not 600.
    • There is no cost to changing the timeout, timeout values can also be variables
  • Beware of expect-lite using regex, when creating lines like: <0.005 secs (5 micro secs)
    • The parentheses is used by the regex engine, instead escape these characters: <0.005 secs \(5 micro secs\) (see Use of Regex in expect-lite)
    • or use '<<' which does not use regex, and does not require escaping: <<0.005 secs (5 micro secs) (see using non-regex evaluation)
  • Use the expect character '<' or '<<' often. Check for valid results when possible. A script which expects nothing will never fail! (see Test Failure)
  • Use printable comments ';'  and ';;' often. Think of it as writing a note to oneself, it will make reading log files much easier. As of version 3.7.0 printable comments will be coloured blue (this is user configurable).
  • Variable assignments use no spaces e.g. $var=value. Note no spaces around the equals sign, this permits leading spaces in the value such as $var=  myvalue

Debugging Expect-lite Scripts

Expect-lite was written to permit quick and easy automation of repetitive tasks. However, sometimes a little additional debugging is required.

Instant-Interact (as of version 3.5.0) or instant breakpoint, provides instant debugging assistance. Pressing ^\ (control+backslash) will force expect-lite to drop into interact mode, a mode which turns control of the keyboard over to the user, so that one may type directly to the process on the remote host. To return to the script, type '+++' (three pluses). During interactive mode, timeouts do not apply, *NOFAIL is automatically enabled, and the script will wait indefinitely for the user to exit this debugging mode. Switching between multiple sessions (see *FORK) is possible via typing the *FORK command in interact mode (see *FORK debugging).

Additionally, the interact mode (or Integrated Debugger Environment) performs three primary functions: 1) connecting the user to the remote host or device under test, 2) monitoring special commands prefaced with the escape key for stepping, and other functions, and 3) the debugger will allow expect-lite script lines to be executed by either typing directly or pasting them into the IDE.

See the Tips and Techniques Guide for additional expect-lite script debugging information.




Reference

Use of Regex in expect-lite

Support of regular expressions in expect-lite are limited by:
  1. Support of regex in standard expect (e.g. including anchors, char-classes and repeats)
  2. Support of regex meta characters (e.g. \t \n \d)
  3. Expect-lite only evaluates lines using regex which begin with '<'  '-<'  '~<' and '+' (dynamic variables)
The following is a table of supported meta-characters, and repeats

Meta-Char
Description
\d
any digit
\D
any non-digit
\w
any word char
\W
any non-word char
\s
any white space
\S
any non-white space
\t
tab
\r
return
\n
new line
.
any character
|
regex OR
Repeats

?
0 or 1
*
0 or more
+
1 or more


As an example, a range of numbers is valid for an IP address. The following would permit the last octet of the IP address to be 2 digits, but not 3.
> /sbin/ifconfig eth0
<inet addr:10\.29\.200\.\d\d\s
The periods are preceded by a backslash to indicate to regex that a period must be returned rather than the regex dot '.' which indicates any character. The \d indicates any single digit.

Because regex is always enabled for expected results, some 'escaping' of characters must be done when using '<'. The following characters must be escaped with a backslash to convey their literal meaning:

Example
Escaped
Character
(abc) \(abc\) parenthesis
[abc]
\[abc\] 
square brackets
\
\\
backslash
. \.
period
$ \$
dollar sign
+ \+ plus
* \* asterisk (or star)
| \| pipe

Regex is a very powerful syntax unto itself, and one can create quite complex regex strings. There is an excellent regex cheat sheet created by David Child, which is very helpful. That said, one does not need to be a regex expert to use expect-lite, it is possible to use regex in a very basic way, see examples for more info.

Using << for non-regex evaluation

There are times when the use of regex is not desired, or 'escaping' characters is burdensome. As of version 3.6.0, expect lines can begin with '<<' which will not use regex, but rather literal evaluation of expected results. 

Handling HTML/XML Tags in expect lines
The << may cause a backward compatibility problem, if the script was expecting an HTML tag, such as
>grep body my_html_file.html
<<body.+>
In the above example, the script is expecting a body tag with 1 or more characters between body and the closing greater-than of the tag. However the new feature will not interpret this line as regex, but rather interpret the beginning of the line as <<, and evaluate it as a literal. The method to force expect-lite to evaluate the line as regex, is to change the line is slightly:
>grep body my_html_file.html
<[<]body.+>
By making the first expected character as a regex character class, it signals to expect-lite that this line should be evaluated as regex, rather than a literal.

Using Variables

Expect-lite supports two types of variables:
  • Static, which are bound at invocation
  • Dynamic, which are bound during execution of the script
Lines starting with a '$' indicate variables which are assigned and set at script invocation, and therefore are static. In expect-lite variable names will always be preceded with a '$'. Variable names are limited to the following set of characters [A-Za-z0-9_]. When making variable assignments, there must NOT be a space between the variable name and the equals sign:
$myvar=myvalue
There are times when it is necessary to unassign or unset a variable, because, it is desired to have the variable be dereferenced by the underlying shell (see bash shell), or that pseudo- arrays are being used (see pseudo-arrays). To unassign a variable (v4.8.0), merely assign it an empty value.
$myvar=

Dynamic Variables

Sometimes it is desirable to bind a variable during the execution of the script. Dynamic variables fill this need by utilizing Expect's built-in regex capture mechanism. Only the portion of the match inside the parenthesis will be bound to the variable. The format of the line is as follows:
+$somevar=text that is not put into the var (text which is put into the var)

Dynamic Variables are always bound to Expect output, meaning text which is returned from the remote host. Therefore something must be sent to the remote host, before text can be returned. A typical usage would be:
> command
+$myvar=command output (capture value) more command output

If the value of the var is successfully captured then expect-lite will print:
Assigned Var:somevar=sometext which is put into the var

If, however the dynamic var is not successfully captured, expect-lite will print:
Assigned Var:somevar=__NO_STRING_CAPTURED__
Avoid using an overly general capture as it will tend to capture too much, or the wrong info. e.g:
> command
+$myvar=\n(.*)
Instead use a specific capture when possible. When capturing the current directory, the current directory will always start with '/', and the path will be made up of a known character set:
> echo $PWD
+$mypwd=\n(/[a-zA-Z0-9/\-_]+)

Expect-lite will print to stdout the value of the assigned variable, assisting the script writer in understanding the value that is bound to the variable during runtime. The value of "__NO_STRING_CAPTURED__" indicates that the regex pattern used did not match the return data. It is best to start using this feature with short scripts targeted at capturing the desired information. Examples can be found in the example section of this document.

Math Functions with Variables

There are times when writing a script it may be necessary to perform arithmetic or bitwise operations, also known as math functions. Expect-lite inherits expect/tcl math functions and natively supports math/bitwise operations on a variable, using the following slightly odd notation:
$myvar=1
=$myvar + 2 * 10
In the above example, 2 will be multiplied by 10 and added to value of $myvar (1). The result of 21 will be placed in $myvar, overwriting the previous value. Since multiplication has a higher precedence than addition, (2 * 10) will be performed before the addition to $myvar.

Parenthesis may be used to enforce user defined precedence. In the following example, the result will be 23:
$myvar=1
=$myvar + 2 * (10 + 1)

Both bitwise and arithmetic operators are supported with the following precedence (left to right):
bitwise
<< >> & ^ | shift left, shift right, bit and, bit exclusive or, or
arithmetic  * / % + -
multiply, divide, modulo, add, subtract


If $myvar does not exist before the math function, it will be initialized to blank and then math functions will be performed. In the following example the result, $myvar, will be 20:
=$myvar + 2 * 10
However, realistically, only the '+' operator works as expected in the above example. Using other operators will more than likely yield a syntax error:
Warning: Expect-lite: Unable to interpret =$answer1 / 2 * (10 + 1) 
syntax error in expression " / 2 * (10 + 1)": unexpected operator /

Because of the expect/tcl inheritance of math functions, all operations will be performed as integers (or whole numbers) unless a decimal (or real) number is used explicitly. In the following example the result is 2.5:
$five=5
=$myvar + $five / 2.0
However, the following (because of the use of integers) example will have the result of 3:
$five=5
=$myvar + $five / 2

Math functions can be used with static and dynamic variables.

String Math Functions with Variables

With the introduction of version 4.6.0, String Math functions have been added. The functions which enable simple manipulation of strings are:
  • Concatenation using +
  • Removal using -
  • Search and Replace using / or //
In order to use string math functions, the variable must already be defined as a string (think: non-numeric). The following is an example of concatenation:
$mylist=abc,def,hig
$lmnvar=,lmn
=$mylist + $lmnvar
This will result in $mylist being equal to abc,def,hig,lmn

String removal will remove all instances of the substring
$mylist=abc,def,hig
$lmnvar=,def
=$mylist - $lmnvar
The result of $mylist will be abc,hig

Search and Replace

Search and replace is the most powerful of functions and uses the slash '/' operator. The form is as follows:
=$myvar /search/replace/
There is a regex / and non-regex // form.

Regex
=$var /search/replace/
Regex support for search and replace,
includes char class, meta chars, capture
Non-regex =$var //search/replace/
Literal search and replace

The non-regex is straight forward, in the following example, one might want to replace commas with spaces (to be used in foreach loop list):
$mylist=abc,def,hig
=$mylist //,/ /
The result would be: abc def hig

The regex version can do more complex operations such as removal of the subnet for IPv4:
$myip=192.168.0.50
=$myip /\d+\.\d+\.\d+\.(\d+)/\1/
The above example searches for a string of digits, dot, digits, dot, digits, dot, then lastly captures the last set of digits. It then replaces the entire IP address with captured value (using the \1), resulting in $myip equaling 50.


Pseudo Arrays

Expect-lite supports pseudo arrays, which are different from a real array (e.g. item[1], item[2],...). Pseudo arrays are made possible by the fact that variables on the left side of the equals can be dereferenced at assignment time. For example, below, the variable $count will be dereferenced (or resolved) prior to assignment:
$myarray$count=some $value
When placed in a loop with a looping variable of $count, a pseudo array of $myarray1, $myarray2, $myarray3,... will be created. Individual values can be retrieved from the array by using the index (or $count) such as $myarray23.

The pseudo array name (variable + index) must be a valid variable name, in the set of characters (A to Z, a to z, 0 to 9, and underscore) and no spaces.

Constants

Expect-lite Constants are represented as variables which are passed into expect-lite at runtime or retrieved from environment variables starting with EL_. Constants will override any script variable already defined inside the script, are immutable, and cannot be changed. Constants can be used to change the behaviour of the script.

For example, one might invoke the following test as:
./expect-lite remote_host=remote-host-018 cmd_file=basic_ping_test.elt \
user_dir=/etc local_eth=eth2

Inside the script basic_ping_test.txt any reference to $local_eth would be set to eth2, thus allowing the script actions to be changed based on the constant passed at invocation. Constants will override any script variable already defined inside the script.

Shell Variables

Expect-lite allows the use of shell variables, which can be more powerful than the built-in variable mechanism. However shell variables will only be resolved by the shell. For example, assigning current working directory to a shell var:
> PWD=`pwd`
> echo $PWD
However, the following test would fail, since $PWD is a shell variable, not an expect-lite variable:
> PWD=`pwd`
>pwd
< $PWD
Shell variables (or environment variables) must be dereferenced by the shell. Similarly, expect-lite variables must be dereferenced by expect-lite.

It is possible to read a shell variable into an expect-lite variable by using the dynamic variable method:
> echo $PWD
+$mypwd=\n(/[a-zA-Z0-9/\-_]+)

Undefined Variables

Since expect-lite allows the use of shell variables, it is possible that a variable will not be known to expect-lite (but known to the shell). Expect-lite will attempt to dereference all variables, the ones which are not defined remain in variable form (perhaps it is a shell variable). However, it can be useful to test if a variable is an expect-lite variable. As of version 4.1.2, in a conditional (if statement) undefined variables will be automatically defined as blank ( or ""). See conditional for more information.

Environment Variables starting with EL_

When expect-lite starts up it will automatically search the environment variables (aka shell variables), and import any found starting with EL_ as constants. This enables setting environment variables as passwords, or remote hosts.

For example, the environment variable EL_sudopass might be set to "mysecret". In the script, the variable would be used as:
>sudo tcpdump -i eth0
<<[sudo] password
>>$EL_sudopass

Using Conditionals & Labels

Conditional (if/then/else) statements are natively supported in expect-lite. The conditional uses a c-style syntax with a question mark (?) at the beginning of the line, and double colon to indicate the else statement, using the format ?cond?action::else_action

Although spaces are not required around the conditional characters (? and ::), it is recommended for ease of reading. The comparison operators are: '==',  '!=' ,'>=',  '<=',  '>' and '<'. If the compared values can be evaluated as a numbers, then larger and less than will yield expected results. A simple conditional example:
$age=56
? $age >= 55 ? >echo "freedom at 55!" :: > echo "keep working!"
In the above example if $age is larger than or equal to 55 then the action 'echo "freedom at 55!". If $age is less than 55 then the action 'echo "keep working!"' will be sent.

Each action or else_action is as if it began on a separate line. Since in the above example '>' is used (e.g. >echo) the echo line will be sent to the remote host. Any expect-lite action (see Special Characters) can be placed after the second question mark. For example, an include file may be executed based on a conditional:
$platform=ppc
? $platform == i386 ? ~connect_i386.inc :: ~connect_ppc.inc
In order to make log files more understandable, a message will be printed when a conditional is evaluated, such as:
If: expr { "ppc" == "i386" } |then ~connect_i386.inc|else ~connect_ppc.inc|result0
The message prints: what was evaluated (expr { "ppc" == "i386" }),  'then', and 'else' actions, as well as a result (true or false)

IPv6 Addresses

Because IPv6 addresses are quite long, it is a common short hand to represent a long string of zeros with a double colon '::', such as 2001:db8::f00d. The conditional also uses a double colon to signal the else portion of the conditional, so what happens when:
$platform=ppc
? $platform == i386 ? $my_addr=2001:db8::f00d :: $my_addr=2001:db8::feed
Fortunately, as of version 4.2.2, it will be as expected, $my_addr will be set to 2001:db8::feed

If in doubt, be sure to place spaces around the else double colon, as spaces are never allowed in an IPv6 address.

Undefined variables

Undefined variables can be tested in a conditional (as of version 4.1.2). In and only in a conditional, variables unknown to expect-lite are dereferenced as blank. This allows testing if a variable has been defined such as:
$blank=
?if $TEST ==$blank? ; === echo this :: ; === echo that
?if $TEST != ? ; === echo this :: ; === echo that
The first example, is setting a variable to blank and then testing for equality. The second example demonstrates testing inequality with blank. Either conditional will work, and it is a matter of style which one is preferred.


Labels

Conditionals are limited to a single line. Sometimes this is too limiting, as it would be nice to have several commands be executed based on the success of a conditional. To support this, the concept of labels has been introduced. A label is defined as having the first character a '%'. Although the label line itself does nothing, it provides a location to the conditional to Jump To Label. The following is a simple example of using a conditional in conjunction with a label:
$kmodules_inserted=true
? $kmodules_inserted == false ? %SKIP_CHECK_KMODS
>lsmod | grep nfs
<nfsd
<exportfs
<sunrpc
%SKIP_CHECK_KMODS
When a label is the action of a conditional, there is an implied Jump to Label. Lines between the conditional and the label will be skipped. The Jump to Label action is no longer limited to only jumping forward. (see Looping with Conditionals)

Multiple labels with the same name are permitted. For example, a conditional action may be Jump to Label %SKIP. However this is only recommended for looping (as of version 4.1.1 expect-lite searches backwards for labels) Of course it is easier for the script reader if clearer label names are used. Labels may contain spaces, as in this example:
; === Test conditional jump to label
? $jump == true ?%move along, nothing to see
>echo "1"
>echo "2"
>echo "3"
%move along, nothing to see
To assist reading the logs, the action Jump to Label will generate a message to standard out (captured in the log). From the above example, the following would be printed:
Jumping to label:%move along, nothing to see

Looping with Conditionals & Labels

Simple looping is supported by allowing jump to label backwards in the script. The Repeat Loop is the easiest loop to implement:
; ======== Incrementing Loop ========
$max=5
$count=3
%REPEAT_INC_LOOP
>echo $count
# increment variable
+$count
?if $count <= $max ?%REPEAT_INC_LOOP
Because of jump to label default behaviour has been changed in version 4.1.1 to jump backwards, it is no longer important to assign unique looping labels. Non-looping labels, as illustrated in the previous section, must (as of version 4.1.1) be unique.

Also included in the above example is incrementing an expect-lite variable: +$count
This will add 1 to the value of $count. If $count is not an integer, the value of $count will remain unchanged (can't add 1 to a string).

As part of the looping enhancement, there is infinite loop protection. The maximum amount of looping is defined with the directive *INFINITE_LOOP N. This value is decremented with each iteration of all loops for the entire script. Typically this would be in the range of 1000 to 10000 to be safe. For example, if a complex expect-lite script had 4 loops each with 100 iterations, the directive *INFINITE_LOOP should be set larger than 400. As of version 4.5.0, the environment variable EL_INFINITE_LOOP is used to override the default value.

Using Code Blocks with Conditionals, While Loops and Foreach Loops

expect-lite is a line oriented scripting language, but there are times when it is convenient to group lines together. As of version 4.2.0, the grouping of lines can be achieved through code blocks. A quick example using a code block and a conditional:
; === Conditional using code block
? $case == true ? [
>echo "1"
>echo "2"
>echo "3"
]
Conditionals

A code block begins with a [ (left square bracket) and ends with a ] (right square bracket). When used with an if statement, it allows more than one "line" to be executed based on the evaluation of the if statement.

Conditionals with code blocks also support the else construct, such as:
; === Conditional using code block as else statement
? $case == true ? >echo "true" :: [
>echo "false 1"
>echo "false 2"
>echo "false 3"
]
As of version 4.5.0 Code blocks are supported in both the then and the else sections of a conditional:
; === Conditional using code block then and another code block else statement
? $case == true ? [
>echo " 1"
>echo " 2"
>echo " 3"
]::[
; else code block
>echo " 4"
>echo " 5"
>echo " 6"
]
However, using the square brackets on the same line is still not supported, since it appears ambiguous:
; === NOT SUPPORTED Conditional using code block then and else statement
? $case == true ? [ :: [
>echo " 1"
]
Nested code blocks can also be used. In the following example each if statement is using a code block for the then statement, and a non-code block for the else statement:
$test=3
; === nested ifs using code blocks
?if 1 < $test ? [ :: >echo "else 1"
?if 2 < $test ? [ :: >echo "else 2"
>echo "if 2"
]
]

While Loops

Code blocks can also be used for quick looping using an implied while loop construct. In the following example, the expression at the beginning of the code block will be evaluated, if true, the lines inside the code block will be executed. The expression (e.g. $i < 3) will be evaluated each time the code block has completed execution. Once the expression evaluates false (in the example $i is equal to 3), the script will continue on the line after the bottom of the code block. The <test> operators are:
Test
Action
==
equal
!=
non equal
<=
less than or equal
>=
greater than or equal
<
less than
>
greater than

The following example tests if $i is less than 3, and continues to loop until $i is equal or greater than 3:
; === while loop
$i=0
[ $i < 3
>echo "hello $i"
<hello $i
+$i
]
Nesting of while loops are also supported:
;; == Nested while loops, should repeat "hello ken" 3 times
;; == with 2 "bye now's inside each outer loop
>
$i=0
[ $i < 3
>echo "hello ken"
<ken
$j=0
[ $j < 2
>echo "bye now"
>
+$j
]
+$i
]
Foreach Loops

Code blocks also support foreach loops, where a loop will iterate of a known number of items. Each iteration, the variable will be set to the next item in the list. Items in a list must be separated by spaces. As of version 4.6.1, the list variable will be normalized (extra spaces removed) before executing the loop. Use string math functions to create a list of items separated by spaces.

In the following example, a list of network connected devices are named after planets. The variable $planet will be set to mercury, then ping it. Next time through the loop, $planet will be set to venus, then ping it, and so on. After the last item, neptune, the loop will complete and the script will continue after the bottom of the code block.
; === foreach loop
[ $planet=mercury venus earth mars jupiter saturn uranus neptune
>ping -c1 $planet
]
Of course, foreach loops can be nested with other foreach loops, or while loops.

Infinite loop protection applies both the "jump-to-label" method as well as to the "while-loop" and "foreach-loop" methods, each iteration of the loop decrements the infinite loop counter (default is 5000). It may be a good idea to set the infinite loop value to something low, when first writing code block loops, just in case:
*INFINITE_LOOP 20

Slowing Loops with sleep

For years, I have resisted putting a sleep function in expect-lite. Sleep is often abused by scripters who blindly put in a 60 second sleep (or longer). I have seen 5 minute (600 seconds) sleeps put into scripts because the scripter didn't want to poll for a change in device status. Long sleeps are just bad programming, and should be avoided.

Version 4.9.0 adds native support for sleep. There are very valid reasons why in the polling loop, the scripter may want to slow things down, and only check device status, say every 5 seconds. This can easily be done with a while loop:
; === while loop with sleep
$eth_status=DOWN
[ $eth_status == DOWN
>ip link show dev eth0
+$eth_status=(DOWN|UP)
:5
]
The script will check eth_status of eth0 every 5 seconds, and continue (exit the while loop) after $eth_status is no longer DOWN.

While the script is sleeping, if *INFO is enabled, expect-lite will print out a ticker of "dots" to indicate that the script is sleeping:

$ ip link show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

Assigned Var:eth_status=UP

Sleeping: 5
....+
Sleep "dots" include dots, plus (every 5 seconds) and a number (every 10 seconds), to show the progress of the sleep.
Sleeping: ....+....10....+....20.... 
Mili-second sleep is also supported by using a decimal value for sleep, e.g. 0.005 equals 5 mili-seconds
#sleep 1/10th of a second
:.1
Of course, sleep value can also be a variable:
#sleep using a variable
$sleeper=5.5
:$sleeper
Please use sleep wisely.

Script Help with --help -h

Once the script running, it might be handy to add help to make it more user-friendly or refresh the mind when running the script at a later date. As of version 4.3.0, t is possible to use the build-in script help function which when --help or -h are typed on the command line, a help section and assigned variables will be printed.

The help section is defined in the script as a block of text which starts with a triple-semi-colon, and ends with a triple-semi-colon, text inside this block will not be executed by the script. For example:
;;;
Test: Test bash here doc

Assumptions: bash
Platforms: anywhere bash runs
;;;
Then when running the script, with the --help or -h cli argument, the following will be printed
$ ./test_bash_here_doc.txt -h
Help for: ./test_bash_here_doc.txt

Test: Test bash here doc

Assumptions: bash
Platforms: anywhere bash runs

Displaying assigned variables, which can be overridden with
a Constant on command line e.g var=value
tempfile=junk
The second portion of the help is a list of variables inside the script which can be overridden as constants (see Constants) on the command line. The assigned variables section of the help will always be printed regardless if a triple-semi-colon help block has been defined in the script. This is useful for older scripts which do not have a triple-semi-colon help block.

Constants are a very powerful feature of expect-lite which allow the behaviour of the script to change without having to edit the script. In the example above, if the script were run with the following arguments:
$ ./test_bash_here_doc.txt tempfile=/tmp/mydata.txt
When the script runs, rather than creating a file called 'junk', it will instead create a file in the /tmp directory called mydata.txt


Expect-lite & Bash

Although not limited to working with bash, bash is invoked upon logging into the remote host, and therefore will be discussed more here.

Since the bash shell is well documented, and supported, and therefore can be leveraged to assist in expect-lite's limitations such as looping and branching (see limitations below). A simple bash loop inside expect-lite can be created for example:
$set=1 2 3 4 5
>for i in $set #expanded by expect-lite
>{
>echo $i #expanded by bash
>}
In the above example, $set is an expect-lite variable, not a bash variable. An expect-lite variable is always declared before its use (e.g. $set= 1 2 3 4 5). If a variable cannot be dereference by expect-lite it is passed to the shell. The loop will execute after the final "}" line is sent to the remote-host. Because of this "execute after" effect, an expect-lite '<' line can not be used within the bash loop.

Another example is the bash while loop:
$max=5
>i=0
>while [ $i -lt $max ]
>do
>echo $i
>let i+=1
>done
Conditionals via bash are also supported, as in this example:
$max=4
>if [ $max -ne 5 ]; then
>echo "max is not equal to 5"
>else
>echo "we have a winner"
>fi
More complex bash assists can be constructed. For example, one may want to set a looping variable based on the processor type of machine at the time (some hosts may be faster than others). The answer is echoed, and then captured using a dynamic variable. Note that that 'proc' is prepended to the answer, as it enables the capture to be much more specific.
>PROC=`uname -i`
>if [ "$PROC" == "i386" ]; then
>echo proc20
>else
>echo proc100
>fi
+$loop_counter=\nproc([0-9]+)
Note that 'proc' is outside the parenthesis, and therefore will not be captured into the expect-lite dynamic variable $loop_counter.

Using Bash to create executable expect-lite scripts
It is possible to make expect-lite scripts executable. The old way was to use an embedded bash script, but version 4.0.1 makes this task more standard. Insert the following at the top of the expect-lite script:
#!/usr/bin/env expect-lite
The final step to making the script executable is to use a chmod command:
chmod 775 my_expect-lite_script.elt
Now the expect-lite script can be run my typing the following:
$ ./my_expect-lite_script.elt -r host-15 -u craig max_loop_count=100
The old bash scriptlet method can be found in expect-lite.proj/examples/test_masquerade.txt

Include Files

Include files are a quick way to develop script snippets which can be included into larger scripts or to include a common variable file. When an include file is executed, it is as if the file were just pasted into the script file, and therefore has access to the variable space of the main script, and can modify that variable space as well. In this example, a common variable file is "sourced":
# Source common variable file
~asic_vars.inc

Common functions, such as telnet'ing to the DUT, are a good use of include files. With version 4.3.0, it is possible to pass var=value parameters to the include file. The var=value operate just like constants overriding variables inside the include file (with a scope of just the include file). In the following example it is possible to override the variable 'user' and assign the value of 'root':
; === Connect to DUT 
~dut_connect.inc user=root
Include filenames can also be assigned in a variable, such that the file names can be declared at the top of the script but used later within the script. For example:
# Source Var file to be used
$asic_include=asic_vars.inc
...
~$asic_include
Include files can also be used for a simple regression, including each sub-script (see regression example) . If the *NOFAIL directive is used, and something inside an include file should fail, a failure message will be reported at the completion of the include file:
##Include Result: FAILED: dns_connect.inc    
Limitations on include file names

With the version of 4.7.0, a line beginning wih tilde character has a dual purpose, marking a fuzzy expect line, and also include files. This means that an include file name must NOT start with a greater than '<' or an equals '='. All other characters should be allowed in an include file name.

Embedded Expect

There are situations when expect-lite cannot provide a solution. Rather than force the user to abandon the simplicity of expect-lite, an embedded expect is supported. This functionality is provided to assist expect-lite rather than replace it. Lines beginning with '!' will be interpreted as native expect (see limitations below).

Embedded Expect requires the script writer to know  native expect. It runs with some variable protection, and provides access to expect-lite variables and constants, as well as the expect variable timeout. This permits the script writer to query the timeout, store it away, and then reset the timeout later.
# Preserve old timeout using embedded expect
!set user_namespace(TIMEOUT) $timeout
# set timeout to 15 seconds - max time to wait for login prompts
@15
>telnet $dut_ip
<login
>>root
<assword
>>my_secret
>
# Set the timeout back to old timeout
@$TIMEOUT
Expect-lite variable/values are stored in the tcl array user_namespace() with the variable name as the index name. Constants are stored similarly in the tcl array cli_namespace().

Expect code is collected in islands and executed after an island shore is reached. The island shore is represented by lines that start with any special character, except '#' and blank lines. However, it is recommended that the '>' be used, as this implies a "wait for prompt."

Branching and looping are supported in embedded expect e.g.:
!if { $arch == "ppc" } { 
! puts "\narch is $arch\n"
!}
>
And looping:
!for {set j 1} {$j<6} {incr j} {
! if {$j == 1} {set type "abc" }
! if {$j == 2} {set type "def" }
! if {$j == 3} {set type "hij" }
! if {$j == 4} {set type "lmn" }
! if {$j == 5} {set type "qrs" }
!
! puts "$j>$type"
!}
>
Provides access to expect send & expect commands:
!send "ls\n"
!expect -re "test" { puts "got test"} timeout {puts "got nothing"}
Embedded expect can fail the entire expect-lite script by calling the built-in _el_fail_test function:
!if { $arch == "ppc" } { 
! puts "\narch IS $arch\n"
! } else {
! puts "\narch is NOT ppc, but $arch\n"
! _el_fail_test
! }
>
TCL files can be sourced, and functions can be declared and called from within embedded expect.

Limitations of Embedded Expect:
  • Limited support of expect global variables. To declare a variable at the top level as global use "set ::<var> <value>" The double colon '::' makes var global (this is a TCL standard)
  • tcl/expect switch command not supported
  • expect statements must be on one line (as in the example above)

Please remember, this functionality is to provide assistance to expect-lite rather than replace it. (see What's Missing)

Expect-lite Directives '*'

Expect-lite directives are indicated by lines which start with the asterisk, '*'. The directives change the behaviour of expect-lite in a user defined manner. These directives include:
  • fail script - run when the script fails *~include_fail_script
  • user defined prompt - for those non-unix/linux remote hosts */regex/
  • stopping the script - *TERM & * FAIL
  • setting the default shell - **SHELL
  • multiple sessions - for hard to script environments *FORK
  • managing logging - *INFO, *NOINFO, *WARN, *NOWARN, *DEBUG, *NODEBUG, *NOCOLOR
  • debugging aid - *INTERACT (see debugging tips)
  • print all expect-lite variables - *SHOW VARS
  • control end of line character(s) - *EOLS LF, *EOLS CRLF
  • control dynamic variable capture method to require a prompt or not *DVPROMPT, *NODVPROMPT
  • allow script to run to completion regardless of failure status *NOFAIL
  • ignore breakpoints (used for regression) *NOINTERACT
Directives can also be placed on the command line at run time. This is useful for controlling logging, such as *NOINFO or *NOCOLOUR. Additionally, *NOFAIL can be useful when debugging a script, since the script will not stop prematurely.

Fail Script    *~include_fail_script

The purpose of the fail script is to clean up after the failed script, reseting times, deleting temporary files and such. It is a special include script which is declared near the top of the script:
*~clean_up.inc
Should the script fail, the fail script will be "sourced". 

The scope of the fail script is always local to the running script. This allows normal include scripts to declare a separate fail script from the main script. Thus enabling a different clean up mechanism for the include script.

The fail script mechanism should no longer be needed  for script development. The technique was useful before debugging tools like instant-interact and the powerful IDE built into version 4.0.1.

User Defined Prompt   */prompt/

With each '>' command an implied "wait for prompt" occurs (see implementation details). The predefined prompts are based on standard shell prompts (>%$#). However, it is quite possible that expect-lite will not be interacting with a shell, but another application (such as gdb) or device which does not issue a shell-like prompt.

As of version 3.1.5, the user may define a custom prompt. User defined prompts are defined from the newline (beginning of line) to the end of the prompt. For example:
*/my_prompt /
The succeeding  '>' lines will interpret a prompt as the standard shell prompts and 'my_prompt '. The user defined prompt definition between the slashes is interpreted as regex, and therefore regex rules such as escaping applies. For example, to set a user defined prompt for gdb the following would be used:
*/\(gdb\) /
Only one (1) user defined prompt can  be active at a time, however multiple prompts can be represented using the regex OR '|', for example, creating a gdb prompt and my_prompt:
*/\(gdb\) |my_prompt /
Another application of the user defined prompt is to make telnet logins easier:
*/login: |.+ssword: /
>telnet remote-host-018
<Connected to
>$username
>$password
>
Clearing a user defined prompt. There may be times where a previously user defined prompt is causing problems by output which falsely triggers the user defined prompt. It is possible to clear the user defined prompt by:
*//
Debug the user defined prompt by using the -vv or --verbose cli parameter. In the example below, the user defined prompt has been incorrectly */\(gbb\) / while running gdb (a linux debugger). The debug output shows the user defined prompt and the string expect-lite is searching for the prompt.
(gdb) Warning:Prompt Timed Out!
DEBUG: User Defined Prompt:<<\(gbb\) >>
Not Found in<<
(gdb)
>>

The scope of the user defined prompt is global, that is, it extends to the main script as well as all include files referenced.

Setting the default shell with **SHELL=<shell>

expect-lite has been tested with bash, however it should operate with other shells. Based on the user's preference, a shell can be selected with the **SHELL directive. Because the shell is invoked immediately after login to the remote host, this is a configuration directive with is read before the script executes. Therefore, it can be placed anywhere in the script, and is not limited to the first line.

Stopping the script early with *TERM, *PASS & *FAIL

There are times when it is desirable to terminate the execution of a script early (earlier than the end of the script). For example, while debugging a script, it is convenient to stop execution mid way in the script while focusing a troublesome section. Or to perform a test, such as this is Monday, and terminate the script.

*PASS will stop execution and return 0, or Pass. This is useful for debugging. *TERM <N> (as of version 4.8.0) will terminate the script early, and return N (0..255). User defined return can be useful for signalling to the calling program that the script returned abnormally (e.g. lab setup was not correct).

*FAIL will stop execution and return 1, or Fail. If a fail script has been declared earlier, it will be executed before terminating execution. If *NOFAIL has been used prior to *FAIL, then the script will run to completion, but will declare fail at that time, and return 1.

No stopping the script with *NOFAIL

There are times, especially when debugging a script, where it is desirable to not stop the script due to a failure, but instead allow the script to run until completion. Setting the directive *NOFAIL will prevent the script from terminating prematurely. Once invoked, there is no disabling *NOFAIL.

Upon completion, if there are failures during the script, expect-lite will return a 1, and indicate that there was a failure.

More no stopping with *NOINTERACT

When running regression scripts, it is not desirable to have each include script stop at breakpoints (*INTERACT). Rather than force the re-editing of each include file, it is possible to ignore the breakpoints with *NOINTERACT.

This directive is sticky, and can not be turned off once it is invoked.

Debugging by examining expect-lite variables with *SHOW VARS

It can be useful when debugging a script to see what the value of variables are at some point in the script. Additionally, overriding constants will also be printed. This can be used with *INTERACT, or pausing the script, however using the debugger command <esc>v is less typing.

Multiple Sessions *FORK

Until version release 3.5.0, expect-lite has been intentionally limited to a single session keeping it simple. However, there are certain environments where it might be advantageous for a single script to control/monitor both a client (e.g wget) and a server (httpd log). Without multiple session support this type of environment would be difficult to automate with expect-lite.

What is a new session? A new session starts with a new shell on the remote host. All commands '>' and received text '<' are constrained to that session. It is possible to use multiple sessions on the localhost using r=none, however r=none support is limited due to timing issues (the timing issues have been removed in 4.7). If multiple sessions on the localhost (the same host where expect-lite is running), it is better to ssh to the localhost by specifying r=localhost. This loop back method ensures that the timing between commands and received text stay in sync.

With the version of 3.5.0, expect-lite supports nearly limitless multiple sessions. However, it is a good guideline to use as few sessions as needed. For backward compatibility, the first session is the "default" session. Additional sessions are assigned a name at invocation with the *FORK <session_name> directive:
*FORK Server
INFO: FORK session is: Server
expect-lite will print an "INFO" line stating the current session name
Session names may not contain spaces, and must be unique for each session. To switch back to a previously started session, re-use the session name. The session name "default" (without quotes) is reserved for the first session.
*FORK default
INFO: FORK session is: default
Print the current session name by using the *FORK with no session name:
*FORK
INFO: FORK session is: Server
*FORK and variables. It is possible to assign a session name to a variable, and then create a new session with that name. For example:
$my_session=Client
*FORK $my_session
INFO: FORK session is: Client
All sessions will be automatically closed when the script terminates.

Debugging in a multi-session environment: When in INTERACT, either via the *INTERACT command in the script or via instant-interact (by pressing ^\), the *FORK command is still available, permitting the switching of sessions while in INTERACT mode. For example:
expect-lite directive: *INTERACT 

Press '+++' to end interact session & return to script

% *FORK CONSOLE

INFO: FORK session is: CONSOLE, Active sessions are: default CONSOLE
#

With the addition of multiple sessions, decoding the output will become more difficult. Remember, although multiple sessions are supported, expect-lite is still single threaded, meaning commands will always be executed the order as they appear in the script.

Managing Logging *LOG, *LOGAPPEND, *NOLOG, *INFO, *NOINFO, *EXP_INFO, *NOEXP_INFO, *WARN, *NOWARN, *DEBUG, *NODEBUG, *NOCOLOR

Expect-lite provides additional information (logging) which makes it easier to debug failures. As of version 4.4.0 expect-lite supports native saving output to a file (logging). Using *LOG, *LOGAPPEND, and *NOLOG, logging can be controlled within a script. On the CLI,  the *LOG commands will override those inside the script.

The following logging commands are supported:
  • *LOG: with no file name, will automatically create a log file with the script name + ".log" in the same directory as the script. If a path/file name is supplied, expect-lite will write standard out and additional info (INFO, WARN, DEBUG) to the path/file name file. Because parameters can be entered in any order on the CLI, the path/file_name must end in ".log" to prevent ambiguity.
  • *NOLOG: closes any open logging files, and disables any additional output from be saved to a file. When used on the CLI, all logging to a file will be disabled.
  • *LOGAPPEND: Will cause output to be appended to an existing file, or create a new file if existing file doesn't exist. Output will be appended to path/file_name, when using *LOGAPPEND <path/file_name>.
The output is coloured based on three levels of information desired: info, warnings, and debug. By default, info and warnings are turned on and exp_info is turned off.  However, using the log information directives: *INFO, *NOINFO, *WARN, *NOWARN, *DEBUG, *NODEBUG, these messages can be managed. Additionally, *NOCOLOR, will turn off color output.

Example of the different information levels are:
  • INFO: Entering FORK sessions, Constant substitution, Conditional Statement Results, Jumping to labels, Including files, Dynamic Variable Assignments, User Defined Prompt assignments, Sleep progress dots and ##Overal Result.
  • EXP_INFO: prints "expected" text (lines with < or <<). Also enabled with -V flag on the command line.
  • WARN: Instant-Interact feature disabled, Wait-for-prompt timeouts, Using localhost (r=none), Unable to parse Conditional, Dynamic Variable Capture timeouts, Unable to parse math variable statement, Unable to parse FORK session name, Unable to parse Conditional Statement
  • DEBUG: Additional debugging information for: User Defined Prompt, FORK session instantiation/switching, '<' searching, Dynamic Variable capture
In addition to the Logging Directives, invoking expect-lite with a -v or -vv will force INFO & WARN on or WARN & DEBUG on, respectively. It should be possible to direct the amount of logging desired by using a combination of the command line arguments (-v or -vv) and the Logging Directives.

In addition to the three logging levels above, there is another more serious level of ERROR, which can not be suppressed, and will terminate expect-lite with a non-zero return code. Examples of ERRORs are: unable to connect or lost connection with remote host, include file not found, or infinite loop detection.

It may be tempting to turn off logging once a script is debugged. However, if the script is being run in an automated environment (like crontab), it is recommended that INFO and WARN be turned on, as the additional information will make debugging failures much easier.

Controlling End of Line Sequence with *EOLS

There are some remote devices which require carriage return + line feed at the end of each line sent. In past versions of expect-lite, only line feed was appended to each '>' or '>>' line. This limitation made it difficult to automate tasks with this class of remote devices.

With version 3.7.0, the end of line sequence can be controlled with the following expect-lite directives: *EOLS LF & *EOLS CRLF. The default is *EOLS LF.

These directives can be placed anywhere in the script and the setting remains in effect until another *EOLS directive is encountered. For example, enabling a script to control a linux machine (using *EOLS LF), and later in the script controlling a remote power-switch device (using *EOLS CRLF).

Controlling Dynamic Variable capture method with *DVPROMPT/*NODVPROMPT

The default Dynamic Variable Capture method requires a trailing prompt to signal when to look for the text of interest. This is based on the assumption that a command would be issued, and somewhere in the output is the text of interest. For example:
$ date
Sat Jul 19 18:19:58 EDT 2014
$
A Dynamic Variable capture script might be looking for the month.
>date
+$month=\n\w+ (\w+)
With the default capture method, *DVPROMPT, expect-lite will wait for the trailing prompt (after the date output) before searching the output for $month. If the text of interest (e.g. month) is not found, then a warning will be issued, and the script will move on quickly.

There are times when there is no prompt (e.g. it is just the running output of /var/log/messages), and the default dynamic capture method will not find the desired text.

*NODVPROMPT turns off the requirement of the trailing prompt for these instances. The Dynamic Variable capture method will search the output until the timeout value (set with @, e.g. @5) expires. Caution: this can slow down the script considerably, if the expect timeout is set to a large value, say 600 (10 minutes), and the text of interest doesn't appear. The script will just sit and wait for 600 seconds, before issuing a warming and finally moving on.

Like all directives, *DVPROMPT/*NODVPROMPT can be used at anytime in the script,  on the command line when invoking the script, or even in the Debugger.

Examples of Expect-lite

Below are some examples to better illustrate what can be accomplished with expect-lite:

Setting an IP address
Setting the IP address on a secondary interface of the remote host (be sure to escape the dot's in the IP address)
$local_eth=eth1
$ip_addr=192\.168\.10\.2
# change timeout value to 10 sec
@10
>ifconfig $local_eth $ip_addr
#check to ensure that we set the right IP address
>ifconfig
<$ip_addr
Telnet to another host
Starting a telnet to another host
@2
>telnet remote-host-018
<Connected to
<login
>>root
<assword
>>secret_password
# issue a command once logged in
>pwd
>^]
>quit

ssh to another host
Starting a ssh session to another host
>ssh root@host-021
<assword:
>>secret_password
# issue a command once logged in
>ls
>exit

Use Screen to keep session alive
Use of 'screen' command to keep a command alive after test completes
; ==== Use Screen command for to keep application running ====
>screen
> $pmm
# Check that HW is alive
>ping
<Successfully pinged the PM H/W
; ==== Keep pmm application active, and detach from this screen
>>^A
>>^D
<detached
>
Assigning a dynamic variable
Assigning a dynamic variable $host using regex to capture the hostname from 'uname -a' command
>uname -a
+$host=Linux ([a-z\-0-9]+)
Assigned Var:host=remote-host-008
>
Assigning multiple dynamic variables using regex to capture environment variables $HOME and $SHELL, while expecting $TERM=xterm
>env | sort
+$home=\nHOME=([a-z/]+)
+$shell=\nSHELL=([a-z/]+)
<TERM=xterm
Assigned Var:home=/home/joe
Assigned Var:shell=/bin/bash
>
Assigning a dynamic variable to a list of items (or multiple lines). The value of the dynamic variable will be a space delimited list (replacing end of lines with spaces):
#prompt is $ and space after $
$prompt=\$
>env | sort
+$env_list=((.|\n|\r)*)$prompt

Allowing multiple responses with RegEx
Using regex in expect-lite to expect users OR wheel
>id
# allow either groups users or wheel
<(users|wheel)
Use of bash-here-doc to create temp file
Create a temporary file with unique name including script name and date. The predefined variable $arg0 is contains the expect-lite script name.
# date stamp of script run time
>echo `date +_%Y_%m_%d_%H-%M-%S`
+$DATE=\n(_[0-9_-]+)

$temp_file=$arg0$DATE
; === Create a temp file using bash "here doc" method
>cat > $temp_file <<'+++'
>line 1
>line 2
>+++
>
Force arguments with help
Creating script help. First, require a parameter be passed via the CLI, such as sw_build:
./upgrade_script.elt sw_build=0015
Inside the script itself, default $sw_build to 0, the CLI constant will override.
$sw_build=0000
; === Check passed parameters
>
*NOINFO
?if $sw_build != 0000 ? %CONTINUE
;; ##############################
;; $arg0 Requires parameters: sw_build, IP
;; e.g. $arg0 sw_build=0015 IP=10.3.5.4
;; ##############################
# quit the script without continuing
*FAIL
%CONTINUE
*INFO

Now when just the script is executed, help will be displayed, and the script will terminate.
./upgrade_script.elt

=== Check passed parameters

expect-lite directive: *NOINFO
##############################
upgrade_script.elt Requires parameters: sw_build, IP
e.g. upgrade_script.elt sw_build=0015 IP=10.3.5.4
##############################

expect-lite directive: *FAIL

User Defined Help
As of version 4.3.0, User defined help can make scripts more user-friendly. Text between triple-semi-colons ';;;' will be displayed when the -h or --help option is used on the command line
;;;
Test: Test bash here doc

Assumptions: bash
Platforms: anywhere bash runs
;;;
Then running the script, the help text as well as variables  in the script which can be overridden on the command line (as constants).
$ ./test_bash_here_doc.txt -h
Help for: ./test_bash_here_doc.txt

Test: Test bash here doc

Assumptions: bash
Platforms: anywhere bash runs

Displaying assigned variables, which can be overridden with
a Constant on command line e.g var=value
tempfile=junk

Sample Regression Script
It is possible to use expect-lite to run simple regressions by including multiple include files. It is also possible to pass var=value parameters to the include files changing the nature of the test. In the following example, pseudo-arrays and a while loop is used to create a quick regression.
# ignore breakpoints in include files
*NOINTERACT
# do not stop on failures
*NOFAIL

######### List of tests to run in regression
$blank=
$tc0=ipv6_telnet.elt
$tc1=ipv6_ndp.elt
$tc2=ipv6_self_syslog.elt mode=ipv4
$tc3=ipv6_self_syslog.elt mode=ipv6
$tc4=

$start=0
$count=$start

[ $tc$count != $blank
; ============== starting $tc$count
~$tc$count
; ============== end $tc$count

# increment variable
+$count
]

Run the regression script and log it using the unix/linux 'tee' command. By defining 'start' as a constant on the command line, it is possible to start the regression with the second test.
./regression.elt start=1 *LOG

Test Failure?!?

When will expect-lite fail a test? It will only fail a test when an expected result (after issuing a command) does not appear in the specified timeout period. For example in the following command file:
$ip_addr=192.168.10.99
>/sbin/ifconfig eth0 $ip_addr
>/sbin/ifconfig eth0
< $ip_addr
The first line will be sent (blindly), since there is no expected return. On line 3, the ifconfig command is sent again, and the script is looking for a desired result of  192.168.10.99 (using the var $ip_addr). If for some reason the ip address is different than ip address was returned, the script would fail.

Not Expect
A test can also fail should text appear that is unexpected (such as an error). This does not clear the expect input buffer, and should be used before a valid expect. How long should the script wait for the un-expected? In order to reduce delays in script running time, the Not Expect feature only waits for 100ms. This is usually enough time to detect quick error responses such as "file not found". 

For example: Fail device doesn't exist
>ls -l /dev/linux
-<No such file
In another example: We don't like Mondays
; === fail if today is 'Monday'
>date +%A
-<Monday
Since the expect input buffer is not consumed, valid expects can still be performed. in the following example, ttyS2 will still be found.
>ls -1 /dev/ttyS*
-<ttyX
<ttyS2
It is recommended that Not Expect lines be used before valid expect lines. There is an exception to this rule, when looking at output which may be slow (longer than 100ms). For example there may be a linux machine with several interfaces (eth0 thru eth4). You may want to ensure that eth4 is not down.
>ip link show
<eth4
-<state DOWN
In the example above, the 100ms counter will start after finding 'eth4'

Fuzzy Expect
Fuzzy expect is expecting an approximate number (either decimal or hex). There are situations where an approximate value, or a range of numbers is acceptable. An example, one might be looking at cpu load (it is OK if it is between 0 and 2, but the test should fail if it over 2).

Fuzzy expect operates in two parts. The first part is to set a range or tolerance (of fuzziness). This defines a range, for example entering 7, will match + or - 7. Set the range by:
; set the fuzzy expect range to 7
~=7
This value will be in effect until it is changed.

The second part is to use fuzzy expect. In the following example, it is important to the script to run in the first 2 weeks of the month. Using the date command to print out the date/day of the month.
; Check the date/day of the month
>date "+the day is: %d"
~<the day is: (7)
The value in parens will be checked against the range earlier defined (plus or minus). In the above example, if the date command returns the string "the day is : 13", it will be pass, since 13 < 7+7

Of course this could have been done in a more programmatic way, of capturing the value returned by date (using Dynamic Variables), then using a conditional (if statement) to do the comparison. But fuzzy expect is much simpler to use.

Additional important details are: it consumes the expect buffer, hexidecimal numbers must start with 0x (e.g. 0xbeef),  parens in the string which are not matching the fuzzy value must be escaped with a backslash (e.g. \(15\) ), regex evaluation is active (like '<'), and the default range is 10.

How can it help?

Expect-lite can be used to quickly create automated tests. It can collapse complicated tests into a single command line. It can also be used in nightly regression testing, performing simple functional tests providing confidence that core functionality has not been broken by the previous day's submissions.

Limitations


Expect-lite is limited to what a person could do in one (1) terminal window (or xterm). It cannot start a program in one window and run a different program in a separate window. However it does support multiple sessions via *FORK which should meet the needs of most.  

Standard programming constructs such as looping and branching are supported as of version 3.0.0. Real Expect can do these things and more. That said, looping is limited to a repeat loop and while loop and branching can be accomplished with Labels.

Expect-lite has been purposely limited to keep scripts simple, easy to use and maintain. Although more complex scripts can be created, basic expect-lite scripts can be created by simply cutting and pasting text from a terminal window into a script and adding '>' '<' characters.

Implementation Details

Variables

Variable names must be restricted to the following set of characters [A-Za-z0-9_]. Variable values may include spaces, and quoting is not required nor permitted. For example:
$bell=For whom the bell tolls
>echo $bell
For whom the bell tolls

Implied "wait for prompt"

Using the '>' character implies waiting for the prompt before sending the command. The definition of a shell prompt a: $, #, % or >. It is often useful to follow a '<' with a '>' for force the script to wait for the prompt:
> $some_long_command
<critical value
# wait for prompt
>
Prompt detection can be problematic, and by no means is expect-lite's method perfect. Expect-lite looks for a prompt character ($, #, % or >) followed by a space at the end of line. This should work for most standard prompts, however coloured or fancy prompts may not be detected. As of version 3.1.5, user defined prompt is supported.

It may be useful to create files on the fly which will be used in the test. Since expect-lite does not read/write files directly, this function can be done with the linux command 'cat' and the bash here doc method. Creating a regex file for example:
; ==== Create Test File on the fly
> cat > $regex_test_file << '+++'
>T00001/a(?$XL.*)/tag=0x08000001
>T00002/b(?$XR.*)/tag=0x08000002
>T00003/c(?$XO[0-7]*)/tag=0x08000003
>T00004/d(?$XH[0-9A-F]*)/tag=0x08000004
>T00005/e(?$XD[0-9]*)/tag=0x08000005
>+++
>
In the above example the script initiates a cat to a file (referenced by $regex_test_file ). Normally the '>' would be used to send lines to the remote host. However, there is an inherent "wait for prompt", which will politely wait for the next line. The last '>' is there to wait for the prompt to return after the cat command.

Predefined variables

Expect-lite has minimized the use of predefined variables to allow the user the widest latitude in selecting variable names. However a few predefined variables are useful and listed below:
Predefined variable
Usage
$arg0 expect-lite script name. Set to the c= or cmd_file= value. Useful for creating temporary files unique to the test script.
$timeout_multiplier Typically used as a constant (passed in from the command line) which the integer value is multiplied by each of the timeouts in the script. i.e. @10 x $timeout_multiplier = real timeout. The value default is 1.

Timeouts

Setting the expect-lite timeout to 0 isn't actually zero as this will cause the keyboard input buffer on the remote machine to overflow (and lose characters). Expect-lite adds a 50ms delay between lines when the @0 is used to prevent the overflow.

Expect Input Buffer

Expect by default places strings that are sent back (from the remote host) into an expect buffer ($expect_out(buffer)). Expect-lite leverages this built-in function, and uses it to perform multiple searches through the buffer when using the '<' , '+<' or '-<' actions. The actions '>' and '>>' will clear expect buffer to prevent spurious matches.

The default size of the expect_out(buffer) is 10000 bytes. This has been selected as a compromise between performance (larger buffer slows down searches), and large enough for most users. If searches (using '<' , '+<' or '-<') or dynamic variable captures are being truncated, it may be required to increase the size of the expect_out(buffer) size. Caution: This should only be a last resort (as it will impact performance). Place the following near the top of the script.
; === set expect buffer big
!match_max -d 20000

Timing is everything: delay_wait_for_host

When communicating with a remote host, enough time must be provided for the remote host to respond. There is an internal global value of $delay_wait_for_host (in mili-seconds) which can be tuned to your environment using the environment variable EL_DELAY_WAIT_FOR_HOST (see sample bashrc). This value is also used by the Expect Not feature to determine how long Expect-Not should wait for the unexpected.

Unsynchronized messages disclaimer

Effort has been put into making the output as synchronous as possible. However because what is seen on stdout or screen is data returned/echoed from the remote host and printable comments are directly printed to stdout, it is possible to have a printable comment appear in the stdout before the action it references is returned from the remote host. This lack of synchronization can sometimes be minimized by adding an additional '>' after the printable comment as shown in the following:
; === starting long command
>
>start_long_command

Instant-Interact: More than just a pretty prompt

When the user presses ^\ (control+backslash) during the running of a script, expect-lite will drop into interactive mode aka the integrated debugger. In version 4.0.1, the debugger has been greatly enhanced and performs three primary functions: 1) connecting the user to the remote host or device under test, 2) monitoring special commands prefaced with the escape key for stepping, and other functions, and 3) the debugger will allow expect-lite script lines to be executed by either typing directly or pasting them into the debugger. The TCL package TclX is no longer required.


Control will be returned to the script when '+++' is typed. In addition to returning to the script, the current command in the script will be "Passed", preventing it from failing the script. This is useful when debugging a script, the user notices that the response is not as expected, and presses ^\ to move to interact mode. Returning from interact mode, the line which would have failed the script (because of the response not being as expected) will be "Passed" and the script will continue. This Passing action is line oriented, and other '<' lines in the script can still fail the script.

 In the IDE (debugger), expect-lite script commands can be typed, even assigning variables or displaying them inside the paused script, for example:
$MYVAR=today

*SHOW VARS
Var:0 Value:0
Var:MYVAR Value:today
Var:TEST Value:/proc/cpuinfo

>>pwd
pwd
/home/user
Additionally, expect-lite lines (code fragments) can be copied from your favourite editor and pasted directly into the debugger. All variables will be automatically dereferenced (assuming they were set in the first place). This is very useful for testing Dynamic Variable capture lines, which can be tricky to define.

When lines are pasted into the debugger, or the Step or sKip function are used, the debugger clumps lines which begin with '>' and any following lines with start with '<' or '+$' This enables expected results and dynamic variables to be interpretted correctly. For example:
# clump 1
>ls /etc/
<group
<passwd

#clump 2
>date "+TIME: %H:%M:%S"
+$time=\nTIME: ([0-9:]+)
If the above lines are copied and pasted into the debugger, in fact they will be executed as 2 clumps. Usually this is of no concern, however when stepping/skipping through an expect-lite script, one will notice that clumps are executed rather than indivdual lines.

*FORK: what happens behind the scenes

A new *FORK session actually creates an additional login(s) to the remote host from the localhost using the same connect method as the first (or default) session. It then sets a sane prompt, and waits for the first script command in that session. This provides multiple sessions on the remote host, which then could be used to drive separate programs (see test_fork_web_client_server.txt for example). Graphically it would appear as:
Localhost Remote Host
+----New_Session1--->Client App
+----New_Session2--->Server App

What's missing

In order to maintain an easy-to-learn, simple-to-use expect like language, looping has been intentionally kept simple and intuitive. Most expect-lite scripts will be to be top down, single pass scripts, making the scripts easy to create and read. These limitations preserve the KISS principle.

If more complexity is required, consider using embedded expect inside expect-lite, or writing a native expect script.

Why Expect-lite

Expect-lite was written so you don't have to learn expect. It provides a quick way to write simple expect scripts using just the > and < characters.



15 August 2015
http://expect-lite.sourceforge.net/

Subscribe to expect-list-users Discussion List

this document for version 4.9.0 and above