Expect-Lite

Making automation scripting Simple

Introduction

What is expect-lite? expect-lite is a quick and easy command line automation tool.

Written in expect, it is designed to directly map an interactive terminal session into an automation script. expect-lite scripts use special character(s) at the beginning of each line to indicate the action. Basic expect-lite scripts can be created by simply cutting and pasting text from a terminal window into a script, and adding '>' '<' characters. No knowledge of expect is required!

Expect-lite is targeted at the verification testing environment, and will produce a Pass/Fail result at the end of the script. However, its use is not limited to this environment.

Features

Expect-lite has many features which make expect-style script writing easy. Such as:

Automatic login to remote host (supporting telnet, ssh with password, and ssh with keys)
Simplicity using one character for send and expect '>' '<'
This creates many lines of expect with minimal "code"
Regular expression (regex) evaluation in expected results
Static and Dynamic Variables with math functions and pseudo arrays
Constants, passed on the command line which override variables in the script
Leverages Bash shell
Include mechanism to read other script files (sourcing)
Embedded Expect - using native expect embedded in an expect-lite script
Not Expect - a feature to look for unexpected text/strings
Conditional Branching a if/then/else with a c-like syntax
Simple looping using conditional branching, a repeat loop can be constructed with infinite loop protection
Scripts can be executable with the help of a small embedded bash script
User defined prompt allows interaction with a wider range of hosts (routers, switches, etc)
Multiple Sessions enabling interaction with a client and a server in one script - new in version 3.5.0
Instant-Interact empowers on the spot debugging new in version 3.5.0

BSD-Style License

Copyright (C) Freescale Semiconductor, Inc. 2005-2007

See the COPYING for information on usage and redistribution
of this file, and for a DISCLAIMER OF ALL WARRANTIES.

Download

Expect-lite downloads can be found on the Expect-lite Project Page: http://sourceforge.net/projects/expect-lite
The package includes expect-lite, as well as several examples.

Installation

Place expect-lite in your path. Test by typing at the prompt:

expect-lite

If you see the help, then it is ready to go, installation is complete!

See the Installation & Quick Start Guide for more help with tuning expect-lite to your environment.

Automatic Login

Based on the command line arguments passed, expect-lite will do the following:

Automatically log into the remote host via telnet or ssh (if ssh keys have been previously setup)
Change directory to the passed directory (if included)

In the following example of expect-lite:

./expect-lite remote_host=remote-host-018 cmd_file=basic_ping_test.elt user_dir=/etc

Or the shorter version

./expect-lite r=remote-host-018 c=basic_ping_test.elt d=/etc

expect-lite will log into remote-host-018, change directory to /etc/ and begin to send commands from basic_ping_test.elt to the remote-host. The parameters of remote_host and cmd_file are required, as expect-lite must know what host to log into, and what script (aka command file) to run.

Special Character Sequences

Expect-lite interprets the following special characters at the beginning of each line in the script as:

Char	Action
>	send string to the remote host
>>	send string to remote host, without waiting for prompt (see implementation details)
<	string/regex MUST be received from the remote host in the alloted timeout or the script will FAIL!
-<	if string/regex IS received from the remote host the script will FAIL! (see Not Expect)
#	used to indicate comment lines, and have no effect
;	are also used to indicate comment lines, but are printed to stdout (for logging)
;;	similar to above, but no extra newlines are printed (useful for printing script help)
@num	changes the expect timeout to num of seconds
$var=	static variable assignment at script invocation (see variables for details)
+$var=	dynamic variable assignment (see variables)
+$var	increment value of $var by 1 decimal (see repeat loop)
-$var	decrement value of $var by 1 decimal
=$var	math functions, perform bitwise and arithmetic operations: << >> & \| ^ * / % + - (see math functions)
!	indicates an embedded expect line (see embedded expect)
?	c-style if/then/else in the format ?cond?action::else_action (see conditionals)
%	label - used for jumping to labels (see conditionals)
~filename	includes a expect-lite automation file, useful for creation of common variable files, or 'subprograms/subroutines'

In addition to the above special characters, blank lines or lines starting with any non-special character are allowed to make the script file more readable.

Sometimes control characters must be sent. Currently all control characters are supported (from ^@ to ^\):

^C or break, when it is desired to stop a running program e.g. >^C
^] or escape from telnet, e.g. >^] This will bring the script to the telnet prompt, then use: >quit
^<any char> will send a control character to the remote-host, such as ^D to logout

Control characters must be the first two characters on the line (e.g.[^][A-Z]). The sequence of ^C anywhere else in the line will not be interpreted as a control-C, but rather the two characters '^' 'C'.

Lastly, any line beginning with an asterick '*' will be interpreted as an expect-lite directive. Directives change the behaviour of expect-lite.

	Expect-lite Directives
*~filename	Include a fail script, which expect-lite runs only if the main script fails (see fail script)
*/prompt/	Set a user defined prompt (see user defined prompt)
*TERM	Immediately terminates script, but returns 0 (pass) (see stopping execution early)
*FAIL	Immediately fails script, and returns 1 (fail) (see stopping execution early)
**SHELL=<shell>	Configuration directive which sets the shell imediately after automatic login to remote host (see setting the shell)
*INTERACT	Places user in interact mode, which pauses the script and turns control of the keyboard over to the user (see Tips & Techniques: Interact)
*FORK <session>	Multiple session support. Directs expect-lite to open a new session and spawns a newshell (see multiple sessions)
*SHOW VARS	Debug information, displays all expect-lite variables. Can be used in interact mode.
	Logging Directives
INFO NOINFO	Enable/disable informational messages (see managing logging)
WARN NOWARN	Enable/disable warning messages
DEBUG NODEBUG	Enable/disable debug messages

General Tips for writing scripts

Here are some simple tips when script writing:

Use reasonable timeouts, if 30 seconds is needed to get a response, set the timeout at 45 or 60 seconds, not 600.

There is no cost to changing the timeout, timeout values can also be variables

Beware of expect-lite using regex, when creating lines like: <0.005 secs (5 micro secs)

The parentheses is used by the regex engine, instead escape these characters: <0.005 secs $5 micro secs$ (see Use of Regex in expect-lite)

Use the expect character '<' often. Check for valid results when possible. A script which expects nothing will never fail! (see Test Failure)
Use printable comments ';' often. Think of it as writing a note to oneself, it will make reading log files much easier.

Debugging Expect-lite Scripts

Expect-lite was written to permit quick and easy automation of repetitive tasks. However, sometimes a little additional debugging is required.

Instant-Interact (as of version 3.5.0), provides instant debugging assistance. Pressing ^\ (control+backslash) will force expect-lite to drop into interact mode, a mode which turns control of the keyboard over to the user, so that one may type directly to the process on the remote host. To return to the script, type '+++' (three pluses). During interactive mode, timeouts do not apply, and the script will wait indefinitely for the user to exit this debugging mode. Switching between multiple sessions (see *FORK) is possible via typing the *FORK command in interact mode (see *FORK debugging).

Additionally, the include scripts, debug.inc and show_vars.inc, have been added to the release (as of version 3.1.3) to assist in debugging scripts.

See the Tips and Techniques Guide for additional expect-lite script debugging information.

Use of Regex in expect-lite

Support of regular expressions in expect-lite are limited by:

Support of regex in standard expect (e.g. including anchors, char-classes and repeats)
Support of regex meta characters (e.g. \t \n are supported, \d is not)
Expect-lite only evaluates lines using regex which begin with '<' '-<' and '+' (dynamic variables)

As an example, a range of numbers is valid for an IP address. The following would permit the last octet of the IP address to be 2 digits, but not 3.

> /sbin/ifconfig eth0
<inet addr:10\.29\.200\.[0-9][0-9][ ]

The periods are preceded by a backslash to indicate to regex that a period must be returned rather than the regex dot '.' which indicates any character.

Because regex is always enabled for expected results, some 'escaping' of characters must be done when using '<'. The following characters must be escaped with a backslash to convey their literal meaning:

Example	Escaped	Character
(abc)	$abc$	parenthesis
[abc]	\[abc\]	square brackets
\	\\	backslash
.	\.	period
$	\$	dollar sign

Using Variables

Expect-lite supports two types of variables:

Static, which are bound at invocation
Dynamic, which are bound during execution of the script

Lines starting with a '$' indicate variables which are assigned and set at script invocation, and therefore are static. In expect-lite variable names will always be preceded with a '$'.

Sometimes it is desirable to bind a variable during the execution of the script. Dynamic variables fill this need by utilizing Expect's built-in regex capture mechanism. Only the portion of the match inside the parenthesis will be bound to the variable. The format of the line is as follows:

+$somevar=text that is not put into the var (text which is put into the var)

Dynamic Variables are always bound to Expect output, meaning text which is returned from the remote host. Therefore something must be sent to the remote host, before text can be returned. A typical usage would be:

> command
+$myvar=command output (capture value) more command output

If the value of the var is successfully captured then expect-lite will print:

Assigned Var:somevar=sometext which is put into the var

If, however the dynamic var is not successfully captured, expect-lite will print:

Assigned Var:somevar=__NO_STRING_CAPTURED__

Avoid using an overly general capture as it will tend to capture too much, or the wrong info. e.g:

> command
+$myvar=\n(.*)

Instead use a specific capture when possible. When capturing the current directory, the current directory will always start with '/', and the path will be made up of a known character set:

> echo $PWD
+$mypwd=\n(/[a-zA-Z0-9/\-_]+)

Expect-lite will print to stdout the value of the assigned variable, assisting the script writer in understanding the value that is bound to the variable during runtime. The value of "__NO_STRING_CAPTURED__" indicates that the regex pattern used did not match the return data. It is best to start using this feature with short scripts targeted at capturing the desired information. Examples can be found in the example section of this document.

Math Functions with Variables

There are times when writing a script it may be necessary to perform arithmetic or bitwise operations, also known as math functions. Expect-lite inherits expect/tcl math functions and natively supports math/bitwise operations on a variable, using the following slightly odd notation:

$myvar=1
=$myvar + 2 * 10

In the above example, 2 will be multiplied by 10 and added to value of $myvar (1). The result of 21 will be placed in $myvar, overwriting the previous value. Since multiplication has a higher precedence than addition, (2 * 10) will be performed before the addition to $myvar.

Parenthesis may be used to enforce user defined precedence. In the following example, the result will be 23:

$myvar=1
=$myvar + 2 * (10 + 1)

Both bitwise and arithmetic operators are supported with the following precedence (left to right):

bitwise	<< >> & ^ \|	shift left, shift right, bit and, bit exclusive or, or
arithmetic	* / % + -	multiply, divide, modulo, add, subtract

If $myvar does not exist before the math function, it will be initialized to blank and then math functions will be performed. In the following example the result, $myvar, will be 20:

=$myvar + 2 * 10

However, realistically, only the '+' operator works as expected in the above example. Using other operators will more than likely yield a syntax error:

Warning: Expect-lite: Unable to interpret =$answer1 / 2 * (10 + 1) 
 syntax error in expression " / 2 * (10 + 1)": unexpected operator /

Because of the expect/tcl inheritance of math functions, all operations will be performed as integers (or whole numbers) unless a decimal (or real) number is used explicitly. In the following example the result is 2.5:

$five=5
=$myvar + $five / 2.0

However, the following (because of the use of integers) example will have the result of 3:

$five=5
=$myvar + $five / 2

Math functions can be used with static and dynamic variables.

Pseudo Arrays

Expect-lite supports pseudo arrays, which are different from a real array (e.g. item[1], item[2],...). Pseudo arrays are made possible by the fact that variables on the left side of the equals can be dereferenced at assignment time. For example, below, the variable $count will be dereferenced (or resolved) prior to assignment:

$myarray$count=some $value

When placed in a loop with a looping variable of $count, a pseudo array of $myarray1, $myarray2, $myarray3,... will be created. Individual values can be retrieved from the array by using the index (or $count) such as $myarray23.

The pseudo array name (variable + index) must be a valid variable name, in the set of characters (A to Z, a to z, 0 to 9, and underscore) and no spaces.

Constants

Expect-lite Constants are represented as variables which are passed into expect-lite at runtime. Constants will override any script variable already defined inside the script, are immutable, and cannot be changed. Constants can be used to change the behaviour of the script.

For example, one might invoke the following test as:

./expect-lite remote_host=remote-host-018 cmd_file=basic_ping_test.elt user_dir=/etc local_eth=eth2

Inside the script basic_ping_test.txt any reference to $local_eth would be set to eth2, thus allowing the script actions to be changed based on the constant passed at invocation. Constants will override any script variable already defined inside the script.

Shell Variables

Expect-lite allows the use of shell variables, which can be more powerful than the built-in variable mechanism. However shell variables will only be resolved by the shell. For example, assigning current working directory to a shell var:

> PWD=`pwd`
> echo $PWD

However, the following test would fail, since $PWD is a shell variable, not an expect-lite variable:

> PWD=`pwd`
>pwd
< $PWD

Shell variables (or environment variables) must be dereferenced by the shell. Similarly, expect-lite variables must be dereferenced by expect-lite.

It is possible to read a shell variable into an expect-lite variable by using the dynamic variable method:

> echo $PWD
+$mypwd=\n(/[a-zA-Z0-9/\-_]+)

Using Conditionals & Labels

Conditional (if/then/else) statements are natively supported in expect-lite. The conditional uses a c-style syntax with a question mark (?) at the beginning of the line, and double colon to indicate the else statement, using the format ?cond?action::else_action

Although spaces are not required around the conditional characters (? and ::), it is recommended for ease of reading. The comparison operators are: '==', '!=' ,'>=', '<=', '>' and '<'. If the compared values can be evaluated as a numbers, then larger and less than will yield expected results. A simple conditional example:

$age=56
? $age >= 55 ? >echo "freedom at 55!" :: > echo "keep working!"

In the above example if $age is larger than or equal to 55 then the action 'echo "freedom at 55!". If $age is less than 55 then the action 'echo "keep working!"' will be sent.

Each action or else_action is as if it began on a separate line. Since in the above example '>' is used (e.g. >echo) the echo line will be sent to the remote host. Any expect-lite action (see Special Characters) can be placed after the second question mark. For example, an include file may be executed based on a conditional:

$platform=ppc
? $platform == i386 ? ~connect_i386.inc :: ~connect_ppc.inc

In order to make log files more understandable, a message will be printed when a conditional is evaluated, such as:

If: expr { "ppc" == "i386" } |then ~connect_i386.inc|else ~connect_ppc.inc|result0

The message prints: what was evaluated (expr { "ppc" == "i386" }), 'then', and 'else' actions, as well as a result (0=false)

Labels

Conditionals are limited to a single line. Sometimes this is too limiting, as it would be nice to have several commands be executed based on the success of a conditional. To support this, the concept of labels has been introduced. A label is defined as having the first character a '%'. Although the label line itself does nothing, it provides a location to the conditional to Jump To Label. The following is a simple example of using a conditional in conjunction with a label:

$kmodules_inserted=true
? $kmodules_inserted == false ? %SKIP_CHECK_KMODS
>lsmod | grep nfs
<nfsd
<exportfs
<sunrpc
%SKIP_CHECK_KMODS

When a label is the action of a conditional, there is an implied Jump to Label. Lines between the conditional and the label will be skipped. The Jump to Label action is no longer limited to only jumping forward. (see Looping with Conditionals)

Multiple labels with the same name are permitted. For example, a conditional action may be Jump to Label %SKIP. Although multiple labels of %SKIP may occur in the script, in this example, expect-lite will jump to the next line containing %SKIP. Of course it is easier for the script reader if clearer label names are used. Labels may contain spaces, as in this example:

; === Test conditional jump to label
? $jump == true ?%move along, nothing to see
>echo "1"
>echo "2"
>echo "3"
%move along, nothing to see

To assist reading the logs, the action Jump to Label will generate a message to standard out (captured in the log). From the above example, the following would be printed:

Jumping to label:%move along, nothing to see

Looping with Conditionals & Labels

Simple looping is now supported by allowing jump to label backwards. The Repeat Loop is the easiest loop to implement:

; ======== Incrementing Loop ========
$max=5
$count=3
%REPEAT_INC_LOOP
	>echo $count
	# increment variable
	+$count
?if $count <= $max ?%REPEAT_INC_LOOP

Because of jump to label can jump backwards, it is important to assign unique looping labels, such as %REPEAT_INC_LOOP. Unexpected results will occur if non-unique loop label names are used. Non-looping labels, as illustrated in the previous section, are not required to be unique.

Also included in the above example is incrementing an expect-lite variable: +$count
This will add 1 to the value of $count. If $count is not an integer, the value of $count will remain unchanged (can't add 1 to a string).

As part of the looping enhancement, there is infinite loop protection. The maximum amount of looping is defined in expect-lite itself with the variable _el_infinite_loop. This value is decremented with each iteration of all loops for the entire script. Typically this would be in the range of 100 to 1000 to be safe. For example, if a complex expect-lite script had 4 loops each with 100 iterations, the _el_infinite_loop should be set larger than 400.

Expect-lite & Bash

Although not limited to working with bash, bash is invoked upon logging into the remote host, and therefore will be discussed more here.

Since the bash shell is well documented, and supported, and therefore can be leveraged to assist in expect-lite's limitations such as looping and branching (see limitations below). A simple bash loop inside expect-lite can be created for example:

$set=1 2 3 4 5
>for i in $set #expanded by expect-lite
>{
>echo $i #expanded by bash
>}

In the above example, $set is an expect-lite variable, not a bash variable. An expect-lite variable is always declared before its use (e.g. $set= 1 2 3 4 5). If a variable cannot be dereference by expect-lite it is passed to the shell. The loop will execute after the final "}" line is sent to the remote-host. Because of this "execute after" effect, an expect-lite '<' line can not be used within the bash loop.

Another example is the while loop:

$max=5
>i=0
>while [ $i -lt $max ]
>do
>echo $i
>let i+=1
>done

Conditionals via bash are also supported, as in this example:

$max=4
>if [ $max -ne 5 ]; then
>echo "max is not equal to 5"
>else
>echo "we have a winner"
>fi

More complex bash assists can be constructed. For example, one may want to set a looping variable based on the processor type of machine at the time (some hosts may be faster than others). The answer is echoed, and then captured using a dynamic variable. Note that that 'proc' is prepended to the answer, as it enables the capture to be much more specific.

>PROC=`uname -i`
>if [ "$PROC" == "i386" ]; then
>echo proc20
>else
>echo proc100
>fi
+loop_counter=\nproc([0-9]+)

Note that 'proc' is outside the parenthesis, and therefore will not be captured into the expect-lite dynamic variable $loop_counter.

Using Bash to create executable expect-lite scripts
It is possible to make expect-lite scripts executable, with the help of a small embedded bash helper script. Insert the following at the top of the expect-lite script:

#!/usr/bin/env bash
# make this auto-runnable! 
# Little bootstrap bash script to run kick start expect-lite script
# Convert --parms to expect-lite param=value format
PARAMS=`echo $* | /bin/sed -r 's;--([a-z]+) ([0-9a-zA-Z]+);\1=\2;g'`
echo $PARAMS
expect-lite r=none c=$0 $PARAMS
exit $?

The bash scriptlet does the following:

Converts command line arguments from --arg value to arg=value format, making the script more linux-like
Call the expect-lite script with default arguments (in this example that is r=none) and converted arguments
Exit with the same exit code of the expect-lite script (0 success, 1 failure)

Expect-lite ignores the bash scriptlet when invoked, because none of the lines begin with expect-lite special characters.

The final step to making the script executable is to use a chmod command:

chmod 775 my_expect-lite_script.elt

Now the expect-lite script can be executed with --arguments or the standard expect-lite argument format (mixed and matched).

$ ./my_expect-lite_script.elt --r host-15 --user_name craig --max_loop_count 100

This bash scriptlet example can be found in expect-lite.proj/Examples/test_masquerade.txt

Include Files

Include files are a quick way to develop script snippets which can be included into larger scripts or to include a common variable file. When an include file is executed, it is as if the file were just pasted into the script file, and therefore has access to the variable space of the main script, and can modify that variable space as well. In this example, a common variable file is "sourced":

# Source common variable file
~asic_vars.inc

Common functions, such as telnet'ing to the DUT, are a good use of include files:

; === Connect to DUT 
~dut_connect.inc

Include filenames can also be assigned in a variable, such that the file names can be declared at the top of the script but used later within the script. For example:

# Source Var file to be used
$asic_include=asic_vars.inc
...
~$asic_include

Embedded Expect

There are situations when expect-lite cannot provide a solution. Rather than force the user to abandon the simplicity of expect-lite, an embedded expect is supported. This functionality is provided to assist expect-lite rather than replace it. Lines beginning with '!' will be interpreted as native expect (see limitations below).

Embedded Expect requires the script writer to know native expect. It runs with some variable protection, and provides access to expect-lite variables and constants, as well as the expect variable timeout. This permits the script writer to query the timeout, store it away, and then reset the timeout later.

# Preserve old timeout using embedded expect
!set user_namespace(TIMEOUT) $timeout
# set timeout to 15 seconds - max time to wait for login prompts
@15
>telnet $dut_ip
<login
>>root
<assword
>>my_secret
>
# Set the timeout back to old timeout
@$TIMEOUT

Expect-lite variable/values are stored in the tcl array user_namespace() with the variable name as the index name. Constants are stored similarly in the tcl array cli_namespace().

Expect code is collected in islands and executed after an island shore is reached. The island shore is represented by lines that start with any special character, except '#' and blank lines. However, it is recommended that the '>' be used, as this implies a "wait for prompt."

Branching and looping are supported in embedded expect e.g.:

!if { $arch == "ppc" } { 
! puts "\narch is $arch\n"
!} 
>

And looping:

!for {set j 1} {$j<6} {incr j} {
! if {$j == 1} {set type "abc" } 
! if {$j == 2} {set type "def" }
! if {$j == 3} {set type "hij" }
! if {$j == 4} {set type "lmn" }
! if {$j == 5} {set type "qrs" }
! 
! puts "$j>$type"
!}
>

Provides access to expect send & expect commands:

!send "ls\n"
!expect -re "test" { puts "got test"} "rand2" { puts "got rand" } default {puts "got nothing"}

Embedded expect can fail the entire expect-lite script by calling the built-in _el_fail_test function:

!if { $arch == "ppc" } { 
! puts "\narch IS $arch\n" 
! } else { 
! puts "\narch is NOT ppc, but $arch\n" 
! _el_fail_test 
! } 
>

TCL files can be sourced, and functions can be declared and called from within embedded expect.

Limitations of Embedded Expect:

Limited support of expect global variables. To declare a variable at the top level as global use "set ::<var> <value>" The double colon '::' makes var global (this is a TCL standard)
tcl/expect switch command not supported
expect statements must be on one line (as in the example above)

Please remember, this functionality is to provide assistance to expect-lite rather than replace it. (see What's Missing)

Expect-lite Directives '*'

Expect-lite directives are indicated by lines which start with the asterick, '*'. The directives change the behaviour of expect-lite in a user defined manner. These directives include:

fail script - run when the script fails *~include_fail_script
user defined prompt - for those non-unix/linux remote hosts */regex/
stopping the script - *TERM & * FAIL
setting the default shell - **SHELL
multiple sessions - for hard to script environments *FORK
managing logging - *INFO, *NOINFO, *WARN, *NOWARN, *DEBUG, *NODEBUG
debugging aid - *INTERACT (see debugging)

Fail Script *~include_fail_script

The purpose of the fail script is to clean up after the failed script, reseting times, deleting temporary files and such. It is a special include script which is declared near the top of the script:

*~clean_up.inc

Should the script fail, the fail script will be "sourced".

The scope of the fail script is always local to the running script. This allows normal include scripts to declare a separate fail script from the main script. Thus enabling a different clean up mechanism for the include script.

The fail script mechanism can also be used for script development. One the script fails, it is possible to drop expect-lite into interactive mode allowing the script developer to interact the DUT interactively. debug.inc in the following example:

*INTERACT
>
# allow script to continue
!set _el(continue) 1
>
; === Continuing Script
>

The above debug fail script places the developer into interactive mode. Once the developer has completed with interactive mode, '+++' is typed and the script continues. In this particular example, a flag is set _el(continue), which allows the main script to continue as if there were no failure.

User Defined Prompt */prompt/

With each '>' command an implied "wait for prompt" occurs (see implementation details). The predefined prompts are based on standard shell prompts (>%$#). However, it is quite possible that expect-lite will not be interacting with a shell, but another application (such as gdb) or device which does not issue a shell-like prompt.

As of version 3.1.5, the user may define a custom prompt using the following command:

*/my_prompt /

The succeeding '>' lines will interpret a prompt as the standard shell prompts and 'my_prompt '. The user defined prompt definition between the slashes is interpreted as regex, and therefore regex rules such as escaping applies. For example, to set a user defined prompt for gdb the following would be used:

*/\(gdb\) /

Only one (1) user defined prompt can be active at a time, however multiple prompts can be represented using the regex OR '|', for example, creating a gdb prompt and my_prompt:

*/\(gdb\) |my_prompt /

Another application of the user defined prompt is to make telnet logins easier:

*/login: |ssword: /
>telnet remote-host-018
<Connected to
>$username
>$password
>

Clearing a user defined prompt. There may be times where a previously user defined prompt is causing problems by output which falsely triggers the user defined prompt. It is possible to clear the user defined prompt by:

*//

Debug the user defined prompt by using the -vv or --verbose cli parameter. In the example below, the user defined prompt has been incorrectly */$gbb$ / while running gdb (a linux debugger). The debug output shows the user defined prompt and the string expect-lite is searching for the prompt.

(gdb) Warning:Prompt Timed Out!
DEBUG: User Defined Prompt:<<\(gbb\) >>
  Not Found in<<
(gdb) 
>>

The scope of the user defined prompt is global, that is, it extends to the main script as well as all include files referenced.

Setting the default shell with **SHELL=<shell>

expect-lite has been tested with bash, however it should operate with other shells. Based on the user's preference, a shell can be selected with the **SHELL directive. Because the shell is invoked immediately after login to the remote host, this is a configuration directive with is read before the script executes. Therefore, it can be placed anywhere in the script, and is not limited to the first line.

Stopping the script early with TERM & FAIL

There are times when it is desirable to terminate the execution of a script early (earlier than the end of the script). For example, while debugging a script, it is convenient to stop execution mid way in the script while focusing a troublesome section. Or to perform a test, such as this is Monday, and terminate the script.

*TERM will stop execution and return 0, or Pass. This is useful for debugging. *FAIL will stop execution and return 1, or Fail. If a fail script has been declared earlier, it will be executed before terminating execution.

Multiple Sessions *FORK

Until version release 3.5.0, expect-lite has been intentionally limited to a single session keeping it simple. However, there are certain environments where it might be advantageous for a single script to control/monitor both a client (e.g wget) and a server (httpd log). Without multiple session support this type of environment would be difficult to automate with expect-lite.

What is a new session? A new session starts with a new shell on the remote host. All commands '>' and received text '<' are constrained to that session. It is possible to use multiple sessions on the localhost using r=none, however r=none support is limted due to timing issues. If multiple sessions on the localhost (the same host where expect-lite is running), it is better to ssh to the localhost by specifying r=localhost. This loop back method ensures that the timing between commands and received text stay in sync.

With the version of 3.5.0, expect-lite supports nearly limitless multiple sessions. However, it is a good guideline to use as few sessions as needed. For backward compatibility, the first session is the "default" session. Additional sessions are assigned a name at invocation with the *FORK <session_name> directive:

*FORK Server
INFO: FORK session is: Server

expect-lite will print an "INFO" line stating the current session name
Session names may not contain spaces, and must be unique for each session. To switch back to a previously started session, re-use the session name. The session name "default" (without quotes) is reserved for the first session.

*FORK default
INFO: FORK session is: default

Print the current session name by using the *FORK with no session name:

*FORK
INFO: FORK session is: Server

*FORK and variables. It is possible to assign a session name to a variable, and then create a new session with that name. For example:

$my_session=Client
*FORK $my_session
INFO: FORK session is: Client

All sessions will be automatically closed when the script terminates.

Debugging in a multi-session environtment: When in INTERACT, either via the *INTERACT command in the script or via instant-interact (by pressing ^\), the *FORK command is still available, permitting the switching of sessions while in INTERACT mode. For example:

expect-lite directive: *INTERACT 

 Press '+++' to end interact session & return to script 

% *FORK CONSOLE

INFO: FORK session is: CONSOLE, Active sessions are: default CONSOLE
#

With the addition of multiple sessions, decoding the output will become more difficult. Remember, although multiple sessions are supported, expect-lite is still single threaded, meaning commands will always be executed the order as they appear in the script.

Managing Logging INFO, NOINFO, WARN, NOWARN, DEBUG, NODEBUG

Expect-lite provides additional information (logging) which makes it easier to debug failures. There are three levels of logging: info, warnings, and debug. By default, info and warnings are turned on. However, using the logging directives: *INFO, *NOINFO, *WARN, *NOWARN, *DEBUG, *NODEBUG, these messages can be managed.

Example of the different logging levels are:

INFO: Entering FORK sessions, Constant substitution, Conditional Statement Results, Jumping to lables, Including files, Dynamic Variable Assignments, User Defined Prompt assignments
WARN: Instant-Interact feature disabled, Wait-for-prompt timeouts, Using localhost (r=none), Unable to parse Conditional, Dynamic Variable Capture timeouts, Unable to parse math variable statement, Unable to parse FORK session, Lost connection to remote host
DEBUG: Additional debugging information for: User Defined Prompt, FORK session instantiation/switching, '<' searching, Dynamic Variable capture

In addition to the Logging Directives, invoking expect-lite with a -v or -vv will force INFO & WARN on or WARN & DEBUG on, respectively. It should be possible to direct the amount of logging desired by using a combination of the command line arguments (-v or -vv) and the Logging Directives.

In addition to the three logging levels above, there is another more serious level of ERROR, which can not be supressed, and will terminate expect-lite with a non-zero return code. Examples of ERRORs are: unable to connect or lost connection with remote host, include file not found, bad timeout value (non-numeric) or infinite loop detection.

It may be tempting to turn off logging once a script is debugged. However, if the script is being run in an automated environment (like crontab), it is recommended that INFO and WARN be turned on, as the additional infomation will make debugging failures much easier.

Examples of Expect-lite

Below are some examples to better illustrate what can be accomplished with expect-lite:

Setting the IP address on a secondary interface of the remote host (be sure to escape the dot's in the IP address)

$local_eth=eth1
$ip_addr=192\.168\.10\.2
# change timeout value to 10 sec
@10
>ifconfig $local_eth $ip_addr
#check to ensure that we set the right IP address
>ifconfig
<$ip_addr

Starting a telnet to another host

@2
>telnet remote-host-018
<Connected to
<login
>>root
<assword
>>secret_password
# issue a command once logged in
>pwd
>^]
>quit

Starting a ssh session to another host

>ssh root@host-021
<assword:
>secret_password
# issue a command once logged in
>ls
>exit

Use of 'screen' command to keep a command alive after test completes

; ==== Use Screen command for to keep application running ====
>screen 
> $pmm 
# Check that HW is alive
>ping
<Successfully pinged the PM H/W
; ==== Keep pmm application active, and detach from this screen
>^A
>^D
<detached
>

Assigning a dynamic variable $host using regex to capture the hostname from 'uname -a' command

>uname -a
+$host=Linux ([a-z\-0-9]+)
Assigned Var:host=remote-host-008
>

Assigning multiple dynamic variables using regex to capture environment variables $HOME and $SHELL, while expecting $TERM=xterm

>env | sort
+$home=\nHOME=([a-z/]+)
+$shell=\nSHELL=([a-z/]+)
<TERM=xterm
Assigned Var:home=/home/joe
Assigned Var:shell=/bin/bash
>

Manage multiple Screen commands with a dynamic variable and regex capture

# show the last screen in the list 
>screen -list | grep pts | sort | tail -1
# regex capture to dynamically assign the result to var $myscreen
+$myscreen=([0-9]+\.pts.[0-9]+.[a-z\- 0-9]+)
# connect to $myscreen
> screen -r $myscreen
# inside existing screen, kick it to get a prompt
>^M
> sleep 3
> ls
>
>^A
>^D
<detached
>

Using Regex in expect-lite to allow multiple responses

>id
# allow either groups users or wheel
<(users|wheel)

Create a temporary file with unique name including script name and date. The predefined variable $arg0 is contains the expect-lite script name.

# date stamp of script run time
>echo `date +_%Y_%m_%d_%H-%M-%S`
+$DATE=\n(_[0-9_-]+)

$temp_file=$arg0$DATE
; === Create a temp file using bash "here doc" method
>cat > $temp_file <<'+++'
>line 1
>line 2
>+++
>

Test Failure?!?

When will expect-lite fail a test? It will only fail a test when an expected result (after issuing a command) does not appear in the specified timeout period. For example in the following command file:

$ip_addr=192.168.10.99
>/sbin/ifconfig eth0 $ip_addr
>/sbin/ifconfig eth0 
< $ip_addr

The first line will be sent (blindly), since there is no expected return. On line 3, the ifconfig command is sent again, and the script is looking for a desired result of 192.168.10.99 (using the var $ip_addr). If for some reason the ip address is different than ip address was returned, the script would fail.

Not Expect
A test can also fail should text appear that is unexpected (such as an error). This does not clear the expect input buffer, and should be used before a valid expect. How long should the script wait for the un-expected? In order to reduce delays in script running time, the Not Expect feature only waits for 100ms. This is usually enough time to detect quick error responses such as "file not found".

For example: Fail device doesn't exist

>ls -l /dev/linux
-<No such file

In another example: We don't like Mondays

; === fail if today is 'Monday'
>date +%A
-<Monday

Since the expect input buffer is not consumed, valid expects can still be performed. in the following example, ttyS2 will still be found.

>ls -1 /dev/ttyS*
-<ttyX
<ttyS2

It is recommended that Not Expect lines be used before valid expect lines.

How can it help?

Expect-lite can be used to quickly create automated tests. It can collapse complicated tests into a single command line. It can also be used in nightly regression testing, performing simple functional tests providing confidence that core functionality has not been broken by the previous day's submissions.

Limitations

Expect-lite is limited to what a person could do in one (1) terminal window (or xterm). It cannot start a program in one window and run a different program in a separate window. However it does support multiple sessions via *FORK which should meet the needs of most.

Standard programming constructs such as looping and branching are supported as of version 3.0.0. Real Expect can do these things and more. That said, looping is limited to a repeat loop and branching can be accomplished with Labels.

Expect-lite has been purposely limited to keep scripts simple, easy to use and maintain. Although more complex scripts can be created, basic expect-lite scripts can be created by simply cutting and pasting text from a terminal window into a script and adding '>' '<' characters.

Implementation Details

Variables

Variable names must be restricted to the following set of characters [A-Za-z0-9_]. Variable values may include spaces, and quoting is not required nor permitted. For example:

$bell=For whom the bell tolls
>echo $bell
For whom the bell tolls

Implied "wait for prompt"

Using the '>' character implies waiting for the prompt before sending the command. The definition of a shell prompt a: $, #, % or >. It is often useful to follow a '<' with a '>' for force the script to wait for the prompt:

> $some_long_command
<critical value
# wait for prompt
>

Prompt detection can be problematic, and by no means is expect-lite's method perfect. Expect-lite looks for a prompt character ($, #, % or >) followed by a space at the end of line. This should work for most standard prompts, however coloured or fancy prompts may not be detected. As of version 3.1.5, user defined prompt is supported without modifying expect-lite.

It may be useful to create files on the fly which will be used in the test. Since expect-lite does not read/write files directly, this function can be done with the linux command 'cat' and the bash here doc method. Creating a regex file for example:

; ==== Create Test File on the fly
> cat > $regex_test_file << '+++'
>T00001/a(?$XL.*)/tag=0x08000001
>T00002/b(?$XR.*)/tag=0x08000002
>T00003/c(?$XO[0-7]*)/tag=0x08000003
>T00004/d(?$XH[0-9A-F]*)/tag=0x08000004
>T00005/e(?$XD[0-9]*)/tag=0x08000005
>+++
>

In the above example the script initiates a cat to a file (referenced by $regex_test_file ). Normally the '>' would be used to send lines to the remote host. However, there is an inherent "wait for prompt", which will politely wait for the next line. The last '>' is there to wait for the prompt to return after the cat command.

Predefined variables

Expect-lite has minimized the use of predefined variables to allow the user the widest latitude in selecting variable names. However a few predefined variables are useful and listed below:

Predefined variable	Usage
$arg0	expect-lite script name. Set to the c= or cmd_file= value. Useful for creating temporary files unique to the test script.
$timeout_multiplier	Typically used as a constant (passed in from the command line) which the integer value is multiplied by each of the timeouts in the script. i.e. @10 x $timeout_multiplier = real timeout. The value default is 1.

Timeouts

Setting the expect-lite timeout to 0 isn't actually zero as this will cause the keyboard input buffer on the remote machine to overflow (and lose characters). Expect-lite adds a 50ms delay between lines when the @0 is used to prevent the overflow.

Expect Input Buffer

Expect by default places strings that are sent back (from the remote host) into an expect buffer ($expect_out(buffer)). Expect-lite leverages this built in function, and uses it to perform multiple searches through the buffer when using the '<' , '+<' or '-<' actions. The actions '>' and '>>' will clear expect buffer to prevent spurious matches.

Timing is everything: delay_wait_for_host

When communicating with a remote host, enough time must be provided for the remote host to respond. There is an internal global value of $delay_wait_for_host (in mili-seconds) which can be tuned to your environment. This value is also used by the Expect Not feature to determine how long Expect-Not should wait for the unexpected.

Unsynchronized messages disclaimer

Effort has been put into making the output as synchronous as possible. However because what is seen on stdout or screen is data returned/echoed from the remote host and printable comments are directly printed to stdout, it is possible to have a printable comment appear in the stdout before the action it references is returned from the remote host. This lack of synchronization can sometimes be minimized by adding an additional '>' after the printable comment as shown in the following:

; === starting long command
>
>start_long_command

Instant-Interact: More than just a pretty prompt

When the user presses ^\ (control+backslash) during the running of a script, expect-lite will drop into interactive mode. In version 3.5.0, this the underlying expect's standard interact mode. This feature requires an additional tcl package Tclx, which is automatically detected at script runtime. If the Tclx package is not available, the instant-interact feature will be disabled, and a warning will be printed.

Control will be returned to the script when '+++' is typed. In addition to returning to the script, the current command in the script will be "Passed", disabling it from failing the script. This is useful when debugging a script, the user notes that the response is not as expected, and presses ^\ to moved to interact mode. Returning from interact mode, the line which would have failed the script (because of the response not being as expected) will be "Passed" and the script will continue. This Passing action is line oriented, and other '<' lines in the script can still fail the script.

There is experimental (read: could be better tested) access to expect-lite commands or directives.

*FORK: what happens behind the scenes

A new *FORK session actually creates an additional login(s) to the remote host from the localhost using the same connect method as the first (or default) session. It then sets a sane prompt, and waits for the first script command in that session. This provides multiple sessions on the remote host, which then could be used to drive separate programs (see test_fork_web_client_server.txt for example). Graphically it would appear as:

Localhost                     Remote Host
         +----New_Session1--->Client App
         +----New_Session2--->Server App

What's missing

In order to maintain an easy-to-learn, simple-to-use expect like language, looping has been intentionally kept simple and intuitive. Most expect-lite scripts will be to be top down, single pass scripts, making the scripts easy to create and read. These limitations preserve the KISS principle.

If more complexity is required, consider using embedded expect inside expect-lite, or writing a native expect script.

Why Expect-lite

Expect-lite was written so you don't have to learn expect. It provides a quick way to write simple expect scripts using just the > and < characters.

COPYING

Copyright (c) 2007, Freescale Semiconductor, Inc.
All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are
permitted provided that the following conditions are met:

 * Redistributions of source code must retain the above copyright notice, this list
	of conditions and the following disclaimer.
 * Redistributions in binary form must reproduce the above copyright notice, this
	list of conditions and the following disclaimer in the documentation and/or other
	materials provided with the distribution.
 * Neither the name of the Freescale Semiconductor nor the names of its contributors
	may be used to endorse or promote products derived from this software without
	specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

15 January 2009
http://expect-lite.sourceforge.net/

Subscribe to expect-list-users Discussion List

this document for version 3.5.0 and above