expect-lite

Automation for the rest of us

Tips and Techniques: Simplifying the Debugging of expect-lite scripts

Introduction

Over the years of using and improving expect-lite, a few debugging techniques have proven quite useful. The purpose of the document is to provide some additional information to assist the user of expect-lite in debugging scripts, and tips to make expect-lite script writing even easier. Familiarity with expect-lite is assumed, if not, please refer to the expect-lite documentation.

Some Tips and Techniques for expect-lite:

Interact Setting a breakpoint, the easiest debugging aid to use
IDE Integrated Debugger Environment new in 4.0.1
Finding In... --verbose, a debugging aid that displays what expect-lite '<' statements are finding
Pasting into a running script via the IDE
Showing Variable Values Displaying expect-lite variables & values
Showing Environment Values Displaying expect-lite environment e.g. directives, shell environment variables, infinite loop counter new in 4.8.0
Running the script to completion Good for debugging
Define '<' narrowly Avoid mis-matches by being specific on what is expected
Decreasing log chatter
Removing colour from output
Adding web colour to output with HTML tags
General Tips Some rules of thumb when writing expect-lite scripts

Interact - setting breakpoints

Interact may be the quickest, easiest, and overall best debugging aid. It is equivalent to setting a breakpoint in the script. Interact is a mode which turns control of the keyboard over to the user, so that one may type directly to the process on the remote host With version 3.5.0 there are two methods to invoke Interact: programmatic, and instant-interact.

Programmatic Interact is called in the script with the following command:

*INTERACT

Expect-lite will pause at this point in the script, and connect the keyboard to the remote session (which may be at a prompt). Any command may be entered and responses observed. Typing '+++' (3 pluses) will return control to the script, and it will continue. This is very helpful for automation assist, allowing the script to perform complicated setup commands, before turning control over to the user for an interactive session.

The other method is instant-interact. This feature no longer requires the tcl package Tclx to be installed. The user can press '^\' (control+backslash) at anytime creating a breakpoint on the fly. This is the easiest, and fastest way to debug a script, dropping the user into the IDE.

Debugging with the IDE

To make debugging of scripts even easier, both methods of interact support the new and powerful expect-lite Integrated Debugger Environment (IDE). Inside the IDE, it is possible to do many things:

Type command directly to the remote host (router, embedded system, Linux box)
Step or sKip line by line through the expect-lite script
Show all expect-lite variables and constants
Show the most recently executed lines, or the next few lines of the script
Paste expect-lite script line(s) into the IDE and have them execute. This is really cool, since it will dereference any variables in the pasted text, or allow you to run just a few lines of script over and over
The IDE can't die, since *NOFAIL automatically enabled while in the IDE.
Type expect-lite command directly while in the IDE, including >, <, $var=, +$var,@n (set timeout).

Stepping, sKipping and Showing
In the above list, #1 is the same as previous versions of expect-lite. Stepping, sKipping, and Showing are new in the IDE (#2, #3, #4). These commands are all two-keystroke commands. The first key is the escape key (or esc) followed by s, k, c, v, e, h and 0 through 9. These keys are not echoed to the screen, nor are they received by the remote host. To represent the key presses [type <esc>h] has been added to the text for completeness.

[type <esc>h]
IDE: Help
  Key          Action
  ----        ------
  <esc>s      Step
  <esc>k      sKip next step
  <esc>c      Continue
  <esc>v      show Vars
  <esc>e      show Env
  <esc>0to9   Show N next lines of script
  <esc>-1to-9 Show N previous lines of script
  ctrl+D      Quit & Exit expect-lite
  <esc>h      this Help

Showing the lines of the script running up to the breakpoint, the current line, or the next few lines of the script can all be displayed by typing <esc> and a number. If the number is negative, say -3, then the last 3 lines will be displayed. The following example is displaying the next 5 lines of the script:

[type <esc>5]
$ Printing next 5 line(s) of script:
> echo hello
?if 1 == 1 ? %PAU
>> pwd
+$P=\n/home/([a-z]+)
>echo "test_include.txt"

Typing <esc>0 will display the current line in the script:

[type <esc>0]
$ Printing current line of script:
*INTERACT

To exit the IDE and continue the execution of the script type <esc>c This is the same as typing '+++'

Typing commands directly into the IDE
In the IDE, expect-lite script commands can be typed directly into the terminal, even assigning variables and displaying them, inside the paused script, for example:

$MYVAR=today
[type <esc>v]
$ DEBUG Info: Printing all expect-lite variables
Var:MYVAR                 Value:today
Var:junk                  Value:orbit-root
Var:manarch               Value:i386                   Const:ppc

>>pwd
pwd
/home/user

Note that variables are printed with any overriding constants.

Pasting into the IDE
Of course, typing is useful, but copy and paste is much easier. Pasting lines into the IDE can be really helpful in debugging a script. For example, you may want to debug a dynamic variable assignment. The script may have the following lines:

>ifconfig
# get IP address of loopback interface
<lo
+$ip_addr=inet addr:([0-9.]+)

This code snippet displays all the interfaces on this machine, expecting lo will clear the capture buffer above 'lo', thus removing the info from the eth0 interface. The dynamic variable will capture any numbers and periods after the words 'inet addr:'

By copying the lines from the script and pasting them into the IDE the terminall will display following:

$ ifconfig
eth0      Link encap:Ethernet  HWaddr 00:30:65:98:e5:4a  
          inet addr:10.1.1.125  Bcast:10.1.1.255  Mask:255.255.255.0
          inet6 addr: fe80::230:65ff:fe98:e54a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          ...
          Interrupt:41 Base address:0x6c00 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          ...
Assigned Var:ip_addr=127.0.0.1
$

And of course, code blocks can be pasted into the IDE as well. Take care to paste the entire code block to get the desired result. Pasting the $i=0 is important, since it initializes the counter in the while loop.

; === while loop
$i=0
[ $i < 3
	>echo "hello $i"
	<hello $i
	+$i
]

Pasting in the modern age
As of 4.8.0 some additional constraints have been added when pasting into the IDE, as the IDE was interfering with typing to the remote host. To prevent interference with typing to the remote host, you should start your paste with of the following lines:

variable assignment ($var=value)
comment (single semi colon)
send (>)

By starting your paste with one of these lines, it triggers the IDE to treat the additional lines as a paste rather than typing to the remote host.

Additionally, there are few subtleties which the user should be aware when typing or pasting expect-lite lines into the IDE:

variable assignment must include equal sign e.g. $var=
comment lines must have a space after the semi-colon ; this comment
conditional statements must have the english word 'if' when pasteing e.g. ?if this != that?

conditionals do not require the optional 'if' when running in a script, only applies to pasting the first line

IDE, It lives!
It is possible to paste in lines which would normally fail the script. However getting kicked out of the script while one is debugging is not only irritating but not all that useful. For example, using similar script lines to above:

>ifconfig
<eth2

Since there is no eth2 interface on this machine the script snippet will fail. However, the *NOFAIL directive is automatically enabled inside the IDE, there pasting the snippet results in:

$ ifconfig
eth0      Link encap:Ethernet  HWaddr 00:30:65:98:e5:4a  
          inet addr:10.1.1.125  Bcast:10.1.1.255  Mask:255.255.255.0
          ...
 FAILED COMMAND: 
    Expect Failed:eth2
$

Instead of the expect-lite exiting, and declaring the script has failed, it remains in the IDE (at the *INTERACT, or breakpoint), ready to receive another command.

Running to Completion

The normal operation of expect-lite is to stop with any failure. However, this may not always be the desired behaviour. Using the *NOFAIL directive, will prevent the expect-lite script from stopping prior to completion. The directive *NOFAIL can be placed in the script, or on the command line (as well as any other directive).

expect-lite will still detect failures and report them at the end with the message:

##Overall Result: FAILED (*NOFAIL on)

Once *NOFAIL has been directed, it remains on for the rest of the script. Conceptually it is the opposite of *FAIL, which fails the script immediately (if *NOFAIL isn't set).

Finding In... Verbose

Since most expect-lite scripts are composed of > and < lines, it can be difficult to understand why expect-lite isn't finding the desired text. Using the -vv or --verbose option (in version 3.1.4 or later) will display the string that is to be matched (find) and the entire string that is being searched (in).

For example, for the given expect-lite code:

>cat $tmp_file
<Proto \[LAN\]Addr:Port => Addr:Port \[WAN\]Addr:Port <= Addr:Port
<6[ \t]+[0-9a-f:]{11,14}[ \t]+[0-9a-f]{8}:22

With the --verbose option, the output will appear with the additional lines:

cat /proj/regression/tmp/junk
Dec 31 1969 19:43:12:025952:INFO :CLI :cli.c:2945:_cli_handle_short_cmd:
 Execute command: "fwdstat".
Proto [LAN]Addr:Port => Addr:Port [WAN]Addr:Port <= Addr:Port
 6 c0a85001:41696 c0a81401:22 c0a81401:22 c0a81402:32770

find<<Proto \[LAN\]Addr:Port => Addr:Port \[WAN\]Addr:Port <= Addr:Port>>
 in<<cat /proj/regression/tmp/junk
Dec 31 1969 19:43:12:025952:INFO :CLI :cli.c:2945:_cli_handle_short_cmd:
 Execute command: "fwdstat".
Proto [LAN]Addr:Port => Addr:Port [WAN]Addr:Port <= Addr:Port>>

find<<6[ \t]+[0-9a-f:]{11,14}[ \t]+[0-9a-f]{8}:22>>
 in<<
 6 c0a85001:41696 c0a81401:22>>

Both the 'find' and 'in' text are wrapped in << >> to show any line feeds. The 'find', if successful, will almost always be at the bottom of the 'in' text. Often when there is a unexpected failure, the '<' is defined too broadly, and expect-lite has matched an unexpected piece of 'in' text. (see Define '<' narrowly)

As of expect-lite version 3.1.4, -vv or --verbose will also display 'find and 'in' for dynamic variable capture, thus providing a bit more information to assist in debugging this feature as well.

As of expect-lite version 3.5.0, -vv or --verbose will display all warnings and debug information (including user defined prompt debug info)

Showing Variables, Values & Constants with *SHOW VARS

Sometimes when troubleshooting a script, it is really useful to display the value of an expect-lite variable. The easiest method to display a variable is to use the printable comment ';' For example, there is a variable named '$max', by adding the following line to the script, $max will be dereferenced and displayed:

; The value of max is:$max

The above method works well for a few variables, but it may be necessary to view all the expect-lite variables. In previous versions of expect-lite, a script (show_vars.inc) was required to view all variable values. As of version 4.0.x, the expect-lite directive: *SHOW VARS now displays variables, values, and any overriding constants.

All variables can now be displayed by using the line:

*SHOW VARS

The output will appear like the following:

expect-lite directive: *SHOW VARS 
DEBUG Info: Printing all expect-lite variables
Var:arg0                  Value:test_script.elt
Var:count                 Value:14
Var:mac_da10              Value:00:7F:DE:6A:FF:FA     Const:00:30:65:98:e5:4a
Var:mac_da11              Value:88:45:3D:2A:95:63
Var:mac_da12              Value:38:8B:50:49:AE:0D
Var:mac_da13              Value:38:E4:EB:0C:DE:4B

*SHOW VARS can also be used with *INTERACT and instant-interact to view the values all assigned variables, however, <esc>v performs the same action and is less typing.

Showing Environment with *SHOW ENV

The goal is to keep it simple. However, expect-lite has matured enough that is may be hard to keep track of which directives are enabled, and how many loops have occurred in the script. To make the script writer's job a little easier, all that info is now printed with the directive *SHOW ENV.

$ *SHOW ENV
DEBUG Info: Printing expect-lite directives/environment
Environment:          Value:
CURRENT_FORK_SESSION  default
DEBUG                 off
DVPROMPT              on
EOLS                  LF
EXP_INFO              on
INFINTE_LOOP_COUNT    5000
INFO                  on
LOG                   off
LOG_EXT               .log
NOFAIL                on
NOINCLUDE             off
NOINTERACT            off
TIMESTAMP             off
USER_DEFINED_PROMPT   .*$ $
WARN                  on
fuzzy_range           10
timeout               2
EL_CONNECT_METHOD     ssh_key
EL_REMOTE_HOST        localhost

The directives are listed in sorted order, fuzzy expect range, and timeout are printed. Then finally any Shell environment variables are printed.

The *SHOW ENV command can be added to the expect-lite script to display the above. This can aid in debugging overnight regression logs, or even supporting others running your scripts. The expect-lite environment can also be displayed from within the IDE by typing <esc> e.

Define '<' narrowly

Do not define '<' too broadly. For example, DO NOT use the following:

<\n.*

The above defines 0 or more of any character after a newline. This will match just about anything, and more than likely not what is intended.

Instead use a more specific '<'. If the expected string is only composed of numbers at the beginning of a line, use:

<\n\d+

Use with the --verbose option to see what is being matched, and to assist in refining the '<' statements.

Decreasing the log chatter

After one's scripts are succesfully debugged, it may not be desireable to see all the log chatter which expect-lite produces, such as warnings, conditional reports, dynamic variable assignments. This chatter can be reduced by using the expect-lite logging directives at or near the beginning of the script, such as:

*NOWARN

If while debugging the script, it is desirable to see the additional logging info (chatter), it can be turned on via the command line with the -v parameter:

expect-lite -v r=myhost c=myscript

Removing Colour from output

In an automated environment, it is probably not desirable to have ANSI color sequences embedded in the logs. Although it is possible to use the expect-lite directive *NOCOLOUR in each script to turn off colour, there is an easier way. Define the terminal type to a dumb terminal, such as tty.

export TERM=tty

expect-lite will detect the dumb terminal type and turn off colour output automatically.

Or as of version 4.0.x, *DIRECTIVES can be placed on on the command line, therefore it is possible to turn off colour on the command line:

expect-lite -r myhost -c myscript *NOCOLOUR

Adding Web Colour to output

Depending on your regression reporting engine (e.g. Jenkins), it may be desirable to display colour but in a web browser such as Firefox. expect-lite looks for a special terminal type, web, and enables HTML colour codes in the output.

export TERM=web

Merging expect-lite and the web is still considered experimental. Currently there is no directive equivalent.

General Tips

Here are some simple tips when script writing:

Use reasonable timeouts, if 30 seconds is needed to get a response, set the timeout at 45 or 60 seconds, not 600.

There is no cost to changing the timeout, timeout values can also be variables

Beware of expect-lite using regex, when creating lines like: <0.005 secs (5 micro secs)

The parentheses is used by the regex engine, instead escape these characters: <0.005 secs $5 micro secs$
or use '<<' which does not use regex, and does not require escaping: <<0.005 secs (5 micro secs)

Use the expect character '<' or '<<' often. Check for valid results when possible. A script which expects nothing will never fail!
Use printable comments ';' and ';;' often. Think of it as writing a note to oneself, it will make reading log files much easier. As of version 3.7.0 printable comments will be coloured blue (this is user configurable).
Variable assignments use no spaces e.g. $var=value. Note no spaces around the equals sign, this permits leading spaces in the value such as $var= myvalue

Summary

By using these troubleshooting aids, it should be even easier to write and debug expect-lite scripts. Feel free to send me any tips. cvmiller at gmail dot com

Why Expect-lite

Expect-lite was written to create quick and easy automation of repetitive tasks.

15 January 2015
http://expect-lite.sourceforge.net/

this document for version 4.8.0 or later