Q: Help, I'm stuck! I've been using your articles to write Expect scripts to send documents with SFTP. When everything goes according to plan, it works fine. However, when something goes wrong, my Expect script doesn't stop, and no error is reported to my CL program--the exit status is zero. I've looked everywhere for an answer. Help me Obi Wan Kenobi. You're my only hope!

A: You're right that OpenSSH's SFTP utility will return a non-zero exit status to its caller if it fails. However, your CL program isn't what's calling SFTP--the Expect script is. So if your Expect script is not detecting an error but rather is returning zero to the CL program, the CL program has no way to tell that something failed.

This article shows you how to add error handling to your Expect script and how to make Expect report errors back to a CL program.

Review: How the Original Script Worked

The basic thing you need to know is that Expect scripts work by searching for a string. They sit and keep reading data from SFTP (or whichever program you're controlling) and adding that data to a buffer, and then searching that buffer for whatever they're looking for. That's pretty much all Expect does, read and compare, read and compare, read and compare.

So how does it know the difference between no data arriving, and what you'd consider an error? It doesn't. It knows when SFTP ends, and it knows when a timeout occurs, and those should be treated as errors, otherwise all it knows is that it hasn't found the string you're searching for.

Here's the example of an Expect script I gave in an earlier article, except that I've added line numbers to the left-hand side:

   #!/usr/local/bin/expect -f
1  spawn sftp klemscot@ssh.example.com
2  expect "Connecting to ssh.example.com..."
3  expect "Password:"
4  send "bigboy\n"
5  expect "sftp>"
6  send "put myfile.csv\n"
7  expect "sftp>"
8  send "quit\n"
9  exit

That example was intended to be as simple as possible, to give you a starting point. Here's what it does:

1. The spawn command starts the sftp program, telling it to connect to ssh.example.com, and sign-in as userid klemscot.
   
2. Sit and wait ("read and compare" in a loop) until the string "Connecting to ssh.example.com..." is found.
   
3. Sit and wait (read and compare) until the string "Password:" is found.
   
4. Send my password "bigboy" followed by \n, which symbolizes the Enter key.
   
5. Sit and wait until the string "sftp>" appears.
   
6. Send the string "put myfile.csv\n"
   
7. Sit and wait for "sftp>" again.
   
8. Send "quit\n"
9. Exit the Expect script, returning 0 to the caller.
   

It's easy to understand, but is it good enough? The answer is probably not, because when unexpected messages come up, Expect continues to sit and wait, still hoping beyond all hope that it'll get the string I told it to expect.

Checking for Other Conditions

Fortunately, you can do better than this. The expect keyword is not limited to only one string to wait for, as I demonstrated above. It can wait for multiple strings and take separate actions depending on which one it receives. It also can have a "default" action that occurs whenever either the SFTP program ends or when the expected string(s) do not appears in a suitable amount of time. For example, suppose the first few lines of the script look like this:

#!/usr/local/bin/expect -f
set timeout 20
spawn sftp klemscot@ssh.example.com
expect { 
   default {exit 2}
   "Connecting to ssh.example.com..." 
}

The "set timeout 20" tells it to time out after 20 seconds. It spawns SFTP as it did before, but notice that the "expect" command is completely different. The curly braces identify a group of strings that it will wait for. In this example, there are two options: the string "Connecting to ssh.example.com..." could be found, or the default action could happen. The default action will happen when the timeout period is reached or when SFTP ends. You'll notice that "exit 2" is listed under the default action--this tells Expect to exit and set the exit status to 2. If your CL program is checking the exit status, it'll see the number 2 and know that something went wrong.

After connecting, the old script waited for the string "Password:" before sending the password. There's a problem with that, however: sometimes password is spelled with a capital P, and sometimes it's lowercase. There's another problem with this script, in that the first time you connect to a given host, it asks you if you're sure you want to connect. If that prompt came up, the original script would be flummoxed, since it would still be in "read and compare" mode looking for "Password:". Here's how I'd solve that:

expect {
   default {exit 2}
   "continue connecting (yes/no)?" {send "yes\n"; exp_continue}
   "assword:"
}
send  "bigboy\n"

As you can see, I've added the default action, so if there's a timeout or SFTP ends, it'll exit the Expect script with exit status 2. Furthermore, I've told it to wait for two different strings, one is the string asking if we want to continue connecting, the other is the actual password prompt. If the "continue connecting" prompt does appear, it sends "yes&92;n" and then does "exp_continue", which tells it to continue the same expect statement. This puts it back into "read and compare" mode, and therefore it'll continue waiting until the password prompt finally appears, or the default action happens.

Hopefully that makes sense. By coding a separate condition for each thing you expect to see, as well as a default condition, you can handle all the different things that could happen, and therefore handle errors smoothly.

Putting It All Together

Here's the updated version of the script in its entirety:

#!/usr/local/bin/expect -f
set timeout 20
spawn sftp klemscot@ssh.example.com
expect {
   default {exit 2}
   "Connecting to ssh.example.com..."
}
expect {
   default {exit 2}
   "continue connecting (yes/no)?" {send "yes\n"; exp_continue}
   "assword:"
}
send  "bigboy\n"
expect {
   default {exit 2}
   "sftp>"
}
send "put myfile.csv\n"
expect {
   default {exit 2}
   "not found" {exit 3}
   "sftp>"
}
send "quit\n"
exit 0

How to Enter the Script

When entering a PASE shell script, keep in mind these things:

• sftp and Expect run under PASE.
• PASE is a Unix environment. It lets you run AIX programs on IBM i.
• Unix environments are case-sensitive.
• Unix environments work in ASCII, not EBCDIC.
• Unix scripts end each line of code with a LF character(not CRLF like Windows does).

With all that in mind, it's important to create your script so that it'll be in ASCII and it'll have the correct end-of-line convention. In my opinion, the easiest way to do that is to create the script from PASE. Since PASE understands ASCII only, it'll mark your file with ASCII.

To create the file from PASE, use the PASE echo command to write the first line of the script. To do that, type the following to start an interactive PASE environment in which you can type PASE commands:

CALL QP2TERM

Now use echo to write the first line of the script:

echo "#!/usr/local/bin/expect -f" > /tmp/expectScript.txt

Now press F3 to exit back to your ordinary IBM i environment, and open up your new script file in an editor. If you have WDSC or RDi, you can edit the file by going into the IFS area in Remote Systems Explorer, find the new file, and edit it. To do it from green-screen, use the EDTF command instead.

EDTF STMF('/tmp/expectScript.txt')

EDTF works very much like SEU. You insert new lines by placing an I in the blanks on the left-hand side of the screen. You delete with D, you copy with C, and so on, just like with SEU. When you're done, press F2 to save your changes followed by F3 to exit.

Running the Script Interactively

You can try out your Expect script by starting an interactive PASE session and then invoking the script. This is an especially good idea when debugging, because it's so easy to see what's happening when the script is running. To start the interactive session, type

CALL QP2TERM

To run the script, type:

/usr/local/bin/expect -f /tmp/expectScript.txt

If you need more debugging information--that is, you want to see what Expect is "thinking" when it's running your script, including which data it's reading and comparing--you can add the -d switch to make it display more.

/usr/local/bin/expect -df /tmp/expectScript.txt

If the script finished successfully, you want the exit status to be 0, and if it failed, you want it to be a positive number that symbolizes the reason for failure. When running the script interactively, you can type the following to see what the exit status is:

echo $?

Calling Your Script from CL

Now that you have your shell script, you'll want to know how to invoke it from your CL programs. CL will need to run the script and then retrieve the exit status to see if it went well. You could use QP2TERM from CL, but keep in mind that QP2TERM only works interactively, so a better choice is probably QP2SHELL, which can be used interactively or in batch.

QP2SHELL receives the shell name in its first parameter. The Expect program functions as a shell script interpreter and therefore is the name of the shell in this example. The remaining parameters to QP2SHELL are the parameters you'd pass to Expect.

When QP2SHELL ends, it'll set the ILE return code, which is located in the job information, to the exit status of the Expect script. To retrieve it, you'll need to call the QUSRJOBI API. Here's an example:

PGM

    DCL VAR(&RCVVAR)    TYPE(*CHAR) LEN(200)
    DCL VAR(&RCVVARLEN) TYPE(*CHAR) LEN(4)

    CALL PGM(QP2SHELL) PARM('/usr/local/bin/expect' +
                            '-f'                     +
                            '/tmp/expectScript.txt'  )

    CHGVAR VAR(%BIN(&RCVVARLEN)) VALUE(200)   /* SIZE OF &RCVVAR */

    CALL PGM(QUSRJOBI) PARM(&RCVVAR    +
                            &RCVVARLEN +
                            'JOBI0600' +
                            '*'        +
                            ' '        )
    IF (%BIN(&RCVVAR 109 4) *NE 0) DO
       SNDPGMMSG MSGID(CPF9897) MSGTYPE(*ESCAPE) MSGF(QCPFMSG) +
                 MSGDTA('COMMAND FAILED')
    ENDDO

ENDPGM

Previous Articles

I have written the following previous articles about using SFTP on i:

• The SSH, SCP, and SFTP Tools from OpenSSH
• Error Handling in SFTP Scripts
• An Easy Way to Put Variables in an Expect Script for SFTP