Q: My company is switching from FTP to SFTP (from OpenSSH), and I need to convert the scripts. I'm using the article "The SSH, SCP, and SFTP Tools from OpenSSH" (January 10, 2008, article ID 56131) as a guideline, and everything works nicely, except that I can't find a way to verify that my transfers were successful. How do I know if they fail?

A: The OpenSSH tools (both scp and SFTP) are Unix programs. By convention, the way that Unix software tells you whether it succeeded or failed is by setting something called the "exit status." The exit status is an integer that programs set when they finish running. The next program can then check the value of that exit status. In Unix, a program sets the exit status to 0 to indicate success and to a higher number to indicate failure.

Exit Status

For example, type CALL QP2TERM to enter an interactive PASE shell. Then try typing a Unix command. In this example, I use the ls command, which lists the contents of a directory:

  ls /QIBM

PASE responds to this command by listing the files in the /QIBM directory of my IFS. But it has also set an exit status. To display that exit status interactively, I can type the following command:

  echo $?

The echo command prints something on the screen, and $? is a special variable that represents the exit status. Assuming that my ls command succeeded, it should print the number 0 to tell me that all was well. On the other hand, try listing a directory that doesn't exist:

> ls /fuzzbuzz
  ls: 0653-341 The file /fuzzbuzz does not exist.

> echo $?
  2

Because the ls command failed, the exit status is now set to 2 instead of 0. I can use that exit status any time I want to determine whether the command succeeded or failed.

The same is true when I use the scp tool from OpenSSH to copy a file over a network. If the file transfer is successful, the exit status is 0. If it fails, the value is something else:

> scp /home/klemscot/myfile.csv scottk@test.example.com:
  ssh: test.example.com: Hostname and service name not provided or found
  lost connection                                                       

> echo $?
  1

So anytime I want to know whether scp succeeded or failed, I can check the exit status. If it's set to 0, I know that my scp command was successful. If it's set to anything else, the scp command failed.

Using the Exit Status from CL

You need to be able to check the exit status from a CL program, of course. The easiest way to do that is to run scp from QShell. When a QShell command finishes running, the system sends you a completion message QSH0005, and the exit status of the command you ran is provided as a binary field in the first four bytes of the message data. Therefore, you can write code such as this:

PGM

    DCL VAR(&SCP)    TYPE(*CHAR) LEN(50) +
           VALUE('/QOpenSys/usr/bin/scp')
    DCL VAR(&CMD)    TYPE(*CHAR) LEN(500)
    DCL VAR(&MSGDTA) TYPE(*CHAR) LEN(4)
    DCL VAR(&MSGID)  TYPE(*CHAR) LEN(7)

    CHGVAR VAR(&CMD) VALUE(&SCP *BCAT '/home/klemscot/myfile.csv +
                           scottk@test.example.com:')
    STRQSH CMD(&CMD)

    RCVMSG MSGTYPE(*COMP) MSGDTA(&MSGDTA) MSGID(&MSGID)
    IF (&MSGID *NE 'QSH0005' *OR %BIN(&MSGDTA) *NE 0) DO
       SNDPGMMSG MSGID(CPF9897) MSGTYPE(*ESCAPE) MSGF(QCPFMSG) +
                 MSGDTA('SCP failed!')
    ENDDO

ENDPGM

In the preceding code, the RCVMSG command retrieves the completion message that QShell sent to my CL program. If my program received a QSH0005 message in which the first four bytes of the message data were set to x'00000000', I know that my scp command was successful. If they're set to anything else, my scp command has failed. For the sake of example, I send an *ESCAPE message when it fails.

The Exit Status and SFTP

When the OpenSSH team designed SFTP, it had a bit of a dilemma. SFTP can run many commands as a script. What do you do if some commands succeed and others fail?

The team decided on this simple but effective solution. If one of the following SFTP commands fails, the script is aborted, and the exit status indicates a failure: get, put, rename, ln, rm, mkdir, chdir, ls, lchdir, chmod, chown, chgrp, lpwd, df, and lmkdir.

Note that the put and get commands are in that list. So if you try to do a file transfer and it fails, the SFTP tool immediately stops your batch SFTP script and sets the exit status to a value other than 0, which you can check for from a CL program.

Unfortunately, a bug was in the initial release of OpenSSH for i5/OS, and it prevented scripts from aborting when an error occurred. IBM fixed this bug in the following PTFs:

Assuming that you have these PTFs installed, you can check for errors in your SFTP scripts the same way that you check them for scp scripts.

PGM

    DCL VAR(&SFTP)   TYPE(*CHAR) LEN(50) +
           VALUE('/QOpenSys/usr/bin/sftp')
    DCL VAR(&CMD)    TYPE(*CHAR) LEN(500)
    DCL VAR(&MSGDTA) TYPE(*CHAR) LEN(4)
    DCL VAR(&MSGID)  TYPE(*CHAR) LEN(7)

    CHGVAR VAR(&CMD) VALUE(&SFTP *BCAT '-b /tmp/script.txt +
                           scottk@test.example.com')
    STRQSH CMD(&CMD)

    RCVMSG MSGTYPE(*COMP) MSGDTA(&MSGDTA) MSGID(&MSGID)
    IF (&MSGID *NE 'QSH0005' *OR %BIN(&MSGDTA) *NE 0) DO
       SNDPGMMSG MSGID(CPF9897) MSGTYPE(*ESCAPE) MSGF(QCPFMSG) +
                 MSGDTA('SFTP script failed!')
    ENDDO

ENDPGM

Ignoring Failed Commands

In some situations, you don't want a script to abort if a command fails. Consider the following script:

chdir /home/scottk/daily
rm yesterday.csv
rename dailyfile.csv yesterday.csv
put dailyfile.csv

This script wants to keep two files on the SFTP server. Each time it uploads a new file, it renames the previous copy to "yesterday.csv." If there's already a yesterday.csv, it deletes it ("rm" is short for "remove"). However, what if there isn't already a yesterday.csv? In the preceding example, the script would fail on the rm command, which isn't what I want! I want my script to proceed, even if there's no yesterday.csv file.

To tell SFTP to ignore a failed command, you place a hyphen ( - ) in front of the command in your script. To make my preceding example work correctly, I change it to look like this:

chdir /home/scottk/daily
-rm yesterday.csv
-rename dailyfile.csv yesterday.csv
put dailyfile.csv

With those changes, all of the following statements are true:

• If the /home/scottk/daily directory doesn't exist, the script aborts, and none of the rm, rename, or put commands will run. My CL program will see that it failed.
• If the yesterday.csv file exists it will be deleted. If it doesn't exist, the script will continue without error.
• If the dailyfile.csv file exists, it will be renamed to yesterday.csv. If it doesn't exist, the script will continue without error.
• The dailyfile.csv file will be uploaded. If it fails, the script will abort, and my CL program will see that it failed.

That's pretty cool, isn't it? Replicating this type of behavior with a standard FTP script would be difficult. With a standard FTP script, when the chdir command fails, an error is written to the output log, but the script continues, possibly resulting in deletion of the wrong file or upload of the file to the wrong place! Ouch. Plus, I have to write complex code to analyze the output log to know whether things succeeded or failed. With SFTP, I have far more versatility, and the code to implement it is much simpler.

Using PASE Directly Instead of QShell

The preceding examples use QShell to run my scp or SFTP scripts. Why did I use QShell instead of calling PASE directly with QP2SHELL? Because QShell is easier to use. QShell reports its exit status by sending me the QSH0005 message. PASE doesn't do that.

However, you might not have QShell installed. Strictly speaking, QShell is not required in order to run OpenSSH on the System i. Also, running a PASE command with the QP2SHELL API is much faster than QShell, because QShell spawns separate jobs for each command, whereas QP2SHELL runs the PASE program in your current job.

QP2SHELL does report the exit status back to your program; it just does it a little differently. Instead of sending you a completion message like QShell does, it sets your job's User Return Code attribute. You can check this attribute's value by calling the QUSRJOBI API:

PGM 

    DCL VAR(&RCVVAR)    TYPE(*CHAR) LEN(200)
    DCL VAR(&RCVVARLEN) TYPE(*CHAR) LEN(4)

    CALL PGM(QP2SHELL) PARM('/QOpenSys/usr/bin/sftp' +
                            '-b'                     +
                            '/tmp/script.txt'        )

    CHGVAR VAR(%BIN(&RCVVARLEN)) VALUE(200)   /* SIZE OF &RCVVAR */

    CALL PGM(QUSRJOBI) PARM(&RCVVAR    +
                            &RCVVARLEN +
                            'JOBI0600' +
                            '*'        +
                            ' '        )
    IF (%BIN(&RCVVAR 109 4) *NE 0) DO
       SNDPGMMSG MSGID(CPF9897) MSGTYPE(*ESCAPE) MSGF(QCPFMSG) +
                 MSGDTA('COMMAND FAILED')
    ENDDO

ENDPGM

Or, if you want to get the same functionality as QShell (i.e., have full shell support, have the command running in a background job, etc.) you can use the QP2TERM API to run a PASE shell and pass the command as a parameter:

PGM 

    DCL VAR(&CMD)       TYPE(*CHAR) LEN(3000)
    DCL VAR(&CMDZ)      TYPE(*CHAR) LEN(3001)
    DCL VAR(&NULL)      TYPE(*CHAR) LEN(1)   VALUE(X'00')
    DCL VAR(&RCVVAR)    TYPE(*CHAR) LEN(200)
    DCL VAR(&RCVVARLEN) TYPE(*CHAR) LEN(4)


    CHGVAR VAR(&CMD) VALUE(&SFTP *BCAT '-b /tmp/script.txt +
                           scottk@test.example.com')


    CHGVAR VAR(%BIN(&RCVVARLEN)) VALUE(200)   /* SIZE OF &RCVVAR */
    CHGVAR VAR(&CMDZ) VALUE(&CMD *TCAT &NULL)

    CALL PGM(QP2TERM)  PARM('/QOpenSys/usr/bin/sh' +
                            '-c'                   +
                            &CMDZ                  )

    CALL PGM(QUSRJOBI) PARM(&RCVVAR    +
                            &RCVVARLEN +
                            'JOBI0600' +
                            '*'        +
                            ' '        )
    IF (%BIN(&RCVVAR 109 4) *NE 0) DO
       SNDPGMMSG MSGID(CPF9897) MSGTYPE(*ESCAPE) MSGF(QCPFMSG) +
                 MSGDTA('COMMAND FAILED')
    ENDDO

ENDPGM

Good luck!