Balancing the Workload in a Batch Environment with Data Queues

Article ID: 53698

Posted December 14th, 2006

By:

One of the challenges of my job is to balance the workload of the nightly batch job so that the core business tasks always have the system resources that they need to accomplish their functions. Those tasks are not usually limited by the processor. Instead, they're generally limited by the available memory and I/O throughput speed.

The business logic for my nightly batch job is a long-running process that executes complex operations over data to produce reports mainly used by our corporate offices. Typically, these are decision-support applications in which large amounts of data must be analyzed to create one- or two-page reports.

The corporate executives demand that their reports be in an elegant format. They accept only PDF and HTML documents with corporate logos, or Excel spreadsheets received via e-mail.

Although improved performance is an important objective, another objective equally important to me is the maintainability of the programs. For that reason, I separate the data processing modules from the report-formatting modules. Fortunately, a tool that we've had for ages (an ancient toolset!) is a great help in doing this: Data queues!

Using Data Queues

Data queues were born with i5/OS. Data queues make transferring data asynchronously between processes possible. One or more jobs send information to a data queue. This information is stored in the form of data queue entries. Those jobs can be called "producers." Another job (the "consumer") receives the data queue entries from the data queue (in a FIFO, LIFO, or keyed sequence) as the producers send them. Both the producers and the consumers must agree on the format of the data inside each entry in the data queue.

To create data queues, you use the CRTDTAQ command. A few APIs are available to manipulate the data queue object:

QCLRDTAQ - Clear Data Queue
QSNDDTAQ - Send to Data Queue
QRCVDTAQ - Receive Data Queue

From my point of view, there's an interesting relationship between data queues and output queues. When you create an output queue, a bunch of additional parameters exist, including one named DTAQ. If you use this parameter, each time a spool file with ready status is put in the output queue, i5/OS automatically generates an entry in the associated data queue with information about the spool file. This entry contains the complete identification of the spooled file.

With the DTAQ parameter set, i5/OS calls the QSNDDTAQ API automatically when a spooled file is ready. This way, my program can wait for the data queue entries to appear. When the long-running batch job is complete, I can retrieve the spooled file, convert it to PDF or HTML, and send it via e-mail. Essentially, I've created a relationship between my batch job's output queue and my data queue.

Information about the format for the data queue entry is at the following link:
http://publib.boulder.ibm.com/infocenter/iseries/v5r4/index.jsp?topic=/rzalu/rzaludataq.htm

My approach requires the nightly batch process to carry out the following steps:

Create a data queue and associate it with an output queue. Later, the batch job establishes that this queue is its default output queue.
```
CRTDTAQ DTAQ(MYLIB/NIGHTDTAQ) MAXLEN(128) SEQ(*FIFO)
CRTOUTQ OUTQ(MYLIB/NIGHTOUTQ) DTAQ(MYLIB/NIGTHDTAQ)
CHGJOB OUTQ(MYLIB/NIGHTOUTQ)
```
Start the consumer job by running the SBMJOB command. This job (which is CPU bound) runs in a separate subsystem with low run priority and other performance tunings (e.g., adjusted storage pool settings). This way, I give more system resources to jobs doing business-critical work in the batch subsystem and fewer system resources to jobs doing less important work. (Naturally, what is more important or less important depends on the point of view: For the corporate folks, the most important thing is to have all the reports at 9:00 a.m.). The key point is that jobs in both subsystems live and work together in a "peaceful coexistence."

Processing the Data Queue entries

The steps that the consumer job carries out are as follows:

Retrieve the data queue entries using the QRCVDTAQ API.
Determine who the owner of the report is.
Call the module that transforms the spool file into a PDF or HTML document.
E-mail the document to the receiver.

I'm a Cobol programmer, so my sample program is written in Cobol. However, you could write it in any language, including RPG, C, C++, and even CL.

Using the QRCVDTAQ API

In my sample program, the required parameter group for the QRCVDTAQ API is declared in the WS-QRCVDTAQ-PARMS data structure. The Wait Time parameter is set to -1, meaning the API waits indefinitely for a new data queue entry to arrive before returning control to my program. When a special record containing the value "Exit" is received, the program ends in a controlled way.

The relationship between the spool file name and the e-mail address of the receiver is obtained from a database table. The module that reads the database file is not included in the sample program, because that task is trivial, so I leave it as an exercise for the reader. However, you should note that this solution assumes that each spool file has a unique, nonrepeatable name. In ILE Cobol, this uniqueness is accomplished by associating the name of the printer file with the FORMATFILE device:

SELECT file-name       ASSIGN TO FORMATFILE-filey

In the preceding example, filey is an externally described printer file (with object attribute PRTF) created before compiling the Cobol program. The associated DDS contains the description of the printer file records (perhaps a heading record, a detail record, and a footer record) and all its output fields. At compile time, you can set several attributes, such as page size, fonts, and so forth. At runtime, the spool file name created by the program is filey.

Some old-style Cobol programs use the PRINTER device and associate it with some IBM-provided standard printer files, such us QPRINT or QLIST:

SELECT file-name       ASSIGN TO PRINTER-qprint

But this way, the spooled file name is not unique, and the solution I propose here won't work.

In my sample code, the task of transforming the spool file to a PDF or HTML document is performed by the SNDSPLMAIL tool. Please note that's not a native command. It's a trademarked tool from Gumbo Software Inc. You could use this or any other such tool from the several ISVs in the market, or perhaps you could develop your own.