Overcoming the Limits of Single-Level Storage in Cobol

By:

I don't know about you, but I find that visualizing extremely large (or extremely small) numbers is difficult. For example, when I read in the news that a financial company has sought a $40 billion bridge loan from the Federal Reserve as a lifeline, I ask myself, "$40 billion? How much is that?" Similarly, when I read that computer scientists think in terms of terabytes (one trillion bytes), I think, "Yikes! I'm better off concentrating on business applications!" However, terabytes are useful, even in the business world. They're no longer solely the province of computer nerds.

For example, suppose that one program needs to create an information structure for containing all the records of a database file and this structure must be passed as a parameter to a subprocedure. Assume further that, after accessing it, the subprocedure destroys this data structure. In this example, the database file contains 20,000 records, and each record is 1,000 bytes long.

In memory, I define a data structure that consists of a set of elements, each of which corresponds to a record in the database file. Each element consists of several fields that correspond to the fields in the file's record format. Therefore, the program needs to allocate 20 MB of storage in which to store that structure.

As you might expect, my first attempt was to create a one-dimensional array of 20,000 elements. But, that failed with a compile error because the length of this data structure is greater than the 16 MB limit imposed by the ILE Cobol compiler. Indeed, this 16 MB limit is applicable to any space in the single-level storage.

To overcome the 16 MB limit, ILE provides a feature called teraspace, which lets you allocate up to 1 terabyte of private storage for a job. By "private," I mean that this area is normally accessible only by the job that allocated it, and by "temporary" I mean that it's automatically freed up if the job ends.

A teraspace-enabled program is one that can access a teraspace address through a pointer. Starting with V4R4, ILE Cobol can generate teraspace-compatible code. Because this behavior is the default, it is unnecessary to use any special parameters at compile time. Several C/C++ functions are available to allocate memory at a teraspace address: _C_TS_malloc(), _C_TS_malloc64(), _C_TS_calloc(), and _C_TS_realloc(). They're the teraspace-enabled versions of the dynamic memory management APIs malloc(), calloc(), and realloc(), respectively. ILE also provides a teraspace-enabled version of the free() API called _C_TS_free().

In my previous articles titled "Cobol and Dynamic Memory Allocation" (January 24, 2008, article ID 56196 at SystemiNetwork.com) and "Cobol and Dynamic Memory Allocation, Part 2" (April 24, 2008, article ID 56601), I provide detail about how you can call the dynamic memory management APIs from a Cobol program. The rules detailed there are also applicable to their teraspace-enabled equivalents. As such, this article only recaps the fundamentals that those articles discuss.

Dynamic memory-management APIs are part of the ILE C/C++ runtime library. For that reason, when you create the Cobol object program--using the CRTBNDCBL or the CRTPGM commands--you must indicate the QC2LE binding directory in the BNDDIR parameter of that commands. This binding directory contains the list of service programs that make up the ILE C/C++ runtime environment.

Cobol programs must call these industry-standard APIs, passing the required parameters by value, indirectly. In this way, the value of the data object is copied to a temporary location, and the address of this copy is placed into the parameter list.

For our sample program, we need a large address space on which to build an information structure. This information structure must represent all the records in a large database file. In addition, this data structure should be available for use by a subprocedure. For these reasons, we decide to build a list structure. This data structure contains a header segment and a list segment. The header segment contains information such as the following:

Where the list begins--the offset of the list segment
The number of entries--because of application requirements, the value of this field is equal to the number of records in the database file
The length of each entry--for the same reasons, the value of this field is equal to the record length in the database file

The list segment is made up of a series of entries, where each entry is the image of a database record.

The Cobol program example (downloadable via the link at the end of this article) defines the header segment in the LINKAGE SECTION in this way:

01   header-segment.
     03 number-of-entries    PIC 9(9) BINARY.
     03 size-each-entry      PIC 9(9) BINARY.
     03 offset-list-data     PIC 9(9) BINARY.

This segment occupies 12 bytes. Assuming that the database file contains 20,000 records and the record length is 1,000 bytes, we'll need 20,000 x 1,000 + 12 bytes to hold the list structure.

The program obtains the teraspace storage, calling the _C_TS_malloc() function in this way:

      
     COMPUTE required-storage = 20000 * 1000 + 12.
     SET header-segment-ptr TO NULL.

     CALL "_C_TS_malloc"
          USING BY VALUE required-storage
          RETURNING header-segment-ptr
     END-CALL.

     IF header-segment-ptr = NULL
          DISPLAY "_C_TS_malloc() function failed"
     END-IF.

It builds the data structure using the following approach:

Set the address of the header segment to the teraspace-pointer returned by the _C_TS_ malloc() function.
Initialize the three fields of the header segment. The offset of the data section (i.e., the offset-list-data field) is initialized with the length of the header segment.
Add the offset of the data section to the space pointer, to get the start position of the list segment.
In a loop, each database record is put in consecutive portions of the memory, adding the size of each entry (i.e., the size-each-entry field) to the space pointer to get the starting position of the next entry.

Mutatis mutandis, a similar approach is to use the subprocedure to read the data structure. The major difference is that the subprocedure receives the teraspace pointer as a parameter from the main program. It frees the teraspace storage by using the _C_TS_free() function, after the information has been processed:

 CALL "_C_TS_free"
      USING BY VALUE header-segment-ptr
 END-CALL.

As you surely have noticed, the techniques I present here are nothing new. Many list APIs return information in a data structure composed of a header segment and a list segment. In addition, the header segment in the list APIs contains the same fields used in my program, plus much more additional information.

We could have used a linked list technique, as taught by Donald Knuth in his seminal work The Art of Computer Programming. However, in this case, each list entry should have an additional pointer field, containing the address of the next list entry. The pointer of the last entry should have the null value, indicating the end of the list.

For the sake of brevity, the sample program doesn't read a database file. It simply writes a numeric 6-byte field and fills the rest of the entry list with spaces. Similarly, the subprocedure displays the value of the first and the last list entries to ensure that it correctly accessed the data structure.

More information:

You can read about teraspace concepts in Chapter 4 of the iSeries ILE Concepts book: "Teraspace and single-level store." You can find the book at:
http://publib.boulder.ibm.com/infocenter/iseries/v5r3/topic/books/sc415606.pdf [2]

You can download this article's sample program at:
http://www.pentontech.com/IBMContent/Documents/article/57289_685_CobolTeraspace.zip [3]