Is it 1983 all over again? When the first IBM PCs came out, they had a whopping 16K of memory, and if you had the cash, you could upgrade to an unheard of 64K of RAM.

Jump ahead to 2007 and RPG IV. Did you know that IBM quietly increased the maximum length of a field in RPG IV to 64K (from 32K) during Version 4 of i5/OS? It did. Of course, you remember that in RPG III we had a 256-character limit for field lengths, yikes!

Today however, with one megabyte BLOB or signature fields in SQL-created database tables, XML being received and processed by RPG IV, and HTML being generated, the 64K limit seems, well, small.

But as with RPG III, most programmers in the know were not restricted by the 256-character limitation; instead, they worked around it. In RPG IV we regularly work around that 64K limitation using based fields and user spaces or even memory allocations.

Using memory allocations or user spaces gives us a huge 16-megabyte ceiling (well, huge by 1978 standards). But due to a number of highly optimized and skillfully engineered routines circa 1976-1978, we are pretty much stuck with 16 MBs.

Here's the problem: RPG IV's opcodes and built-in functions today support up to 64K. So even if you use, for example, a user space with 16 megabytes of XML stored in it, the %SUBST, EVAL, MOVE/MOVEL, and other built-in functions and opcodes work with data lengths of up to 64K only.

Try it. Allocate 100K to a pointer, then try to substring more than 64K of the data in the memory location. Or try to copy it. It will give you a "Range or Subscript" error.

The good news is that all the underlying support structures do not know RPG IV has artificial restrictions built into it, and support (at least) 16-megabyte limitations. The other good news is you can easily call them from within RPG IV and use them in lieu of EVAL or MOVE.

There are several instructions on System i that allow you to move megabytes of data around quickly. The two I want you to consider using are the C runtime function and an MI instruction.

memcpy (copy bytes from one memory location to another)

The memcpy function is pretty simple. You give it a target location and a source location, and then tell it the number of bytes you want to copy (up to 16 MBs), and it copies it very quickly. The syntax and prototype for memcpy follows:

        memcpy( %addr(target) :  %addr(source) : 100000);

     D  memcpy         PR              *   extProc('__memcpy')
     D   pTarget                       *   Value  
     D   pSource                       *   Value  
     D   nLength                     10U 0 Value 
 

The first parameter is the target location ("result field" if you will) where the data is copied.

The second parameter is the source location ("Factor 2" if you will) from which the data is being copied.

The third parameter is the number of bytes to copy.

Unlike RPG that will ensure safeguards against memory overruns or access areas of RAM that are not owned by a specific field or program, memcpy does what it is told. So you need to make sure that you know how many bytes you are copying, because if you tell it to copy 100,000 bytes and the field being copied is only 50 bytes long, it will copy those first 50 bytes and the 99,950 bytes that follow it. So if you do not pay attention, you will be a "Certified Windows" programmer.

Using memcpy is not difficult -- just remember to wrap the fields you are using in %ADDR. This takes the address of the field and passes that to the subprocedure. If that feels funny to you, you can use this alternative prototype, which allows you to avoid passing the address and just pass the field like you are used to:

     D memcpy          PR              *   extProc('__memcpy')
     D  target                        1A   OPTIONS(*VARSIZE) 
     D  source                        1A   OPTIONS(*VARSIZE)
     D  length                       10U 0 Value 

This rather unorthodox prototype uses a little trick I discovered in my quest to break the 64K barrier. If you need a parameter that is unlimited in length, you can declare it as any length you want (I choose 1 in this example) and then specify OPTIONS(*VARSIZE). By doing this, the address of the field you specify is passed (same effect as using %ADDR), and the underlying code ignores the prototype definition's parameter lengths. So using this new prototype, you could call that same memcpy example I specified earlier, as follows:

        memcpy(target:  source : 100000);

The memcpy function is a C runtime function, but IBM has mapped it directly into the MI layer (there is an MI instruction that corresponds to memcpy). So performance is nearly as fast as anything IBM could code.

I say "nearly" because there is another, similar option that is even faster. In fact, for large volumes of data, it should be the right choice when performance is an issue, as it is at least twice as fast as memcpy, and that is CPYBYTES.

CPYBYTES is supposed to perform its copying twice as fast as memcpy or other MI instructions because it will not copy pointers that are contained in the data. This gives the optimizer a bit of an advantage, as it does not have to do the pointer authentication it would normally do when handling data that might contain a pointer.

Virtually none of the data in RPG programs contains pointers, so using CPYBYTES instead of memcpy is probably fine in 99 percent of the situations.

Calling CPYBYTES is identical to calling memcpy: You specify the address of the target, followed by the address of the source, followed by the number of bytes to copy. The prototype follows:

     D  cpybytes       PR                  extProc('_CPYBYTES')
     D   pTarget                       *   Value
     D   pSource                       *   Value
     D   nLength                     10U 0 Value

Using either of these interfaces requires that you proactively know the length being copied; you never want to copy a larger field to a shorter field because you will overwrite memory that you should not be overwriting. To work around this shortcoming, use something like the following to calculate the length of the copy.

      /free
           if (%size(target) > %size(source));
              len = %size(source);
           else;
              len = %size(target);
           endif;
           cpybytes(%addr(target) :  %addr(source) : len);
      /end-free

While IBM works to formally break the 16 MB barrier, the geniuses who built the System i platform back in the 1970s have set up the hooks so that we programmers do not have to wait for RPG to catch up with the times. If you think about the System/38, when it was announced it had 256K of main storage, and developers in the lab were working on it with 128K of RAM. At that time, 64K was a lot of storage, and 16 MB was probably more than they thought they would ever need.

Of course, the contemporary teraspace addressing scheme breaks the 16 MB barrier but does not seem to be as efficient. One thing is for certain: the System i will continue to lead the industry with the latest capabilities. Now if we can just get RPG up to speed.

Want to know more? Join us at RPG World 2008 in Las Vegas.