HOW INCONVENIENCED WOULD YOU be if you couldn't access your checking or savings accounts for a week? How irritated would you be if you went to the ATM at your bank for cash and found it wasn't operating today — just as it hadn't been yesterday or the day before?

Nightmarish as that scenario might be for you and me, it's even more horrific from a banker's point of view. Thousands of customers who can't get at their money and who've promised themselves they're going to switch banks the minute they do get access is the doomsday scenario Granite Bank of Keene, New Hampshire, found itself confronting after reviewing its disaster recovery plan in the late summer of 2000. Although the bank has never had a serious computer or business service outage, a contingency planning review brought home already mounting concern that a system failure could lead to an unacceptably lengthy recovery period. The bank had long since contracted with a disaster-recovery service to which it could have moved its operations in an emergency. But systems had become so large and interdependent that even a hot site wasn't enough insurance.

"The plan showed us it would take several days to get our major systems up at a remote site," notes Don LaFontaine, Granite Bank's administrative vice president of systems, "and restoring online and ATM operations would have taken even longer."

Granite Bank is the largest state-chartered independent bank in New Hampshire and has assets of more than $1 billion. It serves customers from 19 offices and 44 ATMs across New Hampshire and also offers online banking services via the Internet. With this volume of business, any significant system downtime could have a major financial impact.

As it reviewed downtime costs, Granite also found its systems were experiencing an estimated total of 214 hours of downtime in an average year, 192 hours of which were due to planned system upgrade and maintenance activities. Although this was only one percent of total system time, bank management realized this was unacceptable in a market where customers expect to be able to access and manage their funds on a 24-hour basis.

Finding a Reliable Solution

Granite didn't have to look very far for proposed solutions. Granite Bank uses Jack Henry Associates' (JHA's) Silverlake System banking application on their iSeries systems, and JHA already had an alliance with Lakeview Technology, Inc., under which both companies were working to integrate Silverlake System with Lakeview's MIMIX FastPath Services to cluster-enable the JHA application. Granite reasoned that working with JHA and Lakeview could provide Granite with a solution that wouldn't require any significant software changes. At the same time, Granite's disk storage supplier advocated a massive hard-drive upgrade at Granite's headquarters as an alternative solution.

After researching high availability and looking at other available products in the iSeries market, LaFontaine and Systems Officer David Perley decided in favor of MIMIX. The existing integration with JHA applications was a plus, a cost analysis showed that a storage hardware upgrade would be a significantly more expensive option, and other software solutions would be more costly to integrate with Silverlake than MIMIX.

LaFontaine and Perley formulated a plan that involved upgrading iSeries machines at the Keene headquarters, setting up a remote hot site at the bank's Milford, New Hampshire, office, and using MIMIX Cluster Server to operate the network. The plan went to Granite Bank's standing technology committee, a group of 13 representatives from all segments of the bank that, among other duties, reviews major technology capital expenditures. By showing the financial costs of lost business and the expense of recovering from just one downtime incident, LaFontaine was able to easily justify the much smaller cost of implementing a two-site replication and clustering project using MIMIX. The technology committee recommended the purchase to the bank's executive committee of the president and three vice presidents, which also approved funding for a complete implementation.

Unexpected Events Interfere with the Best Plans

Granite upgraded its iSeries systems in Keene and Milford to model 820s running V5R1 in early 2001 and installed the Lakeview software in August 2001. The actual clustering implementation fell victim to a hard-to-pinpoint communications problem between Keene and Milford. The implementation team traced the problem to a single communications software flag. Unfortunately, a bachelor party put the communications line company technician who was supposedly on call to troubleshoot such problems out of commission on the critical day. That delay caused the first cut at implementation to be rescheduled for September 12, 2001 — an event that was in turn postponed following the World Trade Center disaster.

Despite these setbacks, Granite implemented clustering between its sites in November 2001, with JHA and Lakeview personnel handling most of the work. Although three days of training were required for operations personnel, no other training was necessary, and the change was totally transparent to the bank's end users and customers. After a weekend of switchover testing by 15 bank employees, the system was ready to fly, despite some minor initial glitches.

"The first trial switchover took just 13 minutes," LaFontaine recalls. "We had thought we'd be able to do all switchovers that quickly, but clustering issues extended some trials to more than two hours." These were eventually resolved. "We also expected we'd be able to do database reorganizations with no downtime using MIMIX," LaFontaine remembers, "but it turned out that only applied to IBM databases, not the databases supporting Silverlake. Those reorganizations currently require two hours a month."

Although made lengthy by circumstances, Granite bank was happy with the implementation. "I was very impressed by how easy it was to install," LaFontaine remarks. "Ongoing maintenance is also simple. We just leave a single screen running in two or three locations, and from there we can monitor everything on both the primary and backup systems. JHA and Lakeview were very pleasant to work with."

A Feeling of Comfort

Fortunately, aside from testing and practice, Granite still has yet to use its new high-availability and clustering solution to react to a system outage. "However, it's a great comfort to have the totally redundant system in Milford," LaFontaine admits. And the bank is able to use MIMIX to schedule switchovers to give itself larger backup windows for both the primary and secondary systems. Although the initial estimate was that Granite Bank would reduce its total annual downtime to just 40 hours for necessary maintenance, in practice MIMIX Cluster Server lets the bank switch users in even less time than projected, and the bank expects its actual downtime in 2002 will be less than 40 hours.

Implementing MIMIX also provided some direct savings. By setting up a remote system at one of its own facilities, Granite was able to cancel its hot-site contract, saving $59,000 a year.

Overall, Granite Bank is satisfied with MIMIX Cluster Server. "There were no negative effects, and we've been able to implement small tweaks and improvements gradually," LaFontaine notes. "System upgrades and maintenance tasks are no longer a problem to schedule and carry out." And now, Granite's customers can access their accounts any time, without having to worry about observing banker's hours.