BANK OF AMERICA'S MASTERNET SYSTEM: A CASE STUDY IN RISK ASSESSMENT

By Jeffery G. Szilagyi

Abstract

In 1982, Bank of America initiated the development of the Master Net trust accounting system. After $78 million in losses on the project, the bank announced in 1988 that its trust business was being given to a subsidiary because it could no longer handle the operational requirements. MasterNet quickly became known within the information system industry as a classic case of a system that had fallen far short of expectations. The failure was particularly difficult for Bank of America with its rich history of technological successes.

While information system successes have received substantial trade press and academic coverage, system failures have drawn significantly less attention. Only recently has academia started examining the causes of information system development failure. Clemons has developed a five risk framework for assessing the total risk of a project in an attempt to understand the possible sources of failure. This thesis will examine the MasterNet project using Clemons' five risk framework as a basis of analysis. This analysis will demonstrate that Bank of America did very little to manage the total project risk spanning the five dimensions. Consequently, the project was a likely candidate for failure.

Bank of America's Master Net Story

This section will provide brief coverage of Bank of America's 1970s history and then proceed through the 1980s with a closer examination of the MasterNet story.

A. Tom Clausen's Reign of Neglect

On January 1, 1970, Tom Clausen took the reins as Bank of America's president. At the time, Bank of America- tended a stable and profitable retail business that served two and a half million customers. The decentralized retail business proved very easy to run for the bank's corporate management because an effective set of controls had been established. Thus Clausen's two predecessors -- S. Clark Beise and Rudolph Peterson -- both looked to diversify and nurture new business. As a result, BofA's corporate finance and international 'lending grew during these years to become significant enterprises. The move toward making large loans to large corporations worked against A.P Giannini's philosophy that emphasized the "common man," but the new business seemed appropriate for a company that had been successful in traditional commercial banking areas.

Clausen took what Beise and Peterson started and pushed the expansion of large corporate and international lending. This policy fostered several negative effects. First, the overseas operation grew so quickly that the resultant organization was disorganized and lacked the necessary controls for prudent lending. BofA sent inexperienced managers into new countries who, without an understanding of local conditions, made poor credit decisions. Second, Clausen, in his push for a consistent 10% yearly growth in profits, pressured credit officers into extending lower quality loans.

These two conditions significantly weakened BofA's loan portfolio. Third, the retail operation was ignored and neglected. While other banks expanded and modernized their retail operations, BofA spent few resources on the retail side, instead choosing to commit to the new corporate and international lending efforts.

The economic downturn of the late 1970s and early 1980s strained BofA's weak loan portfolio enough to slice into profits. The first quarter of 1981 brought BofA's announcement of the first decrease in profits in fourteen years. Clausen had taken steps to ensure steady and predictable profit increases through the 1980s, quite often using accounting manipulations to adjust earnings.

These manipulations were stockpiled as weapons against poor quarters, but by the early 1980s, the stockpile was exhausted. In 1981, Sam Armacost took over as president of Bank of America while Clausen moved on to become head of the World Bank. At the time, BofA's loan portfolio was falling apart, the bank was large and bureaucratic at the corporate level, and internal systems were strained under increasing pressure.

Arrnacost understood the need for stronger operational systems and planned a technological spending program in an attempt to push BofA back into the lead. The phrase that he liked to use was "leap-frogging into the 1990s." One area that Armacost emphasized was the trust department because it was mired in a 1965 vintage batch processing system. In 1982, Armacost named Clyde R. Claus executive vice president in charge of the trust department. Claus was given the orders to either fix the outdated department or close the business. BofA's internal staff had attempted to develop a modern system in 1981, but failed after spending $6 million and a full years' time.

The complete trust-business encompassed three broad areas:

1. Corporate Trust --- the bank acts as registrar and transfer agent for stock and bond issues.

2. Employee Benefits --- the bank holds and manages assets of private retirement plans.

3. Personal Trust --- the bank helps manage the money for wealthy individuals and estates.

The employee benefits area is subdivided into master trust and master custodial accounts. Also, the corporate trust and employee benefit. areas are sometimes referred to together as institutional trusts. The size of the accounts can be as small as a few thousand dollars for individuals, up to billions of dollars for pension funds. Corporate trust is a relatively simple activity requiring minimal resources. Employee benefits and personal trust are more complicated because they involve investments in real estate, stocks, bonds, commodities, and other legitimate financial investments.

Master trust clients are mostly corporate pension plans that fall within the stricter regulatory reporting requirements of the Employees Retirement Income Security Act of 1975 and related laws.

Master custodials provide the same services as the master trusts but management is retained by the client. Master trust and custodials provide such trust services as securities lending, portfolio analysis, performance measurement, benefits disbursement, and record keeping. For both types, the operation is difficult because of the complexity involved in tracking each account given the varied needs of each client and because of the complexity and dynamics of the government regulations.

Consequently, the trust department must provide extensive records of transactions, current positions, and statements of explanation for all actions taken. BofA's $38 billion of institutional trust assets was split about in half between corporate trusts and employee benefit trusts.

Claus quickly discovered that the bank's trust department was necessary to maintain solid client relationships. Quite often, a client that maintains a large corporate account keeps a. trust account for convenience. After the system development failure in 1981, Claus was reluctant to turn to his internal staff and proceeded to search for an outside vendor to contract the project.

The trust industry held its annual convention in the fall of 1982. At that time two key data processing executives under Claus met Stephen Katz of. Premier Systems. Katz had recently formed Premier after leaving SEI Corp., an organization that he and Alfred West began. SEI sold a software system based on concepts in Katz's MBA's thesis to about 300 banks through the 1970s, and Katz was entering the same business with Premier. At the convention, Katz met with some BofA officials and began to hammer out a deal for Premier to develop a trust accounting system for BofA.

BofA and Katz then proceeded in 1983 to create a consortium of banks in order to share the development costs and risks while participating in the development. The consortium of banks agreed to advance money to Premier to develop a state of the art trust accounting system. The other banks -- Seattle-First National, United Virginia, and Philadelphia National -- were all using an SEI system at the time. Seattle-First National was a subsidiary of Seafirst Corp. BankAmerica, BofA's parent holding company, acquired Seafirst early in 1983 when Seafirst began to falter with losses in its substantial holdings of energy loans. Thus, Seattle-First National became a subsidiary of Bank-America and a sister bank of BofA, although BofA remained a significant part of the total business.

B. Development of the MasterNet System

Katz and Premier researched the project until March of 1984, when Claus presented Premier's proposal to BofA's management committee. The system, called MasterNet, was to consist of a large trust accounting system, called TrustPlus, plus eight smaller systems that augmented the core system. Each system would be integrated to provide the full complement of trust automation and would be accessible to remote clients on a real-time basis. BofA's ultimate goal was to sell the trust accounting services of the system to small and mid-size banks. MasterNet's initial budget was $20 million and the completion date was set at December 31, 1984.

Katz's design engineers at Premier were to work with BofA's systems engineering and trust department to develop MasterNet. The design and implementation of the computer system followed a reasonably standard plan. A committee of all departments affected by the- new system, including.those from the consortium banks, met monthly to define requirements. BofA's data-processing staff met weekly with Premier's designers to discuss progress and needs. Also, expert users were assigned from all banks to become involved with the design process and provide continuity from design to implementation. Data processing executives felt that the degree of camaraderie and cooperation were exceptional throughout the design process. Also, BofA took significant steps to ease conversion to the new system. A comprehensive training program was implemented that assigned user/trainers to develop courses including videotape, classroom, and hands-on terminals. The material was so well designed that BofA earned a $1.5 million grant from a California state' program that rewards companies for committing resources to employee retraining. Later, these knowledgeable trainers served as code testers and certifiers.

Though the initial December 31, 1984 deadline passed, Claus was not worried because of the progress in the system's development. Meanwhile, BofA was restructuring the organization of the bank under the guidance of Armacost's personally hired management consultant, Ichak Adizes.

Armacost dreamed of a united team of executives supporting his vision of the bank and hired Adizes to orchestrate the effort. One outcome of Adizes'work was the formation of BASE --- BankAmerica Systems Engineering --- a consolidation of the assortment of systems engineering departments at BofA.

BASE was headed by Max Hopper of American Airlines' SABRE fame and would be responsible for the management, development, and application of technology. BASE also spearheaded a $5 billion, five year technological spending program that the bank announced at the same time.

Clausen's neglect of systems during his reign allowed a collection of over sixty networks to develop worldwide at BofA. Although the systems were reliable, the whole situation needed consolidation in order to allow orderly expansion and improvement. Organizing BASE was a first step toward this goal by bringing together the various groups responsible for the bank's domestic and international computer and telecommunications operations. BofA expected that BASE would not only improve operations, but sharpen the bank's competitive edge as well.

As a result of the formation of BASE, Claus lost authority over his portion of the systems engineering department. He also lost the securities clearing operation due to other restructuring activities. Consequently, he found it increasingly difficult to get these two groups and his trust department to work together. Despite this, MasterNet development continued through 1985 and into 1986 until March when Claus thought that the system was ready to be publicly announced. In May, BofA staged a lavish two-day demonstration of the system. Many of the bank's most important corporate clients attended the $75,000 party that touted the "industry's most sophisticated technology" for handling trust accounts. Claus was positive about the presentation and felt that clients were impressed by the system's advanced technology. Claus apparently believed that the system was ready for production use, while other executives felt the show was staged merely to appease anxious customers. During 1986, Claus' department began to prepare for the conversion to MasterNet. The first step called for the movement of the $38 billion worth of. institutional trust customers first. The smaller consumer division accounts would be converted later. The department made repeated attempts in late 1986, but were continually stopped by technical problems. The most glaring problems included poor response time and days-long system crashes. Despite the problems, Claus' department pressed on, putting in long hours in continuous attempts to get the system up and running. The enormity of the conversion task made it particularly difficult. Every single client asset had to be classified into one of the approximately 128 asset types. Personal notes kept by trust employees had to be gathered, reviewed, updated, and loaded into the system, entirely separate from the financial data.

Meanwhile, Bank of America as a whole was having significant problems. 1985 and 1986 brought the bank losses of $337 million and $518 million respectively. Since Armacost had become president, problem loans had been uncovered at an increasing rate. Also, news of the bank's losses and bad loans fueled rumors that BofA was in danger of failing. These rumors caused confusion in California where consumers began to close their accounts, causing a loss of more than $2 billion out of $44 billion of domestic deposits. These problems forced BofA's board of directors to oust Armacost and replace him and his vision of technology with the returning Tom Clausen. In October of 1986, Clausen regained control and immediately began cutting costs and selling assets in an effort to restore profitability.

C. MasterNet's Failure and Postmortem

In early 1987, the system stability had improved enough to consider a serious conversion effort. A date of March 2 was set as the conversion deadline. As the systems engineering staff of sixteen rushed to complete the preparations, half of the group was pulled off of the project. In an effort to offset losses, BofA sold its consumer trust business to Wells Fargo for $100 million. The proceeds could be booked in the first quarter if the deal was closed by March 31, so the half of the staff that was pulled was assigned to transfer the accounts to the SF-1-based Wells Fargo system.

The remaining staff worked continuously up to March 2 and all of the institutional accounts were transferred by the deadline. Six days later, the first of over a dozen Prime disk drives failed. The staff spent a weekend downloading back-up data and the rest of the month dealing with the remaining failures. It turned out that the faults inherent in the Prime disk drives didn't show up in the first months of testing. Eventually, 21 of the 24 disk drives were replaced.

BofA decided in April of 1986 to move BASE to Concord, a suburb about half an hour east of San Francisco, in an attempt to tap the synergies of a large group of technologists and hopefully create better programs faster. It was an opportunity for the bank to replace antiquated technology with cutting edge technology and consolidate sixteen separate groups spread across San Francisco. The new technology center was also intended to attract recent college graduates.

Several key employees quit and morale sank as the pressure and stress of the last few months combined with the move announcement pushed them over the edge. Similar difficulties had arisen in the securities clearing operation based in Los Angeles the previous month. This group was responsible for the tracking and reconciliation of the purchase or sale of stocks or other securities. In March, BofA announced that this group was to be moved to San Francisco. The reaction in San Francisco, similar to that of the data processing group in Los Angeles, was employee dissatisfaction and defection. The bank eventually put the decision on hold for three months, but the damage to morale had been done. In order to fill the gap in Los Angeles, high-paid consultants were brought in to complete the work.

Meanwhile, improvements to MasterNet were being made and many of the problems of the previous year were cleared up. Unfortunately, new problems surfaced-for example, two new types of system halts were discovered in spring. Also, because the processors were fully utilized an additional two units were added to the current three. The hardware and software problems created an operational backlog that delayed processes such as monthly statement generation. Some accounts received monthly statements as much as two months late. Also, the backlog made it difficult for the trust department to maintain current data. In an effort to keep updated, fund managers were forced to get current information on portfolios directly from investment managers instead of through MasterNet. Many began to consider switching to other organizations to handle their trust business.

In July of 1987, BofA announced that it was reserving $23 million related to losses due to MasterNet problems. This reserve covered costs related to the hiring of consultants and accountants needed to clean up the damage, expected loss of fees, compensation claims for securities delivered late, and potential losses due to transaction discrepancies. While BofA publicly assured others that the problems would be solved, it was quietly seeking a buyer for the institutional trust business. In October of 1987, both Claus and Mertes resigned in wake of the problems. The bank immediately assembled a seven member team to handle the MasterNet problem. Michael Simmons, formerly in charge of computers and telecommunications at Fidelity Investments in Boston, was hired in July of 1988 to replace Mertes as head of BASE. The stock market crash of late October exacerbated the problems with the transaction processing and statement generation portion of the system.

Software industry sources said none of the other banks involved in MasterNet had become as deeply enmeshed with MasterNet as BofA. Philadelphia National Bank dropped out of the consortium two years before. United Virginia Bank used MasterNet only for its custodial trust business, which is less complicated than master trust services, and Seattle-First National appeared to have delayed adopting MasterNet in its employee benefits business.

In January of 1988, BofA announced that an additional $35 million would be added to the current $23 million reserve for MasterNet malfunctions. In comparison, BofA earned $60 million in the fourth quarter of 1987. Business began to evaporate as clients pulled their accounts from BofA. The number of accounts dropped from 800 to about 700 and total institutional assets declined to $34 billion from $38 billion. A few days after this first announcement, BofA announced that 95% of its institutional trust services clients would be shifted to BankAmerica's Seattle-First National subsidiary. The remaining 29 clients, representing BofA's largest and most complex accounts, would be given outright to State Street Bank and Trust Co.

In May of 1988, BofA completed the conversion of its trust account processing to a service bureau system running at the SEI data center while Seattle-First National handled day-to-day trust operations for its sister subsidiary. Thus, four years after the start of the project, not only was the system a major failure, but the whole trust business was lost. A total of $80 million was spent on the "stillborn" system and BofA received substantial press coverage of the debacle. The LA. Times said in a front page story that BofA slipped in its "bid to leap into 1990s technology," while ComputerWorld, an information systems trade newspaper, called MasterNet an "$80 Million MIS Disaster." These words were especially harsh given BofA's technological leadership throughout most of its history.

At a time when BofA was struggling to regain profitability, the bad press helped convince the world that BofA really did have significant internal problems. At the same time, BofA was also getting press coverage for its loan losses and attempts to stay afloat as one of the largest banks in the country. MasterNet is now considered a classic example of an information technology failure.

IV. A Risk Assessment of MasterNet

This section will assess the risks that Bank of America was exposed to with the MasterNet project.

First, Kemerer and Sosa's work and Clemons' follow-up work in the area of risk management will be reviewed. Then, the MasterNet project will be analyzed using Clemons' five-point risk assessment structure.

A. Review of Previous Works

In their 1991 paper "Systems Development Risks in Strategic Information Systems," Kemerey and Sosa examine strategic information systems (SISs) that have fallen short of expectations. In contrast, current writings use such. well-known successes as American Hospital Supply's ASAP order entry system as encoura ment for the use of SISs. Kemerer and Sosa chose to examine failed SISs in an attempt to identify the barriers that prevent successful SIS development. Their thesis states that "there exist significant systems development challenger, that present risks or even barriers to some organizations' attempts to use IT strategically, and that executives and systems developers who are.considering an SIS development must plan carefully to avoid these. pitfalls and increase the likelihood of a successful SIS."

Kemerer and Sosa illustrated their position by collecting a broad array of SIS failures and categorizing the types of problems that plagued them. These problems were divided. into the- three phases of the systems development life-cycle model: definition, implementation, and maintenance. Kemerer and Sosa then developed a matrix that corresponds the particular pitfalls with the firm's relative position along three dimensions: monetary resources, technological sophistication, and organizational flexibility The purpose of the matrix is to allow firms to identify their most likely pitfalls according to the characteristics of their organization and then adjust their. plans to balance risks as they desire.

Clemons' 199l paper "Evaluation of Strategic Investments in Information Technology" took the evaluation process one step further and developed seven lessons that can be applied to the evaluation of a firm's investment decision in SISs. Clemons' third lesson --- it is necessary to balance many forms of risk --- lists five basic components of risk which must be managed and traded off depending on the firm's Comfort level. These five risks are classified as financial, technical, project, functional, and systemic.

B. The Five-Risk Analysis

This section is devoted to assessing the MasterNet project along Clemons' five-risk framework.

Five subsections follow, each devoted to a particular risk.

1. Financial Risk

The first of Clemons' five-risks is financial risk. Clemons defines excessive financial risk as unacceptable financial exposure or costs that are out of line with expected benefits. The examination of Master-Net's financial risk will concentrate mainly on Bank of America's weak overall financial condition and the significant downside financial exposure of MasterNet.

During the MasterNet project period, BofA's finances can best be described as weak. In the early 1980s, Bank of America experienced a severe downturn in its financial condition. In 1982, BankAmerica (the parent holding company whose most substantial holding is BofA) posted a $457 million profit on $4.2 billion of revenues. This is a mere 2.2% increase from the year before. At this time, the discovery of bad loans was accelerating and its true exposure to bad credit was just being realized. BankAmerica posted even lower profits in 1983 and 1984, until it posted a $337 million loss in 1985. The organization bottomed out in 1987 with close to a $1 billion loss.

Mentioned earlier, the bank set out on an aggressive $5 billion, five year systems spending plan in 1985. With revenues of $5.3 billion in the same year, Armacost and Hopper planned on spending nearly 20% of revenues on systems maintenance and improvement alone. The $1 billion per year figure is over 25% higher than the spending in 1984 of $780 million. This spending was concentrated in BASE, a group consisting of less than 10% of the bank's employees. Relative to $5 billion, the $20 million spent on MasterNet is a 0.4% portion.

While Armacost committed huge sums to technology, he was also slashing costs in other parts of the bank in an attempt to streamline. Armacost withdrew services, closed about 200 branches since 1981, and jettisoned entire lines of business including the FinanceAmerica subsidiary, a national consumer loan network. Armacost was taking BofA in an entirely new direction, shifting expenses away from the bloated retail, business and corporate bureaucracy into systems spending.

Other banks in the master trust industry were also spending significant money on their systems.

From 1983 to 1987, First Pennsylvania Bank spent $8 million to develop a new master trust system.

The bank had $31 billion in managed master trust and custodial as sets as of mid-1987. Also, Banker's Trust planned on spending $10 million in the late 1980s to update its master trust capabilities. It had $118 billion in master trust and custodial assets as the third largest master trust bank in the country. Banker's had a 1977 vintage system that worked efficiently --- the money was budgeted for incremental improvements, such as the ability to handle more sophisticated securities.

While the basic MasterNet project was budgeted for $20 million, the potential loss due to system failure could be higher due to other, auxiliary factors. These potential factors include:

 •  Completion of Work - A system failure such as MasterNet's forces the bank to perform the intended services in an ad hoc manner, usually via manual labor. For example, in 1983, BofA's investment Securities Division (BISD) installed a new computer system to handle the volume of transactions flooding the operation. The new system was started up while the old system was shut down. Unfortunately, the new system worked improperly, misrecording thousands of transactions.

Teams of auditors were brought in to sort through every transaction by hand. In this case, a $20 million reserve was set up to cover losses and audit expenses.

 •  Inaccurate Transaction Recording -When a bank cannot accurately record transactions, it must resort to "blind settling" -the practice of accepting the transaction terms that the counter-party recorded. If the comparison of BofA's records and the counter-party's records indicate a discrepancy, then the counter-party is normally not paid until verification. When blind settling, the counter-party is automatically paid and the exception is verified manually. BofA had such a huge backlog of exceptions that it would be impossible to retrieve all overpayments due to the extreme delay.

 •  Inaccurate Asset Tracking -The MasterNet system was designed to track the asset position of its accounts. Since-BofA managed $38 billion institutional dollars, the exposure to mistakes was significant. The personal trust business was never transferred to the system, so it was not at risk.

 •  Loss of Managed Assets - Before the institutional trust business was given away, the total managed assets had slid from $38 billion to $34 billion. This represents a 10% decrease and an approximate revenue loss of 10%.

 •  Loss of the Business -Obviously, the giveaway of the institutional business to Seattle-First National and State Street represents a total loss of all future cashflows. BofA was fortunate to sell the personal trust business and obtain compensation.

 •  Loss of Peripheral Business -The press coverage of the MasterNet project damaged BofA's reputation. While the effect on BofA's development of new business was unknown, the event certainly had a negative effect.

 •  Litigation and Fines -In the regulated trust business, banks can be fined by the government for not adhering to the regulations. Litigation costs include lawyers and court costs.

In retrospect, BofA spent an additional $58 million over the original $20 million in direct costs cleaning up the MasterNet mess.

The basic financial benefit of the MasterNet project is not so much the generation of new cashflows, but the continuation of current cashfiows. In 1987, BofA made nearly $100 million on its trust business despite the MasterNet problems. MasterNet was primarily intended to be a "catch-up" project with the highest priority of continuing the old system's operation, albeit more efficiently. A secondary goal of the MasterNet system was to add enough functionality to attract new clients. This new business could consist of additional trust accounting or the sale of the trust accounting services.

Management thought the real value of the project rested in the new technology. If by end of the 1980s BofA had a 1965 vintage system, it is clear that the bank would have little strategic edge in this highly competitive arena.

Because of the risks involved, the bank did attempt to reduce the financial exposure. The consortium was formed primarily to reduce exposure by spreading costs among the group, but the inclusion of Seattle-First National made this diversification less effective. Seattle-First National, like BofA, was a subsidiary of BankAmerica (the parent holding company). Therefore, the financial risk was not really reduced for BankAmerica, because the finances of its BofA and Seattle-First National subsidiaries are consolidated into its own financial statements. Keeping the banks involved with the project was another problem. In 1985, Philadelphia National Bank dropped out, leaving just three banks remaining.

Bank of America found it necessary to undertake the MasterNet project. In assessing BofA financial risk with MasterNet, two items in particular must be considered. First, the overall financial health of the bank was weak and deteriorating. Second, the downside costs of failure were significant given that MasterNet handled billions of dollars of client assets.

2. Technical Risk

Clemons defines the second risk, technical risk, as the possibility that the project just cannot be accomplished because the supporting technology is not available. This section will take a broad view of this definition and include not just the chosen technology, but the process of converting from the old to the new. In this context, technology refers to the basic processing platform, the surrounding telecommunications network, and the necessary software.

MasterNet was an aggressive attempt to provide a full complement of trust automation. Most of this functionality was directed at the master trust and custodial trust business, both of which are significantly more complicated than the corporate trust business. The core trust accounting system turned out to be the most difficult to implement. In fact, six of the eight subsystems were successfully implemented and used within the bank. MasterNet planned functionality was grand: an international network, full customer access, and on-line reporting. Initial designs called for the use of a single eight megabyte, one MIP Prime processor. As database and functionality requirements, grew through the development, the system grew to three sixteen megabyte, eight MIP Prime processors on the conversion date in March of 1987. This represents a 24-times increase in computing power. Two of the processors were used as front-end devices, while a single processor connected to thirteen 600 megabyte disk drives served as the back-end. This configuration required development of various new components: a special data communications channel between CPUS, changes to Prime's PRIMOS operating system, changes to Premier's NEXIJS database, and special RAM disk storage for paging. The development of these components stressed the design groups at BofA, Premier, and Prime. As the operational backlog grew through 1987, two more processors were added to expedite statement generation.

The decision to use Prime hardware was driven by Steven Katz of Premier. BofA succumbed to Katz despite the fact that BofA had been an IBM house since 1955. At the time, BofA had been using nearly 20 IBM or plug-compatible mainframes. During the conversion in March of 1987, 21 of 24 Prime disk drives had to be replaced due to defects that didn't show up until after nine-months of operation.

The MasterNet network connected 35 sites with up to 900 asynchronous, terminals connected through 64 multiplexors. The sites were located primarily in California. The local lines were connected to five high speed data nodes that were spread around the state. The network was designed so that any one node could go down without destroying the integrity of the system. The multiplexors were state of the art and the bank had not previously used the high speed communication protocol chosen. Despite initial problems, the network design eventually stabilized and withstood stress and response time tests.

The software development effort was vast. Premier had over 100 programmers on its staff for MasterNet development. The end result of the project was a system with 3.5 million lines of code, a large amount by any standard. This huge amount of code caused substantial system paging as the number of users increased. Thus, the development team added the@ RAM disk storage to help improve paging rates.

The enormity of the system development task is approached in difficulty only by the conversion process. To simplify the process, BofA hired several conversion professionals to work with the developers. These professionals handled the coordination of data gathering and translation. The pre-loading of static data (e.g., names, addresses, etc.) began years prior to conversion. As mentioned previously, some particularly difficult aspects of the conversion included the gathering, updating, and loading of personal notes and the alignment of existing asset positions with the approximately 130 asset types.

The development task was enormous and expensive. New hardware, software, and communications had to be developed and integrated. Almost all aspects of the project were developed from scratch - a tremendous, challenge.

3. Project Risk

According to Clemons, project risk is the possibility that the firm cannot execute the task. He cites such possibilities. as the project is too large or complex or the skills and expertise of the staff do not match the needs of the project. Project risk has the broadest reach and thus demands detail along many different dimensions. Therefore, this section will be broken into five subsections.

a. Management Philosophy and Vision

Armacost began his presidency with a commitment to improving Bank of America's systems. His phrase "leap-frogging into the 1990s" characterized his desire to improve and his acknowledgment of the bank's deficiencies. Armacost's creation of BASE and the $5 billion technological spending program exemplified his commitment. Armacost also wanted BASE to be the focal point of attention; therefore the announcement was well-publicized. He also put Max Hopper in charge of BASE-Hopper was a high-profile systems manager who had fantastic success in developing American Airlines' SABRE reservation system. Armacost hoped that Hopper could bring his experience and success to BofA. Armacost had a vision for technological success, but because he was a banker with little information system experience, he didn't thoroughly understand the technical issues.

As for other managers, Clyde Claus quickly recognized the trust department's need and immediately set out to develop an aggressive new system. Max Hopper was a terrific visionary in the airline industry, having put in place one of the most successful strategic information systems in business history. Although he understood technology, his lack of in-depth banking knowledge slowed him. It is also not clear whether the position he held reserved any significant power to implement daring strategies. BofA's early success with technology was driven by men like S. Clark Beise and A. R. Zipf. These two men alone were responsible for many of the innovations of the 1950s. They were both hands-on managers who understood technology and its limitations. They both also understood that technology was the key to future competitiveness, and that taking technical risks was necessary for long-term success.

When Tom Clausen took over in 1970, the technological vision became clouded. Under Clausen, innovation came to a virtual stand still and investment in computers and telecommunications was delayed. His return in 1987 signaled a return to old times despite his increased awareness of the need for long-term system spending. Clausen's immediate concern was to get the bank back on its feet again and he took whatever short-term measures were necessary.

b. Consultants

Bank of America made significant use of consultants throughout the MasterNet project. In fact, the basic design of the system was given to Steven Katz of Premier Systems. In the 1950s, Bank of America established with the success of the ERMA project that consultant relationships can be very rewarding. The extraordinary cooperation and vision assured some level of success in that undertaking.

The selection of Katz was based mostly on his past accomplishments. As a partner with Alfred West, he successfully sold software systems to small banks through the 1970s. When Katz split off to form Premier, he took his experience with him and needed an opportunity to develop a new product. Sharing development costs allowed Katz to minimize Premier's start-up risk and guaranteed some, customers for the new product. The relationship gave BofA an experienced hand in the trust business.

There are some indications that the relationships between Premier and BofA were not as cordial as they appeared at the upper levels. In their first meeting, Katz insisted on using Prime Computer while BofA wanted to stick with their more familiar IBM hardware. Katz had a solid relationship with Prime. Katz also asserted that he could deliver a product in less than two years.

BofA representatives argued that that assertion was ridiculous. Some of the bank's personnel also found Katz difficult to deal with, especially on technical subjects. While staff were encouraged to learn aspects of other businesses on the ERMA project, Katz proclaimed "Don't give us the solutions, just tell us the problems." At lower levels of management and staff, it seemed as though relationships were cooperative and supportive.

A major distinction can be drawn between ERMA and MasterNet in terms of consultant location. SRI was located within an hour's drive from BofA headquarters, while Premier was located in Pennsylvania. It was a goal of 1950s BofA management to find a local organization that could handle the work. Another distinction that can be drawn is the motivations of the consultants. SRI was an academic think-tank, interested in performing meaningful research. Premier was a young.company that needed to establish some credible business. A contract with one of the largest banks in the country would give prestige to the Premier name as well as provide it with an infusion of capital.

c. Management Capability and Continuity

In April of 1981, Sam Armacost became, at age forty-one, the youngest man to run the bank since A. P Giannini's early days. Armacost was considered charming and personable, in direct contrast to the aloof Clausen. Armacost had moved quickly through the bank's management hierarchy, never staying at a job for more than two years. He led a charmed career, having never been affected by an economic downturn, and was pushed to the top of BofA. Even he admitted in early 1981 that it might be too early for him to become president.

Throughout the 1982-1988 period, Armacost was constantly, occupied with some controversial events that put pressure on his tenure:

 •  Armacost completed the purchase of Charles Schwab in 1982 and troubled Seafirst Corp. in 1983.

 •   In 1984 the failure of Continental Illinois Corp. forced the Comptroller's office to clamp down on BofA and their problem loans. As government auditors from the Comptroller's office swarmed, Armacost remained defensive and protective of the bank's books, fearing loss of control.

 •  In early 1985, an Armacost attempt to cover up a $95 million mortgage scandal at a Los Angeles branch generated criticism from the press.

 •  In 1985, Armacost began to have problems with the bank's board of directors that escalated through 1986.

 •  Early in 1986, Sanford Weill, a successful securities businessman who built the brokerage arm of Shearson Lehman Brothers, offered to become the president of BofA and inject $1 billion of capital into the bank. As a defense against the friendly bid, Armacost considered a merger with First Interstate Bancorp. Weill subsequently withdrew his offer.

Meetings and sessions with Ichak Adizes consumed significant amounts of Armacost's and his executives' time. From late 1982 through early 1985, Adizes spent significant time with all of BofA's executives. Adizes' meetings no doubt helped communication among the top executives, but typically occupied up to 20% of their time. The process eventually hurt morale and drew insults from the lower ranks of managers.

The unsettled picture at the top of BofA continued through 1986. Armacost was dismissed as president and replaced with Tom Clausen. Clausen's return continued the turmoil. Tom Cooper, appointed Chief Operating Officer by Armacost in February of 1986, did not get along with Clausen. Cooper was known as a dedicated cost-cutter and eschewed the luxuries of his office. Clausen, on the other hand, enjoyed the personal luxuries of his position, such as use of a limousine and company plane. As an executive, Clausen often consolidated his authority, and Cooper resigned in May of 1987 as his power was slowly being drained away.

Closer to the trust business, Clausen had lost a portion of his authority with the formation of BASE. In late 1986, Hopper left BASE to return to American Airlines. In late 1987, Armacost launched an internal investigation of the MasterNet debacle. The report, by BofA senior vice president and director of litigation Winslow Christian, was not intended to lay blame on any individual, but detail exactly what went wrong with the project's development. Shortly after the delivery of the report, both Clausen and Mertes resigned in October of 1987.

Through the MasterNet period of 1982-88, BofA's management underwent the stress of a variety of events both internal and external to the bank. Upper management's preoccupation with these events and the shuffling of executives resulted in a loss of continuity and control for the whole bank. Thus MasterNet, like many other projects and organizations in the bank, found itself without the guidance and direction of a strong upper management team.

d. Organizational Factors

Bank of America was concerned that it would be difficult to change a system that had been in place for twenty years. First, there are the problems associated with any move to a new system.

Second, there are the problems related to the use of such drastically modern technology. The old system was based on batch processing, while the new system was real-time on-line reporting. To combat these problems, BOEA created, a comprehensive training program that eventually won grants from California's retraining program.

During this period, financial problems at the bank caused management to evaluate and make changes to the inner structure of the organization. Thus, the 1981-1988 period was a time of significant change and turmoil. The origination of BASE consolidated the bank system engineering departments, but removed some locality of authority. The movement of the BASE employees also put stress on the employees working on MasterNet. The stress of system problems had been growing for years, and the pressure to deliver to increasingly dissatisfied customers grew. Also, the poorly timed sale of the consumer trust department did not enhance employee's view of upper managements understanding of the MasterNet problem.

e. Other System Projects

As a large bank, Bank of America participated in many systems development projects. Directly related to the trust business and previously mentioned, the bank attempted to develop a new trust system in 1981 only to lose $6 million and a year's time. Other examples not related to the trust business exemplify the difficulty of systems development in the financial arena:

 •  The RISD failure detailed in the financial risk section put $20 million at risk in an area that performs functions similar to the trust business. Fortunately for BofA, the eventual losses turned out to be minor.

 •  In 1983, Hopper wanted to implement a system used by the airlines called TPF, or Transaction Processing Facility. TPF was a high-speed, expensive system built for massive volumes. Its high power was not necessary for 1983 volumes, but would be needed if optimistic projections were correct. Unfortunately, the projections were terribly inflated, and the investment in TPF was not immediately necessary. Now, almost nine years later, the investment seems to be paying dividends but only as the rest of BOLA@ systems have caught up. TPF currently handles BofA's bank card authorization inquiries and serves as a switching facility for the ATM system.

 •  In late 1987, BofA dropped an effort to integrate a software package called TradePro into its securities clearing operation. The bank spent $5 million on the package and shipped personnel in from across the country. BASE was responsible for the development of the software.

 •  Other failures in the 1984-85 period include a system that incorrectly sent statements to corporate clients and the collapse of a foreign exchange program that forced traders to use pencil and paper.

 •  Despite a late start into the area, BofA initiated an intense effort to introduce ATMs into its California market in 1981. After spending over $100 million in a single year, the system was essentially complete and working correctly.

 •  In 1984, BofA went on line with their point of sale system named Interlink. A successful real-time debit card service, BofA developed the product with three other west coast banks.

 •  BofA developed a retail Customer Information File (CIF) system in 1988. The system integrates multiple databases to allow transparent access to account data. This system allows the cross-selling of products and services and the development of new products. Renamed Customer On-Line Information Network (COIN), the system won an award for systems excellence from the Society for Information Management.

 •  By 1989, BofA had seen success in its attempt to consolidate networks. The California Data Network (CON) now connects the three major data centers in San Francisco, Los Angeles, and Concord with high-speed digital TI communications lines. The new utility should've reduced costs by $5-10 million annually and provided faster response times.

While BofA has had its share of failed projects, it has also had some significant successes.

Clearly, BofA did have some systems expertise, but either did not appreciate the skills of the successful project managers or didn't have the organizational controls necessary to place these valuable people in strategic positions. Either way, BofA's past record provides substance to the charge of organizational weakness.

4. Functional Risk

Clemons defines functional risk as the possibility that the completed project either does not do what the user wants or users needs have changed enough to make the system useless. This section will detail the difficulty of operating a trust accounting business and how well MasterNet met this functionality.

The administration and accounting of a trust business is a difficult business due to the wide variety of assets, client needs, and government regulations. The master and custodial trust business in particular are tightly regulated and require extensive reporting. These various demands of the business forced banks to upgrade both hardware and software systems continually.

The Employees Retirement Income Security Act (ERISA) of 1974 initiated the growth of the master trust industry. This act required the detailed reporting of trust activities for pension funds.

Corporations looked to banks to handle this difficult responsibility. Additional changes and new regulations -- such as the Tax Equity and Fiscal Responsibility Act of 1982 (TEFRA) - made adherence to the law even more difficult. A Trust Company Bank Of Atlanta vice president stated "Every time Congress gets together, they make a change. And it's difficult to keep up with all the changing regulations." To deal with the complexity, banks established programs to stay up-to-date with master trust technology, such as State Street's continuing professional education for master trust personnel begun in 1987. Custodial trusts have similar, but less stringent requirements.

A good example of the growth of sophisticated financial instruments is the explosion of the Mortgage-backed security market. These securities have floating interest rates principal that can be prepayed at any time, and monthly payments (as opposed to more typical semi-annual). These properties create variable monthly payments that are difficult to track and different from normal corporate bonds. In fact, there are several instances of funds having mortgage-backed problems with various trust accounting banks. For example, State Street lost an $8.6 billion Teamsters account because of mortgage-backed security problems. Other securities that have gained favor with clients are options, futures, and international securities, all of which are more complicated than more traditional stocks and bonds.

The high costs of maintaining sophisticated computer systems necessary to handle trust business scared many major players out of the market. Through the eighties, major banks including J. P. Morgan, Wells Fargo, First National Bank of Chicago, and Crocker National all left some portion of the business. The business has also become highly competitive. This competition has sliced into profits and made the margin of comfort slimmer.

MasterNet's functionality was additionally complex because it had to satisfy so many different groups. These groups included the four trust departments of the consortium banks and Premier personnel. One executive felt that the diversity of interests caused developers to accommodate all needs instead of limiting functionality. Also, there is evidence that system functionality was not sufficiently tested. Both the bank's internal auditor and Ernst & Whinney, its outside auditor, felt that a conversion shouldn't take place because MasterNet was not adequately tested. An anonymous letter from a staff member sent to the trust department's internal newsletter pointed out that the system was not ready for use.

The trust accounting business is a complex, technology-driven arena. Continual systems investment is necessary to keep up-to-date with current government regulation and financial innovation. While BofA recognized the need for this investment, many environmental factors made the systems functionality quite complicated. The intrinsic complexity of the business was compounded by increasingly sophisticated financial instruments and the broad needs of many groups. In an attempt to satisfy everyone, the project became bogged down in massive amounts of code that contributed to its downfall.

5. Systemic Risk

Systemic risk is different in that it implies successful implementation. Clemons says that systemic risk is the possibility that the system has such a large impact that it alters the environment and all assumptions about costs and benefits. By definition, this type of risk is entirely unpredictable.

Therefore, very little can be done to combat it. For argument's sake, the design of MasterNet was unlikely to dominate the business. Features such as customer access were already implemented in other banks' trust systems, and some banks and companies were already selling these services. As for MasterNet's grand functionality, clients stress that accuracy and timeliness of reports are the most important factor in quality trust service, and that broad functionality is less critical. Also, other banks in the business operated systems approaching the sophistication of MasterNet and had been since the technology was available.

C. Summary

Through this five-risk framework, it is apparent that BofA was not prepared to take on a project of such magnitude. Of the four controllable risks, BofA failed to sufficiently assess and manage all four:

 •  Financially, the bank was weak and getting weaker.

 •  The technical aspects presented a tremendous challenge even for the most sophisticated of organizations.

 •   The management and organization were not well suited at the time to handle a project of such magnitude.

 •  Functionally complex from the beginning, the project became more difficult due to greater user demands and market changes.

BofA made two critical mistakes in its handling of MasterNet. First, it did not realize the difficulty and scope of Masternet and sufficiently assess its risks when it was proposed. Second, after the project rolled out of control, the bank failed to take the necessary steps to bring it under control.

Only when MasterNet performed so poorly that it merited a $23 million loss coverage did management begin to take a closer look. The MasterNet debacle not only depicts poor project management unto itself, but is representative of the general state of affairs within the bank as a whole.

V. Lessons Learned

The MasterNet case makes obvious a lot of basic system management lessons. This section will briefly mention some of these lessons.

One of the most interesting contrasts between MasterNet and ERMA is the existence of visionaries. ERMA had two high-level managers, Beise and Zipf, who not only understood the technology, but were driven to include it in the operations of the bank. These men understood that technology. was necessary for future competitiveness, and that the investment had to be made immediately. MasterNet had Armacost, who did have a vision of technology, but was .both not well-versed technically and too busy with other problems to focus on the project. Clausen was motivated but did not have power within the organization to provide strong guidance.

The idea behind the visionary is more than strong guidance. The organization that sees upper management concerned and excited about its project will be motivated to perform to high standards.

MasterNet staff members were generally shown little concern for their welfare being split and moved.

Another important concept that becomes painfully obvious with MasterNet is continual investment. Through the 1970s, BofA neglected its systems development. It then expected to jump back in and immediately catch up with its competitors. This cannot be done for a variety of reasons.

First, the resulting size and complexity of the new project far outreaches anything done before.

Second, the necessary skills for success in the organization just aren't present. Lastly, a whole shift in the organization's attitude is necessary for the change to be successful. BofA attempted this huge jump without sufficiently preparing itself - it takes more than money to breed success.

The MasterNet project also violated some basic rules of large systems development. Modular design, limited functionality, and full-load testing were all disregarded. The project also failed miserably in converting smoothly. The staff should have recognized that the system could not keep up before the conversion attempt was made. BofA also failed to keep a semblance of continuity in the project's management. Many of the errors BofA made with respect to MasterNet are not specifically related to information systems but apply to all project management. The additional technical complexity of MasterNet only made the project management that much more complicated.