Running Oracle Exadata, Brown Brothers Harriman delivers information at high speed, transforming global custody and portfolio accounting.
by Fred Sandsmark, August 2013
On paper, the job of a custody accountant hasn’t changed much since Chris Remondi joined Brown Brothers Harriman (BBH) in 1994. Then as now, accountants at custodian banks such as BBH—which hold assets and perform transactions for large investment firms—tallied the daily closing price of all shares within a fund or portfolio to calculate its net asset value (NAV). For US-based funds, NAV calculations must be completed by 6:05 p.m. Eastern Time—just a hair over two hours after the markets close—to meet publishing deadlines.
But that job description doesn’t tell the whole story. “When I started here, you probably could have calculated the NAV of a portfolio manually if you really had to,” says Remondi, global head of information technology and partner at BBH. “But today, you would have absolutely zero ability to do so.” Today, a single fund portfolio can hold thousands of different securities and other financial instruments that accrue interest and dividends in complex ways. Currency changes, shareholder activity, and other factors must be accounted for within the NAV deadline.
So calculating NAVs today requires powerful IT. Since the early 2000s, accountants at BBH—one of the world’s largest custodian banks—have successfully used a specialized accounting system, built on Oracle Database and Oracle’s hardware, for their fund accounting platform. They hit their deadlines with that system, but in recent years doing so grew increasingly difficult because of the complexity and size of the task. “The custody-related accounting business at BBH has been growing north of 30 percent per year, every year,” says Remondi. “We needed to make that application run more efficiently.”
So in late 2011—after some encouraging early tests—Remondi and his partners decided to migrate BBH’s custodian accounting platform to Oracle Exadata. Business and IT staffs collaborated on a thorough, aggressive testing and transition plan that culminated in a flawless cutover in fall 2012. Moving to Oracle Exadata has slashed NAV calculation time, improved service to BBH’s demanding customers, boosted system availability and IT productivity, and bolstered business continuity. “We’ve been able to transform our work with Oracle Exadata,” Remondi says.
Remondi uses the word transform consciously. In fact, he guides his department with a three-word mantra—run, deliver, transform—in that order. Remondi says IT must run its operations every single day without defect. It must deliver exactly what its clients need, such as new functionality or data-driven business intelligence. And it needs to constantly transform itself and its tools to meet changing needs.
Transformation is natural at BBH, which has prospered through innovation for nearly 200 years. The company began as a textile trading company in 1818; expanded into trade financing; and moved into foreign currency, global securities, and financial services. Today, BBH is the oldest and largest private bank in the United States and a leading provider of global custody services. Throughout its storied history, BBH leadership has leveraged new technologies—from steamships to telegraphs to transatlantic cables to the internet—to enable the business.
Custodian banking, which BBH entered in the 1920s and which now accounts for more than 70 percent of the bank’s business, is a complicated line of work that’s nearly invisible to people outside the industry. As a custodian bank, BBH serves providers of collective investment vehicles (such as mutual funds and pension plans). It holds all their assets in safekeeping, executes trades, settles transactions, and provides related services such as accounting. (Some funds hire one company as custodian and a different company for fund accounting and/or fund administration services.) BBH’s move into the custody business coincided with the growing importance of IT in financial services, Remondi notes. “Years ago IT was probably considered a support function,” he says. “But now, our value lies in how effectively we can process data and transactions—particularly in the global custody space. Today, anyone looking at BBH would say IT is our business.”
About 500 BBH accountants worldwide use the custodian accounting platform; many of the leading mutual fund providers in the world also tap into it. The platform processes data from about 1 million transactions a day and is used to calculate the value of portfolios totaling more than US$1trillion dollars in assets. Speed and accuracy are essential, explains Jonathan Mack, BBH senior vice president responsible for client service accounting. “Delivering NAVs is something that we just need to get right, and we need to get done on time—even when there are big market events,” Mack says.
As custodian banking has globalized—60 percent of BBH’s business comes from outside the United States—quickly calculating NAVs has become even more challenging and important. Global markets open and close around the clock, and funds in different geographies hold global securities and value those securities using different snapshots.
Transactions that had taken us hours to produce were happening in minutes, and reports that we ran overnight were coming back in 10 or 15 minutes. It was just massive improvement in performance.
Globalization also brings volatility. When the unexpected occurs—such as when the stock market in Cyprus was suddenly shut down in March 2013—NAV calculation becomes even more complicated. “Our window gets tightened with every single market event,” Mack says.
To see whether an Oracle engineered system could help pry open that tightening window, BBH accounting and IT staffs sent sample cases and data to the Global Oracle Solution Center in Santa Clara, California, in fall 2011. “We got the results back, and we thought they weren’t correct,” Remondi recalls after tests were run on Oracle Exadata. “Transactions that had taken us hours to produce were happening in minutes, and reports that we ran overnight were coming back in 10 or 15 minutes. It was just massive improvement in performance.”
“We were really excited when we saw some of the results,” Mack confirms, but he acknowledges that he remained skeptical. “We often see drastically improved performance in test cases, but when we get to the actual production environment it doesn’t translate,” he says. “We were cautiously optimistic.”
BBH leadership decided to move ahead. ROI analysis supported their decision: by retiring hardware and support contracts, saving on storage, and other improvements, BBH’s team determined that Oracle Exadata would deliver positive return on investment over five years. “It’s not huge,” Remondi says of the ROI, “but we were really pleased that we could both improve performance and reduce costs.”
A 90-day target was set for transitioning the accounting platform to Oracle Exadata. “It was a stretch goal, with an aggressive timeframe to accomplish the project,” says Faisal Azam (known at the company as fAz), BBH’s chief technology officer for fund accounting platform. The transition needed to begin and end in the middle of months to prevent any disruption of month-end closings, and it needed to finish well before the end of the calendar year in case the transition hit a snag.
Mack, who leads client service accounting and whose staff helped IT with the transition, made his wishes clear. “The first thing I said was, ‘We need to hit this deadline, because we can’t extend it any further without a large business interruption,’” he recalls. Mack’s team also insisted on thorough testing of both client- and internal-facing integrations. Some tests—such as creation of realized gain/loss reports covering thousands of securities—were chosen specifically to stress the system. The testing and transition project kicked off the week of July 15, 2012.
The transition team adopted a modified agile approach to their project that also leveraged strengths of BBH’s in-house development process. Three groups were established: A core team of around 45 people, mostly from the 900-person IT group, who met daily during the transition; an extended team of about 50 people, including representatives from the business side, whose membership evolved according to the task or test at hand; and a 25-member steering committee that provided overall direction and executive sponsorship.
To reduce risk and distractions, all development work on the existing accounting platform was frozen during the transition. “Testing needs to be conducted in a controlled environment,” Azam explains. “It’s an exact science—you can’t just say, ‘This test works in this way with this data,’ and then expect the same results using different circumstances.” The core team used Oracle Real Application Testing and Oracle Flashback technologies to perform their tests.
|Total assets that BBH has under custody and administration|
|Value of assets calculated by BBH’s Oracle Exadata–based accounting system every day|
|Number of markets in the world where assets in BBH custody are held|
|Number of locations worldwide where BBH has offices|
The team also used Oracle GoldenGate to replicate data from the existing Oracle9i production database to the new Oracle Database 11g test database. “When we went to Oracle Database 11g, we didn’t just copy everything over from Oracle9i Database,” Azam says. “We actually created all the objects and all the data in Oracle Database 11g. Every bit of our 15-terabyte database had to be precisely mapped and validated.” The team also used Oracle GoldenGate to replicate from the test database to a separate backup Oracle9i database—one that Azam and the team called “the parachute we never wanted to use,” because it would be required only if the final cutover was unsuccessful.
After weeks of testing and dry runs, the actual conversion—a detailed playlist of about 2,100 discrete steps and activities with precise timelines—began at 3 a.m. on a Saturday morning. Over the weekend, the accounting platform and several smaller applications were all migrated to Oracle Exadata. “This project worked perfectly, as planned, to the minute,” Azam says, beaming. BBH’s customers noticed no disruption in service.
But the difference was immediately apparent to BBH’s accounting staff, including the cautiously optimistic Mack. “Once we saw it in production, we were happy with what we saw,” he says. “Some reports that formerly took hours now take 15 or 20 minutes, which puts us in a much better state when we run year-end, quarter-end, and semiannual reports for all of our funds.”
The new system’s speed allows Mack and his colleagues to release NAVs more quickly. This, in turn, frees time to perform accuracy and logic checks, analyze data, provide client consultations, and perform other value-added services. “If we get information to our clients earlier, everybody’s happier,” Mack says, because it allows BBH’s clients to focus on their core business of managing assets.
IT tasks at BBH are faster and easier with the new Oracle Exadata–based system as well. Oracle Automatic Storage Management and other built-in resource management tools have cut down on manual tasks, and Exadata Hybrid Columnar Compression has shrunk BBH’s database from 15 terabytes to 6 terabytes. “Our backups are faster, and our replications are faster,” Azam says. “Everything is different. There’s a far- reaching impact, in positive terms, from this change in size.”
Developers, who trained on new tools during the transition, began tackling a prioritized list of projects once the development freeze was lifted. Oracle Database 11g’s ability to accommodate rolling migrations—that is, to apply upgrades and changes without restarting the software—has enabled developers to improve the accounting platform without interrupting BBH’s 24/7 operations. “Our accounting platform is extremely visible and must be highly available,” Azam says. “Our development activities often involve changing the database objects and structures, so rolling migrations are extremely important to us.”
BBH’s IT team is also enjoying the improved visibility into database operations that the new platform provides. With the old system, reviewing database health and performance required poring over logs, but with Oracle Exadata that information is presented in quick-to-read, web-based visual formats. “It’s phenomenal,” Azam says of the increased visibility. “We can go into much more detail about many aspects of the system’s operations. It helps us monitor the system much more proactively and address issues before they become problems.”
All of these IT benefits are the product of an engineered system, Azam says. “Oracle Exadata is an appliance,” he explains. “It’s a combination of hardware and software. With an appliance, we have highly optimized hardware that works with our Oracle software.”
The hardware itself—two Oracle Exadata Database Machines—is housed in two separate BBH data centers. “We are active/active within each site today, and will be active/active between the two sites within a year,” Remondi says—a definite boost to BBH’s business continuity.
“Moving to Oracle Exadata aligns with our overall IT strategy,” Remondi concludes. “It improves our ability to run our accounting platform, enhances our ability to deliver new functionality, and really transforms us, from a continuity standpoint.”
Fred Sandsmark is a regular contributor to Profit.
Extreme Performance Calls for Extreme Testing
The accounting team at Brown Brothers Harriman (BBH) is respected for its ability to quickly and accurately calculate the value of a portfolio. Before moving BBH’s accounting platform to Oracle Exadata, the company’s IT team turned to another organization respected for quick, accurate information: the Oracle Solution Center.
At 11 sites around the world, the Oracle Solution Center helps approximately 5,000 customers like BBH choose, configure, and deploy Oracle data center solutions every year. The center’s expert team provides architecture design, capacity planning, performance and scalability testing, and proofs of concept using Oracle hardware and software. To provide truly accurate and agnostic testing, third-party and Oracle partner hardware and software are frequently added to test systems.
The Global Oracle Solution Center in Santa Clara, California—an 18,000-square-foot, 300-rack secure facility—assisted BBH with application testing in 2011. The test used BBH’s actual production data and accounting application—not an anonymized equivalent—running on an Oracle Exadata system identical to the one proposed for BBH. “We use extreme data-handling procedures to ensure the security and integrity of customer data,” says Patrick Cyril, senior director of technical sales and consulting at Oracle. BBH’s data was shipped from its New Jersey data center to Oracle via a “data mule”—a secure, encrypted storage server that only BBH employees could unlock. When the tests were complete, BBH’s data was erased using US Department of Defense–compliant data destruction technology—as data is after every test at the Oracle Solution Center.
“There was excellent collaboration between the team at Oracle and BBH,” says Chris Remondi, global head of information technology and partner at BBH. “I think Oracle understood our key priorities very well, and knew that they could be met.”
Not every customer is eligible to use the Oracle Solution Center. Oracle account teams collaborate with the center’s staff to determine the best candidates. However, all Oracle customers benefit from the center’s work, because the lessons learned there are used in reference architectures available to all customers. Those select customers that work with the center can engage its services in several ways—remotely, onsite, or blended—and that engagement can span from one day to several weeks, depending on the size and scope of testing. Learn more about the Oracle Solution Center at oracle.com/osc.