Why MPEx is better than fiat institutions, part 349085 : we don't use Excel.

Quoting from this reportⁱ explaining why the First Federal Bank of the US (formerly JP Morgan) lost about 6 billion worth of taxpayer's dollars in some spectacularly ill thought derivative trades :

VaR is a metric that attempts to estimate the risk of loss on a portfolio of assets. A portfolio’s VaR represents an estimate of the maximum expected mark-to-market loss over a specified time period, generally one day, at a stated confidence level, assuming historical market conditions. Through January 2012, the VaR for the Synthetic Credit Portfolio was calculated using a “linear sensitivity model,” also known within the Firm as the “Basel I model,” because it was used for purposes of Basel I capital calculations and for external reporting purposes.

The Basel I model captured the major risk facing the Synthetic Credit Portfolio at the time, which was the potential for loss attributable to movements in credit spreads. However, the model was limited in the manner in which it estimated correlation risk: that is, the risk that defaults of the components within the index would correlate. As the tranche positions in the Synthetic Credit Portfolio increased, this limitation became more significant, as the value of the tranche positions was driven in large part by the extent to which the positions in the index were correlated to each other. The main risk with the tranche positions was that regardless of credit risk in general, defaults might be more or less correlated.

This limitation meant that the Basel I model likely would not comply with the requirements of Basel II.5, which originally had been expected to be formally adopted in theUnited States at the end of 2011. One of the traders responsible for the Synthetic Credit Portfolio therefore instructed an expert in quantitative finance within the Quantitative Research team for CIO International to develop a new VaR model for the Synthetic Credit Portfolio that would comply with the requirements of Basel II.5. That individual (henceforth referred to in this Report as “the modeler”) began work on developing that model in or around August 2011.

The trader who had instructed the modeler to develop the new VaR model (and to whom the modeler reported at the time), CIO Market Risk, and the modeler himself also believed that the Basel I model was too conservative – that is, that it was producing a higher VaR than was appropriate.ⁱⁱ The modeler believed that an improved model should both (1) adequately capture correlation risk in the Synthetic Credit Portfolio, and (2) produce a lower and more accurate VaR.

A. Development of the New VaR Model

The modeler is a London-based quantitative expert, mathematician and model developer. In addition to the considerable responsibility of developing a new VaR model, he continued to perform his existing responsibilities in providing analytical support to the Synthetic Credit Portfolio traders. On a number of occasions, he asked the trader to whom he reported for additional resources to support his work on the VaR model, but he did not receive any.

Early in the development process, CIO considered and rejected a proposal to adopt the VaR model used by the Investment Bank’s credit hybrids business for the Synthetic Credit Portfolio. Because the Investment Bank traded many bespoke (i.e., customized), illiquid CDS, its VaR model mapped individual instruments to a combination of indices and single name proxies, which CIO Market Risk viewed as less accurate for CIO’s purposes than mapping to the index as a whole. He believed that, because the Synthetic Credit Portfolio, unlike the Investment Bank, traded indices and index tranches, the Investment Bank’s approach was not appropriate for CIO. The Model Review Group agreed and, in an early draft of its approval of the model, described CIO’s model as “superior” to that used by the Investment Bank “in that it [was] a full revaluation approach.”

From September to November 2011, the modeler corresponded regularly with the relevant individuals from the Model Review Group, and on November 25, 2011, he submitted his new methodology (known internally as the “full revaluation” or “Basel II.5 model”) for formal approval. The Model Review Group performed only limited back-testing of the model, comparing the VaR under the new model computed using historical data to the daily profit-and-loss over a subset of trading days during a two-month period. The modeler informed the Model Review Group that CIO lacked the data necessary for more extensive back-testing of the model (running the comparison required position data for the 264 previous trading days, meaning that a back-test for September 2011 would require position data from September 2010). Neither the Model Review Group nor CIO Market Risk expressed concerns about the lack of more extensive historical position data.

During the review process, additional operational issues became apparent. For example, the model operated through a series of Excel spreadsheets, which had to be completed manually, by a process of copying and pasting data from one spreadsheet to another. In addition, many of the tranches were less liquid, and therefore, the same price was given for those tranches on multiple consecutive days, leading the model to convey a lack of volatility. While there was some effort to map less liquid instruments to more liquid ones (i.e., calculate price changes in the less liquid instruments derived from price changes in more liquid ones), this effort was not organized or consistent.

By the end of 2011, some of the pressure to complete the review of the new model appears to have abated because it became clear that Basel II.5 would not be implemented on the previously anticipated timetable. However, as described in Section II.D.1, CIO exceeded its Global 10-Q VaR limit at several points between January 16 and January 26, 2012, which in turn caused a breach in the overall Firm 10-Q VaR limit. The Synthetic Credit Portfolio was the primary driver of each of those excessions. A temporary limit increase was requestedⁱⁱⁱ and required approval of senior Firm management. CIO recommended a temporary limit increase on the grounds that it was taking steps to reduce the VaR and that, in any event, the newly developed model was about to come online that would show a substantially reduced VaR.

Mr. Weiland and another member of CIO Market Risk contacted the Model Review Group regularly in the last two weeks of January to inquire into the progress of the model approval and, in a January 23, 2012 e-mail to the modeler, the trader to whom the modeler reported wrote that he should “keep the pressure on our friends in Model Validation and [Quantitative Research].” There is some evidence the Model Review Group accelerated its review as a result of this pressure, and in so doing it may have been more willing to overlook the operational flaws apparent during the approval process.

On January 26, the Model Review Group discovered that, for purposes of a pricing step used in the VaR calculation, CIO was using something called the “West End” analytic suite rather than Numerix, an approved vendor model that the Model Review Group had thought was being used. The Model Review Group had never reviewed or approved West End, which (like Numerix) had been developed by the modeler.^iv CIO provided the Model Review Group with a reconciliation test, based on a limited number of days, showing that the valuations from West End and Numerix were in “good agreement,” and the Model Review Group committed to conduct a full review of West End separately, but not before approving the VaR model. The Model Review Group did not examine West End until early May 2012 (the results of which are discussed below).

On January 30, the Model Review Group authorized CIO Market Risk to use the new model for purposes of calculating the VaR for the Synthetic Credit Portfolio beginning the previous trading day (January 27). Once the new model was implemented, the Firm-wide 10-Q VaR limit was no longer exceeded. Formal approval of the model followed on February 1. The formal approval states that the VaR calculation would utilize West End and that West End in turn would utilize the Gaussian Copula model^v to calculate hazard rates^vi and correlations. It is unclear what, if anything, either the Model Review Group or CIO Market Risk did at the time to validate the assertion that West End would utilize the Gaussian Copula model as opposed to some other model, but that assertion later proved to be inaccurate.^vii

As part of its approval of the new model, the Model Review Group included an action plan with respect to two of the risk areas that were identified. First, it mandated automation of the VaR model by January 31, 2012 (i.e., contemporaneously with the model’s approval).

Second, it required monitoring of illiquid tranches to assess whether mapping to more liquid tranches would be necessary, and ultimately development and submission to the Model Review Group of a risk mapping methodology. Neither of these action plans was completed. The Model Review Group and CIO Market Risk apparently believed that work was already underway to complete automation but took no steps to determine that automation had in fact been completed. The modeler likewise did not submit, nor was he ever required to submit, a complete risk mapping methodology.

B. Operation of the VaR Model

From February to April, the new VaR model was in operation. A CIO employee who reported to the modeler was responsible for daily data entry and operation of the new model. In April, an employee from the IT Department (who had previous experience as a senior quantitative developer) also began to provide assistance with these tasks. Notwithstanding this additional assistance, a spreadsheet error caused the VaR for April 10 to fail to reflect the day’s $400 million loss in the Synthetic Credit Portfolio. This error was noticed, first by personnel in the Investment Bank^viii, and by the modeler and CIO Market Risk, and was corrected promptly. Because it was viewed as a one-off error, it did not trigger further inquiry.

C. Discovery of Problems with the New VaR Model and Discontinuance

In early May 2012, in response to the recent losses in the Synthetic Credit Portfolio, Mr. Venkatakrishnan asked an employee in the Model Review Group to perform a review of the West End analytic suite, which, as noted, the VaR model used for the initial steps of its calculations. The West End analytic had two options for calculating hazard rates and correlations: a traditional Gaussian Copula model and a so-called Uniform Rate model, an alternative created by the modeler. The spreadsheet that ran West End included a cell that allowed the user to switch between the Gaussian Copula and Uniform Rate models.

The Model Review Group employee discovered that West End defaulted to running Uniform Rate rather than Gaussian Copula in this cell, including for purposes of calculating the VaR, contrary to the language in the Model Review Group approval. Although this error did not have a significant effect on the VaR, the incident focused the reviewer’s attention on the VaR model and ultimately led to the discovery of additional problems with it.

After this re-review, a decision was made to stop using the Basel II.5 model and not to rely on it for purposes of reporting CIO VaR in the Firm’s first-quarter Form 10-Q. Following that decision, further errors were discovered in the Basel II.5 model, including, most significantly, an operational error in the calculation of the relative changes in hazard rates and correlation estimates. Specifically, after subtracting the old rate from the new rate, the spreadsheet divided by their sum instead of their average, as the modeler had intended. This error likely had the effect of muting volatility by a factor of two and of lowering the VaR, although it is unclear by exactly what amount, particularly given that it is unclear whether this error was present in the VaR calculation for every instrument, and that it would have been offset to some extent by correlation changes. It also remains unclear when this error was introduced in the calculation.

Aren't you impressed ?

For anyone keeping score, the relevant commentary re Excel is found in the article "Law enforcement never fails to unintentionally entertain". In any case Excel is not "one of the greatest, most powerful, most important software applications of all time", as I've seen it described by some random clueless comentator.^ix GCC is one of the greatest, most powerful, most important pieces of software of all time. Vi is one of the greatest, most powerful, most important pieces of software of all time.^x GnuPG is one of the greatest, most powerful, most important pieces of software of all time. Yes, this list goes on for a while. No, nothing on it runs on Windows (or Mac, for that matter). The reason is that nothing on Windows is great, powerful or important. Not this time, not any time, certainly not all time. Maybe at some point in the (likely very distant) future the miracle happens and something powerful, or maybe important, or possibly even great is written for Windows (Mac is really unlikely). Until such a time however...

And in closing, a quote from the discussion of MPEx innards :

This is what we mean when we say things like “Bitcoin will revolutionize the banking industry”. It’s not something along the lines of Amir “Retard” Taaki going “rabble rabble maffs rabble rabble down with the banks”. It’s something along the lines of, “here’s this complete solution to the one problem you people don’t know how to solve”. From there on the future is pretty clear : either they move to the light side or fall by the wayside. Nobody can continue to offer flawed services once somebody opens up shop doing the Right Thing. That’s revolution for you, and it’s certainly enabled by Bitcoin : if it wasn’t for Bitcoin I wouldn’t have bothered to get involved, and if I wouldn’t have bothered to get involved practical deployment of PGP for financial services would have been still waiting for whatever it was awaiting up to this point. It’s 1990s technology after all.

Apparently not only we win because we do not have the problems of security that have rendered the entire credit card system useless^xi but we also win because... we do not use Excel.

Incredible how easy winning is when your competitor is the nationalised financial industry.

1. Conducted in house, in large part because past attempts to conduct outside reviews resulted in expensive, flawed and ocasionally nonsensical reports. You know, just like ROTA worked out. [↩]
2. As noted above, VaR is a metric that attempts to estimate the risk of loss on a portfolio of assets. Both the modeler and a member of the CIO Market Risk team who was also involved in the new model’s development were of the view that the Basel I model might be overstating the VaR for the Synthetic Credit Portfolio, in part because the amount of losses had exceeded the stated VaR limit less frequently than would be expected based on the stated confidence level. [↩]
3. Firm-wide Market Risk raised the possibility of a temporary limit increase to Mr. Hogan on January 20, 2012. On January 21, 2012, the then-head of the Risk Reporting and Finance function – told Mr. Hogan “We are working towards a temporary one-off for CIO and the Firm proposed as follows: JPMC $140mm (vs. $125mm permanent limit) CIO $105mm (vs. $95mm permanent limit.” Mr. Weiland also e-mailed Mr. Hogan on January 22, 2012 regarding a proposed temporary VaR limit increase. [↩]
4. The modeler had previously worked at Numerix. While there, the Numerix repricing model was developed under his supervision. [↩]
5. The Gaussian Copula is a commonly accepted model used to map the approximate correlation between
   two variables. (Accepted, yes, convenient very, sound not so much - n.Tr.) [↩]
6. A hazard rate is the probability of failure per unit of time of items in operation, sometimes estimated as a ratio of the number of failures to the accumulated operating time for the items. For purposes of the model, the hazard rate estimated the probability of default for a unit of time for each of the underlying names in the portfolio. [↩]
7. A March 30, 2012 Internal Audit report on the Market Risk and Valuation Practices in CIO’s credit portfolios (including the Synthetic Credit Portfolio) assigned a rating of ‘Needs Improvement’ due in part to CIO’s use of “unapproved models in the calculation of risk (including VaR).” The reference to the use of “unapproved models” in the calculation of the VaR is to West End, which, as the Internal Audit report noted, had not been submitted to the Model Review Group for Review. The Internal Audit report included an action plan for CIO to document the West End analytics engine and submit to the Model Review Group with a target completion date of June 30, 2012. While the Internal Audit report also noted problems with the control processes surrounding the VaR calculation, Internal Audit found no specific examples of incomplete or inaccurate data. [↩]
8. On April 18, a member of the market risk team for the Investment Bank obtained information on the Firm-wide and CIO VaR calculations to determine the impact of the April 10 loss on the Firm-wide VaR. Upon discovering that the loss was not reflected in the CIO VaR, he reported his findings to Firm-wide Market Risk, who in turn reported to Mr. Hogan that CIO’s VaR appeared to have an error. [↩]
9. James Kwak, associate professor at the University of Connecticut School of Law, if you must know who. [↩]
10. I don't have any idea what "software application" is supposed to mean. Possibly the author is in dire need of some hardware application to the occiput, or else inherited his professorship somehow without needing to attend any school at all. [↩]
11. When something between one third and half of all credit cards ever issued are available for criminal use you can't claim the system is still working in any sense of the term. [↩]