On August 24 the Financial Services Agency of Japan issued a business improvement order to Tokyo Stock Exchange, Inc. (TSE) pursuant to the provisions of Article 153 of the Financial Instruments and Exchange Act.
Please allow us to offer once again our deepest apologies to investors and parties concerned for the inconvenience caused by the system failure related to our derivatives trading system.
We take the order from the authorities with utmost seriousness and will make every effort to prevent a recurrence of the incident while taking all possible measures to ensure stable market operations.
1. Content of business improvement order
(1) Improvement measures established and reported by TSE shall be subject to verification by external experts in order to ensure effectiveness.
(2) Swift and effective implementation of measures that have been verified by external experts.
(3) Regular reports on the progress and status of the above (1) and (2).
2. Account of incident
At 9:18 a.m. on August 7, 2012, a hardware failure occurred at Layer 3 Switch #1, a derivatives trading system network device, which was the primary device in a primary-standby configuration. A system failure task force was formed immediately to identify the nature of the incident and promptly recover system operations.
The derivatives system adopts a redundant configuration for such network devices, and in case of a failure at the primary device (#1), operations were expected to automatically switch to the standby device (#2). However, a defect in the device hindered normal switching and disrupted the transmission of order information, trade execution information, and market information for all derivatives products between trade participants, ISVs, and the exchange. At 9:22 a.m. trading was suspended for all derivatives products.
TSE confirmed that the recovery operations for the derivatives system were completed at 10:20 a.m. Order acceptance began at 10:40 a.m. and trading resumed at 10:55 a.m. Note) A Layer 3 switch is a type of network routing device. Its primary function is to sort data for each destination. The L3 switches in our system send and receive order information, trade execution information and market information to and from participant systems and connect to servers within the derivatives system.
3. Cause of failure and countermeasures
In the incident, when the hardware failure occurred at the L3 Switch #1 (primary device), the L3 Switch #2 (standby device) automatically switched itself to primary state as expected. However, the L3 Switch #1 failed to detect the partial hardware failure within it and remained in primary state. This unexpected behavior left both devices operating in primary state. As a result, connecting servers could not ascertain which switch to send messages to and transmission failed.
We plan to install, in September, a revised version of the built-in program in the device which failed after thorough verification in our test environment. (M easures are already in place to ensure a smooth and prompt switching even if a similar incident occurs.)
4. Preventive measures against recurrence
Although the direct cause of the system failure was a defective L3 switch, the recent incident in the derivatives system occurred shortly after the failure of our equity trading system on February 2 this year. TSE recognizes the huge responsibility it bears in this matter with regard to the loss of trading opportunities in the derivatives market. In order to fulfill our responsibilities and restore confidence in the Tokyo market at home and abroad, TSE will implement measures from the following two perspectives.
(1) Preventative measures From the perspective of business continuity, the following measures will be implemented to prevent any incident leading to a suspension of trading. Before implementation, there will be a third-party check by external experts to confirm the appropriateness of the measures.
The disposition regarding this case shall be as follows.
(1) Reduction of executive officer compensation President & CEO Atsushi Saito 30% of monthly compensation for 2 month
Representative Senior Managing Director & COO Hiroyuki Iwakuma 30% of monthly compensation for 1 month
Senior Managing Director & CIO (IT Unit) Yoshinori Suzuki 30% of monthly compensation for 1 month
Executive Officer Yasuhiro Yoshida 30% of monthly compensation for 1 month
(2) Strict admonishment
A strict admonishment was issued to Senior Managing Director Yoshinori Suzuki
and IT Development Department Trading System Director Hiroaki Uji.