System Transformation Portal

Contingency Planning Methodology

Appendix D:  Contingency Planning Options

Business Based Alternatives - general guidelines

1. If a system failure is imminent and the IT team has no way to fix it, consider:

• selling off a low margin business unit or division where this dramatically reduces upgrade efforts
• shifting to a similar in-house application that is already servicing a different segment of the business
• changing to the underlying business model to eliminate or reduce the need for an existing system

2. If a system failure occurs and the IT team has no way to fix it, consider:

• a change to the underlying business model could eliminate or reduce the need for a system
• if users can shift to a similar in-house application that is already servicing a different segment of the business

3. If a system is going to be down for a brief period and repairs are underway consider:

• having end-users work with a paper trail (built as a contingency) remain productive while system is fixed (works best with accounting or billing systems)
• determining if manual processes could be applied by business analysts during an interim period while the system is fixed
• turning system off and working around it until impact window closes (if system has forward / backward date referencing limited to 30 days or less)

4. If a product fails that was acquired from a supplier:

• determine if a replacement can be obtained in time to correct the problem
• obtain product from a backup supplier (that should be researched and established during this planning process)
• assess options to have business work without the availability of the product (this is time dependent in many cases)

5. If a service supplier fails to continue to deliver that service:

• assess how long the client can continue to function in the absence of this service
• move to a backup source for that service (power generator, performing service in-house, etc.)
• determine if a replacement service can be obtained (establish options based on criticality and probability of failure)

6. If a product fails that was sold to a customer:

• build a hot-line service to accommodate customer inquiries
• send out a replacement offering quickly (if available and monetarily feasible)
• for discontinued product lines, determine best course of action based on input from legal team and marketing

7. If a business partner fails:

• work with that partner to correct the problem as quickly as possible
• seek out replacement partners to fulfill these requirements
• determine if the function can be brought in-house
• determine if another existing business partner can fulfill these service commitments
• eliminate the function / business unit that depends on this service (this is a proactive contingency plan)

Technology Based Alternatives

1. For each application system in the group, list the correction priorities to be applied if the system fails:

• prior to the failure date: one day turnaround
• after potential failure date: one-to-two day turnaround if pre-date
• after date: three-to-five day turnaround (due to prediction of slow post-date response capability)

2. If a data file or data base is contaminated:

• backup to an earlier version
• isolate and stop the source of errant data
• correct the error in the data
• replace the source of the data

3. For each external data interface that fails, assess options to:

• correct the data after receiving it
• run the system without the data until corrected
• work with external source to correct the problem (weighed against dropping them)
• establish an alternative source for that data
• discontinuing use of data based on marginal value

4. For the hardware device supporting various application systems, identify how support areas would respond if that hardware has problems:

• assess impact on application areas
• determine maximum allowable downtime
• seek alternative hardware or processing options (disaster recovery?)

5. For each application package supporting this system, list the sequence of responses that would be required to fix or supplement the functions of that package:

• fix the system in-house
• work with vendor to correct problem
• drop in a quick replacement option
• use another in-house system or package
• discontinue use of system functions in business area

6. Where potential embedded technology failures might occur, list the likely response including:

• backup options from other business areas
• replacement or upgrade options
• pursuit of alternate business options

7. If reporting or query functions fail, consider rebuild options that could supplement the data being provided by those functions.

Common 3rd Party Contingency Plans for specific industries

Financial Institutions

1. Data interchange interruptions or errors

• work jointly with creators of data to correct the problem
• switch to alternate source of data from a backup data supplier
• attempt to correct data errors once information has been received
• for low impact systems, freeze processing of this data until problem is resolved
• work with industry association to freeze trading of financial instruments for a period of time until problem is resolved (when no alternatives exist)

2. Credit card processing failure in banking system

• if volume is low, work around the failure manually until system is corrected
• determine if card supplier or another source could provide a backup processing system
• discontinue acceptance of this type of card
• work with credit card company to assess other alternatives

3. Trading system failure

• determine if a second trading system could be modified to accept these trades
• for low impact or non-strategic monetary instruments, halt trading until situation is resolved
• work with a third party to have them process trades as a service to the client

4. Clearing house processing failure

• work with clearing house to correct problem
• switch to alternate clearing house if possible
• work with SEC to determine if alternate clearing house can be used or established

Insurance Companies

1. EDI data interruption halts claims processing

• deploy backup data interchange plan that utilizes more traditional communication exchange
• launch Internet data exchange system
• process claims manually until source of problem is corrected
• work with company (depends on size and strategic position) to resolve problem

2. Claims processing

• process claims manually while system is corrected
• switch processing to alternate claims processing system (determined in advance)
• if low margin or non-profitable business unit, discontinue service

3. Broker system failure

• insurance company may discontinue use of that broker
• insurance technical support could help fix or replace broker system
• insurance company may recommend that broker obtains a system from another broker for a price

Health Care Providers

1. Failure in diagnostic equipment embedded technology

• utilize alternative diagnostic techniques until problem is corrected
• immediately replace most critical products
• send patients to another hospital during interim period

2. Failure in heart defibrillator

• refer heart patients to another hospital until situation is fixed
• order new product from manufacturer
• use alternative equipment or techniques until the product is replaced

3. Data interchange / EDI failure

• perform data exchange with providers, doctors and insurance companies manually until problem is corrected
• seek help from partners, insurance providers or other sources to correct problem

4. Major patient, billing & processing system failure

• admit, bill and process patients manually until system is fixed
• buy or lease an alternative system as soon as possible (requires retraining personnel)
• sell the clients hospital to another hospital that has corrected the problem - typically when profit margins prevent remediation

Manufacturing & Retail Industries

1. Programmable logic controller fails, shutting down production line

• attempt to replace embedded technology as soon as possible (hopefully with a component that was already procured based on contingency plans)
• ship product that may have been backlogged as a contingency plan
• determine if an alternate product can be shipped to fulfill customer requirements
• if possible, determine if manual processes can supplement automated production
• determine if the client can buy product elsewhere and pass it on to customers at a loss until problem is resolved

2. Inventory control system miscalculates stock levels

• inventory levels should be stocked to allow a 3-4 week window, if possible, while system is being repaired
• begin fixing system immediately
• determine if business analysts can manage inventory levels manually for a period of time

3. Manufacturing supply chain shuts down – the client cannot obtain parts

• contact backup supplier identified during contingency planning process
• work with intermediate suppliers between the clients company and the company that failed to come up with alternate product solutions
• buy the company that failed and move the production process in-house
• continue making products with backlogged supplies as long as the client can

4. Retail supplier shuts down

• for strategic supplier, put a team onsite to help correct the problem
• for medium impact supplier, consider replacing supplier with new supplier
• for low impact supplier, consider dropping product offering

Service Industries

1. Systems failure shuts down law firm

• work around the problems manually, ideally with a paper trail that was created as a contingency strategy
• try to fix system with systems supplier
• see if a replacement system can be installed

2. Reservations system fails at an airline

• keep planes flying & issue guidelines to on-line reservations clerks
• tell people to come to the airport and they will be handled on a first come / first serve basis
• continue to try to fix the system as quickly as possible
• issue customer bonuses and other perks to work with the client during the crisis
• determine if the client can lease reservation system time from another airline

3. Restaurant point of sale system fails

• issue guidelines on how to handle the problem
• work with system provider or in-house teams to correct and re-deploy the system
• have replacement system identified and ready to deploy immediately

4. Older PCs at a radio station fail, shutting down broadcast capability

• triage the old system
• replace with new system - if the budget is available
• sell the station if the money is unavailable

Utilities & Telecommunications

1. Power company's power grid computer shuts off power to millions of people

• put a full time team on the problem immediately
• establish a hot-line for customers communicating exactly what is happening and how they should respond
• establish a web site for customers (for access by people with battery powered PCs) to find out what is happening
• attempt to procure backup power, if applicable, from alternate sources
• work with industry groups to correct problem more quickly
• get to various news outlets to communicate what is happening and when it will be fixed

2. Nuclear power plant control system will not be compliant in time

• create backup power sources in advance – just in case there are problems
• build reinforcements around the plant, including under it, as a fail safe strategy

3. Telecommunication data interchange fails and fouls up millions of billing records

• open a hot-line and web site to communicate what happened
• put a team on the problem to fix it as soon as possible
• offer discounted service for a certain period after the problem is fixed

4. Embedded communication equipment fails

• put team on problem immediately
• look for backup communication lines that could be leased from another provider
• if problem goes on for a long time, consider business unit to a competitor

Government Strategies

1. Revenue collection & tax system fails

• convert graduated tax code to a flat tax or national sales tax if current system cannot be corrected in time
• create public relations campaign to make sure the public understands the process

2. Entitlements disbursement system crashes, checks cannot be issued

• fix the system if the client can
• rig system to write checks manually with equal amounts (probably resulting in overpayments) going out to all parties to avoid underpayment
• pass a law to overhaul the system so that a new, simplified disbursements system can be implemented
• create a hot-line to communicate issues to general public

3. Weather monitoring equipment fails

• attempt to fix the system as quickly as possible
• triage outdated equipment and replace it with new equipment

4. Real time weapon systems fail during a post-date test

• communicate this information on a need-to-know basis only (which includes allies that are using the same equipment)
• triage older systems in favor of newer replacement systems (could be a big money saver)
• eliminate the system / weapons completely if defense hierarchy deems this acceptable