EDUCAUSE | Business Continuity Planning

Business Continuity Planning: It's Much More Than Just Disaster Recovery

renaudr — Fri, 06 Jun 2008 12:47:40 -0500

Business Continuity Planning: It's Much More Than Just Disaster Recovery

Session Details

Systems Evolution

Wednesday, May 28, 2008

3:30 p.m. - 4:45 p.m.

Michigan Room (4th Floor)

Speaker(s)

William Hardy, Director of Business Continuity, California State University, Northridge

Christopher Xanthos, Senior Director, Project Management & Application Services, California State University, Northridge

Session convener: Terry R. Mollett, Director, User Services, Dickinson College

Rather than repeat the content of the presentation, I have attached the PowerPoint slides that were used. I thank the presenters for their willingness to share these materials.

The presenters experienced in January 1994 an earthquake devastated the campus of California State University, Northridge. The campus lost major buildings including the campus data center and there was no effective disaster recovery plan in effect. I found the following issues brought up by the presenters to be particularly interesting.

Nature of Disasters

In the past we have tended to think in terms of natural disasters such as earthquakes, fires, and floods. While we need to continue to consider these threats, we also need to think about events caused by humans, such as the tragedy at Virginia State.

Emergency and Business Continuity Segments

The presenters distinguished between the emergency segment, the initial response to a disaster, and business continuity, the time needed to return to normal operations. The emergency segment tends to be shorter than the business continuity segment was 157 hours. For example, in the case of the February 2007 ice storm that affected Jetblue, the Emergency Segment of the Event was 11 hours and the

Business Continuity Segment was 157 hours.

Building Business Continuity into Processes

We normally think about business continuity after processes have been designed and implemented. Instead, we need to build business continuity into processes and systems as they are being designed. This greatly reduces the cost and complexity associated with business continuity and disaster recovery.

Cash

We can forget that in the event of a serious disaster such as the Northridge earthquake our access to funds may be cut off. In the case of that disaster the university needed to borrow money from local vendors and other sources just to pay their staff and maintain operations. We need to think about storing cash reserves sufficient to maintain operations for extended durations.

Assume the Worst

Many disaster recovery plans assume limited events. It is preferable to plan for scenarios where large scale disasters occur. It is easier to plan for a large disaster and then experience a small one than for the reverse.

Podcast: Weathering the Storm—Preparing for, Responding to, and Recovering from Emergencies

Carie417 — Mon, 02 Jul 2007 16:58:58 -0500

This one-hour podcast was recorded at the EDUCAUSE 2007 Southeast Regional Conference. This keynote panel discussion, entitled “Weathering the Storm – Preparing for, Responding to, and Recovering from Emergencies,” features Betty Hawkins, program manager at the University of South Carolina, David Sliman, directory of technology at the University of Southern Mississippi, and Frank O’Quinn, deputy policy and IT disaster recovery officer at Louisiana State University. Using examples from their own institutional experiences, they share real advice for emergency preparedness and response, answering the question, “How can we plan and prepare in a way that is realistic and meaningful?”

EDUCAUSE Southeast Regional: Weathering the Storm Panel

llarsen — Wed, 20 Jun 2007 11:05:36 -0500

Summary: Weathering the Storm -- Preparing for, Responding to, and Recovering from Emergencies

This session was recorded for podcast.

General Session
Tuesday, June 12, 2007
Atlanta, Georgia

Session moderator:
Kathryn F. Gates, Chief Information Officer, University of Mississippi
Panel:
Betty Hawkins, Program Manager II, University of South Carolina
David J. Sliman, Director of Technology-Gulf Coast, University of Southern Mississippi
Frank O'Quinn, Deputy Policy and IT Disaster Recovery Officer, Louisiana State University

Abstract
Along with Charley, Ivan, and Katrina, hurricanes and other natural disasters are a part of life in the Southeast. Beyond the weather, many other calamities might hit us at any time. While many of us have been involved in efforts to develop emergency preparedness, disaster recovery, and business continuity plans, often these sit on a shelf and have little to do with how the institution really responds in an emergency. How can we plan and prepare in a way that is realistic and meaningful? How can we collaborate in advance so that we will be prepared to help each other when disaster strikes? What are the critical functions of IT before, during, and after an emergency? How can IT best facilitate long-term recovery efforts? This panel will feature several IT professionals who have experienced disastrous situations first-hand and can offer valuable insights to others.

NOTES

Kathy Gates introduced the three perspectives of disaster recovery and business continuity from three different institutions. The themes of the panel will cover planning, collaboration and partnership, and managing technology in emergencies.

==========
David Sliman, whose campus (University of Southern Mississippi) and home were totally destroyed ("everything south of 100 miles of railroad track was gone"), began the session with descriptions of the hurricane damage, his gratitude for all of the support that continues to be offered, and his lessons learned from the experience.
1) Have your personal life in order (insurance docs and all)
2) Plan how far to evacuate - no more than ½ tank gas away
3) It may be difficult to get back so who will be in charge and who has the right documentation to be in charge. Get in touch with officials ahead of time for the right documentation to make sure you can get back on site.
4) Health and welfare of co-workers should be first and paramount
5) Know how you will contact your employees. It may be that you'll have no cell phone towers. Perhaps only text messages will work. Have alternative meeting places.
6) Communications are important:
a. Develop a phone tree for communications (divide and delegate) NOAA has a call in service (1-800 number)
b. Make sure you have your employee's spouse or relative's contact information
c. Also have your employee's children's information
d. Activate text messaging for everyone
e. Have off-site web services arranged or answering service check-ins
7) Have good records of your inventory for insurance (home and work) and take them with you when you evacuate
8) Secure intellectual property and this includes faculty data/papers/books/
9) Disasters come in many flavors. It is the planning not the plan that is most important
10) Be prepared - ahead of time!

There were a few good things like
- Slab Boils instead of Crab Boils (parties on the slabs of concrete that were left)
- Trunk'r'treat - Halloween was trunk by car trunk instead of door to door
- Sculptures have been created from many of the damaged trees

===========
Frank O'Quinn, who was appointed the LSU IT disaster recovery guy one week before Katrina, hunkered down at home with gas in the car, filled propane tanks, batteries and candles, had peanut butter and extra water, games to play with the children, and waited for the power to go out. After the flooding the campus closed and everyone stayed home and cleaned up from the storm and waited for power to return. Then it was a matter of being of service to those critically affected.

LSU's role in these situations is to:
1) Take care of students and employees first - closed the campus
2) Provide a social services location for a special needs shelter.
3) Use their basketball arena to house the shelter which always opens prior to storms for people to use.
LSU & Katrina was a different situation
- There were more people affected and many more responders arriving.
- No one was in charge of these "many more people"
- Determined tasks, organized volunteers, activated an emergency operations center

The temporary medical operations and staging area (TMOSA) that they set up processed over 40K patients. They had to keep the TMOSA open until these people had some place else to go. They were still open when Rita hit and finally closed after 52 days.

LSU had no detailed written Memorandum of Understandings (MOUs) and was not a federally recognized emergency operations center and so they spent over $1 million on everything from telephones to tracking patients. LSU absorbed that 1 million in hard cash - not reimbursed.

Everyone was very emotionally involved but it was also very important to return to the primary mission
- Classes resumed Tuesday following labor day
- Home football game switched to away
- Need to encourage guests of dorm residents to find other arrangements - Faculty staff students swap to get families out of dorms into homes of LSU community - Everyone gathered on the football field with signs of who they could take

Board of regents - declared any student enrolled in an affected state 4 yr school permitted to enroll in any other state 4 year school
For the enrolled displaced students
- Timing was just right - few missed days
- IT had to modify processes, no fee admits
- Existing WEB portal unfeasible
- No dorm rooms, apartments, houses
- Constantly learning lessons and applying them immediately into the process.

On top of enrolling displaced students they were still running the shelter and the requests after Katrina increased for services such as phone, networking connections, pc laptops printers and more. They went with VOIP phones and felt it was a good decision. Their vendors delivered equipment and Frank reminded us of the importance of maintaining good relationships with vendors.

They took on many tasks that were outside their normal roles - like broadcast emails, call center, patients database, volunteer database (many students), map of campus for media, responders. They continued to adjust administrative applications and had twice daily meetings with the chancellor for status checks.

A few of the changes since Katrina - They now have a formal LSU Emergency Operations Center and Formal MOUs and agreements with state agencies and private sector services for 4 days of food, fuel, and water on campus - with secondary suppliers backing up primary ones.
They also have a full time generator for their basketball arena and have pre-planned emergency logistics.

In 2006 they prepared for repeat hurricanes, especially critical now that people were living in mobile homes. IT disaster recovery now has a tested hot-site for mainframe activities and the Chancellor requested written plans from all units on campus. These plans are being updated for 2007

Observations from their experiences:
- Things will be unpredictable. They won't unfold as you planned and you need to be fluid.
- Priorities change by the minute
- Some projects will be aborted after much effort which will cause disappointment and hurt feelings.
- You have to watch out for ideas being suggested that are illegal (ie HIPPA)
- Keep your supplies stocked
- Think about what people still aren't thinking about. People still don't comprehend the unpredictability and they never believe it will happen to them
- Don't wait for the money
- Do the right thing.

Know the big picture
- your mission
- your resources
- your people

===========
Betty Hawkins had no direct experience with Katrina or Rita but as she discussed,
Katrina was just one disaster but things happen every day to different people around the world. She has worked in the area of emergency preparedness for many years and has seen both disasters and incidents of many types.

Disasters: long term or permanent damage
- Fire
- Floods
- Power failures
- Blizzards & ice storms

Incidents: short term interruption
- Environmental
- Hardware failures
- Software failures
- Sabotage

Experiencing a disaster throws you into continuous process improvement.
- People don't plan to fail - they just fail to plan. Lack of planning adds to the mayhem.
- It is important to build relationships with all groups in your setting. This may include groups beyond the local area.
- Plans are effective if people know about them
- Your emergency planning should include marketing avoidance or otherwise stated: how to prevent an incident or disaster.
- Each of us should also have a plan for home emergencies.

Creating the plan is the easy part. It's important to engage others to think about the following layers of the plan:
- Prevention
- Incident recovery
- Disaster recovery

Lessons she has learned over the years include:
- If you complete the plan today, it will be obsolete tomorrow. Continuous planning and preparation is needed.
- Someone needs to own "disaster recovery" at the institution level and the unit level. They must do it and practice it
- Plan on at least one FTE for your IT organization.
- No two disasters are alike
- Be prepared for emotional fallout
- Understand that panic spreads quickly
- Every organization is different so try different things
- Planning and conducting mock disasters is the best way to demonstrate preparedness
- Create an email suggestion box
- Provide positive recognition - volunteers, best ideas for improvement, certificates

Ideas to share information
- At the University of South Carolina she publishes a newsletter every semester. She brings in local EMS, Fire, Police, Red Cross, etc., to write articles or speak on campus and they love to do this.
- Changing the mindset is important so that people think about the little things they can do everyday - her illustration was to put a spare car key with your ID badge.
- University wide business impact analysis (BIA) - circulating this form will generate thought
- Give brown bag presentations to Business managers and keep it easy to understand
- Start a users group and invite various experts to come and speak
- Get the DR & BC people talking with their counterparts at other colleges and universities. (USC's role in Katrina - in less than 24 hrs the IT group set up a place to do IDs for evacuees and talk to state services)
- Reciprocal agreements between organizations should be looked at very closely and determine what are the resources to manage over time
- Create a web site and remind people to visit often. Make sure it includes what IT will provide for recovery and how IT will help in an emergency.

Panel Discussion
Topic: CIO perspective on emergency planning - How should the CIO prioritize?
- Betty suggested that CIOs think about it now and brainstorm with others/stakeholders to develop plan
- Frank believes that communications comes first (from the business side of the house receipts come first but employees think payroll comes first)
- David said communication plans are first

Comment from Monte Luehlfing: You find yourself doing things you never thought you'd be doing - like running a day care center. You have to take care of people first.

Topic: Disaster Exercises -
- Betty indicated that they do hot site exercises (strictly IT) and they find something wrong every time. The exercises are to find out what's wrong. They are not to document what's right. If your exercise is perfect then you haven't succeeded. You should set up a different goal each time you do an exercise.
- Frank said the minimum is that the smoke detectors go off once a month. However they pay visits to their hot site to bring up the backup regularly. Their A Team tests and finds a number of things that they then work on tweaking.

Question: What should the DR technology be? Should we have a black box or a remote site or something else?
- Frank said if main building goes down the alternate site goes up. However, distance is important, 200 yards away not enough. Multiple sites and farther away may be best.

Question from Sandy Schaeffer: Where does the external motivation come from to do emergency planning?
- Kathy indicated it was the State in her situation
- Betty said it was the recognition of a vulnerable point - (happens because it is important)
- Frank suggested that it comes from within the organization but people are "too busy" to do anything individually so organizations have to bring it down from the top. Plus it is "almost equivalent" to a requirement in the state in LA.

Final Question: What should we take away as action items to do immediately upon return to our institutions or key lessons?
- Depends on where you are - look at your own group and where you are vulnerable - low hanging fruit
- People became leaders after Katrina: grow from where you are
- Emotional impact is significant so you must figure out how to deal with the emotional and psychological issues. Keep staff busy so they don't think about other things.
- LSU has changed everything they do and now asks for a DR budget as well (see http://www.educause.edu/SERC07/Program/12202?Product_Code=SERC07/SESS01)

New Business Continuity Planning Page Posted at EDUCAUSE Connect

cluckett — Thu, 24 May 2007 15:17:06 -0500

EDUCAUSE has identified links concerning business continuity that may be useful to the higher education community on the new Business Continuity Planning resource page, including EDUCAUSE Review and EDUCAUSE Quarterly articles, federal government policies, and university resources.

ECAR Releases New Study on IT and Business Continuity in Higher Education

cluckett — Thu, 29 Mar 2007 17:54:06 -0500

The latest EDUCAUSE Center for Applied Research (ECAR) study, “Shelter from the Storm: IT and Business Continuity in Higher Education,” looks at IT unit readiness to foster and support the functioning of colleges and universities that are challenged by disruption. Responding to a well-documented increase of interest in business continuity and disaster recovery issues among higher education chief information officers (CIOs), ECAR designed the study to inform executives about how institutions approach continuity issues and to identify practices that are associated with good business continuity outcomes.

The study methodology included a literature review; consultation with a select group of CIOs and business continuity experts for the purpose of identifying and validating research questions; a quantitative survey of IT administrators (mostly CIOs) at 340 higher education institutions; postsurvey interviews with 15 executives and IT staff members involved in business continuity; a quantitative survey of institutional business officers (mostly CBOs/CFOs) at 247 member institutions of the National Association of College and University Business Officers (NACUBO); and four case studies looking at business continuity planning and operations Florida State University, New York University, Pace University, UC Davis, and UCLA.

The study is available to ECAR subscribers and through purchase, but the key findings and roadmap are publicly available. Visit ECAR on the Web.

Tune In Feb. 27: Free EDUCAUSE Live! Web Seminar on Selecting a Business Continuity Panning Tool

cluckett — Tue, 20 Feb 2007 16:07:18 -0600

Business continuity and disaster recovery planning is becoming an increasing priority for higher education institutions. To facilitate this planning, many are exploring the use of business continuity planning tools, which establish a planning framework, offer direction in the planning process, and provide a common platform for the various campus groups involved. In this free February 27 EDUCAUSE Live! Web seminar, speakers from three institutions will discuss the business continuity planning tool selection process at their institutions, focusing on the criteria used and lessons learned. Those unable to tune in can visit the archives after the event.

EDUCAUSE2006 Podcast: Higher Education IT Readiness

Carie417 — Fri, 16 Feb 2007 09:31:39 -0600

In this 50-minute recording from the 2006 EDUCAUSE Annual Conference, we'll hear from Ronald Yanosky in a session entitled Higher Education IT Readiness for Business Continuity. Using results from a survey conducted by the EDUCAUSE Center for Applied Research, Yanosky takes a look at at business continuity planning, policy, and infrastructure.

EDUCAUSE Launches Business Continuity Constituent Group Listserv

cluckett — Thu, 21 Dec 2006 11:01:08 -0600

EDUCAUSE has launched the Business Continuity Management constituent group listserv, an open-subscription forum for strategic and tactical discussions on restoring business and academic services after disruptions to normal operations. Peruse other EDUCAUSE constituent groups.

New EDUCAUSE Quarterly Article Provides Seven Strategies for Improving Emergency Response

vvogel — Wed, 08 Nov 2006 14:14:47 -0600

In the recent article "Learning the Hard Way" (EDUCAUSE Quarterly, Volume 29, Number 4, 2006), Security Task Force Co-Chair Joy R. Hughes and co-author Keith R. Bushey discuss strategic improvements in George Mason University's emergency response.

Business Continuity Planning at George Mason University

vvogel — Wed, 13 Sep 2006 14:02:23 -0500

Learn more about George Mason University's executive enterprise risk management approach in the September 2006 University Business article, "Business Continuity Planning", by Joy R. Hughes (Security Task Force Co-Chair), et al.