EDUCAUSE | Disaster Recovery Planning

National Campus Safety and Security Project Launched

vvogel — Thu, 21 Feb 2008 16:41:19 -0600

EDUCAUSE has joined with NACUBO and several other higher education associations to launch a national campus safety and security project. The project team will conduct an 18-month study that will examine threats of every nature facing colleges and universities and develop guidelines for campus emergency management plans for prevention, preparedness, response, and recovery. Mark Luker, vice president of EDUCAUSE, underscores the importance of addressing campus threats through a multi-organizational effort: "EDUCAUSE, for example, has investigated cyber security and assisted institutions in solving related problems. Expanding on our past efforts in the most meaningful way to colleges and universities requires more of an enterprise approach. We’re looking forward to this opportunity to integrate the concerns and work of EDUCAUSE with those of our partners in the higher education community who represent other components of campus life."

E07 Podcast: It Can't Happen Here - Developing a Unique Three-Tiered Administrative IT Disaster Recovery Plan

kellywalker — Fri, 23 Nov 2007 22:29:26 -0600

This 36-minute podcast recorded during the EDUCAUSE 2007 Annual Conference features Kenneth Richard Schroyer, ITS Disaster Recovery Strategy Manager, The Pennsylvania State University in a session titled "It Can't Happen Here: Developing a Unique Three-Tiered Administrative IT Disaster Recovery Plan."

The session abstract:

What would you do if a broken water pipe temporally disabled your IT capability, or a fire destroyed your primary or secondary IT facilities? Penn State has developed a plan addressing three unique event levels--incidents, disasters, and catastrophes--with three plan phases, each of which initiates a unique recovery response.

Podcast: Weathering the Storm—Preparing for, Responding to, and Recovering from Emergencies

Carie417 — Mon, 02 Jul 2007 16:58:58 -0500

This one-hour podcast was recorded at the EDUCAUSE 2007 Southeast Regional Conference. This keynote panel discussion, entitled “Weathering the Storm – Preparing for, Responding to, and Recovering from Emergencies,” features Betty Hawkins, program manager at the University of South Carolina, David Sliman, directory of technology at the University of Southern Mississippi, and Frank O’Quinn, deputy policy and IT disaster recovery officer at Louisiana State University. Using examples from their own institutional experiences, they share real advice for emergency preparedness and response, answering the question, “How can we plan and prepare in a way that is realistic and meaningful?”

EDUCAUSE Southeast Regional: Weathering the Storm Panel

llarsen — Wed, 20 Jun 2007 11:05:36 -0500

Summary: Weathering the Storm -- Preparing for, Responding to, and Recovering from Emergencies

This session was recorded for podcast.

General Session
Tuesday, June 12, 2007
Atlanta, Georgia

Session moderator:
Kathryn F. Gates, Chief Information Officer, University of Mississippi
Panel:
Betty Hawkins, Program Manager II, University of South Carolina
David J. Sliman, Director of Technology-Gulf Coast, University of Southern Mississippi
Frank O'Quinn, Deputy Policy and IT Disaster Recovery Officer, Louisiana State University

Abstract
Along with Charley, Ivan, and Katrina, hurricanes and other natural disasters are a part of life in the Southeast. Beyond the weather, many other calamities might hit us at any time. While many of us have been involved in efforts to develop emergency preparedness, disaster recovery, and business continuity plans, often these sit on a shelf and have little to do with how the institution really responds in an emergency. How can we plan and prepare in a way that is realistic and meaningful? How can we collaborate in advance so that we will be prepared to help each other when disaster strikes? What are the critical functions of IT before, during, and after an emergency? How can IT best facilitate long-term recovery efforts? This panel will feature several IT professionals who have experienced disastrous situations first-hand and can offer valuable insights to others.

NOTES

Kathy Gates introduced the three perspectives of disaster recovery and business continuity from three different institutions. The themes of the panel will cover planning, collaboration and partnership, and managing technology in emergencies.

==========
David Sliman, whose campus (University of Southern Mississippi) and home were totally destroyed ("everything south of 100 miles of railroad track was gone"), began the session with descriptions of the hurricane damage, his gratitude for all of the support that continues to be offered, and his lessons learned from the experience.
1) Have your personal life in order (insurance docs and all)
2) Plan how far to evacuate - no more than ½ tank gas away
3) It may be difficult to get back so who will be in charge and who has the right documentation to be in charge. Get in touch with officials ahead of time for the right documentation to make sure you can get back on site.
4) Health and welfare of co-workers should be first and paramount
5) Know how you will contact your employees. It may be that you'll have no cell phone towers. Perhaps only text messages will work. Have alternative meeting places.
6) Communications are important:
a. Develop a phone tree for communications (divide and delegate) NOAA has a call in service (1-800 number)
b. Make sure you have your employee's spouse or relative's contact information
c. Also have your employee's children's information
d. Activate text messaging for everyone
e. Have off-site web services arranged or answering service check-ins
7) Have good records of your inventory for insurance (home and work) and take them with you when you evacuate
8) Secure intellectual property and this includes faculty data/papers/books/
9) Disasters come in many flavors. It is the planning not the plan that is most important
10) Be prepared - ahead of time!

There were a few good things like
- Slab Boils instead of Crab Boils (parties on the slabs of concrete that were left)
- Trunk'r'treat - Halloween was trunk by car trunk instead of door to door
- Sculptures have been created from many of the damaged trees

===========
Frank O'Quinn, who was appointed the LSU IT disaster recovery guy one week before Katrina, hunkered down at home with gas in the car, filled propane tanks, batteries and candles, had peanut butter and extra water, games to play with the children, and waited for the power to go out. After the flooding the campus closed and everyone stayed home and cleaned up from the storm and waited for power to return. Then it was a matter of being of service to those critically affected.

LSU's role in these situations is to:
1) Take care of students and employees first - closed the campus
2) Provide a social services location for a special needs shelter.
3) Use their basketball arena to house the shelter which always opens prior to storms for people to use.
LSU & Katrina was a different situation
- There were more people affected and many more responders arriving.
- No one was in charge of these "many more people"
- Determined tasks, organized volunteers, activated an emergency operations center

The temporary medical operations and staging area (TMOSA) that they set up processed over 40K patients. They had to keep the TMOSA open until these people had some place else to go. They were still open when Rita hit and finally closed after 52 days.

LSU had no detailed written Memorandum of Understandings (MOUs) and was not a federally recognized emergency operations center and so they spent over $1 million on everything from telephones to tracking patients. LSU absorbed that 1 million in hard cash - not reimbursed.

Everyone was very emotionally involved but it was also very important to return to the primary mission
- Classes resumed Tuesday following labor day
- Home football game switched to away
- Need to encourage guests of dorm residents to find other arrangements - Faculty staff students swap to get families out of dorms into homes of LSU community - Everyone gathered on the football field with signs of who they could take

Board of regents - declared any student enrolled in an affected state 4 yr school permitted to enroll in any other state 4 year school
For the enrolled displaced students
- Timing was just right - few missed days
- IT had to modify processes, no fee admits
- Existing WEB portal unfeasible
- No dorm rooms, apartments, houses
- Constantly learning lessons and applying them immediately into the process.

On top of enrolling displaced students they were still running the shelter and the requests after Katrina increased for services such as phone, networking connections, pc laptops printers and more. They went with VOIP phones and felt it was a good decision. Their vendors delivered equipment and Frank reminded us of the importance of maintaining good relationships with vendors.

They took on many tasks that were outside their normal roles - like broadcast emails, call center, patients database, volunteer database (many students), map of campus for media, responders. They continued to adjust administrative applications and had twice daily meetings with the chancellor for status checks.

A few of the changes since Katrina - They now have a formal LSU Emergency Operations Center and Formal MOUs and agreements with state agencies and private sector services for 4 days of food, fuel, and water on campus - with secondary suppliers backing up primary ones.
They also have a full time generator for their basketball arena and have pre-planned emergency logistics.

In 2006 they prepared for repeat hurricanes, especially critical now that people were living in mobile homes. IT disaster recovery now has a tested hot-site for mainframe activities and the Chancellor requested written plans from all units on campus. These plans are being updated for 2007

Observations from their experiences:
- Things will be unpredictable. They won't unfold as you planned and you need to be fluid.
- Priorities change by the minute
- Some projects will be aborted after much effort which will cause disappointment and hurt feelings.
- You have to watch out for ideas being suggested that are illegal (ie HIPPA)
- Keep your supplies stocked
- Think about what people still aren't thinking about. People still don't comprehend the unpredictability and they never believe it will happen to them
- Don't wait for the money
- Do the right thing.

Know the big picture
- your mission
- your resources
- your people

===========
Betty Hawkins had no direct experience with Katrina or Rita but as she discussed,
Katrina was just one disaster but things happen every day to different people around the world. She has worked in the area of emergency preparedness for many years and has seen both disasters and incidents of many types.

Disasters: long term or permanent damage
- Fire
- Floods
- Power failures
- Blizzards & ice storms

Incidents: short term interruption
- Environmental
- Hardware failures
- Software failures
- Sabotage

Experiencing a disaster throws you into continuous process improvement.
- People don't plan to fail - they just fail to plan. Lack of planning adds to the mayhem.
- It is important to build relationships with all groups in your setting. This may include groups beyond the local area.
- Plans are effective if people know about them
- Your emergency planning should include marketing avoidance or otherwise stated: how to prevent an incident or disaster.
- Each of us should also have a plan for home emergencies.

Creating the plan is the easy part. It's important to engage others to think about the following layers of the plan:
- Prevention
- Incident recovery
- Disaster recovery

Lessons she has learned over the years include:
- If you complete the plan today, it will be obsolete tomorrow. Continuous planning and preparation is needed.
- Someone needs to own "disaster recovery" at the institution level and the unit level. They must do it and practice it
- Plan on at least one FTE for your IT organization.
- No two disasters are alike
- Be prepared for emotional fallout
- Understand that panic spreads quickly
- Every organization is different so try different things
- Planning and conducting mock disasters is the best way to demonstrate preparedness
- Create an email suggestion box
- Provide positive recognition - volunteers, best ideas for improvement, certificates

Ideas to share information
- At the University of South Carolina she publishes a newsletter every semester. She brings in local EMS, Fire, Police, Red Cross, etc., to write articles or speak on campus and they love to do this.
- Changing the mindset is important so that people think about the little things they can do everyday - her illustration was to put a spare car key with your ID badge.
- University wide business impact analysis (BIA) - circulating this form will generate thought
- Give brown bag presentations to Business managers and keep it easy to understand
- Start a users group and invite various experts to come and speak
- Get the DR & BC people talking with their counterparts at other colleges and universities. (USC's role in Katrina - in less than 24 hrs the IT group set up a place to do IDs for evacuees and talk to state services)
- Reciprocal agreements between organizations should be looked at very closely and determine what are the resources to manage over time
- Create a web site and remind people to visit often. Make sure it includes what IT will provide for recovery and how IT will help in an emergency.

Panel Discussion
Topic: CIO perspective on emergency planning - How should the CIO prioritize?
- Betty suggested that CIOs think about it now and brainstorm with others/stakeholders to develop plan
- Frank believes that communications comes first (from the business side of the house receipts come first but employees think payroll comes first)
- David said communication plans are first

Comment from Monte Luehlfing: You find yourself doing things you never thought you'd be doing - like running a day care center. You have to take care of people first.

Topic: Disaster Exercises -
- Betty indicated that they do hot site exercises (strictly IT) and they find something wrong every time. The exercises are to find out what's wrong. They are not to document what's right. If your exercise is perfect then you haven't succeeded. You should set up a different goal each time you do an exercise.
- Frank said the minimum is that the smoke detectors go off once a month. However they pay visits to their hot site to bring up the backup regularly. Their A Team tests and finds a number of things that they then work on tweaking.

Question: What should the DR technology be? Should we have a black box or a remote site or something else?
- Frank said if main building goes down the alternate site goes up. However, distance is important, 200 yards away not enough. Multiple sites and farther away may be best.

Question from Sandy Schaeffer: Where does the external motivation come from to do emergency planning?
- Kathy indicated it was the State in her situation
- Betty said it was the recognition of a vulnerable point - (happens because it is important)
- Frank suggested that it comes from within the organization but people are "too busy" to do anything individually so organizations have to bring it down from the top. Plus it is "almost equivalent" to a requirement in the state in LA.

Final Question: What should we take away as action items to do immediately upon return to our institutions or key lessons?
- Depends on where you are - look at your own group and where you are vulnerable - low hanging fruit
- People became leaders after Katrina: grow from where you are
- Emotional impact is significant so you must figure out how to deal with the emotional and psychological issues. Keep staff busy so they don't think about other things.
- LSU has changed everything they do and now asks for a DR budget as well (see http://www.educause.edu/SERC07/Program/12202?Product_Code=SERC07/SESS01)

2007 Enterprise Conference: The Adaptable University

gbayne — Fri, 15 Jun 2007 17:20:48 -0500

This podcast features a keynote address from the 2007 Enterprise Conference in Chicago, Illinois. Our speaker is H. David Lambert, Vice President for Information Services and Chief Information Officer at Georgetown University. His speech is titled “The Adaptable University” and runs approximately 53 minutes.

Institutions of higher education have become increasingly concerned about their ability to maintain critical services to faculty, students, and staff in the event of a major disruption. As we build or buy new information systems, enhance new facilities, and design new programs, we must be concerned with business continuity and creating resilient, adaptable institutions that consider academic sustainability in every aspect of the enterprise.

New Business Continuity Planning Page Posted at EDUCAUSE Connect

cluckett — Thu, 24 May 2007 15:17:06 -0500

EDUCAUSE has identified links concerning business continuity that may be useful to the higher education community on the new Business Continuity Planning resource page, including EDUCAUSE Review and EDUCAUSE Quarterly articles, federal government policies, and university resources.

ECAR Releases New Study on IT and Business Continuity in Higher Education

cluckett — Thu, 29 Mar 2007 17:54:06 -0500

The latest EDUCAUSE Center for Applied Research (ECAR) study, “Shelter from the Storm: IT and Business Continuity in Higher Education,” looks at IT unit readiness to foster and support the functioning of colleges and universities that are challenged by disruption. Responding to a well-documented increase of interest in business continuity and disaster recovery issues among higher education chief information officers (CIOs), ECAR designed the study to inform executives about how institutions approach continuity issues and to identify practices that are associated with good business continuity outcomes.

The study methodology included a literature review; consultation with a select group of CIOs and business continuity experts for the purpose of identifying and validating research questions; a quantitative survey of IT administrators (mostly CIOs) at 340 higher education institutions; postsurvey interviews with 15 executives and IT staff members involved in business continuity; a quantitative survey of institutional business officers (mostly CBOs/CFOs) at 247 member institutions of the National Association of College and University Business Officers (NACUBO); and four case studies looking at business continuity planning and operations Florida State University, New York University, Pace University, UC Davis, and UCLA.

The study is available to ECAR subscribers and through purchase, but the key findings and roadmap are publicly available. Visit ECAR on the Web.

EDUCAUSE Enterprise Conference 2007 Takes a Broad Look at IT's Role in the Institution

cluckett — Fri, 23 Feb 2007 10:35:18 -0600

The 2007 Enterprise Conference—a small, specialized event for leaders and managers of administrative and enterprise IT in higher education—will be held May 23–24, in Chicago, Illinois. View the program and register by April 25 to save money with low, early-bird rates.

The conference will focus on the following topic areas:

Challenges Where IT Can Make a Difference
Project and Change Management/Centralized Services
Security/Disaster Recovery/Business Continuity
Running the Organization
Policy and Governance

View the list of speakers, who will address pertinent topics such as data privacy in higher education, the changing role of enterprise systems, federated identity management, assessment and accreditation challenges, and disaster recovery planning in breakout and plenary sessions.

Browse EDUCAUSE resources on Disaster Recovery Planning, Security Management, IT Governance, and Change Management.

Tune In Feb. 27: Free EDUCAUSE Live! Web Seminar on Selecting a Business Continuity Panning Tool

cluckett — Tue, 20 Feb 2007 16:07:18 -0600

Business continuity and disaster recovery planning is becoming an increasing priority for higher education institutions. To facilitate this planning, many are exploring the use of business continuity planning tools, which establish a planning framework, offer direction in the planning process, and provide a common platform for the various campus groups involved. In this free February 27 EDUCAUSE Live! Web seminar, speakers from three institutions will discuss the business continuity planning tool selection process at their institutions, focusing on the criteria used and lessons learned. Those unable to tune in can visit the archives after the event.

EDUCAUSE2006 Podcast: Statewide Disaster Recovery

Carie417 — Fri, 16 Feb 2007 09:52:14 -0600

In this 49-minute recording from the 2006 EDUCAUSE Annual Conference, we'll hear from Mitchell Dysart and John Ellinger in a session entitled Statewide Disaster Recovery for Ohio's Higher Education. They will share details from a collaboration between The Ohio State University and the University of Cincinnati to establish the necessary infrastructure to support redundant disaster recovery for any higher education institution in Ohio.