http://webmail.uaa.alaska.edu
http://webmail.uaa.alaska.edu Many individuals experienced at least intermittent problems with their email service the week of July 5 due to difficulties that UAA’s IT Services engineering group had with a major upgrade of the SunOne messaging and directory server.
UAA has planned a major upgrade of the Sentinel mail and directory system for approximately six months. The current SunOne software environment has reached end-of-life and additional hardware performance is required to support the increased messaging load experienced by the university for students, faculty, and staff. The upgrade involved replacing server hardware and upgrading to the newest version of the SunOne Messaging and Directory server which UAA has used for the past five years. A planned outage was announced for July 1-4 over the Independence Day holiday break to do the actual hardware/software replacement and migration of all user message stores. Prior to the planned outage, the engineering staff tested the new hardware and software without incident.
The new hardware/software were installed and all message stores were re-pointed to the new system by Saturday, July 3. Throughout Saturday and Sunday over 120,000 email messages were processed by the new system without incident. On Monday, the new system was subjected to heavy production loads causing record-locking problems which were unacceptable. The record-locking problems were due to the fact that the software could not operate correctly under load with the particular hardware configuration that had been implemented even though everything ran flawlessly during the testing process. The decision was made to implement a planned fallback to return to the old Sentinel hardware/software environment. The fallback required a little over one hour to complete, but it also required re-building of mail account indexes and message stores. The re-build process was unexpected and required 4-5 days to complete. In summary, a planned outage from July 1-4 extended to an unplanned outage from July 5-11.
During the unplanned outage email operation (send/receive) was intermittently interrupted for all users and users of webmail/IMAP mail services experienced more severe service interruptions. Perhaps the most significant impact was that on-line mail folders belonging to webmail and IMAP users were inaccessible for an extended period of time. All email services to all UAA users were returned to normal operations early Sunday, July 11.
A post-project review is planned for this week. Currently, UAA’s Sentinel mail and directory system is operating on legacy hardware and software in a very stable state. Engineering staff will begin re-evaluating and planning for the hardware/software migration which will take place at a future semester break to be announced. Preliminary indications are that future problems can be avoided by replacing a current network file service with dedicated file storage which will require funding.
IT Services annually completes roughly a dozen large-scale projects similar to the Sentinel email and directory system upgrade in the course of routine operations. Planning, design review, testing and vendor review are keys to success in these projects. The engineering and technical staff is relatively small for the size and complexity of UAA’s technology infrastructure and struggle with challenges associated with funding and resource management as do other UAA organizations. Still, most of these major projects, the multitude of minor projects and daily operations take place with an absolute minimum of impact on service availability.
A full incident report by IT Services engineering staff will be published at http://technology.uaa.alaska.edu/kb/Incident_Reports/ later this week. Contact IT Services Call Center at 786-4646 with further questions concerning this or any other technology matter.
