Internet Performance
Dartmouth has two major network (Internet) providers: Level 3 and VTEL. Level 3's path out of Dartmouth to the Internet backbones is through Vermont and Albany, NY. The network fiber is often buried, with other utilities, alongside major highways; portions were damaged due to road and bridge outages caused by tropical storm Irene. Level 3 had expected that it would be out of service for a week, but was able to make repairs much faster than anticipated; that path is once again open and Dartmouth is sending network traffic via the route.
VTEL uses two paths, one to Boston and the other through Vermont. Dartmouth's VTEL traffic goes to Boston; while Level 3 was down, all Dartmouth Internet traffic was using this VTEL route. On August 29, we discovered congestion associated with the network operating system serving Boston. We adjusted traffic flow through this path and believe it will improve network responsiveness.
While both paths are now working, and the changes should improve network responsiveness, we are contacting other network providers to establish a third link from campus.
Computing Services realizes how important Internet access is for everyone in our community, and we continue to work with our providers to improve your access
William F. Garrity
Associate CIO
(New) Blitz Unavailable
From about 3:00 to 6:00 p.m., all customers of Microsoft Office 365, which Dartmouth uses for our new Blitz system for e-mail and calendars, lost connectivity to the service. The press is reporting that this is the first failure of Microsoft's cloud services in more than a year.
Dell Server Failure; Route Flapping
The Dell server which hosts Citrix and R25 experienced an Ethernet switch failure on its backplane. When that switch failed, the router began to broadcast two addresses on our network. The result was “route flapping.” See Wikipedia for a description of route flapping.
The routers on the network became confused, and all routers began to act up. The result was that systems appeared to be available, and then unavailable. Network Services needed to identify and isolate the server with the failed switch before the network could be stabilized. The problem began around 9:00 a.m. and was resolved around 2:00 pm.
CISCO UCS Failure
During an upgrade to the Cisco UCS (Unified Computing System), one component of the infrastructure serving as the directory for all of the systems that run within the UCS environment experienced a directory file corruption. This was an anomaly that Cisco had not seen before and it took many hours for Dartmouth and Cisco to isolate the problem and to develop the steps to correct it. Dartmouth had successfully upgraded the UCS at the off-campus data center, and we had been expecting the same smooth path in the on-campus datacenter. However, because of the type of problem and how this technology works between the two data centers, we could not failover to the off-campus data center with any confidence that applications would be operable. Most Oracle based systems and on-premises email were down from 7:30 a.m. to 8:00 p.m.
Microsoft Online Services provides document sharing, video and web conferencing, and 10GB of storage for each user. Microsoft’s strong commitment to privacy for users and information security for the institution was a key factor in the decision to select Microsoft over Google and other services.
The new Microsoft service will replace Dartmouth’s “BlitzMail” e-mail system. Although once highly innovative and beloved by many, it is more than 20 years old and no longer meets our needs. We know that Google and its various applications have many supporters. Yet, after careful evaluation, we have decided that Microsoft offers the most secure and best integrated service on the market today. We are confident that it is a robust solution that will allow us to provide modern and protected service to our community.
For more information, see:
