Wednesday, September 5, 2007

Email Challenges & Network Conflicts

All Los Rios Computer Users,

As you are probably painfully aware, our Los Rios Windows’ infrastructure has been having difficulty over the past few days. This problem has unfortunately manifested itself in periods when users and servers can not authenticate successfully and are therefore not allowed their rightful service. This has also resulted in outages of Outlook Web Access, Exchange and various other Windows-based services. At this point the underlying problem appears to be resolved but due to the nebulous nature of the problem we not ready to guarantee that all symptoms of this problem are gone. We will be watching closely for the next day or two to confirm that all symptoms of this problem have been eliminated. Please report any new occurrences of this problem to the Help Desk.

A more technical explanation of the problem (if interested): the Los Rios Windows infrastructure is composed of a network of users, workstations, applications, servers, and controllers that use a distributed directory service (Active Directory) to manage the security between all Windows devices and objects. This security mechanism is time-sensitive and requires that all Windows servers and controllers share a consistent and accurate Time source. Unfortunately, one of the primary controllers in the Los Rios network obtained a “bad” Time source yesterday morning (Tuesday) at approximately 8:30 AM. This “bad” time-stamp was then rapidly propagated across the Los Rios Windows’ infrastructure resulting in sporadic authentication problems and Windows’ service outages.

Though this problem was recognized within minutes of its first occurrence, due to the distributed nature the Windows’ infrastructure (directory, controllers, servers, etc.) it was very difficult to find and correct all objects with Time-stamps. At several times over the past two days the problem was believed to have been resolved, only to see it reappear in a different branch of the Los Rios network. At this time, neither our District Office Windows’ Administrators nor Microsoft Technical Support can see any further evidence of Time-stamp problems. We will be monitoring the Windows systems very closely over the next several hours.

Thank you for your patience during this difficult period.

Brian Roach
Director, Technical Services
Los Rios Community College District