TDS recently sat down with Jeff, our director of Network Configuration and Technical Support, and asked him to explain these scheduled outages of service.
What is a maintenance window?
A maintenance window is a period designated in advance, generally between midnight and 6 a.m. This is when our technical staff work to grow our network and perform preventative maintenance to keep it healthy and predictable.
When possible, maintenance windows are strategically planned to cause as little inconvenience as possible for our customers and when our usage is at its lowest.
What types of maintenance windows does TDS have?
We typically have standard and emergency changes. Standard changes are planned seven days in advance, communicated to customers, and include activities that expand our network capabilities. Activities like upgrades and network additions (or removals) are usually considered standard changes.
Emergency changes are genuine network emergencies that need to be done that night. This is usually to fully restore a failed or impaired network device to proper working condition. Those changes often come on the heels of an outage or from having worked a network failure during the day so we can get customers back online.
Is the maintenance window always from midnight to 6 a.m.?
It is, but sometimes the work is significant enough that we start at 10 p.m. Our goal is to ensure we get the activities done in a single night.
What are some challenges business customers face with maintenance windows?
Some businesses have nightly backups or run 24-hour operations, so our projects must be done with advance notice. We really work with businesses such as hospitals, schools, financial institutions, and law firms to plan and provide them plenty of notice and to accommodate them where we can.
How long does most work take?
Depends on the complexity. Some simple software upgrades can take 30 minutes. We try to start at the top of the maintenance window (midnight) in case something goes wrong so we have time to get the help we may need to get back online before 6 a.m.
If we are upgrading GPON rings and there are a dozen nodes, it could take the entire night to go through the whole project. That’s an example of when we would use the whole window.
Are there TDS associates who just do maintenance window work?
We do have a small, select number of individuals who only work during the maintenance windows but that consists of our NOC third shift and a couple of folks in Deployment and ATS. Generally, most of the maintenance window work is done by people with an 8 to 5 p.m. shift. Many of our operators will work several maintenance windows in a month to keep up with workload. They come in at 10 p.m., prep for the project, do the work, and then clean up.
We know this is disruptive to their sleep and family schedule, and much of this work is very detail oriented so we need folks to be sharp in these windows. They are making a personal sacrifice that’s necessary for our network and our customers, but we try to afford a lot of flexibility for them to keep a reasonable balance.
What types of projects are done?
Everything from a battery replacement, circuit or backhaul additions, customer or traffic moves, software upgrades, coordinated hardware upgrades … you name it. Anything that affects the network and could potentially have a negative impact to customers is done during a maintenance window.
Where is the work done?
Almost all the work by centralized teams is done at their home office. About 30 to 40% of the time, field services or contractors are supporting the effort on-site at a DSA or central office. These projects are done across the country.
How far in advance do you plan?
Generally speaking, most of our planned work is requested seven days in advance and then we can notify customers five days ahead. There are emergency maintenance projects that need to be done immediately. For example, if we had an outage and we did a quick fix on it, but then a permanent fix is needed, we will do it that night.
How do you track these changes?
We have a calendar where we have anywhere from zero to 35 or more updates a day. First thing every morning, we have a Change Advisory Board that meets for about 15 minutes. It’s really a good way to communicate and it keeps us all in sync.
No comments yet.