Willamette Integrated Technology Services (WITS) experienced a number of outages and issues during the month of November, including wifi outages in residence halls, intermittent WUPrint outages, mail group issues and SAGE slowness during class registration. These issues created difficulties for students in both their personal and academic lives. Despite these problems all occurring over the span of less than two weeks, they were unrelated to each other. Since, WITS has fixed each issue and conducted an internal review, which resulted in several plans to improve its technology and systems to prevent such problems from occurring again in the future.
WUPrint was first reported to be down on Nov. 1, and intermittent outages continued through Nov. 6. “The WUPrint outage was eventually determined to be a memory tuning issue with the application in order to handle a heavier load,” said Casey Feskens, Willamette’s director of infrastructure services. Memory tuning is the adjustment of memory in software applications to improve functionality. It was recommended by the vendor for WUPrint because its memory was insufficient for its amount of use.
The cause of the outages were hard to identify, which prompted WITS to contact WUPrint’s vendor, A.N.D. Technologies, Inc., on Nov. 4 for support. With the help, the issue was resolved on Nov. 6. When asked about what the University is doing to prevent this issue from arising again, Feskens said WITS has “discussed looking at other products to compare against the value we are getting from WUPrint.”
Additionally, WITS’ network monitoring system reported wifi outages in buildings Lausanne, Doney and Olin on the morning of Nov. 7. “This outage was determined to be due to damage to the fiber optic cables that connect buildings to the Willamette network,” Feskens said.
Fiber optic cables are just one way in which an area can connect to the internet, similar to the old standard copper wires. The damage was mitigated and connectivity was restored to the affected residence halls in a manner of hours by transitioning service from the damaged cables to other cables between the buildings that were not damaged. This measure is only temporary.
“The vendor has determined that the fiber cannot be repaired due to its age. We are currently pursuing quotes to replace the fiber with newer, higher bandwidth fiber and will replace it as funding becomes available,” Feskens said.
When asked if the University was planning on investing in any new technology to prevent a similar situation from happening again, Feskens said: “Over the last several years, WITS has made significant investments in modernizing our fiber optic cable plant and our network for greater throughput and resiliency. The University has also been investing significantly in best-of-breed web applications and cybersecurity and future technology investments are planned.”
Mail groups were also not working for several students from Nov. 3 to Nov. 4. The issue was the result of an operating system upgrade for outdated software on Nov. 3, “which changed some of the underlying software libraries upon which the mail group system relies,” Feskens said. “While our staff testing following the upgrade did not reveal any issues, some mailgroups with custom access control began denying authorized senders the ability to send to some groups.”
Moving forward, WITS is looking to replace mailgroups with Google Groups by the end of 2020.
SAGE went down for a period of time during junior class registration a week after the WUPrint outage, mail group issue and wifi outage. According to Senior Programmer Analyst Bryan Cook, “The issue was traced to a database connection bottleneck. Each semester at Willamette, we increase the number of students allowed to register at one time. The load caused by the increased number of students overwhelmed the configured number of database connections. The connection configuration was diagnosed and corrected within an hour.”
WITS did not receive any further reports of SAGE slowness or downage during the week of Nov. 18, when registration opened for sophomores.
In the future, WITS plans to “automate SAGE deployment and testing processes,” Cook said. SAGE is updated four times a year to improve functionality, but this process is manual and slow. Automating the process would allow WITS to run simulations to test how well the system can handle a heavier load and verify that the configuration is correct.
“Our staff is always assessing service offerings for improvement and we typically follow up any major outage with a review as to how we can reduce or eliminate repeat issues. Our staff has assessed each of these outages for possible improvements,” Feskens said.
If you have any questions, feedback or suggestions for WITS, email <wits> or call them at 503-370-6004. Phone numbers for individual WITS employees can be found on the WITS staff page on the Willamette website.