Intermittent IVR Issue
Incident Report for Digital Pharmacist


On Tuesday, May 28 2019, at 9:27 the CS Team notified Engineering Support that some IVR phone numbers were intermittently getting “dead air”. As the day progressed, more clients began to experience the same issue, but would also report that it had begun working again. This made it difficult to resolve due to the intermittent symptoms.

Analysis & Discovery

Reviewing our application monitoring dashboard, it was clear that there was a CPU performance issue on our DBMS server that was occurring every 15 minutes. Since the issue was intermittent and the CPU performance issue appeared to coincide with our DBMS replication service, we attempted several solutions to resolve the issue. However, none of solutions we attempted resolved the issue. Reviewing the metrics from our monitoring dashboard from the last two weeks, we identified a pattern that began Thursday May 23rd where the CPU utilization began to increase incrementally. This coincided with an update made on that date. At this point, we were able to locate the specific change made that was causing the issue and resolve it.

Posted May 29, 2019 - 15:57 CDT

This incident has been resolved.
Posted May 29, 2019 - 15:40 CDT
The issue was found and resolved. One of the changes made to resolve voicemail issues introduced a bug into the system.
Posted May 29, 2019 - 07:55 CDT
We have identified the source of the problem. We continue to investigate specifics within the code.
Posted May 28, 2019 - 17:18 CDT
We are currently investigating this issue.
Posted May 28, 2019 - 14:13 CDT
This incident affected: IVR Services (Digital Pharmacist IVR Services (Legacy Cloud)).