Ask JADA AI Employee proves to be faster and 85% more accurate than Human Call Center Agents
Summary
A Case Study on Error Rates Among English-Speaking Call Center Agents at Rogers Communications based in North America
This study adheres to fair dealing and privacy standards under Canadian law.
Chances are, if you've engaged with a customer service representative in recent years, you’ve likely experienced subpar service. Recent data indicates that customer service quality has declined due to cognitive burnout, high employee turnover, and the need for constant upskilling [1] [2] [3]. These challenges are further compounded by documented cognitive decline in verbal, mathematical, and matrix reasoning skills [5] [8], making it increasingly difficult for individuals to meet the demands of customer service roles.
Reduced verbal reasoning skills can lead to misunderstandings and miscommunications between customer service representatives and clients, resulting in lower satisfaction. Similarly, a decline in mathematical problem-solving abilities can affect the handling of billing inquiries, technical issues, and other tasks requiring numerical accuracy and logical thinking. Lastly, decreases in matrix reasoning can hinder the ability to recognize patterns in customer complaints and resolve issues efficiently. All of which you will see play a key role in the human error exhibited throughout the results of our study below.
This poses the question: are humans still the best choice for tele-operated customer service roles today?
Study Objective
To assess human error rates, MomentPort Inc. (MomentPort) conducted a study at Rogers Communications (Rogers). In theory, Rogers should have a low error rate due to its North American-based agents, which help mitigate language and cultural differences while ensuring standardized education and training—challenges often encountered in outsourced operations [4] [9]. Furthermore, an employee confirmed that Rogers provides a centralized source for internal documentation, enabling agents to answer questions more accurately and effectively.
Study Methodology
As part of the assessment, a MomentPort researcher conducted ten back-to-back calls to Rogers customer care, posing a common and critical question for which we already knew the answer: 'How fast will my internet speed be if I sign up with Rogers 5G Home Internet?'
This question was chosen because it is both a frequent consumer inquiry and a key factor influencing purchasing decisions. Moreover, it is a straightforward question that should be easily answered by any customer service representative, given that Rogers 5G Home Internet was launched approximately a year ago at the time of writing, meaning Rogers 5G Home Internet is by no means a new product.
The Correct Answer: Baseline for Comparison
To give you a baseline, here is the correct answer:
Rogers offers a 5G wireless-based internet service called Rogers 5G Home Internet. As of July 24, 2024, this service claims to offer speeds around 500 GB at download speeds of up to 100 Mbps and upload speeds of up to 10 Mbps. After exceeding 500 GB, download speeds are throttled to around 10 Mbps and upload speeds will be limited to 2 Mbps. Most notably, all detected video streams will be throttled to 3 Mbps each, equivalent to streaming at 720p HD video. These facts are listed in online marketing material on their website rogers.com, as well as the Terms & Conditions under Internet Traffic Management Practice (ITMP) for Wireless Home Internet, Wireless Business Internet, and Rogers 5G Home Internet [7].
Furthermore, we also discovered, thanks to posts on Rogers' own community forums, that they also throttle speeds on VPN traffic such as OpenVPN and Wireguard protocols, making this service nearly unusable for remote workers [6]. The throttling of VPN traffic was confirmed during a 42-minute call with a Rogers manager who stated that many types of high-consumption traffic, such as gaming, music streaming, and most notably VPN traffic, are throttled.
Results and Analysis
Armed with the knowledge above, we developed a rating scale based on two key questions asked during each of the 10 calls to Rogers within 1.5 hours on July 24, 2024, resulting in 20 total evaluations. Each correct answer accounted for 5% of a total success rate, benchmarked at 100%.
Key Questions Asked
The Questions:
- Did the customer service agent state that the download speed for all detected video streams is throttled to 3Mbps when asked directly?
- Did the customer service agent know that VPN traffic is throttled when asked directly?
The chart below highlights a significant error rate among Rogers' customer service agents when answering basic questions about internet speeds and bandwidth limitations, even when prompted.
Notably, 70% of the human agents failed to correctly state the throttling of video streaming speeds despite the details being documented in internal, public marketing, and legal material. Furthermore, 100% of agents failed to mention the throttling of VPN traffic, requiring 40 minutes of discussions with multiple representatives and advanced technical knowledge to uncover the truth.
This gives human representatives an overall success rate of only 15% in correctly answering the basic questions about internet service speeds.
Comparison with Ask JADA AI Voice
In November 2024, MomentPort released an AI representative called Ask JADA. This product uses a proprietary parsing engine to train JADA’s behavior and memory based on employee documentation, marketing materials, phone calls, emails, and other sources. Ask JADA supports both Voice and SMS (Text Messaging) as communication methods with customers. But in this case, we will focus exclusively on comparing Ask JADA’s Voice module to the performance of human customer service representatives at Rogers.
Training Data for Ask JADA AI
For educational purposes and to facilitate this study, we trained our AI, JADA, using publicly available information from the Rogers website and the 40 minute phone call that was required to learn the truth about VPN throttling. As internal Rogers employee knowledge was not accessible, no additional data was used, meaning the AI operated at less than peak efficacy.
To ensure a fair test, we reviewed transcripts from each call with human representatives at Rogers and asked those same questions to JADA, the AI. JADA consistently provided accurate information about Rogers 5G Home Internet bandwidth limitations and related details based on its initial training. Additionally, JADA utilized our proprietary memory engine – JADA Memory – to learn from each call recording, further enhancing the consistency of its responses in both content and linguistic accuracy.
Sources:
Legal Compliance Statement
The Ask JADA AI employee used in this study was developed using only publicly available data related to Rogers Communications. This data was collected and utilized strictly for educational and scientific research purposes, with the sole intent of benchmarking performance in customer service. No confidential, proprietary, or personally identifiable information was accessed, used, or incorporated in this project.
All related datasets were collected lawfully and have been permanently deleted upon the conclusion of the educational study. At no time was this data used for commercial purposes, nor was it integrated into any product or service intended for commercial deployment.
Rogers Communications is neither a customer, partner, nor affiliate of Momentport Inc., and no such relationship is implied by the inclusion of the Rogers name or trademark in this research. The use of the Rogers name and trademark is solely for descriptive purposes to identify the subject of the public benchmark, in accordance with fair dealing principles under Canadian law.
This research was conducted in good faith and in compliance with all applicable copyright, trademark, and privacy laws. For further inquiries or concerns, please contact Momentport Inc.
Meet JADA. Your business's first AI Representative.
