Published: July 2019
$4 billion in annual revenue. That’s the value of orders Cisco still receives via fax or email--almost 10% of our total revenue. We have a dedicated team that enters faxed orders into our commerce platform so that customers receive what they need, when they need it.
“Manual processing is costly and stressful, especially at quarter end,” says Gaurav Goyal, Cisco IT commerce architect. Before entering an order, employees need to validate SKUs, configurations, discounts, billing addresses, and so on. Manually entering a single order can take hours--up to several days for large purchase orders (POs) spanning 100 pages or more.
In May 2019, we rolled out a solution that automates fax order entry using deep learning. This case study describes how we’ve automated 65% of fax orders--reducing annual support costs by 50%. The underlying technology is robotic process automation with deep learning.
To automate fax order entry, we had to solve three challenges:
Starting with the first challenge, our first thought was to use optical character recognition (OCR), but we quickly realized our folly.
Our customers use thousands of order templates and change them often. OCR can’t tell whether the number in the first column includes the buyer part number, vendor part number, or discount. It can’t recognize that a value spanning two or more columns is actually just one value. It doesn’t recognize whether the address at the bottom is a shipping or billing address, nor does it raise an eyebrow at an 80% discount.
In our first experiment, we tried a combination of OCR and machine learning. Initially we used spaCy, an open-source natural language processing algorithm. We trained the algorithm to identify values in the fax orders and map them to typical PO field labels: Billing Address, Item Name, Quantity, Unit List Price, Total Price, PO Number, etc. Then, we used OCR to read the information in the cells.
“Machine learning with OCR worked well for simple fields such as PO number,” says Dharmesh Panchmatia, senior director of eCommerce. “But it didn’t do well with variable fields like customer addresses, for example, which can span 3-5 lines.”
To increase the accuracy of address recognition we added another machine learning algorithm, Naive Bayes. Accuracy increased to 60%, but the solution still could not recognize cell values if:
Next, we experimented with deep learning, a specialized type of machine learning that uses neural networks to make sense of unstructured data. We started by using the single-shot detection (SSD) algorithm to detect the content associated with each label.
To train the system we selected approximately 5000 POs in different formats and used Python LabelImg to graphically annotate each cell with its label--for example, shipping address, PO#, or SKU. Model training requires heavy processing power. Initially, we used a public cloud, later switching to the Cisco UCS C480 ML for better performance. The training continued for millions of iterations.
Then, we used SSD to detect content. The verdict: “SSD worked well for header information but not for other information,” Goyal says. Replacing SSD with Faster Regional Convolution Neural Network (R-CNN) increased accuracy 70 to 75% (see Table 1). But Faster R-CNN could not identify cell values when a single line on the PO contained multiple cells. After 1.8 million iterations, the algorithm stopped improving. We’d reached a plateau.
Table 1. Comparing the accuracy of deep learning techniques for fax order processing
|Mean average precision||.51||.70+|
Sorry, no results matched your search criteria(s). Please try again.
Third try--success! Machine learning plus deep learning
The winning approach was combining machine learning (Naïve Bayes and spaCy) and deep learning (Faster R-CNN). “We think of it as machine learning with a deep learning algorithm layered on top,” says Goyal.
Initially, the hybrid solution could accurately read individual cell labels such as PO number, PO date, and payment terms, but not labels that appeared on the same line as other values, such as Quantity and Net Price. Here’s how we trained the system to do what we needed:
With the hybrid approach, accuracy increased from 70% to more than 90%. "Even for the remaining orders processing is faster, "Panchmatia says. "Machine learning captures most of the information, so less work is left for human agents."
Machine learning requires very fast graphics processing units (GPUs). Initially, we used public cloud servers to train our models. Leasing processing space costs less because the training process was very intensive, time-bound, and didn’t require dedicated compute resources. Performance increased significantly when we switched to Cisco UCS C480 ML server, which is optimized for machine learning and has eight NVIDIA V100 GPUs.
"The UCS was 200-300% faster than the public cloud platform we tried originally," says Ken Schroeder, Cisco IT principal engineer. "We knew that the GPUs we’d use for fax order automation would be idle much of the time, so we decided to build an internal solution in our private cloud that we could offer ‘GPU as a service,’ billing different departments according to their usage."
Faxed orders sometimes contain errors, such as shipping address, SKU, or invalid configurations. Before we enter the data that the system has recognized, we apply business rules to make sure the data is accurate. For example, suppose the order is for a UCS server. We make a call to one system to make sure the discount is valid, another call to another system to validate the configuration (storage, memory, CPU, services), and still more calls to other systems.
After the PO is validated, we create an order for human agents to review. Agents compare the order and PO to confirm they match. If any information on the order is missing or incorrect, agents can override the automatically generated order. When this occurs, we feed the PO back into the system to train the machine learning algorithms. In this way, accuracy continually improves.
"We're making the process more efficient—but humans remain involved," says Goyal. "The system is an example of augmented intelligence. That is, humans can do more, and enjoy their work more, because machines take over the tedious and mundane work."
“We're making the process more efficient but humans remain involved. The system is an example of augmented intelligence. That is, humans can do more, and enjoy their work more, because machines take over the tedious and mundane work.”Gaurav Goyal, Cisco IT commerce architect, Cisco
The Fax Order Automation solution went live in May 2019. Currently, it automates entry of 65% of fax orders. Results to date:
We’re currently getting ready to offer machine learning as a service to other Cisco teams. Our customers will be billed about half of what they’d pay for the same GPU capacity from a major public cloud provider. "Training that took three days in the public cloud takes just one day on the UCS C480 ML," says Panchmatia. "Productivity gains are especially important for data scientists because machine learning is a trial-and-error process."