Have an account?

  •   Personalized content
  •   Your products and support

Need an account?

Create an account

Fax Order Automation with Machine Learning and Deep Learning


Published: July 2019

$4 billion in annual revenue. That’s the value of orders Cisco still receives via fax or email--almost 10% of our total revenue. We have a dedicated team that enters faxed orders into our commerce platform so that customers receive what they need, when they need it.

“Manual processing is costly and stressful, especially at quarter end,” says Gaurav Goyal, Cisco IT commerce architect. Before entering an order, employees need to validate SKUs, configurations, discounts, billing addresses, and so on. Manually entering a single order can take hours--up to several days for large purchase orders (POs) spanning 100 pages or more.

In May 2019, we rolled out a solution that automates fax order entry using deep learning. This case study describes how we’ve automated 65% of fax orders--reducing annual support costs by 50%. The underlying technology is robotic process automation with deep learning.

A three-part challenge

To automate fax order entry, we had to solve three challenges:

  • Extracting the required data (PO number, SKU, etc.) from unstructured document images like faxes and emailed PDFs
  • Validating the accuracy of the data such as discounts and SKUs by checking our other systems
  • Entering the data in our commerce platform

Recognizing data in unstructured documents

Starting with the first challenge, our first thought was to use optical character recognition (OCR), but we quickly realized our folly.

Our customers use thousands of order templates and change them often. OCR can’t tell whether the number in the first column includes the buyer part number, vendor part number, or discount. It can’t recognize that a value spanning two or more columns is actually just one value. It doesn’t recognize whether the address at the bottom is a shipping or billing address, nor does it raise an eyebrow at an 80% discount.

First try: machine learning

In our first experiment, we tried a combination of OCR and machine learning. Initially we used spaCy, an open-source natural language processing algorithm. We trained the algorithm to identify values in the fax orders and map them to typical PO field labels: Billing Address, Item Name, Quantity, Unit List Price, Total Price, PO Number, etc. Then, we used OCR to read the information in the cells.

“Machine learning with OCR worked well for simple fields such as PO number,” says Dharmesh Panchmatia, senior director of eCommerce. “But it didn’t do well with variable fields like customer addresses, for example, which can span 3-5 lines.”

To increase the accuracy of address recognition we added another machine learning algorithm, Naive Bayes. Accuracy increased to 60%, but the solution still could not recognize cell values if:

  • Lines contained multiple cells—for example, quantity and price of a particular item.
  • Information overlapped two or more cells.
  • Labels were in languages other than English.
  • Values didn’t have a label—for example, when a table continued onto a second page.

Second try: deep learning

Next, we experimented with deep learning, a specialized type of machine learning that uses neural networks to make sense of unstructured data. We started by using the single-shot detection (SSD) algorithm to detect the content associated with each label.

To train the system we selected approximately 5000 POs in different formats and used Python LabelImg to graphically annotate each cell with its label--for example, shipping address, PO#, or SKU. Model training requires heavy processing power. Initially, we used a public cloud, later switching to the Cisco UCS C480 ML for better performance. The training continued for millions of iterations.

Then, we used SSD to detect content. The verdict: “SSD worked well for header information but not for other information,” Goyal says. Replacing SSD with Faster Regional Convolution Neural Network (R-CNN) increased accuracy 70 to 75% (see Table 1). But Faster R-CNN could not identify cell values when a single line on the PO contained multiple cells. After 1.8 million iterations, the algorithm stopped improving. We’d reached a plateau.

Table 1. Comparing the accuracy of deep learning techniques for fax order processing

  SSD Faster R-CNN
Classification loss 1.9 1.00
Localization loss .95 .51
Mean average precision .51 .70+

Sorry, no results matched your search criteria(s). Please try again.

Third try--success! Machine learning plus deep learning

The winning approach was combining machine learning (Naïve Bayes and spaCy) and deep learning (Faster R-CNN). “We think of it as machine learning with a deep learning algorithm layered on top,” says Goyal.

Initially, the hybrid solution could accurately read individual cell labels such as PO number, PO date, and payment terms, but not labels that appeared on the same line as other values, such as Quantity and Net Price. Here’s how we trained the system to do what we needed:

  • Annotated nearly 4000 PDF order images using Python LabelImg, drawing a boundary around the attributes that the system missed.
  • Used Faster R-CNN algorithm to identify labels and cell boundaries.
  • Superimposed output of deep learning onto cells (as shown on the right side of Figure 1).
  • Used machine learning to identify the values of missing fields by understanding the field labels on other lines.

With the hybrid approach, accuracy increased from 70% to more than 90%. "Even for the remaining orders processing is faster, "Panchmatia says. "Machine learning captures most of the information, so less work is left for human agents."

High-performance platform: UCS server purpose-built for machine learning

Machine learning requires very fast graphics processing units (GPUs). Initially, we used public cloud servers to train our models. Leasing processing space costs less because the training process was very intensive, time-bound, and didn’t require dedicated compute resources. Performance increased significantly when we switched to Cisco UCS C480 ML server, which is optimized for machine learning and has eight NVIDIA V100 GPUs.

"The UCS was 200-300% faster than the public cloud platform we tried originally," says Ken Schroeder, Cisco IT principal engineer. "We knew that the GPUs we’d use for fax order automation would be idle much of the time, so we decided to build an internal solution in our private cloud that we could offer ‘GPU as a service,’ billing different departments according to their usage."

Entering orders into the commerce system: robotic process automation

Faxed orders sometimes contain errors, such as shipping address, SKU, or invalid configurations. Before we enter the data that the system has recognized, we apply business rules to make sure the data is accurate. For example, suppose the order is for a UCS server. We make a call to one system to make sure the discount is valid, another call to another system to validate the configuration (storage, memory, CPU, services), and still more calls to other systems.

After the PO is validated, we create an order for human agents to review. Agents compare the order and PO to confirm they match. If any information on the order is missing or incorrect, agents can override the automatically generated order. When this occurs, we feed the PO back into the system to train the machine learning algorithms. In this way, accuracy continually improves.

"We're making the process more efficient—but humans remain involved," says Goyal. "The system is an example of augmented intelligence. That is, humans can do more, and enjoy their work more, because machines take over the tedious and mundane work."

We're making the process more efficient but humans remain involved. The system is an example of augmented intelligence. That is, humans can do more, and enjoy their work more, because machines take over the tedious and mundane work.

Gaurav Goyal, Cisco IT commerce architect, Cisco

Business outcomes

The Fax Order Automation solution went live in May 2019. Currently, it automates entry of 65% of fax orders. Results to date:

  • More than 50% productivity increase--freeing staff to be more responsive to customer inquiries and more quickly enter unusual orders.
  • More than 90% overall accuracy.
  • Increased customer satisfaction, the result of 50% faster order cycles and automatic acknowledgement of order receipt.

Next steps: machine learning "as-a-service" for other Cisco teams

We’re currently getting ready to offer machine learning as a service to other Cisco teams. Our customers will be billed about half of what they’d pay for the same GPU capacity from a major public cloud provider. "Training that took three days in the public cloud takes just one day on the UCS C480 ML," says Panchmatia. "Productivity gains are especially important for data scientists because machine learning is a trial-and-error process."

For More Information

Fax Order Automation with Machine Learning and Deep Learning (PDF)

Cisco UCS C480 ML Server

Blog: AI in Cisco IT Operations

Blog: Machine Learning, Beyond the Buzzword