Inside Cisco's $2 Billion Blunder

By: Paul Kaihla Issue: March 2002 Print Article | Email This Article
How the world's most admired supply chain screwed up, and how CEO John Chambers plans to fix it.
As corporate humiliations go, it had to be one of the worst. In May 2001, Cisco Systems (CSCO) announced the largest inventory write-down in history: $2.2 billion erased from its balance sheet for components it ordered but couldn't use. The gaffe was made all the more embarrassing by waves of prior publicity about Cisco's brilliant integration of its vast information systems. The network was so responsive, gushed Cisco CEO John Chambers, that the company could close its books in 24 hours, any day of the year. Yet the system wasn't responsive enough to stop building billions of dollars' worth of stuff nobody wanted. Cisco blamed the fiasco on a plunge in technology spending that Chambers called as unforeseeable as "a 100-year flood." If company forecasters had only been able to see this coming, Cisco implied, the supply-chain system would have worked perfectly. But Business 2.0 has learned that, in fact, flaws in the system contributed significantly to the breakdown. Cisco recognized many of these problems even before last year's inventory bubble, and ever since, an elite group of execs and engineers has been working on a top-secret remediation program called eHub. Here's the first inside look at how this ambitious, multimillion-dollar project could help avoid a repeat of last year's disaster ... if it works as planned. Inflating the Bubble During the late 1990s, Cisco became famous for being the hardware maker that doesn't make hardware. Instead, Cisco farms out most of its routers and switches to electronics contract manufacturers. This arrangement has several advantages. For one, it allows Cisco to concentrate on marketing and product innovation. It also liberates Cisco from much of the hassle and expense of maintaining inventory, as Cisco's information systems make it possible to ship fully assembled machines directly from the factory to buyers, more or less on demand. But the Great Inventory Wreck of 2001 highlighted some ugly bugs in the system. Cisco's supply chain is basically structured like a pyramid, with Cisco at the point. On the second tier reside a handful of contract manufacturers -- including Celestica (CLS), Flextronics (FLEX), and Solectron (SLR) -- responsible for final assembly. These manufacturers are fed by a larger sub-tier supplying components such as processor chips (Intel and Xilinx) and optical gear (JDS Uniphase and Corning). Those companies, in turn, draw on an even larger base of commodity suppliers scattered all over the globe. Communication gaps between the tiers eventually got Cisco into trouble. To lock in supplies of scarce components during the boom, Cisco ordered large quantities well in advance, based on demand projections from the company's sales force. What the forecasters didn't notice, however, was that many of their projections were inflated artificially. With network gear hard to come by, many Cisco customers also ordered similar equipment from Cisco's competitors, knowing that they'd ultimately make just one purchase -- from whoever could deliver the goods first. The result was double and triple ordering, which bloated demand forecasts and put the squeeze on component supplies. A missing link in Cisco's supply-chain management system magnified this problem. Suppose Cisco projected sales of 10,000 units of a particular router. Each of the company's contract manufacturers would compete to fill the entire order, and to gain an edge, they often tried to lock up supplies of scarce components. Suppliers would be swamped with orders, but Cisco's supply-chain system couldn't show that the spike in demand

represented overlapping orders. If, say, three manufacturers were competing to build those 10,000 routers, to chipmakers it looked like a sudden demand for 30,000 machines. Cisco became enmeshed in a vicious cycle of artificially inflated sales forecasts, artificially inflated demand for key components, higher costs, and bad communication throughout the supply chain. Eventually, the bubble burst. Better Living Through Automation Cisco's inventory woes highlighted the shortcomings of a communication system that stopped only partway down the pyramid. That's where eHub comes in. As it happens, work on eHub began in 2000, when the last thing on anyone's mind was a slump in demand. Instead, the project was intended to help eliminate bidding wars for thenscarce components. "It was created for scaling upward," says Carl Redfield, Cisco's head of worldwide manufacturing and logistics. "It was put in to ensure that enough material would always be on hand." By the summer of 2000, Cisco had produced a 3-inch-thick specification binder that served as a blueprint for eHub. Its central nervous system would be run by Irvine, Calif., supply-chain integration firm Viacore, which would take Cisco's already advanced supply-chain engine to a new level. Private exchanges that link members of a supply chain are nothing new. In the past, companies like Dell (DELL) and GM (GM) have created electronic hubs that feed supply-chain data to outsource manufacturers and suppliers, and vice versa. These exchanges typically provide a Web interface where vendors manually type in things like sales forecasts, purchase orders, and shipping schedules. The systems are not real-time, and they're plagued by dataentry errors. eHub practically eliminates the need for human intervention. Instead, the system automates the flow of information between Cisco, its contract manufacturers, and its component suppliers. The key ingredient is an XML technology called Partner Interface Process, or PIP. eHub's dozen or so PIPs indicate whether a document requires a response -- and if so, how quickly. For example, a PIP purchase order might stipulate that the recipient's system must send a confirmation two hours after receipt and a confirmed acceptance within 24 hours. If the recipient's system fails to meet these deadlines, the purchase order is considered null and void. Under eHub, Cisco's production cycle begins when a demand forecast PIP is sent out, showing cumulative orders. That forecast goes not only to contract manufacturers but also to chipmakers like Philips Semiconductors and Altera Corp. (ALTR) "Before, if Celestica, Flextronics, and Solectron all came to Philips at the same time, and each said they wanted 10,000 of a certain chip, that was a total of 30,000 chips," says one senior engineer who worked on the eHub project. "Now, Philips can say, 'Hold on, I'm on eHub. I know that total aggregated demand is only 10,000.'" By requiring all the systems in the supply network to talk to each other, eHub ferrets out inventory shortfalls, production blackouts, and other screwups almost as fast as they occur. Complexity and cost have put eHub a bit behind schedule. Cisco originally planned to connect 250 contractors and suppliers by the end of 2001. Instead it linked roughly 60, including Agilent Technologies (A), Hitachi, IBM (IBM), Intel (INTC), LSI Logic (LSI), Motorola (MOT), and Xilinx (XLNX). This year the number should rise to 150 or more, and Cisco's ultimate goal is to integrate as many as 650 supply-chain participants. eHub is just the first stage of Cisco's plans for the future. Ultimately, Cisco hopes to automate the whole enchilada: A customer purchases a product online, and that order goes into both

Cisco's financial database and supply-chain system simultaneously. For now, however, eHub will be plagued by a timeless software limitation -- it's only as good as the data it receives. Garbage in, garbage out. "If the inputs are wrong, the world's best supply chain can't save you," says Steve Kammen, an analyst who covers Cisco for CIBC World Markets. Nevertheless, when the next tech boom comes, Cisco expects eHub to provide the parts it needs; the next bust, eHub might keep it from getting stuck with so much unwanted stuff. Cisco isn't ready to boast about all of this -yet -- but the buzz is building. "I walk into our customers today," marvels Viacore CEO Fadi Chehade. "They say, 'Oh, we heard that you are operating the Cisco eHub. Can you copy it for us?'"