John I. Alber is the full time technology partner for Bryan Cave LLP and also leads that firm’s Client Technology Group. The Client Technology Group includes lawyers, business analysts, web developers, graphics and multimedia professionals, litigation support specialists and others. The group works with Bryan Cave lawyers and clients to develop innovative tools to help prevent or resolve legal problems. John joined Bryan Cave in 1981 after judicial clerkship. In 1988 he left the practice to become CEO of a transportation industry technology company. He rejoined the firm as technology partner in 1999.

Summary

Knowledge management projects undertaken based on good intentions alone can succeed only by accident. Success is far more likely if projects are undertaken with specific, measurable goals in mind. Predicting and then monitoring return on investment is a classic means of increasing the likelihood of success in IT projects. However, rather than using abstract ROI calculations that are too easily manipulated by project proponents, it is preferable to use measures that closely track a firm’s business and the interests of clients. Leverage, effective rate and profit component are such measures. Knowledge management projects should not be undertaken until the business intelligence tools necessary to measure results are in place. It is preferable to undertake knowledge management projects incrementally, both to mitigate risk and to permit managing such projects (using business intelligence tools) toward a higher net benefit to the firm and to clients..

Allocating Scarce Resources for Knowledge Management

The legal literature, both in the United States and abroad, is rife with pronouncements on the value of knowledge management. This is typical:

Knowledge management is an increasingly important source of competitive advantage for organizations. Knowledge embedded in the organization’s business processes and the employee’s skills provides the firm with unique capabilities to deliver customers with a product or service. (Gottschalk 1999).

The legal marketplace has responded to such pronouncements with a wide variety of knowledge management products. Perhaps because of the perceived value of knowledge management, pricing for many of these products is quite high, and relatively unrelated to their development costs.¹ For example, it is possible for a large law firm to spend a half million dollars acquiring a knowledge management suite and then spend hundreds of thousands of dollars more a year maintaining that products suite. And firms are spending such sums nowadays. Moreover, some large firms have also gotten very aggressive at developing their own knowledge management products, in some cases, systems costing millions of dollars. (see, e.g. Jones 2002, Parsons 2004).

Too often, such knowledge management projects are undertaken on good intentions alone. Firm management is gripped with a broad, amorphous sense that is essential to do something about knowledge management.² The result is very often a decision to spend money—and very likely a great deal of money. These investments consume scarce funds that might otherwise be used for marketing, client development, building practices or other activities at the core of a law firm’s business.³ Such investments can be justified only on the assumption that they will have a profoundly positive impact on the operation of a law firm. That assumption, however, needs to be quantified and then tested and the best means of doing so is to do some form of return on investment analysis.⁴

Here is a key question with respect to such analysis, however: do the firms making large investments in knowledge management have any idea what sorts of return on investment they are achieving with such projects? I think the answer, generally, is that they cannot have such information because they lack the tools to measure return on investment for knowledge management products. I believe that the very first step in any knowledge management initiative should be to acquire or create the tools necessary to measure the relative success of such an initiative. Such tools, if selected or constructed correctly, will do far more than merely measure the success or failure of a knowledge management tool suite. They can provide law firm owners with key information about their core business activities.

The ROI Spreadsheet Game

Information technologists are not babes in the woods. They know that savvy managers will demand a return on investment analysis before spending what may amount to millions of dollars on a knowledge management initiative—at least one would hope they would demand that. Professional information technologists have therefore developed a more than passing acquaintance with return on investment techniques. Given a little time and a spreadsheet or two, they can demonstrate a positive return on investment.

The problem with traditional ROI analysis is that it is very rarely predictive. It is most often used as part of the sales process designed to convince firm management that a large-scale project is necessary. And it is hardly ever subjected to a post-project analysis.

Getting funding for an IT project (and in most firms, KM projects fall in the IT domain) is, in the eyes of many IT professionals, a “competition” and ROI analyses are tools to use in that competition:

In the ongoing IT investment drought, technology managers are finding that a return on investment (ROI) analysis is no longer just a helpful tool to assess the viability of a technology investment, but a prerequisite for funding. If you want to win the competition for capital playing out in every firm, it’s a necessity that you build your ROI model right. (Scheer 2002).

As you might expect, a very significant number of ROI analyses done to justify IT projects show a positive result:

ROI numbers often come from the guesswork IT managers use to justify projects to senior management. A recent study from Jupiter Media Metrix points out that e-business ROI studies are often conducted by in-house staff–workers who may not be sufficiently objective or adequately trained to conduct such analysis. The study found that 59 percent of in-house ROI studies generate a positive result and that only 17 percent of the surveyed companies hired outside firms to conduct or oversee their ROI studies. (Mello 2001).

There are many, many ways to calculate what some call “return on investment,” but the classic approach is to determine the net benefits of a project in dollars and then divide that in some manner by the costs of the project. The result is expressed as a percentage. Even within this definition of ROI, there is quite a bit of latitude.

To get an idea of just how much variation is possible in ROI calculations, it will help to examine a scenario and calculate ROI several different ways:

ROI Calculation Scenario

Let’s assume a firm is considering a new knowledge management package that will cost about $1,000,000 to set up. That might be typical of a full-scale, off-the-shelf system for a global firm. Once installed, it will cost about $250,000 a year to maintain but yield $750,000 in alleged annual benefits. The result is a net benefit of about $500,000 a year. Let’s assume that benefits will be measured over a three year period.

You can calculate the expected return on such a project at least three different ways:

Cumulative ROI

Cumulative ROI = 150%

This is calculated as:
Total net benefits/ Initial costs = $1,500,000/ $1,000,000

This formula adds the project’s net benefits over three years and divides by the initial cost. This calculation, often called cumulative ROI (CROI), yields the highest number, making it rather popular. But CROI puts together several years of returns instead of breaking them out. Think of it as your local bank advertising a 3% return on passbook savings committed for three years, when they are really paying just 1% a year. This method also ignores the time-value of money.

Discounted ROI

Discounted ROI = 124%

This is calculated as:
Present value of net benefits/ Initial costs = $1,243,000/ $1,000,000

To calculate Discounted ROI, one first chooses a discount factor—a number that reflects the cost of tying up a firm’s cash—and then applies that discount factor to a stream of revenue (in this case, the three year net benefit stream). Typically, a discount factor approximates opportunity cost, or the return sacrificed by devoting funds to this project rather than to other investments (e.g., marketing). A discount factor of 10% applied to the $1,500,000 three year net benefits yields total benefits of roughly $1,243,000.

Average ROI

Average ROI = 50%

This is calculated as:
Average annual net benefit/ Initial costs = $500,000/ $1,000,000

By averaging the yearly benefits, this ROI determination comes closest to the textbook definition of an accounting rate of return. It is also a conservative number. Like other types of ROI, however, it tells you little about the payback period—how long before the project recovers its start-up costs. In this case, the payback period is two years.

As you can see, changing the calculus can yield widely different results. That is troubling enough in itself. There is plenty of room to bamboozle a technology committee or a management group, just in the math alone. But there is a more fundamental problem with any ROI calculation: while the upfront costs are easy to determine, the net benefits are, essentially, an exercise in creating fiction. The more intangible the benefits involved, the more likely they are to be wrong:

ROI analysis works best for projects that are easy to measure and are likely to yield such tangible cost benefits as lower headcount, reduced transaction costs, or lower inventory levels. ROI is not an effective tool for evaluating e-business initiatives that produce intangible benefits such as customer satisfaction or improved communication. ROI favors cost-cutting initiatives over projects aimed at revenue growth. ROI analysis may do nothing for a company that is grappling with a change in the competitive landscape or overall business environment. For example, research and development projects may not make a strong ROI case, but companies that fail to innovate will eventually be outpaced by competitors or miss growth opportunities. (Mello 2001).

Where there are very tangible figures at hand, ROI calculations can provide some useful information. Say, for example, that a project is aimed at reducing costs associated with storing archived paper files. You can easily calculate the costs of paper, duplication, floor space, etc. that would be avoided by electronic storage of those files. You might have some faith in the benefits calculus in such a case.

It is quite another matter if the benefits case is projecting revenue increases. Though knowledge management projects have sometimes been sold as cost avoidance initiatives (e.g., if you catalog work product so that you can find it more easily, you should be able to create similar work product for less), knowledge management projects are inherently revenue centric. They may lead to a faster, better, cheaper way to do something, but bringing those reduced costs to the bottom line requires tinkering with the very fundamentals of the business—with hourly rate, timekeeping and other factors at the core of a firm’s business model. Such endeavors are inherently speculative—intangible by definition. So what to do—should we abandon any pretense of determining ROI? Should we give up?

Measure What’s Important to the Firm and Clients

I suggest that in setting goals for knowledge management projects, we should strive to relate initiatives as closely as possible to the firm’s core business model and to clients’ interests. We should not accept vague promises of revenue increases. Rather, we should use measures of success that immediately translate into benefits for the firm and clients. Fortunately, there are such measures.

The three measures that accomplish this best in my view are:

(1) Leverage
(2) Effective rate delivered to the client, and
(3) Profit component⁵

Leverage

There are may ways to measure leverage. Perhaps the simplest entails creating a ratio of partner hours to non-partner hours. You can add many subtleties to this calculation, but the net result is still to determine the effectiveness with which work in a particular group is moved to younger or less experienced lawyers.

At first blush, high leverage sounds like a bad deal for clients. However, in a well-managed firm, quite the contrary should be the case. If associates and young partners are well-trained and provided with tools that make it possible for them to accomplish expert work to high standards of quality and to do so sooner than is the case at peer firms, clients benefit by having a larger pool of talent available to do their work. They also benefit by getting a lower effective rate for the work they have done (because of the lower cost of younger lawyers).

Effective Rate

Effective rate is an average of all hours billed to a client. To calculate it, simply divide total fees by total hours on an engagement or for work done by a particular group across engagements. If you agree to a blended rate when serving a client and you do a good job of managing the engagement, then the effective rate ought not to be higher than the agreed blended rate.

Profit Component

When an engagement is concluded (and probably before), it makes some positive or negative contribution to the profits of a firm. Allocate that on a partner basis, or a matter basis or a group basis, and you have a profit component. (See Laws 2003). This can be roughly equivalent to calculating earnings per product or per business unit in a publicly held corporation. Doing it successfully requires the ability to allocate costs very precisely. For example, if you want to look at the profit component for a group of lawyers using a new knowledge management tool, you will have to track costs across all the matters on which they use the tool and then arrive at a profit component for the group. That can be difficult, both technically and culturally (firms have to decide whether they want to know profit components by group).

If a knowledge management product is accomplishing something useful, it should have a positive impact on at least one of the measures above. An example may illustrate the case:

A group of lawyers within the firm specializes in doing work on behalf of regional banks in connection with loans made by those banks. The bulk of the work involves documenting loans. The agreements involved are complex and important to these clients. However, across all of the loan transactions handled by the group there is a good deal of repetition.

The group proposes to develop a knowledge management system jointly with a group of clients that would automate large parts of the documentation process. As transactions come together, the client would enter information into the system, answer questions posed by the system, and, occasionally, consult by phone on the particulars of the transaction.

The hope in using this new system is that far fewer partner hours will be required to complete the transactions. As a consequence, fees to the client can be held down. Indeed, it is hoped that a flat fee billing arrangement will become possible.

Clearly, a goal when designing such a system is to make possible the creation of high quality documents that meet the clients’ needs, and to do so with less partner time involved. Should that happen, leverage and effective rate will surely be impacted to the benefit of the client. If the firm is successful in pricing this automated work correctly, then there should also be an increase in the profit component for the work.⁶

If that proves the case, then this firm will have achieved an optimal result from a knowledge management initiative: the clients will have benefited from lower rates and from a broader pool of talent to apply to their work. Likewise, the firm will have benefited by achieving revenue with a lower cost basis than was previously the case. The more such work the firm can do the more profitable it will become. However, in contrast with the situation very often occurring with work billed by the hour, the firm’s increased profits are not in conflict with the interests of clients – they are simply the result of better management. Hourly billing pits the firm’s interest in increasing hours against the client’s interest in containing hours. This model avoids all that and in the process rewards both sides of the transaction.

The Primacy of Business Intelligence

It is possible to imagine knowledge management projects like the one described above going very well. But it is also possible to imagine them going very badly wrong. In fact, it seems likely that some significant number of knowledge management projects will go wrong from time to time. That is the nature of the beast. Innovation involves risk of failure.

The question is, when such projects go wrong, how do you know that? Project sponsors have a significant incentive to paint a rosy picture, and they may well be the only ones in command of the facts necessary to determine the success or failure of an initiative. Without a light to shine on such projects, you’ll likely be in the dark.

Hence, before any major knowledge management initiatives are undertaken, a firm must first develop the means to obtain critical business intelligence. Put another way, business intelligence tools must be the first knowledge management initiatives undertaken. And such business intelligence tools must give firms the ability to assess the relative success of other initiatives, including knowledge management initiatives. A starting point would be the ability to measure leverage, effective rate and profit component.

My own firm created a business intelligence suite that permits every partner to obtain very detailed information about the clients and matters for which that partner is managing work. Because a vast and well-thought-out data warehouse underlies that suite, it is also possible to analyze leverage, effective rate and profit component for knowledge management projects.

However, it is no longer necessary to create business intelligence tools from scratch. Several firms are now providing data warehouse-based analytic tools that, with some modification, can be used to analyze KM efforts. See for example the tools created by Redwood Analytics and The Satori Group.

Creating, or even buying, business intelligence tools is not a matter for the faint of heart. It requires deep consideration of a firm’s business processes, its cultural values and its goals. And it takes time…a lot of time. Nonetheless, if a firm is to assess the impact of its innovation, such work is indispensable.

Minimizing Risk

Purchasing an off-the-shelf knowledge management package may lead firms into attempting to adopt wholesale, firmwide knowledge management initiatives. Such an approach is hardly a formula for success. It is far preferable to initiate knowledge management projects on a small-scale, one group at a time.⁷ Business intelligence tools can then be used to monitor the groups closely as the initiative progresses. Where results are not as expected, then steps can be taken to redefine processes, substitute different people or otherwise adjust the project to optimize results.

This iterative approach also permits much more granular risk management. Very broad initiatives can get completely derailed by small and very predictable failures that occur early in the process. Breaking a firm’s knowledge management efforts up into smaller pieces not only contains the effects of failure, but it makes successes more easily repeatable. Knowledge management efforts conducted by this method are best viewed as “experiments.” As in any experimental environment, some will succeed and some will fail. The key is to be able to determine which is which and have enough intelligence (both data and insight) to be able to duplicate successes and avoid failures going forward.

Resources

Gottschalk, P (2002), “Law Firm Clients as Drivers of Law Firm Change”, The Journal of Information, Law and Technology (JILT) 2002 http://elj.warwick.ac.uk/jilt/02-1/gottschalk.html.

Gottschalk P, “Use of IT for Knowledge Management in Law Firms”, 1999, The Journal of Information, Law and Technology (JILT), http://elj.warwick.ac.uk/jilt/99-3/gottschalk.html.

Jones, A, “Brobeck Rolls Out Legal Knowledge Management System,” Law.com, April 8, 2002, http://www.law.com/jsp/statearchive.jsp?type=Article&oldid=ZZZNO7UMLZC.

Kay, S. “Cost, Value and ROI for Knowledge Management in Law Firms,” LLRX.com, August 31, 2003, //www.llrx.com/features/kmcost.htm.

Laws, T, “Profit Components,” reprinted from Legal Times, June 30, 2003, http://www.hildebrandt.com/Documents.aspx?Doc_ID=1260.

Martin, K, “Show Me the Money – Measuring the Return on Knowledge Management,” LLRX.com, October 15, 2002, //www.llrx.com/features/kmroi.htm.

Mello, A, “Why ROI Can Sometimes Lie,” ZDNet TechUpdate October 3, 2001, http://techupdate.zdnet.com/techupdate/stories/main/0,14179,2816181,00.html.

Parsons, M, “Knowledge Management: Measuring Success,” LegalIT, January 4, 2004, http://www.legalit.net/ViewItem.asp?id=15487.

Scheer, B, “How to Build the ROI Case for IT Projects,” Internet Week, March 26 2002 http://www.internetwk.com/columns01/bscheer032602.htm.

Footnotes

¹ Pricing is also a function of who is entering the KM arena. Many of the vendors are old line legal vendors, who have developed cash cows over the years. One suspects their consequent dependencies on cash flow distort their value calculations with new products. The marketplace seems ready for an audacious startup that can more closely match price with value. <back to text>

² See Kay 2003 for a defense of the intrinsic value of KM. Kay views the ROI case as a necessity to satisfy firm management. <back to text>

³ This is not to suggest that knowledge management is not also at the core of a law firm’s business. Rather, the point is that allocation of capital involves choice based on a prioritization of needs. <back to text>

⁴ This is far from an accepted proposition among knowledge management professionals. See Kay 2003. <back to text>

⁵ See Martin 2002 for a more thorough look at such measures. <back to text>

⁶ Our own experience at my firm is that business intelligence tools not only help monitor the projects, but they also make it easier to raise ROI by giving partners tools to analyze pricing and staffing decisions. For example, in a real estate lending undertaking very similar to the scenario described above, the partner in charge used BIS tools to closely analyze staffing ratios and pricing, with very positive results. <back to text>

⁷ Vendors may not always make an incremental approach easy. Their preferred pricing models are often based on firmwide enterprise licenses, which can make projects undertaken on behalf of small groups disproportionately expensive. <back to text>