Paul Petruccelli is Chief Marketing & E-Commerce Counsel for Kraft Foods North America, Inc. Paul has been a practicing attorney for Kraft Foods and the Federal Trade Commission for the past 23 years and a constant user of the Internet for a number of years. You should take these suggestions as those of a busy practitioner, but not those of an expert online legal researcher.
This article is a companion piece to Sabrina I. Pacifici’s Getting It Right: Verifying Sources on the Net , published March 1, 2002.
Introduction
The development of the internet and its accompanying online research tools have both simplified and complicated the task of legal research for professionals. The task is simpler in the sense that it is now possible to search simultaneously through multiple sources of potentially relevant information on a topic or issue, culling out those of greatest relevance or interest; saving them; printing them; sharing them with colleagues; embedding them in documents that are being prepared for submission to a court or governmental agency, a client or opponent. All of these tasks are in many ways streamlined and enhanced by the technologies now available to us.
At the same time, locating information that is not only apparently relevant, but also reliable, has been made more difficult by the advent of online legal research. When all of these materials were available in the setting of a physical law library, in actual hard-bound case reports, law journals, treatises and the like, merely picking up the materials provided some assurance of their reliability. An article appearing in a hard-copy publication, edited by legal professionals familiar with the subject matter, disclosing most typically the identity of and some background about the author, published by a familiar legal publishing house of long standing – these were all indicia of reliability so consistently present that they went almost unnoticed by most practitioners.
In the virtual world, things are a bit different. The best sites, of course, make efforts to supply the online equivalent of the same indicia of reliability that were so often available in the offline world. In fact, the LLRX website is a terrific example of a site that strives to accomplish this by practicing what it preaches. They identify their authors, as well as the authors’ qualifications to write on the subjects they address (thus explaining my own disclaimer above). All of the articles are clearly dated. LLRX discloses who the editors and other individuals associated with the site are, as well as information about each of them. The site is well-organized and easily navigable, the links are reliable, there’s a search feature, and so on.
The online world also differs in the amount of chaff one may have to wade through before getting to the wheat. In the online world, you conduct a search. Perhaps you start at www.findlaw.com or one of the other legal search engines that are available.[1] You enter your search query, and the site returns 1,524 links to you. How do you wade through all of this material, and how do you assess whether the information you’ve reviewed is reliable enough to justify stopping your review of the remaining links? One answer, of course, is that if the site returned 1,524 links to you, then your search needs to be narrowed. But suppose you’ve narrowed it to even 75 links, or 50, or 30. Who has time to look at all of this material, searching for that which is actually reliable? If you’re as busy as most of us are most of the time, once you’ve found even a few items that seem reliable, you want to be able to stop right then and there, or at least conclude that you’re on the right track and that further exploration of the additional links returned by the search engine is unnecessary.
In her accompanying article, Sabrina I. Pacifici offers up a checklist of some 20 or so questions to ask about any given site in an effort to assess its reliability. All are excellent questions, and the answers will leave you better informed not only about the site’s reliability, but also about its ease of use and overall appeal to you. Sabrina also supplies a list of resources that can be used to cross-check a site by learning more about the people associated with it, by discovering website reviews that may have commented about the value of the site, and other matters.
If you’ve discovered a new website in your practice area that you’re considering using on a regular basis, a full-blown evaluation using the sources and techniques Sabrina notes is probably time well invested. After all, you want to be able to return to the site again and again for valuable information, and you want to make your quality evaluation once at the outset of your relationship, rather than repeatedly as you visit the site.
But let’s assume that you’re just searching for information on a topic with which you’re not totally familiar, and you run across an unfamiliar source. Let’s also assume you aren’t interested in a long-term relationship with this site, and you have neither the time for nor any interest in conducting a full-blown evaluation. You just want a “quick and dirty” sense of whether this material looks reliable or not.
The Busy Practitioner’s Quick Reality Check
What you need is a short set of criteria by which a web site should be judged. Ideally, these criteria should not be fundamentally different than the ones you’d apply in the offline world. There probably have been almost as many different sets of criteria propounded for this purpose as there have been proponents. By and large, however, most sources would include at least the following among the appropriate evaluative criteria for application in the offline world:
· Accuracy
· Authority
· Objectivity
· Currency
· Coverage
In one way or another, the same criteria are applicable to information found online. The difficulty is that in the online world, where anyone can publish anything at any time and at very low cost, some of these elements are more difficult to assess. Moreover, it is the nature of online research and search engine technology to generate large numbers of links that may be relevant to the topic of the search. To fully evaluate all five of these elements every time we click on one of those returned links and encounter an unfamiliar resource would effectively stop us all in our tracks.
As an alternative to full evaluation using the traditional criteria, I offer the following, even shorter list of evaluative criteria. When visiting an unfamiliar website in connection with research, I would not spend another minute on the site if it doesn’t satisfy the following minimal tests:
· Authority information
· Who wrote or supplied this information, and what can I learn quickly – on the site – about that person or organization?
· Contact information
· Am I given email or other contact information for the author?
· Timeliness information
· Is the information of interest to me clearly dated, so that I can judge its currency?
I’m willing to take a chance on my ability to judge objectivity, accuracy and other factors “on the fly,” as I continue to review and evaluate information on the site and on other sites that I may visit concerning the topic. But if the site isn’t offering me even the minimal information listed above, then it’s just not a facially reliable site in my view and I won’t spend more time on it. If it doesn’t meet my A-C-T test , I’m gone.
Test Driving the Busy Practitioner’s Reality Check
So let’s test out this proposition by actually looking at a couple of websites. I started with a web search query on Findlaw that related to the effect of the Americans with Disabilities Act on website design matters. Since it was a narrow search, it produced a reasonably limited number of hits. After discarding two or three obviously unhelpful sites, I came to The Disability Rights Activist website. Now I know from the name that this is an “activist” website, but maybe that’s what I’m looking for, or maybe I’m interested in learning what the activist community thinks about this issue. So I’ve learned something about the site’s objectivity already, even though it’s not one of my “reality check” criteria.
In any event, let’s assume I’m not dissuaded by the name, and want to apply the reality check test to the site. Interestingly, there’s nothing on the site about this organization, if indeed it even is an organization. If I scroll to the bottom and click to their Homepage , it doesn’t get any better – I still don’t see any information about the people who run this website. Can I get more information? Sure. I can use some of the tips in Sabrina’s checklist to find out who handles the site, what their background is, and a whole host of other information. Am I likely to? No. There may be some very useful information on the site, and I can always stay there if I choose to override my own instincts on the matter. But I’m more likely to go back to the list of sites produced by my search and see what else is available.
So I do. Next on my list is the site for the International Center for Disability Resources on the Internet . Immediately, I’m feeling more comfortable. Why? Because I know the site’s association, which is clearly identified on the page I’ve been taken to (which is not the site’s homepage). That same page clearly shows a link to an “About Us” section, so I go there. It tells me a little bit about the organization (it’s a nonprofit); identifies the Board of Directors and their backgrounds; lists Advisory Board members and their backgrounds; and further identifies the organization’s staff members and their backgrounds. Do I necessarily review all of this material at this particular moment? Maybe not. But do I take added comfort about being on the site from the mere presence of all of this disclosure about the organization and its staff and their various backgrounds and affiliations? You bet I do. And it probably took me less than one minute to reach this conclusion.
So I click back to the article to which I was originally linked by Findlaw. And what does that tell me? Well, right off the bat, the authors are identified at the outset of the piece. If I click on their names, I’m linked to biographical information about them at the end of the piece. That information includes the email address and phone number of one of the authors, although not of the other. It also includes a link to a piece that’s about work that was done by the other author. Not perfect on the issue of Contact information, but not bad.
So far I’m feeling pretty good about Authority information, and reasonably comfortable about Contact information. What about Timeliness information? Not surprisingly, that’s there too. There’s a 1998 copyright at the end of the article, and a disclosure that the piece was originally published in the November 1998 issue of The Internet Lawyer newsletter.
In little more than a few strokes, I’ve satisfied myself that the site is worth examining, and educated myself a bit about the authority and timeliness of the article of interest. On a busy day, that may be as much time as I can invest. If ADA issues are an area of broader or increasing interest to me, I’ll probably bookmark the site, and may return later to one or more of the broader list of questions Sabrina suggests in her article.
Some Additional Considerations
The reality check suggested here is a simple enough test, and it’s effective as far as it goes. But bear in mind that it’s only a reality check. There is a great deal of additional source material available for more comprehensive assessments of the reliability of information you may encounter on the web. In addition, there are some commonsense rules you should follow if you’re engaged in online research and want to feel reasonably confident about its reliability. On the commonsense front, consider the following three questions as you pursue online research:
1. Are you even in the right place? If you’re searching for cases, you should be at the website for the court that originally published the opinions.[2] If you’re searching for statutes, go to the website for the state, or to the Thomas website or to FirstGov for federal materials. Don’t use secondary sources for material that’s readily available online from the actual source.
2. Is this the best source for this information? Are you culling statistics from an article or website you’ve found? Why? Wouldn’t it make more sense to find the real statistical source for the data? Wouldn’t Official City Sites be a good place to look for officially sanctioned information about cities, towns, counties, police departments, and myriad other information of a local nature? Wouldn’t the Census Bureau’s SMSA Data Book be the ideal place to look for statistics on social and economic conditions at the state and local level? Or better still, the Census Bureau’s new American FactFinder website, which even takes the trouble to help you create tables that illustrate your search results?
3. Can you verify the information you’ve found? The fact that someone is quoting a figure to you, or alleging that certain conduct took place, ought to make you insistent on the need for a separate, independent source for the information. For some examples of the importance of verification, as well as a discussion of online information quality in general, see Genie Tyburski’s excellent piece on the Virtual Chase website.
Finally, don’t overlook the wealth of additional source information that’s available on the topic of online information quality. For relatively quick reading, consider the Tyburski piece mentioned above, T is for Thinking by John Henderson for the Ithaca College library, or Diana Botluk’s column from LLRX.
If you really want to invest heavily in this topic, take a look at the Information Quality Virtual Library , with pointers to a variety of criteria for evaluating online information resources. You may also be interested in the Internet Detective . It’s an interactive, online tutorial that teaches the skills required for critical evaluation of online information quality.
A Philosophy for Online Reliability
In the offline world, a lack of information reliability tends to have economic consequences. If a book or treatise or journal is deemed unreliable, consumers stop purchasing the item and it goes out of print. In the online world, while it’s more difficult for such Darwinistic effects to be seen, it’s equally important that they eventually occur. Websites that generate traffic attract advertisers, whose dollars continue to support the websites. If you find that a site doesn’t satisfy even minimal tests for reliability, don’t go there. Eventually, as with similarly unreliable or outdated things in this world, those websites should become extinct.
[1] For truly legal research, I would not recommend beginning with a broad-based search engine like www.google.com. While I like the Google search engine and am a frequent user of it, it really is more effective to conduct legal searches on the law-related search sites, as they are less likely to return unrelated material.
[2] There are mountains of court websites, as well as websites for statutes and regulations at the state and federal levels. For exceptional lists of these sites, I highly recommend The Essential Guide to the Best and Worst Legal Sites on the Web, by Robert J. Ambrogi, Esq.