Features – Evaluating Foreign and International Legal Databases on the Internet

Mirela Roznovschi is the Reference Librarian for International and Foreign Law at New York University School of Law Library. She holds a M.A. from the University of Bucharest (Romania), a M.L.S. from Pratt Institute, and a Certificate in Internet Technologies from New York University. Her activities include monitoring and evaluating foreign and international legal databases on the Internet, training law faculty and students to use the Internet for legal research, advising developing democracies on the building of electronic law libraries, and training librarians from developing democracies. She is in charge of the library’s home page, Guide to International and Foreign Law Databases. She also serves as a member of the Index to Foreign Legal Periodicals Advisory Committee.

Table of Contents

1. Introduction

2. Criteria of Evaluation:

Accuracy Archiving Author/Publisher Completeness
Cost over Print Format Coverage Currency Language
Licensing Search Quality Source of Data Stability
User Interactivity Workability

3. Conclusion

Introduction

During the last three or four years, legal documents on foreign and international law have begun to appear on the Internet. As a result, information specialists from all over the world have started to evaluate electronic sources of legal data, applying standards both different and the same as those used for print data. The main concern is to achieve the highest quality of access and information throughout the WWW virtual legal library.1 Unfortunately, ultimate standards have yet to be agreed, even though some criteria do already exist – such as The Argus Clearing-House Ratings Systems2, The American Association of Law Libraries Standing Committee on Access to Electronic Legal Information criteria, and NELLCO Criteria Worksheet for Electronic/Intranet Acquisitions. One might ask why it is taking so long to develop them. The answer is simply that it has not been possible to adapt traditional evaluation criteria because the very newness itself of electronic sources demands new standards. Therefore, the hunt is still on.

Criteria of Evaluation

In his “Introduction to Reference Work”, William A. Katz3 discusses the evaluation process for print reference sources – taking into consideration criteria such as purpose, authority, scope, currency, audience, cost, format, objectivity, annotated indices, and guides to reference. Later, in a comment about “Evaluation of Databases”4, he emphasized that “effective evaluation of databases depends on methods similar to those employed for evaluating printed works, such as purpose, authority and scope. In addition, he makes reference to various formats for storage of database data, how data is accessed, how often the database is updated, “what hardware and software is necessary to make use of the database to its fullest, and to the speed and efficiency of search. Because of many unexpected factors, these criteria unfortunately proved insufficient, and sometimes actually impossible, to apply to legal databases in the Internet.

The Internet carries both non-professional and professional information. In the former case, a “site” will contain almost any type of data, with Home Pages usually not disclosing either author or publisher, and the source of the information may be uncertain (for example, neither the reliability nor the authority for the data may be specified – such as in some versions of French Journal Officiel). Reliable legal databases fall in the latter professional category.

There is a distinct difference between a “site” and a “database”. An Internet “site” can be any URL in the Internet, which contains a document (e.g. a treaty, a decision, a law), a collection of data (e.g. International Court of Justice documents at Cornell server), or just an index to some sources gathered by some good- hearted persons. There are many good Samaritans on the Internet. By contrast, “database” means subject-oriented data gathered and maintained in a server by an official entity, displaying a high interactivity with the user as well as search engine capability, data records files, query or sales transactions, product catalogs etc. This is how are constituted the Hague International Court of Justice Offical Database, as well as the databases of the United Nations, ODS, European Union, World Trade Organization, and AUSTLII-Australasian Legal Information Institute. A “database” is a collection of data organized in such a way that its content can be easily accessed, managed, and updated. Generally, a database has its own server hosting HTML, PDF, CGI files. In practice, we work mainly with legal databases using the HTML format and not with Internet sites. Surprisingly, and in fact, the “Internet” has a pejorative meaning in our profession. In summary, to say that we evaluate legal databases in HTML format is, I believe, more accurate than saying we evaluate legal sites on the Internet.

The process of monitoring and evaluating foreign and international legal databases from so many jurisdictions is a question of content expertise and access, and requires certain skills. They range from weighing and authenticating documents found (to use Richard Danner’s words from a recent article “Redefining a Profession”5), identifying value-added features, and engaging in establishing database policies, to understanding copyright issues, maintaining links or implementing mirror sites, and dealing with different languages and various formats. Some consider that “source evaluation is an art,”6 which can only be achieved by first acquiring an extraordinary level of understanding. As we know, reference librarians for foreign and international law have to work with databases located in far away servers, from different jurisdictions, in many languages, in various formats, and displaying different copyright issues – all of which contributes to the complexity of our task.

Though we are still in process of creating the full set of necessary standards, at least for the time being we are able to define those which can guide us to the right information, from the right source, at the right time, and in the form most suitable to use. In addition to the evaluation forms I have designed and use on a daily basis, and which are available at the Home Page of New York University School of Law Library – Guide to Foreign and International Legal Databases, I would like to bring to your attention a checklist for evaluating foreign and international legal databases in HTML format, including the following questions to be settled:

Completeness

Is this a full text database? Or, index and abstracts? Or, index only?

Database completeness is also important because many of them are still under construction or in process of expanding the amount of information. A relevant example is the European Union mega-database in which nearly every month are added new data. Even though non-full text databases are not so acceptable, an index or abstract can often be very helpful in locating the full text of a law in electronic or print format from a foreign jurisdiction. Foreign databases displaying a Summary or an Index of laws are very valuable tools. As examples, I would cite:

  • Gazette du Palais, the French Official Journal, offering a summary of its issues and the famous Les Tables de la Jurisprudence. (Ex: Go to Gazette du Palais –> Documentation juridique –> Les tables de la jurisprudence –> Abandon de Famille). [By comparison with the print 1997 issues, the result is extraordinary].

  • Magnus, a comprehensive fee-based Danish legal database, in the vernacular – with prices ranging from $300 to $450 per month, which allows free access to its valuable index of legal news.

  • TAXLINE -Gazzetta Ufficiale (Italy Official Journal) contains useful daily summaries of Italian laws, decrees, and presidential ordinances. Because of the huge amount of every day changes in Italian law, such an Index is a blessing. Fee-based databases are not always available at our discretion, but they may nevertheless be valuable tools because of their free-to-see indices.

  • COLEX-DATA, a fee-based database, which gives us a valuable Index of the new Spanish laws (last month) and a helpful abstract (See Ultimas Novedades en las Bases de Datos — Jurisprudencia; Legislacion; Derecho Fiscal.

Who is the Author/Publisher

Does the database have standing authority? – Government? University? Legal institution? Private vendor? a Samaritan provider?

In the case of foreign and international substantive law, the ideal situation is that both author and publisher be the same entity: as example – a government, parliament, department of justice, international organization, or a legal institute working under government supervision. A fee-based database of a private vendor-providing guarantee of authenticity of legal documents could be viewed as preferable to a database that may be free but does not provide such guarantee. The quality of the publisher is a guarantee in “creating quality-controlled collections.”7

What is the source of data

Is there a guarantee of content authenticity, e.g., electronic signature on imaged pages – such as at:

Does the database indicate the page of the printed document?

More and more databases are trying to offer the page of the printed document. REFACT is one of them. In Canada, QUICKLAW and the new database launched recently by eCarswell — LAW.PRO are fighting for the rights of pagination. eCarswell have just announced that will offer soon legal documents in html and pdf (the imaged page of the reporter with the printed pagination).

Do we have the imaged version of the official page in ?pdf? format (portable document format)?

The ideal, of course, is to obtain the original version in the vernacular in pdf format (image of the original text) together with the English translation. This is often hard and expensive to acquire – though some international organizations such as the UN, supranational organizations such as the EU, governments such as the German and Swiss are already providing the data in this way. We also see situations where data is only in the vernacular, with no other version provided. Such a case is Mercosur (see Guia del Mercosur and Mercosur.) Nevertheless, it is preferable to have the original version of a legal document, instead of an English translation that may be without any guarantee of accuracy.

Language

Vernacular? Is there an English, French, Spanish, German abstract or an abstract in a well-known language?

Many databases have no legal information in English, but they are provided in other European languages; such is the case of Confederatio Helvetica (the Swiss government database), which offers legal documents in French, Italian, and German – the official languages of the confederation. On the other hand, many Chinese databases such as Ceilaw and Great China Web do not give us any hint about the content in any other language. Some databases provides us with an English service (Juristisches InternetProject Saarbrucken) or with English abstracts, like in the case of GLAW (German Case Law Project).

Accuracy

  • Is the information correct? Fully or partly? Is it comprehensive?
  • Is the content as reliable as its print counterpart?
  • Does the resource have an organization or expert backing it?
  • Are there any obvious errors or omissions in the documents?
  • Are all sides of controversial issues presented or is it necessary to seek alternative views?
  • Are there spelling or grammatical errors?
  • Is the information of consistent quality?

Gene L. Wilkinson stresses the importance of objectivity and the avoidance of “indications of careless or hasty preparation.”8 D. Scott Brandt considers as an important point of source evaluation the ways in which “information gets filtered,”9 written and/or issued by an authoritative source (federal government, reliable organization); authenticated as part of an editorial or peer review process by a publisher; evaluated by experts, reviewers or subject specialists/librarians as part of a collection development. Aimee Glassel10 stressed the importance of selection criteria to filter resources and recommended: “Start with a site recognized for its authority; look for references to new resources; evaluate new resources as either a potential site to review, or as a new source for further scouting.” In any case, it is very worthwhile whenever possible to obtain the views and opinions of scholars from different countries concerning their own country’s databases.

Currency

Currency should be an important consideration, since Internet sources have an advantage over print sources. For online databases, currency also means more than questions such as:

  • Is the document page or Home Page dated?
  • When was the last time updated?
  • Are resources maintained?
  • How often are they updated?
  • Timeliness? 11
  • Does the organization have a commitment to ongoing maintenance and stability of the resource?

Some work is timeless, explains Robert Harris, like the classic novels and stories; other work has a limited useful life because of advances in the discipline, and some work is quickly outdated – as in the field of common law. Therefore, we must decide whether the database information is still of value, and of how much value. Harris stressed: “An important idea connected with timeliness is the dynamic, fluid nature of information and the fact that constant change means constant changes in timeliness. The facts we learn today may be timely now, but tomorrow will not be.”

Often a document includes the date when the information was gathered, or clearly refers to dated information, to a “publication date or a last updated date,” or to a date of copyright.12

To determine the date of last updating is one of the most difficult tasks – especially when dealing with civil law jurisdictions where no date is displayed on the Home Page, and sometimes the date refers to anything but to the question of updating. In many cases, even if webmasters do not care to inform us, the machines themselves can be very informative – because they can neither lie nor ignore this problem due to their built-in software. The Hypertext Transfer Protocol (HTTP) is the language which Web clients and Web servers use to communicate with each other. We can ask the server directly about the currency of data whenever there is a doubt related to any database or a document. Here is an example. In trying to determine when a database was last updated, we use the HTTP protocol. Whenever “talking” directly with the server we use telnet. The technique is to omit http:// but to add 80 at the end of the address, which is the standard port for any server in the world, as in the following example:

1. At the prompt: telnet%
Enter the name of the host you wish to connect to: www.admin.ch 80
Trying 193.5.216.31…
Connected to adminsrv.admin.ch.
Escape character is ‘^]’.

2. At this prompt type the following command (^ means leave a space!):
HEAD^/^ HTTP/1.0

3. Hit enter key twice to leave a blank line. This is what is received:
HTTP/1.1 200 OK
Date: Mon, 11 Jan 1999 21:24:02 GMT
Server: Apache/1.3.3 (Unix) mod_perl/1.16 PHP/3.0.6
Last-Modified: Wed, 16 Dec 1998 07:31:48 GMT
ETag: “31326-f7c-367761e4”
Accept-Ranges: bytes
Content-Length: 3964
Connection: close
Content-Type: text/html

To find out more about a specific file (Ex: http://www.admin.ch/ch/f/rs/rs.html), follow the same steps with the difference that at step no. 2 type in the specific directory’s address:

1. Telnet — the same as the above.

2. Type in the file?s address containing links to Recueil systematique:
HEAD^/^rs.html/HTTP/1.0

3. Hit the enter key twice to leave a blank line:
HTTP/1.1 200 OK
Date: Mon, 11 Jan 1999 21:24:02 GMT
Server: Apache/1.3.3 (Unix) mod_perl/1.16 PHP/3.0.6
Last-Modified: Wed, 16 Dec 1998 07:31:48 GMT
ETag: “31326-f7c-367761e4”
Accept-Ranges: bytes
Content-Length: 3964
Connection: close
Content-Type: text/html

The server notified that the French directory of the database was updated in the same day. The shortcoming of this technique is that if the server indicates the date of change in the physical file, this may not reflect the currency of information.13 The date is updated automatically every time any change is made to the file, no matter how insignificant – as for example adding a comma.

Coverage

Dates covered

Databases cover sometime in full text only the last few weeks, such as in the case of French Official Journal (official version), or the last few years, such as the International Court of Justice database. UN ODS announces what coverage of resolutions is available from 1946, providing the rest of the documents from 1992.

Archiving

  • How long will the provider keep data? Unlimited time? Limited cumulation?
  • Will the data be archived by another provider? Who?
  • Do agreements state archival responsibility?
  • Do agreements permit the library to make/obtain digital copies of content for archiving and for use in perpetuity?

For example, EUR-Lex keeps E.U. Official Journal issues only for 20 days after the date of publication (the period of time was recently extended to almost two months) , while Norway’s legal database Lovdata offers Supreme Court decisions shortly after release by the Court, and each decision will remain available for about two months only.

Search quality

  • Is there a link to an external search engine or is there a search engine embedded in the database?
  • Is there a built in Thesaurus?
  • Ease of use?
  • Search engine capability:
    • Boolean logic?
    • Fuzzy logic?
    • Natural language processing?
    • Concept based searching?
    • Any restrictions (date, court, region)?

Excellent search engines can be seen in:

Serena Fenton rates “Search Engine Intelligence” as one of the most important criteria in evaluating a database.14

Workability

Alastair Smith15 launched this term related to the resource convenience and its effectiveness to use. Workability can include:

  • user friendliness;
  • resource access with standard equipment or special software such as for Japanese and Chinese databases;
  • password;
  • network requirements;
  • browsability and organization;
  • navigation features;
  • a logical manner to locate the resources; (Eur-Lex site map is outstanding.)
  • a good organizational scheme to help (e.g., by subject, format, chronology, etc).

A perfect example of difficulties that could be encountered offers the Italian Supreme Court database — Corte Suprema di Cassazione, a fee based database that costs $1.500/year for one paswword. EasyFind is the software that has to be dowloaded in order to access the database. The downloading can be done with a speciall password, which is different from the one given for accessing the database. The browser and the computer configurations have to be changed whenever accessing the database. The downloading process and the connection to the database took me more than one week and many back and forth emails to Italy. Special assistance from the Library’s computer support was needed. Any information regarding the database was only in Italian. The database handbook, written in 1996, was not covering all user questions.I think that the database has to be adjusted to the new characteristic of the electronic environment in order to be world wide accepted. The difficulties in accessing the database are high and the required configurations are time consuming and primitive.

Stability

Is there a mirror site? Does database use fluctuate during different times of day?

One may be better off to use more than one location such as in the case of CISG (United Nations Convention on Contracts for the International Sale of Goods). The CISG Online Project is maintained by the Institute of Foreign and International Private Law at the University of Freiburg, Germany. It provides the status of the convention, text, cases, bibliographic references and German case law on the United Nations Convention on Contracts for the International Sale of Goods (CISG). There is also the International Trade Database at Pace University CISG-UN Convention on Contracts for the International Sale of Goods with a similar content.

Interactivity with the user

Are there interactive files such as forms for queries, comments, requesting documents and ordering publications (using cgi scripts, perl programs)? Are they helpful? Is there a user support?

Cost over prints format

  • Are there hidden costs?
  • Other costs that are not hidden?
  • How much cost is attributable to:
    1. licensing of the content?
    2. providing access?

My belief is that a database should cost no more than its print analog, and that the ideal is to maintain one paper copy in addition to the electronic version. However, in this transition period, print may cover more than databases, so we may be paying a lot for a database that is still incomplete or growing, even though reliable (UN ODS). It is preferable to have this kind of policy for two reasons – firstly, because electronic databases represent the future format in which legal documents will exist, and because today electronic databases represent an excellent tool for accessing the most up-to-date legal information.

There are different approaches in pricing policy for almost every fee-based database. Many of them offer a subscription for a number of records and require other payments for any additional record, such as in the case of Chinese CEIlaw database.

Licensing

  • Does the database permit fair use of all information for non-commercial educational, instructional and research purposes by authorized users, including unlimited viewing, downloading, and printing?
  • Is the number of simultaneous users limited?

As we know, licensing agreements have become more prominent in the electronic era. A licensing agreement should cover those circumstances which a content owner and user/purchaser agree upon for the use of certain specified content in a digital environment. First, we have to understand the nature of licensing content. After that, we have to decide what is the appropriate type of arrangement.

There is no consistency in the way this matter is handled . TAXBASE accepts multi-users for one password, and a single annual subscription provides unlimited access to all the tax news and documents at a cost of $247.50 per year per tax professional in the company – which means that one “tax professional” can make available his password to more than 10 persons working for him. By contrast, ODS and CELEX only allow one access per password at a time.

In addition, “fair use” does not have the same meaning in Europe as it does in the U.S,. The fair use exemption in Europe is, generally speaking, more limited than in the U.S. For example, in the U.K. it appears that there is no “educational” fair use. A very interesting Home Page – monitoring intellectual property and copyright issues in Europe – can be found at European Commission Legal Advisory Board.

Conclusion

It is becoming clear that the future in providing legal information will be in electronic format. Therefore, it is critically important to establish from the outset clear standards for publication over the Internet. The purpose of this paper is to define standards such as they exist to date, in spite of the fact that much work remains to be done in developing them for international use.

The information and research environments make the evaluation process a problem of content, access, and point of view. The evaluation depends on our patrons’ needs, the realities of our workplace and upon requests. The main concern in the evaluation process is the Quality of Information found in the Internet. Criteria of evaluation are as well starting points in building electronic resource collections, developing Internet-based information services, or instructing users in the effective use of digital information.

***************

Footnotes

  1. Ciolek,T. Matthew. “Information Quality WWW Virtual Library. The Internet Guide to Construction of Quality Online Resources.” [5th of May, 1998; Last updated January 1999]. <back to text>
  2. “The Argus Clearinghouse Ratings systems.” [5th May, 1998]. <back to text>
  3. Katz, A. William. Introduction to Reference Work. Vol 1 Basic Information Sources, Sixth Edition. New York: McGraw-Hill, Inc.1992 p.23. <back to text>
  4. Ibid. p.50. <back to text>
  5. Danner, A. Richard. “Redefining a Profession.” [1998]. <back to text>
  6. Harris, Robert. “Evaluating Internet Research Resources.” 17 November 1997. Online Internet. [April 23, 1998]. <back to text>
  7. 64th IFLA General Conference, August 16-21, 1998. Conference Program and Proceedings. Look for161A UDT Core Programme: Workshop. Entry nr.1 (PDF). <back to text>
  8. Wilkinson, Gene L. “Consolidated Listing of Evaluation Criteria and Quality Indicators.” [1st May, 1998]. <back to text>
  9. Brandt, D. Scott. “Evaluating Information on the Internet.” [1st May, 1998]. <back to text>
  10. Glassel, Aimee. 64th IFLA General Conference, August 16-21, 1998, Conference Programme and Proceeding. Look for 161A UDT Core Programme: Workshop. Entry nr.4 (PDF). <back to text>
  11. Harris, Robert ibid. <back to text>
  12. Kirk, Elisabeth E., “Evaluating Information Found on the Internet.” [1st May, 1998]. <back to text>
  13. Smith, Alastair G. “Testing the surf: Criteria for Evaluating Internet Information Resources.” The Public-Access computer systems Review 8, no.3 (1997) [May 1st, 1998]. <back to text>
  14. Fenton, Serena Jardine.”Information Quality: Is the truth out there?” [23 April, 1998]. <back to text>
  15. Smith, Alastair. “Criteria for evaluation of Internet Information Resources.” [1st May, 1998]. <back to text>
LinkedIn
Posted in: Features, International Legal Research
➤