Advanced Internet Searching

It was difficult to decide which pre-conference program to attend. It turned out that Advanced Internet Searching with Major Net Finding Aids presented by Greg Notess, was a good choice. Greg Notess, a frequent speaker on Internet searching, is a columnist for ONLINE and DATABASE. And lucky for you, much of the information he presented is available at his Web site, which is frequently updated.

When conducting a search Greg recommends the “Stop and Think” method of search planning:

1. Think about who is the likely source of the information you’re looking for, then search for that site. For example, if you need the population of Montana, you’d go to the Census home page. If you can identify an organization that would produce/collect the information you’re looking for, you need to find that organization.
2. Go to a subject directory like Yahoo.
3. Use a search engine when you’re looking for something very specific.

Greg regularly runs comparison searches on major search engines on a monthly basis. A new search engine, called Northern Light, was the talk of the conference, and made Greg’s top 3 list of search engines in November, along with Altavista and Hotbot. Northern Light ranks number 3 in number of pages indexed.

But each search engine has it’s quirks making it difficult to know what you’re actually searching. Altavista doesn’t always report everything. If there’s a lot of traffic at a particular moment in time, it sometimes sends you only partial results. You can get different numbers from Hotbot depending upon the time of day. If they’re doing a software upgrade, your search results may be affected. The moral is that if you’re trying to be comprehensive, you have to run the search at different times of day, and on different days.

Then there’s the issue of database overlap. There is surprisingly little overlap between the major search engines in terms of which pages are retrieved. Greg ran a test search in October which showed that of 440 hits found, 220 were unique pages. Of those, only 12 pages were found by all four search engines. This makes an excellent case for using more than one search engine.

Another surprising finding was the chart of database change over time. In some cases, searches run a year ago retrieved more hits than those run recently. Hotbot took a dip in search results in September, perhaps due to a hardware upgrade, and recovered in October.

Northern Light, one of the newer search engines has a very large web database. It presents search results into search “folders” based on the subject matter, location or individual host. Northern Light also offers “special collection” material taken from the IAC database, for a fee.
What’s not included in search engines?

• PDF files
• Content requiring log-in, such as many of the newspaper databases
• CGI output, i.e. Department of Labor statistics
• Intranets
• Commercial resources with domain limitations
• Any file with a ROBOTS.TXT file at their site, which indicates to search engines not to index the site, such as the Superintendent of Documents page
• Non-web resources

Among other pre-conference workshops that were well-received was Hope Tillman and Walt Howe’s Web Graphics Design for Librarians and Internet Tips and Tricks.

11% of the people using the Web a year ago are no longer using it, stating that it was too difficult to find what they were looking for.

The average searcher only uses one search engine, varying his search, but not his search engine in an effort to find what he’s looking for.

Greg’s Top 3 Search Engines by Size of Database (11/97)

1. Altavista
2. Hotbot
3. Northern Light

Ask Jeeves tries a natural language approach to Internet searching.

An interesting entry in the search engine field is a new demo site called Rankdex. Rankdex provides you with search results ranked based on how many sites link to the sites retrieved. The problem with Rankdex is that the database is too small and isn’t updated frequently.

Among the suggested multiple search engines, which can be useful if you have very unique search terms, were Dogpile and SavvySearch.

Audio tapes of the general sessions are available from:

Conference Audio Services
3901 Edenvale Place
Oakland, CA 94605
510-562-9731

fax 510-633-0929
email: [email protected]

Be aware that the quality of the audio tapes varied.

“The future is always a mix of things predicted and things unimagined.”

John Coates

‘Everything that frustrates us about the Web today represents job security for tomorrow.”

Reva Basch

Reva’s outline, as well as many of the other Powerpoint presentations, is available at on the Information Today’s Web site.

General Conference Programs

My favorite part of the conference schedule was the timing of the programs. There was a break every 45 minutes, the first one in the morning or afternoon was a full half hour exhibit hall break. I found it much easier to concentrate in these shorter sessions.

Each of the regular conference days included 3 different tracks. On Monday I attended the Intranet Track, Managing Organizational Knowledge, while occasionally hopping into the Tools and Techniques track. It was interesting hearing what some of the large corporations and technology companies were doing with their intranets. Among the lessons learned by one speaker:

• Take charge of the product and make sure you’re a player
• Get a champion at the highest level possible
• Get broad organizational support – need IS and Human Resources.
• Present the project as an experiment
• Find a killer application
• Educate and involve users
• Make friends with IS
• Vent

On Tuesday, the “Searching and the Web” track started with Reva Basch giving an update of interviews and information presented in her book “Secrets of the Super Net Searchers.” According to Reva, there has been a major attitude change by information professional since the book came out. There has been a co-evolution of the net and information professionals.

Some of Reva’s comments –

Is everything on the web? Has gopher, FTP, etc. moved to the Web? Generally yes, except for some very specialized materials and some non-U.S. resources.

There has also been a backlash against over-designed web sites. Just because you CAN do something in html, doesn’t mean you should.

Favorite search engines include:

• Altavista
• Hotbot
• Infoseek
• Excite because of it’s concept based search
• Deja News
• Savvy Search and other multiple search engines such as Dogpile

The hot search engine of the month is Northern Light. But there’s no such thing as brand loyalty when it comes to search engines.

What is the Internet really good for?

• Timely information
• Opinion vs. fact
• Connects you with people
• Provides alternative views

On to the Exhibits

The exhibit hall included a variety of vendors. In some ways, the selection was quite similar to a general library conference. After all, almost all publishers, automation software vendors and online services have some kind of Internet product.

Wrap Up

One thing was made quite clear from the 3 plus days of programs, and the abundance of attendees. The Internet has arrived as a serious research tool, a serious publishing tool and a serious online tool.