Why librarians, not computer scientists, should rule the 'Net
R. Anders Schneiderman, Ph.D.

Arecent article in Business Week titled "Has The Net Hit The Wall?" complained that it is harder and harder to find anything on the Internet. One solution that holds great promise, it said, is using artificial intelligence to catalog the World Wide Web.

Meanwhile, back at the lab, scientists were finding that easier said than done. The National Center for Supercomputing Applications (NCSA), the people who brought us the first popular software for creating and browsing the Web, tackled a relatively small collection of documents (10 million abstracts from an engineering library). But even this small set overwhelmed their powerful workstation computers; eventually, they had to run their programs on a massive supercomputer for four days.

NCSA’s experience is a good reminder of the central problems with the Internet. Most of us think of libraries as quaint, antiquated places, home of "Marian the Librarian." The reality is that librarians have a lot to offer the Information Age.

Librarians have been managing complex information for over 200 years. If we were smart we’d let librarians rule the Internet.

Let’s start with the issue of searching. Until recently, computer scientists argued that the best way to search for information on the Web was by using keyword searching: You type in a word or two, and the computer searches documents for them. But keyword searching often fails miserably.

If I’m interested in poems about love, what do I search for? If I simply searched for "love," I would miss many famous love poems. Clearly, keyword searching isn’t enough. Information needs to be catalogued.

Given how quickly the Web grew, no systems of cataloging would have worked perfectly. But if librarians had been in charge, they would have insisted that every Web author have access to simple programs that helped them briefly catalog any document or collection of documents they put up on the Web.

That way, every document would have at least been identified by author, title, date and a subject heading, according to at least one standard scheme of categorization. It would have given us a fighting chance to find the information we really need no matter how vast the Web becomes.

There are a number of similar issues where librarians would have saved us from pain and suffering. For example, one of the really irritating aspects of the Web is that if someone moves their Web site, there is no easy way to find it. This is because it never occurred to the Web’s creators that documents might move, and so they didn’t put in a way to keep track of them.

Perhaps the most tragic aspect of having computer scientists, rather than librarians, rule the ‘Net is a result of the differences between the cultures of these two professions. Both believe in providing information for free, but they do so using very different methods.

Computer programmers operate by what might be called the "treehouse" ethic of sharing. The ‘Net contains a wealth of computer resources—programming languages, programs, Frequently Asked Question (FAQ) lists—that are free for the taking. But at the same time, there is no sense that everyone should have the right to join the club.

Libraries, in contrast, are built around the idea that they need to serve everyone. They focus on resources that everyone will be able to use, and they strongly believe in ensuring universal access.

Unfortunately, far from being in charge of the rapidly expanding ‘Net, libraries and librarians are simply struggling to survive. While the federal government pours millions into questionable experiments with "digital libraries," funding for real libraries continues to suffer.

The ‘Net also poses a direct threat to libraries through the battle over "fair use." Libraries work because they are allowed to freely lend out books and other items they have purchased. However, on the World Wide Web, if you make one copy freely available, you’ve essentially made millions of free copies. Not surprisingly, the publishing industry wants to radically restrict "fair use," outlawing making any freely available copies.

In the long run, the only way the ‘Net will rise to its true potential is if librarians become an integral part of the discussion of the ‘Net’s future. In the meantime, we need to fight to make sure that libraries survive and thrive in the new Information Age, and we need to start giving librarians the respect they are due.

This article appeared in the July 11-17, 1997, issue of the Dallas Business Journal. It has been reprinted with the permission of the Dallas Business Journal. Any further reproduction in whole or part is strictly prohibited. Copyright 1997 by the Dallas Business Journal.