The Invisible Web
also referred to as
The Hidden or Deep Web

by Rebecca Landau, MLS

The term "invisible" refers to information on the web that search engines cannot or will not search.

• Information in a database
• Dynamically generated pages (Responses to a specific query)
  • Dictionary definitions
  • Cataloged library holdings
  • Late breaking news
• Sites requiring a password, login, or fee
• Pages with a question mark, "?" in the URL
• Certain file formats (e.g., pdf, power point, word perfect, etc.)

Web Search Engines

Google and Alta Vista have spiders that search static pages. It is estimated that web search engines only search 40 to 50 % of the web. [Anne Clyde]

Web search engines:

• Do not find dynamic pages
• Respond to a specific query
• Do not search all file formats

General Periodical Sites

Find Articles.com: http://www.findarticles.com/PI/index.jhtml

Find Articles.com is a vast archive of published articles that you can search for free.

Constantly updated, it contains articles dating back to 1998 from more than 300 magazines and journals. This database is funded by advertising and is a good supplement to EBSCO.

Find Articles is text only. There are no pictures or images in Find Articles.

Magportal.com: http://www.magportal.com/

Magportal.com has a help page http://www.magportal.com/help/ that explains the sort functions for retrieving articles (quality, date, publication, category).

Directories and Search Engines

Directories can lead to content that is hidden in a database. Use term "database or databases" when searching a topic in a directory (e.g., Yahoo) or a search engine (e.g., Google) Example: plane crash database

Librarians' Index to the Internet: http://lii.org/

Librarians' Index to the Internet is a directory. Example: literature and databases "library catalog" and databases.

Single Topic Website FindLaw: http://www.findlaw.com/

FindLaw offers information on federal and state laws, bankruptcy, lawsuits and traffic tickets, lawyers and law firms.

Example of a Search in FindLaw:

Clicked on Legal Subjects. Clicked the Subject Heading "Torts and Injuries" Clicked on Databases. Led to a list of Databases. Clicked on CHID Online Database Base and connected with a search screen.

Invisible Web Databases

Animal Search: http://animalsearch.net/

Is a family-safe animal-related sites.

ERIC: http://www.eric.ed.gov/

This is a free bibliographic database relating to education.The database has more than 1.1 million citations going back to 1966 and includes more than 107,000 full-text non-journal documents (issued 1993-2004).

GPO Access: http://www.gpoaccess.gov/

Provides free, on-line use of over 2, 200 databases of the U.S. Federal Government in over 80 areas. Databases are updated based on the print equivalent and generally date back to 1994.

References on the Invisible Web

1. Robert J. Lackie, Those Dark Hiding Places: The "Invisible Web" Revealed, http://www.robertlackie.com/invisible/index.html
2. Invisible Web: What it is, Why it exists, How to find it, and Its inherent ambiguity, http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/InvisibleWeb.html

Books and Articles

1. The Invisible Web: How to find the hidden resources that most search tools skip over, by Walter Minkel, School Library Journal, December 2002. p. 29
2. Gary Price and Chris Sherman. The Invisible Web: Uncovering Information Sources Search Engines Can't See. CyberAge Books, July 2001. ISBN: 091096551X (Paper $29.95).
3. The Invisible Web by Anne Clyde, Teacher Librarian, April 2002, Vol. 29 number 4.

http://www.niedermayer.ca/~ral/

Updated April 11, 2008 by Rebecca Landau.