By now you should know that a thing called the "Invisible Web" exists and that the spiders, robots and crawlers search engines use to index Web content cannot find pages on the Invisible Web as a result of things like there being no links to or from a Web page.  But that is not the only reason some Web pages are not found by search engines.  To a certain extent, pages that have "'disconnected URLs'" compose a good majority of Invisible Web content -- about 20 percent according to a study put on by IBM, Altavista and Compaq in 2000 (Sherman and Price 295).

    Yet there are other reasons certain Web pages remain invisible.  While specific terms have been coined to describe these degrees of inaccessibility, for simplicity sake, I will discuss the two basic causes for content ending up on the Invisible Web:

1. Pages cannot or are not indexed by search engines
2. Pages do not want to be indexed by search engines