Deep Web also called Deepnet, Invisible Web, Undernet or Hidden Web, is a portion of the Internet which belongs to the World Wide Web. Deep Web is not part of the Surface Web, which is indexed by standard search engines. It should not be confused with the dark Internet, or the Darknet file hosting and distribution network.
Most of the information on the Internet is hidden in dynamically generated sites, and so traditional search engines can’t find it. In 2001, the deep web was a few dozen times larger than the superficial Internet. Deep Web, or “deep internet”, is a bunch of online services or sites that are not indexed by search engines. These are mostly legal, but they are not indexed either because of intellectual property or for other reasons relied on by companies.
Netflix, for example, is easy to find online through search engines, but they “don’t see” what’s beyond the homepage. This is an example of a Deep Web service. Many other similar services are also in the Deep Web: we cannot “see” what is happening inside PayPal payment service, banks’ internet banking services, Gmail, online storage services such as Dropbox or OneDrive or within. Messaging applications like WhatsApp, where we can’t see who wrote what.
In essence, any proprietary activity is usually hidden, even though the service may be free. Ordinary web pages can be easily accessed by any user connected to the Internet and are often indexed by traditional search engines. Unlike the surface web, search engine results do not include deep web pages, which are therefore free of regular queries (for example pages that offer paid subscription-based content).
The Internet that is accessible to ordinary users is a large part of our daily lives, but there is another kind of Internet and can be accessed only through programs such as the Tor browser or the “Tails” operating system.
Even though the mystery around it is often associated with illicit activities, most of the Deep Web is harmless, being used for activities such as posting and reviewing articles on blogs, web page design changes, or academic journals.
An example that greatly simplifies the understanding of concepts is that of an iceberg, where the visible part of the water surface is web, followed by the part immediately below – deep web, and the invisible part of the deep is dark web, accessible only to professional divers who are guided to reach those places.
Search engines use a special robot that navigates through hyperlinks and indexes the content of the web pages that appear, placing their content and hyperlinks in their databases. After finding links to other pages on the indexed web page, the search bot navigates through them and indexes the content of each page found, finds new hyperlinks, and hovers over them for indexing;
As a result of the links that go beyond the indexed pages, the number of indexed web pages is constantly increasing. To get to web pages that are not mentioned by other pages, the search engine cannot, because of the content of these pages that is not indexed. Therefore, without knowing the URL of the site or web page of “Deep Web”, a regular user cannot access them.
Deep Web Search Engines:
– Infomine: it was made by several libraries in the USA; information from databases, electronic journals, e-books, newsletters, mailing lists, online catalogs, articles, and other resources.
– The WWW Virtual Library: considered to be the oldest web catalog
– All: search engine that indexes the sites of prestigious universities, which provide resources for study and research.
– Complete Planet: indexes around 70,000 databases whose content varies from agriculture to the military.
– Infoplease: contains encyclopedias, almanacs, atlases, and biographies.
– DeepPeep: access the search within several domains: auto, aviation, books, biology, hotel, jobs, rentals.
– IncyWincy: Use other search engines and then filter the results.
– DeepWebTech: offers five search engines for certain topics; among the fields covered are science, medicine, and business.
– Scirus: powerful search engine focused only on science; indexes journals, personal sites of scholars, course materials and institutional networks.
– TechXtra: search engine focused on engineering, mathematics and IT; indexes news from these fields, job offers, technical reports, electronic documents, study and research resources, other relevant articles and information.