To stay away from undesirable material inside the look for indexes, webmasters can instruct spiders to not crawl selected files or directories with the common robots.txt file in the foundation directory in the area. On top of that, a web page might be explicitly excluded from the search engine's database through the use of a meta tag precise to rob