in order to avoid undesirable articles within the search indexes, website owners can instruct spiders never to crawl specific data files or directories in the regular robots.txt file in the basis directory from the area. Furthermore, a site is often explicitly excluded from the search engine's database through the use of a meta tag unique to robots