To block the search engine robots crawling your website- hackers use search engine too
We all know that we need SEO/SEF website. We rely so much for the search engine or google ranking and even wants our website to be in the first page of each search engine. The problem with this hackers also use search engine. They have robots that crawls your website relentlessly and they have good information about your website vulnerability. Plus the fact that there are Joomla users who wants to use their website to a certain area or locality. For example some people wants to use Joomla in the school of Japan or Australia or just for certain group of trusted people they know or just a local Church. Why would they want to open their website to the world using search engine optimization? Opening the your website to the world also open your website to the hackers from all over the world. Blocking the search engine using codes is not an option for some Joomla users who are non-coders. One of the power of Joomla is that it is becoming more and more friendlier to non-coders. Joomla can dominate its competitors by reaching more people who are non-coders.
Another thing that is good about this option is that Joomla would have a better privacy. Search engine robots don't care about people information be it private or for public use because they are robots after all they just want to do is crawl and do we really know where would they crawl? Hackers can use this crawling ability and it is only sepearted by time that they find a way to use it for their advantage. Hackers find ways everyday researching and developing sophisticated tools. With this option we are one step ahead of them.
Another reason why we want to block this search engine robots is because many Joomla users can only target certain area they reside in. Many Joomla users can't target the world using their website because the cost to maintain that kind of website that serves the world is a lot of money. For example if you want to build a social network only for your Church or small group of people, if you open your website to the world using search engine robots you not only bring hackers all over the world you also increase the bandwidth the RAM of your webhost so you pay more money for the people whom you don't want to use your website.
du you know the robots.txt file?