There are times when we want part of our website to be excluded from Google Search result with reasons, such as when we are setting up a publicly accessible development site or demo site with lots of dummy contents. One way of doing that is to set up a disallow rule in robots.txt to the respective site or add a meta noindex tag to prevent googlebot or any searchbot from crawling those dummy contents.
However, as forgetful as some people are, just like me, when we haven't already put the disallow rule or noindex meta tag onto our sites, those diligent searchbots had already started crawling our sites at full speed. Here is the crawl stat of googlebot on this site over the past few weeks:
I have had 2 subdirectories of this site removed from Google Search listing recently, one of the sub-site was used to showcase my Joomla Jobboard Component which contains Joomla default sample contents and Jobboard Component dummy contents and another one was used to test Sweetcron Lifestream software, because many of its links are unnecessarily cluttering the Google Search result with not-so-worthy contents.
I used Google Webmaster to remove those 2 sub-sites, if you haven't got started using Google Webmaster yet, go do that, it's free and effective!
Once logged in, there are a few sections within the dashboard and since we're dealing with content removal, let's head to Tools ยป Remove URLs and the rest are self-explanatory.
Here are the removed site urls:
As referenced from Google Webmaster Help, you have to make sure that the contents that you'd like to remove meet one of the following criteria:
- Make sure the content is no longer live on the web. Requests for the page must return an HTTP 404 (not found) or 410 status code.
- Block the content using a robots.txt file.
- Block the content using a meta noindex tag.
If you haven't met the above criteria when submitting content removal, you'll get the following and you need to re-submit again.
Subscribe to this comment's feed
