This list is beginning to sound like a promotion for this guy, but frankly, he deserves it. Yes it is Joost de Valk again. The Robots Meta plugin is a great way of configuring your site to allow and disallow the search engines from indexing particular pages.
The reason for this is quite simple. It means that you can advise the search engines what pages are unique and important for them to look at and which are not through adding commands such as Index, No Index, Follow and No Follow helping to avoid duplicate content issues on search engine results page.
No need for extended explanations on how to copnfigure the plugin as Joost has done it already in this video.
Author: Joost de Valk
WordPress Robots Meta Plugin.