If you need an all-purpose way to block certain crawlers, bots, scrapers etc from your Magento site, here’s a simple way to get the job done. Backup your .htaccess file and add the folowing lines at the top of the file: Block unwanted Crawler Bots ##### RewriteEngine On RewriteCond %{HTTP_USER_AGENT} ^.*(Baidu|spider|Yandex|robot|crawl|wget).*$ [NC] RewriteCond %{HTTP_USER_AGENT} !^.*(Google|MSN|Yahoo|Bing).*$