# CRELoaded Generated Robots.txt # Robot Exclusion File -- robots.txt # Author: CRELoaded Team # Last Updated : September 11th, 2007 User-Agent: * Disallow: /admin/ Disallow: /common/ Disallow: /video/ Disallow: /genre/ Disallow: /mp3/ Disallow: /login/ Disallow: /includes/ Disallow: /MultiPowUploadSource/ Disallow: /nobius_platinum/ Disallow: /Player/ #To prevent only a select bots from any scans #User-agent: crawlers.looksmart.com #Disallow: / #To prenet any images from being scanned #Disallow: *.gif #Disallow: *.jpg #Disallow: *.png #To block main page due to size from bots #Disallow: /index.php #To shut down site completely from bots use # Disallow: /