User-agent: Mediapartners-Google* Disallow: User-Agent: Googlebot #Disallow: / Disallow: /comments Disallow: /user Disallow: /poll Disallow: /print Disallow: /search User-Agent: Cogger #Disallow: / Disallow: /comments Disallow: /user Disallow: /poll Disallow: /print Disallow: /search # Alexa Archver, allow them User-Agent: ia_archiver Disallow: /comments Disallow: /user Disallow: /poll Disallow: /print Disallow: /search # Internet Archives open source crawler # Has gone nuts on us before. User-Agent: os-heritrix Disallow: /comments Disallow: /user Disallow: /poll Disallow: /print Disallow: /search User-agent: YahooFeedSeeker Disallow: /comments Disallow: /user Disallow: /poll Disallow: /print Disallow: /search User-agent: BlogPulse Disallow: /comments Disallow: /user Disallow: /poll Disallow: /print Disallow: /search User-agent: kinjabot Disallow: /comments Disallow: /user Disallow: /poll Disallow: /print Disallow: /search User-Agent: Teoma #Disallow: / Disallow: /comments Disallow: /user Disallow: /poll Disallow: /print Disallow: /search User-Agent: Sphere Scout #Disallow: / Disallow: /comments Disallow: /user Disallow: /poll Disallow: /print Disallow: /search User-Agent: * Disallow: /