User-agent: * Disallow: /content/ # This is an infinite virtual URL space User-agent: TurnitinBot Disallow: / User-agent: ConveraCrawler Disallow: / User-agent: QuepasaCreep Disallow: / User-agent: Jetbot Disallow: /