# Turnitin downloaded a 20MB dmg file, and then # scraped the entire freakin' site over the next # hour. You need to throttle back on those giant # files, and spread the scrapage across multiple # days. User-agent: turnitinbot Disallow: / # I see this bot crawl me often, with no apparent # gain for me or its customers User-agent: NPBot Disallow: /