User-agent: * Crawl-delay: 10 Request-rate: 1/10 # Maximum rate is one page every 10 seconds Disallow: /demo/