User-agent: *
Disallow: /
# /robots.txt file for gsa-crawler (This is a comment line.) 
User-agent: gsa-crawler (This names the user-agent that the file targets.) 
Disallow: /applications(The gsa-crawler will not be allowed to crawl any CGI files.) 
Allow: /career/training/ce