Add robots.txt to better crawl content

We should do a better job of directing what pages are crawled. I added a robots.txt to prevent the crawling of legacy. Something similar should be added for teiid.github.com to allow only the crawling of master.