http://danielrm26.disqus.com/dmiessler.com/about/enThu, 08 Feb 2007 08:38:06 -0000http://dmiessler.com/blog/the-whitehousegov-websites-robotstxt-file-has-1839-lines-in-it#comment-4353125Yup, even I noted it sometime back as an excellent sitemap. ;-)DeepakThu, 08 Feb 2007 08:38:06 -0000http://dmiessler.com/blog/the-whitehousegov-websites-robotstxt-file-has-1839-lines-in-it#comment-4353126Ooooh, /secret/ directories. *nods head*ghost16825Fri, 26 Jan 2007 05:03:46 -0000http://dmiessler.com/blog/the-whitehousegov-websites-robotstxt-file-has-1839-lines-in-it#comment-4353123Search in Google for 'robots.txt' shows whitehouse.gov at position 5sergeiThu, 25 Jan 2007 13:11:44 -0000http://dmiessler.com/blog/the-whitehousegov-websites-robotstxt-file-has-1839-lines-in-it#comment-4353124Looking at most of those entries, it looks like they're excluding pages which look to be designed for text only browsers/screen readers.. nearly every directory ends in /text<br><br>Disallow: /asia/2005/photoessay/china/text<br>Disallow: /asia/2005/photoessay/japan/text<br>Disallow: /asia/2005/photoessay/korea/text<br>Disallow: /asia/2005/photoessay/mongolia/text<br>Disallow: /asia/2005/photoessay/mrsbush1/text<br>Disallow: /asia/2005/photoessay/mrsbush2/text<br><br><br>and if you browse up one directory, you get the same story with pictures..<br><br>I'd say it looks like they are doing it to work around for a poor file structure or possibly to keep search engines from finding duplicate text (although without pictures)<br><br>*shrugs* I'm all for pointing out when the administration does something crooked, but I can't see fault in this one.. (granted, I've only checked out 20 or so of the links.. the only one that didn't go anywhere for me was /video/text )Some JokerTue, 23 Jan 2007 17:06:03 -0000