DISQUS

Hello! dmiessler.com | grep understanding is using DISQUS, a powerful comment system, to manage its comments. Learn more.

Community Page

dmiessler.com/ Jump to website »

Subscribe
- New Comments
Community
Top Commenters
- Daniel Miessler
  1031 comments · 2 points
- CarlM
  514 comments · 2 points
- cooperati
  179 comments · 1 points
- arikb
  89 comments · 1 points
- brooksgarrett
  52 comments · 1 points
Popular Threads
- Considering Using danielmiessler.com Instead of dmiessler.com
  3 veckor sedan · 8 comments
- Republicans: The Modern Dixiecrats?
  3 veckor sedan · 8 comments
- Music | Mastodon: Crack the Skye
  4 veckor sedan · 4 comments
- Outbound Number Translation Coming to Google Voice Soon
  3 veckor sedan · 3 comments
- Wireless: WPA2 Enterprise Integration With Active Directory 2008
  3 veckor sedan · 3 comments
Recent Comments
- I think this is a very helpful link. It fixes all the keyboard mapping problems. Even for VMware server 2.0 http://communities.vmware.com/message/1091425
  3 veckor sedan by Mario R Mena
  
  in VMware Server 2.0 For Linux Doesn’t Have a “Send Ctrl-Alt-Del” Button
- Test.
  3 veckor sedan by Daniel Miessler
  
  in Testing the New Domain
- "Yes, it helps people through altered perception of the world, but so do three glasses of wine." ___ Oh, so atheists do not drink alcohol because it is too much like religion?...
  3 veckor sedan by SayBlade
  
  in Atheism Argued Successfully by Two Quotations
- burden is on those who make the assertion, not on those who deny
  3 veckor sedan by Adam
  
  in Atheism Argued Successfully by Two Quotations
- • Theism is not a religion. Atheism is not a religion for the same reason. 'Theism' is an abstract noun which refers collectively to each organized religion which espouses the existence of...
  3 veckor sedan by anti-supernaturalist
  
  in The Definition Of Atheism, And Why It Is *NOT* A Belief

dmiessler.com | grep understanding

dmiessler.com/about/

Jump to original thread »

The Whitehouse.gov Website’s Robots.txt File Has 1839 Lines In It

Started by Daniel Miessler · 7 månader sedan

1839?

I’m sure they’re not trying to hide anything, as doing so with a robots.txt file would be asinine, but 1839 entries is just insane. At some point you have to start wondering whether the content should be online at all if you don’t want it indexed. ... Continue reading »

4 comments

Some Joker
2 år sedan

Looking at most of those entries, it looks like they're excluding pages which look to be designed for text only browsers/screen readers.. nearly every directory ends in /text

Disallow: /asia/2005/photoessay/china/text
Disallow: /asia/2005/photoessay/japan/text
Disallow: /asia/2005/photoessay/korea/text
Disallow: /asia/2005/photoessay/mongolia/text
Disallow: /asia/2005/photoessay/mrsbush1/text
Disallow: /asia/2005/photoessay/mrsbush2/text

and if you browse up one directory, you get the same story with pictures..

I'd say it looks like they are doing it to work around for a poor file structure or possibly to keep search engines from finding duplicate text (although without pictures)

*shrugs* I'm all for pointing out when the administration does something crooked, but I can't see fault in this one.. (granted, I've only checked out 20 or so of the links.. the only one that didn't go anywhere for me was /video/text )

reply