[Pdweb] puredata.info robots.txt disallows everything

[Hans-Christoph Steiner]

On Jul 1, 2010, at 3:23 PM, Hans-Christoph Steiner wrote:

>
> On Jul 1, 2010, at 4:21 AM, IOhannes m zmoelnig wrote:
>
>> On 2010-07-01 03:42, Hans-Christoph Steiner wrote:
>>>
>>>
>>> Um, why not? There is no magic to home internet connections that
>>> protects them from DDoS.
>>>
>>
>> there is, and it's called "speed".
>>
>> the term "DoS" might be a bit harsh here (as crawlers don't really
>> intend to attack your host), but the effect is the same.
>>
>> fgamsd
>> IOhannes
>
> Oh, you're saying my home cable modem is faster than IEM's internet  
> connection?

What about specifying the large files in the robots.txt file and  
letting the rest be indexed?  It seems to me that /docs, /dev, and / 
exhibition are all just text/HTML and maybe some images so they could  
be allowed.  Or maybe just turn off indexing on /Members and allow it  
everywhere else.

.hc



----------------------------------------------------------------------------

Terrorism is not an enemy.  It cannot be defeated.  It's a tactic.   
It's about as sensible to say we declare war on night attacks and  
expect we're going to win that war.  We're not going to win the war on  
terrorism.        - retired U.S. Army general, William Odom