[Pdweb] puredata.info robots.txt disallows everything

[Hans-Christoph Steiner]

On Jun 29, 2010, at 11:26 AM, IOhannes m zmoelnig wrote:

> On 2010-06-28 20:01, Hans-Christoph Steiner wrote:
>
>> I'm sure the robots are also interested in not constantly pulling 4GB
>> files down, I imagine they try to avoid that.
>
> but usually they don't really check.

A robot can easily drop the download once its gotten a few megs.  Are  
you sure that the bots are actually downloading the whole file?  I  
doubt they do.

>> I'm fine with a text only robots but it seems not worth the effort  
>> and I
>> wouldn't know how to do it.  I do know that I have many gigs of  
>> files on
>> my own website, and robots are constantly hitting it.  My site is  
>> hosted
>> on my home internet connection and I have never noticed a problem  
>> with
>> robots.
>
>
> that might be the clue: on your homeinternet connection you might  
> never
> get into the range of (D)DoS.


Um, why not? There is no magic to home internet connections that  
protects them from DDoS.

.hc




----------------------------------------------------------------------------

'You people have such restrictive dress for women,’ she said, hobbling  
away in three inch heels and panty hose to finish out another pink- 
collar temp pool day.  - “Hijab Scene #2", by Mohja Kahf