[Pdweb] puredata.info robots.txt disallows everything

[IOhannes m zmoelnig]

On 2010-06-15 20:33, Hans-Christoph Steiner wrote:
> 
> I'd like to change the robots.txt for puredata.info so that it allows
> everything, and search engines can index it.  Are we going by lazy
> consensus on this?
> 

i would prefer if robots would not constantly pull 4GB of data.
this is mainly relevant for media-data (patches, movies, presentations,
...) which is probably not so relevant for searchbots either.

so i would consent on a robots.txt that does not allow everything for
everyone but rather restrict bots to text-pages.

fmasdr
IOhannes

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3636 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.puredata.info/pipermail/pdweb/attachments/20100628/ef9d558c/attachment.bin>