[PD-dev] urlloader, a web-based abstraction loader

[Stephen Sinclair]

Hello,

I have just submitted patch #, which implements a new loader for Pd.
This allows one to specify in a patch a "base URL" in which missing
abstractions are expected to be found.  The loader downloads an index
at each listed URL, and if it knows about an abstraction that hasn't
yet been loaded, it will download the patch file to a cache and open
it from there.  This is an idea I've been toying with for a while, and
I've given it some consideration on several levels.  I'm not even
convinced it's a great idea, but I think it's something some people
might be interested in.  Lately I haven't had time to work on it, but
I've got it working well enough that perhaps I should publish it
before it starts to just stagnate in my home folder.

I thought I should just send a follow-up email explaining how to use
it and expanding on some of the points listed in the patch comments:

First, to test the patch you'll need to do a few things.

1) Get it compiled and loaded.

The code is in externals/loaders/urlloader.  Put it in your startup
path and make sure it gets loaded.  (pd should post some messages when
it is loaded.)  You'll need the libcurl-dev package installed.

2) Make the files accessible through http.

There is a directory under /urlloader called /remote.  These files
must go to a web server.  For testing, I have a local Apache running
on my laptop, and the files are hosted at http://localhost/pd.  The
/pd folder is just a symlink from /var/www/pd to the /remote folder.
Note that the test patch (test.pd) points to this location, so if you
decide to use a different location, you'll have to modify it.

3) Sign the patches in the remote folder.

The urlloader requires GPG to work (more on this below).  So you'll
need to have gpg working to test this code.  If you have a gpg key,
use it, otherwise make one.  (Not hard.)
You want files ending in .asc, so issue these two commands:

gpg -b --armor -s unknown.pd
gpg -b --armor -s another.pd

4) Open test.pd

It should open the test patch, and then the [another] and [unknown]
objects should be instantiated.  Open them and you'll see that the
actual location of the files is in a cache directory in your /home
folder.


Some points:

- Requires GPG to verify downloaded patches before loading them.

Since this patch essentially allows code downloaded from a website to
run locally on your machine, it can obviously be a screaming security
risk.  Therefore I did not want to publish this code before
implementing this feature.  However, it's got a few ugly user
interface implications.  Of course not everyone knows how to use GPG
(I learned how just for this actually).  But also, what if you don't
have the key to verify a signature?  Currently it just 'posts' this to
the pd window, and you have to go download it.  The person publishing
the abstractions must provide their public key for this purpose.  You
then have to manually import it into your keyring with 'gpg --import'.

- Also verifies the files MD5 sums

I think this is probably redundant with the GPG stuff mentioned above.
 Originally I was just going to use the MD5 sum of a file to verify
its integrity.  However, I realized that trust issues go much deeper
than that, since a loaded abstraction can easily specify another URL
and then download patches from the new location.  You wouldn't want
your computer automatically following and downloading anything it
sees, so I saw it as a necessity to support some kind of trust
infrastructure.  But I left the MD5 code in the patch for now anyways.
 The pd.index file that sits in /remote is just the output of the
'md5sum' command.

On the TODO list:

- Port to non-Linux operating systems.

Currently I developed this on Ubuntu.  It should be portable to other
systems, provided libcurl and gpg work there.

- Determine proper caching rules.

Currently patches are cached for 100 seconds, then downloaded again
after the expire.  This is just for testing purposes.  In real-world a
more intelligent caching system would be necessary.  (That might be a
good use of the MD5 sum in the index file, now that I think about it.)

- Aliases for URLs which could be prepended to object names.
  (e.g., urlA:myobject)

If more than one URL has the same object listed, the first one will be
downloaded, which may as well be considered undeterministic to the
user.  It would be nice to support a way for the user to specify what
URL an object belongs to.  Using some kind of alias prepended to the
object name might be a way of doing it, with the alias specified in
the [baseurl] object.  In fact, these should be related only to the
canvas the [baseurl] objects reside in, whereas right now everything
is pretty much global.  (It would probably require modification of the
g_list data structure, if I understand right.)

- Deal sanely with versioning.  (support svn URLs?)

This brings up a problem with versions.  What if a patch is designed
that depends on an abstraction at some server, but then the owner of
that server makes an incompatible change which breaks the person's
patch.  It would be nice to support some kind of versioning, so you
could say not only, 'this patch depends on xyz', but 'this patch
depends on version 2.5 of xyz'.  I was thinking that one way to
support this might be to include support for subversion URLs, or at
least websvn, so that specific versions of files would be downloaded,
but still easily updated to a newer version.

- Determine whether this is even a good idea.

I'm not completely convinced that downloading patches and running them
directly is the best way to deal with abstractions hosted in the
'cloud'.  It might be better to have a coherent and trusted repository
system like APT.


Please try it and give some feedback.  If you'd like me to commit this
work to the svn, please feel free to grant me access, though I'll
likely continue developing it on my own either way.


Steve