[PD-dev] new loader replacement progress report

[Claude Heiland-Allen]

Hi everyone,

I've been working on some algorithms to eventually replace the way Pd 
finds and loads and creates objects.  Sorry for the length of this mail, 
but you can skip some sections if you don't care about implementation 
details and/or user interface details :)


Table of Contents
   * Specifications
   * Example Loaders
   * Clearer Error Messages
   * Test Inputs
   * Test Outputs


Specifications
--------------

As agreed by everyone so far, the new load order will be roughly:

     for path in paths do -- the core does this bit
       for loader in loaders do
         loader(path, libray, object)

instead of the old, problematic:

    for loader in loaders do
      for path in paths do -- the loader does this bit
        loader(path, library, object)

I restructured the loader plugin mechanism to be (pseudocode):

    typedef err libloadfun(symbol *dir, symbol *lib);
    typedef err objloadfun(symbol *dir, symbol *lib, symbol *obj);
    struct loader {
      symbol *name;
      float priority;
      libloadfun *libfun;
      objloadfun *objfun;
    }


Example Loaders
---------------

   -- binary loader is super simple
   Loader.new("bin", 70,
     function(dir, lib)      return loadbin(dir, lib) end,
     function(dir, lib, obj) return loadbin(dir, obj) end)

   -- directory loader is needed if there is no main library file
   -- the core code checks the results to see if it succeeded
   Loader.new("dir", 10,
     function(dir, lib)
       if dir:sub(-1 - #lib, -1) == ("/" .. lib) then
         return function() end
       else
         return nil, "dir doesn't match lib"
       end
     end,
     nil -- directory loader can't load objects
   )

   -- the abstraction loader is quite complicated
   -- if the file exists it creates a callback that
   -- registers the class, with a constructor that
   -- runs the abstraction itself
   -- a Context here is essentially a 'canvas'
   Loader.new("txt", 30,
     nil, -- abstractions cannot register new classes
     function(dir, lib, obj)
       local src = dir .. "/" .. obj .. ".txt.lua"
       local f, err = fileexists(src)
       if f then
         local ldir, lobj = dir, obj
         return function()
           register(lobj, src, function(o, arguments)
             Context.push(Context.new({ ldir }))
             runtxt(ldir, lobj)
             Context.pop()
           end, src)
         end
       else
         return nil, err
       end
     end)

The current implementation is entirely in Lua, in an unpublished git 
repository.  I'll upload it somewhere if there is any interest.


Clearer Error Messages
----------------------

The main upshot so far is clearer error messages instead of random 
possibly-working-but-perhaps-not implementation defined behaviour when 
more than one library defines the same name:


OBJECT    [obj4]
     obj4 ... couldn't create (ambiguous)
                              (found in: 'lib4')
                              (found in: 'lib3')
                              (found in: 'lib1')
                              (found in: 'lib5')
                              (found in: 'lib2')
              disambiguate by [import lib2/obj4]
              replacing lib2 by the library you want to import from

OBJECT    [obj2]
     obj2 ... couldn't create (ambiguous qualified import)
                              (imported: 'lib2/obj2')
                              (imported: 'lib1/obj2')
              disambiguate by removing extra qualified imports


The error messages for objects that are not found are better too:


OBJECT    [obj8]
     obj8 ... couldn't create (no such object)
                              (looked in: 'lib4')
                              (looked in: 'vanilla')
                              (looked in: 'lib3')
                              (looked in: 'lib1')
                              (looked in: 'lib5')
                              (looked in: 'lib2')

OBJECT    [lib1/obj8]
     lib1/obj8 ... couldn't create (no such object in library)

OBJECT    [lib6/obj1]
     lib6/obj1 ... couldn't create (no such library)


Note for that last [libc6/obj1] example: if libc6 actually did exist in 
the path but was not imported, the error would currently be the same.  I 
want to make it so that [somelibrary/someobject] would work even when 
[somelibrary] is not imported (but such "implicit imports" would be 
deprecated, and warnings emitted).


Test Inputs
-----------

The test input directory structure (.bin.lua approximates .pd_linux etc, 
.txt.lua approximates .pd patches/abstractions):
----8<----
.:
global  prj1

./global:
lib1  lib2  lib3  lib3.bin.lua  lib4  lib5.bin.lua  vanilla

./global/lib1:
obj1.txt.lua  obj3.txt.lua  obj4.txt.lua  obj5.txt.lua  obj7.txt.lua
obj2.txt.lua  obj4.bin.lua  obj5.bin.lua  obj6.bin.lua

./global/lib2:
lib2.bin.lua  obj2.txt.lua  obj4.bin.lua  obj5.bin.lua  obj6.bin.lua
obj1.txt.lua  obj3.txt.lua  obj4.txt.lua  obj5.txt.lua  obj7.txt.lua

./global/lib3:
obj1.txt.lua  obj3.txt.lua  obj4.txt.lua  obj5.txt.lua  obj7.txt.lua
obj2.txt.lua  obj4.bin.lua  obj5.bin.lua  obj6.bin.lua

./global/lib4:
obj1.txt.lua  obj3.txt.lua  obj4.txt.lua  obj5.txt.lua  obj7.txt.lua
obj2.txt.lua  obj4.bin.lua  obj5.bin.lua  obj6.bin.lua

./global/vanilla:
debug.bin.lua

./prj1:
prj1.txt.lua
----8<----

The test input file (prj1.txt.lua) imports stuff and creates objects:
----8<----
import({
   "lib1", "lib1/obj1", "lib1/obj2",
   "lib2", "lib2/obj2", "lib2/obj3",
   "lib3",
   "lib4",
   "lib5"
})
for o = 1,8 do
   object("obj" .. o)
end
for l = 1,6 do for o = 1,8 do
   object("lib" .. l .. "/" .. "obj" .. o)
end end
----8<----


Test Outputs
------------

The full raw output:
http://claudiusmaximus.goto10.org/t/pd/pd-new-loader-test-4-verbose.txt

The output with the STAT lines filtered out:
http://claudiusmaximus.goto10.org/t/pd/pd-new-loader-test-4-quiet.txt

The output with the STAT lines sorted and duplicates counted:
http://claudiusmaximus.goto10.org/t/pd/pd-new-loader-test-4-statistics.txt
(The counts are low because the results are cached, this will need to be 
changed for live-coding abstractions etc...)


Hopefully I'm heading in the right direction with this, if not then yell 
loudly!

Thanks,


Claude
-- 
http://claudiusmaximus.goto10.org