[PD-dev] call for discussion: native video for Pd

[Miller Puckette]

To Pd developers,

After years of worrying about it, I'm thinking it's time to add video
support natively to Pd.  My basic idea is to add a feature to the block~
object so that windows of DSP computation can be triggered from external
messages.  That way, video I/O objects could be designed to spit out
frames or portions of frames on demand, and the user gets the ability to
explicitly schedule how the video computations should be run.  

(My favorite
example of non-obvious scheduling of video computation is low-latency
analysis of incoming video, where you can actually use the 1/60-second
fields without having to wait the additional 1/60-th second for a complete
frame to arrive.)

I think there will have to be some state added to the DSP chain
mechanism so that objects explicitly designed to work  on video streams 
can find out what part of the image they're looking at.  This could be
as simple as a global data structure describing the current vector's
position in an image; existing tilde objects would simply ignore the
information.

To begin with at least, I'm hoping to be able to do all video operations
except storage using floating-point numbers, thereby re-using the usual
tilde objects.  The disadvantage of this approach is that, if you want to
"sample" an image, to get decent cache behavior you'd want the possibility
of storing it in a data-reduced way, as 8-bit integers or perhaps using
YUYV packing.  (for example, a 512x768 color video frame takes almost 5MB
to store in floating point, but only about 0.7 MB in 8-bit YUYV.)

So there will probably have to be a suite of new objects for storing 2D
arrays in fixed point formats, variously trading off memory compaction with
processing time needed to get the data in and out.  There will probably also
want to be a choice of interpolation strategies.  Probably the design can
look like the table/tabwrite/tabread/tabread4/tabwrite~/tabread~/tabread4~
objects.  

Also like "table" objects I want to publish an API so that pdp and gem can
read and write into image storage buffers. 

I'll probably base the actual video I/O objects on the way Tom Shouten did
it in pdp - perhaps pdp would then be able to use pd's video objects directly
if I can get the design just right.

ideas and opinions welcome!

cheers
Miller