[PD-dev] call for discussion: native video for Pd

[geiger]

On Sat, 27 May 2006, Miller Puckette wrote:
> To begin with at least, I'm hoping to be able to do all video operations
> except storage using floating-point numbers, thereby re-using the usual
> tilde objects.  The disadvantage of this approach is that, if you want to
> "sample" an image, to get decent cache behavior you'd want the possibility
> of storing it in a data-reduced way, as 8-bit integers or perhaps using
> YUYV packing.  (for example, a 512x768 color video frame takes almost 5MB
> to store in floating point, but only about 0.7 MB in 8-bit YUYV.)

I sort of have the same feelings as Christian and others. If you do
video processing, you probably want it to be as fast as possible, because
it still takes a considerable amount of a modern CPU's power.
Considering that most building blocks will be different from the ones
that are used in audio processing, you will have to reimplement them
anyhow, and the best would be to implement them as fast as possible.

> So there will probably have to be a suite of new objects for storing 2D
> arrays in fixed point formats, variously trading off memory compaction with
> processing time needed to get the data in and out.  There will probably also
> want to be a choice of interpolation strategies.  Probably the design can
> look like the table/tabwrite/tabread/tabread4/tabwrite~/tabread~/tabread4~
> objects.
>
> Also like "table" objects I want to publish an API so that pdp and gem can
> read and write into image storage buffers.
>
> I'll probably base the actual video I/O objects on the way Tom Shouten did
> it in pdp - perhaps pdp would then be able to use pd's video objects directly
> if I can get the design just right.

I am not sure if I understand what you mean with video I/O objects, but
what I understand (output to screen and input from video sources) is the
part of pdp that  I think is not well designed. I find it strange having
to use pdp_xv or pdp_glx as output object, depending on what hardware I
have. This decision should be taken by the system, or changeable by
messages, not hardcoded in a patch.

> ideas and opinions welcome!

If I understand correctly, what you want to do is line by line processing
for images. I am not a video specialist, by this might make some
algorithms a lot harder to implement. Do you have some ideas about
how a convolution with a matrix would look like, for the user and
the internal implementation ?

And then, asuming that you have to use a compact representation of the
image in memory, because otherwise cache behaviour will destroy
performance completely, you would also have to pack and unpack the
image data on every step for algorithms that are based on different
frames in time.

Well, sorry if I didn't really get what it is all about, but I also try
to figure out why we need yet another video extension to pd, instead
of fixing the ones that exist.

Couldn't we just incorporate pdp and make it run on windows and macosx ?

Cheers,
Günter

>
> cheers
> Miller
>
> _______________________________________________
> PD-dev mailing list
> PD-dev at iem.at
> http://lists.puredata.info/listinfo/pd-dev
>