[PD-dev] call for discussion: native video for Pd

[Hans-Christoph Steiner]

On May 28, 2006, at 4:26 PM, geiger wrote:

> On Sat, 27 May 2006, Miller Puckette wrote:
>> To begin with at least, I'm hoping to be able to do all video  
>> operations
>> except storage using floating-point numbers, thereby re-using the  
>> usual
>> tilde objects.  The disadvantage of this approach is that, if you  
>> want to
>> "sample" an image, to get decent cache behavior you'd want the  
>> possibility
>> of storing it in a data-reduced way, as 8-bit integers or perhaps  
>> using
>> YUYV packing.  (for example, a 512x768 color video frame takes  
>> almost 5MB
>> to store in floating point, but only about 0.7 MB in 8-bit YUYV.)
>
> I sort of have the same feelings as Christian and others. If you do
> video processing, you probably want it to be as fast as possible,  
> because
> it still takes a considerable amount of a modern CPU's power.
> Considering that most building blocks will be different from the ones
> that are used in audio processing, you will have to reimplement them
> anyhow, and the best would be to implement them as fast as possible.

Since we already have GEM and PDP, it seems that there should be more  
leeway for experimentation in terms of a floating point video format  
for Pd.  I think that it would at the very least be an interesting  
experiment to try to have a unified system for handling media data.   
Yes it will be slow, but in a couple of years, it could be something  
really unique.

If you need fast, you could use GEM or PDP in the meantime.  Speaking  
of, how about releasing that PDP port for Windows?

.hc

>> So there will probably have to be a suite of new objects for  
>> storing 2D
>> arrays in fixed point formats, variously trading off memory  
>> compaction with
>> processing time needed to get the data in and out.  There will  
>> probably also
>> want to be a choice of interpolation strategies.  Probably the  
>> design can
>> look like the table/tabwrite/tabread/tabread4/tabwrite~/tabread~/ 
>> tabread4~
>> objects.
>>
>> Also like "table" objects I want to publish an API so that pdp and  
>> gem can
>> read and write into image storage buffers.
>>
>> I'll probably base the actual video I/O objects on the way Tom  
>> Shouten did
>> it in pdp - perhaps pdp would then be able to use pd's video  
>> objects directly
>> if I can get the design just right.
>
> I am not sure if I understand what you mean with video I/O objects,  
> but
> what I understand (output to screen and input from video sources)  
> is the
> part of pdp that  I think is not well designed. I find it strange  
> having
> to use pdp_xv or pdp_glx as output object, depending on what  
> hardware I
> have. This decision should be taken by the system, or changeable by
> messages, not hardcoded in a patch.
>
>> ideas and opinions welcome!
>
> If I understand correctly, what you want to do is line by line  
> processing
> for images. I am not a video specialist, by this might make some
> algorithms a lot harder to implement. Do you have some ideas about
> how a convolution with a matrix would look like, for the user and
> the internal implementation ?
>
> And then, asuming that you have to use a compact representation of the
> image in memory, because otherwise cache behaviour will destroy
> performance completely, you would also have to pack and unpack the
> image data on every step for algorithms that are based on different
> frames in time.
>
> Well, sorry if I didn't really get what it is all about, but I also  
> try
> to figure out why we need yet another video extension to pd, instead
> of fixing the ones that exist.
>
> Couldn't we just incorporate pdp and make it run on windows and  
> macosx ?
>
> Cheers,
> Günter
>
>>
>> cheers
>> Miller
>>
>> _______________________________________________
>> PD-dev mailing list
>> PD-dev at iem.at
>> http://lists.puredata.info/listinfo/pd-dev
>>
>
> _______________________________________________
> PD-dev mailing list
> PD-dev at iem.at
> http://lists.puredata.info/listinfo/pd-dev


________________________________________________________________________ 
____

"I have the audacity to believe that peoples everywhere can have  
three meals a day for their bodies, education and culture for their  
minds, and dignity, equality and freedom for their spirits."
                                             - Martin Luther King, Jr.