[PD-dev] [GEM] more on YUV (was: postrendering crashes)

[daniel heckenberg]

hi folks,

i've been having a bit of break from the keyboard so apologies if i'm
rehashing here... it looks like the yuv stuff is causing some really good
discussion about GEM development.  excellent!

i might hazard my opinion on some of the issues with yuv:
- in general i suspect it's more costly to convert from one colour space to
another to do a particular operation (say chromakey or lumakey) than it is
to simply use that operation in the other colour space, even if it's a
little awkward.  this is even more the case if you then want to convert
back to the original colour space.  in the most extreme case, doing a pixel
by pixel format conversion in the inner loop of the algorithm is going to
be better than doing an image conversion and then running the algorithm.  
- as far as i can see, the only issues of platform/machine dependence with
yuv are in the film codec and texture upload stages.  format conversions
can easily be included in the relevant objects when the underlying software
and hardware do not provide the feature natively. 

for these reasons (and for others more capricious), i would argue:
- that all objects should be pix_XXXX objects.  
- all pix_ objects should operate on all image types (luminance, rgb, rgba,
yuv), with the exception of the few objects which are truly format
restricted (only those explicitly using the alpha channel i think) which
should print an (accurate!) error message. 
- format conversions should only be done with explicit objects (pix_toyuv),
except in image source (e.g. pix_film) and image sink (e.g. pix_texture)
object where they may be done implicitly.  
- image sources should allow their format to be chosen through an argument
or a message.  they should announce the format when the image or film is
loaded.  should their default be system dependent to maximise performance,
or system independent to maximise consistency?
- a pix_info object should provide information at its outputs about the
image passing through it: pixel dimensions, format and anything else
relevant

it would be relatively easy to add pix_ base class automatic format
conversion support as a stop-gap.  this can be done with a nasty
out-of-place conversion and reconversion with a performance warning printed
to the console.  this would allow this general behaviour to be acheived in
the short term before implementing each pix_ object for each image format. 

hmm i wonder if there are clever ways to use generic programming techniques
to allow the algorithms to be implemented almost independently of the image
format... and to allow new image formats to be added without adding code to
the pix_s?

in general, i don't think that there's a really easy solution to automatic
format negotiation in video data flow trees is there?  i've done a bit of
work with directshow on windows which tries to be very general and succeeds
in being rather comlicated and hard to use...

daniel

chris clepper writes:

> >as ben pointed out: how about sending a "want YUV"-message to a proper
> >[pix_film] and not having to worry whether you should need a 
> >[pix_filmBGRA_ext]
> >or [pix_filmLUMINANCE] ?
> 
> that would work, but the question of feedback to the user remains. 
> how do you make it clear which objects work with which color-space 
> properly?  and when do you need to convert?   would this be automatic 
> or user controlled?
> 
> >  >
> >>  the complexity really depends on the operation.  for proper
> >>  chroma-keying and color adjustment one must know about yuv color
> >>  space.  other functions like blurring, mixing and masking are just
> >>  like their rgb counterparts.
> >so why segregating them ?
> 
> i think i've said this before:
> 
> At 6:35 PM -0600 12/16/02, chris clepper wrote:
> >here is my reasoning for making the [yuv_] set objects:
> >
> >- a separate set of objects would be easier to leave out of compiles 
> >for platforms that don't support it.
> ...
> 
> >- there is not a one to one correlation between yuv and rgb color-spaces.
> ...
> 
> >- Gem already had the pix_ naming convention which made it clear 
> >that pix = image processing and that stringing pix_objects together 
> >would result in a functional chain.  my initial attempts to extend 
> >pix to yuv seemed to destroy that, so i decided that yuv_objects 
> >would make their own chains to avoid confusion.
>   ...
> 
> 
> >i was rather thinking of something like
> >[pix_film] (on demand, tries to load YUV, if it fails (because the codec does
> >not support it (is this possible), loads RGBA) -> [pix_yuv] (if pix_film
> >returns YUV this does *nothing*, if we only have RGBA it converts to YUV)
> >-> [pix_luma-key] (everything done in YUV-space) -> [pix_textureformat]
> >(converts into a format that can be textured by the openGL-implementation - on
> >macOS it might do nothing) -> [pix_texture]
> 
> ok. how about this:  pix_film -> pix_luma_key -> pix_alpha -> 
> pix_composite -> pix_texture
> 
> that's a total mess to mix and match yuv and rgba when you have alpha 
> in one and not the other.  i don't immediately see a good solution to 
> this apart from lots of conversions that either happen in the objects 
> automatically or a bunch of conversion objects added to the chain.
> 
> i can go on and on with examples like this, and would have to test 
> all of these combinations if yuv_ is integrated into pix_.  pix_ 
> objects imply that they all work together, and that should be the 
> case if yuv is added.  also it seems like if conversion is going to 
> be necessary for most situations and platforms then maybe it 
> shouldn't be included in the CVS at all.
> 
> >i will not have linux-only or macOS-only stuff in Gem as it is.
> 
> i just don't see how that's possible to maintain.  osx is not linux 
> which is not windows, so there has to be some things that can be done 
> on one platform and not another.  otherwise GEM just becomes a lowest 
> common denominator of these platforms, and not a full featured app on 
> any of them.  if you don't take advantage of the strengths that each 
> platform has, then why even bother making the app run on it in the 
> first place?
> 
> cgc
> 
> 
> 
> -- 
> 
> _______________________________________________
> PD-dev mailing list
> PD-dev at iem.kug.ac.at
> http://iem.kug.ac.at/cgi-bin/mailman/listinfo/pd-dev