[PD-dev] sound + video : real time issues

[Ricardo Fabbri]

Hi,

I am building instalations/instruments which control sounds
through the webcam, using a color detector that I submitted to this
list a couple of days ago. Things work well, and I am just facing real
time/delay issues now.
When I try to generate sound with any pix
processing, I am getting jumps in the sound (like on/off sounds at
about 5-20Hz). Turning off any pix processing, the sound is then back
to normal, even if I show the raw video from the camera on a gemwin.
By pix processing I mean anything with even only 1 pass through the
image on the CPU. I am using a ps3eye camera, which makes things
better as it has a high framerate, but I still get interruptions in
the sound.

Why is this lag in the sound? The image with the output of the pix
processing doesn't seem to be moving any slower with or without sound.

What are your experiences in trying to get gem pix processing, video,
and DSP to work in real time? My application is interactive, so that I
really need the sound to be as smooth as possible. Here are some
factors I will investigate, let me know if they make any sense to you:

- It could be just too much data, even a single pass on each pixel
could be causing delays/jumps. I could downsample the image prior to
any pix processing
- The kernel is too slow in processing the video stream; I tried '-rt'
with no success
- Perhaps there is a problem in Pd sound scheduling vs pix processing
- Perhaps its just the CPU. In fact, I used OSC to do the video on one
machine, send the parameters to a second machine which then processes
the audio there. This effectively solved the problem. Question is, is
this due to CPU or to another factor?

I would really appreciate hearing ideas from you, as I am new to this.
Best,
Ricardo Fabbri
--
Linux registered user #175401
www.lems.brown.edu/~rfabbri
labmacambira.sf.net