[PD] GEM: video analysis

Tue Jul 25 18:10:33 CEST 2006

hi.
sorry for the late reply. i was on holiday....

Florian Walter wrote:
> Hallo,
> I would like to do some video analysis to control the playback of
> audio files. pix_movement and pix_blob work quite well, but now I
> would like to go into details and have some questions:
> 
> - How can I apply a color filter to an image?
>   I would like to distinguish the movement of different colors...

use [pix_rgb2hsv] to convert the image into HSV-colorspace (the actual 
colorspace is still GL_RGBA, but the "red" channel will be "hue",...)
then use [pix_curve] to mask out the colors ("hue") that you don't want.
finally use [pix_blob] only on the 1st channel ("hue").

i attach i (quite chaotic) patch that should show you the idea (it does 
NOT do motion-tracking but simple colortracking instead)

> 
> - Is there an object that returns the total brightness of an image? Or
>   are there other possibilities to detect the "amount" of movment in
>   a video?

the last outlet of [pix_blob] is the size of the blob, which is 
proportional to the amount of movement.

>   (I read about pix_mean_color in the archives but I can't find it in
>   gem 0.90.1...)

there is an experimental binary for w32 available on the download site 
at http://gem.iem.at

for self-compiling check out the CVS.

> 
> - I would like to recognize whether big blocks of a single color or
>   fine noise-like structures are moving. Downsamling might be a good
>   solution to get away the fine structures and to reduce the number of
>   pixels inside of a colored block that are not recognized as moving.
>     pix_resize didn't change anything (are there any parameters I
>   can pass to that object?) and needs lots of computing power...
>   Any ideas how to solve this problem?

try to capture with the lowest feasible resolution in the first place.

[pix_resize 32 32] should resize your image to 32x32 pixels.
[pix_movement] takes an optional arugment to set a threshold: if the 
movement in a pixel is smaller than the threshold, it is not taken into 
account. (but you still cannot detect, whether you have a number of 
small blobs or as single bigger one)
with [pix_convolve] you can do lowpass-filtering of the image (and 
hopefully get rid of noise)

[pix_multiblob] (i doubt whether this is in the 0.90.1 release) is a bit 
more sophisticated, as it returns not only the size ("total brightness") 
of each individual blob but also the area covered by the blob

hope this helps.

mfga.sdr
IOhannes

-------------- next part --------------
#N canvas 209 0 853 891 10;
#X floatatom 237 298 5 0 0 0 - - -;
#X floatatom 290 300 5 0 0 0 - - -;
#X obj 54 92 gemwin;
#X msg 106 74 destroy;
#X msg 86 31 1;
#X msg 88 52 0;
#X floatatom 373 301 5 0 0 0 - - -;
#X obj 54 131 gemhead;
#X obj 54 157 pix_video 64 64;
#X text 248 41 a movement detector;
#X obj 54 404 pix_texture;
#X obj 54 441 square 4;
#X obj 211 393 gemhead;
#X obj 211 447 translateXYZ;
#X obj 237 344 * 8;
#X obj 237 366 - 4;
#X obj 290 344 * 8;
#X obj 290 366 - 4;
#X msg 153 126 dimen 256 256;
#X msg 259 120 mode PAL \$1;
#X floatatom 284 103 5 0 0 0 - - -;
#X floatatom 112 237 5 0 0 0 - - -;
#X obj 320 228 pix_curve hue sat val;
#X obj 480 45 table hue 256;
#X obj 480 75 table sat 256;
#X obj 476 101 table val 256;
#X msg 623 626 \; sat const 255;
#X obj 630 353 i;
#X obj 658 351 + 1;
#X obj 692 353 % 256;
#X msg 633 378 \$1 \$1;
#X obj 626 315 until;
#X msg 627 290 256;
#X obj 634 441 tabwrite sat;
#X msg 625 664 \; val const 255;
#X obj 324 201 pix_rgb2hsv;
#X obj 426 131 tgl 15 0 empty empty empty 0 -6 0 8 -262144 -1 -1 0
1;
#X obj 404 131 tgl 15 0 empty empty empty 0 -6 0 8 -262144 -1 -1 0
1;
#X obj 632 149 i;
#X obj 660 147 + 1;
#X obj 694 149 % 256;
#X obj 632 122 until;
#X msg 625 56 256;
#N canvas 0 0 450 450 0set1 0;
#X obj 94 68 inlet;
#X obj 138 386 outlet;
#X text 139 69 center range;
#X obj 94 134 unpack;
#X obj 180 205 / 2;
#X obj 180 229 -;
#X obj 180 253 * -1;
#X obj 138 299 until;
#X obj 138 183 t f b f;
#X obj 138 323 i;
#X obj 166 324 + 1;
#X msg 147 278 0;
#X obj 138 346 +;
#X obj 94 158 moses 0;
#X msg 94 107 \$2 \$1;
#X connect 0 0 14 0;
#X connect 3 0 13 0;
#X connect 3 1 5 1;
#X connect 4 0 5 0;
#X connect 5 0 6 0;
#X connect 6 0 12 1;
#X connect 7 0 9 0;
#X connect 8 0 7 0;
#X connect 8 1 11 0;
#X connect 8 2 4 0;
#X connect 9 0 10 0;
#X connect 9 0 12 0;
#X connect 10 0 9 1;
#X connect 11 0 9 1;
#X connect 12 0 1 0;
#X connect 13 1 8 0;
#X connect 14 0 3 0;
#X restore 420 588 pd 0set1;
#X obj 420 566 t l b;
#X obj 420 613 mod 256;
#X obj 420 657 tabwrite hue;
#X msg 491 584 \; hue const 0;
#X obj 478 502 t b f;
#X floatatom 420 490 5 0 0 0 - - -;
#X floatatom 476 480 5 0 0 0 - - -;
#X obj 390 481 bng 15 250 50 0 empty empty empty 0 -6 0 8 -262144 -1
-1;
#X msg 630 532 \; sat const 0;
#X msg 632 570 \; val const 0;
#X obj 632 207 tabwrite val;
#X msg 632 177 \$1 \$1;
#X obj 320 251 pix_histo h s v;
#X obj 86 535 table h 256;
#X obj 86 565 table s 256;
#X obj 82 591 table v 256;
#X msg 420 634 256 \$1;
#X obj 247 698 hsv2rgb;
#X msg 246 673 \$1 1 1;
#X obj 247 720 unpack 0 0 0;
#X floatatom 247 746 5 0 0 0 - - -;
#X floatatom 289 746 5 0 0 0 - - -;
#X floatatom 329 746 5 0 0 0 - - -;
#X obj 245 810 unpack 0 0 0;
#X floatatom 245 836 5 0 0 0 - - -;
#X floatatom 287 836 5 0 0 0 - - -;
#X floatatom 327 836 5 0 0 0 - - -;
#X obj 245 766 pack 0 0 0;
#X obj 245 788 rgb2hsv;
#X floatatom 247 652 5 0 0 0 - - -;
#X obj 327 770 t b;
#X obj 211 853 * 256;
#X floatatom 209 873 5 0 0 0 - - -;
#X obj 282 392 * -1;
#X msg 10 101 reset;
#X msg 54 6 create \, 1;
#X msg 282 83 1;
#X msg 184 69 bang;
#X obj 312 274 pix_blob 1;
#X obj 211 540 color 0.5 0.5 0.5;
#X obj 211 568 sphere 0.5;
#N canvas 0 0 450 300 color2hsv 0;
#X obj 96 68 inlet;
#X obj 102 258 outlet;
#X obj 108 110 symbol;
#X msg 101 233 151 8;
#X msg 157 231 93 16;
#X msg 235 227 42 28;
#X obj 108 172 select blue green yellow red;
#X msg 297 223 0 20;
#X connect 0 0 2 0;
#X connect 2 0 6 0;
#X connect 3 0 1 0;
#X connect 4 0 1 0;
#X connect 5 0 1 0;
#X connect 6 0 3 0;
#X connect 6 1 4 0;
#X connect 6 2 5 0;
#X connect 6 3 7 0;
#X connect 7 0 1 0;
#X restore 393 430 pd color2hsv;
#X obj 399 452 unpack;
#X floatatom 449 206 5 0 0 0 - - -;
#X obj 441 784 unpack 0 0 0;
#X floatatom 546 807 5 0 0 0 - - -;
#X floatatom 492 807 5 0 0 0 - - -;
#X floatatom 441 807 5 0 0 0 - - -;
#X text 555 785 color-list;
#X obj 440 764 rgb2hsv;
#X obj 441 828 * 256;
#X obj 401 93 tgl 15 0 empty empty empty 0 -6 0 8 -262144 -1 -1 0 1
;
#X obj 53 191 t a a;
#X obj 72 286 pix_texture;
#X obj 72 323 square 4;
#X obj 69 365 spigot;
#X obj 131 359 tgl 15 0 empty empty empty 0 -6 0 8 -262144 -1 -1 0
1;
#X obj 145 307 gemmouse;
#X obj 157 327 change;
#X obj 157 349 select 1;
#X obj 211 474 color 0.5 0.5 0.5;
#X obj 211 516 square 0.25;
#X obj 283 497 * 1.5;
#X obj 342 479 t f;
#X obj 420 534 pack 0 10;
#X obj 557 748 t a a;
#X obj 393 352 cnv 15 120 60 empty empty empty 20 12 0 14 -233017 -66577
0;
#X msg 398 365 blue;
#X msg 394 391 green;
#X msg 450 365 yellow;
#X msg 466 385 red;
#X obj 553 714 cnv 15 200 30 empty empty empty 20 12 0 14 -233017 -66577
0;
#N canvas 0 0 647 439 colorselect 0;
#X obj 92 131 spigot;
#X obj 78 101 pack;
#X obj 162 261 pix_data;
#X obj 204 82 gemhead;
#X obj 228 186 pix_snap;
#X obj 206 121 spigot;
#X obj 220 220 unpack;
#X obj 86 57 gemmouse 1 1;
#X text 329 281 gray-value;
#X floatatom 288 240 5 0 0 0 - - -;
#X floatatom 264 262 5 0 0 0 - - -;
#X floatatom 148 118 5 0 0 0 - - -;
#X obj 190 152 t b b a;
#X msg 195 183 snap;
#X msg 257 135 500 500;
#X obj 184 298 outlet;
#X obj 438 260 - 1;
#X obj 438 296 * -1;
#X obj 257 112 loadbang;
#X connect 0 0 6 0;
#X connect 1 0 0 0;
#X connect 2 1 15 0;
#X connect 3 0 5 0;
#X connect 4 0 2 1;
#X connect 5 0 12 0;
#X connect 6 0 2 2;
#X connect 6 0 10 0;
#X connect 6 1 9 0;
#X connect 6 1 16 0;
#X connect 7 0 1 0;
#X connect 7 1 1 1;
#X connect 7 2 0 1;
#X connect 7 2 5 1;
#X connect 7 2 11 0;
#X connect 12 0 2 0;
#X connect 12 1 13 0;
#X connect 12 2 4 0;
#X connect 13 0 4 0;
#X connect 14 0 4 2;
#X connect 16 0 17 0;
#X connect 17 0 2 3;
#X connect 18 0 14 0;
#X restore 555 721 pd colorselect by mouse;
#X connect 0 0 14 0;
#X connect 1 0 16 0;
#X connect 3 0 2 0;
#X connect 4 0 2 0;
#X connect 5 0 2 0;
#X connect 6 0 107 0;
#X connect 7 0 8 0;
#X connect 8 0 96 0;
#X connect 10 0 11 0;
#X connect 12 0 13 0;
#X connect 13 0 104 0;
#X connect 14 0 15 0;
#X connect 15 0 13 1;
#X connect 16 0 17 0;
#X connect 17 0 77 0;
#X connect 18 0 8 0;
#X connect 19 0 8 0;
#X connect 20 0 19 0;
#X connect 21 0 82 1;
#X connect 22 0 56 0;
#X connect 27 0 28 0;
#X connect 27 0 30 0;
#X connect 28 0 29 0;
#X connect 29 0 27 1;
#X connect 30 0 33 0;
#X connect 31 0 27 0;
#X connect 32 0 31 0;
#X connect 35 0 22 0;
#X connect 36 0 22 0;
#X connect 37 0 35 0;
#X connect 38 0 39 0;
#X connect 38 0 55 0;
#X connect 39 0 40 0;
#X connect 40 0 38 1;
#X connect 41 0 38 0;
#X connect 42 0 41 0;
#X connect 43 0 45 0;
#X connect 44 0 43 0;
#X connect 44 1 47 0;
#X connect 45 0 60 0;
#X connect 48 0 108 0;
#X connect 48 1 108 1;
#X connect 49 0 108 0;
#X connect 50 0 48 0;
#X connect 51 0 49 0;
#X connect 55 0 54 0;
#X connect 56 0 82 0;
#X connect 60 0 46 0;
#X connect 61 0 63 0;
#X connect 62 0 61 0;
#X connect 63 0 64 0;
#X connect 63 1 65 0;
#X connect 63 2 66 0;
#X connect 64 0 71 0;
#X connect 65 0 71 1;
#X connect 65 0 74 0;
#X connect 66 0 71 2;
#X connect 66 0 74 0;
#X connect 67 0 68 0;
#X connect 67 1 69 0;
#X connect 67 2 70 0;
#X connect 68 0 75 0;
#X connect 71 0 72 0;
#X connect 72 0 67 0;
#X connect 73 0 62 0;
#X connect 74 0 64 0;
#X connect 75 0 76 0;
#X connect 77 0 13 2;
#X connect 78 0 2 0;
#X connect 79 0 2 0;
#X connect 80 0 20 0;
#X connect 81 0 18 0;
#X connect 81 0 80 0;
#X connect 82 0 99 0;
#X connect 82 1 0 0;
#X connect 82 2 1 0;
#X connect 82 3 6 0;
#X connect 83 0 84 0;
#X connect 85 0 86 0;
#X connect 86 0 49 0;
#X connect 86 1 50 0;
#X connect 88 0 91 0;
#X connect 88 1 90 0;
#X connect 88 2 89 0;
#X connect 91 0 94 0;
#X connect 93 0 88 0;
#X connect 94 0 49 0;
#X connect 95 0 37 0;
#X connect 95 0 36 0;
#X connect 96 0 35 0;
#X connect 96 1 97 0;
#X connect 97 0 98 0;
#X connect 99 0 10 0;
#X connect 100 0 99 1;
#X connect 101 4 102 0;
#X connect 102 0 103 0;
#X connect 103 0 100 0;
#X connect 104 0 105 0;
#X connect 105 0 83 0;
#X connect 106 0 105 1;
#X connect 107 0 106 0;
#X connect 107 0 84 1;
#X connect 108 0 44 0;
#X connect 109 0 83 1;
#X connect 109 1 93 0;
#X connect 111 0 85 0;
#X connect 112 0 85 0;
#X connect 113 0 85 0;
#X connect 114 0 85 0;
#X connect 116 0 109 0;