[PD-dev] [PD] Pd 0.50-2 on macos 11.0 BigSur beta

[IOhannes m zmoelnig]

On 06.07.20 19:39, Martin Peach wrote:
> The build farm used by Pd-extended worked very well for me. It would
> be nice to have such a setup integrated with the deken system, so
> anyone could upload their externals and they would be built and
> packaged online instead of each developer needing to acquire one of
> each machine on a 3-year rolling plan.

we have something like this [0] setup at the iem, based on gitlab-ci[1].

the system is targetting pd-lib-builder projects (though it will
probably will work with most build-systems that follow a "make && make
install" paradigm)

whenever someone pushes a new commit to a CI-enabled Pd-project, it will
build the external for the following architectures:
- Darwin/amd64 (aka "64bit macOS")
- Windows/amd64 (aka "64bit windows")
- Windows/i386 (aka "32bit Windows")
- Linux/amd64
- Linux/i386
- Linux/armhf (aka "Raspberry")
- Linux/arm64 (the future)

the build artifacts are collected into a single deken package, which can
be downloaded manually.
if the commit was "tagged" (which we take as a "release" indicator), the
deken package will be automatically uploaded to deken.puredata.info.

there's a system to declare build dependencies (via "apt" on linux,
MSYS' "pacman" on Windows, and "brew" on macOS).
for systems that don't come with a proper package manager installed by
default (macOS, Windows), the runtime dependencies are collected and
embedded in the deken package.

some libraries cannot be built into a single multi-arch library because
of file conflicts (this becomes fun quickly when dealing with dependency
libraries and dependencies thereof). there's an option to create
multiple single-architecture deken packages for this case.

in general, the configuration is about 3-lines (e.g. [2])
a bit more, if you want to enable automatic uploads to puredata.info.


so everythings cool.
what are the caveats?

## Darwin/i386
we currently don't build for Darwin/i386 (nor Darwin/amd64).
while we have older macOS builders (there's builders for both
macOS/Sierra and macOS/Mojave, the default being macOS/Mojave) with
*not* the newest XCode, and should therefore still be able to build i386
binaries without problems, these are currently disabled because of the
automatic dependency management (on macOS we use "brew" to install
dependencies; brew has decided to stop supporting i386 binaries a while
ago :-()


## accountability
deken is a quite insecure system, as there isn't really a way to
"verify" a package before installing it.
the best you can get is that you know the <username> who uploaded a
given package (in theory there's also the gpg-signature; but who ever
checked that?)
now, with automated uploads this becomes a bit muddled, as there is no
longer any direct involvement of a human (whom you can trust, or not) in
creating the binaries.
in theory we know the user who pushed the last commit that triggered the
build process.
but currently there's no easy way to make this user show up in the deken
results.


## passwords
to upload any package to deken.puredata.info, you currently need a
username/password.
if you want to upload automatically, the build-process needs to know
your username/password.
i really don't want to encourage people to store their credential for
one web-service in any other web-service (even if both services are
operated by the same entity (me!)).
(the credentials would be stored in some database and *NOT* in a file in
the repository, so they are not directly exposed to ordinary humans; but
nevertheless the credentials need to be plain-text (as in: unhashed)
which is a security breach...).
for my personal use i have created a "zmoelnigbot" account on
puredata.info to manage those uploads (and keep it separate from my
other account). but that doesn't scale well.


## resources
it's not like we have a big compile farm or so, but we take interest
(and pride) in running all the builds on-premises.
currently there's 3 machines:
- an oldish (but dedicated) MacMini (that needs to run all the
Darwin-builds for license reasons; and which also does Windows builds)
- a dedicated linux box (Windows and Linux builds)
- a virtual machine (Linux builds)

having dedicated machines means, that the builds will (mostly) start
right away (as opposed to e.g. travis-ci where it can take considerable
time (as in: half an hour) until the builds start).
having very few machines available means that building for all the
different architectures will take longer than just running "make" locally.
it also means that if multiple projects want to build at the same time,
it will take longer (and we have some in-house projects where each build
takes about 1hour; if they happen to be started before your project, it
will take a while).

it's probably super-easy for volunteers to add their own machines to the
build farm - at least for the Linux builds (that use docker internally).
it's much more complicated for Windows and Darwin builds (which
currently use manually pre-configured VirtualBox images that have to be
regularly updated (manually), which is always a pain in the ass).
of course, the Windows and Darwin builds are those where the automatic
build becomes really interesting.


## resources (legally)
the service is currently run by the iem, at the iem and for the iem.
i would have to talk with my boss about making it available to a wider
audience.
it would probably be fine (but of course, i really don't know).


mgfdasr
IOhannes



[0] https://git.iem.at/pd/iem-ci/
[1] https://docs.gitlab.com/ee/ci/
[2] https://git.iem.at/pd/iemnet/-/blob/master/.git-ci/gitlab-iem.yml

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.puredata.info/pipermail/pd-dev/attachments/20200707/51409171/attachment.sig>