[PD] [else/dir] leaks file handles (was: "too many open files" error in 0.48.1)

Tue Jul 24 00:09:22 CEST 2018

Hey all

Not only by creating 1020 instances of [else/dir] you can crash Pd, but
also by sending it a 337 'reset, open .' messages. I guess the reason
is the same: [else/dir] is leaking file handles. 

It appears I tried  [else/dir] for the similar reasons as you, Liam. I
am also using [ggee/getdir] and [moocow/readdir] now and thought about
reducing dependencies by switching to else. It turns out I can't switch
yet.

Roman

On Sun, 2018-05-27 at 19:26 +0000, Liam Goodacre wrote:
> OK, I've just spent several hours on this, testing upwards of 200,000
> parallel abstractions and crashing my computer almost as many times.
> But I managed to figure it out.
> 
> (Most of) you will be relieved to hear that the problem does not
> originate with the PD core, but with externals, as IOhannes
> suggested. Alex will not be relieved to hear that the culprit is
> [else/dir]. It seems that the 1019th instance of this object crashes
> PD! How strange.
> 
> I'm filing a bug report on git and moving on with my life.
> 
> Liam
> 
> 
> From: Pd-list <pd-list-bounces at lists.iem.at> on behalf of Liam
> Goodacre <liamg_uw at hotmail.com>
> Sent: 22 May 2018 05:36
> To: pd-list at lists.iem.at
> Subject: Re: [PD] "too many open files" error in 0.48.1
>  
> Thank you for the input. It will be a few days before I can do any
> serious testing, but I can answer some of your questions now.
> 
> IOhannes: yes, I upgraded PD, my externals, and my OS, all at the
> same time. I'll try it out with various different versions of the
> same and let you know what happens. But just as a heads-up, it's
> quite possible that I am trying to load more than 10,000 abstractions
> at once. Quite difficult to count them though!
> 
> If neither the externals nor the OS turn out to be the problem,
> I'll try to come up with a simple test case and get back to you.
> From: Pd-list <pd-list-bounces at lists.iem.at> on behalf of IOhannes m
> zmölnig <zmoelnig at iem.at>
> Sent: 21 May 2018 20:50
> To: pd-list at lists.iem.at
> Subject: Re: [PD] "too many open files" error in 0.48.1
>  
> On 05/21/2018 09:37 PM, ub at xdv wrote:
> > On 21.05.2018 20:54, Claude Heiland-Allen wrote:
> >>
> >>
> >> On 21/05/18 19:35, ub at xdv wrote:
> >>> hello,
> >>>
> >>> On 20.05.2018 06:50, Liam Goodacre wrote:
> >>>> In 0.48.1 on Ubuntu, I'm getting a horrible scenario where PD
> refuses to
> >>>> open patches or create any more abstractions for me. I get an
> error
> >>>> message saying "too many open files". Granted I have a lot open,
> but
> >>>> this is a serious problem as it means I can't access all of my
> old
> >>>> performances. They worked fine in 0.47.
> >>>>
> >>>> Any ideas?
> >>> this is not really a problem with pd
> >>
> >> I disagree. The most common cause of "too many open files" is a
> bug in
> >> closing files properly, 
> 
> intuitively i would agree.
> 
> > possible. never occured to me. is it in the issue tracker?
> 
> no (afaik). it hasn't been reported before.
> 
> > 
> >> because 1024 simultaneously-open files should be
> >> enough for most use cases.definitely, but when someone says
> "Granted I have a lot open", that
> > could be an understatement. or a regression of some sort.
> 
> so i did a test run, with some heavily embedded abstraction, that is
> loaded a total of 10000 times.
> $ ulimit -Sn
> 1024
> $ valgrind --track-fds=yes pd -noprefs -oss -nosound -nomidi -nrt
> -stderr -open test.pd
> [...]
> ==29030== FILE DESCRIPTORS: 3 open at exit.
> ==29030== Open file descriptor 2: /dev/pts/12
> ==29030==    <inherited from parent>
> ==29030==
> ==29030== Open file descriptor 1: /dev/pts/12
> ==29030==    <inherited from parent>
> ==29030==
> ==29030== Open file descriptor 0: /dev/pts/12
> ==29030==    <inherited from parent>
> ==29030==
> [...]
> $
> 
> so there seems to be neither a temporary file-handle leak (because my
> 10000 abstractions load just fine) nor an absolute file-handle leak
> (since the only handles that are left open by the program are stdin,
> stdout, stderr)
> 
> 
> so i guess there must be something wrong with the patch.
> 
> @liam are there any externals in use? did you upgrade the entire
> system
> or just Pd? when using an older version of Pd, does the problem
> persist?
> 
> gfmtdsar
> IOhannes
> 
> 
> > 
> > 
> >>
> >>
> >> Claude
> >>
> >>> , but with your shell- or system
> >>> configuration limiting the number of open file descriptors.
> >>>
> >>> check your current shell-limit with
> >>> ulimit -Sn
> >>> -S is for soft limit (you can lower, but not raise, the hard
> limit).
> >>> raise the limit in your shell with
> >>> ulimit -n 65536
> >>> start pd from that shell and see if the situation improves.
> >>>
> >>> if that doesn't help, you can check the kernel limits with
> >>> cat /proc/sys/fs/file-nr
> >>> which returns 3 numbers, the first of which is the number of open
> files,
> >>> the last of which is the limit.
> >>>
> >>> increase the value of "nofile" in  /etc/security/limits.conf
> >>> do
> >>> sudo sysctl -p
> >>>
> >>> additional steps are required to make these settings permanent,
> the
> >>> ulimit -n 65536 would have to go into your .bashrc so it's
> executed at
> >>> system startup. if you normally start pd from your GUI you'd have
> to
> >>> restart your system (actually just xorg) so your master shell
> knows
> >>> about the new value. there's other ways, one of which would be to
> start
> >>> pd from a wrapper script, or probably gnome provides that sort of
> >>> environmental setup for it's program shortcuts.
> >>>
> >>> there's more information
> >>> -
> >>> https://askubuntu.com/questions/162229/how-do-i-increase-the-open
> -files-limit-for-a-non-root-user
> >>>
> >>> -
> >>> https://unix.stackexchange.com/questions/29577/ulimit-difference-
> between-hard-and-soft-limits
> >>>
> >>>
> >>> hope that helps ... cheers,
> >>> ub
> >>>
> >>>> Liam
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Pd-list at lists.iem.at mailing list
> >>>> UNSUBSCRIBE and account-management ->
> >>>> https://lists.puredata.info/listinfo/pd-list
> >>>>
> >>>
> >>> _______________________________________________
> >>> Pd-list at lists.iem.at mailing list
> >>> UNSUBSCRIBE and account-management ->
> >>> https://lists.puredata.info/listinfo/pd-list
> > 
> > 
> > _______________________________________________
> > Pd-list at lists.iem.at mailing list
> > UNSUBSCRIBE and account-management -> https://lists.puredata.info/l
> istinfo/pd-list
> > 
> 
> 
> _______________________________________________
> Pd-list at lists.iem.at mailing list
> UNSUBSCRIBE and account-management -> https://lists.puredata.info/lis
> tinfo/pd-list
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part
URL: <http://lists.puredata.info/pipermail/pd-list/attachments/20180724/1600adb5/attachment.sig>