[PD] Large File Support on Linux

[Miller Puckette]

OK... now I'm trying to fix the bug but I can't reproduce it.  I made a
5 GB soundfile (wave format, 60 channels, 4-bit floats, 450 seconds long)
and opened and am reading it using readsf~.  This is on a 64 bit linux
box.

Question for Charles Goyard: were you on a 32-bit linux machine?  Or did
you do something else besides just read the file from its beginning?

thanks
Miller

On Wed, Mar 20, 2013 at 06:20:59PM -0700, Miller Puckette wrote:
> I believe not - every extern that used open_bia_path0 would have to be
> fixed at the source level to use lseek64(), fstat64() in place of
> lseek() and fstat().  I can't see any way to "fix" this source-compatibly.
> 
> I checked open64() on 32-bit Raspbian (ARM) and it worked fine - so I think
> it's save to say open64 exists on all modern linuxes, both 64 and 32 bit.
> 
> cheers
> Miller
> On Wed, Mar 20, 2013 at 09:14:42PM -0400, Ivica Ico Bukvic wrote:
> > Miller,
> > 
> > Does this mean if one were not so concerned with backwards compatibility and
> > included define you listed below in the s_path.c that this would fix the LFS
> > issue albeit at the expense of backwards/cross-platform compatibility? Also,
> > in Linux is open64 safe for both 32-bit and 64-bit OS variants?
> > 
> > > -----Original Message-----
> > > From: pd-list-bounces at iem.at [mailto:pd-list-bounces at iem.at] On Behalf Of
> > > Miller Puckette
> > > Sent: Wednesday, March 20, 2013 6:46 PM
> > > To: IOhannes m zmölnig
> > > Cc: pd-list at iem.at
> > > Subject: Re: [PD] Large File Support on Linux
> > > 
> > > OK.. I think I'm sold (for 0.45 :) on trying to get sys_open, etc., to do
> > > the "right" thing.  I guess on Windows this would have to be somehow both
> > > UTF8 and 64-bit safe - all the more reason to do as you suggest and hide
> > the
> > > whole mes behind sys_this_and_that() variants.
> > > 
> > > I thought exactly the same thing about a find_via_path routine.  It seems
> > to
> > > me that open_via_path, which tries not just to verify that the file exists
> > but
> > > also that it can actually be opened, is perhaps working too hard; if a
> > file
> > > that proves impossible to open is in an earlier spot on the path, the best
> > > thing might be to try and fail to open it instead of going out to another,
> > > perhaps incorrect, version of the file.
> > > 
> > > Also, it could be fixed to take the path as an argument so that
> > d_soundfile.c
> > > can call it from other threads safely (using separate copies of the path).
> > > 
> > > cheers
> > > Miller
> > > 
> > > On Wed, Mar 20, 2013 at 09:42:42PM +0100, IOhannes m zmölnig wrote:
> > > > On 03/20/2013 18:02, Miller Puckette wrote:
> > > > > On further thought, I don't think it's even possible to change
> > > open_via_path()
> > > > > to use open64() and maintain even source compatibility for externs -
> > if
> > > > > one of them calls lseek or fstat we're sunk - and I don't see any
> > robust
> > > > > way of aliasing those calls to lseek64() etc.
> > > >
> > > > yep.
> > > > that's why i said that the only real solution would be to provide a
> > > > more-or-less complete set of fileIO-functions: sys_lseek, sys_stat
> > > > (including f-variants).
> > > >
> > > > >
> > > > > I'm now realizing too that I don't know what approaches the Mac and
> > > Microsoft
> > > > > pltforms are taking to large file support - I think any solution will
> > have to
> > > > > somehow address all of the platforms.
> > > >
> > > > which would be possible if we provided the full set. of file accessors.
> > > > (but to repeat, this is not really a Pd-problem, and could (and maybe
> > > > should) be solved in an auxiliary lib).
> > > >
> > > > > For right now I'd like a way to fix 0.44's sfread~ - I'm thinking I
> > can do
> > > > > that internally to d_soundfile.c so as to cause the least possible
> > disruption
> > > > > in a 'bugfix' pd release (probably 0.44-3).
> > > >
> > > > which i think is a good enough approach.
> > > > but of course there's a catch: sys_open() will gracefully handle UTF-8
> > > > filenames, whereas on some platforms open() will not.
> > > > so with your solution, soundfiles with non-ascii chars will fail to open
> > > > on W32.
> > > >
> > > > since open_via_path() is so often used to determine the full path of a
> > > > file somewhere in the searchpath, it might be a good idea to wrap that
> > > > functionality into a find_via_path() function that returns the
> > > > canonicalized filename. it would be great if that filename would be
> > > > mangled in a way so UTF8 doesn't make problems any more, but i think
> > > > this is simply not possible with the way w32 handles opening such
> > > filenames.
> > > >
> > > >
> > > > fgmadsr
> > > > IOhannes
> > > >
> > > > _______________________________________________
> > > > Pd-list at iem.at mailing list
> > > > UNSUBSCRIBE and account-management ->
> > > http://lists.puredata.info/listinfo/pd-list
> > > 
> > > _______________________________________________
> > > Pd-list at iem.at mailing list
> > > UNSUBSCRIBE and account-management ->
> > > http://lists.puredata.info/listinfo/pd-list
> > 
> 
> _______________________________________________
> Pd-list at iem.at mailing list
> UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list