[PD-dev] broken msgs cause Tcl traces, how to debug it?

[Hans-Christoph Steiner]

On Jan 23, 2012, at 12:42 PM, Mathieu Bouchard wrote:

> Le 2012-01-22 à 21:54:00, Miller Puckette a écrit :
>> On Mon, Jan 23, 2012 at 12:45:09AM -0500, Mathieu Bouchard wrote:
>>> That's called a newline... not preceded by a backslash. The thing
>>> with fconfigure -buffering line is that it doesn't care about
>>> backslashes, whereas eval does, and if you use both together, you
>>> need to account for that difference.
> 
> Sorry, the problem is indeed more complex than just looking for backslashes before newlines. It has to do with counting braces. The most efficient way to do it would be a parser that keeps its own state so that after returning from [info complete] you don't need to call it again from scratch, but that does not exist in Tcl (it's possible that there's a Tcl extension that you could compile for that, but I wouldn't know about it).
> 
> The way Pd is designed, sys_gui has never any obligation to contain a whole statement, so there isn't much that you could do in sys_gui to help the situation for all externals at once. If it were the case, you could for example use nul-terminators ('\0' aka (char)0) to make the parsing much easier on the receiving end.
> 
> However, if you make the assumption that any statement started within a wb-function must be terminated in the same wb-function, then the caller of the wb-function can call some kind of cleanup that adds a nul-terminator iff number of bytes written by sys_gui >= 0 (since last termination). Same goes with t_guicallbackfn.
> 
> But there are a number of cases of sys_gui being called from elsewhere. Considering end of t_clock run and end of t_pollfn run as being ends of statement could help, as well as _setup function. Is there any other case that wouldn't be covered by that ?

The problem isn't related to sys_gui not enforcing full Tcl commands since the breaks don't happen along the lines of sys_gui command contents.  The only time I've seen a problem is when the commands are broken in the middle of a sys_gui post.


Here's a transcript of my discussion on ##tcl which has some things to try:

Hans-Christoph Steiner
I'm having a weird socket buffering issue in my app, Pd.  It is two processes: a pure Tcl GUI (pd-gui) and a C engine (pd) which communication over a TCP socket.  In pd-gui, I setup the socket to get the data like this:
   fconfigure $sock -blocking 0 -buffering line -encoding utf-8;
   fileevent $sock readable {::pd_connect::pd_readsocket ""}

then pd_readsocket reads from the socket using [read $pd_socket]. pd sends lots of Tcl commands to pd-gui which are then executed in pd_readsocket by doing:

   if {[catch {uplevel #0 $cmd_from_pd} errorname]} {

Mostly this works well, but occasionally, the $cmd_from_pd contents are not broken at the new line, but instead split in this middle of a Tcl command.

<thommey> you could use [gets] instead of [reads] and process by-line
<Gotisch> shouldnt -buffering line prevent that?
<Gotisch> i mean the problem he is descripting

Hans-Christoph Steiner
actually, mostly, we use "-buffering none"

<Gotisch> descripting!
<Gotisch> terrible

Hans-Christoph Steiner
but "-buffering line" doesn't seem to change the behavior

<Gotisch> well shouldn't -buffering line do exactly that
<Gotisch> mh maybe it just means
<Gotisch> be only readable if at least a line is ready to be read

aku
Hans - The OS is free to break packets wherever it wants. You cannot rely on the data in one puts arriving as a single read. This is outsaide of Tcl's control. You have to [read] into a buffer, and use [info complete] to see when a command is complete and can be executed.

Hans-Christoph Steiner
the funny little detail is that when it breaks in the middle of a Tcl command, it is always exactly at the 48kb mark. exactly.

<Gotisch> i dont understand it then
<Gotisch> "If newValue is line, then the I/O system will automatically flush output for the channel whenever a newline character is output."

aku
Some OS setting I guess, or default, or whatnot. The -buffering on the sender side only controls when Tcl pushes data into the OS. It doesn't prevent the OS from packaging things as it likes

<Gotisch> doesnt that mean you get the readable directly after end of line?
<Gotisch> oh
<Gotisch> well then i would go with thommeys suggestion

aku
[gets] can help, assuming that each command is a single line. If not, you still have to accumulate lines until [info complete] on the buffers singlas that the command is complete

<Gotisch> would need to check for linebreaks in buffer too though
<Gotisch> or you could go from 0.5 commands to 1.5 commands and so on
<Gotisch> well very unrealistic scenario

Hans-Christoph Steiner
unfortunately, there can be commands with escaped newlines in them, so I don't think that [gets] will work

<Gotisch> i dont think gets would stop at escaped new lines

Hans-Christoph Steiner
oh, really? something to try

<thommey> [gets] performs no output processing, you will still have to check for \ at the end of the line yourself and then read the next one and buffer

Hans-Christoph Steiner
thommey: that seems doable, better than running [info complete] each time, I suppose
<Gotisch> i guess i didnt know what i was talking about again 

Hans-Christoph Steiner
the 48k size doesn't ring any bells for anyone?

<Gotisch> maybe its the buffer size
<Gotisch> on the c side that is

Hans-Christoph Steiner
I was playing around with the buffring on the C/pd side, and that didn't change the 48k break size

<Gotisch> ah

Hans-Christoph Steiner
I changed it anywhere from 1k to 128k

kbk
It's someone's buffer limit.  Whose, not sure.  But the solutions that aku is describing will work.  Use [gets] and [info complete] -- and be prepared to handle the [fblocked] return from [gets]

Hans-Christoph Steiner
kbk: so this [gets] technique, I would need to "fconfigure -blocking 1"? would I still use:

   fileevent $sock readable {::pd_connect::pd_readsocket ""}   ?

aku
No, it is still -blocking 0 on the receiver side. Small example, see:
http://www.activestate.com/blog/2010/05/concurrency-tcl-events-without-getting-twisted
[gets] returns -1 if the line is incomplete. You can then simply try again at the next event i.e. see if the line is has become complete.

.hc

----------------------------------------------------------------------------

If nature has made any one thing less susceptible than all others of exclusive property, it is the action of the thinking power called an idea, which an individual may exclusively possess as long as he keeps it to himself; but the moment it is divulged, it forces itself into the possession of everyone, and the receiver cannot dispossess himself of it.            - Thomas Jefferson