Command line shell not flushing stderr when interactive

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Command line shell not flushing stderr when interactive

Christopher Wellons

When the shell is set to interactive (i.e. "-interactive"), the output
(stdout) is flushed with every prompt (shell.c:422) but stderr is not.
In some situations this leads to no error messages being displayed until
the stderr buffer fills.

This happens when running the official sqlite3 binary as subprocess of
Emacs under Windows 7. The error messages do not appear in a timely
fashion. I was unable to trigger the misbehavior in a plain shell so my
only demo is a bit of Emacs Lisp. When this Elisp code below is run, a
buffer will pop up that *should* contain the output of .help. Under
Windows it does not. The same occurs even when it's launched via a shell
subprocess using "2>&1", so it's not simply an issue with Emacs not
reading from the subprocess's stderr output fast enough.

    (let* ((buffer (generate-new-buffer "sqlite"))
           (proc (start-process "sqlite" buffer "sqlite3" "-interactive")))
      (process-send-string proc ".help\n")
      (pop-to-buffer buffer))

I suspect it has to do with being compiled without readline, which is
why it behaves better elsewhere. I couldn't figure out how to link with
libreadline on Windows, though, so I couldn't test this.

With the following change to the amalgamation release I got the behavior
I was looking for: timely error messages from the SQLite command line
shell. I understand this is probably not the Right Way to do this, but
it's just a demonstation of a possible fix.

--- a/shell.c
+++ b/shell.c
@@ -418,6 +418,7 @@ static char *one_input_line(FILE *in, char *zPrior, int isCont
     zResult = readline(zPrompt);
     if( zResult && *zResult ) add_history(zResult);
 #else
+    fflush(stderr);
     printf("%s", zPrompt);
     fflush(stdout);
     zResult = local_getline(zPrior, stdin);
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Command line shell not flushing stderr when interactive

Richard Hipp-3
On Sun, Jan 19, 2014 at 11:10 AM, Christopher Wellons <
[hidden email]> wrote:

>
> When the shell is set to interactive (i.e. "-interactive"), the output
> (stdout) is flushed with every prompt (shell.c:422) but stderr is not.
>

Stderr is suppose to be unbuffered so that flushing is not required.  Or is
that different for windows?



> In some situations this leads to no error messages being displayed until
> the stderr buffer fills.
>
> This happens when running the official sqlite3 binary as subprocess of
> Emacs under Windows 7. The error messages do not appear in a timely
> fashion. I was unable to trigger the misbehavior in a plain shell so my
> only demo is a bit of Emacs Lisp. When this Elisp code below is run, a
> buffer will pop up that *should* contain the output of .help. Under
> Windows it does not. The same occurs even when it's launched via a shell
> subprocess using "2>&1", so it's not simply an issue with Emacs not
> reading from the subprocess's stderr output fast enough.
>
>     (let* ((buffer (generate-new-buffer "sqlite"))
>            (proc (start-process "sqlite" buffer "sqlite3" "-interactive")))
>       (process-send-string proc ".help\n")
>       (pop-to-buffer buffer))
>
> I suspect it has to do with being compiled without readline, which is
> why it behaves better elsewhere. I couldn't figure out how to link with
> libreadline on Windows, though, so I couldn't test this.
>
> With the following change to the amalgamation release I got the behavior
> I was looking for: timely error messages from the SQLite command line
> shell. I understand this is probably not the Right Way to do this, but
> it's just a demonstation of a possible fix.
>
> --- a/shell.c
> +++ b/shell.c
> @@ -418,6 +418,7 @@ static char *one_input_line(FILE *in, char *zPrior,
> int isCont
>      zResult = readline(zPrompt);
>      if( zResult && *zResult ) add_history(zResult);
>  #else
> +    fflush(stderr);
>      printf("%s", zPrompt);
>      fflush(stdout);
>      zResult = local_getline(zPrior, stdin);
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Command line shell not flushing stderr when interactive

Christopher Wellons

>> When the shell is set to interactive (i.e. "-interactive"), the output
>> (stdout) is flushed with every prompt (shell.c:422) but stderr is not.

> Stderr is suppose to be unbuffered so that flushing is not required.  Or is
> that different for windows?

According to the stderr Linux man page stderr is unbuffered, which would
be why I'm not having a problem in Linux:

> The stream stderr is unbuffered.  The  stream  stdout  is  line-buffered
> when  it  points  to  a  terminal.

I'm unable to find any documentation about this for Windows, but since
I'm seeing stderr buffering it must not be unbuffered in Windows.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Command line shell not flushing stderr when interactive

Luuk
On 19-01-2014 19:59, Christopher Wellons wrote:

>
>>> When the shell is set to interactive (i.e. "-interactive"), the output
>>> (stdout) is flushed with every prompt (shell.c:422) but stderr is not.
>
>> Stderr is suppose to be unbuffered so that flushing is not required.  Or is
>> that different for windows?
>
> According to the stderr Linux man page stderr is unbuffered, which would
> be why I'm not having a problem in Linux:
>
>> The stream stderr is unbuffered.  The  stream  stdout  is  line-buffered
>> when  it  points  to  a  terminal.
>
> I'm unable to find any documentation about this for Windows, but since
> I'm seeing stderr buffering it must not be unbuffered in Windows.
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>

Is this not biting you?


It is acceptable—and normal—for standard output and standard error to be
directed to the same destination, such as the text terminal. Messages
appear in the same order as the program writes them, unless buffering is
involved. (For example, a common situation is when the standard error
stream is unbuffered but the standard output stream is line-buffered; in
this case, text written to standard error later may appear on the
terminal earlier, if the standard output stream's buffer is not yet full.)

source:
http://en.wikipedia.org/wiki/Standard_streams#Standard_error_.28stderr.29

_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Command line shell not flushing stderr when interactive

Simon Slavin-3

On 19 Jan 2014, at 7:32pm, Luuk <[hidden email]> wrote:

> It is acceptable—and normal—for standard output and standard error to be directed to the same destination, such as the text terminal. Messages appear in the same order as the program writes them, unless buffering is involved. (For example, a common situation is when the standard error stream is unbuffered but the standard output stream is line-buffered; in this case, text written to standard error later may appear on the terminal earlier, if the standard output stream's buffer is not yet full.)
>
> source:
> http://en.wikipedia.org/wiki/Standard_streams#Standard_error_.28stderr.29

Buffering matters only if an app is going to use stderr as a warning stream instead of its original purpose of "I'm about to crash and here's why.".

When stderr was thought up, a program wrote some text to it just before it quit.  The question of buffering wasn't important because any buffer would be flushed an instant later when the program that wrote it quit.  So it didn't matter whether stderr was buffered or not.

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Command line shell not flushing stderr when interactive

Peter Aronson-3
In reply to this post by Christopher Wellons
Microsoft seems to only make the stderr stream unbuffered when writing to a character device: "The stdout and stderr functions are flushed whenever they are full or, if you are writing to a character device, after each library call."  It doesn't seem to consider pipe that emacs is reading from a character device.  This seems to violate the ISO C standard, which I believe requires stderr to be unbuffered or line-buffered at start-up, but I doubt that Microsoft is losing any sleep over that.
 
My office mate, who also uses Emacs on Windows, modified our local copy of shell.c to deal with this.  First, he made a similar addition of a fflush that you did (our changes are bracketed by ifdef ESRI):
 
static int process_input(struct callback_data *p, FILE *in){
  char *zLine = 0;
  char *zSql = 0;
  int nSql = 0;
  int nSqlPrior = 0;
  char *zErrMsg;
  int rc;
  int errCnt = 0;
  int lineno = 0;
  int startline = 0;
  while( errCnt==0 || !bail_on_error || (in==0 && stdin_is_interactive) ){
#ifdef ESRI
    fflush(stderr);
#endif
    fflush(p->out);
    free(zLine);
 
And made this addition to main to get interactive behavior when running is an Emacs *shell* window:
 
int main(int argc, char **argv){
  char *zErrMsg = 0;
  struct callback_data data;
  const char *zInitFile = 0;
  char *zFirstCmd = 0;
  int i;
  int rc = 0;
  if( strcmp(sqlite3_sourceid(),SQLITE_SOURCE_ID)!=0 ){
    fprintf(stderr, "SQLite header and source version mismatch\n%s\n%s\n",
            sqlite3_sourceid(), SQLITE_SOURCE_ID);
    exit(1);
  }
  Argv0 = argv[0];
  main_init(&data);
  stdin_is_interactive = isatty(0);
#ifdef ESRI
  if (!stdin_is_interactive) {
    /* If Emacs shell window's TERM is set to "emacs".
    ** Then set interactive mode on to redirect STDIN to Emacs shell window.
    */
    char *env_var = getenv("TERM");
    if (env_var) {
      if (!strcmp (env_var, "emacs"))
        stdin_is_interactive = 1;
    }
  }
#endif

Peter

From: Christopher Wellons <[hidden email]>

>To: [hidden email]
>Sent: Sunday, January 19, 2014 9:10 AM
>Subject: [sqlite] Command line shell not flushing stderr when interactive
>
>
>
>When the shell is set to interactive (i.e. "-interactive"), the output
>(stdout) is flushed with every prompt (shell.c:422) but stderr is not.
>In some situations this leads to no error messages being displayed until
>the stderr buffer fills.
>
>This happens when running the official sqlite3 binary as subprocess of
>Emacs under Windows 7. The error messages do not appear in a timely
>fashion. I was unable to trigger the misbehavior in a plain shell so my
>only demo is a bit of Emacs Lisp. When this Elisp code below is run, a
>buffer will pop up that *should* contain the output of .help. Under
>Windows it does not. The same occurs even when it's launched via a shell
>subprocess using "2>&1", so it's not simply an issue with Emacs not
>reading from the subprocess's stderr output fast enough.
>
>    (let* ((buffer (generate-new-buffer "sqlite"))
>          (proc (start-process "sqlite" buffer "sqlite3" "-interactive")))
>      (process-send-string proc ".help\n")
>      (pop-to-buffer buffer))
>
>I suspect it has to do with being compiled without readline, which is
>why it behaves better elsewhere. I couldn't figure out how to link with
>libreadline on Windows, though, so I couldn't test this.
>
>With the following change to the amalgamation release I got the behavior
>I was looking for: timely error messages from the SQLite command line
>shell. I understand this is probably not the Right Way to do this, but
>it's just a demonstation of a possible fix.
>
>--- a/shell.c
>+++ b/shell.c
>@@ -418,6 +418,7 @@ static char *one_input_line(FILE *in, char *zPrior, int isCont
>    zResult = readline(zPrompt);
>    if( zResult && *zResult ) add_history(zResult);
>#else
>+    fflush(stderr);
>    printf("%s", zPrompt);
>    fflush(stdout);
>    zResult = local_getline(zPrior, stdin);
>_______________________________________________
>sqlite-users mailing list
>[hidden email]
>http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
>
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users