Quantcast

strange behaviour on sqlite shell output…

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

strange behaviour on sqlite shell output…

Andreas Otto-5
Hi…i have written a VTAB extension and using SHELL to view the results…

+ valgrind --leak-check=full --quiet gen/cltdb -column -echo -header
gen/.my1 'select Reporter, Partner, Year, PartnerCode from [EXEC-BtH0]
where Reporter = '\''AGO'\'' and
select Reporter, Partner, Year, PartnerCode from [EXEC-BtH0] where
Reporter = 'AGO' and Partner = 'AUT'
Reporter    Partner     Year        PartnerCode
----------  ----------  ----------  ----------------
AGO         AUT         2000        EO EU WE €O $H
AGO         AUT         2001        EO EU WE €O $H
AGO         AUT         2002        EO EU WE €O $H
AGO         AUT         2003        EO EU WE €O $H
AGO         AUT         2004        EO EU WE €O $H
AGO         AUT         2005        EO EU WE €O $H
AGO         AUT         2006        EO EU WE €O $H
AGO         AUT         2007        EO EU WE €O $H
AGO         AUT         2008        EO EU WE €O $H
AGO         AUT         2009        EO EU WE €O $H
AGO         AUT         2010        EO EU WE €O $H
AGO         AUT         2011        EO EU WE €O $H
AGO         AUT         2012        EO EU WE €O $H
AGO         AUT         2013        EO EU WE €O $H
AGO         AUT         2014        EO EU WE €O $H
AGO         AUT         2015        EO EU WE €O $H

→ I have problem with the "PartnerCode" column… with the € sign… for
DETAIL "where"
(where Reporter = '\''AGO'\'' and Partner = '\''AUT'\''') everything
looks fine… "EO EU WE €O $H"

with LESS DETAIL "where"

+ valgrind --leak-check=full --quiet gen/cltdb -column -echo -header
gen/.my1 'select Reporter, Partner, Year, PartnerCode from [EXEC-BtH0]
where Reporter = '\''AGO'\'''
select Reporter, Partner, Year, PartnerCode from [EXEC-BtH0] where
Reporter = 'AGO'
Reporter    Partner     Year        PartnerCode
----------  ----------  ----------  -----------
AGO         ABW         2000        CA $H
AGO         ABW         2001        CA $H
AGO         ABW         2002        CA $H
...
AGO         AUS         2009        OC FR E5 $H
AGO         AUS         2010        OC FR E5 $H
AGO         AUS         2011        OC FR E5 $H
AGO         AUS         2012        OC FR E5 $H
AGO         AUS         2013        OC FR E5 $H
AGO         AUS         2014        OC FR E5 $H
AGO         AUS         2015        OC FR E5 $H
AGO         AUT         2000        EO EU WE 342202
AGO         AUT         2001        EO EU WE 342202
AGO         AUT         2002        EO EU WE 342202
AGO         AUT         2003        EO EU WE 342202
AGO         AUT         2004        EO EU WE 342202
AGO         AUT         2005        EO EU WE 342202
AGO         AUT         2006        EO EU WE 342202
AGO         AUT         2007        EO EU WE 342202
AGO         AUT         2008        EO EU WE 342202
AGO         AUT         2009        EO EU WE 342202
AGO         AUT         2010        EO EU WE 342202
AGO         AUT         2011        EO EU WE 342202
AGO         AUT         2012        EO EU WE 342202
AGO         AUT         2013        EO EU WE 342202
AGO         AUT         2014        EO EU WE 342202
AGO         AUT         2015        EO EU WE 342202
AGO         AZE         2000        AS ZW $h
AGO         AZE         2001        AS ZW $h

 > i get strange numbers…"342202"

→ valgrind leakcheck says…everything is fine.

my compile command is…

ccache gcc -DNUM2 -c -o gen/shell.o -ggdb -Wall -Wextra -Isqlite
-Ilibmsgque -Igen -I../libclt/build -D_HAVE_SQLITE_CONFIG_H -DDEBUG=1
-DHAVE_READLINE=1 \
         -Wno-extra -Wno-unused-value -Wno-unused-but-set-variable
-Wno-unused-function \
         shell.c
bin/Header.bash cltraw.c 1>gen/cltraw.h
ccache gcc -DNUM1 -c -o gen/cltdb.o -ggdb -Wall -Wextra -Isqlite
-Ilibmsgque -Igen -I../libclt/build -D_HAVE_SQLITE_CONFIG_H -DDEBUG=1
-Wno-unused-function cltdb.c
ccache gcc -DNUM1 -c -o gen/cltraw.o -ggdb -Wall -Wextra -Isqlite
-Ilibmsgque -Igen -I../libclt/build -D_HAVE_SQLITE_CONFIG_H -DDEBUG=1
-Wno-unused-function cltraw.c
ccache gcc -o gen/cltdb -ggdb -lreadline -lncurses gen/shell.o
gen/cltdb.o gen/cltraw.o ../libclt/build/libclt.a
libmsgque/.libs/libmsgque.a
ccache gcc -DNUM3 -std=gnu99 -c -o gen/abrain.o -ggdb -Wall -Wextra
-Isqlite -Ilibmsgque -Igen -I../libclt/build -D_HAVE_SQLITE_CONFIG_H  
-DDEBUG=1 -DMQ_IGNORE_EXTERN \
         -Wno-ignored-qualifiers -Wno-unused-parameter \
         abrain.c
ccache gcc -o gen/msqdb -ggdb gen/abrain.o gen/cltdb.o gen/cltraw.o
../libclt/build/libclt.a libmsgque/.libs/libmsgque.a
make[1]: Leaving directory '/home/dev1usr/WorldBank/clt/db'


thanks for your support !!



_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: strange behaviour on sqlite shell output…

Andreas Otto-5
Am 15.04.2017 um 09:49 schrieb aotto:
> select Reporter, Partner, Year, PartnerCode from [EXEC-BtH0] where
> Reporter = 'AGO'

Hi,

I add the following code in shell.c
         if( w<0 ){
           utf8_printf(p->out,"%*.*s%s",-w,-w,
               azArg[i] ? azArg[i] : p->nullValue,
               i==nArg-1 ? rowSep : "  ");
         }else{
           utf8_printf(p->out,"%-*.*s%s",w,w,
               azArg[i] ? azArg[i] : p->nullValue,
               i==nArg-1 ? rowSep : "  ");
printC(azArg[i])
         }
using "printC" (this is a macro end op with…
→ #define printC(var)     MVP(MQ_FORMAT_C,var)
 > #define MVP(f,v) fprintf(MQ_OUT,"%s(%s:%d:%d) -> %p:" #v "<" f ">\n",
__func__, __FILE__, __LINE__, mq_getpid(), \
               v,v);fflush(MQ_OUT);
→#define MQ_OUT p->out
→ is is just a "fprintf"…

I get the following lines…

AGO         shell_callback(shell.c:1094:1) -> 0xa0b9c0:azArg[i]<AGO>
AUT         shell_callback(shell.c:1094:1) -> 0x9e7f88:azArg[i]<AUT>
2000        shell_callback(shell.c:1094:1) -> 0x5a9de78:azArg[i]<2000>
EO EU WE 342202
shell_callback(shell.c:1094:1) -> 0x9e7fab:azArg[i]<EO EU WE €O $H>
AGO         shell_callback(shell.c:1094:1) -> 0xa0b9c0:azArg[i]<AGO>
AUT         shell_callback(shell.c:1094:1) -> 0x9e7f88:azArg[i]<AUT>
2001        shell_callback(shell.c:1094:1) -> 0x5a9de78:azArg[i]<2001>
EO EU WE 342202
shell_callback(shell.c:1094:1) -> 0x9e7fab:azArg[i]<EO EU WE €O $H>

summary: the default "fprintf" write the right "€" sign and the sqlite
"utf8_printf"
seems to do something wrong.


mfg ao.

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: strange behaviour on sqlite shell output…

Andreas Otto-5
Am 17.04.2017 um 10:26 schrieb aotto:

> Am 15.04.2017 um 09:49 schrieb aotto:
>> select Reporter, Partner, Year, PartnerCode from [EXEC-BtH0] where
>> Reporter = 'AGO'
>
> Hi,
>
> I add the following code in shell.c
>         if( w<0 ){
>           utf8_printf(p->out,"%*.*s%s",-w,-w,
>               azArg[i] ? azArg[i] : p->nullValue,
>               i==nArg-1 ? rowSep : "  ");
>         }else{
>           utf8_printf(p->out,"%-*.*s%s",w,w,
>               azArg[i] ? azArg[i] : p->nullValue,
>               i==nArg-1 ? rowSep : "  ");
> printC(azArg[i])
>         }
> using "printC" (this is a macro end op with…
> → #define printC(var)     MVP(MQ_FORMAT_C,var)
> > #define MVP(f,v) fprintf(MQ_OUT,"%s(%s:%d:%d) -> %p:" #v "<" f
> ">\n", __func__, __FILE__, __LINE__, mq_getpid(), \
>               v,v);fflush(MQ_OUT);
> →#define MQ_OUT p->out
> → is is just a "fprintf"…
>
> I get the following lines…
>
> AGO         shell_callback(shell.c:1094:1) -> 0xa0b9c0:azArg[i]<AGO>
> AUT         shell_callback(shell.c:1094:1) -> 0x9e7f88:azArg[i]<AUT>
> 2000        shell_callback(shell.c:1094:1) -> 0x5a9de78:azArg[i]<2000>
> EO EU WE 342202
> shell_callback(shell.c:1094:1) -> 0x9e7fab:azArg[i]<EO EU WE €O $H>
> AGO         shell_callback(shell.c:1094:1) -> 0xa0b9c0:azArg[i]<AGO>
> AUT         shell_callback(shell.c:1094:1) -> 0x9e7f88:azArg[i]<AUT>
> 2001        shell_callback(shell.c:1094:1) -> 0x5a9de78:azArg[i]<2001>
> EO EU WE 342202
> shell_callback(shell.c:1094:1) -> 0x9e7fab:azArg[i]<EO EU WE €O $H>
>
> summary: the default "fprintf" write the right "€" sign and the sqlite
> "utf8_printf"
> seems to do something wrong.
>
>
> mfg ao.
>
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


additional info… this is the code AFTER preprocessing…

       for(i=0; i<nArg; i++){
         int w;
         if( i<(int)(sizeof(p->actualWidth)/sizeof(p->actualWidth[0])) ){
            w = p->actualWidth[i];
         }else{
            w = 10;
         }
         if( p->cMode==9 && azArg[i] && strlen30(azArg[i])>w ){
           w = strlen30(azArg[i]);
         }
         if( i==1 && p->aiIndent && p->pStmt ){
           if( p->iIndent<p->nIndent ){
             fprintf(p->out, "%*.s", p->aiIndent[p->iIndent], "");
           }
           p->iIndent++;
         }
         if( w<0 ){
           fprintf(p->out,"%*.*s%s",-w,-w,
               azArg[i] ? azArg[i] : p->nullValue,
               i==nArg-1 ? rowSep : "  ");
         }else{
           fprintf(p->out,"%-*.*s%s",w,w,
               azArg[i] ? azArg[i] : p->nullValue,
               i==nArg-1 ? rowSep : "  ");
fprintf(p->out,"%s(%s:%d:%d) -> %p:" "azArg[i]" "<" "%s" ">\n",
__func__, "shell.c", 1094, 1, azArg[i],azArg[i]);fflush(p->out);
         }
       }

mfg a.o

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: strange behaviour on sqlite shell output…

Andreas Otto-5
Am 17.04.2017 um 10:34 schrieb aotto:
………
This is the isolated test case this shows the BUG

#include "stdio.h"

int
main ( int argc, char **argv ) {
      // code

     int i=3;
     int w=11; // → ??????
     int nArg=4;
     char *rowSep="\n";
     char *azArg[4]  = {NULL,NULL,NULL,"EO EU WE €O $H"};

     fprintf(stdout,"%-*.*s%s",w,w,
               azArg[i] ? azArg[i] : NULL,
                 i==nArg-1 ? rowSep : "  ");

      return 0; // Indicates that everything went well.
}


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: strange behaviour on sqlite shell output…

Hick Gunter
I don't see any calls to sqlite3() functions in your "isolated test case". Maybe you are having problems with character encoding outside of sqlite?

-----Ursprüngliche Nachricht-----
Von: sqlite-users [mailto:[hidden email]] Im Auftrag von aotto
Gesendet: Montag, 17. April 2017 10:55
An: SQLite mailing list <[hidden email]>
Betreff: Re: [sqlite] strange behaviour on sqlite shell output…

Am 17.04.2017 um 10:34 schrieb aotto:
………
This is the isolated test case this shows the BUG

#include "stdio.h"

int
main ( int argc, char **argv ) {
      // code

     int i=3;
     int w=11; // → ??????
     int nArg=4;
     char *rowSep="\n";
     char *azArg[4]  = {NULL,NULL,NULL,"EO EU WE €O $H"};

     fprintf(stdout,"%-*.*s%s",w,w,
               azArg[i] ? azArg[i] : NULL,
                 i==nArg-1 ? rowSep : "  ");

      return 0; // Indicates that everything went well.
}


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___________________________________________
 Gunter Hick
Software Engineer
Scientific Games International GmbH
FN 157284 a, HG Wien
Klitschgasse 2-4, A-1130 Vienna, Austria
Tel: +43 1 80100 0
E-Mail: [hidden email]

This communication (including any attachments) is intended for the use of the intended recipient(s) only and may contain information that is confidential, privileged or legally protected. Any unauthorized use or dissemination of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender by return e-mail message and delete all copies of the original communication. Thank you for your cooperation.


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: strange behaviour on sqlite shell output…

Richard Hipp-3
On 4/18/17, Hick Gunter <[hidden email]> wrote:
> I don't see any calls to sqlite3() functions in your "isolated test case".
> Maybe you are having problems with character encoding outside of sqlite?

I think the OP is referring to a problem that comes up because the
field width and precision of a printf() format are measured in bytes,
not characters, and if the input is multi-byte UTF then it is possible
for a single character to be cut in half, resulting in goofy output.

I checked in a fix for this yesterday.  See
https://www.sqlite.org/src/timeline?c=f508aff8

--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: strange behaviour on sqlite shell output…

Hick Gunter
>On 4/18/17, Hick Gunter <[hidden email]> wrote:
>> I don't see any calls to sqlite3() functions in your "isolated test case".
>> Maybe you are having problems with character encoding outside of sqlite?
>
>I think the OP is referring to a problem that comes up because the field width and precision of a printf() format are measured in bytes, not characters, and if the input is multi-byte UTF then it is possible for a single character to >be cut in half, resulting in goofy output.
>
>I checked in a fix for this yesterday.  See
>https://www.sqlite.org/src/timeline?c=f508aff8
>

Damn UTF8 ... my "favorite" error is a data source defined as having ISO encoding actually containing (2 byte) UTF8 characters, which, after calling iconv() ISO->UTF8, yield invalid codepoints (2 x 2 bytes) that produce goofy output on terminals and cause mysterious crashes within XML parsers, preferably called from within perl procedures that silently terminate without any indication of the cause.


___________________________________________
 Gunter Hick
Software Engineer
Scientific Games International GmbH
FN 157284 a, HG Wien
Klitschgasse 2-4, A-1130 Vienna, Austria
Tel: +43 1 80100 0
E-Mail: [hidden email]

This communication (including any attachments) is intended for the use of the intended recipient(s) only and may contain information that is confidential, privileged or legally protected. Any unauthorized use or dissemination of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender by return e-mail message and delete all copies of the original communication. Thank you for your cooperation.


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: strange behaviour on sqlite shell output…

Rowan Worth-2
On 18 April 2017 at 16:28, Hick Gunter <[hidden email]> wrote:

> Richard Hipp wrote:
> >I think the OP is referring to a problem that comes up because the field
> width and precision of a printf() format are measured in bytes, not
> characters, and if the input is multi-byte UTF then it is possible for a
> single character to >be cut in half, resulting in goofy output.
> >
> >I checked in a fix for this yesterday.  See
> >https://www.sqlite.org/src/timeline?c=f508aff8
> >
>
> Damn UTF8 ... my "favorite" error is a data source defined as having ISO
> encoding actually containing (2 byte) UTF8 characters, which, after calling
> iconv() ISO->UTF8, yield invalid codepoints (2 x 2 bytes) that produce
> goofy output on terminals and cause mysterious crashes within XML parsers,
> preferably called from within perl procedures that silently terminate
> without any indication of the cause.
>

Right, so
1. the encoding is mislabeled
2. iconv is invoked to change the encoding
3. garbage in garbage out
4. this is utf-8's fault?

The elegance of utf-8 lies in the fact that (a) the entire unicode space is
representable using the one encoding and (b) it's backwards compatible with
ASCII. A corollary of (a) is that in a utf-8 system no transcoding is
required - ie. as soon as you invoke iconv you are complaining about
*other* encodings, not utf-8 :P

-Rowan (I did find the cascade of errors amusing :) )
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: strange behaviour on sqlite shell output…

Hick Gunter
If only Hollerith had remembered his lingual heritage...

-----Ursprüngliche Nachricht-----
Von: sqlite-users [mailto:[hidden email]] Im Auftrag von Rowan Worth
Gesendet: Dienstag, 18. April 2017 10:41
An: SQLite mailing list <[hidden email]>
Betreff: Re: [sqlite] strange behaviour on sqlite shell output…

On 18 April 2017 at 16:28, Hick Gunter <[hidden email]> wrote:

> Richard Hipp wrote:
> >I think the OP is referring to a problem that comes up because the
> >field
> width and precision of a printf() format are measured in bytes, not
> characters, and if the input is multi-byte UTF then it is possible for
> a single character to >be cut in half, resulting in goofy output.
> >
> >I checked in a fix for this yesterday.  See
> >https://www.sqlite.org/src/timeline?c=f508aff8
> >
>
> Damn UTF8 ... my "favorite" error is a data source defined as having
> ISO encoding actually containing (2 byte) UTF8 characters, which,
> after calling
> iconv() ISO->UTF8, yield invalid codepoints (2 x 2 bytes) that produce
> goofy output on terminals and cause mysterious crashes within XML
> parsers, preferably called from within perl procedures that silently
> terminate without any indication of the cause.
>

Right, so
1. the encoding is mislabeled
2. iconv is invoked to change the encoding 3. garbage in garbage out 4. this is utf-8's fault?

The elegance of utf-8 lies in the fact that (a) the entire unicode space is representable using the one encoding and (b) it's backwards compatible with ASCII. A corollary of (a) is that in a utf-8 system no transcoding is required - ie. as soon as you invoke iconv you are complaining about
*other* encodings, not utf-8 :P

-Rowan (I did find the cascade of errors amusing :) ) _______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___________________________________________
 Gunter Hick
Software Engineer
Scientific Games International GmbH
FN 157284 a, HG Wien
Klitschgasse 2-4, A-1130 Vienna, Austria
Tel: +43 1 80100 0
E-Mail: [hidden email]

This communication (including any attachments) is intended for the use of the intended recipient(s) only and may contain information that is confidential, privileged or legally protected. Any unauthorized use or dissemination of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender by return e-mail message and delete all copies of the original communication. Thank you for your cooperation.


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Loading...