Using .testcase and .check in continuous integration test

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Using .testcase and .check in continuous integration test

Lodewijk Duymaer van Twist-2
I would like use .testcase and .check in our GitLab Continuous Integration test.

GitLab pipelines will check process return code for success or fail.

Consider a simple test:
lodewijk@DebianDev:~$ sqlite3 database.db3 < test.sql
testcase-100 ok
testcase-110 ok
lodewijk@DebianDev:~$ echo $?
0

Now if I would have a failure the return value of the sqlite3 process will also be 0:
lodewijk@DebianDev:~$ sqlite3 octalarm.db3 < test/test-languages.sql
testcase-100 FAILED
Expected: [66]
Got: [67
]
lodewijk@DebianDev:~$ echo $?
0

Is there a nice elegant way of making my CI stop on a failure?

Kind regards,

Lodewijk

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Using .testcase and .check in continuous integration test

Dominique Devienne
On Mon, Oct 16, 2017 at 12:32 PM, Lodewijk Duymaer van Twist <
[hidden email]> wrote:

> I would like use .testcase and .check in our GitLab Continuous Integration
> test.
>
> GitLab pipelines will check process return code for success or fail.
>
> Consider a simple test:
> lodewijk@DebianDev:~$ sqlite3 database.db3 < test.sql
> testcase-100 ok
> testcase-110 ok
> lodewijk@DebianDev:~$ echo $?
> 0
>
> Now if I would have a failure the return value of the sqlite3 process will
> also be 0:
> lodewijk@DebianDev:~$ sqlite3 octalarm.db3 < test/test-languages.sql
> testcase-100 FAILED
> Expected: [66]
> Got: [67
> ]
> lodewijk@DebianDev:~$ echo $?
> 0
>
> Is there a nice elegant way of making my CI stop on a failure?
>

C:\Users\ddevienne>sqlite3 -bail bad.db "create table foo(id)" && echo OK
OK

C:\Users\ddevienne>sqlite3 -bail bad.db "create table bar" && echo OK
Error: near "bar": syntax error

C:\Users\ddevienne>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Using .testcase and .check in continuous integration test

Lodewijk Duymaer van Twist-2
Hi Dominique,

That would be an other way, but what I'm looking for is using the Command Line Shell ".testcase" and ".check" method.

Create an example test file:
echo ".testcase 100
SELECT (1,2,3)<(1,2,4), (1,2,3)<(1,NULL,4), (1,2,3)<(2,NULL,1);
.check 1||1

.testcase 110
SELECT (1,2,3)<(1,2,4), (1,2,3)<(1,NULL,4), (1,2,3)<(2,NULL,1);
.check 1|5|1" > test-script.sql

Execute the test:
sqlite3 < test-script.sql

Bail will not work for that.

Kind regards,

Lodewijk


Van: "Dominique Devienne" <[hidden email]>
Aan: "sqlite-users" <[hidden email]>
Verzonden: Maandag 16 oktober 2017 13:49:18
Onderwerp: Re: [sqlite] Using .testcase and .check in continuous integration test

On Mon, Oct 16, 2017 at 12:32 PM, Lodewijk Duymaer van Twist <
[hidden email]> wrote:

> I would like use .testcase and .check in our GitLab Continuous Integration
> test.
>
> GitLab pipelines will check process return code for success or fail.
>
> Consider a simple test:
> lodewijk@DebianDev:~$ sqlite3 database.db3 < test.sql
> testcase-100 ok
> testcase-110 ok
> lodewijk@DebianDev:~$ echo $?
> 0
>
> Now if I would have a failure the return value of the sqlite3 process will
> also be 0:
> lodewijk@DebianDev:~$ sqlite3 octalarm.db3 < test/test-languages.sql
> testcase-100 FAILED
> Expected: [66]
> Got: [67
> ]
> lodewijk@DebianDev:~$ echo $?
> 0
>
> Is there a nice elegant way of making my CI stop on a failure?
>

C:\Users\ddevienne>sqlite3 -bail bad.db "create table foo(id)" && echo OK
OK

C:\Users\ddevienne>sqlite3 -bail bad.db "create table bar" && echo OK
Error: near "bar": syntax error

C:\Users\ddevienne>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users 
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Using .testcase and .check in continuous integration test

Dominique Devienne
On Tue, Oct 17, 2017 at 12:30 PM, Lodewijk Duymaer van Twist <
[hidden email]> wrote:

> That would be an other way, but what I'm looking for is using the Command
> Line Shell ".testcase" and ".check" method.
>

OK. That's new information :)


> Create an example test file:
> echo ".testcase 100
> SELECT (1,2,3)<(1,2,4), (1,2,3)<(1,NULL,4), (1,2,3)<(2,NULL,1);
> .check 1||1
>
> .testcase 110
> SELECT (1,2,3)<(1,2,4), (1,2,3)<(1,NULL,4), (1,2,3)<(2,NULL,1);
> .check 1|5|1" > test-script.sql
>
> Execute the test:
> sqlite3 < test-script.sql
>
> Bail will not work for that.
>

shell.c needs to change for that, by 1 character I believe, in
do_meta_command.
In case of a failure, set rc to 1 instead of 2. That way the shell no
longer exits
with 0, but 1, i.e. an error.

I'm actually surprised the result of do_meta_command() is not subject to
.bail,
and always exits if non-zero.

Either do those changes yourself in your locally compiled shell,
or hope Richard agrees the behavior should change. --DD

        rc = do_meta_command(azCmd[i], &data);
        if( rc ) return rc==2 ? 0 : rc;

static int do_meta_command(char *zLine, ShellState *p){
...
  /* Cancel output redirection, if it is currently set (by .testcase)
  ** Then read the content of the testcase-out.txt file and compare against
  ** azArg[1].  If there are differences, report an error and exit.
  */
  if( c=='c' && n>=3 && strncmp(azArg[0], "check", n)==0 ){
    char *zRes = 0;
    output_reset(p);
    if( nArg!=2 ){
      raw_printf(stderr, "Usage: .check GLOB-PATTERN\n");
      rc = 2;
    }else if( (zRes = readFile("testcase-out.txt", 0))==0 ){
      raw_printf(stderr, "Error: cannot read 'testcase-out.txt'\n");
      rc = 2;
    }else if( testcase_glob(azArg[1],zRes)==0 ){
      utf8_printf(stderr,
                 "testcase-%s FAILED\n Expected: [%s]\n      Got: [%s]\n",
                 p->zTestcase, azArg[1], zRes);
      rc = 2; <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< change to 1
    }else{
      utf8_printf(stdout, "testcase-%s ok\n", p->zTestcase);
      p->nCheck++;
    }
    sqlite3_free(zRes);
  }else
...

meta_command_exit:
  if( p->outCount ){
    p->outCount--;
    if( p->outCount==0 ) output_reset(p);
  }
  return rc;
}
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Using .testcase and .check in continuous integration test

Lodewijk Duymaer van Twist-2
Hi Dominique,

Thank you for investigating. You're fix works. Should I repost this as a bug with your fix, or will this be picked up as is right now?

Kind regard,

Lodewijk


Van: "Dominique Devienne" <[hidden email]>
Aan: "sqlite-users" <[hidden email]>
Verzonden: Dinsdag 17 oktober 2017 13:38:13
Onderwerp: Re: [sqlite] Using .testcase and .check in continuous integration test

On Tue, Oct 17, 2017 at 12:30 PM, Lodewijk Duymaer van Twist <
[hidden email]> wrote:

> That would be an other way, but what I'm looking for is using the Command
> Line Shell ".testcase" and ".check" method.
>

OK. That's new information :)


> Create an example test file:
> echo ".testcase 100
> SELECT (1,2,3)<(1,2,4), (1,2,3)<(1,NULL,4), (1,2,3)<(2,NULL,1);
> .check 1||1
>
> .testcase 110
> SELECT (1,2,3)<(1,2,4), (1,2,3)<(1,NULL,4), (1,2,3)<(2,NULL,1);
> .check 1|5|1" > test-script.sql
>
> Execute the test:
> sqlite3 < test-script.sql
>
> Bail will not work for that.
>

shell.c needs to change for that, by 1 character I believe, in
do_meta_command.
In case of a failure, set rc to 1 instead of 2. That way the shell no
longer exits
with 0, but 1, i.e. an error.

I'm actually surprised the result of do_meta_command() is not subject to
.bail,
and always exits if non-zero.

Either do those changes yourself in your locally compiled shell,
or hope Richard agrees the behavior should change. --DD

rc = do_meta_command(azCmd[i], &data);
if( rc ) return rc==2 ? 0 : rc;

static int do_meta_command(char *zLine, ShellState *p){
...
/* Cancel output redirection, if it is currently set (by .testcase)
** Then read the content of the testcase-out.txt file and compare against
** azArg[1]. If there are differences, report an error and exit.
*/
if( c=='c' && n>=3 && strncmp(azArg[0], "check", n)==0 ){
char *zRes = 0;
output_reset(p);
if( nArg!=2 ){
raw_printf(stderr, "Usage: .check GLOB-PATTERN\n");
rc = 2;
}else if( (zRes = readFile("testcase-out.txt", 0))==0 ){
raw_printf(stderr, "Error: cannot read 'testcase-out.txt'\n");
rc = 2;
}else if( testcase_glob(azArg[1],zRes)==0 ){
utf8_printf(stderr,
"testcase-%s FAILED\n Expected: [%s]\n Got: [%s]\n",
p->zTestcase, azArg[1], zRes);
rc = 2; <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< change to 1
}else{
utf8_printf(stdout, "testcase-%s ok\n", p->zTestcase);
p->nCheck++;
}
sqlite3_free(zRes);
}else
...

meta_command_exit:
if( p->outCount ){
p->outCount--;
if( p->outCount==0 ) output_reset(p);
}
return rc;
}
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users 
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Using .testcase and .check in continuous integration test

Dominique Devienne
On Tue, Oct 17, 2017 at 3:16 PM, Lodewijk Duymaer van Twist <
[hidden email]> wrote:

> Thank you for investigating. You're fix works. Should I repost this as a
> bug with your fix, or will this be picked up as is right now?
>

Glad it did. Just sit tight and again wait and see if Dr Hipp agrees the
behavior should change or not.
I'm hopeful he might, but if he doesn't, there's little you can do about
it. I don't think reposting is necessary for now. --DD
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Using .testcase and .check in continuous integration test

Dominique Devienne
On Tue, Oct 17, 2017 at 3:52 PM Dominique Devienne <[hidden email]>
wrote:

> On Tue, Oct 17, 2017 at 3:16 PM, Lodewijk Duymaer van Twist <
> [hidden email]> wrote:
>
>> Thank you for investigating. You're fix works. Should I repost this as a
>> bug with your fix, or will this be picked up as is right now?
>>
>
> Glad it did. Just sit tight and again wait and see if Dr Hipp agrees the
> behavior should change or not.
> I'm hopeful he might, but if he doesn't, there's little you can do about
> it. I don't think reposting is necessary for now. --DD
>

FYI, Richard checked the change in the day of that post.
Only noticed it today, thus this late answer.  Thank you Richard.
http://www.sqlite.org/src/info/e2af0cc6ef5fafc7
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users