Daily test coverage results published online

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Daily test coverage results published online

Harri Porten
Dear SQLite developers,

as of earlier this week you'll find the daily code coverage results of the
tests bundled with SQLite online at:

   http://www.opencoverage.net/sqlite

With over 89% condition coverage (measured at source code level) your
project is a clear leader among the other projects hosted on that site.
Congratulations!

The site can feature a post-commit analysis of individual patches (for
OpenSSL for example[1]). We just didn't get around integrating
fossil-based repositories, yet. Something that could be done if
considered useful.

We would love to display results split per source file but failed to
compile the code when using the --disable-amalgamation configure switch. I
can dig up the details of the error we got.

If you have any questions about the results or believe that the tests may
not be running properly, yet, please let us[2] know.

Thanks,

Harri.

[1] http://www.opencoverage.net/openssl/analysis/patch1.html

[2] We are makers of a code coverage tool named Coco and consider above
site as our little contribution to open source software as well as a show
case of the tool's capabilities.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Daily test coverage results published online

Richard Hipp-3
On 3/7/18, Harri Porten <[hidden email]> wrote:

> Dear SQLite developers,
>
> as of earlier this week you'll find the daily code coverage results of the
> tests bundled with SQLite online at:
>
>    http://www.opencoverage.net/sqlite
>
> With over 89% condition coverage (measured at source code level) your
> project is a clear leader among the other projects hosted on that site.
> Congratulations!

Thanks for running the tests.  Cool site!

Just so that you will know, we actually run 100% condition coverage
(100% MC/DC) using a separate proprietary test suite.  See
https://www.sqlite.org/testing.html#test_coverage for additional
information.

>
> The site can feature a post-commit analysis of individual patches (for
> OpenSSL for example[1]). We just didn't get around integrating
> fossil-based repositories, yet. Something that could be done if
> considered useful.

If you are interested in adding Fossil repository tracking, please
know that you can always pull down a tarball of the latest trunk
check-in using this url:

    https://sqlite.org/src/tarball/sqlite.tar.gz?id=trunk

If the client you use to query that URL supports standard HTTP caching
using either If-Modified-Since or If-None-Match, then the URL above
will return a 304 Not Modified if there have been no new trunk
check-ins since your last attempt.

Using tarballs such as the above might be an easy way for you to
integrate support for Fossil.  This is, in fact, what OSSFuzz does.
However, to minimize bandwidth, it would be better to have a "fossil"
executable on hand so that you can clone the repository and then
periodically do "fossil update trunk" to get the latest trunk code.
Fossil is a single self-contained executable that you merely place on
your $PATH so it isn't difficult to install.

>
> We would love to display results split per source file but failed to
> compile the code when using the --disable-amalgamation configure switch. I
> can dig up the details of the error we got.

I just ran:

    ./configure --disable-amalgamation; make test

And it ran fine for me.  No errors.  This is on ubuntu.  If you
continue to have problems, please send additional information and we
will look into it.

--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Daily test coverage results published online

Harri Porten
Hi Richard,

thanks for the feedback. And sorry about the late reply:

On Wed, 7 Mar 2018, Richard Hipp wrote:

>> as of earlier this week you'll find the daily code coverage results of the
>> tests bundled with SQLite online at:
>>
>>    http://www.opencoverage.net/sqlite
>>
>> With over 89% condition coverage (measured at source code level) your
>> project is a clear leader among the other projects hosted on that site.
>> Congratulations!
>
> Thanks for running the tests.  Cool site!
>
> Just so that you will know, we actually run 100% condition coverage
> (100% MC/DC) using a separate proprietary test suite.  See
> https://www.sqlite.org/testing.html#test_coverage for additional
> information.

Saw that. And this statement made us interested in SQLite as the project
obviously pays attention to testing.

>> The site can feature a post-commit analysis of individual patches (for
>> OpenSSL for example[1]). We just didn't get around integrating
>> fossil-based repositories, yet. Something that could be done if
>> considered useful.
>
> If you are interested in adding Fossil repository tracking, please
> know that you can always pull down a tarball of the latest trunk
> check-in using this url:
>
>    https://sqlite.org/src/tarball/sqlite.tar.gz?id=trunk

We are manging the repository updates with the pre-build version of the
Fossil tool just fine. What's merely missing is an extraction of the last
e.g. 10 patches with an analysis of their coverage.  Shouldn't be too
difficult once someone finds spare 30 minutes.

>> We would love to display results split per source file but failed to
>> compile the code when using the --disable-amalgamation configure switch. I
>> can dig up the details of the error we got.
>
> I just ran:
>
>    ./configure --disable-amalgamation; make test
>
> And it ran fine for me.  No errors.  This is on ubuntu.  If you
> continue to have problems, please send additional information and we
> will look into it.

You are right. Sorry about the false alarm. This was an internal
communication and scripting error. The results are now based on the
original source files and therefore much easier to read:

   http://www.opencoverage.net/sqlite/index_html/sources.html

And a filter is now set that limits the analysis to the sqlite/src
sub-directory.

Harri.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users