SQLite 3.16.0 enters testing

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

SQLite 3.16.0 enters testing

Richard Hipp-3
SQLite version 3.16.0 has entered testing.  Only bug fixes will go on
trunk until the release.

Please download and test the latest pre-release snapshot from
https://www.sqlite.org/download.html and report any problems to this
mailing list, or directly to me.

Change log for the 3.16.0 release:
https://www.sqlite.org/draft/releaselog/3_16_0.html

Draft 3.16.0 website: https://www.sqlite.org/draft/

A new release checklist is up at
https://www.sqlite.org/checklists/3160000/index - the release will
occur when that checklist goes all green.

--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: SQLite 3.16.0 enters testing

Donald Griggs
Tiny typo in docs:

Page: https://www.sqlite.org/draft/imposter.html

"This will not cause any immediately problems"  (remove "ly")


>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: SQLite 3.16.0 enters testing

Richard Hipp-3
On 12/29/16, Donald Griggs <[hidden email]> wrote:
> Tiny typo in docs:
>
> "This will not cause any immediately problems"  (remove "ly")

Fixed.  Thanks for reporting.

--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: SQLite 3.16.0 enters testing

Donald Griggs
In reply to this post by Donald Griggs
Sorry to have dribbled these out:

From the same page: https://www.sqlite.org/draft/imposter.html

"can be accessed as if it where a "   (were)

"table can be queries to see" (queried)

"Each table and each index in SQLite is stored"  (It's a trivial nit at any
rate, but I suspect should be "are stored."
http://www.quickanddirtytips.com/education/grammar/compound-subjects )



On Thu, Dec 29, 2016 at 2:17 PM, Donald Griggs <[hidden email]> wrote:

> Tiny typo in docs:
>
> Page: https://www.sqlite.org/draft/imposter.html
>
> "This will not cause any immediately problems"  (remove "ly")
>
>
>>
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: SQLite 3.16.0 enters testing

Simon Slavin-3
In reply to this post by Richard Hipp-3

On 29 Dec 2016, at 6:52pm, Richard Hipp <[hidden email]> wrote:

> Change log for the 3.16.0 release:
> https://www.sqlite.org/draft/releaselog/3_16_0.html

In which I find ...

> • This feature could be used to implement information schema by first creating a separate schema using
>
> ATTACH
>  ':memory:' AS 'information_schema';
>
> Then creating a VIEWs in that schema that implement the official information schema tables using table-valued PRAGMA functions.

I anticipate three different implementations of this in the next three months.

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: SQLite 3.16.0 enters testing

Simon Slavin-3
In reply to this post by Richard Hipp-3

On 29 Dec 2016, at 6:52pm, Richard Hipp <[hidden email]> wrote:

> Draft 3.16.0 website: https://www.sqlite.org/draft/

The documentation for the shell tool

<https://www.sqlite.org/draft/cli.html>

does not yet reflect the new dot commands (.imposter and .mode quote ?  others ?).

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: SQLite 3.16.0 enters testing

Richard Hipp-3
All corrections should be caught up now.  Please continue proofreading!


--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: SQLite 3.16.0 enters testing

Simon Slavin-3

On 29 Dec 2016, at 8:09pm, Richard Hipp <[hidden email]> wrote:

> All corrections should be caught up now.  Please continue proofreading!

You fixed the problem I noted.  I’m itching to see if I can make '.mode quote' do the wrong thing with some of my test data.

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: SQLite 3.16.0 enters testing

Bob Friesenhahn
In reply to this post by Richard Hipp-3
On Thu, 29 Dec 2016, Richard Hipp wrote:
>
> Change log for the 3.16.0 release:
> https://www.sqlite.org/draft/releaselog/3_16_0.html

What caught my attention the most about the release log was the "Uses
9% fewer CPU cycles" and link to https://www.sqlite.org/draft/cpu.html 
where it describes using valgrind's cachegrind to meausure CPU cycle
usage.

While the CPU is instrumented (by valgrind) at the instruction level
and cache misses are recorded, it seems to me that cachegrind is an
advanced simulation.  Is there a way to know how well cachegrind CPU
cycles map to real-world CPU usage?  If sqlite3 consumes 2X less "CPU
cycles" since 2009 is there a way to measure how much less actual CPU
time that it takes in order to validate that there is a clear
relationship?

Bob
--
Bob Friesenhahn
[hidden email], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: SQLite 3.16.0 enters testing

Richard Hipp-3
On 12/29/16, Bob Friesenhahn <[hidden email]> wrote:
> Is there a way to know how well cachegrind CPU
> cycles map to real-world CPU usage?

Not that I know of.  If you have any suggestions, please speak up.

I tried to make clear in the docs that I consider cachegrind to be a
proxy for CPU usage rather than a precise measurement.
--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: SQLite 3.16.0 enters testing

Bob Friesenhahn
On Thu, 29 Dec 2016, Richard Hipp wrote:

> On 12/29/16, Bob Friesenhahn <[hidden email]> wrote:
>> Is there a way to know how well cachegrind CPU
>> cycles map to real-world CPU usage?
>
> Not that I know of.  If you have any suggestions, please speak up.
>
> I tried to make clear in the docs that I consider cachegrind to be a
> proxy for CPU usage rather than a precise measurement.

If the proxy maps well to real results, then the improved results are
a considerable accomplishment.

At which point in the timeline shown at
https://www.sqlite.org/draft/cpu.html did sqlite developers start
tuning the implementation based on cachegrind results?

Thanks,

Bob
--
Bob Friesenhahn
[hidden email], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: SQLite 3.16.0 enters testing

Richard Hipp-3
On 12/29/16, Bob Friesenhahn <[hidden email]> wrote:
>
> At which point in the timeline shown at
> https://www.sqlite.org/draft/cpu.html did sqlite developers start
> tuning the implementation based on cachegrind results?
>

I discovered this technique starting at about 3.8.0.
--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: SQLite 3.16.0 enters testing

Richard Hipp-3
On 12/29/16, Richard Hipp <[hidden email]> wrote:
> On 12/29/16, Bob Friesenhahn <[hidden email]> wrote:
>>
>> At which point in the timeline shown at
>> https://www.sqlite.org/draft/cpu.html did sqlite developers start
>> tuning the implementation based on cachegrind results?
>>
>
> I discovered this technique starting at about 3.8.0.

Just to confirm:  I looked back through the old release checklists
(https://www.sqlite.org/checklists/) and the first reference to
cachegrind seems to be for 3.8.0, in the checklist item 25b.

--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: SQLite 3.16.0 enters testing

Bob Friesenhahn
On Thu, 29 Dec 2016, Richard Hipp wrote:
>>
>> I discovered this technique starting at about 3.8.0.
>
> Just to confirm:  I looked back through the old release checklists
> (https://www.sqlite.org/checklists/) and the first reference to
> cachegrind seems to be for 3.8.0, in the checklist item 25b.

That would explain the inflection point and more rapid progress
starting at that time.

Bob
--
Bob Friesenhahn
[hidden email], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: SQLite 3.16.0 enters testing

Dominique Pellé
In reply to this post by Richard Hipp-3
Richard Hipp <[hidden email]> wrote:

> On 12/29/16, Bob Friesenhahn <[hidden email]> wrote:
>> Is there a way to know how well cachegrind CPU
>> cycles map to real-world CPU usage?
>
> Not that I know of.  If you have any suggestions, please speak up.

The 'stat' command of Linux perf tool would be a good
candidate to replace cachgrind stats. It's much faster
than cachegrind. I suppose that it also gives more realistic
results. See:

https://perf.wiki.kernel.org/index.php/Tutorial#Counting_with_perf_stat

Dominique
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: SQLite 3.16.0 enters testing

Darko Volaric
What are you basing that theory on?

On Thu, Dec 29, 2016 at 10:29 PM, Dominique Pellé <[hidden email]
> wrote:

> Richard Hipp <[hidden email]> wrote:
>
> > On 12/29/16, Bob Friesenhahn <[hidden email]> wrote:
> >> Is there a way to know how well cachegrind CPU
> >> cycles map to real-world CPU usage?
> >
> > Not that I know of.  If you have any suggestions, please speak up.
>
> The 'stat' command of Linux perf tool would be a good
> candidate to replace cachgrind stats. It's much faster
> than cachegrind. I suppose that it also gives more realistic
> results. See:
>
> https://perf.wiki.kernel.org/index.php/Tutorial#Counting_with_perf_stat
>
> Dominique
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: SQLite 3.16.0 enters testing

Bob Friesenhahn
On Thu, 29 Dec 2016, Darko Volaric wrote:

> What are you basing that theory on?

Perf is claimed to provide very good results but they are real results
based on real measurements.  Due to this, the measured results are
very different for the first time the program is executed and the
second time it is executed.  Any other factor on the machine would
impact perf results.

It seems that cachegrind produces absolutely consistent results which
do not depend on I/O, multi-core, or VM artifacts.

Bob
--
Bob Friesenhahn
[hidden email], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: SQLite 3.16.0 enters testing

Dominique Pellé
Bob Friesenhahn <[hidden email]> wrote:

> On Thu, 29 Dec 2016, Darko Volaric wrote:
>
>> What are you basing that theory on?
>
>
> Perf is claimed to provide very good results but they are real results based
> on real measurements.  Due to this, the measured results are very different
> for the first time the program is executed and the second time it is
> executed.  Any other factor on the machine would impact perf results.
>
> It seems that cachegrind produces absolutely consistent results which do not
> depend on I/O, multi-core, or VM artifacts.

You're right. Consistency can matter more where measuring
small improvements that add up.

I just tried "valgrind --tool=cachegrind ..." and "perf stat ..."
with the same command. Valgrind result was more indeed
more consistent across multiple runs.

Regarding speed of measurement, the same command
took 13.8 sec with cachegrind vs only 0.28 sec with "perf stat"
and 0.27 sec with neither cachegrind nor perf stat. So
perf stat has almost no overhead whereas cachegrind has a
big overhead, making it impractical when measuring slow
commands.

Dominique
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: SQLite 3.16.0 enters testing

Richard Hipp-3
In reply to this post by Bob Friesenhahn
On 12/29/16, Bob Friesenhahn <[hidden email]> wrote:
>
> Perf is claimed to provide very good results but they are real results
> based on real measurements.  Due to this, the measured results are
> very different for the first time the program is executed and the
> second time it is executed.

If this is true, then perf is entirely unsuitable for
microoptimization, since microoptimization depends on having
reproducible results.
--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: SQLite 3.16.0 enters testing

Eric Grange
In reply to this post by Dominique Pellé
> If this is true, then perf is entirely unsuitable for
> microoptimization, since microoptimization depends on having
> reproducible results.

Reproducibility can be misleading however if it is achieved by simulated
CPU cache and instruction pipelines, as this could lead to favoring the
wrong microoptimizations.

Perf stat variability (and other "real" measurements) can be minimized by
making multiple runs and taking the minimum, but also by keeping an eye on
the variance of realtime runs.

A high variance is often the sign that either the host machine is busy
(which can be dealt with seperately) or that the code is relying on
behaviors whose performance is "fragile", such as kernel context switches,
thread switches, mutexes, interrupts, I/O "coincidences" or even race
conditions.

For instance a spinlock may never exceed its spin count to enter sleep/wait
states in a simulated, deterministic environment, but could occasionnally
do in a more realistic setting, which could lead to a very different
performance profile when that happens, and ultimately favor different
optimization strategies ("slower but more robust").

Eric

On Fri, Dec 30, 2016 at 12:03 AM, Dominique Pellé <[hidden email]
> wrote:

> Bob Friesenhahn <[hidden email]> wrote:
>
> > On Thu, 29 Dec 2016, Darko Volaric wrote:
> >
> >> What are you basing that theory on?
> >
> >
> > Perf is claimed to provide very good results but they are real results
> based
> > on real measurements.  Due to this, the measured results are very
> different
> > for the first time the program is executed and the second time it is
> > executed.  Any other factor on the machine would impact perf results.
> >
> > It seems that cachegrind produces absolutely consistent results which do
> not
> > depend on I/O, multi-core, or VM artifacts.
>
> You're right. Consistency can matter more where measuring
> small improvements that add up.
>
> I just tried "valgrind --tool=cachegrind ..." and "perf stat ..."
> with the same command. Valgrind result was more indeed
> more consistent across multiple runs.
>
> Regarding speed of measurement, the same command
> took 13.8 sec with cachegrind vs only 0.28 sec with "perf stat"
> and 0.27 sec with neither cachegrind nor perf stat. So
> perf stat has almost no overhead whereas cachegrind has a
> big overhead, making it impractical when measuring slow
> commands.
>
> Dominique
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users