Article about pointer abuse in SQLite

classic Classic list List threaded Threaded
51 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Article about pointer abuse in SQLite

Simon Slavin-3
<http://blog.regehr.org/archives/1292>

"SQLite is a carefully engineered and thoroughly tested piece of software. Even so, it contains undefined behaviors because, until recently, no good checker for these behaviors existed. If anything is going to save us from UB hell, it’s tools combined with developers who care to listen to them. "

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Article about pointer abuse in SQLite

Keith Medcalf

There is no such thing as "undefined behaviour".  The machine code does exactly what it is told to do in exactly the manner in which it has been told to do it and obtains exactly the correct answer every time.

That the computation is "advanced beyond the realm of understanding of the observer" does not make the behaviour undefined.  It is perfectly defined, however, it is occasionally necessary to describe things as "undefined", oftentimes because it is too complicated to explain.  Just because someone says something as "undefined" does not mean that is so.  It is simply a euphemism for "I don't understand how it did that/what it is supposed to be doing (or, more often an appeal to self-proclaimed authority which said that such behaviour was undefined" without  having to admit fault, much in the same way that "supported" is a euphemism for "make money from".

Things will only be non-deterministic and perhaps undefined when run on Quantum Computers using Heisenberg registers for intermediate results.

> <http://blog.regehr.org/archives/1292>
 
> "SQLite is a carefully engineered and thoroughly tested piece of software.
> Even so, it contains undefined behaviors because, until recently, no good
> checker for these behaviors existed. If anything is going to save us from
> UB hell, it’s tools combined with developers who care to listen to them. "
 
> Simon.




_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Article about pointer abuse in SQLite

Marc L. Allen
I had a long response to this, but it occurs to me that you're just being pedantic for fun.  Am I wrong?

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Keith Medcalf
Sent: Friday, March 18, 2016 4:41 PM
To: SQLite mailing list <[hidden email]>
Subject: Re: [sqlite] Article about pointer abuse in SQLite


There is no such thing as "undefined behaviour".  The machine code does exactly what it is told to do in exactly the manner in which it has been told to do it and obtains exactly the correct answer every time.

That the computation is "advanced beyond the realm of understanding of the observer" does not make the behaviour undefined.  It is perfectly defined, however, it is occasionally necessary to describe things as "undefined", oftentimes because it is too complicated to explain.  Just because someone says something as "undefined" does not mean that is so.  It is simply a euphemism for "I don't understand how it did that/what it is supposed to be doing (or, more often an appeal to self-proclaimed authority which said that such behaviour was undefined" without  having to admit fault, much in the same way that "supported" is a euphemism for "make money from".

Things will only be non-deterministic and perhaps undefined when run on Quantum Computers using Heisenberg registers for intermediate results.

> <http://blog.regehr.org/archives/1292>
 
> "SQLite is a carefully engineered and thoroughly tested piece of software.
> Even so, it contains undefined behaviors because, until recently, no
> good checker for these behaviors existed. If anything is going to save
> us from UB hell, it’s tools combined with developers who care to listen to them. "
 
> Simon.




_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users



This email and any attachments are only for use by the intended recipient(s) and may contain legally privileged, confidential, proprietary or otherwise private information. Any unauthorized use, reproduction, dissemination, distribution or other disclosure of the contents of this e-mail or its attachments is strictly prohibited. If you have received this email in error, please notify the sender immediately and delete the original.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Article about pointer abuse in SQLite

Scott Hess
In reply to this post by Keith Medcalf
Not sure where you're going with this.  "Undefined behavior" in this case
is obviously referring to things defined by the C standard.  Things not
defined by the standard can (and do) change over time as compilers advance,
and also often differ between compilers from different vendors.

-scott


On Fri, Mar 18, 2016 at 1:40 PM, Keith Medcalf <[hidden email]> wrote:

>
> There is no such thing as "undefined behaviour".  The machine code does
> exactly what it is told to do in exactly the manner in which it has been
> told to do it and obtains exactly the correct answer every time.
>
> That the computation is "advanced beyond the realm of understanding of the
> observer" does not make the behaviour undefined.  It is perfectly defined,
> however, it is occasionally necessary to describe things as "undefined",
> oftentimes because it is too complicated to explain.  Just because someone
> says something as "undefined" does not mean that is so.  It is simply a
> euphemism for "I don't understand how it did that/what it is supposed to be
> doing (or, more often an appeal to self-proclaimed authority which said
> that such behaviour was undefined" without  having to admit fault, much in
> the same way that "supported" is a euphemism for "make money from".
>
> Things will only be non-deterministic and perhaps undefined when run on
> Quantum Computers using Heisenberg registers for intermediate results.
>
> > <http://blog.regehr.org/archives/1292>
>
> > "SQLite is a carefully engineered and thoroughly tested piece of
> software.
> > Even so, it contains undefined behaviors because, until recently, no good
> > checker for these behaviors existed. If anything is going to save us from
> > UB hell, it’s tools combined with developers who care to listen to them.
> "
>
> > Simon.
>
>
>
>
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Article about pointer abuse in SQLite

Igor Tandetnik-2
In reply to this post by Keith Medcalf
On 3/18/2016 4:40 PM, Keith Medcalf wrote:
> There is no such thing as "undefined behaviour".  The machine code does exactly what it is told to do

But SQLite is not written in machine code. It is (largely) written in C.
And C language most certainly has the concept of undefined behavior -
roughly, a language construct or situation for which a C compiler is not
obliged to produce any particular machine code; or in other words, can
produce machine code that does anything at all. See also:

https://en.wikipedia.org/wiki/Undefined_behavior
http://catb.org/jargon/html/N/nasal-demons.html

> Things will only be non-deterministic and perhaps undefined when run on Quantum Computers using Heisenberg registers for intermediate results.

There's plenty of non-deterministic behavior on regular von Neumann
machines with more than one core. See e.g.
https://en.wikipedia.org/wiki/Race_condition . In any case, "undefined
behavior" is not at all the same thing as "non-deterministic behavior".
A conforming C (or C++) program does not (by definition of "conforming")
exhibit undefined behavior, but may very well be non-deterministic.
--
Igor Tandetnik

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Article about pointer abuse in SQLite

Richard Hipp-3
On 3/18/16, Igor Tandetnik <[hidden email]> wrote:
> On 3/18/2016 4:40 PM, Keith Medcalf wrote:
>> There is no such thing as "undefined behaviour".  The machine code does
>> exactly what it is told to do
>
> But SQLite is not written in machine code. It is (largely) written in C.

SQLite is written in C, but the focus of testing is the resulting
machine code.  So, there is a point of view that says that it doesn't
really matter if the C code is "undefined" or not.  What matters is
whether or not the resulting machine code computes the correct answer.
That's what we test: does the machine code compute the correct answer.

Our testing standard is to cause every machine-code branch instruction
to take both decisions (jump and fall through) at least once, in an
as-deployed configuration.  That means every instruction in the
machine code is tested and testing occurs with no special compile-time
options or flags.  The code that gets tested is exactly byte-for-byte
the same machine code that gets deployed in the field.  This is what
they mean by "fly what you test and test what you fly".

That said, we also do source-code validation on SQLite.  We work to
make SQLite compile without warnings, we run it through static
analyzers, and I address any "undefined behaviors" in the source code
that John Regehr or others report.  SQLite contains thousands of
assert() statements which are really another form of source-code
validation (as the asserts do not appear in deployment builds.)  Just
the other day, I wrote a custom static code analyzer for SQLite that
runs on every "make" looking for a particular kind of programming
error (see https://www.sqlite.org/src/artifact/4f65e1a6748e42f2).  We
also compile and test SQLite on as many different compilers as we can,
with all kinds of different compiler settings, and verify that they
all get the same answer, which is yet another kind of source-code
validation.

All of this source-code validation is well and good, and I recommend
it.  But the real focus of testing is SQLite is the machine code.  And
so to a first approximation, Keith is correct: "undefined behavior" at
the source code level does not matter.

You cannot say that a program is correct by only looking at source
code.  You have to validate the machine code.  Compilers make
mistakes.  SQLite has, at various times, hit bugs in each of GCC,
Clang, and MSVC, all of which have now been fixed.  Do not trust your
compiler.

One point of view is that compilers that attempt to take advantage of
obscure "undefined behavior" to boost performance, but instead break
programs, are in fact buggy.  (Compiler writers tend to disagree with
this viewpoint.  I'm not saying it is the correct point of view, just
a point of view that is widely held.)  So watching out for obscure
"undefined behavior" that breaks your program is really the same thing
as watching out for bugs in your compiler.  Both of these happen, and
in my experience, with about the same frequency.

--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Article about pointer abuse in SQLite

Scott Robison-2
On Fri, Mar 18, 2016 at 4:03 PM, Richard Hipp <[hidden email]> wrote:

> On 3/18/16, Igor Tandetnik <[hidden email]> wrote:
> > On 3/18/2016 4:40 PM, Keith Medcalf wrote:
> >> There is no such thing as "undefined behaviour".  The machine code does
> >> exactly what it is told to do
> >
> > But SQLite is not written in machine code. It is (largely) written in C.
>
> SQLite is written in C, but the focus of testing is the resulting
> machine code.  So, there is a point of view that says that it doesn't
> really matter if the C code is "undefined" or not.  What matters is
> whether or not the resulting machine code computes the correct answer.
> That's what we test: does the machine code compute the correct answer.
>
> That said, we also do source-code validation on SQLite.  We work to
> make SQLite compile without warnings, we run it through static
> analyzers, and I address any "undefined behaviors" in the source code
> that John Regehr or others report.  SQLite contains thousands of
>

I'd rather have code that might use some "undefined behavior" and generates
the right answer than code that always conformed to defined behavior yet
was logically flawed. Mind you, I don't often have to worry about my code
being compiled in a huge number of environments.

Also, it seems a lot of people today are critical of programs that utilize
defined behavior in the context of the standard to which they were written,
but not to a later standard (such as ANSI C / C89 / C90 vs C99 or C11 or
whatever). It's great to conform to both when reasonable, but it may not
always be reasonable (as I can't necessarily conceive of every possibility).

--
Scott Robison
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Article about pointer abuse in SQLite

Simon Slavin-3

On 18 Mar 2016, at 10:33pm, Scott Robison <[hidden email]> wrote:

> I'd rather have code that might use some "undefined behavior" and generates
> the right answer than code that always conformed to defined behavior yet
> was logically flawed. Mind you, I don't often have to worry about my code
> being compiled in a huge number of environments.

That's the problem.

The development team don't know what compilers their code is going to be compiled on.  They can argue that a compiler which violates /documented/ behaviour is broken.  But they can't argue that a compiler which correctly compiles C code is broken.  There is, by definition, nothing wrong with such a compiler and if it causes your code to do The Wrong Thing then your code is broken.

However nobody has come up with such a compiler.  Someone could write one, I suppose[1].  Then they'd compile SQLite with it.  Then they'd find a test case which caused SQLite to do The Wrong Thing.  Then they'd submit a bug report.  Until that has happened, nobody has proved there's anything wrong with SQLite.

Frankly the fact that I can take a cross-compiler for an embedded processor in ... say, a smart thermostat ... compile the amalgamation version on it and be sure it's going to work is very pleasing.

Simon.

[1] Or find an open source compiler and make some simple modifications to it.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Article about pointer abuse in SQLite

jose isaias cabrera-2
In reply to this post by Scott Robison-2

Scott Robison wrote...

> Also, it seems a lot of people today are critical of programs that utilize
> defined behavior in the context of the standard to which they were
> written,
> but not to a later standard (such as ANSI C / C89 / C90 vs C99 or C11 or
> whatever). It's great to conform to both when reasonable, but it may not
> always be reasonable (as I can't necessarily conceive of every
> possibility).

Good thought, Scott. I have come to the conclusion that
"political-correctness" has even invaded computer science.  Ever since
classes started to be pushed by teachers, all of these "ideal methods" have
taken more power than they should have.  It's not that they are bad, but,
they are just other methods for some folks to be able to choose. I have so
much to say, but so little time.  Thank Simon for bringing this theme.

josé

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Article about pointer abuse in SQLite

James K. Lowden
In reply to this post by Scott Robison-2
On Fri, 18 Mar 2016 16:33:56 -0600
Scott Robison <[hidden email]> wrote:

> I'd rather have code that might use some "undefined behavior" and
> generates the right answer than code that always conformed to defined
> behavior yet was logically flawed.

Code that falls under undefined behavior *is* logically flawed, by
definition.  Whether or not it works, it's not specified to.  The
compiler may have generated perfectly correct machine code, but another
compiler or some future version of your present compiler may not.  

You might share my beef with the compiler writers, though: lots things
that are left undefined shouldn't be.  Because hardware architecture
varies, some practices that do work and have worked and are expected to
work on a wide variety of machines are UB.  A recent thread on using
void* for a function pointer is an example: dlsym(2) returns a function
pointer defined as void*, but the C standard says void* can only refer
to data, not functions!  

Machines exist for which the size of a function pointer is not
sizeof(void*).  Source code that assumes they are the same size is not
portable to those architectures.  Fine.  But a particular compiler
generates code for a particular architecture.  On x86 hardware, all
pointers have always been and will always be the same size.  All
Linux/Posix code relies on that, too, along with a host of other
assumptions. If that ever changed, a boat load of code would have to be
changed.  Why does the compiler writer feel it's in his interest or
mine to warn me about that not-happening eventuality?  For the machine
I'm compilng for, the code is *not* in error.  For some future machine,
maybe it will be; let's leave that until then.  

I was looking at John Regehr's blog the other day.  I think it was
there that I learned that the practice of dropping UB code on the floor
has been going on longer than I'd realized; it's just that gcc has been
more aggressive in recent years.  I think it was there I saw this
construction:

        if( p < p + n)
                error

where p is a pointer.  On lots of architectures, for large n, p + n can
be negative.  The test works.  Or did.  The C standard says that's
UB, though. It doesn't promise the pointer will go negative.  It doesn't
promise it won't.  It doesn't promise not to tell your mother about
it.  And, in one recent version, it doesn't compile it.  Warning?  No.
Error? No.  Machine code?  No!  It's UB, so no code is generated (ergo,
no error handling)!  Even though the hardware instructions that would
be -- that used to be -- generated work as implied by the code.    

Postel's Law is to be liberal in what you accept and conservative in
what you emit.  The compilers have been practicing the opposite,
thwarting common longstanding practice just because they "can".  

Dan Bernstein is calling for a new C compiler that is 100%
deterministic: no UB.  All UB per the standard would be defined by the
compiler.  And maybe a few goodies, like zero-initialized automatic
(stack) variables.  

Such a compiler would enjoy great popularity, even if it imposed, say,
a 5% performance penalty, because C programmers would have greater
confidence in their code working as expected. They'd have some
assurance that the compiler wouldn't cut them off at the knees in its
next release.  As he says, there's not real choice between fast and
correct  If the "always defined befavior" compiler got off the ground,
may it would finally drive gcc & friends in the direction of working
with their users for a change.  Or make them irrelevant.  

--jkl


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Article about pointer abuse in SQLite

Scott Robison-2
On Mar 18, 2016 11:12 PM, "James K. Lowden" <[hidden email]>
wrote:

>
> On Fri, 18 Mar 2016 16:33:56 -0600
> Scott Robison <[hidden email]> wrote:
>
> > I'd rather have code that might use some "undefined behavior" and
> > generates the right answer than code that always conformed to defined
> > behavior yet was logically flawed.
>
> Code that falls under undefined behavior *is* logically flawed, by
> definition.  Whether or not it works, it's not specified to.  The
> compiler may have generated perfectly correct machine code, but another
> compiler or some future version of your present compiler may not.

Perhaps I should have said "undefined behavior as per the standard". Code
that does what is intended for the intended target environment utilizing a
specific tool chain is not logically flawed just because the standard calls
a construct "undefined behavior".

> You might share my beef with the compiler writers, though: lots things
> that are left undefined shouldn't be.

Not just compiler writers but standards organizations. Lots of overlap to
be sure, but not 100%.

Because hardware architecture
> varies, some practices that do work and have worked and are expected to
> work on a wide variety of machines are UB.  A recent thread on using
> void* for a function pointer is an example: dlsym(2) returns a function
> pointer defined as void*, but the C standard says void* can only refer
> to data, not functions!

So I guess casting between function pointers might be arguably safer if the
void* form of a function was called void(*)(void) though even then there
can be multiple forms of function pointers (like near vs far pointers in
x86 real mode).

>
> Machines exist for which the size of a function pointer is not
> sizeof(void*).  Source code that assumes they are the same size is not
> portable to those architectures.  Fine.  But a particular compiler
> generates code for a particular architecture.  On x86 hardware, all
> pointers have always been and will always be the same size.  All
> Linux/Posix code relies on that, too, along with a host of other
> assumptions. If that ever changed, a boat load of code would have to be
> changed.  Why does the compiler writer feel it's in his interest or
> mine to warn me about that not-happening eventuality?  For the machine
> I'm compilng for, the code is *not* in error.  For some future machine,
> maybe it will be; let's leave that until then.
>
> I was looking at John Regehr's blog the other day.  I think it was
> there that I learned that the practice of dropping UB code on the floor
> has been going on longer than I'd realized; it's just that gcc has been
> more aggressive in recent years.  I think it was there I saw this
> construction:
>
>         if( p < p + n)
>                 error
>
> where p is a pointer.  On lots of architectures, for large n, p + n can
> be negative.  The test works.  Or did.  The C standard says that's
> UB, though. It doesn't promise the pointer will go negative.  It doesn't
> promise it won't.  It doesn't promise not to tell your mother about
> it.  And, in one recent version, it doesn't compile it.  Warning?  No.
> Error? No.  Machine code?  No!  It's UB, so no code is generated (ergo,
> no error handling)!  Even though the hardware instructions that would
> be -- that used to be -- generated work as implied by the code.

Yes, these are the sorts of things that are frustrating.

> Postel's Law is to be liberal in what you accept and conservative in
> what you emit.  The compilers have been practicing the opposite,
> thwarting common longstanding practice just because they "can".
>
> Dan Bernstein is calling for a new C compiler that is 100%
> deterministic: no UB.  All UB per the standard would be defined by the
> compiler.  And maybe a few goodies, like zero-initialized automatic
> (stack) variables.

Neat idea, though undefined behavior isn't always bad. Interesting article
at http://blog.llvm.org/2011/05/what-every-c-programmer-should-know.html
that goes into some considerations.

>
> Such a compiler would enjoy great popularity, even if it imposed, say,
> a 5% performance penalty, because C programmers would have greater
> confidence in their code working as expected. They'd have some
> assurance that the compiler wouldn't cut them off at the knees in its
> next release.  As he says, there's not real choice between fast and
> correct

Except that testing can verify something is correct for a given environment.

  If the "always defined befavior" compiler got off the ground,
> may it would finally drive gcc & friends in the direction of working
> with their users for a change.  Or make them irrelevant.

I think they'd continue to be popular with people looking to eek out as
much performance as possible.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Article about pointer abuse in SQLite

James K. Lowden
On Sat, 19 Mar 2016 02:04:35 -0600
Scott Robison <[hidden email]> wrote:

> As he says, there's not real choice between fast and
> > correct
>
> Except that testing can verify something is correct for a given
> environment.  

That's actually not true, on a couple of levels.  

        "[T]esting can be used very effectively to show the presence of
bugs but never to show their absence."
        -- EWD303

I think that should be called Dijkstra's Dictum.  It's not just quip;
it's a concise insight into limits of testing versus proving
correctness.  

Second, you can't test the future.  If the correctness of the code is
subject to change by the compiler's interpretation of the language, how
is the programmer to prevent it?  

> > finally drive gcc & friends in the direction of working
> > with their users for a change.  Or make them irrelevant.
>
> I think they'd continue to be popular with people looking to eek out
> as much performance as possible.

You may be right.  As a consultant I've often felt I was hired to
sprinkle magic pixie performance dust on the system.  People want to
believe that performance is found in tools.  How come there's no -O5?  

In truth, every performance problem I've encountered was a design
problem, often obvious, always unnecessary.  "Use a better compiler"
has never been the solution.  Unloading mounds of unnecessary
processing with a pitchfork is.  

Doubtless there are some well tested, highly stable applications run at
scale, for which 5% is a measurable and meaningful gain.  IMO they're
actually the ones driving UB treatment by compiler writers.  The
other 99% stand to gain from a compiler that emphasizes correctness and
predictable behavior.  

--jkl

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Article about pointer abuse in SQLite

Richard Hipp-3
On 3/19/16, James K. Lowden <[hidden email]> wrote:
>
> Second, you can't test the future.  If the correctness of the code is
> subject to change by the compiler's interpretation of the language, how
> is the programmer to prevent it?
>

Indeed.  Every bit of the code examined by Prof. Regehr was
well-defined according to K&R.  But subsequent revisions of the
C-language standards changed that.  How does one write code that will
comply with language standards that keep changing out from under you?
--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Article about pointer abuse in SQLite

Scott Robison-2
In reply to this post by James K. Lowden
On Mar 19, 2016 1:19 PM, "James K. Lowden" <[hidden email]> wrote:

>
> On Sat, 19 Mar 2016 02:04:35 -0600
> Scott Robison <[hidden email]> wrote:
>
> > As he says, there's not real choice between fast and
> > > correct
> >
> > Except that testing can verify something is correct for a given
> > environment.
>
> That's actually not true, on a couple of levels.

True, and that was a sloppy phrasing on my part. Please replace "correct"
with "sufficiently correct for the use case". If you are writing software
for one environment you are in a very different situation than if you are
trying to write truly portable software intended to be built and used in
any arbitrary environment.

> Second, you can't test the future.  If the correctness of the code is
> subject to change by the compiler's interpretation of the language, how
> is the programmer to prevent it?

In addition to DRH's comment about the moving target set by evolving
standards: even if there was nothing undefined by the standard you can't
trust a new release of a compiler or library to have not introduced some
bug that impacts you. Hence the need to test the heck out of any intended
release. It's not proof of correctness but it is evidence of a likely
sufficiently correct program.

>
> > > finally drive gcc & friends in the direction of working
> > > with their users for a change.  Or make them irrelevant.
> >
> > I think they'd continue to be popular with people looking to eek out
> > as much performance as possible.
>
> You may be right.  As a consultant I've often felt I was hired to
> sprinkle magic pixie performance dust on the system.  People want to
> believe that performance is found in tools.  How come there's no -O5?
>
> In truth, every performance problem I've encountered was a design
> problem, often obvious, always unnecessary.  "Use a better compiler"
> has never been the solution.  Unloading mounds of unnecessary
> processing with a pitchfork is.

I guess it depends on what you're doing. Some applications (I'm thinking
games at the moment) need both quality design and quality code generation.
When you know your application is being targeted for a platform you'll
utilize undefined behavior if it eeks out a little bit more performance. If
necessary you'll hide UB in conditional blocks so that you get the best out
of different platforms.

I agree that most applications do not need anything like this, and I think
it is a good idea to steer clear of UB, but it is not universally true.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Article about pointer abuse in SQLite

KlaasV
In reply to this post by Simon Slavin-3
>> On 3/19/16, James K. Lowden <[hidden email]> wrote:

>> ... If the correctness of the code is
>> subject to change by the compiler's interpretation of the language, how
>> is the programmer to prevent it?

> On Sat, 19 Mar 2016 15:50:43 -0400 Richard Hipp <[hidden email]> wrote:

> ... But subsequent revisions of the
> C-language standards changed that.  How does one write code that will
> comply with language standards that keep changing out from under you?

It's like trying to live according to the law while they're changing the constitution.


Kind regards/Vriendelijke groeten.
Klaas `Z4us` Van B., CEO/CIO LI#437429414
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Klaas "Z4us" V, MetaDBA at InnocentIsArt.EU
Reply | Threaded
Open this post in threaded view
|

Re: Article about pointer abuse in SQLite

Scott Perry
On Mar 21, 2016, at 3:17 AM, Klaas Van B. <[hidden email]> wrote:

>
>>> On 3/19/16, James K. Lowden <[hidden email]> wrote:
>
>>> ... If the correctness of the code is
>>> subject to change by the compiler's interpretation of the language, how
>>> is the programmer to prevent it?
>
>> On Sat, 19 Mar 2016 15:50:43 -0400 Richard Hipp <[hidden email]> wrote:
>
>> ... But subsequent revisions of the
>> C-language standards changed that.  How does one write code that will
>> comply with language standards that keep changing out from under you?
>
> It's like trying to live according to the law while they're changing the constitution.

This is a false dichotomy. Compilers allow you to choose your standard; --std=c11 means something very specific (and unchanging) about the behaviour you can expect to be defined.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Article about pointer abuse in SQLite

Scott Perry
On Mar 21, 2016, at 1:48 PM, Scott Perry <[hidden email]> wrote:

>
> On Mar 21, 2016, at 3:17 AM, Klaas Van B. <[hidden email]> wrote:
>>
>>>> On 3/19/16, James K. Lowden <[hidden email]> wrote:
>>
>>>> ... If the correctness of the code is
>>>> subject to change by the compiler's interpretation of the language, how
>>>> is the programmer to prevent it?
>>
>>> On Sat, 19 Mar 2016 15:50:43 -0400 Richard Hipp <[hidden email]> wrote:
>>
>>> ... But subsequent revisions of the
>>> C-language standards changed that.  How does one write code that will
>>> comply with language standards that keep changing out from under you?
>>
>> It's like trying to live according to the law while they're changing the constitution.
>
> This is a false dichotomy. Compilers allow you to choose your standard; --std=c11 means something very specific (and unchanging) about the behaviour you can expect to be defined.

(and yes, I did mean to say false equivalence in my response)
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Article about pointer abuse in SQLite

Scott Robison-2
In reply to this post by Scott Perry
On Mar 21, 2016 2:48 PM, "Scott Perry" <[hidden email]> wrote:
>
> On Mar 21, 2016, at 3:17 AM, Klaas Van B. <[hidden email]> wrote:
> >
> >>> On 3/19/16, James K. Lowden <[hidden email]> wrote:
> >
> >>> ... If the correctness of the code is
> >>> subject to change by the compiler's interpretation of the language,
how
> >>> is the programmer to prevent it?
> >
> >> On Sat, 19 Mar 2016 15:50:43 -0400 Richard Hipp <[hidden email]> wrote:
> >
> >> ... But subsequent revisions of the
> >> C-language standards changed that.  How does one write code that will
> >> comply with language standards that keep changing out from under you?
> >
> > It's like trying to live according to the law while they're changing
the constitution.
>
> This is a false dichotomy. Compilers allow you to choose your standard;
--std=c11 means something very specific (and unchanging) about the
behaviour you can expect to be defined.

Unfortunately, gcc library header files warn about perfectly valid code in
C89 mode due to changes in C99. It's not a big deal until people take
compiler warnings as prima facie evidence that code is broken.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Article about pointer abuse in SQLite

James K. Lowden
In reply to this post by Scott Perry
On Mon, 21 Mar 2016 13:48:06 -0700
Scott Perry <[hidden email]> wrote:

> Compilers allow you to choose your standard; --std=c11 means
> something very specific (and unchanging)

They do.  And that covers what the standard covers.  The standard also
has limits.  It includes constructs that are syntactically permitted
but whose behavior is left undefined, known by the scarred as "UB" for
"undefined behavior". An example from Clang's discussion is

        int i = 10 << 31;

The standard says << is a shift operator.  It places no limit on the
number of bits to be shifted.  If that number is so large that the
product cannot be represented by the assigned variable, that is *not*
an error.  The standard allows the compiler to do anything or nothing
with it.  As you may imagine, the varieties of anything and nothing are
many.  

Compiler writers are well aware that "nothing" is faster done than
"something".  Over time, they have gotten more aggressive in simply
deleting UB code.  As a consequence, programmers who thought they wrote
standards-conforming code get burned when they upgrade/change
compilers.  Mysterious and sometimes subtle errors are introduced by
the compiler for the user's benefit.  

Your googlefu will turn up lots of discussion.  One I liked that wasn't
on Page 1:

        http://blog.frama-c.com/index.php?post/2013/10/09/Overflow-float-integer

--jkl
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Article about pointer abuse in SQLite

J Decker
So far I just see analysis tools fail for the same sorts of valid code...

this is a bit of C# but the same idea causes the same warnings and
there's nothign tecniclally wrong with this.



class test
{
   struct large_struct { public int x; }
   bool arbitrary_true_false = true;
   void method()
   {
      bool initialized = false;
      large_struct s;
      if( arbitrary_true_false )
      {
         initialized = true;
         s.x = 1;
      }
      if( initialized )
      {
         Console.WriteLine( "this fails(during compile) as
uninitialized: {0}", s.x );
      }
   }
}

On Mon, Mar 21, 2016 at 4:35 PM, James K. Lowden
<[hidden email]> wrote:

> On Mon, 21 Mar 2016 13:48:06 -0700
> Scott Perry <[hidden email]> wrote:
>
>> Compilers allow you to choose your standard; --std=c11 means
>> something very specific (and unchanging)
>
> They do.  And that covers what the standard covers.  The standard also
> has limits.  It includes constructs that are syntactically permitted
> but whose behavior is left undefined, known by the scarred as "UB" for
> "undefined behavior". An example from Clang's discussion is
>
>         int i = 10 << 31;
>
> The standard says << is a shift operator.  It places no limit on the
> number of bits to be shifted.  If that number is so large that the
> product cannot be represented by the assigned variable, that is *not*
> an error.  The standard allows the compiler to do anything or nothing
> with it.  As you may imagine, the varieties of anything and nothing are
> many.
>
> Compiler writers are well aware that "nothing" is faster done than
> "something".  Over time, they have gotten more aggressive in simply
> deleting UB code.  As a consequence, programmers who thought they wrote
> standards-conforming code get burned when they upgrade/change
> compilers.  Mysterious and sometimes subtle errors are introduced by
> the compiler for the user's benefit.
>
> Your googlefu will turn up lots of discussion.  One I liked that wasn't
> on Page 1:
>
>         http://blog.frama-c.com/index.php?post/2013/10/09/Overflow-float-integer
>
> --jkl
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
123