Fill empty space with random

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Fill empty space with random

J Decker
Is there maybe a compile option for sqlite to fill empty space in a db with
random data rather than 0 ?
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Fill empty space with random

Bob Gailer
On Oct 14, 2018 7:57 AM, "J Decker" <[hidden email]> wrote:
>
> Is there maybe a compile option for sqlite to fill empty space in a db
with
> random data rather than 0 ?

I don't know about compile options.

You could do a 1 time update to set all 0 columns to random(1,99), and
create a trigger to do the same on insert.

This, of course is on a table-by-table basis.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Fill empty space with random

J Decker
On Sun, Oct 14, 2018 at 5:48 AM Bob Gailer <[hidden email]> wrote:

> On Oct 14, 2018 7:57 AM, "J Decker" <[hidden email]> wrote:
> >
> > Is there maybe a compile option for sqlite to fill empty space in a db
> with
> > random data rather than 0 ?
>
> I don't know about compile options.
>
> You could do a 1 time update to set all 0 columns to random(1,99), and
> create a trigger to do the same on insert.
>
> This, of course is on a table-by-table basis.
>
Doesn't work for all the empty space outside of the table; between tables;
not unused pages from deleted records, ....

> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Fill empty space with random

Simon Slavin-3
In reply to this post by J Decker
On 14 Oct 2018, at 12:56pm, J Decker <[hidden email]> wrote:

> Is there maybe a compile option for sqlite to fill empty space in a db with random data rather than 0 ?

There is not.  But

(A) It may be an easy change to the source code
(B) Your operating system may have a setting to do this automatically to freed blocks on a storage device.
(C) Your device driver may have a setting to do this automatically to freed blocks on the device.

That type of security is normally done at OS or device level, not by each individual app.

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Fill empty space with random

Luuk
On 14-10-2018 16:17, Simon Slavin wrote:

> On 14 Oct 2018, at 12:56pm, J Decker <[hidden email]> wrote:
>
>> Is there maybe a compile option for sqlite to fill empty space in a db with random data rather than 0 ?
> There is not.  But
>
> (A) It may be an easy change to the source code
> (B) Your operating system may have a setting to do this automatically to freed blocks on a storage device.
> (C) Your device driver may have a setting to do this automatically to freed blocks on the device.
>
> That type of security is normally done at OS or device level, not by each individual app.
>
> Simon.
>
Can you give any hints on why it would be a security issue to fill
'empty space' with 0, and why 'random data' should be used?

?


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Fill empty space with random

J Decker
In reply to this post by Simon Slavin-3
On Sun, Oct 14, 2018 at 7:17 AM Simon Slavin <[hidden email]> wrote:

> On 14 Oct 2018, at 12:56pm, J Decker <[hidden email]> wrote:
>
> > Is there maybe a compile option for sqlite to fill empty space in a db
> with random data rather than 0 ?
>
> There is not.  But
>
> (A) It may be an easy change to the source code
>
Sure; figured I'd toss out the question to see if there was at least a
springboard


> (B) Your operating system may have a setting to do this automatically to
> freed blocks on a storage device.
> (C) Your device driver may have a setting to do this automatically to
> freed blocks on the device.
>
> B and C would apply if there was a vacuum also; adding data, and
deleteting data, the db ends up with lots of zeros....
Also between non-integral pages; messages that are say 700 bytes; so 4096%
700 is 596; which is all filled with zeros...
if I end up with messages that are say 2200 bytes; nearly half the page is
blank for a while.


> That type of security is normally done at OS or device level, not by each
> individual app.
>
> Simon.
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Fill empty space with random

J Decker
In reply to this post by Luuk
On Sun, Oct 14, 2018 at 7:24 AM Luuk <[hidden email]> wrote:

> On 14-10-2018 16:17, Simon Slavin wrote:
> > On 14 Oct 2018, at 12:56pm, J Decker <[hidden email]> wrote:
> >
> >> Is there maybe a compile option for sqlite to fill empty space in a db
> with random data rather than 0 ?
> > There is not.  But
> >
> > (A) It may be an easy change to the source code
> > (B) Your operating system may have a setting to do this automatically to
> freed blocks on a storage device.
> > (C) Your device driver may have a setting to do this automatically to
> freed blocks on the device.
> >
> > That type of security is normally done at OS or device level, not by
> each individual app.
> >
> > Simon.
> >
> Can you give any hints on why it would be a security issue to fill
> 'empty space' with 0, and why 'random data' should be used?
>
> ?
>
I hesitate to describe the real scenario; and want to instead manufacture
one; but in either case I feel there will be more comments about the
underlaying system than on Sqlite itself.

In the simple case, the VFS that the sqlite Db is mounted in is encrypted
with a long key.  The key has cycles at 4096(A) and 16(B1-Bn) bytes
(4096/16 = 256 cycles of Bn); such that each sector is masked with
A^B1(256x), A^B2(256x), ... all together there is no repetition because the
change from Bn to B(n+1) at the 4096 boundary makes the stream overall
appear continuously random.
Only data that is written is actually masked...

Sqlite likes to write 0's in large splotches (in my usage); which leaks key
information; (only slightly more than the data stored in tables typically,
which is a lot of the same bytes (0, 1 for instance and A-Z, a-z less-so;
but all of that has upper bit(s) that are 0... )

And even is a specific sector (or several) is 'cracked' it doesn't do any
good for any other page... but if LOTS of pages are found, it becomes
easier to find what the overall A key is, which makes finding sector keys
that you only need a few 32-64 bytes of 0's to reveal the sector specific
key (for later use?)

The keys are a procedurally generated with a PRNG sha2 bit streams based;
so 512 bits (16 bytes) at a time; and sha algorithms generates VERY good PR
numbers. which can be consumed as end-to-end bit streams.

I might look into it; there are certainly a great test suite available to
reveal issues; but I expect Sqlite 'expects' memory to be 0 initialized
(even when filled from disk) and that it will be a HUGE can of worms.


>
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Fill empty space with random

Simon Slavin-3
In reply to this post by J Decker
On 14 Oct 2018, at 3:24pm, J Decker <[hidden email]> wrote:

> B and C would apply if there was a vacuum also; adding data, and deleteting data, the db ends up with lots of zeros....
> Also between non-integral pages; messages that are say 700 bytes; so 4096% 700 is 596; which is all filled with zeros...

True.  And note the non-intuitive usage patterns of solid-state storage devices, where changing one byte of a 'sector' can result in the old 'sector' being freed and a new one used.  Changes to SQLite (or anything else at app level) cannot help with this sort of thing.

It comes down to why you're concerned about security, and whether you're concerned that someone may physically steal your hardware, or whether you're just sharing a virtual machine host with a possibly-hostile organisation.

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Fill empty space with random

J Decker
In reply to this post by J Decker
(sorry for the math err s/16/32/g and s/512/256/   - I double and halfed
the wrong directions.)

On Sun, Oct 14, 2018 at 7:57 AM J Decker <[hidden email]> wrote:

>
>
> On Sun, Oct 14, 2018 at 7:24 AM Luuk <[hidden email]> wrote:
>
>> On 14-10-2018 16:17, Simon Slavin wrote:
>> > On 14 Oct 2018, at 12:56pm, J Decker <[hidden email]> wrote:
>> >
>> >> Is there maybe a compile option for sqlite to fill empty space in a db
>> with random data rather than 0 ?
>> > There is not.  But
>> >
>> > (A) It may be an easy change to the source code
>> > (B) Your operating system may have a setting to do this automatically
>> to freed blocks on a storage device.
>> > (C) Your device driver may have a setting to do this automatically to
>> freed blocks on the device.
>> >
>> > That type of security is normally done at OS or device level, not by
>> each individual app.
>> >
>> > Simon.
>> >
>> Can you give any hints on why it would be a security issue to fill
>> 'empty space' with 0, and why 'random data' should be used?
>>
>> ?
>>
> I hesitate to describe the real scenario; and want to instead manufacture
> one; but in either case I feel there will be more comments about the
> underlaying system than on Sqlite itself.
>
> In the simple case, the VFS that the sqlite Db is mounted in is encrypted
> with a long key.  The key has cycles at 4096(A) and 16(B1-Bn) bytes
> (4096/16 = 256 cycles of Bn); such that each sector is masked with
> A^B1(256x), A^B2(256x), ... all together there is no repetition because the
> change from Bn to B(n+1) at the 4096 boundary makes the stream overall
> appear continuously random.
> Only data that is written is actually masked...
>
> Sqlite likes to write 0's in large splotches (in my usage); which leaks
> key information; (only slightly more than the data stored in tables
> typically, which is a lot of the same bytes (0, 1 for instance and A-Z, a-z
> less-so; but all of that has upper bit(s) that are 0... )
>
> And even is a specific sector (or several) is 'cracked' it doesn't do any
> good for any other page... but if LOTS of pages are found, it becomes
> easier to find what the overall A key is, which makes finding sector keys
> that you only need a few 32-64 bytes of 0's to reveal the sector specific
> key (for later use?)
>
> The keys are a procedurally generated with a PRNG sha2 bit streams based;
> so 512 bits (16 bytes) at a time; and sha algorithms generates VERY good PR
> numbers. which can be consumed as end-to-end bit streams.
>
> I might look into it; there are certainly a great test suite available to
> reveal issues; but I expect Sqlite 'expects' memory to be 0 initialized
> (even when filled from disk) and that it will be a HUGE can of worms.
>
>
>>
>> _______________________________________________
>> sqlite-users mailing list
>> [hidden email]
>> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>>
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Fill empty space with random

Luuk
On 14-10-2018 17:07, J Decker wrote:

> (sorry for the math err s/16/32/g and s/512/256/   - I double and halfed
> the wrong directions.)
>
> On Sun, Oct 14, 2018 at 7:57 AM J Decker <[hidden email]> wrote:
>
>>
>> On Sun, Oct 14, 2018 at 7:24 AM Luuk <[hidden email]> wrote:
>>
>>> On 14-10-2018 16:17, Simon Slavin wrote:
>>>> On 14 Oct 2018, at 12:56pm, J Decker <[hidden email]> wrote:
>>>>
>>>>> Is there maybe a compile option for sqlite to fill empty space in a db
>>> with random data rather than 0 ?
>>>> There is not.  But
>>>>
>>>> (A) It may be an easy change to the source code
>>>> (B) Your operating system may have a setting to do this automatically
>>> to freed blocks on a storage device.
>>>> (C) Your device driver may have a setting to do this automatically to
>>> freed blocks on the device.
>>>> That type of security is normally done at OS or device level, not by
>>> each individual app.
>>>> Simon.
>>>>
>>> Can you give any hints on why it would be a security issue to fill
>>> 'empty space' with 0, and why 'random data' should be used?
>>>
>>> ?
>>>
>> I hesitate to describe the real scenario; and want to instead manufacture
>> one; but in either case I feel there will be more comments about the
>> underlaying system than on Sqlite itself.
>>
>> In the simple case, the VFS that the sqlite Db is mounted in is encrypted
>> with a long key.  The key has cycles at 4096(A) and 16(B1-Bn) bytes
>> (4096/16 = 256 cycles of Bn); such that each sector is masked with
>> A^B1(256x), A^B2(256x), ... all together there is no repetition because the
>> change from Bn to B(n+1) at the 4096 boundary makes the stream overall
>> appear continuously random.
>> Only data that is written is actually masked...
>>
>> Sqlite likes to write 0's in large splotches (in my usage); which leaks
>> key information; (only slightly more than the data stored in tables
>> typically, which is a lot of the same bytes (0, 1 for instance and A-Z, a-z
>> less-so; but all of that has upper bit(s) that are 0... )
>>
>> And even is a specific sector (or several) is 'cracked' it doesn't do any
>> good for any other page... but if LOTS of pages are found, it becomes
>> easier to find what the overall A key is, which makes finding sector keys
>> that you only need a few 32-64 bytes of 0's to reveal the sector specific
>> key (for later use?)
>>
>> The keys are a procedurally generated with a PRNG sha2 bit streams based;
>> so 512 bits (16 bytes) at a time; and sha algorithms generates VERY good PR
>> numbers. which can be consumed as end-to-end bit streams.
>>
>> I might look into it; there are certainly a great test suite available to
>> reveal issues; but I expect Sqlite 'expects' memory to be 0 initialized
>> (even when filled from disk) and that it will be a HUGE can of worms.
>>
>>
>>

Thanks for the explanation ...

I never would have guessed that you "I double and halfedthe wrong
directions. " 😊😊



_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Fill empty space with random

Thilo Jeremias-3
In reply to this post by J Decker
>
> In the simple case, the VFS that the sqlite Db is mounted in is encrypted
> with a long key.  The key has cycles at 4096(A) and 16(B1-Bn) bytes
> (4096/16 = 256 cycles of Bn); such that each sector is masked with
> A^B1(256x), A^B2(256x), ... all together there is no repetition because the
> change from Bn to B(n+1) at the 4096 boundary makes the stream overall
> appear continuously random.
> Only data that is written is actually masked…


Apologies if I’m wrong or am missing something.
This sounds like wrong usage of encryption (ECB).

If the underlaying storage is encrypted properly ( maybe AES in counter mode with the counter being the block number or something),
there is no way to recover a key from learning any other blocks.

Thilo



_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: Fill empty space with random

Hick Gunter
In reply to this post by J Decker
For an encryption scheme to suffer enigma machine type vulnerabilities, the concept behind it must predate WWII.

IIRC the last straw (apart from known clear text like all messages ending with the same greeting) the broke the enigma encoding was the fact that a radio operator on an italian ship was told to transmit a test message. Nobody provided an unencrypted text, and so he proceeded to transmit a message consisting of a long run of the letter L. Due to electricall constraints, the enigma machine never translated a character to itself - thus there was no L in the chipher text, which oddity was detected by the cryptanalyst, and led him to assume a clear text of all L.

-----Ursprüngliche Nachricht-----
Von: sqlite-users [mailto:[hidden email]] Im Auftrag von J Decker
Gesendet: Sonntag, 14. Oktober 2018 16:57
An: General Discussion of SQLite Database <[hidden email]>
Betreff: [EXTERNAL] Re: [sqlite] Fill empty space with random

On Sun, Oct 14, 2018 at 7:24 AM Luuk <[hidden email]> wrote:

> On 14-10-2018 16:17, Simon Slavin wrote:
> > On 14 Oct 2018, at 12:56pm, J Decker <[hidden email]> wrote:
> >
> >> Is there maybe a compile option for sqlite to fill empty space in a
> >> db
> with random data rather than 0 ?
> > There is not.  But
> >
> > (A) It may be an easy change to the source code
> > (B) Your operating system may have a setting to do this
> > automatically to
> freed blocks on a storage device.
> > (C) Your device driver may have a setting to do this automatically
> > to
> freed blocks on the device.
> >
> > That type of security is normally done at OS or device level, not by
> each individual app.
> >
> > Simon.
> >
> Can you give any hints on why it would be a security issue to fill
> 'empty space' with 0, and why 'random data' should be used?
>
> ?
>
I hesitate to describe the real scenario; and want to instead manufacture one; but in either case I feel there will be more comments about the underlaying system than on Sqlite itself.

In the simple case, the VFS that the sqlite Db is mounted in is encrypted with a long key.  The key has cycles at 4096(A) and 16(B1-Bn) bytes
(4096/16 = 256 cycles of Bn); such that each sector is masked with A^B1(256x), A^B2(256x), ... all together there is no repetition because the change from Bn to B(n+1) at the 4096 boundary makes the stream overall appear continuously random.
Only data that is written is actually masked...

Sqlite likes to write 0's in large splotches (in my usage); which leaks key information; (only slightly more than the data stored in tables typically, which is a lot of the same bytes (0, 1 for instance and A-Z, a-z less-so; but all of that has upper bit(s) that are 0... )

And even is a specific sector (or several) is 'cracked' it doesn't do any good for any other page... but if LOTS of pages are found, it becomes easier to find what the overall A key is, which makes finding sector keys that you only need a few 32-64 bytes of 0's to reveal the sector specific key (for later use?)

The keys are a procedurally generated with a PRNG sha2 bit streams based; so 512 bits (16 bytes) at a time; and sha algorithms generates VERY good PR numbers. which can be consumed as end-to-end bit streams.

I might look into it; there are certainly a great test suite available to reveal issues; but I expect Sqlite 'expects' memory to be 0 initialized (even when filled from disk) and that it will be a HUGE can of worms.


>
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___________________________________________
 Gunter Hick | Software Engineer | Scientific Games International GmbH | Klitschgasse 2-4, A-1130 Vienna | FN 157284 a, HG Wien, DVR: 0430013 | (O) +43 1 80100 - 0

May be privileged. May be confidential. Please delete if not the addressee.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users