Enabling MMAP in Android

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Enabling MMAP in Android

Nick
I find MMAP is able to be used for db files(getPageMMap()) but is not enabled
by default.
I wonder if it is recommended to use MMAP in Android? As I think MMAP is one
way to improve the speed performance(I use speedtest1.c to evaluate).

At the same time, Mmap() is only used for reading by default, not writing.
But why? And in what situation I can compile with the
-DSQLITE_MMAP_READWRITE option?

By the way, is there any other recommended compile-options? I find some
references in  http://www.sqlite.org/compile.html
<http://www.sqlite.org/compile.html>  , but I am wondering if there is any
other method to enhance performance?

Thanks for your help.



--
Sent from: http://sqlite.1065341.n5.nabble.com/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Enabling MMAP in Android

Graham Holden
Have you actually got speed issues, or is this just a case of "this should be faster"? If you DO have speed issues, my experience from reading this list is that most can be improved with better SQL, better indices, or better interplay between readers and writers. Only if those have been exhausted should you be looking at compile options.
Graham.


Sent from my Samsung Galaxy S7 - powered by Three
-------- Original message --------From: advancenOO <[hidden email]> Date: 11/12/2017  14:04  (GMT+00:00) To: [hidden email] Subject: [sqlite] Enabling MMAP in Android
I find MMAP is able to be used for db files(getPageMMap()) but is not enabled
by default.
I wonder if it is recommended to use MMAP in Android? As I think MMAP is one
way to improve the speed performance(I use speedtest1.c to evaluate).

At the same time, Mmap() is only used for reading by default, not writing.
But why? And in what situation I can compile with the
-DSQLITE_MMAP_READWRITE option?

By the way, is there any other recommended compile-options? I find some
references in  http://www.sqlite.org/compile.html
<http://www.sqlite.org/compile.html>  , but I am wondering if there is any
other method to enhance performance?

Thanks for your help.



--
Sent from: http://sqlite.1065341.n5.nabble.com/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Enabling MMAP in Android

Richard Hipp-3
In reply to this post by Nick
On 12/11/17, advancenOO <[hidden email]> wrote:
>
> At the same time, Mmap() is only used for reading by default, not writing.
> But why?

If the database is memory-mapped writable, then a stray pointer or
array overrun in the application (NB: not in SQLite - in the
application that uses SQLite) could easily overwrite part of the
database file and corrupt the database file.

It is also possible for this to happen without the use of writable
memory-maps, if the application corrupts internal SQLite data
structures and a subsequent COMMIT occurs.  But that is much less
likely, as SQLite would probably detect the situation before the
COMMIT completes, thus leaving the database intact.

--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Enabling MMAP in Android

Nick
Um, stray pointer or array overrun in upper applications will be treated as
BUG in my system. And I believe all these bugs could be fixed before using
sqlite.
So can I safely enable SQLITE_MMAP_READWRITE directly if I can ignore the
possibility of the BUG?

But I notice that Sqlite may not call msync() in mmap, calls fsync instead.
So I am wondering if it is safe to use SQLITE_MMAP_READWRITE to write to
mmapped memory as msync() is not called.



--
Sent from: http://sqlite.1065341.n5.nabble.com/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Enabling MMAP in Android

Nick
In reply to this post by Graham Holden
I am a developer of system, not application. So I have nothing to do with the
SQL or indices even though I know these may cut orders of magnitude from
execution times.

And that is why I use speedtest1.c of Sqlite to evaluate the performance of
my system.



--
Sent from: http://sqlite.1065341.n5.nabble.com/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Enabling MMAP in Android

Simon Slavin-3
In reply to this post by Nick


On 12 Dec 2017, at 1:32pm, advancenOO <[hidden email]> wrote:

> Um, stray pointer or array overrun in upper applications will be treated as
> BUG in my system. And I believe all these bugs could be fixed before using
> sqlite.
> So can I safely enable SQLITE_MMAP_READWRITE directly if I can ignore the
> possibility of the BUG?

The stray pointer or array overrun is not in the SQLite code.  It’s in the implementation of memory mapping built into the OS.  So you need to ask yourself whether you trust your OS.

Before you answer that question, you should know that both Windows and macOS have been proved to have serious bugs in their memory mapping code.

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Enabling MMAP in Android

Richard Hipp-3
In reply to this post by Nick
On 12/12/17, advancenOO <[hidden email]> wrote:
> Um, stray pointer or array overrun in upper applications will be treated as
> BUG in my system. And I believe all these bugs could be fixed before using
> sqlite.

You have a very optimistic view of application software :-)

Look - if you are interested in maximizing performance, you need to
use the latest Linux kernel with the latest F2FS filesystem, then
compile SQLite with the -DSQLITE_ENABLE_BATCH_ATOMIC_WRITE.  That
combination will get you way more performance boost than you will ever
find by playing games with mmap().
--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Enabling MMAP in Android

Jens Alfke-2
In reply to this post by Simon Slavin-3


> On Dec 12, 2017, at 5:46 AM, Simon Slavin <[hidden email]> wrote:
>
> The stray pointer or array overrun is not in the SQLite code.  It’s in the implementation of memory mapping built into the OS.  So you need to ask yourself whether you trust your OS.

This argument (which is a standard one against using writeable mmap) is about _application_ code issuing stray writes. This wasn’t a topic about OS bugs.

I’m not sure I buy the argument anyway. It’s already possible for app code to write into SQLite’s buffer cache, which can then be written back to disk, corrupting the file. And the buffer cache is located in the malloc heap, which is close to application heap blocks, while a mmap’ed database file is likely to be somewhere way far away in address space.

> Before you answer that question, you should know that both Windows and macOS have been proved to have serious bugs in their memory mapping code.

This has been brought up several times recently, but I’ve never seen any details given about exactly what was wrong with macOS’s mmap implementation. Does anyone have a pointer to authoritative information about this?

—Jens
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Enabling MMAP in Android

Simon Slavin-3


On 12 Dec 2017, at 6:27pm, Jens Alfke <[hidden email]> wrote:

> On Dec 12, 2017, at 5:46 AM, Simon Slavin <[hidden email]> wrote:

>> Before you answer that question, you should know that both Windows and macOS have been proved to have serious bugs in their memory mapping code.
>
> This has been brought up several times recently, but I’ve never seen any details given about exactly what was wrong with macOS’s mmap implementation. Does anyone have a pointer to authoritative information about this?

See this thread:

<http://sqlite.1065341.n5.nabble.com/SQLITE-vs-OSX-mmap-inevitable-catalog-corruption-td85620.html>

This is the thread which led to memory mapping being disabled for writing on macOS, as discussed here:

<http://sqlite.1065341.n5.nabble.com/Re-Database-corruption-and-PRAGMA-fullfsync-on-macOS-td95366i20.html>

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Enabling MMAP in Android

Martin Raiber
On 12.12.2017 19:47 Simon Slavin wrote:

> On 12 Dec 2017, at 6:27pm, Jens Alfke <[hidden email]> wrote:
>
>> On Dec 12, 2017, at 5:46 AM, Simon Slavin <[hidden email]> wrote:
>>> Before you answer that question, you should know that both Windows and macOS have been proved to have serious bugs in their memory mapping code.
>> This has been brought up several times recently, but I’ve never seen any details given about exactly what was wrong with macOS’s mmap implementation. Does anyone have a pointer to authoritative information about this?
> See this thread:
>
> <http://sqlite.1065341.n5.nabble.com/SQLITE-vs-OSX-mmap-inevitable-catalog-corruption-td85620.html>
>
> This is the thread which led to memory mapping being disabled for writing on macOS, as discussed here:
>
> <http://sqlite.1065341.n5.nabble.com/Re-Database-corruption-and-PRAGMA-fullfsync-on-macOS-td95366i20.html>

There might also be a mmap bug in the Android 7.0 fuse layer:
https://www.mail-archive.com/openldap-its@.../msg10970.html


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Enabling MMAP in Android

Jens Alfke-2
In reply to this post by Simon Slavin-3


> On Dec 12, 2017, at 10:47 AM, Simon Slavin <[hidden email]> wrote:
>
> See this thread:
>
> <http://sqlite.1065341.n5.nabble.com/SQLITE-vs-OSX-mmap-inevitable-catalog-corruption-td85620.html <http://sqlite.1065341.n5.nabble.com/SQLITE-vs-OSX-mmap-inevitable-catalog-corruption-td85620.html>>

That thread reported SQLite database corruption when writeable mmap was enabled, but didn't identify what caused it. There was also the following observation by Török Edwin, which no one ever replied to:

>> I notice that SQLite doesn't use msync, and msync(2) says 'Without use of this call there is no guarantee that changes are written back before munmap(2) is called'.
>> Is f(data)sync enough to ensure changed mmap pages are written to the disk?


That sounds pretty suspicious to me; maybe the problem was just that sqlite needed to call msync?

> This is the thread which led to memory mapping being disabled for writing on macOS, as discussed here:
>
> <http://sqlite.1065341.n5.nabble.com/Re-Database-corruption-and-PRAGMA-fullfsync-on-macOS-td95366i20.html <http://sqlite.1065341.n5.nabble.com/Re-Database-corruption-and-PRAGMA-fullfsync-on-macOS-td95366i20.html>>

Yup, and I was heavily involved in that thread :) Again, there was only circumstantial evidence that mmap was causing problems; no one seems to have gotten to the bottom of it.

I’m skeptical about mmap being broken on Macs, since there are other production databases such as Realm* that use it heavily. (Though I am not sure whether Realm uses writeable mappings.)

—Jens

* and LMDB, but I am not sure if LMDB is in use on macOS.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Enabling MMAP in Android

Howard Chu
Jens Alfke wrote:

> I’m skeptical about mmap being broken on Macs, since there are other production databases such as Realm* that use it heavily. (Though I am not sure whether Realm uses writeable mappings.)
>
> —Jens
>
> * and LMDB, but I am not sure if LMDB is in use on macOS.

LMDB is in common use on MacOS, no issues have been reported. On iOS there's
this nagging problem that the virtual address space is still limited to 4GB,
even on 64bit systems.

--
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Enabling MMAP in Android

Howard Chu
In reply to this post by Martin Raiber
Martin Raiber wrote:

> On 12.12.2017 19:47 Simon Slavin wrote:
>> On 12 Dec 2017, at 6:27pm, Jens Alfke <[hidden email]> wrote:
>>
>>> On Dec 12, 2017, at 5:46 AM, Simon Slavin <[hidden email]> wrote:
>>>> Before you answer that question, you should know that both Windows and macOS have been proved to have serious bugs in their memory mapping code.
>>> This has been brought up several times recently, but I’ve never seen any details given about exactly what was wrong with macOS’s mmap implementation. Does anyone have a pointer to authoritative information about this?
>> See this thread:
>>
>> <http://sqlite.1065341.n5.nabble.com/SQLITE-vs-OSX-mmap-inevitable-catalog-corruption-td85620.html>
>>
>> This is the thread which led to memory mapping being disabled for writing on macOS, as discussed here:
>>
>> <http://sqlite.1065341.n5.nabble.com/Re-Database-corruption-and-PRAGMA-fullfsync-on-macOS-td95366i20.html>
>
> There might also be a mmap bug in the Android 7.0 fuse layer:
> https://www.mail-archive.com/openldap-its@.../msg10970.html

There is definitely a bug in Android 7 fuse/mmap. The bug is definitely not
present when bypassing fuse, but only rooted devices can bypass...

--
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Enabling MMAP in Android

J Decker
On Tue, Dec 12, 2017 at 4:35 PM, Howard Chu <[hidden email]> wrote:

> Martin Raiber wrote:
>
>> On 12.12.2017 19:47 Simon Slavin wrote:
>>
>>> On 12 Dec 2017, at 6:27pm, Jens Alfke <[hidden email]> wrote:
>>>
>>> On Dec 12, 2017, at 5:46 AM, Simon Slavin <[hidden email]> wrote:
>>>>
>>>>> Before you answer that question, you should know that both Windows and
>>>>> macOS have been proved to have serious bugs in their memory mapping code.
>>>>>
>>>> This has been brought up several times recently, but I’ve never seen
>>>> any details given about exactly what was wrong with macOS’s mmap
>>>> implementation. Does anyone have a pointer to authoritative information
>>>> about this?
>>>>
>>> See this thread:
>>>
>>> <http://sqlite.1065341.n5.nabble.com/SQLITE-vs-OSX-mmap-inev
>>> itable-catalog-corruption-td85620.html>
>>>
>>> This is the thread which led to memory mapping being disabled for
>>> writing on macOS, as discussed here:
>>>
>>> <http://sqlite.1065341.n5.nabble.com/Re-Database-corruption-
>>> and-PRAGMA-fullfsync-on-macOS-td95366i20.html>
>>>
>>
>> There might also be a mmap bug in the Android 7.0 fuse layer:
>> https://www.mail-archive.com/openldap-its@.../msg10970.html
>>
>
> There is definitely a bug in Android 7 fuse/mmap. The bug is definitely
> not present when bypassing fuse, but only rooted devices can bypass...
>
> My experience was in majority only rooted devices could use fuse.  since
/dev/fuse was rw------- .  (although not all.)

> --
>   -- Howard Chu
>   CTO, Symas Corp.           http://www.symas.com
>   Director, Highland Sun     http://highlandsun.com/hyc/
>   Chief Architect, OpenLDAP  http://www.openldap.org/project/
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Enabling MMAP in Android

Howard Chu
J Decker wrote:

> On Tue, Dec 12, 2017 at 4:35 PM, Howard Chu <[hidden email]> wrote:
>
>> Martin Raiber wrote:
>>
>>> On 12.12.2017 19:47 Simon Slavin wrote:
>>>
>>>> On 12 Dec 2017, at 6:27pm, Jens Alfke <[hidden email]> wrote:
>>>>
>>>> On Dec 12, 2017, at 5:46 AM, Simon Slavin <[hidden email]> wrote:
>>>>>
>>>>>> Before you answer that question, you should know that both Windows and
>>>>>> macOS have been proved to have serious bugs in their memory mapping code.
>>>>>>
>>>>> This has been brought up several times recently, but I’ve never seen
>>>>> any details given about exactly what was wrong with macOS’s mmap
>>>>> implementation. Does anyone have a pointer to authoritative information
>>>>> about this?
>>>>>
>>>> See this thread:
>>>>
>>>> <http://sqlite.1065341.n5.nabble.com/SQLITE-vs-OSX-mmap-inev
>>>> itable-catalog-corruption-td85620.html>
>>>>
>>>> This is the thread which led to memory mapping being disabled for
>>>> writing on macOS, as discussed here:
>>>>
>>>> <http://sqlite.1065341.n5.nabble.com/Re-Database-corruption-
>>>> and-PRAGMA-fullfsync-on-macOS-td95366i20.html>
>>>>
>>>
>>> There might also be a mmap bug in the Android 7.0 fuse layer:
>>> https://www.mail-archive.com/openldap-its@.../msg10970.html
>>>
>>
>> There is definitely a bug in Android 7 fuse/mmap. The bug is definitely
>> not present when bypassing fuse, but only rooted devices can bypass...
>>
>> My experience was in majority only rooted devices could use fuse.  since
> /dev/fuse was rw------- .  (although not all.)

The /sdcard partition is always mounted through fuse because it's a vfat/exfat
filesystem that doesn't support owner/permission bits, and the fuse driver
imposes the Android security model on top of it. Many android devices no
longer include a physical SDcard slot, but still have an internal storage
partition that's labeled /sdcard and it still behaves this way.

--
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Enabling MMAP in Android

Nick
In reply to this post by Simon Slavin-3
Thanks for the notification about the bugs in MacOS.
So the most important reason for disable SQLITE_MMAP_READWRITE is the bugs
in OS, not the stray pointer in upper applications. Is it right?



--
Sent from: http://sqlite.1065341.n5.nabble.com/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Enabling MMAP in Android

Nick
In reply to this post by Richard Hipp-3
Yeah. Thanks for your suggestion about -DSQLITE_ENABLE_BATCH_ATOMIC_WRITE.
But the version of my sqlite is 3.16.2, so I guess I may use it at a later
time int the future.



--
Sent from: http://sqlite.1065341.n5.nabble.com/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Enabling MMAP in Android

Nick
In reply to this post by Howard Chu

OK, I know it is not safe when use mmap in Android 7 fuse.
And do you mean the bug is in fuse kernel framework as you mentioned Android
FUSE
filesystem driver? Which corresponding to Linux 3.10?
 
By the way, do you have any possible repro steps especially in Android
Sqlite?



--
Sent from: http://sqlite.1065341.n5.nabble.com/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users