Why MMAP return ENOMEM in SQLite?

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Why MMAP return ENOMEM in SQLite?

sanhua.zh
I try to mmap a BIG file which is around 1.8GB size, on iOS.
Then I found that [sqlite3OSFetch] doesn’t work at all, which means the mmap is disable.
After debug, I find that [osMmap] in [unixRemapFile] return an error. The error code is 12, named ENOMEM, which means cannot allocate memory.


My question is:
Is it caused by mmap file too big that the address space is not enough?
If yes, then how can I get the allowed mmap file size?
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Why MMAP return ENOMEM in SQLite?

Simon Slavin-3

On 31 Aug 2016, at 4:40am, sanhua.zh <[hidden email]> wrote:

> Is it caused by mmap file too big that the address space is not enough?

Yes.  Under iOS, memory maps can only be as big as the available physical memory.  You're working on a tiny multi-tasking device and memory is at a premium.  You can't afford to hog memory.  Nor can you do data-handling so fast that it runs down the battery or overheats the device.

Also, iDevices before the iPhone 5S in 2013 ran a 32-bit version of iOS, limiting them to 4GB of memory per App.

> If yes, then how can I get the allowed mmap file size?

This figure would be useless since available physical memory varies depending on things your application cannot prevent.  For instance, you may find out how much free memory there is, but then there may be in incoming phonecall.  iOS will then allocate some of that memory to the process dealing with the phonecall.  And, of course you cannot do anything to interfere with a phonecall because that is more important than an App.

Why are you trying to do memory-mapping on a portable device ?  Is it for speed ?  Do you absolutely need it ?  Won't the standard SQLite API do the job well enough ?

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Why MMAP return ENOMEM in SQLite?

sanhua.zh
In reply to this post by sanhua.zh
Thanks for your answer.


Simon Why are you trying to do memory-mapping on a portable device ? Is it for speed ? Do you absolutely need it ?
Yes. even on iDevice, [mmap] can get faster than sequential I/O. Sometime it will be twice faster.


Simon Won't the standard SQLite API do the job well enough ?
I did use the standard SQLite API. It maps the whole db file into address space. When the db file is too big, the [mmap] will failed and back tosequential I/O method.


I have another way to solve this problem.
The address space might be“scttered”. There is not enough space to fit a BIG file. But there are a lot of fragment space to fit multiple small region.
So I can map multiple regions of file, instead of mapping the whole db file. For example, mapping 128 4MB-regions, instead of mapping the whole 1GB file.
This solution should modify the [unixRemapFile] method in the source code of SQLite, also, I should remap the exact region into the pMapRegion variable.
In my testcase, I can only [mmap] a db file at most 1.4GB size. But in this new way, I can map a file at most 3.2GB. The test device is iPhone 6S.




原始邮件
发件人:Simon [hidden email]
收件人:SQLite mailing [hidden email]
发送时间:2016年8月31日(周三) 14:38
主题:Re: [sqlite] Why MMAP return ENOMEM in SQLite?


On 31 Aug 2016, at 4:40am, sanhua.zh [hidden email] wrote:  Is it caused by mmap file too big that the address space is not enough? Yes. Under iOS, memory maps can only be as big as the available physical memory. You're working on a tiny multi-tasking device and memory is at a premium. You can't afford to hog memory. Nor can you do data-handling so fast that it runs down the battery or overheats the device. Also, iDevices before the iPhone 5S in 2013 ran a 32-bit version of iOS, limiting them to 4GB of memory per App.  If yes, then how can I get the allowed mmap file size? This figure would be useless since available physical memory varies depending on things your application cannot prevent. For instance, you may find out how much free memory there is, but then there may be in incoming phonecall. iOS will then allocate some of that memory to the process dealing with the phonecall. And, of course you cannot do anything to interfere with a phonecall because that is more important than an App. Why are you trying to do memory-mapping on a portable device ? Is it for speed ? Do you absolutely need it ? Won't the standard SQLite API do the job well enough ? Simon. _______________________________________________ sqlite-users mailing list [hidden email] http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Why MMAP return ENOMEM in SQLite?

sanhua.zh
In reply to this post by sanhua.zh
Thanks for your answer. Simon Why are you trying to do memory-mapping on a portable device ? Is it for speed ? Do you absolutely need it ? Yes. even on iDevice, [mmap] can get faster than sequential I/O. Sometimes it will be twice faster. Simon Won't the standard SQLite API do the job well enough ? I did use the standard SQLite API. It maps the whole db file into address space. When the db file is too big, the [mmap] will failed and turn back to sequential I/O method. I have another way to solve this problem. The address space might be“scttered”. There is not enough space to fit a BIG file. But there are a lot of fragment space to fit multiple small regions. So I can map multiple regions of file, instead of mapping the whole db file. For example, mapping 256 4MB-regions, instead of mapping the whole 1GB file. This solution should modify the [unixRemapFile] method in the source code of SQLite, also, I should remap the exact region into the pMapRegion variable. In my testcase, I can only [mmap] a db file at most 1.4GB size. But in this new way, I can map a file at most 3.2GB. The test device is iPhone 6S.




原始邮件
发件人:[hidden email]
收件人:SQLite mailing [hidden email]
发送时间:2016年8月31日(周三) 16:39
主题:Re: [sqlite] Why MMAP return ENOMEM in SQLite?


Thanks for your answer. Simon Why are you trying to do memory-mapping on a portable device ? Is it for speed ? Do you absolutely need it ? Yes. even on iDevice, [mmap] can get faster than sequential I/O. Sometime it will be twice faster. Simon Won't the standard SQLite API do the job well enough ? I did use the standard SQLite API. It maps the whole db file into address space. When the db file is too big, the [mmap] will failed and back tosequential I/O method. I have another way to solve this problem. The address space might be“scttered”. There is not enough space to fit a BIG file. But there are a lot of fragment space to fit multiple small region. So I can map multiple regions of file, instead of mapping the whole db file. For example, mapping 128 4MB-regions, instead of mapping the whole 1GB file. This solution should modify the [unixRemapFile] method in the source code of SQLite, also, I should remap the exact region into the pMapRegion variable. In my testcase, I can only [mmap] a db file at most 1.4GB size. But in this new way, I can map a file at most 3.2GB. The test device is iPhone 6S. 原始邮件 发件人:Simon [hidden email] 收件人:SQLite mailing [hidden email] 发送时间:2016年8月31日(周三) 14:38 主题:Re: [sqlite] Why MMAP return ENOMEM in SQLite? On 31 Aug 2016, at 4:40am, sanhua.zh [hidden email] wrote: Is it caused by mmap file too big that the address space is not enough? Yes. Under iOS, memory maps can only be as big as the available physical memory. You're working on a tiny multi-tasking device and memory is at a premium. You can't afford to hog memory. Nor can you do data-handling so fast that it runs down the battery or overheats the device. Also, iDevices before the iPhone 5S in 2013 ran a 32-bit version of iOS, limiting them to 4GB of memory per App. If yes, then how can I get the allowed mmap file size? This figure would be useless since available physical memory varies depending on things your application cannot prevent. For instance, you may find out how much free memory there is, but then there may be in incoming phonecall. iOS will then allocate some of that memory to the process dealing with the phonecall. And, of course you cannot do anything to interfere with a phonecall because that is more important than an App. Why are you trying to do memory-mapping on a portable device ? Is it for speed ? Do you absolutely need it ? Won't the standard SQLite API do the job well enough ? Simon. _______________________________________________ sqlite-users mailing list [hidden email] http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list [hidden email] http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Why MMAP return ENOMEM in SQLite?

Stephan Beal-3
In reply to this post by sanhua.zh
On Wed, Aug 31, 2016 at 10:39 AM, sanhua.zh <[hidden email]> wrote:

> In my testcase, I can only [mmap] a db file at most 1.4GB size. But in
> this new way, I can map a file at most 3.2GB. The test device is iPhone 6S.
>

According to google, the iPhone 6s only has 2GB of RAM, so you can't memmap
3.2G. In any case, as Simone said, _other apps_ require memory of their own
as well. Your app does not have that whole 2G to itself.

--
----- stephan beal
http://wanderinghorse.net/home/stephan/
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Why MMAP return ENOMEM in SQLite?

Stephan Beal-3
On Wed, Aug 31, 2016 at 10:43 AM, Stephan Beal <[hidden email]>
wrote:

> On Wed, Aug 31, 2016 at 10:39 AM, sanhua.zh <[hidden email]> wrote:
>
>> In my testcase, I can only [mmap] a db file at most 1.4GB size. But in
>> this new way, I can map a file at most 3.2GB. The test device is iPhone 6S.
>>
>
> According to google, the iPhone 6s only has 2GB of RAM, so you can't
> memmap 3.2G.
>

Correction: you can map larger than physical memory, up to limits set by
the OS environment. In your case, see:

http://stackoverflow.com/questions/9184773/is-there-a-practical-limit-on-the-number-of-memory-mapped-files-in-ios


> In any case, as Simone said, _other apps_ require memory of their own as
> well.
>

Correction 2: Simon, not Simone (my roommate's name, so i type the 'e' out
of habit)

--
----- stephan beal
http://wanderinghorse.net/home/stephan/
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Why MMAP return ENOMEM in SQLite?

sanhua.zh
In reply to this post by sanhua.zh
Yes, [mmap] can be larger than physical memory.
And what do you think about the new mapping way I mentioned ?




原始邮件
发件人:Stephan [hidden email]
收件人:SQLite mailing [hidden email]
发送时间:2016年8月31日(周三) 16:53
主题:Re: [sqlite] Why MMAP return ENOMEM in SQLite?


On Wed, Aug 31, 2016 at 10:43 AM, Stephan Beal [hidden email] wrote:  On Wed, Aug 31, 2016 at 10:39 AM, sanhua.zh [hidden email] wrote:   In my testcase, I can only [mmap] a db file at most 1.4GB size. But in  this new way, I can map a file at most 3.2GB. The test device is iPhone 6S.    According to google, the iPhone 6s only has 2GB of RAM, so you can't  memmap 3.2G.  Correction: you can map larger than physical memory, up to limits set by the OS environment. In your case, see: http://stackoverflow.com/questions/9184773/is-there-a-practical-limit-on-the-number-of-memory-mapped-files-in-ios  In any case, as Simone said, _other apps_ require memory of their own as  well.  Correction 2: Simon, not Simone (my roommate's name, so i type the 'e' out of habit) -- ----- stephan beal http://wanderinghorse.net/home/stephan/ "Freedom is sloppy. But since tyranny's the only guaranteed byproduct of those who insist on a perfect world, freedom will have to do." -- Bigby Wolf _______________________________________________ sqlite-users mailing list [hidden email] http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Why MMAP return ENOMEM in SQLite?

Stephan Beal-3
On Wed, Aug 31, 2016 at 10:55 AM, sanhua.zh <[hidden email]> wrote:

> Yes, [mmap] can be larger than physical memory.
>

Indeed, my mistake.


> And what do you think about the new mapping way I mentioned ?
>

i think it's a "huge can of worms" - it's asking for more, bigger problems
than the perceived performance problems you have right now. The chances
that something breaks with that approach is, i suspect, very high. You will
eventually corrupt a database and then write back to the list to ask why
that approach corrupted it.

--
----- stephan beal
http://wanderinghorse.net/home/stephan/
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Why MMAP return ENOMEM in SQLite?

sanhua.zh
In reply to this post by sanhua.zh
Why do you think it will corrupt the database?
Can you give me more explainations or examples?




原始邮件
发件人:Stephan [hidden email]
收件人:SQLite mailing [hidden email]
发送时间:2016年8月31日(周三) 16:57
主题:Re: [sqlite] Why MMAP return ENOMEM in SQLite?


On Wed, Aug 31, 2016 at 10:55 AM, sanhua.zh [hidden email] wrote:  Yes, [mmap] can be larger than physical memory.  Indeed, my mistake.  And what do you think about the new mapping way I mentioned ?  i think it's a "huge can of worms" - it's asking for more, bigger problems than the perceived performance problems you have right now. The chances that something breaks with that approach is, i suspect, very high. You will eventually corrupt a database and then write back to the list to ask why that approach corrupted it. -- ----- stephan beal http://wanderinghorse.net/home/stephan/ "Freedom is sloppy. But since tyranny's the only guaranteed byproduct of those who insist on a perfect world, freedom will have to do." -- Bigby Wolf _______________________________________________ sqlite-users mailing list [hidden email] http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Why MMAP return ENOMEM in SQLite?

Stephan Beal-3
On Wed, Aug 31, 2016 at 11:03 AM, sanhua.zh <[hidden email]> wrote:

> Why do you think it will corrupt the database?
> Can you give me more explainations or examples?
>

It's only my intuition - i don't have a concrete example. sqlite and ios
are "well-oiled machines." They do their jobs and they do it well. If you
start interfering with that, trying to take over or abuse their
responsibilities because you think you can do it better, you will
_eventually_ run into problems. In my experience, the chances of a
back-fire when trying to push software beyond what it's designed to do are
very high.

You explicitly want to add complexity to an already complex system.
Additional complexity almost always comes with a higher bug rate. A
telephone is _not_ a high-performance computing platform, but a
_convenience_ platform. Whether a db operation takes 10ms or 800ms should,
for such platforms, be irrelevant. i _suspect_ that you are overestimating
the impact of your perceived performance problem on the end users.

But that's all just my opinion based on experience - i have no facts or
statistics to back it up. Maybe it will work well for you.

--
----- stephan beal
http://wanderinghorse.net/home/stephan/
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Why MMAP return ENOMEM in SQLite?

sanhua.zh
In reply to this post by sanhua.zh
OK, I get your idea.
Thanks for your advice. :)


原始邮件
发件人:Stephan [hidden email]
收件人:SQLite mailing [hidden email]
发送时间:2016年8月31日(周三) 17:09
主题:Re: [sqlite] Why MMAP return ENOMEM in SQLite?


On Wed, Aug 31, 2016 at 11:03 AM, sanhua.zh [hidden email] wrote:  Why do you think it will corrupt the database?  Can you give me more explainations or examples?  It's only my intuition - i don't have a concrete example. sqlite and ios are "well-oiled machines." They do their jobs and they do it well. If you start interfering with that, trying to take over or abuse their responsibilities because you think you can do it better, you will _eventually_ run into problems. In my experience, the chances of a back-fire when trying to push software beyond what it's designed to do are very high. You explicitly want to add complexity to an already complex system. Additional complexity almost always comes with a higher bug rate. A telephone is _not_ a high-performance computing platform, but a _convenience_ platform. Whether a db operation takes 10ms or 800ms should, for such platforms, be irrelevant. i _suspect_ that you are overestimating the impact of your perceived performance problem on the end users. But that's all just my opinion based on experience - i have no facts or statistics to back it up. Maybe it will work well for you. -- ----- stephan beal http://wanderinghorse.net/home/stephan/ "Freedom is sloppy. But since tyranny's the only guaranteed byproduct of those who insist on a perfect world, freedom will have to do." -- Bigby Wolf _______________________________________________ sqlite-users mailing list [hidden email] http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Why MMAP return ENOMEM in SQLite?

Simon Slavin-3
In reply to this post by sanhua.zh

On 31 Aug 2016, at 9:39am, sanhua.zh <[hidden email]> wrote:

> Yes. even on iDevice, [mmap] can get faster than sequential I/O. Sometime it will be twice faster.

Unless your users have complained about speed, this does not matter.  There is no need to make everything happen as fast /as possible/.  Try using just standard SQLite and find out if it is fast /enough/.  Then you don't have to spend lots of programming time learning tiny little details about one small piece of a software library.

> Simon Won't the standard SQLite API do the job well enough ?
> I did use the standard SQLite API. It maps the whole db file into address space.

It does this only if you have told it to.  According to

<https://www.sqlite.org/mmap.html>

"To activate memory-mapped I/O, use the mmap_size pragma and set the mmap_size to some large number"

So do not do this and SQLite will stop trying to memory map the file.

But it is possible that you are using a SQLite library which has a default memory map size.  Can you execute the command "PRAGMA mmap_size" and tell what it outputs ?

If the result is not zero or blank, you can disable memory mapping using the command "PRAGMA mmap_size=0" after you have opened the database.  Please try this and see whether your application still crashes.

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Why MMAP return ENOMEM in SQLite?

sanhua.zh
In reply to this post by sanhua.zh
Unless your users have complained about speed, this does not matter. There is no need to make everything happen as fast /as possible/. Try using just standard SQLite and find out if it is fast /enough/. Then you don't have to spend lots of programming time learning tiny little details about one small piece of a software library.
I have my monitor system to check the lag and delay. I can make sure that I must make it faster.



PRAGMA mmap_size=0x7fff0000
I am using SQLite for a long time, and be familiar with the most of the source code. So you have no need to worry about the basic things.


BTW, what do you think if I mapseparatly instead of the whole db file, which is the way I mentioned before ?




原始邮件
发件人:Simon [hidden email]
收件人:SQLite mailing [hidden email]
发送时间:2016年8月31日(周三) 18:04
主题:Re: [sqlite] Why MMAP return ENOMEM in SQLite?


On 31 Aug 2016, at 9:39am, sanhua.zh [hidden email] wrote:  Yes. even on iDevice, [mmap] can get faster than sequential I/O. Sometime it will be twice faster. Unless your users have complained about speed, this does not matter. There is no need to make everything happen as fast /as possible/. Try using just standard SQLite and find out if it is fast /enough/. Then you don't have to spend lots of programming time learning tiny little details about one small piece of a software library.  Simon Won't the standard SQLite API do the job well enough ?  I did use the standard SQLite API. It maps the whole db file into address space. It does this only if you have told it to. According to https://www.sqlite.org/mmap.html "To activate memory-mapped I/O, use the mmap_size pragma and set the mmap_size to some large number" So do not do this and SQLite will stop trying to memory map the file. But it is possible that you are using a SQLite library which has a default memory map size. Can you execute the command "PRAGMA mmap_size" and tell what it outputs ? If the result is not zero or blank, you can disable memory mapping using the command "PRAGMA mmap_size=0" after you have opened the database. Please try this and see whether your application still crashes. Simon. _______________________________________________ sqlite-users mailing list [hidden email] http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Why MMAP return ENOMEM in SQLite?

Simon Slavin-3

On 31 Aug 2016, at 12:28pm, sanhua.zh <[hidden email]> wrote:

> BTW, what do you think if I mapseparatly instead of the whole db file, which is the way I mentioned before ?

Sorry but I have no experience with that.  I hope other people can advise you.

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Why MMAP return ENOMEM in SQLite?

Scott Robison-2
In reply to this post by sanhua.zh
On Aug 31, 2016 5:29 AM, "sanhua.zh" <[hidden email]> wrote:
>
> BTW, what do you think if I mapseparatly instead of the whole db file, ...

I suspect that it wouldn't really help you much, if any.

One, there is overhead in making that many system calls to map a bunch of 4
MiB buffers.

Two, once you've mapped that many buffers, you now have to determine where
each part of your file is in memory before accessing it. You've lost the
benefit of easy address calculations and now have to perform an indirect
lookup, first finding the correct page, next computing the offset into that
page. It would be potentially worse if a piece of data every spanned two
pages, but I doubt that would be an issue for SQLite.

Three, I have no idea what limits exist for mapping pages at the OS level
per process, but it wouldn't surprise me if maybe this exceeded something
like that.

Four, the one potential benefit is that mapping the pages would avoid the
penalty of copying read data from a kernel buffer to your user space
buffer. I think the additional complexity and my other reasons above make
it a less than ideal solution.

If your problem space requires higher speed access to the data than SQLite
is capable of delivering, it seems to me that you'd be better off with a
data storage solution tailored to your requirements. I don't make that
suggestion lightly, and would have to be really desperate for performance
to do it myself, but a more specialized solution can gain performance at
the expense of not being as generally useful. Even then, it might be
difficult to improve on SQLite, a very optimized library for data storage,
manipulation, and access.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users