Does wal-file support MMAP?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Does wal-file support MMAP?

advancenOO

I am trying to understand how SQLITE_DEFAULT_MMAP_SIZE works and I think db
files can use MMAP by using this compilation option, as Sqlite3PagerGet()
mapped to getPageMMap().
And I know "/the current VFS implementations use a mmapped file for the
wal-index/"

But is there a way that wal-files can use MMAP? Why not? As I guess it can
improve the IO performance in WAL mode.

Thanks for your insights.




--
Sent from: http://sqlite.1065341.n5.nabble.com/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Does wal-file support MMAP?

Dan Kennedy-4
On 11/09/2017 08:59 AM, advancenOO wrote:
> I am trying to understand how SQLITE_DEFAULT_MMAP_SIZE works and I think db
> files can use MMAP by using this compilation option, as Sqlite3PagerGet()
> mapped to getPageMMap().
> And I know "/the current VFS implementations use a mmapped file for the
> wal-index/"
>
> But is there a way that wal-files can use MMAP? Why not? As I guess it can
> improve the IO performance in WAL mode.

No way to do that right now. It would require a VFS interface change I
think.

Dan.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Does wal-file support MMAP?

advancenOO
In multi-thread scenario, I guess that threads may not use MMAP correctly
after one thread truncates wal-file. And that is the reason MMAP can not be
used together with WAL mode.

So, I am wondering if I can simply make the size of Wal-file a fixed size to
support MMAP for wal-file.



--
Sent from: http://sqlite.1065341.n5.nabble.com/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Does wal-file support MMAP?

Simon Slavin-3


On 10 Nov 2017, at 5:35am, advancenOO <[hidden email]> wrote:

> In multi-thread scenario, I guess that threads may not use MMAP correctly
> after one thread truncates wal-file. And that is the reason MMAP can not be
> used together with WAL mode.

Are the threads using the same connection ?

> So, I am wondering if I can simply make the size of Wal-file a fixed size to
> support MMAP for wal-file.

What makes you think you need to use memory mapping ?  It is purely for speed or do you have a hardware-based reason ?  Have you tried your program without memory mapping to see if it’s fast enough ?

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Does wal-file support MMAP?

advancenOO

Purely for speed :)
I hope to improve the performance inside sqlite in order to optimize the
performance of the upper applications. As there are so many different scenes
in APPs.
 



--
Sent from: http://sqlite.1065341.n5.nabble.com/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: Does wal-file support MMAP?

Hick Gunter
I take it you have already fine tuned your database schema and statements to utilize optimum query plans, which may cut orders of magnitude from execution times and found the performance to still be lacking.

-----Ursprüngliche Nachricht-----
Von: sqlite-users [mailto:[hidden email]] Im Auftrag von advancenOO
Gesendet: Freitag, 10. November 2017 09:49
An: [hidden email]
Betreff: [EXTERNAL] Re: [sqlite] Does wal-file support MMAP?


Purely for speed :)
I hope to improve the performance inside sqlite in order to optimize the performance of the upper applications. As there are so many different scenes in APPs.




--
Sent from: http://sqlite.1065341.n5.nabble.com/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___________________________________________
 Gunter Hick | Software Engineer | Scientific Games International GmbH | Klitschgasse 2-4, A-1130 Vienna | FN 157284 a, HG Wien, DVR: 0430013 | (O) +43 1 80100 - 0

May be privileged. May be confidential. Please delete if not the addressee.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Does wal-file support MMAP?

Simon Slavin-3
In reply to this post by advancenOO


On 10 Nov 2017, at 8:49am, advancenOO <[hidden email]> wrote:

> Purely for speed :)
> I hope to improve the performance inside sqlite in order to optimize the
> performance of the upper applications. As there are so many different scenes
> in APPs.

hAve you optimised your column orders ?

Have you created ideal indexes for the WHERE and ORDER BY clauses you use ?

Have you made use of covering indexes ?

Both those things speed up SQLite many times more than using memory mapping.

Have you tried SQLite just as it is, without putting special effort into PRAMGAs ?  SQLite is extremely fast without those things.  There’s a good chance it’s fast enough and you’re just wasting your time.

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Does wal-file support MMAP?

Howard Kapustein
On 10 Nov 2017, at 8:49am, advancenOO <[hidden email]> wrote:

>hAve you optimised your column orders ?
What is optimal?

    - Howard
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Does wal-file support MMAP?

Simon Slavin-3


On 19 Nov 2017, at 6:15am, Howard Kapustein <[hidden email]> wrote:

> On 10 Nov 2017, at 8:49am, advancenOO <[hidden email]> wrote:
>
>> hAve you optimised your column orders ?
> What is optimal?

SQLite reads only up to the last column it needs to execute the command.  So you put your most frequently used columns at the beginning of the table definition.

Have you created indexes ideally suited to your WHERE and ORDER BY clauses ?

We see a lot of people here asking about obscure complicated features of how SQLite works who haven’t done basic optimization things that apply to all SQL engines.  That’s what’s behind the questions advancenOO and I asked.

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users