Make some changes to the source code

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Make some changes to the source code

Nick
I am trying to make some changes to the source code of sqlite. As I found
there will be a little improvement if I support MMAP to wal file. I guess
the optimization is micro but it is useful in my test and this is a good way
for me to study the code of sqlite :D

1.wal+mmap
I could use unixMapfile() to map the wal file while MMAP may cause SIGBUS if
the mapped file is truncated. This could happen when reseting the wal file,
in another word, if journal_size_limit is reached or
SQLITE_CHECKPOINT_TRUNCATE is called. But I guess it works if these two APIs
will always not be called in my application.
So, I want to create file holes to get a 4M wal-file in sqlite3WalOpen(),
and always set journal_size_limit to 4M. Then mmap will be supported by
simply calling unixMapfile(4M) in sqlite3WalOpen(). After that, memcpy()
instead of read() will be used when read the first 4M of wal file.
I am wondering if it is all right in my Android applications?

2.Further more. I know mmap is supported when fetching db file:
To map file:
In getPageMMap(), sqlite3OsFetch() MMAPs the whole db file, and return the
mapped page through *pData. Then pagerAcquireMapPage will obtain a page
reference PgHdr based on the pData.
(A small question here, why pData is needed? As xRead() will always use
memcpy instead of read() after unixMapfile(-1) is called.)

sqlite3OsFileControlHint is called to remap the db file when the db grows as
a result of a checkpoint.

To avoid SIGBUS:
Process will catch the CHANGE of other processes by comparing
pWal->hdr.iChange and the corresponding number in wal-index. Whenever a
read, write or checkpoint operation happens, unixUnmapfile() will be called
if there is a CHANGE.

3.Thus another way of wal+mmap:
I want to use pWal->hdr.unused to catch the CHANGE when other process
truncate the wal file(journal_size_limit or SQLITE_CHECKPOINT_TRUNCATE).
Then I will check the hdr.unused to call unixMapfile(-1) before whenever
sqlite3OsRead(pWal->pWalFd) is called.
Is there a better timing to remap the file? Just like
sqlite3WalBeginReadTransaction and walcheckpoint in db+mmap;

I run sqlite test to check my code, but I find pVfs->szOsFile is 16 when
test_vfs.c is called, which means pRet->pWalFd is no longer a unixFile
struct. At this time, sqlite3OsOpen() binds to tvfsOpen() instead of
unixOpen(). So I cannot use unixMapfile() and the test that uses test_vfs.c
will not pass. So could you give me some advices to pass the test?

Thank you.




--
Sent from: http://sqlite.1065341.n5.nabble.com/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Make some changes to the source code

Simon Slavin-3
On 24 Jan 2018, at 11:35am, Nick <[hidden email]> wrote:

> As I found
> there will be a little improvement if I support MMAP to wal file.

Which OS are you using ?  The developers of SQLite have previously had corruption problems with memory-mapping under some versions of Windows and macOS.  Corruption occurs rarely, and it may appear that your system works fine during testing.

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Make some changes to the source code

Nick
I use sqlite in my Android application. And I tried to run sqlite test on my
MacOS PC. Some cases failed but I can not figure out it is indeed a
corruption.

Do you mean the corruption problems you mentioned will happen in db+mmap? I
guess it should happen in both wal+mmap and db+mmap if it exists. But I have
not found it until now even though I have heard about the mmap+OS problem
from the community.

And Simon, do you have any idea about the test_vfs problem? And is it OK to
change the code as 1.wal+mmap mentioned besides the mmap problem?



--
Sent from: http://sqlite.1065341.n5.nabble.com/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Make some changes to the source code

Nick
In reply to this post by Simon Slavin-3
I ran tests in my MacOS 10.12.6.



--
Sent from: http://sqlite.1065341.n5.nabble.com/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Make some changes to the source code

Simon Slavin-3
In reply to this post by Nick


On 25 Jan 2018, at 1:34am, Nick <[hidden email]> wrote:

> Do you mean the corruption problems you mentioned will happen in db+mmap?

When memory-mapping was used for all file access of SQLite, some users reported that they occasionally got corrupted databases.  When memory mapping was turned off, the problem went away.  There was no error in the source code built into SQLite.  The fault was in the operating system’s implementation of memory mapping.  That’s all I remember.

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Make some changes to the source code

Nick
Yup, I guess I understand you correctly about the mmap problem in the OS. I
have seen some threads about it before.
But I think wal+mmap is still a worthy consideration as db+mmap has already
been supported even though it is disabled by default. At least I think I
could use it in my own application until I find the mmap problem in my
system.

The one thing that bothers me the most is that I have no way to check my
code, as there is a testvfs in sqlite test. So could you please review my
train of thought about my wal+mmap? (Map file when opening it and do not
truncate the file)

By the way, is there a possibly way to submit patch to sqlite?

Thank you Simon.



--
Sent from: http://sqlite.1065341.n5.nabble.com/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Make some changes to the source code

Simon Slavin-3
On 25 Jan 2018, at 5:55am, Nick <[hidden email]> wrote:

> The one thing that bothers me the most is that I have no way to check my
> code, as there is a testvfs in sqlite test. So could you please review my
> train of thought about my wal+mmap? (Map file when opening it and do not
> truncate the file)

That is way beyond my area of competence.  I’m not a good C programmer.  But there are other people on this list who are.

> By the way, is there a possibly way to submit patch to sqlite?

Although SQLite is 'open source' it's not your typical open source project.  Almost all the code in SQLite was written by a small development team.  The team tends to accept development ideas from the outside (e.g. this list) and write the code themselves rather than to accept code written outside the team.  This allows them to enforce a consistent programming style and to ensure code is not copied from other projects under licence.  In fact it's a problem if you show them your code since they then might be accused of copying it.  (Not to accuse you personally of copying code, I'm just layout out the general principle.)

There’s also a requirement of usefulness vs. code size.  One of SQLite's jealously-guarded advantages is that it's tiny.  It has to fit in embedded processors, on phones, and in various hand-held devices with very limited space.  SQLite doesn’t add size to the project just because it’s easy, or because someone wrote the code and it works.  It adds it because it would be useful to a lot of users.  I've had an SQL-conformance change neglected because even though SQLite would conform more closely to SQL it wouldn't allow users to do anything more.

If you find your new facility useful then you can use it yourself, by maintaining your own fork.  But a minor speed improvement for a single platform, dependent on an OS facility, may not get integrated into the main projects.

Sorry if I have disappointed you.

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Make some changes to the source code

Nick
Thank you Simon, I totally understand you.

And still hope for someone to give me some advice about my wal+mmap (Map
file when opening it and do not truncate the file) .

Thanks.



--
Sent from: http://sqlite.1065341.n5.nabble.com/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users