Hot-Journal with VFS

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Hot-Journal with VFS

Bob schwanzer
Hi,
 I'm seeing hot journal frequently in our SQLite installation. We use VFS
which does some checksum and other operations. There are no abnormal
scenarios such as power off, process crash, abandoned transactions... DB is
opened by 10-20 processes each of which can have multiple threads.

Old version of SQLite: 3.7.13
Process A is in write mode. Process B which is reading a value. Process B
used to sees hot-journal.
Now, we upgraded to 3.9.2
Process A is in write mode. Process B is trying to read a value. B  sees a
hot-journal and recovers it.
Process A will get error introduced in new new package: "The xDelete method
in the built-in VFSes now return SQLITE_IOERR_DELETE_NOENT if the file to be
deleted does not exist"

Few things stand out:
Hot journal was reported in 3.7.13. There are no logs to indicate it's
recovery.
Hot journal is seen in 3.9.2. There are logs to indicate it's rollback.
Committer Process(Process A) gets error.

I know some optimizations in HardDisk or OS is creating the problem. What
other layers could contribute to this anomaly? How could installation endup
with Hot journals without system experiencing crash or power loss.









--
Sent from: http://sqlite.1065341.n5.nabble.com/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Hot-Journal with VFS

Simon Slavin-3
On 18 Oct 2018, at 3:27pm, Bob schwanzer <[hidden email]> wrote:

> DB is
> opened by 10-20 processes each of which can have multiple threads.

What OS are you using ?
What programming language are you using ?
Are you calling the SQLite C library directly or using a shim ?

Does your program close each database as it is finished with it ?
Does the result code from _close() come back as SQLITE_OK ?

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Hot-Journal with VFS

Dan Kennedy-4
In reply to this post by Bob schwanzer
On 10/18/2018 09:27 PM, Bob schwanzer wrote:

> Hi,
>  I'm seeing hot journal frequently in our SQLite installation. We use VFS
> which does some checksum and other operations. There are no abnormal
> scenarios such as power off, process crash, abandoned transactions... DB is
> opened by 10-20 processes each of which can have multiple threads.
>
> Old version of SQLite: 3.7.13
> Process A is in write mode. Process B which is reading a value. Process B
> used to sees hot-journal.
> Now, we upgraded to 3.9.2
> Process A is in write mode. Process B is trying to read a value. B  sees a
> hot-journal and recovers it.
> Process A will get error introduced in new new package: "The xDelete method
> in the built-in VFSes now return SQLITE_IOERR_DELETE_NOENT if the file to be
> deleted does not exist"
>
> Few things stand out:
> Hot journal was reported in 3.7.13. There are no logs to indicate it's
> recovery.
> Hot journal is seen in 3.9.2. There are logs to indicate it's rollback.
> Committer Process(Process A) gets error.
>
> I know some optimizations in HardDisk or OS is creating the problem. What
> other layers could contribute to this anomaly?

Sounds like the locking function (sqlite3_io_methods.xLock callback) is
broken. Process A should be holding at least RESERVED on the db file
while creating/using a journal file as part of a write transaction. And
process B requires EXCLUSIVE to rollback and delete the journal. Those
two are supposed to be mutually exclusive:

   https://www.sqlite.org/lockingv3.html

A hunch: If you're (a) using unix, and open()ing and close()ing any file
descriptors on SQLite databases, this POSIX bug may be tripping you up
somehow:

 
https://www.sqlite.org/howtocorrupt.html#_posix_advisory_locks_canceled_by_a_separate_thread_doing_close_



Dan.




How could installation endup

> with Hot journals without system experiencing crash or power loss.
>
>
>
>
>
>
>
>
>
> --
> Sent from: http://sqlite.1065341.n5.nabble.com/
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users