[Question] Non-EXCLUSIVE modes in dot-file locking strategy

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[Question] Non-EXCLUSIVE modes in dot-file locking strategy

Pierre Tempel
Hi,

I'm currently working with a system where dot-file is the only applicable
SQLITE_ENABLE_LOCKING_STYLE and I'm trying to understand how this is
implemented exactly. So I've read through the implementation of this
locking strategy and a few questions came to mind:

First, dot-file actually uses a directory, however, the comments don't seem
to give a reason for this. Why is a directory created, instead of a more
(arguably) traditional lock file (see e.g. git)? dotlockLock creates the
directory with permissions set to 777. As far as I can tell, this could
create a situation where the temporary locking directory is more accessible
than the database file or the db parent directory. Is there a reason the
existing permissions aren't inherited? I suspect this is done to prevent
any issues while deleting the lock directory.

Lastly, it seems that all lock types are promoted to EXCLUSIVE with this
strategy. I wonder if that can be changed to allow all previous lock types,
maybe using lock *files* and judging the lock type by content or file name?
In my mind, this would create a completely portable, file-based locking
strategy that doesn't depend on OS-provided locking mechanisms. But I'm
probably overlooking something here, so please correct me if I'm wrong.

Thanks,
--
*Pierre Tempel*
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: [Question] Non-EXCLUSIVE modes in dot-file locking strategy

Richard Hipp-3
On 12/15/18, Pierre Tempel <[hidden email]> wrote:
>
> First, dot-file actually uses a directory, however, the comments don't seem
> to give a reason for this. Why is a directory created, instead of a more
> (arguably) traditional lock file (see e.g. git)? dotlockLock creates the
> directory with permissions set to 777. As far as I can tell, this could
> create a situation where the temporary locking directory is more accessible
> than the database file or the db parent directory. Is there a reason the
> existing permissions aren't inherited? I suspect this is done to prevent
> any issues while deleting the lock directory.

IIRC, this is because creating a directory is an atomic operation on
NFS, but creating a file is not.

>
> Lastly, it seems that all lock types are promoted to EXCLUSIVE with this
> strategy. I wonder if that can be changed to allow all previous lock types,
> maybe using lock *files* and judging the lock type by content or file name?
> In my mind, this would create a completely portable, file-based locking
> strategy that doesn't depend on OS-provided locking mechanisms. But I'm
> probably overlooking something here, so please correct me if I'm wrong.
>

No.  Unfortunately, if all you have is the presence or absence of a
directory to determine if the file is locked, then that is only one
bit of information.  And there is no way to encode all the other
locking states in a single bit.

--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: [Question] Non-EXCLUSIVE modes in dot-file locking strategy

Pierre Tempel
In reply to this post by Pierre Tempel
> No. Unfortunately, if all you have is the presence or absence of a
> directory to determine if the file is locked, then that is only one
> bit of information. And there is no way to encode all the other
> locking states in a single bit.

Hm. Doesn't this mean the proposed solution still works, but with
directories instead of files for the sake of atomicity? I.e. for each
lock type, create a unique dir name (e.g. ".1.lock") and stat for all
lock types when changing levels?
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: [Question] Non-EXCLUSIVE modes in dot-file locking strategy

Richard Hipp-3
On 12/15/18, Pierre Tempel <[hidden email]> wrote:
>
> Hm. Doesn't this mean the proposed solution still works, but with
> directories instead of files for the sake of atomicity? I.e. for each
> lock type, create a unique dir name (e.g. ".1.lock") and stat for all
> lock types when changing levels?

I think you probably need a counter in order to implement a
reader/writer lock, don't you?

We used to do a stocastic simulation reader/writer locks on win98 by
using 512 separate locking bytes.  But is seems a little excessive to
have 512 separate locking directories.

Does anybody else know how to implement reader/writer locks using lock files?

--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: [Question] Non-EXCLUSIVE modes in dot-file locking strategy

Pierre Tempel
It might be worth investigating some other projects that use lock
files. Git, as mentioned locks the index with a lock file, but AFAIK
without counting. As a reference for the NFS issue, see

https://dwheeler.com/secure-programs/Secure-Programs-HOWTO/avoid-race.html

section 7.11.2.1, which says:

> Second, if the lock file may be on an NFS-mounted filesystem, then
> you have the problem that NFS version 2 doesn’t completely support
> normal file semantics. This can even be a problem for work that’s
> supposed to be “local” to a client, since some clients don’t
> have local disks and may have all files remotely mounted via NFS. The
> manual for open(2) explains how to handle things in this case (which
> also handles the case of root programs):
>
> “... programs which rely on [the O_CREAT and O_EXCL flags of
> open(2) to work on filesystems accessed via NFS version 2] for
> performing locking tasks will contain a race condition. The solution
> for performing atomic file locking using a lockfile is to create a
> unique file on the same filesystem (e.g., incorporating hostname and
> pid), use link(2) to make a link to the lockfile and use stat(2) on
> the unique file to check if its link count has increased to 2. Do not
> use the return value of the link(2) call.”
>

The question is if this applies to current NFS versions (3, 4).
However, If I read this correctly, the link count strategy can be used
to encode more info. However, this again depends on OS-specific
extended attributes, which is contrary to the general file-system based
locking I was looking to create.

As an aside, I don't quite understand why and how a reader/writer lock
requires/uses a counter. I'd appreciate a link to any documentation
about that.

---
Pierre Tempel
On Sa, Dez 15, 2018 at 9:43 , Richard Hipp <[hidden email]> wrote:

> On 12/15/18, Pierre Tempel <[hidden email]> wrote:
>>
>>  Hm. Doesn't this mean the proposed solution still works, but with
>>  directories instead of files for the sake of atomicity? I.e. for
>> each
>>  lock type, create a unique dir name (e.g. ".1.lock") and stat for
>> all
>>  lock types when changing levels?
>
> I think you probably need a counter in order to implement a
> reader/writer lock, don't you?
>
> We used to do a stocastic simulation reader/writer locks on win98 by
> using 512 separate locking bytes.  But is seems a little excessive to
> have 512 separate locking directories.
>
> Does anybody else know how to implement reader/writer locks using
> lock files?
>
> --
> D. Richard Hipp
> [hidden email]
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users



_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: [Question] Non-EXCLUSIVE modes in dot-file locking strategy

Peter da Silva-2
The only way I can think of to have a large counter in a lock is to lock
access to a file containing a counter and I don't know if not-a-file-system
semantics support that.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: [Question] Non-EXCLUSIVE modes in dot-file locking strategy

Rowan Worth-2
In reply to this post by Pierre Tempel
On Sun, 16 Dec 2018 at 05:00, Pierre Tempel <[hidden email]> wrote:

> > “... programs which rely on [the O_CREAT and O_EXCL flags of
> > open(2) to work on filesystems accessed via NFS version 2] for
> > performing locking tasks will contain a race condition. The solution
> > for performing atomic file locking using a lockfile is to create a
> > unique file on the same filesystem (e.g., incorporating hostname and
> > pid), use link(2) to make a link to the lockfile and use stat(2) on
> > the unique file to check if its link count has increased to 2. Do not
> > use the return value of the link(2) call.”
> >
>
> The question is if this applies to current NFS versions (3, 4).
> However, If I read this correctly, the link count strategy can be used
> to encode more info. However, this again depends on OS-specific
> extended attributes, which is contrary to the general file-system based
> locking I was looking to create.
>
> As an aside, I don't quite understand why and how a reader/writer lock
> requires/uses a counter. I'd appreciate a link to any documentation
> about that.
>

For a reader/writer lock, the usual semantics allow a single writer but
arbitrarily many readers. However we're in a situation where we don't have
actual locking primitives. ie. each reader will need its own lockfile -
they can't share one - and this the problem a counter addresses (by
providing unique files).

Of course it also creates another problem, as now how do you atomically
check whether anyone has a read lock? Or prevent two processes
simultaneously creating a read-lockfile and a write-lockfile?

Encoding information into the attributes of a common lockfile seems to
suffer similar race conditions, but lets say there's a way to do it. Now
one process comes and starts a transaction; obtains a read lock. While it's
busy a second process comes and also acquires a read lock. The second query
finishes really quick and goes to relinquish its read lock. However it does
this, it must leave the overall database in a read-locked state, because
the first process is still busy! How would you achieve that without a
counter or equivalent mechanism?

-Rowan
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users