Parallel reading can be slow on APFS

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Parallel reading can be slow on APFS

Simon Slavin-3
This post is about a problem with Apple's new APFS file system.  The problem will affect you only if you have multiple reads/writes happening at the same time.  The problem involves merely slowing of performance, not corruption of databases or incorrect answers being returned by SQLite.

Gregory Szorc, expert on Mercurial (revision control and tracking software) investigated a report that Mercurial was performing slowly on new Macs, and deduced that the cause was that APFS uses a kernel-level global lock, not only on writes (expected) but also on reads when no writing was happening (unexpected).  Technical details can be found here:

<https://gregoryszorc.com/blog/2018/10/29/global-kernel-locks-in-apfs/>

The more parallel operations are trying to access the storage device, the slower it gets.

This post is intended to urge users to avoid premature optimization by parallel processing, and to consider that slow performance may not be SQLite's fault.  This is not a "Macs suck" post.  Please consider the following text from the article (which comes with relevant links):

"While this post is about APFS, this issue of global kernel locks during common I/O operations is not unique to APFS. I already referenced similar issues in AUFS. And I've encountered similar behaviors with Btrfs (although I can't recall exactly which operations). And NTFS has its own bag of problems."

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Parallel reading can be slow on APFS

Darren Duncan
I find it strange that there's a big global lock problem with APFS considering
one of the biggest design and selling points was that APFS did NOT use global
locks like the old HFS+ did. -- Darren Duncan

On 2018-10-29 7:13 PM, Simon Slavin wrote:

> This post is about a problem with Apple's new APFS file system.  The problem will affect you only if you have multiple reads/writes happening at the same time.  The problem involves merely slowing of performance, not corruption of databases or incorrect answers being returned by SQLite.
>
> Gregory Szorc, expert on Mercurial (revision control and tracking software) investigated a report that Mercurial was performing slowly on new Macs, and deduced that the cause was that APFS uses a kernel-level global lock, not only on writes (expected) but also on reads when no writing was happening (unexpected).  Technical details can be found here:
>
> <https://gregoryszorc.com/blog/2018/10/29/global-kernel-locks-in-apfs/>
>
> The more parallel operations are trying to access the storage device, the slower it gets.
>
> This post is intended to urge users to avoid premature optimization by parallel processing, and to consider that slow performance may not be SQLite's fault.  This is not a "Macs suck" post.  Please consider the following text from the article (which comes with relevant links):
>
> "While this post is about APFS, this issue of global kernel locks during common I/O operations is not unique to APFS. I already referenced similar issues in AUFS. And I've encountered similar behaviors with Btrfs (although I can't recall exactly which operations). And NTFS has its own bag of problems."

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Parallel reading can be slow on APFS

Rowan Worth-2
In reply to this post by Simon Slavin-3
"The problem will affect you only if you have multiple reads/writes
happening at the same time."

ie. The problem will only manifest if the user is doing anything at all
with their computer? :P

Interesting analysis - thanks for sharing.
-Rowan



On Tue, 30 Oct 2018 at 10:13, Simon Slavin <[hidden email]> wrote:

> This post is about a problem with Apple's new APFS file system.  The
> problem will affect you only if you have multiple reads/writes happening at
> the same time.  The problem involves merely slowing of performance, not
> corruption of databases or incorrect answers being returned by SQLite.
>
> Gregory Szorc, expert on Mercurial (revision control and tracking
> software) investigated a report that Mercurial was performing slowly on new
> Macs, and deduced that the cause was that APFS uses a kernel-level global
> lock, not only on writes (expected) but also on reads when no writing was
> happening (unexpected).  Technical details can be found here:
>
> <https://gregoryszorc.com/blog/2018/10/29/global-kernel-locks-in-apfs/>
>
> The more parallel operations are trying to access the storage device, the
> slower it gets.
>
> This post is intended to urge users to avoid premature optimization by
> parallel processing, and to consider that slow performance may not be
> SQLite's fault.  This is not a "Macs suck" post.  Please consider the
> following text from the article (which comes with relevant links):
>
> "While this post is about APFS, this issue of global kernel locks during
> common I/O operations is not unique to APFS. I already referenced similar
> issues in AUFS. And I've encountered similar behaviors with Btrfs (although
> I can't recall exactly which operations). And NTFS has its own bag of
> problems."
>
> Simon.
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Parallel reading can be slow on APFS

Jens Alfke-2
In reply to this post by Simon Slavin-3


> On Oct 29, 2018, at 7:13 PM, Simon Slavin <[hidden email]> wrote:
>
> This post is about a problem with Apple's new APFS file system.  The problem will affect you only if you have multiple reads/writes happening at the same time.  The problem involves merely slowing of performance, not corruption of databases or incorrect answers being returned by SQLite.

I finally got around to reading the article — thanks for posting the link, Simon.

However, I don’t think the issue described in the article is relevant to SQLite at all. The article is specifically about slowdowns in the readdir system call, not file I/O. ("I haven't conducted a comprehensive analysis of APFS to determine what other filesystem operations seem to acquire global kernel locks: all I know is readdir() does.”) But there are no calls to readdir in the SQLite 3.25 source code!

Second, it’s good to see that the situation has been improved in macOS 10.14. "It is apparent that macOS 10.14 Mojave has received performance work relative to macOS 10.13! Overall kernel CPU time when performing parallel directory walks has decreased substantially - to ~50% of original on some invocations!” However, "Despite those improvements, APFS is still spending a lot of CPU time in the kernel.”

—Jens
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users