Linux top command and sqlite

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Linux top command and sqlite

Kevin O'Gorman
I'm not at all sure this is the right place to ask, but as it only comes up
when I'm running one of
my sqlite jobs, I thought I'd start here.  I'm running Python 3.5 scripts
in Linux 16.04.1 using the sqlite3 package.  Machine is Core i5, 32GB RAM.

Some of my stuff takes a while to run, and I like to keep tabs on it.
Right now, I'm running one of those, and the Linux top command shows
extremely small CPU usage, and a status ("S" column) of "D" which the man
page defines as "uninterruptable sleep".  Huh?

My code does not use an intentional sleep at all.  It's traversing parts of
a graph, and not interacting with anything else.  As far as I know, anyway,
and I wrote the whole thing.

Now it's true that it's updating the consequences of changes to graph nodes
that took about 3 hours just to install, so taking a while is not a
surprise by itself.  I just wonder what that status means and how it could
arise.

--
word of the year: *kakistocracy*
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Linux top command and sqlite

Richard Hipp-3
On 2/21/17, Kevin O'Gorman <[hidden email]> wrote:
> I'm not at all sure this is the right place to ask, but as it only comes up
> when I'm running one of
> my sqlite jobs, I thought I'd start here.  I'm running Python 3.5 scripts
> in Linux 16.04.1 using the sqlite3 package.  Machine is Core i5, 32GB RAM.
>
> Some of my stuff takes a while to run, and I like to keep tabs on it.
> Right now, I'm running one of those, and the Linux top command shows
> extremely small CPU usage, and a status ("S" column) of "D" which the man
> page defines as "uninterruptable sleep".  Huh?

My guess:  It is busying doing an fsync() after a transaction commit.
To find out, temporarily set "PRAGMA synchronous=off" in your script
and see if that makes the pauses go away.

A better long-term solution would be:

(1) Group multiple changes into a single transaction using BEGIN...COMMIT.
(2) Set PRAGMA journal_mode=WAL with PRAGMA synchronous=NORMAL.


>
> My code does not use an intentional sleep at all.  It's traversing parts of
> a graph, and not interacting with anything else.  As far as I know, anyway,
> and I wrote the whole thing.
>
> Now it's true that it's updating the consequences of changes to graph nodes
> that took about 3 hours just to install, so taking a while is not a
> surprise by itself.  I just wonder what that status means and how it could
> arise.
>
> --
> word of the year: *kakistocracy*
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>


--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Linux top command and sqlite

Kevin O'Gorman
I'll try the synchronous=off the next time I run it.  As it happens, this
one
finished in 57 minutes, which is not bad, considering.

But, I wonder, could i not tell if it's doing lots of commits by looking at
the size of the ?.db-journal file?  It gets pretty large.


On Tue, Feb 21, 2017 at 10:38 AM, Richard Hipp <[hidden email]> wrote:

> On 2/21/17, Kevin O'Gorman <[hidden email]> wrote:
> > I'm not at all sure this is the right place to ask, but as it only comes
> up
> > when I'm running one of
> > my sqlite jobs, I thought I'd start here.  I'm running Python 3.5 scripts
> > in Linux 16.04.1 using the sqlite3 package.  Machine is Core i5, 32GB
> RAM.
> >
> > Some of my stuff takes a while to run, and I like to keep tabs on it.
> > Right now, I'm running one of those, and the Linux top command shows
> > extremely small CPU usage, and a status ("S" column) of "D" which the man
> > page defines as "uninterruptable sleep".  Huh?
>
> My guess:  It is busying doing an fsync() after a transaction commit.
> To find out, temporarily set "PRAGMA synchronous=off" in your script
> and see if that makes the pauses go away.
>
> A better long-term solution would be:
>
> (1) Group multiple changes into a single transaction using BEGIN...COMMIT.
> (2) Set PRAGMA journal_mode=WAL with PRAGMA synchronous=NORMAL.
>
>
> >
> > My code does not use an intentional sleep at all.  It's traversing parts
> of
> > a graph, and not interacting with anything else.  As far as I know,
> anyway,
> > and I wrote the whole thing.
> >
> > Now it's true that it's updating the consequences of changes to graph
> nodes
> > that took about 3 hours just to install, so taking a while is not a
> > surprise by itself.  I just wonder what that status means and how it
> could
> > arise.
> >
> > --
> > word of the year: *kakistocracy*
> > _______________________________________________
> > sqlite-users mailing list
> > [hidden email]
> > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
> >
>
>
> --
> D. Richard Hipp
> [hidden email]
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>



--
word of the year: *kakistocracy*
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Linux top command and sqlite

Roger Binns
In reply to this post by Kevin O'Gorman
On 21/02/17 10:22, Kevin O'Gorman wrote:
> Some of my stuff takes a while to run, and I like to keep tabs on it.
> Right now, I'm running one of those, and the Linux top command shows
> extremely small CPU usage, and a status ("S" column) of "D" which the man
> page defines as "uninterruptable sleep".  Huh?

In this case you can read D as waiting on disk.

Unless you are using multithreading, each query that needs to get data
not already in cache will result in disk activity.  And the code will
not continue until it gets the data from the disk.  The waiting is why
CPU usage is low.

You can use multithreading to allow more concurrent disk I/O although
Python's GIL complicates matters.  You can also tune the caches used by
SQLite.  If your database is smaller than your RAM then you can
prepopulate the OS cache with the database file contents.

Roger


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

signature.asc (169 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Linux top command and sqlite

Graham Holden
In reply to this post by Kevin O'Gorman


-------- Original message --------From: Roger Binns <[hidden email]> Date: 21/02/2017  20:48  (GMT+00:00) To: SQLite mailing list <[hidden email]> Subject: Re: [sqlite] Linux top command and sqlite
On 21/02/17 10:22, Kevin O'Gorman wrote:
> Some of my stuff takes a while to run, and I like to keep tabs on it.
> Right now, I'm running one of those, and the Linux top command shows
> extremely small CPU usage, and a status ("S" column) of "D" which the man
> page defines as "uninterruptable sleep".  Huh?

In this case you can read D as waiting on disk.

<snip>
According to this answer on StackOverflow:  http://stackoverflow.com/a/6685764/2096401 spotting a "D" is relatively rare, although can be more common if NFS or similar is involved. No idea whether that's true or not, nor whether it applies to the OP.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Linux top command and sqlite

Hick Gunter
In reply to this post by Kevin O'Gorman
Definitely 'D' as in "Disk I/O", if you type "1" while runing top, the results for each CPU are displayed separately und you will find at least 1 line (not necessarily always the same line) with a significant percentage of "wa" (I/O wait state).

Apparently your transactions involve very much more data changes than computations. A common reason ist hat you are running. each. change. in. a. separate. transaction. As suggested, try grouping several (the optimal size typically ranges from 100s to 1000s) small updates into a single transaction so that all the changes hit the disk together instead of separately.

-----Ursprüngliche Nachricht-----
Von: sqlite-users [mailto:[hidden email]] Im Auftrag von Kevin O'Gorman
Gesendet: Dienstag, 21. Februar 2017 19:22
An: sqlite-users <[hidden email]>
Betreff: [sqlite] Linux top command and sqlite

I'm not at all sure this is the right place to ask, but as it only comes up when I'm running one of my sqlite jobs, I thought I'd start here.  I'm running Python 3.5 scripts in Linux 16.04.1 using the sqlite3 package.  Machine is Core i5, 32GB RAM.

Some of my stuff takes a while to run, and I like to keep tabs on it.
Right now, I'm running one of those, and the Linux top command shows extremely small CPU usage, and a status ("S" column) of "D" which the man page defines as "uninterruptable sleep".  Huh?

My code does not use an intentional sleep at all.  It's traversing parts of a graph, and not interacting with anything else.  As far as I know, anyway, and I wrote the whole thing.

Now it's true that it's updating the consequences of changes to graph nodes that took about 3 hours just to install, so taking a while is not a surprise by itself.  I just wonder what that status means and how it could arise.

--
word of the year: *kakistocracy*
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___________________________________________
 Gunter Hick
Software Engineer
Scientific Games International GmbH
FN 157284 a, HG Wien
Klitschgasse 2-4, A-1130 Vienna, Austria
Tel: +43 1 80100 0
E-Mail: [hidden email]

This communication (including any attachments) is intended for the use of the intended recipient(s) only and may contain information that is confidential, privileged or legally protected. Any unauthorized use or dissemination of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender by return e-mail message and delete all copies of the original communication. Thank you for your cooperation.


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users