new Error database disk image is malformed

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

new Error database disk image is malformed

Ron Barnes
Hi All,

I keep generating this error and I can't figure out why.  I have deleted and re-created the database but it keeps popping up.

Error....

database disk image is malformed


Any ideas why?

-Ron


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: new Error database disk image is malformed

R Smith-2

On 2018/01/04 9:49 PM, Ron Barnes wrote:

> Hi All,
>
> I keep generating this error and I can't figure out why.  I have deleted and re-created the database but it keeps popping up.
>
> Error....
>
> database disk image is malformed
>
>
> Any ideas why?

This is a wild guess, but I'm thinking it's because the disk image is
malformed...? :)

More seriously, what you probably meant to ask is: "Any idea why my file
gets corrupt in this way?"

To which the best answer is: No idea, but here are some things that are
typical culprits:
https://sqlite.org/howtocorrupt.html

Once you've checked all those, and you still are not sure what could
cause the corruption, please write again but then include all
information, OS, storage media, file system, use case, etc. and likely
someone here would have had similar experience or run a similar system
(or perhaps see some other obvious problem).

Good luck!
Ryan


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: new Error database disk image is malformed

Jens Alfke-2
In reply to this post by Ron Barnes
(1) Try opening & querying the database file using the `sqlite3` tool, to verify whether it’s actually corrupted.
(2) If it is, check the documentation page “How To Corrupt A SQLite Database” and see if you’re making any of those mistakes.

—Jens
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: new Error database disk image is malformed

Ron Barnes
In reply to this post by R Smith-2
Thank you - I will and report back!

-----Original Message-----
From: sqlite-users [mailto:[hidden email]] On Behalf Of R Smith
Sent: Thursday, January 4, 2018 3:14 PM
To: [hidden email]
Subject: Re: [sqlite] new Error database disk image is malformed


On 2018/01/04 9:49 PM, Ron Barnes wrote:

> Hi All,
>
> I keep generating this error and I can't figure out why.  I have deleted and re-created the database but it keeps popping up.
>
> Error....
>
> database disk image is malformed
>
>
> Any ideas why?

This is a wild guess, but I'm thinking it's because the disk image is malformed...? :)

More seriously, what you probably meant to ask is: "Any idea why my file gets corrupt in this way?"

To which the best answer is: No idea, but here are some things that are typical culprits:
https://sqlite.org/howtocorrupt.html

Once you've checked all those, and you still are not sure what could cause the corruption, please write again but then include all information, OS, storage media, file system, use case, etc. and likely someone here would have had similar experience or run a similar system (or perhaps see some other obvious problem).

Good luck!
Ryan


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: new Error database disk image is malformed

Ron Barnes
In reply to this post by R Smith-2
Hi,

Ok. I've read the Document and I think I may be having a rouge thread issue.  I hope not since I use synclock in my code when ever a thread is attempting a write to the database.  That seems like the only issue from that page that I may be doing.  I could have up to 30 or more threads reading from the DB but only one to three active threads writing.  The way my code is set up, though is that each thread would have to wait for the previous thread to finish writing before its turn to write.  

Each process works as follows...

Gather information.
Format the information

Enter the write routine using synclock (Visual Basic)

Write the record
Exit the write routine.

I'm using Visual Studio 2015
Visual Basic
Samsung 2TB SSD
64GB RAM
8 Core AMD Processor

Could any of the 30 or so concurrent reads be messing up my writes?

Here is my Create string...

Dim CreateNewDBConString As String = "Data Source=" + Trim(MyPath) + "\LoAndBehold.DB3;Version=3;New=True;Max Page Count=10485760;Compress=True;journal_mode=WAL;"

Here is my read/write string...

Dim Myconstring As String = "Data Source=" + Trim(MyPath) + "\LoAndBehold.DB3;Version=3;New=False;Max Page Count=10485760;Compress=True;journal_mode=WAL;"


Regards,

-Ron

-----Original Message-----
From: sqlite-users [mailto:[hidden email]] On Behalf Of R Smith
Sent: Thursday, January 4, 2018 3:14 PM
To: [hidden email]
Subject: Re: [sqlite] new Error database disk image is malformed


On 2018/01/04 9:49 PM, Ron Barnes wrote:

> Hi All,
>
> I keep generating this error and I can't figure out why.  I have deleted and re-created the database but it keeps popping up.
>
> Error....
>
> database disk image is malformed
>
>
> Any ideas why?

This is a wild guess, but I'm thinking it's because the disk image is malformed...? :)

More seriously, what you probably meant to ask is: "Any idea why my file gets corrupt in this way?"

To which the best answer is: No idea, but here are some things that are typical culprits:
https://sqlite.org/howtocorrupt.html

Once you've checked all those, and you still are not sure what could cause the corruption, please write again but then include all information, OS, storage media, file system, use case, etc. and likely someone here would have had similar experience or run a similar system (or perhaps see some other obvious problem).

Good luck!
Ryan


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: new Error database disk image is malformed

Jens Alfke-2


> On Jan 4, 2018, at 4:51 PM, Ron Barnes <[hidden email]> wrote:
>
> I hope not since I use synclock in my code when ever a thread is attempting a write to the database.  That seems like the only issue from that page that I may be doing.  I could have up to 30 or more threads reading from the DB but only one to three active threads writing.

It depends on the value of SQLITE_THREADSAFE that SQLite was compiled with. (Check the docs for details.) If it’s set to 1, you can do what you’re doing. Otherwise, you cannot use a single SQLite connection on multiple threads without using your own mutex (if it’s 2), or at all (if it’s 0).

30 threads sounds like overkill, BTW. Generally the appropriate number is equal to the number of CPU cores. And if the task is I/O-bound there’s not much benefit to having more than one since only one of them can be inside SQLite at a time.

—Jens
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: new Error database disk image is malformed

Ron Barnes
Hi Jens,

Thank you for the response.  
I left it at default so it should be a 1.  The threads are not I/O.  They each execute only a few short lines of code to gather network information and then hit the DB for a read using the information they just gathered to populate a key.  After the read they add the formatted data to a table that is eventually displayed to the user.

A quick overview of the code.
I check 64K ports for a response to determine if they are open or not (Nothing nefarious).  I use 30+ threads because it could take up to 10 seconds for a port to respond while other ports respond immediately.

If you are curious to see the app in action and get a better picture of what I'm doing.  It's on the Microsoft store for free.

Do a search of the MS Store for an App named Lo and Behold.

Or click this link..

https://www.microsoft.com/en-us/store/p/lo-and-behold/9nblggh533kc

-Ron

-----Original Message-----
From: sqlite-users [mailto:[hidden email]] On Behalf Of Jens Alfke
Sent: Thursday, January 4, 2018 8:03 PM
To: SQLite mailing list <[hidden email]>
Subject: Re: [sqlite] new Error database disk image is malformed



> On Jan 4, 2018, at 4:51 PM, Ron Barnes <[hidden email]> wrote:
>
> I hope not since I use synclock in my code when ever a thread is attempting a write to the database.  That seems like the only issue from that page that I may be doing.  I could have up to 30 or more threads reading from the DB but only one to three active threads writing.

It depends on the value of SQLITE_THREADSAFE that SQLite was compiled with. (Check the docs for details.) If it’s set to 1, you can do what you’re doing. Otherwise, you cannot use a single SQLite connection on multiple threads without using your own mutex (if it’s 2), or at all (if it’s 0).

30 threads sounds like overkill, BTW. Generally the appropriate number is equal to the number of CPU cores. And if the task is I/O-bound there’s not much benefit to having more than one since only one of them can be inside SQLite at a time.

—Jens
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: new Error database disk image is malformed

Richard Damon
In reply to this post by Jens Alfke-2
On 1/4/18 8:02 PM, Jens Alfke wrote:
>
>> On Jan 4, 2018, at 4:51 PM, Ron Barnes <[hidden email]> wrote:
>>
>> I hope not since I use synclock in my code when ever a thread is attempting a write to the database.  That seems like the only issue from that page that I may be doing.  I could have up to 30 or more threads reading from the DB but only one to three active threads writing.
> It depends on the value of SQLITE_THREADSAFE that SQLite was compiled with. (Check the docs for details.) If it’s set to 1, you can do what you’re doing. Otherwise, you cannot use a single SQLite connection on multiple threads without using your own mutex (if it’s 2), or at all (if it’s 0).
>
> 30 threads sounds like overkill, BTW. Generally the appropriate number is equal to the number of CPU cores. And if the task is I/O-bound there’s not much benefit to having more than one since only one of them can be inside SQLite at a time.
>
> —Jens
Limiting to number of CPU cores make sense for homogeneous threads
(multiple threads doing the same sort of work, so make it go in
parallel). One case where more can make sense is if they are
heterogeneous, each thread doing some unique task to keep the worker
threads simple.

--
Richard Damon

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: new Error database disk image is malformed

Peter da Silva
In reply to this post by Ron Barnes
> Ok. I've read the Document and I think I may be having a rouge thread issue.  I hope not since I use synclock in my code when ever a thread is attempting a write to the database.  That seems like the only issue from that page that I may be doing.  I could have up to 30 or more threads reading from the DB but only one to three active threads writing.  The way my code is set up, though is that each thread would have to wait for the previous thread to finish writing before its turn to write.

Since you're I/O bound on socket connections, and not CPU or database bound, you might want to just have one database thread that communicates using native inter-thread messaging to pass out work and accept responses from the worker threads.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: new Error database disk image is malformed

Jens Alfke-2


> On Jan 4, 2018, at 6:29 PM, Peter Da Silva <[hidden email]> wrote:
>
> Since you're I/O bound on socket connections, and not CPU or database bound, you might want to just have one database thread that communicates using native inter-thread messaging to pass out work and accept responses from the worker threads.

+1. Also, this will make it easy to batch multiple updates into a single transaction, which greatly improves write performance.

—Jens

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: new Error database disk image is malformed

Ron Barnes
I'm coding that now.  I never thought about it.  Thank you!

-----Original Message-----
From: sqlite-users [mailto:[hidden email]] On Behalf Of Jens Alfke
Sent: Friday, January 5, 2018 1:34 PM
To: SQLite mailing list <[hidden email]>
Subject: Re: [sqlite] new Error database disk image is malformed



> On Jan 4, 2018, at 6:29 PM, Peter Da Silva <[hidden email]> wrote:
>
> Since you're I/O bound on socket connections, and not CPU or database bound, you might want to just have one database thread that communicates using native inter-thread messaging to pass out work and accept responses from the worker threads.

+1. Also, this will make it easy to batch multiple updates into a single transaction, which greatly improves write performance.

—Jens

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users