Quantcast

How to minimize or remove need for dirty bitmap memory size?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

How to minimize or remove need for dirty bitmap memory size?

Bjorn Toft Madsen
Hi,

The docs for SQLite state:
- Situations Where Another RDBMS May Work Better
   - Very large datasets
      - [...] [sqlite] has to allocate a bitmap of dirty pages in the
disk file to help it manage its rollback journal. [...]

Aside from a larger page size, what remedies are suggested for
removing (or minimizing) this requirement?

SQLite is more or less perfect for my needs, but I expect hundreds and
hundreds of gigabytes of data so this high memory requirement means I
have to possibly roll my own solution :(

Thanks,
Bjorn
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to minimize or remove need for dirty bitmap memory size?

Dan Kennedy-4

On Apr 14, 2009, at 4:00 PM, Bjorn Toft Madsen wrote:

> Hi,
>
> The docs for SQLite state:
> - Situations Where Another RDBMS May Work Better
>   - Very large datasets
>      - [...] [sqlite] has to allocate a bitmap of dirty pages in the
> disk file to help it manage its rollback journal. [...]
>
> Aside from a larger page size, what remedies are suggested for
> removing (or minimizing) this requirement?

That statement is out of date. Since it was written the bitmap has
been replaced with a better data structure that does not require the
big per-transaction allocation.

Dan.


> SQLite is more or less perfect for my needs, but I expect hundreds and
> hundreds of gigabytes of data so this high memory requirement means I
> have to possibly roll my own solution :(
>
> Thanks,
> Bjorn
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to minimize or remove need for dirty bitmap memory size?

Bjorn Toft Madsen
... and there was much rejoicing! :D

Is it thus safe to say that sqlite can handle 100+ GB of data, or are
other resources consumed exponentially as DB size goes up? (linear
growth I can understand and handle, exponential growth is a problem
for me)

Thanks,
Bjorn

2009/4/14 Dan <[hidden email]>:

>
> On Apr 14, 2009, at 4:00 PM, Bjorn Toft Madsen wrote:
>
>> Hi,
>>
>> The docs for SQLite state:
>> - Situations Where Another RDBMS May Work Better
>>   - Very large datasets
>>      - [...] [sqlite] has to allocate a bitmap of dirty pages in the
>> disk file to help it manage its rollback journal. [...]
>>
>> Aside from a larger page size, what remedies are suggested for
>> removing (or minimizing) this requirement?
>
> That statement is out of date. Since it was written the bitmap has
> been replaced with a better data structure that does not require the
> big per-transaction allocation.
>
> Dan.
>
>
>> SQLite is more or less perfect for my needs, but I expect hundreds and
>> hundreds of gigabytes of data so this high memory requirement means I
>> have to possibly roll my own solution :(
>>
>> Thanks,
>> Bjorn
>> _______________________________________________
>> sqlite-users mailing list
>> [hidden email]
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to minimize or remove need for dirty bitmap memory size?

D. Richard Hipp
In reply to this post by Dan Kennedy-4

On Apr 14, 2009, at 6:33 AM, Dan wrote:

>
> On Apr 14, 2009, at 4:00 PM, Bjorn Toft Madsen wrote:
>
>> Hi,
>>
>> The docs for SQLite state:
>> - Situations Where Another RDBMS May Work Better
>>  - Very large datasets
>>     - [...] [sqlite] has to allocate a bitmap of dirty pages in the
>> disk file to help it manage its rollback journal. [...]
>>
>> Aside from a larger page size, what remedies are suggested for
>> removing (or minimizing) this requirement?
>
> That statement is out of date. Since it was written the bitmap has
> been replaced with a better data structure that does not require the
> big per-transaction allocation.
>

Right.  The change to get rid of that bitmap happened over a year  
ago.  (That change also made transactions run a little faster, which  
was a surprising, though welcomed, side-effect.)  Unfortunately, I  
failed to update the documentation referenced above.  I have just now  
checked in a fix to the documentation.  It now reads that one should  
consider a different database engine if you are contemplating  
databases larger than 2 tebibytes in size.  SQLite should scale up to  
that size now, assuming you have a disk-drive and filesystem that can  
hold a file that big.

D. Richard Hipp
[hidden email]



_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Loading...