General Amalgamation vs Your Own

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

General Amalgamation vs Your Own

Stephen Chrzanowski
With a post that someone made here, and the thought that I've been running
in my head over the past while...

How does one have their own code base for SQLite, with their own customer
logic or functionality or whatever, then, have updates provided by the
SQLite team implemented in when updates and such are provided?

I'm assuming Fossil would handle this kind of process with merging and such?

What is the workflow on making sure the amalgamation source doesn't modify
local changes?
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: General Amalgamation vs Your Own

Richard Hipp-3
On 9/21/19, Stephen Chrzanowski <[hidden email]> wrote:
>
> How does one have their own code base for SQLite, with their own customer
> logic or functionality or whatever, then, have updates provided by the
> SQLite team implemented in when updates and such are provided?
>

https://www.sqlite.org/privatebranch.html

--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: General Amalgamation vs Your Own

Warren Young
In reply to this post by Stephen Chrzanowski
On Sep 21, 2019, at 11:29 AM, Stephen Chrzanowski <[hidden email]> wrote:
>
> How does one have their own code base for SQLite, with their own customer
> logic or functionality or whatever, then, have updates provided by the
> SQLite team implemented in when updates and such are provided?

What kind of code are you talking about?

With custom functions and loadable extensions, you don’t modify the SQLite core at all:

    https://sqlite.org/c3ref/create_function.html
    https://sqlite.org/loadext.html

The latter allows a lot of flexibility, changing collation sequences, adding VFSes, etc.

If you’re modifying the SQLite core, then why?  Maybe it isn’t actually required that you do so, and we could show you a better way.  Or, maybe SQLite extensions could be, ah, *extended* to give you the API you’d need to do this from the outside, without modifying the core.

> I'm assuming Fossil would handle this kind of process with merging and such?

I would not recommend using private branches, even though drh offered it, because it’s not a commonly-used feature of Fossil, so it’s not as well-tested and well-supported as Fossil’s mainstream usage code paths.  If you ever run into trouble using private branches in Fossil, you’re more likely to get “crickets” as a response than with similar trouble within the well-used features of Fossil.

Even when private branches are the right answer, they exist for a different use case than what you’ve got going here: you have check-in rights on the parent repo you cloned from, but you don’t want all local check-ins to sync up to the parent repo.  I don’t see any user with check-ins on the SQLite code repo that looks like your name or your Gmail user name, so I’m assuming you *don’t* have check-in rights on the SQLite code repo.  Therefore, private branches don’t really apply to your case.

I’d suggest one of two other alternatives:

1. As with private branches, keep your SQLite changes in a clone of the main SQLite repo, but set the “autosync” setting to “pullonly” and check your local changes in on a normal branch.  Since you have no check-in ability on the main SQLite repo and autosync is set to not push local changes, you’re set.

The main thing you have to watch out for is that if you ever *do* get check-in rights on the main SQLite repo, that you never use that same forked repo with that account, since the first sync will push all of your historical changes up to SQLite.

All branches in Fossil are effectively private when you don’t have check-in capability.  All the autosync setting does is prevent Fossil from trying things you know will always fail.


2. Keep your local mods in a separate repository — possibly managed by a non-Fossil [D]VCS — and periodically merge changes in from SQLite.  This arrangement is usually called a “vendor branch:”

    http://svnbook.red-bean.com/en/1.7/svn.advanced.vendorbr.html

That document gives it in terms of Subversion, but it works in any decent [D]VCS.

When using two different [D]VCSes, you do the “merge” in part with copy commands:

    $ cd ~/path/to/sqlite/checkout
    $ fossil update
    $ make -j11 sqlite3.c           # optional
    $ cd ~/path/to/private/repo
    $ git checkout sqlite-vendor-branch
    $ cp ~/path/to/sqlite/checkout/src/* 3rd-party/sqlite
    $ git commit -a
    $ git checkout master, merge, etc.

You could then script that process locally, so that you give only a single command, like “tools/update-sqlite”.

> What is the workflow on making sure the amalgamation source doesn't modify
> local changes?

Vendor branches.  You want to do that even with my option #1 above or with drh’s offering of private branches.  In the latter case, SQLite “trunk” is the vendor branch, and your branch off of that is your project’s effective “trunk”.

Keep in mind that in Fossil, “trunk” isn’t all that special.  It just happens to be Fossil’s default when starting a new repo and it’s Fossil’s best guess when you give it no other clues.  Other than that, trunk is just another branch in Fossil.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: General Amalgamation vs Your Own

Stephen Chrzanowski
Thanks for all this info.

No, I have not made a single change to the amalgamation that has been
provided by sqlite.org.  The most I've done is read, line by line, through
the first meg or so of source code, as well as setup an environment where I
specifically can build sqlite3.dll for fun.  The thought was entirely for
theoretical practices, and only came up when I noticed a few threads
suggesting that they're making changes to the code (Be it to test theory or
to enact live in their libraries), then my thoughts wandered to their code
vs future SQLite code enhancements or bug fixes, and how getting the two
different sets of code to work with each other going forward.

On Sat, Sep 21, 2019 at 10:41 PM Warren Young <[hidden email]> wrote:

> On Sep 21, 2019, at 11:29 AM, Stephen Chrzanowski <[hidden email]>
> wrote:
> >
> > How does one have their own code base for SQLite, with their own customer
> > logic or functionality or whatever, then, have updates provided by the
> > SQLite team implemented in when updates and such are provided?
>
> What kind of code are you talking about?
>
> With custom functions and loadable extensions, you don’t modify the SQLite
> core at all:
>
>     https://sqlite.org/c3ref/create_function.html
>     https://sqlite.org/loadext.html
>
> The latter allows a lot of flexibility, changing collation sequences,
> adding VFSes, etc.
>
> If you’re modifying the SQLite core, then why?  Maybe it isn’t actually
> required that you do so, and we could show you a better way.  Or, maybe
> SQLite extensions could be, ah, *extended* to give you the API you’d need
> to do this from the outside, without modifying the core.
>
> > I'm assuming Fossil would handle this kind of process with merging and
> such?
>
> I would not recommend using private branches, even though drh offered it,
> because it’s not a commonly-used feature of Fossil, so it’s not as
> well-tested and well-supported as Fossil’s mainstream usage code paths.  If
> you ever run into trouble using private branches in Fossil, you’re more
> likely to get “crickets” as a response than with similar trouble within the
> well-used features of Fossil.
>
> Even when private branches are the right answer, they exist for a
> different use case than what you’ve got going here: you have check-in
> rights on the parent repo you cloned from, but you don’t want all local
> check-ins to sync up to the parent repo.  I don’t see any user with
> check-ins on the SQLite code repo that looks like your name or your Gmail
> user name, so I’m assuming you *don’t* have check-in rights on the SQLite
> code repo.  Therefore, private branches don’t really apply to your case.
>
> I’d suggest one of two other alternatives:
>
> 1. As with private branches, keep your SQLite changes in a clone of the
> main SQLite repo, but set the “autosync” setting to “pullonly” and check
> your local changes in on a normal branch.  Since you have no check-in
> ability on the main SQLite repo and autosync is set to not push local
> changes, you’re set.
>
> The main thing you have to watch out for is that if you ever *do* get
> check-in rights on the main SQLite repo, that you never use that same
> forked repo with that account, since the first sync will push all of your
> historical changes up to SQLite.
>
> All branches in Fossil are effectively private when you don’t have
> check-in capability.  All the autosync setting does is prevent Fossil from
> trying things you know will always fail.
>
>
> 2. Keep your local mods in a separate repository — possibly managed by a
> non-Fossil [D]VCS — and periodically merge changes in from SQLite.  This
> arrangement is usually called a “vendor branch:”
>
>     http://svnbook.red-bean.com/en/1.7/svn.advanced.vendorbr.html
>
> That document gives it in terms of Subversion, but it works in any decent
> [D]VCS.
>
> When using two different [D]VCSes, you do the “merge” in part with copy
> commands:
>
>     $ cd ~/path/to/sqlite/checkout
>     $ fossil update
>     $ make -j11 sqlite3.c           # optional
>     $ cd ~/path/to/private/repo
>     $ git checkout sqlite-vendor-branch
>     $ cp ~/path/to/sqlite/checkout/src/* 3rd-party/sqlite
>     $ git commit -a
>     $ git checkout master, merge, etc.
>
> You could then script that process locally, so that you give only a single
> command, like “tools/update-sqlite”.
>
> > What is the workflow on making sure the amalgamation source doesn't
> modify
> > local changes?
>
> Vendor branches.  You want to do that even with my option #1 above or with
> drh’s offering of private branches.  In the latter case, SQLite “trunk” is
> the vendor branch, and your branch off of that is your project’s effective
> “trunk”.
>
> Keep in mind that in Fossil, “trunk” isn’t all that special.  It just
> happens to be Fossil’s default when starting a new repo and it’s Fossil’s
> best guess when you give it no other clues.  Other than that, trunk is just
> another branch in Fossil.
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: General Amalgamation vs Your Own

Keith Medcalf

On Saturday, 21 September, 2019 22:35, Stephen Chrzanowski <[hidden email]>:

>Thanks for all this info.

>No, I have not made a single change to the amalgamation that has been
>provided by sqlite.org.  The most I've done is read, line by line,
>through the first meg or so of source code, as well as setup an
>environment where I specifically can build sqlite3.dll for fun.  
>The thought was entirely for theoretical practices, and only came
>up when I noticed a few threads suggesting that they're making
>changes to the code (Be it to test theory or to enact live in
>their libraries), then my thoughts wandered to their code vs future
>SQLite code enhancements or bug fixes, and how getting the two
>different sets of code to work with each other going forward.

I used to have a repository that contained basically the distribution amalgamation/shell code on trunk and a branch that contained the changes I had made.  This was complicated to maintain since the changes end up being somewhat extensive to implement simple things (such as moving extensions from the shell into the core, and adding new builtin extensions).  

Now I simply work on a clone of the distribution repository and have a private branch that has my changes incorporated in source and standard build tools directly.  It is much easier to maintain and so far has not required anywhere near as many manual merge operations to resolve conflicts.




_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users