PRAGMA table_info fails with "no such tokenizer"

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

PRAGMA table_info fails with "no such tokenizer"

Anatoli Babenia
Python 3.7.4 (default, Jul  9 2019, 16:32:37)
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> conn = sqlite3.connect("/home/anatoli/.cache/tracker/meta.db")
>>> cursor = conn.cursor()
>>> cursor.execute('PRAGMA main.table_info("fts5")')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
sqlite3.OperationalError: no such tokenizer: TrackerTokenizer


The `sqlite3 meta.db .dump` works without errors. The `fts5` table
referenced in the dump.

INSERT INTO sqlite_master(type,name,tbl_name,rootpage,sql)VALUES('table','fts5','fts5',0,'CREATE
VIRTUAL TABLE fts5 USING fts5(content="fts_view", "nfo:fileName",
"nco:title", "nfo:genre", "nmm:genre", "mtp:creator",
"nco:nameFamily", "nco:nameGiven", "nco:nameAdditional",
"nco:phoneNumber", "nao:prefLabel", "nao:description", "nco:fullname",
"nco:nickname", "nco:contactGroupName", "nfo:tableOfContents",
"ncal:comment", "ncal:location", "ncal:summary", "ncal:contact",
"ncal:description", "nmo:messageSubject", "nmm:albumTitle",
"nmm:artistName", "nco:department", "nco:role", "nco:note",
"nie:keyword", "nco:imID", "nco:imNickname", "nco:emailAddress",
"nmm:category", "nfo:fontFamily", "nco:region", "nco:country",
"nco:extendedAddress", "nco:streetAddress", "nco:postalcode",
"nco:locality", "nco:county", "nco:district", "nco:pobox",
"nie:title", "nie:subject", "nie:plainTextContent", "nie:description",
"nie:comment", "nid3:title", "nid3:albumTitle", "nid3:contentType",
tokenize=TrackerTokenizer)');


--
Anatoli Babenia
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: PRAGMA table_info fails with "no such tokenizer"

Gwendal Roué-2
Hello,

The fts5 table in your database was created with a custom FTS5 tokenizer
named "TrackerTokenizer" (see
https://www.sqlite.org/fts5.html#custom_tokenizers).

Custom FTS5 tokenizers must be registered in each SQLite connection to the
database file, or you get an error like "no such tokenizer:
TrackerTokenizer".

You thus have two options:

1. Grab information about your Python support for SQLite, and register a
custom tokenizer named "TrackerTokenizer" in your connection. For example,
check https://pypi.org/project/sqlitefts/
2. Ask for a database file that does not use any custom FTS5 tokenizer.

Gwendal Roué

On Sun, Sep 29, 2019 at 12:24 PM Anatoli Babenia <[hidden email]>
wrote:

> Python 3.7.4 (default, Jul  9 2019, 16:32:37)
> [GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import sqlite3
> >>> conn = sqlite3.connect("/home/anatoli/.cache/tracker/meta.db")
> >>> cursor = conn.cursor()
> >>> cursor.execute('PRAGMA main.table_info("fts5")')
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> sqlite3.OperationalError: no such tokenizer: TrackerTokenizer
>
>
> The `sqlite3 meta.db .dump` works without errors. The `fts5` table
> referenced in the dump.
>
> INSERT INTO
> sqlite_master(type,name,tbl_name,rootpage,sql)VALUES('table','fts5','fts5',0,'CREATE
> VIRTUAL TABLE fts5 USING fts5(content="fts_view", "nfo:fileName",
> "nco:title", "nfo:genre", "nmm:genre", "mtp:creator",
> "nco:nameFamily", "nco:nameGiven", "nco:nameAdditional",
> "nco:phoneNumber", "nao:prefLabel", "nao:description", "nco:fullname",
> "nco:nickname", "nco:contactGroupName", "nfo:tableOfContents",
> "ncal:comment", "ncal:location", "ncal:summary", "ncal:contact",
> "ncal:description", "nmo:messageSubject", "nmm:albumTitle",
> "nmm:artistName", "nco:department", "nco:role", "nco:note",
> "nie:keyword", "nco:imID", "nco:imNickname", "nco:emailAddress",
> "nmm:category", "nfo:fontFamily", "nco:region", "nco:country",
> "nco:extendedAddress", "nco:streetAddress", "nco:postalcode",
> "nco:locality", "nco:county", "nco:district", "nco:pobox",
> "nie:title", "nie:subject", "nie:plainTextContent", "nie:description",
> "nie:comment", "nid3:title", "nid3:albumTitle", "nid3:contentType",
> tokenize=TrackerTokenizer)');
>
>
> --
> Anatoli Babenia
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users