# page_content

c2: Query Language Understood by SQLite
c5: lang.html
c4:
SQL As Understood By SQLite

SQLite understands most of the standard SQL
language.  But it does omit some features
while at the same time
adding a few features of its own.  This document attempts to
describe precisely what parts of the SQL language SQLite does
and does not support.  A list of SQL keywords is 
also provided.  The SQL language syntax is described by
syntax diagrams.

The following syntax documentation topics are available:






aggregate functions
ALTER TABLE
ANALYZE
ATTACH DATABASE
BEGIN TRANSACTION
comment
COMMIT TRANSACTION
core functions
CREATE INDEX
CREATE TABLE
CREATE TRIGGER
CREATE VIEW
CREATE VIRTUAL TABLE
date and time functions
DELETE
DETACH DATABASE
DROP INDEX
DROP TABLE
DROP TRIGGER
DROP VIEW
END TRANSACTION
EXPLAIN
expression
INDEXED BY
INSERT
keywords
ON CONFLICT clause
PRAGMA
REINDEX
RELEASE SAVEPOINT
REPLACE
RETURNING clause
ROLLBACK TRANSACTION
SAVEPOINT
SELECT
UPDATE
UPSERT
VACUUM
WITH clause





The routines sqlite3_prepare_v2(), sqlite3_prepare(),
sqlite3_prepare16(), sqlite3_prepare16_v2(), 
sqlite3_exec(), and sqlite3_get_table() accept
an SQL statement list (sql-stmt-list) which is a semicolon-separated
list of statements.

sql-stmt-list:
 

 


Each SQL statement in the statement list is an instance of the
following:

sql-stmt:


c2: SQLite Support Options
c5: support.html
c4:
Public Forum

Connect with SQLite developers and other SQLite users on the
SQLite Forum.  This is the preferred
way to get free help with SQLite.




There was formerly a mailing list devoted to discussions
about SQLite, but as of 2020-03-12, the mailing list is deprecated.
Please use the 
SQLite Forum instead.
Historical mailing lists posts are preserved at

www.mail-archive.com



Paid Professional Support


If you would like professional support for SQLite
or if you want custom modifications performed by the
original author of SQLite, these services are available for a modest fee.
For additional information contact:



D. Richard Hipp 
Hwaci - Applied Software Research 
704.948.4565 
drh@hwaci.com

More Info



Proprietary SQLite Extensions

The core SQLite library found on this website is in the
public domain.  But there also exist
proprietary, licensed extensions to SQLite.



The SQLite Encryption Extension (SEE)
will read and write database files encrypted using
128-bit or 256-bit AES.


The SQLite Compressed and Encrypted Read-Only Database (CEROD)
extension will read a database file that is both compressed
and encrypted.


The ZIPVFS
extension will both read and write database files that are both compressed and optionally
encrypted using application-supplied compression and encryption routines.


The Test Harness #3 (TH3) is an alternative set of test
vectors for SQLite that verify the operation of SQLite in an
as-deployed configuration on embedded platforms with
100% branch test coverage.


c2: The SQLITE_MEMSTAT Virtual Table
c5: memstat.html
c4:
1. Overview


The SQLITE_MEMSTAT extension implements an eponymous-only virtual table that
provides SQL access to the sqlite3_status64() and
sqlite3_db_status() interfaces.



The SQLITE_STMT extension can also be loaded at run-time
by compiling the extension into a shared library or DLL using the source
code at https://sqlite.org/src/file/ext/misc/memstat.c and following the
instructions for how to compile loadable extensions.


2. Usage


The SQLITE_MEMSTAT virtual table is a read-only table that can be
queried to determine performance characteristics (primarily the
amount of memory being used) of the current instance of SQLite.
The SQLITE_MEMSTATE table is essentially a wrapper around the
C-language APIs sqlite3_status64() and sqlite3_db_status().
If the 
memstat.c source
file is compiled with the -DSQLITE_ENABLE_ZIPVFS option, then SQLITE_MEMSTAT
will also do some file-control calls to extract
memory usage information about the 
ZIPVFS subsystem, 
if that subsystem as been licensed, installed, and is in use.


The SQLITE_MEMSTAT table appears to have the following schema:

CREATE TABLE sqlite_memstat(
  name TEXT,
  schema TEXT,
  value INT,
  hiwtr INT
);



Each row of the SQLITE_MEMSTAT table corresponds to a single call to
one of the sqlite3_status64() or sqlite3_db_status() interfaces.
The NAME column of the row identifies which "verb" was passed to those
interfaces.  For example, if sqlite3_status64() is invoked with
SQLITE_STATUS_MEMORY_USED, then the NAME column is 'MEMORY_USED'.
Or if sqlite3_db_status() is invoked with SQLITE_DBSTATUS_CACHE_USED,
then the NAME column is "DB_CACHE_USED".


The SCHEMA column is NULL, except for cases when the sqlite3_file_control()
interface is used to interrogate the ZIPVFS backend.  As this only happens
when the memstat.c module is compiled with -DSQLITE_ENABLE_ZIPVFS and when
ZIPVFS is in use, 
SCHEMA is usually NULL.


The VALUE and HIWTR columns report the current value of the measure and
its "high-water mark".  The high-water mark is the highest value ever seen
for the measurement, at least since the last reset.  The SQLITE_MEMSTAT
virtual table does not provide a mechanism for resetting the high-water mark.


Depending on which parameter is being interrogated, one of the VALUE
or HIWTR mark measurements might be undefined.  For example, only the
high-water mark is meaningful for SQLITE_STATUS_MALLOC_SIZE, and
only the current value is meaningful for SQLITE_DBSTATUS_CACHE_USED.
For rows where one or the other of VALUE or HIWTR is not meaningful,
that value is returned as NULL.
the 

interfaces, with the initial


c2: SQLite Copyright
c5: copyright.html
c4:
SQLite Is Public Domain




SQLite is in the
Public Domain




All of the code and documentation in SQLite has been dedicated to the
public domain
by the authors.
All code authors, and representatives of the companies they work for,
have signed affidavits dedicating their contributions to
the public domain and originals of
those signed affidavits are stored in a firesafe at the main offices
of Hwaci.  All contributors are citizens
of countries that allow creative works to be dedicated into the public
domain.
Anyone is free to copy, modify, publish, use, compile, sell, or distribute
the original SQLite code, either in source code form or as a compiled binary,
for any purpose, commercial or non-commercial, and by any means.



The previous paragraph applies to the deliverable code and documentation
in SQLite - those parts of the SQLite library that you actually bundle and
ship with a larger application.  Some scripts used as part of the
build process (for example the "configure" scripts generated by autoconf)
might fall under other open-source licenses.  Nothing from these build 
scripts ever reaches the final deliverable SQLite library, however, and 
so the licenses associated with those scripts should not be 
a factor in assessing your rights to copy and use the SQLite library.



All of the deliverable code in SQLite has been written from scratch.
No code has been taken from other projects or from the open
internet.  Every line of code can be traced back to its original
author, and all of those authors have public domain dedications
on file.  So the SQLite code base is clean and is
uncontaminated with licensed code from other projects.




Open-Source, not Open-Contribution


SQLite is open-source, meaning that you can make as many copies of it as
you want and do whatever you want with those copies, without limitation.
But SQLite is not open-contribution.  In order to keep SQLite in the public
domain and ensure that the code does not become contaminated with proprietary
or licensed content, the project does not accept patches from people who
have not submitted an affidavit dedicating their contribution into the
public domain.


All of the code in SQLite is original, having been written
specifically for use by SQLite.  No code has been copied from unknown
sources on the internet.










Warranty of Title


SQLite is in the public domain and does not require a license.
Even so, some organizations want legal proof of their right to use
SQLite.  Circumstances where this might occurs include the following:



 Your company desires indemnity against claims of copyright infringement.
 You are using SQLite in a jurisdiction that does not recognize
     the public domain.  
 You are using SQLite in a jurisdiction that does not recognize
     the right of an author to dedicate their work to the public
     domain. 
 You want to hold a tangible legal document
     as evidence that you have the legal right to use and distribute
     SQLite. 
 Your legal department tells you that you must purchase a license.
     



If any of the above circumstances apply to you,
Hwaci, the company that employs 
all the developers of SQLite, will 
sell you
a Warranty of Title for SQLite.
A Warranty of Title is a legal document that asserts that the claimed
authors of SQLite are the true authors, and that the authors 
have the legal right to dedicate the SQLite to the public domain, and 
that Hwaci will vigorously defend against challenges to those claims.
All proceeds from the sale of SQLite Warranties of Title are used to fund
continuing improvement and support of SQLite.


Contributed Code


In order to keep SQLite completely free and unencumbered by copyright,
the project does not accept patches.  If you would like to suggest a
change and you include a patch as a proof-of-concept, that would
be great.  However, please do not be offended if we rewrite your patch
from scratch.


c2: The Schema Table
c5: schematab.html
c4:
1. Introduction

Every SQLite database contains a single "schema table" that stores the
schema for that database.  The schema for a database is a description of
all of the other tables, indexes, triggers, and views that are
contained within the database.  The schema table looks like this:


CREATE TABLE sqlite_schema(
  type text,
  name text,
  tbl_name text,
  rootpage integer,
  sql text
);


The sqlite_schema table contains one row for each table, index, view,
and trigger (collectively "objects") in the schema, except there
is no entry for the sqlite_schema table itself.  See the
schema storage subsection of the file format documentation for
additional information on how SQLite uses the sqlite_schema table
internally.

2. Alternative Names

The schema table can always be referenced using the name "sqlite_schema",
especially if qualifed by the schema name like 
"main.sqlite_schema" or "temp.sqlite_schema".  But for historical
compatibility, some alternative names are also recognized, including:


 sqlite_master
 sqlite_temp_schema
 sqlite_temp_master



Alternatives (2) and (3) only work for the TEMP database associated
with each database connection, but alternative (1) works anywhere.
For historical reasons, callbacks from the sqlite3_set_authorizer()
interface always refer to the schema table using names (1) or (3).

3. Interpretation Of The Schema Table

The meanings of the fields of the schema table are as follows:


type

The sqlite_schema.type column will be one
of the following text strings:  'table', 'index', 'view', or 'trigger'
according to the type of object defined.  The 'table' string is used
for both ordinary and virtual tables.


name

The sqlite_schema.name column will hold the name of the object.
(UNIQUE and PRIMARY KEY constraints on tables cause SQLite to create
internal indexes with names of the form "sqlite_autoindex_TABLE_N"
where TABLE is replaced by the name of the table that contains the
constraint and N is an integer beginning with 1 and increasing by one
with each constraint seen in the table definition.
In a WITHOUT ROWID table, there is no sqlite_schema entry for the
PRIMARY KEY, but the "sqlite_autoindex_TABLE_N" name is set aside
for the PRIMARY KEY as if the sqlite_schema entry did exist.  This
will affect the numbering of subsequent UNIQUE constraints.
The "sqlite_autoindex_TABLE_N" name is never allocated for an
INTEGER PRIMARY KEY, either in rowid tables or WITHOUT ROWID tables.



tbl_name

The sqlite_schema.tbl_name column holds the name of a table or view
that the object is associated with.  For a table or view, the
tbl_name column is a copy of the name column.  For an index, the tbl_name
is the name of the table that is indexed.  For a trigger, the tbl_name
column stores the name of the table or view that causes the trigger 
to fire.


rootpage

The sqlite_schema.rootpage column stores the page number of the root
b-tree page for tables and indexes.  For rows that define views, triggers,
and virtual tables, the rootpage column is 0 or NULL.


sql

The sqlite_schema.sql column stores SQL text that describes the
object.  This SQL text is a CREATE TABLE, CREATE VIRTUAL TABLE,
CREATE INDEX,
CREATE VIEW, or CREATE TRIGGER statement that if evaluated against
the database file when it is the main database of a database connection
would recreate the object.  The text is usually a copy of the original
statement used to create the object but with normalizations applied so
that the text conforms to the following rules:


The CREATE, TABLE, VIEW, TRIGGER, and INDEX keywords at the beginning
of the statement are converted to all upper case letters.
The TEMP or TEMPORARY keyword is removed if it occurs after the 
initial CREATE keyword.
Any database name qualifier that occurs prior to the name of the
object being created is removed.
Leading spaces are removed.
All spaces following the first two keywords are converted into a single
space.


The text in the sqlite_schema.sql column is a copy of the original
CREATE statement text that created the object, except normalized as
described above and as modified by subsequent ALTER TABLE statements.
The sqlite_schema.sql is NULL for the internal indexes that are
automatically created by UNIQUE or PRIMARY KEY constraints.



4. Creation and Modification Of The Schema Table

SQLite creates the schema table upon database creation and modifies
its content as SQLite users submit DDL statements for execution. There
is no need for users to modify it under normal circumstances, and they
bear the risk of database corruption if they do modify it.


c2: How To Compile SQLite
c5: howtocompile.html
c4:
How To Compile SQLite



 Overview 


SQLite is ANSI-C source code.
It must be compiled into machine code before it is useful.
This article is a guide to the various ways of compiling SQLite.


This article does not contain a step-by-step recipe for compiling
SQLite.  That would be difficult since each development situation
is different.
Rather, this article describes and illustrates the principals behind the
compilation of SQLite.  Typical compilation commands are provided as examples
with the expectation that application developers can use these examples
as guidance for developing their own custom compilation procedures.
In other words, this article provides ideas and insights, not turnkey
solutions.

1. Amalgamation Versus Individual Source Files 

SQLite is built from over one hundred files of C code and script
spread across multiple directories.  The implementation of SQLite is pure
ANSI-C, but many of the C-language source code files are either
generated or transformed by auxiliary C programs and AWK, SED, and TCL 
scripts prior to being incorporated into the finished SQLite library.
Building the necessary C programs and transforming and/or creating the
C-language source code for SQLite is a complex process.

To simplify matters, SQLite is also available as a pre-packaged
amalgamation source code file: sqlite3.c.  The amalgamation is
a single file of ANSI-C code that implements the entire SQLite library.
The amalgamation is much easier to deal with.  Everything is contained
within a single code file, so it is easy to drop into the source tree
of a larger C or C++ program.  All the code generation and transformation
steps have already been carried out so there are no auxiliary C programs
to configure and compile and no scripts to run.  And, because the entire
library is contained in a single translation unit, compilers are able to
do more advanced optimizations resulting in a 5% to 10% performance 
improvement.  For these reasons, the amalgamation source file 
("sqlite3.c") is recommended for all applications.


The use of the amalgamation is recommended for all applications.


Building SQLite directly from individual source code files is certainly
possible, but it is not recommended.  For some specialized applications, it
might be necessary to modify the build process in ways that cannot be done
using just the prebuilt amalgamation source file downloaded from the website.
For those situations, it is recommended that a customized amalgamation be
built (as described below)
and used.  In other words, even if a project requires building SQLite 
beginning with individual source files, it is still recommended that an
amalgamation source file be used as an intermediate step.



2. Compiling The Command-Line Interface

A build of the command-line interface requires three source
files:


sqlite3.c: The SQLite amalgamation source file
sqlite3.h: The header files that accompanies sqlite3.c and 
defines the C-language interfaces to SQLite.
shell.c: The command-line interface program itself.
This is the C source code file that contains the definition of
the main() routine and the loop that prompts for user input
and passes that input into the SQLite database engine for processing.


All three of the above source files are contained in the
amalgamation tarball available on the download page.

To build the CLI, simply put these three files in the same directory
and compile them together.  Using MSVC:


cl shell.c sqlite3.c -Fesqlite3.exe


On unix systems (or on Windows using cygwin or mingw+msys)
the command typically looks something like this:


gcc shell.c sqlite3.c -lpthread -ldl -lm -o sqlite3


The pthreads library is needed to make SQLite threadsafe.  But
since the CLI is single threaded,  we could instruct SQLite to build
in a non-threadsafe mode and thereby omit the pthreads library:


gcc -DSQLITE_THREADSAFE=0 shell.c sqlite3.c -ldl -lm -o sqlite3


The -ldl library is needed to support dynamic loading, the
sqlite3_load_extension() interface and the
load_extension() SQL function.  If these features are not required,
then they can be omitted using SQLITE_OMIT_LOAD_EXTENSION compile-time
option:


gcc -DSQLITE_THREADSAFE=0 -DSQLITE_OMIT_LOAD_EXTENSION shell.c sqlite3.c -o sqlite3


One might want to provide other compile-time options such as
-DSQLITE_ENABLE_FTS4 or -DSQLITE_ENABLE_FTS5 for full-text search,
-DSQLITE_ENABLE_RTREE for the R*Tree search engine extension,
-DSQLITE_ENABLE_JSON1 to include JSON SQL functions, or
-DSQLITE_ENABLE_DBSTAT_VTAB for the dbstat virtual table.
In order to see extra commentary in EXPLAIN listings, add the 
-DSQLITE_ENABLE_EXPLAIN_COMMENTS option.
On unix systems, add -DHAVE_USLEEP=1 if the host machine supports the
usleep() system call.  Add -DHAVE_READLINE and the -lreadline
and -lncurses libraries to get command-line editing support.
One might also want to
specify some compiler optimization switches.  (The precompiled
CLI available for download from the SQLite website uses "-Os".)
There are countless possible variations here.  A command to
compile a full-featured shell might look something like this:


gcc -Os -I. -DSQLITE_THREADSAFE=0 -DSQLITE_ENABLE_FTS4 &#92;
   -DSQLITE_ENABLE_FTS5 -DSQLITE_ENABLE_JSON1 &#92;
   -DSQLITE_ENABLE_RTREE -DSQLITE_ENABLE_EXPLAIN_COMMENTS &#92;
   -DHAVE_USLEEP -DHAVE_READLINE &#92;
   shell.c sqlite3.c -ldl -lm -lreadline -lncurses -o sqlite3


The key point is this:  Building the CLI consists of compiling 
together two C-language files.   The shell.c file contains the
definition of the entry point and the user input loop and the
SQLite amalgamation sqlite3.c contains the complete implementation
of the SQLite library.



3. Compiling The TCL Interface

The TCL interface for SQLite is a small module that is added into
the regular amalgamation.  The result is a new amalgamated source
file called "tclsqlite3.c".  This single source file is all that
is needed to generate a shared library that can be loaded into a
standard 
tclsh or 
wish using the 
TCL load command, or to generate a
standalone tclsh that comes with SQLite built in.
A copy of the tcl amalgamation
is included on the download page as a file in the TEA tarball.

To generate a TCL-loadable library for SQLite on Linux, the following
command will suffice:


gcc -o libtclsqlite3.so -shared tclsqlite3.c -lpthread -ldl -ltcl


Building shared libraries for Mac OS X and Windows is not nearly so simple,
unfortunately.  For those platforms it is best to use the configure script
and makefile that is included with the TEA tarball.

To generate a standalone tclsh that is statically linked with SQLite,
use this compiler invocation:


gcc -DTCLSH=1 tclsqlite3.c -ltcl -lpthread -ldl -lz -lm


The trick here is the -DTCLSH=1 option.  The TCL interface module for
SQLite includes a main() procedure that initializes a TCL interpreter
and enters a command-line loop when it is compiled with -DTCLSH=1.  The
command above works on both Linux and Mac OS X, though one may need to adjust
the library options depending on the platform and which version of TCL one
is linking against.



4. Building The Amalgamation

The versions of the SQLite amalgamation that are supplied on the
download page are normally adequate for most users.  However, some
projects may want or need to build their own amalgamations.  A common
reason for building a custom amalgamation is in order to use certain
compile-time options to customize the SQLite library.  Recall that
the SQLite amalgamation contains a lot of C-code that is generated by
auxiliary programs and scripts.  Many of the compile-time
options effect this generated code and must be supplied to the code
generators before the amalgamation is assembled.  The set of 
compile-time options that must be passed into the code generators can
vary from one release of SQLite to the next, but at the time of this
writing (circa SQLite 3.6.20, 2009-11-04) the set of options that must
be known by the code generators includes:


SQLITE_ENABLE_UPDATE_DELETE_LIMIT
SQLITE_OMIT_ALTERTABLE
SQLITE_OMIT_ANALYZE
SQLITE_OMIT_ATTACH
SQLITE_OMIT_AUTOINCREMENT
SQLITE_OMIT_CAST
SQLITE_OMIT_COMPOUND_SELECT
SQLITE_OMIT_EXPLAIN
SQLITE_OMIT_FOREIGN_KEY
SQLITE_OMIT_PRAGMA
SQLITE_OMIT_REINDEX
SQLITE_OMIT_SUBQUERY
SQLITE_OMIT_TEMPDB
SQLITE_OMIT_TRIGGER
SQLITE_OMIT_VACUUM
SQLITE_OMIT_VIEW
SQLITE_OMIT_VIRTUALTABLE


To build a custom amalgamation, first download the original individual
source files onto a unix or unix-like development platform.  
Be sure to get the original source
files not the "preprocessed source files".  One can obtain the complete
set of original source files either from the download page or directly
from the configuration management system.

Suppose the SQLite source tree is stored in a directory named "sqlite".
Plan to construct the amalgamation in a parallel directory named (for
example) "bld".  First construct an appropriate Makefile by either
running the configure script at the top of the SQLite source tree, or by
making a copy of one of the template Makefiles at the top of the source tree.
Then hand edit this Makefile to include the desired compile-time options.
Finally run:


make sqlite3.c


Or on Windows with MSVC:


nmake /f Makefile.msc sqlite3.c


The "sqlite3.c" make target will automatically construct the regular
"sqlite3.c" amalgamation source file, its header file
"sqlite3.h", and the "tclsqlite3.c" amalgamation source
file that includes the TCL interface.
Afterwards, the needed files can be copied into project directories and
compiled according to the procedures outlined above.



5. Building A Windows DLL

To build a DLL of SQLite for use in Windows, first acquire the
appropriate amalgamated source code files, sqlite3.c and sqlite3.h.  
These can either
be downloaded from the SQLite website
or custom generated from sources as shown above.

With source code files in the working directory, a DLL
can be generated using MSVC with the following command:


cl sqlite3.c -link -dll -out:sqlite3.dll


The above command should be run from the MSVC Native Tools Command
Prompt.  If you have MSVC installed on your machine, you probably
have multiple versions of this Command Prompt, for native builds
for x86 and x64, and possibly also for cross-compiling to ARM.
Use the appropriate Command Prompt depending on the desired DLL.

If using the MinGW compiler, the command-line is this:


gcc -shared sqlite3.c -o sqlite3.dll


Note that MinGW generates 32-bit DLLs only.  There is a separate
MinGW64 project that can be used to generate 64-bit DLLs.  Presumably
the command-line syntax is similar.
Also note that recent versions of MSVC generate DLLs that will not work
on WinXP and earlier versions of Windows.  So for maximum compatibility
of your generated DLL, MinGW is recommended.  A good rule-of-thumb
is to generate 32-bit DLLs using MinGW and 64-bit DLLs using MSVC.

In most cases, you will want to supplement the basic commands above with
compile-time options appropriate for your application.  Commonly used
compile-time options include:


-Os - Optimize for size. 
Make the DLL as small as possible.

-O2 - Optimize for speed.  This will make the DLL larger by
unrolling loops and inlining functions.

-DSQLITE_ENABLE_FTS4 -
Include the full-text search engine code in SQLite.

-DSQLITE_ENABLE_RTREE - Include the R-Tree extension.

-DSQLITE_ENABLE_COLUMN_METADATA -
This enables some extra APIs that are required by some common systems,
including Ruby-on-Rails.


c2: SQLite FTS3 and FTS4 Extensions
c5: fts3.html
c4:
Overview


  FTS3 and FTS4 are SQLite virtual table modules that allows users to perform 
  full-text searches on a set of documents. The most common (and effective) 
  way to describe full-text searches is "what Google, Yahoo, and Bing do
  with documents placed on the World Wide Web". Users input a term, or series 
  of terms, perhaps connected by a binary operator or grouped together into a 
  phrase, and the full-text query system finds the set of documents that best 
  matches those terms considering the operators and groupings the user has 
  specified. This article describes the deployment and usage of FTS3 and FTS4.


  FTS1 and FTS2 are obsolete full-text search modules for SQLite.  There are known
  issues with these older modules and their use should be avoided.
  Portions of the original FTS3 code were contributed to the SQLite project 
  by Scott Hess of Google. It is now 
  developed and maintained as part of SQLite.

1. Introduction to FTS3 and FTS4


  The FTS3 and FTS4 extension modules allows users to create special tables with a 
  built-in full-text index (hereafter "FTS tables"). The full-text index
  allows the user to efficiently query the database for all rows that contain
  one or more words (hereafter "tokens"), even if the table
  contains many large documents.


  For example, if each of the 517430 documents in the 
  "Enron E-Mail Dataset"
  is inserted into both an FTS table and an ordinary SQLite table
  created using the following SQL script:

CREATE VIRTUAL TABLE enrondata1 USING fts3(content TEXT);     /* FTS3 table */
CREATE TABLE enrondata2(content TEXT);                        /* Ordinary table */



  Then either of the two queries below may be executed to find the number of
  documents in the database that contain the word "linux" (351). Using one
  desktop PC hardware configuration, the query on the FTS3 table returns in
  approximately 0.03 seconds, versus 22.5 for querying the ordinary table.

SELECT count(*) FROM enrondata1 WHERE content MATCH 'linux';  /* 0.03 seconds */
SELECT count(*) FROM enrondata2 WHERE content LIKE '%linux%'; /* 22.5 seconds */



  Of course, the two queries above are not entirely equivalent. For example
  the LIKE query matches rows that contain terms such as "linuxophobe"
  or "EnterpriseLinux" (as it happens, the Enron E-Mail Dataset does not
  actually contain any such terms), whereas the MATCH query on the FTS3 table
  selects only those rows that contain "linux" as a discrete token. Both 
  searches are case-insensitive. The FTS3 table consumes around 2006 MB on
  disk compared to just 1453 MB for the ordinary table. Using the same
  hardware configuration used to perform the SELECT queries above, the FTS3
  table took just under 31 minutes to populate, versus 25 for the ordinary
  table.

1.1. Differences between FTS3 and FTS4




  FTS3 and FTS4 are nearly identical. They share most of their code in common,
  and their interfaces are the same. The differences are:


   FTS4 contains query performance optimizations that may significantly
       improve the performance of full-text queries that contain terms that are
       very common (present in a large percentage of table rows).

   FTS4 supports some additional options that may used with the matchinfo()
       function. 

   Because it stores extra information on disk in two new 
       shadow tables in order to support the performance
       optimizations and extra matchinfo() options, FTS4 tables may consume more
       disk space than the equivalent table created using FTS3. Usually the overhead
       is 1-2% or less, but may be as high as 10% if the documents stored in the
       FTS table are very small. The overhead may be reduced by specifying the
       directive "matchinfo=fts3" as part of the FTS4 table
       declaration, but this comes at the expense of sacrificing some of the
       extra supported matchinfo() options.

   FTS4 provides hooks (the compress and uncompress 
       options) allowing data to be stored in a compressed 
       form, reducing disk usage and IO.



  FTS4 is an enhancement to FTS3. 
  FTS3 has been available since SQLite version 3.5.0 (2007-09-04)
  The enhancements for FTS4 were added with SQLite version 3.7.4
  (2010-12-07).


  Which module, FTS3 or FTS4, should you use in your application?  FTS4 is
  sometimes significantly faster than FTS3, even orders of magnitude faster
  depending on the query, though in the common case the performance of the two
  modules is similar. FTS4 also offers the enhanced matchinfo() outputs which
  can be useful in ranking the results of a MATCH operation.  On the
  other hand, in the absence of a matchinfo=fts3 directive FTS4 requires a little
  more disk space than FTS3, though only a percent of two in most cases.


  For newer applications, FTS4 is recommended; though if compatibility with older 
  versions of SQLite is important, then FTS3 will usually serve just as well.  

1.2. Creating and Destroying FTS Tables


  Like other virtual table types, new FTS tables are created using a 
  CREATE VIRTUAL TABLE statement. The module name, which follows
  the USING keyword, is either "fts3" or "fts4". The virtual table module arguments may
  be left empty, in which case an FTS table with a single user-defined 
  column named "content" is created. Alternatively, the module arguments
  may be passed a list of comma separated column names. 


  If column names are explicitly provided for the FTS table as part of
  the CREATE VIRTUAL TABLE statement, then a datatype name may be optionally 
  specified for each column. This is pure syntactic sugar, the
  supplied typenames are not used by FTS or the SQLite core for any
  purpose. The same applies to any constraints specified along with an
  FTS column name - they are parsed but not used or recorded by the system
  in any way.

-- Create an FTS table named "data" with one column - "content":
CREATE VIRTUAL TABLE data USING fts3();

-- Create an FTS table named "pages" with three columns:
CREATE VIRTUAL TABLE pages USING fts4(title, keywords, body);

-- Create an FTS table named "mail" with two columns. Datatypes
-- and column constraints are specified along with each column. These
-- are completely ignored by FTS and SQLite. 
CREATE VIRTUAL TABLE mail USING fts3(
  subject VARCHAR(256) NOT NULL,
  body TEXT CHECK(length(body)&lt;10240)
);



  As well as a list of columns, the module arguments passed to a CREATE
  VIRTUAL TABLE statement used to create an FTS table may be used to specify
  a tokenizer. This is done by specifying a string of the form
  "tokenize=&lt;tokenizer name&gt; &lt;tokenizer args&gt;" in place of a column
  name, where &lt;tokenizer name&gt; is the name of the tokenizer to use and
  &lt;tokenizer args&gt; is an optional list of whitespace separated qualifiers
  to pass to the tokenizer implementation. A tokenizer specification may be
  placed anywhere in the column list, but at most one tokenizer declaration is
  allowed for each CREATE VIRTUAL TABLE statement. See below for a 
  detailed description of using (and, if necessary, implementing) a tokenizer.

-- Create an FTS table named "papers" with two columns that uses
-- the tokenizer "porter".
CREATE VIRTUAL TABLE papers USING fts3(author, document, tokenize=porter);

-- Create an FTS table with a single column - "content" - that uses
-- the "simple" tokenizer.
CREATE VIRTUAL TABLE data USING fts4(tokenize=simple);

-- Create an FTS table with two columns that uses the "icu" tokenizer.
-- The qualifier "en_AU" is passed to the tokenizer implementation
CREATE VIRTUAL TABLE names USING fts3(a, b, tokenize=icu en_AU);



  FTS tables may be dropped from the database using an ordinary DROP TABLE
  statement. For example:

-- Create, then immediately drop, an FTS4 table.
CREATE VIRTUAL TABLE data USING fts4();
DROP TABLE data;


1.3. Populating FTS Tables

  
    FTS tables are populated using INSERT, UPDATE and DELETE
    statements in the same way as ordinary SQLite tables are.

  
    As well as the columns named by the user (or the "content" column if no
    module arguments were specified as part of the CREATE VIRTUAL TABLE 
    statement), each FTS table has a "rowid" column. The rowid of an FTS
    table behaves in the same way as the rowid column of an ordinary SQLite 
    table, except that the values stored in the rowid column of an FTS table 
    remain unchanged if the database is rebuilt using the VACUUM command. 
    For FTS tables, "docid" is allowed as an alias along with the usual "rowid",
    "oid" and "_oid_" identifiers. Attempting to insert or update a row with a 
    docid value that already exists in the table is an error, just as it would 
    be with an ordinary SQLite table.

  
    There is one other subtle difference between "docid" and the normal SQLite
    aliases for the rowid column. Normally, if an INSERT or UPDATE statement 
    assigns discrete values to two or more aliases of the rowid column, SQLite 
    writes the rightmost of such values specified in the INSERT or UPDATE
    statement to the database. However, assigning a non-NULL value to both
    the "docid" and one or more of the SQLite rowid aliases when inserting or
    updating an FTS table is considered an error. See below for an example.

-- Create an FTS table
CREATE VIRTUAL TABLE pages USING fts4(title, body);

-- Insert a row with a specific docid value.
INSERT INTO pages(docid, title, body) VALUES(53, 'Home Page', 'SQLite is a software...');

-- Insert a row and allow FTS to assign a docid value using the same algorithm as
-- SQLite uses for ordinary tables. In this case the new docid will be 54,
-- one greater than the largest docid currently present in the table.
INSERT INTO pages(title, body) VALUES('Download', 'All SQLite source code...');

-- Change the title of the row just inserted.
UPDATE pages SET title = 'Download SQLite' WHERE rowid = 54;

-- Delete the entire table contents.
DELETE FROM pages;

-- The following is an error. It is not possible to assign non-NULL values to both
-- the rowid and docid columns of an FTS table.
INSERT INTO pages(rowid, docid, title, body) VALUES(1, 2, 'A title', 'A document body');


  
    To support full-text queries, FTS maintains an inverted index that maps
    from each unique term or word that appears in the dataset to the locations
    in which it appears within the table contents. For the curious, a 
    complete description of the data structure used to store
    this index within the database file appears below. A feature of
    this data structure is that at any time the database may contain not
    one index b-tree, but several different b-trees that are incrementally
    merged as rows are inserted, updated and deleted. This technique improves 
    performance when writing to an FTS table, but causes some overhead for
    full-text queries that use the index. Evaluating the special "optimize" command, 
    an SQL statement of the
    form "INSERT INTO &lt;fts-table&gt;(&lt;fts-table&gt;) VALUES('optimize')",
    causes FTS to merge all existing index b-trees into a single large
    b-tree containing the entire index. This can be an expensive operation,
    but may speed up future queries. 

  
    For example, to optimize the full-text index for an FTS table named
    "docs":

-- Optimize the internal structure of FTS table "docs".
INSERT INTO docs(docs) VALUES('optimize');


  
    The statement above may appear syntactically incorrect to some. Refer to
    the section describing the simple fts queries for an explanation.

  
    There is another, deprecated, method for invoking the optimize 
    operation using a SELECT statement. New code should use statements
    similar to the INSERT above to optimize FTS structures.


1.4. Simple FTS Queries


  As for all other SQLite tables, virtual or otherwise, data is retrieved
  from FTS tables using a SELECT statement.


  FTS tables can be queried efficiently using SELECT statements of two
  different forms:


  
    Query by rowid. If the WHERE clause of the SELECT statement
    contains a sub-clause of the form "rowid = ?", where ? is an SQL expression,
    FTS is able to retrieve the requested row directly using the equivalent 
    of an SQLite INTEGER PRIMARY KEY index.

  
    Full-text query. If the WHERE clause of the SELECT statement contains
    a sub-clause of the form "&lt;column&gt; MATCH ?", FTS is able to use 
    the built-in full-text index to restrict the search to those documents 
    that match the full-text query string specified as the right-hand operand
    of the MATCH clause.



  If neither of these two query strategies can be used, all
  queries on FTS tables are implemented using a linear scan of the entire
  table. If the table contains large amounts of data, this may be an 
  impractical approach (the first example on this page shows that a linear
  scan of 1.5 GB of data takes around 30 seconds using a modern PC).

-- The examples in this block assume the following FTS table:
CREATE VIRTUAL TABLE mail USING fts3(subject, body);

SELECT * FROM mail WHERE rowid = 15;                -- Fast. Rowid lookup.
SELECT * FROM mail WHERE body MATCH 'sqlite';       -- Fast. Full-text query.
SELECT * FROM mail WHERE mail MATCH 'search';       -- Fast. Full-text query.
SELECT * FROM mail WHERE rowid BETWEEN 15 AND 20;   -- Fast. Rowid lookup.
SELECT * FROM mail WHERE subject = 'database';      -- Slow. Linear scan.
SELECT * FROM mail WHERE subject MATCH 'database';  -- Fast. Full-text query.



  In all of the full-text queries above, the right-hand operand of the MATCH
  operator is a string consisting of a single term. In this case, the MATCH
  expression evaluates to true for all documents that contain one or more 
  instances of the specified word ("sqlite", "search" or "database", depending 
  on which example you look at). Specifying a single term as the right-hand
  operand of the MATCH operator results in the simplest and most common type 
  of full-text query possible. However more complicated queries are possible,
  including phrase searches, term-prefix searches and searches for documents 
  containing combinations of terms occurring within a defined proximity of each
  other. The various ways in which the full-text index may be queried are
  described below.


  Normally, full-text queries are case-insensitive. However, this
  is dependent on the specific tokenizer used by the FTS table
  being queried. Refer to the section on tokenizers for details.


  The paragraph above notes that a MATCH operator with a simple term as the
  right-hand operand evaluates to true for all documents that contain the
  specified term. In this context, the "document" may refer to either the 
  data stored in a single column of a row of an FTS table, or to the contents
  of all columns in a single row, depending on the identifier used as the
  left-hand operand to the MATCH operator. If the identifier specified as
  the left-hand operand of the MATCH operator is an FTS table column name,
  then the document that the search term must be contained in is the value
  stored in the specified column. However, if the identifier is the name
  of the FTS table itself, then the MATCH operator evaluates to true
  for each row of the FTS table for which any column contains the search 
  term. The following example demonstrates this:

-- Example schema
CREATE VIRTUAL TABLE mail USING fts3(subject, body);

-- Example table population
INSERT INTO mail(docid, subject, body) VALUES(1, 'software feedback', 'found it too slow');
INSERT INTO mail(docid, subject, body) VALUES(2, 'software feedback', 'no feedback');
INSERT INTO mail(docid, subject, body) VALUES(3, 'slow lunch order',  'was a software problem');

-- Example queries
SELECT * FROM mail WHERE subject MATCH 'software';    -- Selects rows 1 and 2
SELECT * FROM mail WHERE body    MATCH 'feedback';    -- Selects row 2
SELECT * FROM mail WHERE mail    MATCH 'software';    -- Selects rows 1, 2 and 3
SELECT * FROM mail WHERE mail    MATCH 'slow';        -- Selects rows 1 and 3

  

  At first glance, the final two full-text queries in the example above seem
  to be syntactically incorrect, as there is a table name ("mail") used as
  an SQL expression. The reason this is acceptable is that each FTS table
  actually has a HIDDEN column with the same name
  as the table itself (in this case, "mail"). The value stored in this
  column is not meaningful to the application, but can be used as the 
  left-hand operand to a MATCH operator. This special column may also be
  passed as an argument to the FTS auxiliary functions.


  The following example illustrates the above. The expressions "docs", 
  "docs.docs" and "main.docs.docs" all refer to column "docs". However, the 
  expression "main.docs" does not refer to any column. It could be used to 
  refer to a table, but a table name is not allowed in the context in which
  it is used below.

-- Example schema
CREATE VIRTUAL TABLE docs USING fts4(content);

-- Example queries
SELECT * FROM docs WHERE docs MATCH 'sqlite';              -- OK.
SELECT * FROM docs WHERE docs.docs MATCH 'sqlite';         -- OK.
SELECT * FROM docs WHERE main.docs.docs MATCH 'sqlite';    -- OK.
SELECT * FROM docs WHERE main.docs MATCH 'sqlite';         -- Error.

 
1.5. Summary


  From the users point of view, FTS tables are similar to ordinary SQLite
  tables in many ways. Data may be added to, modified within and removed 
  from FTS tables using the INSERT, UPDATE and DELETE commands just as 
  it may be with ordinary tables. Similarly, the SELECT command may be used 
  to query data. The following list summarizes the differences between FTS
  and ordinary tables:


   
    As with all virtual table types, it is not possible to create indices or
    triggers attached to FTS tables. Nor is it possible to use the ALTER TABLE
    command to add extra columns to FTS tables (although it is possible to use
    ALTER TABLE to rename an FTS table).

   
    Data-types specified as part of the "CREATE VIRTUAL TABLE" statement
    used to create an FTS table are ignored completely. Instead of the
    normal rules for applying type affinity to inserted values, all
    values inserted into FTS table columns (except the special rowid
    column) are converted to type TEXT before being stored.

   
    FTS tables permit the special alias "docid" to be used to refer to the
    rowid column supported by all virtual tables.

   
    The FTS MATCH operator is supported for queries based on the built-in
    full-text index. 

   
    The FTS auxiliary functions, snippet(), offsets(), and matchinfo() are 
    available to support full-text queries.

  
    

    Every FTS table has a hidden column with the 
    same name as the table itself. The value contained in each row for the
    hidden column is a blob that is only useful as the left operand of a
    MATCH operator, or as the left-most argument to one
    of the FTS auxiliary functions.
    





2. Compiling and Enabling FTS3 and FTS4


  Although FTS3 and FTS4 are included with the SQLite core source code, they are not
  enabled by default. To build SQLite with FTS functionality enabled, define
  the preprocessor macro SQLITE_ENABLE_FTS3 when compiling. New applications
  should also define the SQLITE_ENABLE_FTS3_PARENTHESIS macro to enable the
  enhanced query syntax (see below). Usually, this is done by adding the 
  following two switches to the compiler command line:

-DSQLITE_ENABLE_FTS3
-DSQLITE_ENABLE_FTS3_PARENTHESIS



  Note that enabling FTS3 also makes FTS4 available.  There is not a separate
  SQLITE_ENABLE_FTS4 compile-time option.  A build of SQLite either supports
  both FTS3 and FTS4 or it supports neither.


  If using the amalgamation autoconf based build system, setting the CPPFLAGS
  environment variable while running the 'configure' script is an easy
  way to set these macros. For example, the following command:

CPPFLAGS="-DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS" ./configure &lt;configure options&gt;



  where &lt;configure options&gt; are those options normally passed to
  the configure script, if any.


  Because FTS3 and FTS4 are virtual tables, The SQLITE_ENABLE_FTS3 compile-time option
  is incompatible with the SQLITE_OMIT_VIRTUALTABLE option.


  If a build of SQLite does not include the FTS modules, then any attempt to prepare an
  SQL statement to create an FTS3 or FTS4 table or to drop or access an existing 
  FTS table in any way will fail. The error message returned will be similar 
  to "no such module: ftsN" (where N is either 3 or 4).


  If the C version of the ICU library
  is available, then FTS may also be compiled with the SQLITE_ENABLE_ICU
  pre-processor macro defined. Compiling with this macro enables an FTS
  tokenizer that uses the ICU library to split a document into terms
  (words) using the conventions for a specified language and locale.

-DSQLITE_ENABLE_ICU

  


3. Full-text Index Queries


  The most useful thing about FTS tables is the queries that may be 
  performed using the built-in full-text index. Full-text queries are 
  performed by specifying a clause of the form 
  "&lt;column&gt; MATCH &lt;full-text query expression&gt;" as part of the WHERE 
  clause of a SELECT statement that reads data from an FTS table. 
  Simple FTS queries that return all documents that 
  contain a given term are described above. In that discussion the right-hand
  operand of the MATCH operator was assumed to be a string consisting of a
  single term. This section describes the more complex query types supported 
  by FTS tables, and how they may be utilized by specifying a more
  complex query expression as the right-hand operand of a MATCH operator.


  FTS tables support three basic query types:


  

  Token or token prefix queries. 
    An FTS table may be queried for all documents that contain a specified
    term (the simple case described above), or for
    all documents that contain a term with a specified prefix. As we have
    seen, the query expression for a specific term is simply the term itself.
    The query expression used to search for a term prefix is the prefix
    itself with a '*' character appended to it. For example:


-- Virtual table declaration
CREATE VIRTUAL TABLE docs USING fts3(title, body);

-- Query for all documents containing the term "linux":
SELECT * FROM docs WHERE docs MATCH 'linux';

-- Query for all documents containing a term with the prefix "lin". This will match
-- all documents that contain "linux", but also those that contain terms "linear",
--"linker", "linguistic" and so on.
SELECT * FROM docs WHERE docs MATCH 'lin*';



  
    Normally, a token or token prefix query is matched against the FTS table 
    column specified as the left-hand side of the MATCH operator. Or, if the
    special column with the same name as the FTS table itself is specified,
    against all columns. This may be overridden by specifying a column-name
    followed by a ":" character before a basic term query. There may be space
    between the ":" and the term to query for, but not between the column-name
    and the ":" character. For example:

   
-- Query the database for documents for which the term "linux" appears in
-- the document title, and the term "problems" appears in either the title
-- or body of the document.
SELECT * FROM docs WHERE docs MATCH 'title:linux problems';

-- Query the database for documents for which the term "linux" appears in
-- the document title, and the term "driver" appears in the body of the document
-- ("driver" may also appear in the title, but this alone will not satisfy the
-- query criteria).
SELECT * FROM docs WHERE body MATCH 'title:linux driver';



  
    If the FTS table is an FTS4 table (not FTS3), a token may also be prefixed
    with a "&#94;" character. In this case, in order to match the token must
    appear as the very first token in any column of the matching row. Examples:


-- All documents for which "linux" is the first token of at least one
-- column.
SELECT * FROM docs WHERE docs MATCH '&#94;linux';

-- All documents for which the first token in column "title" begins with "lin".
SELECT * FROM docs WHERE body MATCH 'title: &#94;lin*';





  Phrase queries.
    A phrase query is a query that retrieves all documents that contain a
    nominated set of terms or term prefixes in a specified order with no
    intervening tokens. Phrase queries are specified by enclosing a space
    separated sequence of terms or term prefixes in double quotes (").
    For example:


-- Query for all documents that contain the phrase "linux applications".
SELECT * FROM docs WHERE docs MATCH '"linux applications"';

-- Query for all documents that contain a phrase that matches "lin* app*". As well as
-- "linux applications", this will match common phrases such as "linoleum appliances"
-- or "link apprentice".
SELECT * FROM docs WHERE docs MATCH '"lin* app*"';





  NEAR queries. 
    A NEAR query is a query that returns documents that contain a two or
    more nominated terms or phrases within a specified proximity of each 
    other (by default with 10 or less intervening terms). A NEAR query is 
    specified by putting the keyword "NEAR" between two phrase, token or 
    token prefix queries. To specify a proximity other than the default,
    an operator of the form "NEAR/&lt;N&gt;" may be used, where
    &lt;N&gt; is the maximum number of intervening terms allowed.
    For example:


-- Virtual table declaration.
CREATE VIRTUAL TABLE docs USING fts4();

-- Virtual table data.
INSERT INTO docs VALUES('SQLite is an ACID compliant embedded relational database management system');

-- Search for a document that contains the terms "sqlite" and "database" with
-- not more than 10 intervening terms. This matches the only document in
-- table docs (since there are only six terms between "SQLite" and "database" 
-- in the document).
SELECT * FROM docs WHERE docs MATCH 'sqlite NEAR database';

-- Search for a document that contains the terms "sqlite" and "database" with
-- not more than 6 intervening terms. This also matches the only document in
-- table docs. Note that the order in which the terms appear in the document
-- does not have to be the same as the order in which they appear in the query.
SELECT * FROM docs WHERE docs MATCH 'database NEAR/6 sqlite';

-- Search for a document that contains the terms "sqlite" and "database" with
-- not more than 5 intervening terms. This query matches no documents.
SELECT * FROM docs WHERE docs MATCH 'database NEAR/5 sqlite';

-- Search for a document that contains the phrase "ACID compliant" and the term
-- "database" with not more than 2 terms separating the two. This matches the
-- document stored in table docs.
SELECT * FROM docs WHERE docs MATCH 'database NEAR/2 "ACID compliant"';

-- Search for a document that contains the phrase "ACID compliant" and the term
-- "sqlite" with not more than 2 terms separating the two. This also matches
-- the only document stored in table docs.
SELECT * FROM docs WHERE docs MATCH '"ACID compliant" NEAR/2 sqlite';



  
    More than one NEAR operator may appear in a single query. In this case each
    pair of terms or phrases separated by a NEAR operator must appear within the
    specified proximity of each other in the document. Using the same table and
    data as in the block of examples above:



-- The following query selects documents that contains an instance of the term 
-- "sqlite" separated by two or fewer terms from an instance of the term "acid",
-- which is in turn separated by two or fewer terms from an instance of the term
-- "relational".
SELECT * FROM docs WHERE docs MATCH 'sqlite NEAR/2 acid NEAR/2 relational';

-- This query matches no documents. There is an instance of the term "sqlite" with
-- sufficient proximity to an instance of "acid" but it is not sufficiently close
-- to an instance of the term "relational".
SELECT * FROM docs WHERE docs MATCH 'acid NEAR/2 sqlite NEAR/2 relational';



  Phrase and NEAR queries may not span multiple columns within a row.


  The three basic query types described above may be used to query the full-text
  index for the set of documents that match the specified criteria. Using the
  FTS query expression language it is possible to perform various set 
  operations on the results of basic queries. There are currently three 
  supported operations:


   The AND operator determines the intersection of two sets of documents.

   The OR operator calculates the union of two sets of documents.

   The NOT operator (or, if using the standard syntax, a unary "-" operator)
       may be used to compute the relative complement of one set of
       documents with respect to another.



  The FTS modules may be compiled to use one of two slightly different versions
  of the full-text query syntax, the "standard" query syntax and the "enhanced" 
  query syntax. The basic term, term-prefix, phrase and NEAR queries described 
  above are the same in both versions of the syntax. The way in which set 
  operations are specified is slightly different. The following two sub-sections 
  describe the part of the two query syntaxes that pertains to set operations. 
  Refer to the description of how to compile fts for compilation notes.


3.1. 
  Set Operations Using The Enhanced Query Syntax


  The enhanced query syntax supports the AND, OR and NOT binary set operators.
  Each of the two operands to an operator may be a basic FTS query, or the
  result of another AND, OR or NOT set operation. Operators must be entered
  using capital letters. Otherwise, they are interpreted as basic term queries
  instead of set operators.


  The AND operator may be implicitly specified. If two basic queries appear 
  with no operator separating them in an FTS query string, the results are
  the same as if the two basic queries were separated by an AND operator.
  For example, the query expression "implicit operator" is a more succinct
  version of "implicit AND operator".

-- Virtual table declaration
CREATE VIRTUAL TABLE docs USING fts3();

-- Virtual table data
INSERT INTO docs(docid, content) VALUES(1, 'a database is a software system');
INSERT INTO docs(docid, content) VALUES(2, 'sqlite is a software system');
INSERT INTO docs(docid, content) VALUES(3, 'sqlite is a database');

-- Return the set of documents that contain the term "sqlite", and the
-- term "database". This query will return the document with docid 3 only.
SELECT * FROM docs WHERE docs MATCH 'sqlite AND database';

-- Again, return the set of documents that contain both "sqlite" and
-- "database". This time, use an implicit AND operator. Again, document
-- 3 is the only document matched by this query. 
SELECT * FROM docs WHERE docs MATCH 'database sqlite';

-- Query for the set of documents that contains either "sqlite" or "database".
-- All three documents in the database are matched by this query.
SELECT * FROM docs WHERE docs MATCH 'sqlite OR database';

-- Query for all documents that contain the term "database", but do not contain
-- the term "sqlite". Document 1 is the only document that matches this criteria.
SELECT * FROM docs WHERE docs MATCH 'database NOT sqlite';

-- The following query matches no documents. Because "and" is in lowercase letters,
-- it is interpreted as a basic term query instead of an operator. Operators must
-- be specified using capital letters. In practice, this query will match any documents
-- that contain each of the three terms "database", "and" and "sqlite" at least once.
-- No documents in the example data above match this criteria.
SELECT * FROM docs WHERE docs MATCH 'database and sqlite';



  The examples above all use basic full-text term queries as both operands of 
  the set operations demonstrated. Phrase and NEAR queries may also be used,
  as may the results of other set operations. When more than one set operation
  is present in an FTS query, the precedence of operators is as follows:


  OperatorEnhanced Query Syntax Precedence
  NOT  Highest precedence (tightest grouping).
  AND 
  OR   Lowest precedence (loosest grouping).



  When using the enhanced query syntax, parenthesis may be used to override
  the default precedence of the various operators. For example:

-- Return the docid values associated with all documents that contain the
-- two terms "sqlite" and "database", and/or contain the term "library".
SELECT docid FROM docs WHERE docs MATCH 'sqlite AND database OR library';

-- This query is equivalent to the above.
SELECT docid FROM docs WHERE docs MATCH 'sqlite AND database'
  UNION
SELECT docid FROM docs WHERE docs MATCH 'library';

-- Query for the set of documents that contains the term "linux", and at least
-- one of the phrases "sqlite database" and "sqlite library".
SELECT docid FROM docs WHERE docs MATCH '("sqlite database" OR "sqlite library") AND linux';

-- This query is equivalent to the above.
SELECT docid FROM docs WHERE docs MATCH 'linux'
  INTERSECT
SELECT docid FROM (
  SELECT docid FROM docs WHERE docs MATCH '"sqlite library"'
    UNION
  SELECT docid FROM docs WHERE docs MATCH '"sqlite database"'
);



3.2. Set Operations Using The Standard Query Syntax


  FTS query set operations using the standard query syntax are similar, but
  not identical, to set operations with the enhanced query syntax. There
  are four differences, as follows:


   Only the implicit version of the AND operator is supported.
    Specifying the string "AND" as part of a standard query syntax query is
    interpreted as a term query for the set of documents containing the term 
    "and".



   Parenthesis are not supported.



   The NOT operator is not supported. Instead of the NOT 
    operator, the standard query syntax supports a unary "-" operator that
    may be applied to basic term and term-prefix queries (but not to phrase
    or NEAR queries). A term or term-prefix that has a unary "-" operator
    attached to it may not appear as an operand to an OR operator. An FTS
    query may not consist entirely of terms or term-prefix queries with unary
    "-" operators attached to them.


-- Search for the set of documents that contain the term "sqlite" but do
-- not contain the term "database".
SELECT * FROM docs WHERE docs MATCH 'sqlite -database';



   The relative precedence of the set operations is different. 
   In particular, using the standard query syntax the "OR" operator has a
   higher precedence than "AND". The precedence of operators when using the
   standard query syntax is: 



  OperatorStandard Query Syntax Precedence
  Unary "-"  Highest precedence (tightest grouping).
  OR  
  AND  Lowest precedence (loosest grouping).



  The following example illustrates precedence of operators using the standard 
  query syntax:


-- Search for documents that contain at least one of the terms "database"
-- and "sqlite", and also contain the term "library". Because of the differences
-- in operator precedences, this query would have a different interpretation using
-- the enhanced query syntax.
SELECT * FROM docs WHERE docs MATCH 'sqlite OR database library';




4. Auxiliary Functions - Snippet, Offsets and Matchinfo


  The FTS3 and FTS4 modules provide three special SQL scalar functions that may be useful
  to the developers of full-text query systems: "snippet", "offsets" and
  "matchinfo". The purpose of the "snippet" and "offsets" functions is to allow
  the user to identify the location of queried terms in the returned documents.
  The "matchinfo" function provides the user with metrics that may be useful
  for filtering or sorting query results according to relevance.


  The first argument to all three special SQL scalar functions
  must be the FTS hidden column of the FTS table that the function is
  applied to.  The FTS hidden column is an automatically-generated column found on
  all FTS tables that has the same name as the FTS table itself.
  For example, given an FTS table named "mail":

SELECT offsets(mail) FROM mail WHERE mail MATCH &lt;full-text query expression&gt;;
SELECT snippet(mail) FROM mail WHERE mail MATCH &lt;full-text query expression&gt;;
SELECT matchinfo(mail) FROM mail WHERE mail MATCH &lt;full-text query expression&gt;;



  The three auxiliary functions are only useful within a SELECT statement that
  uses the FTS table's full-text index. If used within a SELECT that uses
  the "query by rowid" or "linear scan" strategies, then the snippet and
  offsets both return an empty string, and the matchinfo function returns
  a blob value zero bytes in size.


  All three auxiliary functions extract a set of "matchable phrases" from
  the FTS query expression to work with. The set of matchable phrases for
  a given query consists of all phrases (including unquoted tokens and
  token prefixes) in the expression except those that are prefixed with
  a unary "-" operator (standard syntax) or are part of a sub-expression 
  that is used as the right-hand operand of a NOT operator.


  With the following provisos, each series of tokens in the FTS table that
  matches one of the matchable phrases in the query expression is known as a
  "phrase match":


   If a matchable phrase is part of a series of phrases connected by
       NEAR operators in the FTS query expression, then each phrase match
       must be sufficiently close to other phrase matches of the relevant
       types to satisfy the NEAR condition.

   If the matchable phrase in the FTS query is restricted to matching
       data in a specified FTS table column, then only phrase matches that 
       occur within that column are considered.

 


4.1. The Offsets Function


  For a SELECT query that uses the full-text index, the offsets() function 
  returns a text value containing a series of space-separated integers. For
  each term in each phrase match of the current row, 
  there are four integers in the returned list. Each set of four integers is 
  interpreted as follows:


  Integer Interpretation
  0 
      The column number that the term instance occurs in (0 for the
          leftmost column of the FTS table, 1 for the next leftmost, etc.).
  1
      The term number of the matching term within the full-text query
          expression. Terms within a query expression are numbered starting
          from 0 in the order that they occur.
  2
      The byte offset of the matching term within the column.
  3
      The size of the matching term in bytes.



  The following block contains examples that use the offsets function.

CREATE VIRTUAL TABLE mail USING fts3(subject, body);
INSERT INTO mail VALUES('hello world', 'This message is a hello world message.');
INSERT INTO mail VALUES('urgent: serious', 'This mail is seen as a more serious mail');

-- The following query returns a single row (as it matches only the first
-- entry in table "mail". The text returned by the offsets function is
-- "0 0 6 5 1 0 24 5".
--
-- The first set of four integers in the result indicate that column 0
-- contains an instance of term 0 ("world") at byte offset 6. The term instance
-- is 5 bytes in size. The second set of four integers shows that column 1
-- of the matched row contains an instance of term 0 ("world") at byte offset
-- 24. Again, the term instance is 5 bytes in size.
SELECT offsets(mail) FROM mail WHERE mail MATCH 'world';

-- The following query returns also matches only the first row in table "mail".
-- In this case the returned text is "1 0 5 7 1 0 30 7".
SELECT offsets(mail) FROM mail WHERE mail MATCH 'message';

-- The following query matches the second row in table "mail". It returns the
-- text "1 0 28 7 1 1 36 4". Only those occurrences of terms "serious" and "mail"
-- that are part of an instance of the phrase "serious mail" are identified; the
-- other occurrences of "serious" and "mail" are ignored.
SELECT offsets(mail) FROM mail WHERE mail MATCH '"serious mail"';




4.2. The Snippet Function


  The snippet function is used to create formatted fragments of document text
  for display as part of a full-text query results report. The snippet function 
  may be passed between one and six arguments, as follows:


  Argument Default Value Description
  0 N/A
       The first argument to the snippet function must always be the FTS hidden column
           of the FTS table being queried and from which the snippet is to be taken.  The
           FTS hidden column is an automatically generated column with the same name as the
           FTS table itself.
  1 "&lt;b&gt;"
       The "start match" text.
  2 "&lt;/b&gt;"
       The "end match" text.
  3 "&lt;b&gt;...&lt;/b&gt;"
       The "ellipses" text.
  4 -1
       The FTS table column number to extract the returned fragments of
           text from. Columns are numbered from left to right starting with
           zero. A negative value indicates that the text may be extracted
           from any column.
  5 -15
       The absolute value of this integer argument is used as the 
           (approximate) number of tokens to include in the returned text 
           value. The maximum allowable absolute value is 64. The value of
           this argument is referred to as N in the discussion below.



  The snippet function first attempts to find a fragment of text consisting
  of |N| tokens within the current row that contains at least one phrase 
  match for each matchable phrase matched somewhere in the current row, 
  where |N| is the absolute value of the sixth argument passed to the
  snippet function. If the text stored in a single column contains less than
  |N| tokens, then the entire column value is considered. Text fragments 
  may not span multiple columns.


  If such a text fragment can be found, it is returned with the following
  modifications:


   If the text fragment does not begin at the start of a column value,
       the "ellipses" text is prepended to it.
   If the text fragment does not finish at the end of a column value,
       the "ellipses" text is appended to it.
   For each token in the text fragment that is part of a phrase match,
       the "start match" text is inserted into the fragment before the token,
       and the "end match" text is inserted immediately after it.



  If more than one such fragment can be found, then fragments that contain
  a larger number of "extra" phrase matches are favored. The start of
  the selected text fragment may be moved a few tokens forward or backward
  to attempt to concentrate the phrase matches toward the center of the
  fragment.


  Assuming N is a positive value, if no fragments can be found that
  contain a phrase match corresponding to each matchable phrase, the snippet
  function attempts to find two fragments of approximately N/2 tokens
  that between them contain at least one phrase match for each matchable phrase
  matched by the current row. If this fails, attempts are made to find three
  fragments of N/3 tokens each and finally four N/4 token
  fragments. If a set of four fragments cannot be found that encompasses the
  required phrase matches, the four fragments of N/4 tokens that provide
  the best coverage are selected.


  If N is a negative value, and no single fragment can be found 
  containing the required phrase matches, the snippet function searches
  for two fragments of |N| tokens each, then three, then four. In
  other words, if the specified value of N is negative, the sizes
  of the fragments is not decreased if more than one fragment is required
  to provide the desired phrase match coverage.


  After the M fragments have been located, where M is between
  two and four as described in the paragraphs above, they are joined together
  in sorted order with the "ellipses" text separating them. The three 
  modifications enumerated earlier are performed on the text before it is 
  returned.

Note: In this block of examples, newlines and whitespace characters have
been inserted into the document inserted into the FTS table, and the expected
results described in SQL comments. This is done to enhance readability only,
they would not be present in actual SQLite commands or output.

-- Create and populate an FTS table.
CREATE VIRTUAL TABLE text USING fts4();
INSERT INTO text VALUES('
  During 30 Nov-1 Dec, 2-3oC drops. Cool in the upper portion, minimum temperature 14-16oC 
  and cool elsewhere, minimum temperature 17-20oC. Cold to very cold on mountaintops, 
  minimum temperature 6-12oC. Northeasterly winds 15-30 km/hr. After that, temperature 
  increases. Northeasterly winds 15-30 km/hr.     
');

-- The following query returns the text value:
--
--   "&lt;b&gt;...&lt;/b&gt;cool elsewhere, minimum temperature 17-20oC. &lt;b&gt;Cold&lt;/b&gt; to very 
--    &lt;b&gt;cold&lt;/b&gt; on mountaintops, minimum temperature 6&lt;b&gt;...&lt;/b&gt;".
--
SELECT snippet(text) FROM text WHERE text MATCH 'cold';

-- The following query returns the text value:
--
--   "...the upper portion, &#91;minimum&#93; &#91;temperature&#93; 14-16oC and cool elsewhere,
--    &#91;minimum&#93; &#91;temperature&#93; 17-20oC. Cold..."
--
SELECT snippet(text, '&#91;', '&#93;', '...') FROM text WHERE text MATCH '"min* tem*"'



4.3. The Matchinfo Function


  The matchinfo function returns a blob value. If it is used within a query
  that does not use the full-text index (a "query by rowid" or "linear scan"),
  then the blob is zero bytes in size. Otherwise, the blob consists of zero
  or more 32-bit unsigned integers in machine byte-order. The exact number
  of integers in the returned array depends on both the query and the value
  of the second argument (if any) passed to the matchinfo function.


  The matchinfo function is called with either one or two arguments. As for
  all auxiliary functions, the first argument must be the special 
  FTS hidden column. The second argument, if it is specified, must be a text value
  comprised only of the characters 'p', 'c', 'n', 'a', 'l', 's', 'x', 'y' and 'b'.
  If no second argument is explicitly supplied, it defaults to "pcx". The
  second argument is referred to as the "format string" below.


  Characters in the matchinfo format string are processed from left to right. 
  Each character in the format string causes one or more 32-bit unsigned
  integer values to be added to the returned array. The "values" column in
  the following table contains the number of integer values appended to the
  output buffer for each supported format string character. In the formula
  given, cols is the number of columns in the FTS table, and 
  phrases is the number of matchable phrases in 
  the query. 


  CharacterValuesDescription
  p 1 The number of matchable phrases in the query.
  c 1 The number of user defined columns in the FTS
    table (i.e. not including the docid or the FTS hidden column).
  x 3 * cols * phrases 
    

      For each distinct combination of a phrase and table column, the
      following three values:
      
         In the current row, the number of times the phrase appears in 
             the column.
         The total number of times the phrase appears in the column in
             all rows in the FTS table.
         The total number of rows in the FTS table for which the 
             column contains at least one instance of the phrase.
      
      The first set of three values corresponds to the left-most column
      of the table (column 0) and the left-most matchable phrase in the
      query (phrase 0). If the table has more than one column, the second
      set of three values in the output array correspond to phrase 0 and
      column 1. Followed by phrase 0, column 2 and so on for all columns of
      the table. And so on for phrase 1, column 0, then phrase 1, column 1
      etc. In other words, the data for occurrences of phrase p in
      column c may be found using the following formula:

          hits_this_row  = array&#91;3 * (c + p*cols) + 0&#93;
          hits_all_rows  = array&#91;3 * (c + p*cols) + 1&#93;
          docs_with_hits = array&#91;3 * (c + p*cols) + 2&#93;

  ycols * phrases 
    

      For each distinct combination of a phrase and table column, the
      number of usable phrase matches that appear in the column. This is
      usually identical to the first value in each set of three returned by the
      matchinfo 'x' flag. However, the number of hits reported by the
      'y' flag is zero for any phrase that is part of a sub-expression
      that does not match the current row. This makes a difference for
      expressions that contain AND operators that are descendants of OR
      operators. For example, consider the expression: 

          a OR (b AND c)

      and the document:

          "a c d"

      The matchinfo 'x' flag would report a single hit for the phrases "a" and "c".
      However, the 'y' directive reports the number of hits for "c" as zero, as
      it is part of a sub-expression that does not match the document - (b AND c).
      For queries that do not contain AND operators descended from OR
      operators, the result values returned by 'y' are always the same as
      those returned by 'x'.


      The first value in the array of integer values corresponds to the
      leftmost column of the table (column 0) and the first phrase in the query
      (phrase 0). The values corresponding to other column/phrase combinations
      may be located using the following formula:


          hits_for_phrase_p_column_c  = array&#91;c + p*cols&#93;

      For queries that use OR expressions, or those that use LIMIT or return
      many rows, the 'y' matchinfo option may be faster than 'x'.

b((cols+31)/32) * phrases 



  The matchinfo 'b' flag provides similar information to the
  matchinfo 'y' flag, but in a more
  compact form. Instead of the precise number of hits, 'b' provides a single
  boolean flag for each phrase/column combination. If the phrase is present in
  the column at least once (i.e. if the corresponding integer output of 'y' would
  be non-zero), the corresponding flag is set. Otherwise cleared.


  If the table has 32 or fewer columns, a single unsigned integer is output for
  each phrase in the query. The least significant bit of the integer is set if the
  phrase appears at least once in column 0. The second least significant bit is
  set if the phrase appears once or more in column 1. And so on.


  If the table has more than 32 columns, an extra integer is added to the output
  of each phrase for each extra 32 columns or part thereof. Integers
  corresponding to the same phrase are clumped together. For example, if a table
  with 45 columns is queried for two phrases, 4 integers are output. The first
  corresponds to phrase 0 and columns 0-31 of the table. The second integer
  contains data for phrase 0 and columns 32-44, and so on.


  For example, if nCol is the number of columns in the table, to determine if
  phrase p is present in column c:


    p_is_in_c = array&#91;p * ((nCol+31)/32)&#93; & (1 &lt;&lt; (c % 32))


  n 1 The number of rows in the FTS4 table. This value is
    only available when querying FTS4 tables, not FTS3.
  a cols For each column, the average number of
    tokens in the text values stored in the column (considering all rows in
    the FTS4 table). This value is only available when querying FTS4 tables,
    not FTS3.  
  l cols 
    For each column, the length of the value stored in the current row of the
    FTS4 table, in tokens.  This value is only available when querying
    FTS4 tables, not FTS3. And only if the "matchinfo=fts3" directive was not
    specified as part of the "CREATE VIRTUAL TABLE" statement used to create
    the FTS4 table.
  s cols For each column, the length of the longest 
    subsequence of phrase matches that the column value has in common
    with the query text. For example, if a table column contains the text
    'a b c d e' and the query is 'a c "d e"', then the length of the longest
    common subsequence is 2 (phrase "c" followed by phrase "d e").
    
    



  For example:

-- Create and populate an FTS4 table with two columns:
CREATE VIRTUAL TABLE t1 USING fts4(a, b);
INSERT INTO t1 VALUES('transaction default models default', 'Non transaction reads');
INSERT INTO t1 VALUES('the default transaction', 'these semantics present');
INSERT INTO t1 VALUES('single request', 'default data');

-- In the following query, no format string is specified and so it defaults
-- to "pcx". It therefore returns a single row consisting of a single blob
-- value 80 bytes in size (20 32-bit integers - 1 for "p", 1 for "c" and
-- 3*2*3 for "x"). If each block of 4 bytes in the blob is interpreted 
-- as an unsigned integer in machine byte-order, the values will be:
--
--     3 2  1 3 2  0 1 1  1 2 2  0 1 1  0 0 0  1 1 1
--
-- The row returned corresponds to the second entry inserted into table t1.
-- The first two integers in the blob show that the query contained three
-- phrases and the table being queried has two columns. The next block of
-- three integers describes column 0 (in this case column "a") and phrase
-- 0 (in this case "default"). The current row contains 1 hit for "default"
-- in column 0, of a total of 3 hits for "default" that occur in column
-- 0 of any table row. The 3 hits are spread across 2 different rows.
--
-- The next set of three integers (0 1 1) pertain to the hits for "default"
-- in column 1 of the table (0 in this row, 1 in all rows, spread across 
-- 1 rows).
--
SELECT matchinfo(t1) FROM t1 WHERE t1 MATCH 'default transaction "these semantics"';

-- The format string for this query is "ns". The output array will therefore
-- contain 3 integer values - 1 for "n" and 2 for "s". The query returns
-- two rows (the first two rows in the table match). The values returned are:
--
--     3  1 1
--     3  2 0
--
-- The first value in the matchinfo array returned for both rows is 3 (the 
-- number of rows in the table). The following two values are the lengths 
-- of the longest common subsequence of phrase matches in each column.
SELECT matchinfo(t1, 'ns') FROM t1 WHERE t1 MATCH 'default transaction';



  The matchinfo function is much faster than either the snippet or offsets
  functions. This is because the implementation of both snippet and offsets
  is required to retrieve the documents being analyzed from disk, whereas
  all data required by matchinfo is available as part of the same portions
  of the full-text index that are required to implement the full-text query
  itself. This means that of the following two queries, the first may be
  an order of magnitude faster than the second:

SELECT docid, matchinfo(tbl) FROM tbl WHERE tbl MATCH &lt;query expression&gt;;
SELECT docid, offsets(tbl) FROM tbl WHERE tbl MATCH &lt;query expression&gt;;



  The matchinfo function provides all the information required to calculate
  probabilistic "bag-of-words" relevancy scores such as 
  Okapi BM25/BM25F that may
  be used to order results in a full-text search application. Appendix A of this 
  document, "search application tips", contains an example of using the
  matchinfo() function efficiently.


5. Fts4aux - Direct Access to the Full-Text Index


  As of version 3.7.6 (2011-04-12), 
  SQLite includes a new virtual table module called 
  "fts4aux", which can be used to inspect the full-text index of an existing
  FTS table directly. Despite its name, fts4aux works just as well with FTS3
  tables as it does with FTS4 tables. Fts4aux tables are read-only. The only
  way to modify the contents of an fts4aux table is by modifying the
  contents of the associated FTS table. The fts4aux module is automatically
  included in all builds that include FTS.


  An fts4aux virtual table is constructed with one or two arguments.  When
  used with a single argument, that argument is the unqualified name of the
  FTS table that it will be used to access.  To access a table in a different
  database (for example, to create a TEMP fts4aux table that will access an
  FTS3 table in the MAIN database) use the two-argument form and give the
  name of the target database (ex: "main") in the first argument and the name
  of the FTS3/4 table as the second argument.  (The two-argument form of 
  fts4aux was added for SQLite version 3.7.17 (2013-05-20)
  and will throw an error in prior releases.)
  For example:

-- Create an FTS4 table
CREATE VIRTUAL TABLE ft USING fts4(x, y);

-- Create an fts4aux table to access the full-text index for table "ft"
CREATE VIRTUAL TABLE ft_terms USING fts4aux(ft);

-- Create a TEMP fts4aux table accessing the "ft" table in "main"
CREATE VIRTUAL TABLE temp.ft_terms_2 USING fts4aux(main,ft);



  For each term present in the FTS table, there are between 2 and N+1 rows
  in the fts4aux table, where N is the number of user-defined columns in
  the associated FTS table. An fts4aux table always has the same four columns, 
  as follows, from left to right:


  Column NameColumn Contents
  term 
    Contains the text of the term for this row.
  col 
    This column may contain either the text value '*' (i.e. a single 
    character, U+002a) or an integer between 0 and N-1, where N is
    again the number of user-defined columns in the corresponding FTS table.

  documents
    This column always contains an integer value greater than zero.
    
    If the "col" column contains the value '*', then this column
    contains the number of rows of the FTS table that contain at least one
    instance of the term (in any column). If col contains an integer
    value, then this column contains the number of rows of the FTS table that
    contain at least one instance of the term in the column identified by
    the col value. As usual, the columns of the FTS table are numbered
    from left to right, starting with zero.

  occurrences
    This column also always contains an integer value greater than zero.
    
    If the "col" column contains the value '*', then this column
    contains the total number of instances of the term in all rows of the 
    FTS table (in any column). Otherwise, if col contains an integer
    value, then this column contains the total number of instances of the
    term that appear in the FTS table column identified by the col
    value.

  languageid (hidden)
    

    This column determines which languageid is used to
    extract vocabulary from the FTS3/4 table.
    
    The default value for languageid is 0.  If an alternative language
    is specified in WHERE clause constraints, then that alternative is
    used instead of 0.  There can only be a single languageid per query.
    In other words, the WHERE clause cannot contain a range constraint
    or IN operator on the languageid.



  For example, using the tables created above:

INSERT INTO ft(x, y) VALUES('Apple banana', 'Cherry');
INSERT INTO ft(x, y) VALUES('Banana Date Date', 'cherry');
INSERT INTO ft(x, y) VALUES('Cherry Elderberry', 'Elderberry');

-- The following query returns this data:
--
--     apple       |  *  |  1  |  1
--     apple       |  0  |  1  |  1
--     banana      |  *  |  2  |  2
--     banana      |  0  |  2  |  2
--     cherry      |  *  |  3  |  3
--     cherry      |  0  |  1  |  1
--     cherry      |  1  |  2  |  2
--     date        |  *  |  1  |  2
--     date        |  0  |  1  |  2
--     elderberry  |  *  |  1  |  2
--     elderberry  |  0  |  1  |  1
--     elderberry  |  1  |  1  |  1
--
SELECT term, col, documents, occurrences FROM ft_terms;



  In the example, the values in the "term" column are all lower case, 
  even though they were inserted into table "ft" in mixed case. This is because
  an fts4aux table contains the terms as extracted from the document text
  by the tokenizer. In this case, since table "ft" uses the 
  simple tokenizer, this means all terms have been folded to
  lower case. Also, there is (for example) no row with column "term"
  set to "apple" and column "col" set to 1. Since there are no instances
  of the term "apple" in column 1, no row is present in the fts4aux table.


  During a transaction, some of the data written to an FTS table may be 
  cached in memory and written to the database only when the transaction is 
  committed. However the implementation of the fts4aux module is only able 
  to read data from the database. In practice this means that if an fts4aux 
  table is queried from within a transaction in which the associated 
  FTS table has been modified, the results of the query are likely to reflect 
  only a (possibly empty) subset of the changes made.


6. FTS4 Options


  If the "CREATE VIRTUAL TABLE" statement specifies module FTS4 (not FTS3), 
  then special directives - FTS4 options - similar to the "tokenize=*" option
  may also appear in place of column names. An FTS4 option consists of the
  option name, followed by an "=" character, followed by the option value.
  The option value may optionally be enclosed in single or double quotes, with
  embedded quote characters escaped in the same way as for SQL literals. There
  may not be whitespace on either side of the "=" character. For example,
  to create an FTS4 table with the value of option "matchinfo" set to "fts3":

-- Create a reduced-footprint FTS4 table.
CREATE VIRTUAL TABLE papers USING fts4(author, document, matchinfo=fts3);



  FTS4 currently supports the following options:


  OptionInterpretation
  compress
    The compress option is used to specify the compress function. It is an error to
    specify a compress function without also specifying an uncompress
    function. See below for details.

  content
    The content allows the text being indexed to be
    stored in a separate table distinct from the FTS4 table,
    or even outside of SQLite.

  languageid
    The languageid option causes the FTS4 table to have an additional hidden
    integer column that identifies the language of the text contained in
    each row.  The use of the languageid option allows the same FTS4 table
    to hold text in multiple languages or scripts, each with different tokenizer
    rules, and to query each language independently of the others.
    
  matchinfo 
    When set to the value "fts3", the matchinfo option reduces the amount of
    information stored by FTS4 with the consequence that the "l" option of
    matchinfo() is no longer available.

  notindexed 
    This option is used to specify the name of a column for which data is
    not indexed. Values stored in columns that are not indexed are not
    matched by MATCH queries. Nor are they recognized by auxiliary functions.
    A single CREATE VIRTUAL TABLE statement may have any number of notindexed 
    options.

  order
    

    The "order" option may be set to either "DESC" or "ASC" (in upper or
    lower case). If it is set to "DESC", then FTS4 stores its data in such
    a way as to optimize returning results in descending order by docid.
    If it is set to "ASC" (the default), then the data structures are 
    optimized for returning results in ascending order by docid.  In other
    words, if many of the queries run against the FTS4 table use "ORDER BY
    docid DESC", then it may improve performance to add the "order=desc" 
    option to the CREATE VIRTUAL TABLE statement.

  prefix
    This option may be set to a comma-separated list of positive non-zero 
    integers. For each integer N in the list, a separate index is created
    in the database file to optimize prefix queries where
    the query term is N bytes in length, not including the '*' character,
    when encoded using UTF-8. See below for details.

  uncompress
    This option is used to specify the uncompress function. It is an error to
    specify an uncompress function without also specifying a compress
    function. See below for details.



  When using FTS4, specifying a column name that contains an "=" character
  and is not either a "tokenize=*" specification or a recognized FTS4 option
  is an error. With FTS3, the first token in the unrecognized directive is 
  interpreted as a column name. Similarly, specifying multiple "tokenize=*"
  directives in a single table declaration is an error when using FTS4, whereas
  the second and subsequent "tokenize=*" directives are interpreted as column
  names by FTS3. For example:

-- An error. FTS4 does not recognize the directive "xyz=abc".
CREATE VIRTUAL TABLE papers USING fts4(author, document, xyz=abc);

-- Create an FTS3 table with three columns - "author", "document"
-- and "xyz".
CREATE VIRTUAL TABLE papers USING fts3(author, document, xyz=abc);

-- An error. FTS4 does not allow multiple tokenize=* directives
CREATE VIRTUAL TABLE papers USING fts4(tokenize=porter, tokenize=simple);

-- Create an FTS3 table with a single column named "tokenize". The
-- table uses the "porter" tokenizer.
CREATE VIRTUAL TABLE papers USING fts3(tokenize=porter, tokenize=simple);

-- An error. Cannot create a table with two columns named "tokenize".
CREATE VIRTUAL TABLE papers USING fts3(tokenize=porter, tokenize=simple, tokenize=icu);





6.1. The compress= and uncompress= options


  The compress and uncompress options allow FTS4 content to be stored in
  the database in a compressed form. Both options should be set to the name
  of an SQL scalar function registered using sqlite3_create_function()
  that accepts a single argument. 


  The compress function should return a compressed version of the value 
  passed to it as an argument. Each time data is written to the FTS4 table, 
  each column value is passed to the compress function and the result value 
  stored in the database. The compress function may return any type of SQLite 
  value (blob, text, real, integer or null).


  The uncompress function should uncompress data previously compressed by
  the compress function. In other words, for all SQLite values X, it should
  be true that uncompress(compress(X)) equals X. When data that has been
  compressed by the compress function is read from the database by FTS4, it
  is passed to the uncompress function before it is used.


  If the specified compress or uncompress functions do not exist, the table
  may still be created. An error is not returned until the FTS4 table is
  read (if the uncompress function does not exist) or written (if it is the 
  compress function that does not exist).

-- Create an FTS4 table that stores data in compressed form. This
-- assumes that the scalar functions zip() and unzip() have been (or
-- will be) added to the database handle.
CREATE VIRTUAL TABLE papers USING fts4(author, document, compress=zip, uncompress=unzip);



  When implementing the compress and uncompress functions it is important to
  pay attention to data types. Specifically, when a user reads a value from
  a compressed FTS table, the value returned by FTS is exactly the same
  as the value returned by the uncompress function, including the data type.
  If that data type is not the same as the data type of the original value as
  passed to the compress function (for example if the uncompress function is
  returning BLOB when compress was originally passed TEXT), then the users
  query may not function as expected.




6.2. The content= option 


  The content option allows FTS4 to forego storing the text being indexed.
  The content option can be used in two ways:


 The indexed documents are not stored within the SQLite database 
        at all (a "contentless" FTS4 table), or

 The indexed documents are stored in a database table created and
        managed by the user (an "external content" FTS4 table).



  Because the indexed documents themselves are usually much larger than 
  the full-text index, the content option can be used to achieve 
  significant space savings.


6.2.1.  Contentless FTS4 Tables 


  In order to create an FTS4 table that does not store a copy of the indexed
  documents at all, the content option should be set to an empty string.
  For example, the following SQL creates such an FTS4 table with three
  columns - "a", "b", and "c":

CREATE VIRTUAL TABLE t1 USING fts4(content="", a, b, c);



  Data can be inserted into such an FTS4 table using an INSERT statements.
  However, unlike ordinary FTS4 tables, the user must supply an explicit
  integer docid value. For example:

-- This statement is Ok:
INSERT INTO t1(docid, a, b, c) VALUES(1, 'a b c', 'd e f', 'g h i');

-- This statement causes an error, as no docid value has been provided:
INSERT INTO t1(a, b, c) VALUES('j k l', 'm n o', 'p q r');



  It is not possible to UPDATE or DELETE a row stored in a contentless FTS4
  table. Attempting to do so is an error.


  Contentless FTS4 tables also support SELECT statements. However, it is
  an error to attempt to retrieve the value of any table column other than
  the docid column. The auxiliary function matchinfo() may be used, but
  snippet() and offsets() may not. For example:

-- The following statements are Ok:
SELECT docid FROM t1 WHERE t1 MATCH 'xxx';
SELECT docid FROM t1 WHERE a MATCH 'xxx';
SELECT matchinfo(t1) FROM t1 WHERE t1 MATCH 'xxx';

-- The following statements all cause errors, as the value of columns
-- other than docid are required to evaluate them.
SELECT * FROM t1;
SELECT a, b FROM t1 WHERE t1 MATCH 'xxx';
SELECT docid FROM t1 WHERE a LIKE 'xxx%';
SELECT snippet(t1) FROM t1 WHERE t1 MATCH 'xxx';



  Errors related to attempting to retrieve column values other than docid
  are runtime errors that occur within sqlite3_step(). In some cases, for
  example if the MATCH expression in a SELECT query matches zero rows, there
  may be no error at all even if a statement does refer to column values 
  other than docid.


6.2.2.  External Content FTS4 Tables 


  An "external content" FTS4 table is similar to a contentless table, except
  that if evaluation of a query requires the value of a column other than 
  docid, FTS4 attempts to retrieve that value from a table (or view, or 
  virtual table) nominated by the user (hereafter referred to as the "content
  table"). The FTS4 module never writes to the content table, and writing
  to the content table does not affect the full-text index. It is the
  responsibility of the user to ensure that the content table and the 
  full-text index are consistent.


  An external content FTS4 table is created by setting the content option
  to the name of a table (or view, or virtual table) that may be queried by
  FTS4 to retrieve column values when required. If the nominated table does
  not exist, then an external content table behaves in the same way as
  a contentless table. For example:

CREATE TABLE t2(id INTEGER PRIMARY KEY, a, b, c);
CREATE VIRTUAL TABLE t3 USING fts4(content="t2", a, c);



  Assuming the nominated table does exist, then its columns must be the same 
  as or a superset of those defined for the FTS table.  The external table
  must also be in the same database file as the FTS table.  In other words,
  The external table cannot be in a different database file connected using
  ATTACH nor may one of the FTS table and the external content be in the
  TEMP database when the other is in a persistent database file such as MAIN.


  When a users query on the FTS table requires a column value other than
  docid, FTS attempts to read the requested value from the corresponding column of
  the row in the content table with a rowid value equal to the current FTS
  docid. Only the subset of content-table columns duplicated in the FTS/34
  table declaration can be queried for - to retrieve values from any other
  columns the content table must be queried directly. Or, if such a row cannot
  be found in the content table, a NULL value is used instead. For example:

CREATE TABLE t2(id INTEGER PRIMARY KEY, a, b, c);
CREATE VIRTUAL TABLE t3 USING fts4(content="t2", b, c);

INSERT INTO t2 VALUES(2, 'a b', 'c d', 'e f');
INSERT INTO t2 VALUES(3, 'g h', 'i j', 'k l');
INSERT INTO t3(docid, b, c) SELECT id, b, c FROM t2;
-- The following query returns a single row with two columns containing
-- the text values "i j" and "k l".
--
-- The query uses the full-text index to discover that the MATCH 
-- term matches the row with docid=3. It then retrieves the values
-- of columns b and c from the row with rowid=3 in the content table
-- to return.
--
SELECT * FROM t3 WHERE t3 MATCH 'k';

-- Following the UPDATE, the query still returns a single row, this
-- time containing the text values "xxx" and "yyy". This is because the
-- full-text index still indicates that the row with docid=3 matches
-- the FTS4 query 'k', even though the documents stored in the content
-- table have been modified.
--
UPDATE t2 SET b = 'xxx', c = 'yyy' WHERE rowid = 3;
SELECT * FROM t3 WHERE t3 MATCH 'k';

-- Following the DELETE below, the query returns one row containing two
-- NULL values. NULL values are returned because FTS is unable to find
-- a row with rowid=3 within the content table.
--
DELETE FROM t2;
SELECT * FROM t3 WHERE t3 MATCH 'k';



  When a row is deleted from an external content FTS4 table, FTS4 needs to
  retrieve the column values of the row being deleted from the content table.
  This is so that FTS4 can update the full-text index entries for each token
  that occurs within the deleted row to indicate that row has been 
  deleted. If the content table row cannot be found, or if it contains values
  inconsistent with the contents of the FTS index, the results can be difficult
  to predict. The FTS index may be left containing entries corresponding to the
  deleted row, which can lead to seemingly nonsensical results being returned
  by subsequent SELECT queries. The same applies when a row is updated, as
  internally an UPDATE is the same as a DELETE followed by an INSERT.


  This means that in order to keep an FTS in sync with an external content
  table, any UPDATE or DELETE operations must be applied first to the FTS
  table, and then to the external content table. For example:

CREATE TABLE t1_real(id INTEGER PRIMARY KEY, a, b, c, d);
CREATE VIRTUAL TABLE t1_fts USING fts4(content="t1_real", b, c);

-- This works. When the row is removed from the FTS table, FTS retrieves 
-- the row with rowid=123 and tokenizes it in order to determine the entries 
-- that must be removed from the full-text index.
-- 
DELETE FROM t1_fts WHERE rowid = 123;
DELETE FROM t1_real WHERE rowid = 123;

-- This does not work. By the time the FTS table is updated, the row
-- has already been deleted from the underlying content table. As a result
-- FTS is unable to determine the entries to remove from the FTS index and
-- so the index and content table are left out of sync.
--
DELETE FROM t1_real WHERE rowid = 123;
DELETE FROM t1_fts WHERE rowid = 123;


  
  Instead of writing separately to the full-text index and the content table,
  some users may wish to use database triggers to keep the full-text index
  up to date with respect to the set of documents stored in the content table.
  For example, using the tables from earlier examples:

CREATE TRIGGER t2_bu BEFORE UPDATE ON t2 BEGIN
  DELETE FROM t3 WHERE docid=old.rowid;
END;
CREATE TRIGGER t2_bd BEFORE DELETE ON t2 BEGIN
  DELETE FROM t3 WHERE docid=old.rowid;
END;

CREATE TRIGGER t2_au AFTER UPDATE ON t2 BEGIN
  INSERT INTO t3(docid, b, c) VALUES(new.rowid, new.b, new.c);
END;
CREATE TRIGGER t2_ai AFTER INSERT ON t2 BEGIN
  INSERT INTO t3(docid, b, c) VALUES(new.rowid, new.b, new.c);
END;



  The DELETE trigger must be fired before the actual delete takes place
  on the content table. This is so that FTS4 can still retrieve the original
  values in order to update the full-text index. And the INSERT trigger must
  be fired after the new row is inserted, so as to handle the case where the
  rowid is assigned automatically within the system. The UPDATE trigger must
  be split into two parts, one fired before and one after the update of the
  content table, for the same reasons.


  The FTS4 "rebuild" command
  deletes the entire full-text index and rebuilds it based on the current
  set of documents in the content table. Assuming again that "t3" is the
  name of the external content FTS4 table, the rebuild command looks like this:

INSERT INTO t3(t3) VALUES('rebuild');



  This command may also be used with ordinary FTS4 tables, for example if
  the implementation of the tokenizer changes.  It is an
  error to attempt to rebuild the full-text index maintained by a contentless
  FTS4 table, since no content will be available to do the rebuilding.





6.3. The languageid= option


  When the languageid option is present, it specifies the name of
  another hidden column that is added to the FTS4
  table and which is used to specify the language stored in each row
  of the FTS4 table.  The name of the languageid hidden column must
  be distinct from all other column names in the FTS4 table.  Example:

CREATE VIRTUAL TABLE t1 USING fts4(x, y, languageid="lid")



  The default value of a languageid column is 0. Any value inserted
  into a languageid column is converted to a 32-bit (not 64) signed
  integer.


  By default, FTS queries (those that use the MATCH operator)
  consider only those rows with the languageid column set to 0. To
  query for rows with other languageid values, a constraint of the
  form " = " must be added to the queries
  WHERE clause. For example:

SELECT * FROM t1 WHERE t1 MATCH 'abc' AND lid=5;



  It is not possible for a single FTS query to return rows with
  different languageid values. The results of adding WHERE clauses
  that use other operators (e.g.  lid!=5, or lid&lt;=5) are undefined.


  If the content option is used along with the languageid option,
  then the named languageid column must exist in the content= table
  (subject to the usual rules - if a query never needs to read the
  content table then this restriction does not apply).


  When the languageid option is used, SQLite invokes the xLanguageid()
  on the sqlite3_tokenizer_module object immediately after the object
  is created in order to pass in the language id that the
  tokenizer should use.  The xLanguageid() method will never be called
  more than once for any single tokenizer object.  The fact that different
  languages might be tokenized differently is one reason why no single
  FTS query can return rows with different languageid values.
  





6.4. The matchinfo= option


  The matchinfo option may only be set to the value "fts3". 
  Attempting to set matchinfo to anything other than "fts3" is an error.
  If this option is specified, then some of the extra information stored by
  FTS4 is omitted. This reduces the amount of disk space consumed by
  an FTS4 table until it is almost the same as the amount that would
  be used by the equivalent FTS3 table, but also means that the data
  accessed by passing the 'l' flag to the matchinfo() function is
  not available. 




6.5. The notindexed= option


  Normally, the FTS module maintains an inverted index of all terms in
  all columns of the table. This option is used to specify the name of
  a column for which entries should not be added to the index. Multiple
  "notindexed" options may be used to specify that multiple columns should
  be omitted from the index. For example:

-- Create an FTS4 table for which only the contents of columns c2 and c4
-- are tokenized and added to the inverted index.
CREATE VIRTUAL TABLE t1 USING fts4(c1, c2, c3, c4, notindexed=c1, notindexed=c3);



  Values stored in unindexed columns are not eligible to match MATCH 
  operators. They do not influence the results of the offsets() or matchinfo()
  auxiliary functions. Nor will the snippet() function ever return a
  snippet based on a value stored in an unindexed column.




6.6. The prefix= option


  The FTS4 prefix option causes FTS to index term prefixes of specified lengths
  in the same way that it always indexes complete terms.  The prefix option
  must be set to a comma separated list of positive non-zero integers. 
  For each value N in the list, prefixes of length N bytes (when encoded 
  using UTF-8) are indexed.  FTS4 uses term prefix indexes to speed up
  prefix queries. The cost, of course, is that indexing term prefixes as
  well as complete terms increases the database size and slows down write 
  operations on the FTS4 table.


  Prefix indexes may be used to optimize prefix queries in two cases.
  If the query is for a prefix of N bytes, then a prefix index created
  with "prefix=N" provides the best optimization. Or, if no "prefix=N"
  index is available, a "prefix=N+1" index may be used instead. 
  Using a "prefix=N+1" index is less
  efficient than a "prefix=N" index, but is better than no prefix index at all.

-- Create an FTS4 table with indexes to optimize 2 and 4 byte prefix queries.
CREATE VIRTUAL TABLE t1 USING fts4(c1, c2, prefix="2,4");

-- The following two queries are both optimized using the prefix indexes.
SELECT * FROM t1 WHERE t1 MATCH 'ab*';
SELECT * FROM t1 WHERE t1 MATCH 'abcd*';

-- The following two queries are both partially optimized using the prefix
-- indexes. The optimization is not as pronounced as it is for the queries
-- above, but still an improvement over no prefix indexes at all.
SELECT * FROM t1 WHERE t1 MATCH 'a*';
SELECT * FROM t1 WHERE t1 MATCH 'abc*';





7. Special Commands For FTS3 and FTS4


  Special INSERT operates can be used to issue commands to FTS3 and FTS4 tables.
  Every FTS3 and FTS4 has a hidden, read-only column which is the same name as
  the table itself.  INSERTs into this hidden column are interpreted as commands
  to the FTS3/4 table.  For a table with the name "xyz" the following commands
  are supported:


INSERT INTO xyz(xyz) VALUES('optimize');
INSERT INTO xyz(xyz) VALUES('rebuild');
INSERT INTO xyz(xyz) VALUES('integrity-check');
INSERT INTO xyz(xyz) VALUES('merge=X,Y');
INSERT INTO xyz(xyz) VALUES('automerge=N');




7.1. The "optimize" command


  The "optimize" command causes FTS3/4 to merge together all of its
  inverted index b-trees into one large and complete b-tree.  Doing
  an optimize will make subsequent queries run faster since there are
  fewer b-trees to search, and it may reduce disk usage by coalescing
  redundant entries.  However, for a large FTS table, running optimize
  can be as expensive as running VACUUM.  The optimize command
  essentially has to read and write the entire FTS table, resulting
  in a large transaction.


  In batch-mode operation, where an FTS table is initially built up
  using a large number of INSERT operations, then queried repeatedly
  without further changes, it is often a good idea
  to run "optimize" after the last INSERT and before the first query.



7.2. The "rebuild" command


  The "rebuild" command causes SQLite to discard the entire FTS3/4
  table and then rebuild it again from original text.  The concept
  is similar to REINDEX, only that it applies to an
  FTS3/4 table instead of an ordinary index.


  The "rebuild" command should be run whenever the implementation
  of a custom tokenizer changes, so that all content can be retokenized.
  The "rebuild" command is also useful when using the
  FTS4 content option after changes have been made to the original
  content table.



7.3. The "integrity-check" command


  The "integrity-check" command causes SQLite to read and verify
  the accuracy of all inverted indices in an FTS3/4 table by comparing
  those inverted indices against the original content. The 
  "integrity-check" command silently succeeds if the inverted
  indices are all ok, but will fail with an SQLITE_CORRUPT error
  if any problems are found.


  The "integrity-check" command is similar in concept to
 PRAGMA integrity_check.  In a working system, the "integrity-command"
 should always be successful.  Possible causes of integrity-check
 failures include:
  
   The application has made changes to the FTS shadow tables
       directly, without using the FTS3/4 virtual table, causing
       the shadow tables to become out of sync with each other.
   Using the FTS4 content option and failing to manually keep
       the content in sync with the FTS4 inverted indices.
   Bugs in the FTS3/4 virtual table.  (The "integrity-check"
       command was original conceived as part of the test suite
       for FTS3/4.)
   Corruption to the underlying SQLite database file.  (See
       documentation on how to corrupt and SQLite database for
       additional information.)
  



7.4. The "merge=X,Y" command


  The "merge=X,Y" command (where X and Y are integers) causes SQLite
  to do a limited amount of work toward merging the various inverted
  index b-trees of an FTS3/4 table together into one large b-tree.
  The X value is the target number of "blocks" to be merged, and Y is
  the minimum number of b-tree segments on a level required before
  merging will be applied to that level.  The value of Y should
  be between 2 and 16 with a recommended value of 8.  The value of X
  can be any positive integer but values on the order of 100 to 300
  are recommended.


  When an FTS table accumulates 16 b-tree segments at the same level,
  the next INSERT into that table will cause all 16 segments to be
  merged into a single b-tree segment at the next higher level.  The
  effect of these level merges is that most INSERTs into an FTS table
  are very fast and take minimal memory, but an occasional INSERT is
  slow and generates a large transaction because of the need to
  do merging. This results in "spiky" performance of INSERTs.


  To avoid spiky INSERT performance, an application can run the
  "merge=X,Y" command periodically, possibly in an idle thread or
  idle process, to ensure that the FTS table never accumulates
  too many b-tree segments at the same level.  INSERT performance
  spikes can generally be avoided, and performance of FTS3/4 can be
  maximized, by running "merge=X,Y" after every few thousand
  document inserts.  Each "merge=X,Y" command will run in a separate
  transaction (unless they are grouped together using BEGIN...COMMIT,
  of course).  The transactions can be kept small by choosing a value
  for X in the range of 100 to 300.  The idle thread that is running
  the merge commands can know when it is done by checking the difference
  in sqlite3_total_changes() before and after each "merge=X,Y"
  command and stopping the loop when the difference drops below two.



7.5. The "automerge=N" command


  The "automerge=N" command (where N is an integer between 0 and 15,
  inclusive) is used to configure an FTS3/4 tables "automerge" parameter,
  which controls automatic incremental inverted index merging. The default 
  automerge value for new tables is 0, meaning that automatic incremental 
  merging is completely disabled. If the value of the automerge parameter
  is modified using the "automerge=N" command, the new parameter value is
  stored persistently in the database and is used by all subsequently
  established database connections.


  Setting the automerge parameter to a non-zero value enables automatic
  incremental merging. This causes SQLite to do a small amount of inverted 
  index merging after every INSERT operation. The amount of merging 
  performed is designed so that the FTS3/4 table never reaches a point 
  where it has 16 segments at the same level and hence has to do a large 
  merge in order to complete an insert.  In other words, automatic 
  incremental merging is designed to prevent spiky INSERT performance.


  The downside of automatic incremental merging is that it makes
  every INSERT, UPDATE, and DELETE operation on an FTS3/4 table run
  a little slower, since extra time must be used to do the incremental
  merge.  For maximum performance, it is recommended that applications
  disable automatic incremental merge and instead use the 
  "merge" command in an idle process to keep the inverted indices
  well merged.  But if the structure of an application does not easily
  allow for idle processes, the use of automatic incremental merge is
  a very reasonable fallback solution.


  The actual value of the automerge parameter determines the number of
  index segments merged simultaneously by an automatic inverted index
  merge. If the value is set to N, the system waits until there are at
  least N segments on a single level before beginning to incrementally
  merge them. Setting a lower value of N causes segments to be merged more
  quickly, which may speed up full-text queries and, if the workload 
  contains UPDATE or DELETE operations as well as INSERTs, reduce the space
  on disk consumed by the full-text index. However, it also increases the
  amount of data written to disk.


  For general use in cases where the workload contains few UPDATE or DELETE
  operations, a good choice for automerge is 8. If the workload contains
  many UPDATE or DELETE commands, 
  or if query speed is a concern, it may be advantageous to reduce automerge
  to 2.


  For reasons of backwards compatibility, the "automerge=1" command sets
  the automerge parameter to 8, not 1 (a value of 1 would make no sense 
  anyway, as merging data from a single segment is a no-op).



8. Tokenizers


  An FTS tokenizer is a set of rules for extracting terms from a document 
  or basic FTS full-text query. 


  Unless a specific tokenizer is specified as part of the CREATE 
  VIRTUAL TABLE statement used to create the FTS table, the default 
  tokenizer, "simple", is used. The simple tokenizer extracts tokens from
  a document or basic FTS full-text query according to the following 
  rules:


   A term is a contiguous sequence of eligible characters, where 
    eligible characters are all alphanumeric characters and all characters with
    Unicode codepoint values greater than or equal to 128.
    All other characters are
    discarded when splitting a document into terms. Their only contribution is
    to separate adjacent terms.

   All uppercase characters within the ASCII range (Unicode codepoints
    less than 128), are transformed to their lowercase equivalents as part
    of the tokenization process. Thus, full-text queries are
    case-insensitive when using the simple tokenizer.



  For example, when a document containing the text "Right now, they're very
  frustrated.", the terms extracted from the document and added to the 
  full-text index are, in order, "right now they re very frustrated". Such
  a document would match a full-text query such as "MATCH 'Frustrated'", 
  as the simple tokenizer transforms the term in the query to lowercase
  before searching the full-text index.


  As well as the "simple" tokenizer, the FTS source code features a tokenizer 
  that uses the Porter 
  Stemming algorithm. This tokenizer uses the same rules to separate
  the input document into terms including folding all terms into lower case,
  but also uses the Porter Stemming algorithm to reduce related English language
  words to a common root. For example, using the same input document as in the
  paragraph above, the porter tokenizer extracts the following tokens:
  "right now thei veri frustrat". Even though some of these terms are not even
  English words, in some cases using them to build the full-text index is more
  useful than the more intelligible output produced by the simple tokenizer.
  Using the porter tokenizer, the document not only matches full-text queries
  such as "MATCH 'Frustrated'", but also queries such as "MATCH 'Frustration'",
  as the term "Frustration" is reduced by the Porter stemmer algorithm to 
  "frustrat" - just as "Frustrated" is. So, when using the porter tokenizer,
  FTS is able to find not just exact matches for queried terms, but matches
  against similar English language terms. For more information on the 
  Porter Stemmer algorithm, please refer to the page linked above.


  Example illustrating the difference between the "simple" and "porter"
  tokenizers:

-- Create a table using the simple tokenizer. Insert a document into it.
CREATE VIRTUAL TABLE simple USING fts3(tokenize=simple);
INSERT INTO simple VALUES('Right now they''re very frustrated');

-- The first of the following two queries matches the document stored in
-- table "simple". The second does not.
SELECT * FROM simple WHERE simple MATCH 'Frustrated';
SELECT * FROM simple WHERE simple MATCH 'Frustration';

-- Create a table using the porter tokenizer. Insert the same document into it
CREATE VIRTUAL TABLE porter USING fts3(tokenize=porter);
INSERT INTO porter VALUES('Right now they''re very frustrated');

-- Both of the following queries match the document stored in table "porter".
SELECT * FROM porter WHERE porter MATCH 'Frustrated';
SELECT * FROM porter WHERE porter MATCH 'Frustration';



  If this extension is compiled with the SQLITE_ENABLE_ICU pre-processor
  symbol defined, then there exists a built-in tokenizer named "icu"
  implemented using the ICU library. The first argument passed to the
  xCreate() method (see fts3_tokenizer.h) of this tokenizer may be
  an ICU locale identifier. For example "tr_TR" for Turkish as used
  in Turkey, or "en_AU" for English as used in Australia. For example:

CREATE VIRTUAL TABLE thai_text USING fts3(text, tokenize=icu th_TH)



  The ICU tokenizer implementation is very simple. It splits the input
  text according to the ICU rules for finding word boundaries and discards
  any tokens that consist entirely of white-space. This may be suitable
  for some applications in some locales, but not all. If more complex
  processing is required, for example to implement stemming or
  discard punctuation, this can be done by creating a tokenizer
  implementation that uses the ICU tokenizer as part of its implementation.




  The "unicode61" tokenizer is available beginning with SQLite version 3.7.13
  (2012-06-11).
  Unicode61 works very much like "simple" except that it does simple unicode
  case folding according to rules in Unicode Version 6.1 and it recognizes
  unicode space and punctuation characters and uses those to separate tokens.
  The simple tokenizer only does case folding of ASCII characters and only
  recognizes ASCII space and punctuation characters as token separators.


  By default, "unicode61" attempts to remove diacritics from Latin script
  characters. This behaviour can be overridden by adding the tokenizer argument
  "remove_diacritics=0". For example:

-- Create tables that remove alldiacritics from Latin script characters
-- as part of tokenization.
CREATE VIRTUAL TABLE txt1 USING fts4(tokenize=unicode61);
CREATE VIRTUAL TABLE txt2 USING fts4(tokenize=unicode61 "remove_diacritics=2");

-- Create a table that does not remove diacritics from Latin script
-- characters as part of tokenization.
CREATE VIRTUAL TABLE txt3 USING fts4(tokenize=unicode61 "remove_diacritics=0");


The remove_diacritics option may be set to "0", "1" or "2". The default
   value is "1".  If it is set to "1" or "2", then diacritics are removed from
   Latin script characters as described above. However, if it is set to "1",
   then diacritics are not removed in the fairly uncommon case where a single
   unicode codepoint is used to represent a character with more that one
   diacritic. For example, diacritics are not removed from codepoint 0x1ED9
   ("LATIN SMALL LETTER O WITH CIRCUMFLEX AND DOT BELOW"). This is technically
   a bug, but cannot be fixed without creating backwards compatibility
   problems. If this option is set to "2", then diacritics are correctly
   removed from all Latin characters.


  It is also possible to customize the set of codepoints that unicode61 treats
  as separator characters. The "separators=" option may be used to specify one
  or more extra characters that should be treated as separator characters, and
  the "tokenchars=" option may be used to specify one or more extra characters
  that should be treated as part of tokens instead of as separator characters.
  For example:

-- Create a table that uses the unicode61 tokenizer, but considers "."
-- and "=" characters to be part of tokens, and capital "X" characters to
-- function as separators.
CREATE VIRTUAL TABLE txt3 USING fts4(tokenize=unicode61 "tokenchars=.=" "separators=X");

-- Create a table that considers space characters (codepoint 32) to be
-- a token character
CREATE VIRTUAL TABLE txt4 USING fts4(tokenize=unicode61 "tokenchars= ");



  If a character specified as part of the argument to "tokenchars=" is considered
  to be a token character by default, it is ignored. This is true even if it has
  been marked as a separator by an earlier "separators=" option. Similarly, if
  a character specified as part of a "separators=" option is treated as a separator
  character by default, it is ignored. If multiple "tokenchars=" or "separators="
  options are specified, all are processed. For example:

-- Create a table that uses the unicode61 tokenizer, but considers "."
-- and "=" characters to be part of tokens, and capital "X" characters to
-- function as separators. Both of the "tokenchars=" options are processed
-- The "separators=" option ignores the "." passed to it, as "." is by
-- default a separator character, even though it has been marked as a token
-- character by an earlier "tokenchars=" option.
CREATE VIRTUAL TABLE txt5 USING fts4(
    tokenize=unicode61 "tokenchars=." "separators=X." "tokenchars=="
);



  The arguments passed to the "tokenchars=" or "separators=" options are 
  case-sensitive. In the example above, specifying that "X" is a separator
  character does not affect the way "x" is handled.



8.1. Custom (Application Defined) Tokenizers


  In addition to providing built-in "simple", "porter" and (possibly) "icu" and
  "unicode61" tokenizers,
  FTS provides an interface for applications to implement and register custom
  tokenizers written in C.  The interface used to create a new tokenizer is defined
  and described in the fts3_tokenizer.h source file.


  Registering a new FTS tokenizer is similar to registering a new
  virtual table module with SQLite. The user passes a pointer to a
  structure containing pointers to various callback functions that
  make up the implementation of the new tokenizer type. For tokenizers,
  the structure (defined in fts3_tokenizer.h) is called
  "sqlite3_tokenizer_module".


  FTS does not expose a C-function that users call to register new
  tokenizer types with a database handle. Instead, the pointer must
  be encoded as an SQL blob value and passed to FTS through the SQL
  engine by evaluating a special scalar function, "fts3_tokenizer()".
  The fts3_tokenizer() function may be called with one or two arguments,
  as follows:

SELECT fts3_tokenizer(&lt;tokenizer-name&gt;);
SELECT fts3_tokenizer(&lt;tokenizer-name&gt;, &lt;sqlite3_tokenizer_module ptr&gt;);



  Where &lt;tokenizer-name&gt; is parameter to which a string is bound using
  sqlite3_bind_text() where the string identifies the tokenizer and
  &lt;sqlite3_tokenizer_module ptr&gt; is a parameter to which a BLOB is
  bound using sqlite3_bind_blob() where the value of the BLOB is a
  pointer to an sqlite3_tokenizer_module structure.
  If the second argument is present,
  it is registered as tokenizer &lt;tokenizer-name&gt; and a copy of it
  returned. If only one argument is passed, a pointer to the tokenizer
  implementation currently registered as &lt;tokenizer-name&gt; is returned,
  encoded as a blob. Or, if no such tokenizer exists, an SQL exception
  (error) is raised.


  Prior to SQLite version 3.11.0 (2016-02-15), the arguments to
  fts3_tokenizer() could be literal strings or BLOBs. They did not have to
  be bound parameters.  But that could lead to security problems in the
  event of an SQL injection.  Hence, the legacy behavior is now disabled
  by default.  But the old legacy behavior can be enabled, for backwards
  compatibility in applications that really need it, 
  by calling
  sqlite3_db_config(db,SQLITE_DBCONFIG_ENABLE_FTS3_TOKENIZER,1,0).


  The following block contains an example of calling the fts3_tokenizer()
  function from C code:

/*
** Register a tokenizer implementation with FTS3 or FTS4.
*/
int registerTokenizer(
  sqlite3 *db,
  char *zName,
  const sqlite3_tokenizer_module *p
){
  int rc;
  sqlite3_stmt *pStmt;
  const char *zSql = "SELECT fts3_tokenizer(?1, ?2)";

  rc = sqlite3_prepare_v2(db, zSql, -1, &pStmt, 0);
  if( rc!=SQLITE_OK ){
    return rc;
  }

  sqlite3_bind_text(pStmt, 1, zName, -1, SQLITE_STATIC);
  sqlite3_bind_blob(pStmt, 2, &p, sizeof(p), SQLITE_STATIC);
  sqlite3_step(pStmt);

  return sqlite3_finalize(pStmt);
}

/*
** Query FTS for the tokenizer implementation named zName.
*/
int queryTokenizer(
  sqlite3 *db,
  char *zName,
  const sqlite3_tokenizer_module **pp
){
  int rc;
  sqlite3_stmt *pStmt;
  const char *zSql = "SELECT fts3_tokenizer(?)";

  *pp = 0;
  rc = sqlite3_prepare_v2(db, zSql, -1, &pStmt, 0);
  if( rc!=SQLITE_OK ){
    return rc;
  }

  sqlite3_bind_text(pStmt, 1, zName, -1, SQLITE_STATIC);
  if( SQLITE_ROW==sqlite3_step(pStmt) ){
    if( sqlite3_column_type(pStmt, 0)==SQLITE_BLOB ){
      memcpy(pp, sqlite3_column_blob(pStmt, 0), sizeof(*pp));
    }
  }

  return sqlite3_finalize(pStmt);
}





8.2. Querying Tokenizers

The "fts3tokenize" virtual table can be used to directly access any
   tokenizer.  The following SQL demonstrates how to create an instance 
   of the fts3tokenize virtual table:

CREATE VIRTUAL TABLE tok1 USING fts3tokenize('porter');


The name of the desired tokenizer should be substituted in place of
   'porter' in the example, of course.  If the tokenizer requires one or
   more arguments, they should be separated by commas in the fts3tokenize
   declaration (even though they are separated by spaces in declarations
   of regular fts4 tables). The following creates fts4 and fts3tokenize
   tables that use the same tokenizer:
CREATE VIRTUAL TABLE text1 USING fts4(tokenize=icu en_AU);
CREATE VIRTUAL TABLE tokens1 USING fts3tokenize(icu, en_AU);

CREATE VIRTUAL TABLE text2 USING fts4(tokenize=unicode61 "tokenchars=@." "separators=123");
CREATE VIRTUAL TABLE tokens2 USING fts3tokenize(unicode61, "tokenchars=@.", "separators=123");

   

   Once the virtual table is created, it can be queried as follows:

SELECT token, start, end, position 
  FROM tok1
 WHERE input='This is a test sentence.';


The virtual table will return one row of output for each token in the
   input string.  The "token" column is the text of the token.  The "start"
   and "end" columns are the byte offset to the beginning and end of the
   token in the original input string.  
   The "position" column is the sequence number
   of the token in the original input string.  There is also an "input"
   column which is simply a copy of the input string that is specified in
   the WHERE clause.  Note that a constraint of the form "input=?" must
   appear in the WHERE clause or else the virtual table will have no input
   to tokenize and will return no rows.  The example above generates
   the following output:

thi|0|4|0
is|5|7|1
a|8|9|2
test|10|14|3
sentenc|15|23|4


Notice that the tokens in the result set from the fts3tokenize virtual
   table have been transformed according to the rules of the tokenizer.
   Since this example used the "porter" tokenizer, the "This" token was
   converted into "thi".  If the original text of the token is desired,
   it can be retrieved using the "start" and "end" columns with the
   substr() function.  For example:

SELECT substr(input, start+1, end-start), token, position
  FROM tok1
 WHERE input='This is a test sentence.';


The fts3tokenize virtual table can be used on any tokenizer, regardless
   of whether or not there exists an FTS3 or FTS4 table that actually uses
   that tokenizer.

 

9. Data Structures


  This section describes at a high-level the way the FTS module stores its
  index and content in the database. It is not necessary to read or 
  understand the material in this section in order to use FTS in an 
  application. However, it may be useful to application developers attempting 
  to analyze and understand FTS performance characteristics, or to developers 
  contemplating enhancements to the existing FTS feature set.





9.1. Shadow Tables

  For each FTS virtual table in a database, three to five real (non-virtual) tables
  are created to store the underlying data.  These real tables are called "shadow tables".
  The real tables are named "%_content",
  "%_segdir", "%_segments", "%_stat", and "%_docsize", where "%" is replaced by the name
  of the FTS virtual table.


  The leftmost column of the "%_content" table is an INTEGER PRIMARY KEY field
  named "docid". Following this is one column for each column of the FTS
  virtual table as declared by the user, named by prepending the column name
  supplied by the user with "cN", where N is the index of the 
  column within the table, numbered from left to right starting with 0. Data
  types supplied as part of the virtual table declaration are not used as
  part of the %_content table declaration. For example:

-- Virtual table declaration
CREATE VIRTUAL TABLE abc USING fts4(a NUMBER, b TEXT, c);

-- Corresponding %_content table declaration
CREATE TABLE abc_content(docid INTEGER PRIMARY KEY, c0a, c1b, c2c);



  The %_content table contains the unadulterated data inserted by the user 
  into the FTS virtual table by the user. If the user does not explicitly
  supply a "docid" value when inserting records, one is selected automatically
  by the system.


  The %_stat and %_docsize tables are only created if the FTS table uses the
  FTS4 module, not FTS3. Furthermore, the %_docsize table is omitted if the
  FTS4 table is created with the "matchinfo=fts3" directive
  specified as part of the CREATE VIRTUAL TABLE statement. If they are created,
  the schema of the two tables is as follows:
CREATE TABLE %_stat(
  id INTEGER PRIMARY KEY, 
  value BLOB
);

CREATE TABLE %_docsize(
  docid INTEGER PRIMARY KEY,
  size BLOB
);



  For each row in the FTS table, the %_docsize table contains a corresponding
  row with the same "docid" value. The "size" field contains a blob consisting
  of N FTS varints, where N is the number of user-defined columns
  in the table. Each varint in the "size" blob is the number of tokens in the
  corresponding column of the associated row in the FTS table. The %_stat table
  always contains a single row with the "id" column set to 0. The "value" 
  column contains a blob consisting of N+1 FTS varints, where N
  is again the number of user-defined columns in the FTS table. The first
  varint in the blob is set to the total number of rows in the FTS table. The
  second and subsequent varints contain the total number of tokens stored in
  the corresponding column for all rows of the FTS table.


  The two remaining tables, %_segments and %_segdir, are used to store the 
  full-text index. Conceptually, this index is a lookup table that maps each 
  term (word) to the set of docid values corresponding to records in the 
  %_content table that contain one or more occurrences of the term. To
  retrieve all documents that contain a specified term, the FTS module
  queries this index to determine the set of docid values for records that
  contain the term, then retrieves the required documents from the %_content
  table. Regardless of the schema of the FTS virtual table, the %_segments
  and %_segdir tables are always created as follows:

CREATE TABLE %_segments(
  blockid INTEGER PRIMARY KEY,       -- B-tree node id
  block blob                         -- B-tree node data
);

CREATE TABLE %_segdir(
  level INTEGER,
  idx INTEGER,
  start_block INTEGER,               -- Blockid of first node in %_segments
  leaves_end_block INTEGER,          -- Blockid of last leaf node in %_segments
  end_block INTEGER,                 -- Blockid of last node in %_segments
  root BLOB,                         -- B-tree root node
  PRIMARY KEY(level, idx)
);



  The schema depicted above is not designed to store the full-text index 
  directly. Instead, it is used to store one or more b-tree structures. There
  is one b-tree for each row in the %_segdir table. The %_segdir table
  row contains the root node and various meta-data associated with the
  b-tree structure, and the %_segments table contains all other (non-root)
  b-tree nodes. Each b-tree is referred to as a "segment". Once it has
  been created, a segment b-tree is never updated (although it may be
  deleted altogether).


  The keys used by each segment b-tree are terms (words). As well as the
  key, each segment b-tree entry has an associated "doclist" (document list).
  A doclist consists of zero or more entries, where each entry consists of:


   A docid (document id), and
   A list of term offsets, one for each occurrence of the term within
       the document. A term offset indicates the number of tokens (words)
       that occur before the term in question, not the number of characters
       or bytes. For example, the term offset of the term "war" in the
       phrase "Ancestral voices prophesying war!" is 3.



  Entries within a doclist are sorted by docid. Positions within a doclist
  entry are stored in ascending order.


  The contents of the logical full-text index is found by merging the
  contents of all segment b-trees. If a term is present in more than one
  segment b-tree, then it maps to the union of each individual doclist. If,
  for a single term, the same docid occurs in more than one doclist, then only
  the doclist that is part of the most recently created segment b-tree is 
  considered valid. 


  Multiple b-tree structures are used instead of a single b-tree to reduce
  the cost of inserting records into FTS tables. When a new record is 
  inserted into an FTS table that already contains a lot of data, it is
  likely that many of the terms in the new record are already present in
  a large number of existing records. If a single b-tree were used, then
  large doclist structures would have to be loaded from the database,
  amended to include the new docid and term-offset list, then written back
  to the database. Using multiple b-tree tables allows this to be avoided
  by creating a new b-tree which can be merged with the existing b-tree
  (or b-trees) later on. Merging of b-tree structures can be performed as
  a background task, or once a certain number of separate b-tree structures
  have been accumulated. Of course, this scheme makes queries more expensive
  (as the FTS code may have to look up individual terms in more than one
  b-tree and merge the results), but it has been found that in practice this
  overhead is often negligible.
  
9.2. Variable Length Integer (varint) Format


  Integer values stored as part of segment b-tree nodes are encoded using the
  FTS varint format. This encoding is similar, but not identical, to
  the SQLite varint format.


  An encoded FTS varint consumes between one and ten bytes of space. The
  number of bytes required is determined by the sign and magnitude of the
  integer value encoded. More accurately, the number of bytes used to store
  the encoded integer depends on the position of the most significant set bit
  in the 64-bit twos-complement representation of the integer value. Negative
  values always have the most significant bit set (the sign bit), and so are
  always stored using the full ten bytes. Positive integer values may be
  stored using less space.


  The final byte of an encoded FTS varint has its most significant bit 
  cleared. All preceding bytes have the most significant bit set. Data
  is stored in the remaining seven least significant bits of each byte.
  The first byte of the encoded representation contains the least significant
  seven bits of the encoded integer value. The second byte of the encoded
  representation, if it is present, contains the seven next least significant
  bits of the integer value, and so on. The following table contains examples
  of encoded integer values:


  DecimalHexadecimalEncoded Representation
  430x000000000000002B0x2B 
  2008150x000000000003106F0xEF 0xA0 0x0C
  -10xFFFFFFFFFFFFFFFF0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0x01

  

9.3. Segment B-Tree Format


  Segment b-trees are prefix-compressed b+-trees. There is one segment b-tree
  for each row in the %_segdir table (see above). The root node of the segment
  b-tree is stored as a blob in the "root" field of the corresponding row
  of the %_segdir table. All other nodes (if any exist) are stored in the 
  "blob" column of the %_segments table. Nodes within the %_segments table are
  identified by the integer value in the blockid field of the corresponding
  row. The following table describes the fields of the %_segdir table:


  Column           Interpretation
  level             
    Between them, the contents of the "level" and "idx" fields define the
    relative age of the segment b-tree. The smaller the value stored in the
    "level" field, the more recently the segment b-tree was created. If two
    segment b-trees are of the same "level", the segment with the larger
    value stored in the "idx" column is more recent. The PRIMARY KEY constraint
    on the %_segdir table prevents any two segments from having the same value
    for both the "level" and "idx" fields.
  idx               See above.
  start_block      
    The blockid that corresponds to the node with the smallest blockid that 
    belongs to this segment b-tree. Or zero if the entire segment b-tree
    fits on the root node. If it exists, this node is always a leaf node.
  leaves_end_block 
    The blockid that corresponds to the leaf node with the largest blockid 
    that belongs to this segment b-tree. Or zero if the entire segment b-tree
    fits on the root node.
  end_block 
    This field may contain either an integer or a text field consisting of
    two integers separated by a space character (unicode codepoint 0x20).

    The first, or only, integer is the blockid that corresponds to the interior
    node with the largest blockid that belongs to this segment b-tree. Or zero
    if the entire segment b-tree fits on the root node. If it exists, this node
    is always an interior node.

    The second integer, if it is present, is the aggregate size of all data
    stored on leaf pages in bytes. If the value is negative, then the segment
    is the output of an unfinished incremental-merge operation, and the
    absolute value is current size in bytes.

  root             
    Blob containing the root node of the segment b-tree.



  Apart from the root node, the nodes that make up a single segment b-tree are
  always stored using a contiguous sequence of blockids. Furthermore, the
  nodes that make up a single level of the b-tree are themselves stored as
  a contiguous block, in b-tree order. The contiguous sequence of blockids
  used to store the b-tree leaves are allocated starting with the blockid
  value stored in the "start_block" column of the corresponding %_segdir row,
  and finishing at the blockid value stored in the "leaves_end_block"
  field of the same row. It is therefore possible to iterate through all the
  leaves of a segment b-tree, in key order, by traversing the %_segments 
  table in blockid order from "start_block" to "leaves_end_block".  

9.3.1. Segment B-Tree Leaf Nodes


  The following diagram depicts the format of a segment b-tree leaf node.


  
   Segment B-Tree Leaf Node Format



  The first term stored on each node ("Term 1" in the figure above) is
  stored verbatim. Each subsequent term is prefix-compressed with respect
  to its predecessor. Terms are stored within a page in sorted (memcmp)
  order.

9.3.2. Segment B-Tree Interior Nodes


  The following diagram depicts the format of a segment b-tree interior 
  (non-leaf) node.


  
   Segment B-Tree Interior Node Format



9.4. Doclist Format


  A doclist consists of an array of 64-bit signed integers, serialized using
  the FTS varint format. Each doclist entry is made up of a series of two 
  or more integers, as follows:


   The docid value. The first entry in a doclist contains the literal docid
       value. The first field of each subsequent doclist entry contains the 
       difference between the new docid and the previous one (always a positive 
       number).
   Zero or more term-offset lists. A term-offset list is present for each
       column of the FTS virtual table that contains the term. A term-offset
       list consists of the following:
     
        Constant value 1. This field is omitted for any term-offset list
            associated with column 0.
        The column number (1 for the second leftmost column, etc.). This
            field is omitted for any term-offset list associated with column 0.
        A list of term-offsets, sorted from smallest to largest. Instead
            of storing the term-offset value literally, each integer stored 
            is the difference between the current term-offset and the previous 
            one (or zero if the current term-offset is the first), plus 2.
     
   Constant value 0.



  
   FTS3 Doclist Format



  
   FTS Doclist Entry Format



  For doclists for which the term appears in more than one column of the FTS
  virtual table, term-offset lists within the doclist are stored in column 
  number order. This ensures that the term-offset list associated with 
  column 0 (if any) is always first, allowing the first two fields of the
  term-offset list to be omitted in this case.


10. Limitations

10.1.  UTF-16 byte-order-mark problem 

For UTF-16 databases, when using the "simple" tokenizer, it is possible to use
malformed unicode strings to cause the 
integrity-check special command to falsely report
corruption, or for auxiliary functions to return
incorrect results. More specifically, the bug can be triggered by any of the following:


  A UTF-16 byte-order-mark (BOM) is embedded at the beginning of an SQL string
       literal value inserted into an FTS3 table. For example:

INSERT INTO fts_table(col) VALUES(char(0xfeff)||'text...');


  Malformed UTF-8 that SQLite converts to a UTF-16 byte-order-mark is
       embedded at the beginning of an SQL string literal value inserted 
       into an FTS3 table.

  A text value created by casting a blob that begins with the two
       bytes 0xFF and 0xFE, in either possible order, is inserted into an
       FTS3 table. For example:
       
INSERT INTO fts_table(col) VALUES(CAST(X'FEFF' AS TEXT));



Everything works correctly if any of the following are true:

 The database encoding is UTF-8.
 All text strings are insert using one of the
     sqlite3_bind_text() family of functions.
 Literal strings contain no byte-order-marks.
 A tokenizer is used that recognizes byte-order-marks
     as whitespace.  (The default "simple" tokenizer for
     FTS3/4 does not think that BOMs are whitespace, but
     the unicode tokenizer does.)

All of the above conditions must be false in order for problems
to occur.  And even if all of the conditiona above are false,
most things will still operator correctly.  Only the
integrity-check command and the
auxiliary functions might given
unexpected results.



  Appendix A: Search Application Tips



  FTS is primarily designed to support Boolean full-text queries - queries
  to find the set of documents that match a specified criteria. However, many 
  (most?) search applications require that results are somehow ranked in order
  of "relevance", where "relevance" is defined as the likelihood that the user
  who performed the search is interested in a specific element of the returned
  set of documents. When using a search engine to find documents on the world
  wide web, the user expects that the most useful, or "relevant", documents 
  will be returned as the first page of results, and that each subsequent page 
  contains progressively less relevant results. Exactly how a machine can 
  determine document relevance based on a users query is a complicated problem
  and the subject of much ongoing research.


  One very simple scheme might be to count the number of instances of the 
  users search terms in each result document. Those documents that contain
  many instances of the terms are considered more relevant than those with
  a small number of instances of each term. In an FTS application, the 
  number of term instances in each result could be determined by counting
  the number of integers in the return value of the offsets function.
  The following example shows a query that could be used to obtain the
  ten most relevant results for a query entered by the user:

-- This example (and all others in this section) assumes the following schema
CREATE VIRTUAL TABLE documents USING fts3(title, content);

-- Assuming the application has supplied an SQLite user function named "countintegers"
-- that returns the number of space-separated integers contained in its only argument,
-- the following query could be used to return the titles of the 10 documents that contain
-- the greatest number of instances of the users query terms. Hopefully, these 10
-- documents will be those that the users considers more or less the most "relevant".
SELECT title FROM documents 
  WHERE documents MATCH &lt;query&gt;
  ORDER BY countintegers(offsets(documents)) DESC
  LIMIT 10 OFFSET 0



  The query above could be made to run faster by using the FTS matchinfo
  function to determine the number of query term instances that appear in each
  result. The matchinfo function is much more efficient than the offsets 
  function. Furthermore, the matchinfo function provides extra information
  regarding the overall number of occurrences of each query term in the entire
  document set (not just the current row) and the number of documents in which 
  each query term appears. This may be used (for example) to attach a higher
  weight to less common terms which may increase the overall computed relevancy 
  of those results the user considers more interesting.

-- If the application supplies an SQLite user function called "rank" that
-- interprets the blob of data returned by matchinfo and returns a numeric
-- relevancy based on it, then the following SQL may be used to return the
-- titles of the 10 most relevant documents in the dataset for a users query.
SELECT title FROM documents 
  WHERE documents MATCH &lt;query&gt;
  ORDER BY rank(matchinfo(documents)) DESC
  LIMIT 10 OFFSET 0



  The SQL query in the example above uses less CPU than the first example
  in this section, but still has a non-obvious performance problem. SQLite
  satisfies this query by retrieving the value of the "title" column and
  matchinfo data from the FTS module for every row matched by the users
  query before it sorts and limits the results. Because of the way SQLite's
  virtual table interface works, retrieving the value of the "title" column
  requires loading the entire row from disk (including the "content" field,
  which may be quite large). This means that if the users query matches
  several thousand documents, many megabytes of "title" and "content" data
  may be loaded from disk into memory even though they will never be used
  for any purpose. 


  The SQL query in the following example block is one solution to this 
  problem. In SQLite, when a sub-query 
  used in a join contains a LIMIT clause, the results of the sub-query are
  calculated and stored in temporary table before the main query is executed.
  This means that SQLite will load only the docid and matchinfo data for each
  row matching the users query into memory, determine the docid values
  corresponding to the ten most relevant documents, then load only the title
  and content information for those 10 documents only. Because both the matchinfo
  and docid values are gleaned entirely from the full-text index, this results
  in dramatically less data being loaded from the database into memory.

SELECT title FROM documents JOIN ( 
    SELECT docid, rank(matchinfo(documents)) AS rank 
    FROM documents
    WHERE documents MATCH &lt;query&gt;
    ORDER BY rank DESC 
    LIMIT 10 OFFSET 0
) AS ranktable USING(docid)
ORDER BY ranktable.rank DESC



  The next block of SQL enhances the query with solutions to two other problems
  that may arise in developing search applications using FTS:


   
       The snippet function cannot be used with the above query. Because
       the outer query does not include a "WHERE ... MATCH" clause, the snippet 
       function may not be used with it. One solution is to duplicate the WHERE
       clause used by the sub-query in the outer query. The overhead associated
       with this is usually negligible.
   
       The relevancy of a document may depend on something other than just
       the data available in the return value of matchinfo. For example
       each document in the database may be assigned a static weight based
       on factors unrelated to its content (origin, author, age, number
       of references etc.). These values can be stored by the application
       in a separate table that can be joined against the documents table
       in the sub-query so that the rank function may access them.



  This version of the query is very similar to that used by the 
  sqlite.org documentation search 
  application.

-- This table stores the static weight assigned to each document in FTS table
-- "documents". For each row in the documents table there is a corresponding row
-- with the same docid value in this table.
CREATE TABLE documents_data(docid INTEGER PRIMARY KEY, weight);

-- This query is similar to the one in the block above, except that:
--
--   1. It returns a "snippet" of text along with the document title for display. So
--      that the snippet function may be used, the "WHERE ... MATCH ..." clause from
--      the sub-query is duplicated in the outer query.
--
--   2. The sub-query joins the documents table with the document_data table, so that
--      implementation of the rank function has access to the static weight assigned
--      to each document.
SELECT title, snippet(documents) FROM documents JOIN ( 
    SELECT docid, rank(matchinfo(documents), documents_data.weight) AS rank
    FROM documents JOIN documents_data USING(docid)
    WHERE documents MATCH &lt;query&gt;
    ORDER BY rank DESC 
    LIMIT 10 OFFSET 0
) AS ranktable USING(docid)
WHERE documents MATCH &lt;query&gt;
ORDER BY ranktable.rank DESC



  All the example queries above return the ten most relevant query results.
  By modifying the values used with the OFFSET and LIMIT clauses, a query 
  to return (say) the next ten most relevant results is easy to construct. 
  This may be used to obtain the data required for a search applications second
  and subsequent pages of results.


  The next block contains an example rank function that uses matchinfo data 
  implemented in C. Instead of a single weight, it allows a weight to be 
  externally assigned to each column of each document. It may be registered
  with SQLite like any other user function using sqlite3_create_function.

 Security Warning: Because it is just an ordinary SQL function, 
  rank() may be invoked as part of any SQL query in any context. This means
  that the first argument passed may not be a valid matchinfo blob.
  Implementors should take care to handle this case without causing buffer
  overruns or other potential security problems.

/*
** SQLite user defined function to use with matchinfo() to calculate the
** relevancy of an FTS match. The value returned is the relevancy score
** (a real value greater than or equal to zero). A larger value indicates 
** a more relevant document.
**
** The overall relevancy returned is the sum of the relevancies of each 
** column value in the FTS table. The relevancy of a column value is the
** sum of the following for each reportable phrase in the FTS query:
**
**   (&lt;hit count&gt; / &lt;global hit count&gt;) * &lt;column weight&gt;
**
** where &lt;hit count&gt; is the number of instances of the phrase in the
** column value of the current row and &lt;global hit count&gt; is the number
** of instances of the phrase in the same column of all rows in the FTS
** table. The &lt;column weight&gt; is a weighting factor assigned to each
** column by the caller (see below).
**
** The first argument to this function must be the return value of the FTS 
** matchinfo() function. Following this must be one argument for each column 
** of the FTS table containing a numeric weight factor for the corresponding 
** column. Example:
**
**     CREATE VIRTUAL TABLE documents USING fts3(title, content)
**
** The following query returns the docids of documents that match the full-text
** query &lt;query&gt; sorted from most to least relevant. When calculating
** relevance, query term instances in the 'title' column are given twice the
** weighting of those in the 'content' column.
**
**     SELECT docid FROM documents 
**     WHERE documents MATCH &lt;query&gt; 
**     ORDER BY rank(matchinfo(documents), 1.0, 0.5) DESC
*/
static void rankfunc(sqlite3_context *pCtx, int nVal, sqlite3_value **apVal){
  int *aMatchinfo;                /* Return value of matchinfo() */
  int nMatchinfo;                 /* Number of elements in aMatchinfo&#x5B;&#x5D; */
  int nCol = 0;                   /* Number of columns in the table */
  int nPhrase = 0;                /* Number of phrases in the query */
  int iPhrase;                    /* Current phrase */
  double score = 0.0;             /* Value to return */

  assert( sizeof(int)==4 );

  /* Check that the number of arguments passed to this function is correct.
  ** If not, jump to wrong_number_args. Set aMatchinfo to point to the array
  ** of unsigned integer values returned by FTS function matchinfo. Set
  ** nPhrase to contain the number of reportable phrases in the users full-text
  ** query, and nCol to the number of columns in the table. Then check that the
  ** size of the matchinfo blob is as expected. Return an error if it is not.
  */
  if( nVal&lt;1 ) goto wrong_number_args;
  aMatchinfo = (unsigned int *)sqlite3_value_blob(apVal&#x5B;0&#x5D;);
  nMatchinfo = sqlite3_value_bytes(apVal&#x5B;0&#x5D;) / sizeof(int);
  if( nMatchinfo&gt;=2 ){
    nPhrase = aMatchinfo&#x5B;0&#x5D;;
    nCol = aMatchinfo&#x5B;1&#x5D;;
  }
  if( nMatchinfo!=(2+3*nCol*nPhrase) ){
    sqlite3_result_error(pCtx,
      "invalid matchinfo blob passed to function rank()", -1);
    return;
  }
  if( nVal!=(1+nCol) ) goto wrong_number_args;

  /* Iterate through each phrase in the users query. */
  for(iPhrase=0; iPhrase&lt;nPhrase; iPhrase++){
    int iCol;                     /* Current column */

    /* Now iterate through each column in the users query. For each column,
    ** increment the relevancy score by:
    **
    **   (&lt;hit count&gt; / &lt;global hit count&gt;) * &lt;column weight&gt;
    **
    ** aPhraseinfo&#x5B;&#x5D; points to the start of the data for phrase iPhrase. So
    ** the hit count and global hit counts for each column are found in 
    ** aPhraseinfo&#x5B;iCol*3&#x5D; and aPhraseinfo&#x5B;iCol*3+1&#x5D;, respectively.
    */
    int *aPhraseinfo = &aMatchinfo&#x5B;2 + iPhrase*nCol*3&#x5D;;
    for(iCol=0; iCol&lt;nCol; iCol++){
      int nHitCount = aPhraseinfo&#x5B;3*iCol&#x5D;;
      int nGlobalHitCount = aPhraseinfo&#x5B;3*iCol+1&#x5D;;
      double weight = sqlite3_value_double(apVal&#x5B;iCol+1&#x5D;);
      if( nHitCount>0 ){
        score += ((double)nHitCount / (double)nGlobalHitCount) * weight;
      }
    }
  }

  sqlite3_result_double(pCtx, score);
  return;

  /* Jump here if the wrong number of arguments are passed to this function */
wrong_number_args:
  sqlite3_result_error(pCtx, "wrong number of arguments to function rank()", -1);
}


c2: Swarmvtab Virtual Table
c5: swarmvtab.html
c4:
1. Overview

The "swarmvtab" virtual table allows the user to query a large number 
of tables (hereafter "component" tables) with similar schemas but distinct
ranges of rowid values as if they were a single database table. The tables may
be (and usually are) located in different databases. Swarmvtab tables are
read-only.

Component tables must not be declared WITHOUT ROWID, and must all have
the same schema, but may have different names within their databases. In
this context, "the same schema" means that:


  All component tables must have the same set of columns, in the same 
      order.
  The types and default collation sequences attached to each column
      must be the same for all component tables.
  All component tables must have the same PRIMARY KEY declaration (if any).


A swarmvtab table has the same schema as each of its component tables.

A swarmvtab virtual table is created as follows:

CREATE VIRTUAL TABLE temp.&lt;name&gt; USING swarmvtab(&lt;sql-statement&gt;);


Swarmvtab virtual tables must be created in the temp schema. Attempting
to create a swarmvtab in the main or an attached database is an error.

The SQL statement supplied as the argument to the CREATE VIRTUAL TABLE
statement is executed when the table is created. It must return either four
or five columns. Each row returned describes one of the component tables. The
first four columns are interpreted, from first to last, as:


   Database URI. A filename or URI that can be used to open the
  database containing the component table.

   Table name. The name of the component table within its database.

   Minimum rowid. The smallest rowid value that the component
  table may contain.

   Maximum rowid. The smallest rowid value that the component
  table may contain.


The interpretation of the final column, if it is present, is 
described here.

For example, say the SQL statement returns the following data when 
executed:


Database URITable nameMinimum rowidMaximum rowid
test.db1 t1 0 10
test.db2 t2 11 20
test.db3 t1 21 30
test.db4 t1 31 40


and the user queries the swarmvtab table for the row with rowid value
25. The swarmvtab table will open database file "test.db3" and read the
data to return from table "t1" (as 25 falls within the range of rowids
assigned to table "t1" in "test.db3").

Swarmvtab efficiently handles range and equality constraints on the
rowid (or other INTEGER PRIMARY KEY) field only. If a query does not 
contain such a constraint, then swarmvtab finds the results by opening
each database in turn and linearly scanning the component table. Which 
generates a correct result, but is often slow.

There must be no overlapping rowid ranges in the rows returned by
the SQL statement. It is an error if there are.

The swarmvtab implementation may open or close databases at any 
point. By default, it attempts to limit the maximum number of 
simultaneously open database files to nine. This is not a hard limit -
it is possible to construct a scenario that will cause swarmvtab to 
exceed it.


2. Compiling and Using Swarmvtab

The code for the swarmvtab virtual table is found in the
ext/misc/unionvtab.c file of the main SQLite source tree. It may be compiled
into an SQLite loadable extension using a command like:

gcc -g -fPIC -shared unionvtab.c -o unionvtab.so


Alternatively, the unionvtab.c file may be compiled into the application. 
In this case, the following function should be invoked to register the
extension with each new database connection:

int sqlite3_unionvtab_init(sqlite3 *db, void*, void*);


 The first argument passed should be the database handle to register the
extension with. The second and third arguments should both be passed 0.

 The source file and entry point are named for "unionvtab" instead of
"swarmvtab". Unionvtab is a separately documented virtual table 
that is bundled with swarmvtab.


3. Advanced Usage

Most users of swarmvtab will only use the features described above. 
This section describes features designed for more esoteric use cases. These
features all involve specifying extra optional parameters following the SQL
statement as part of the CREATE VIRTUAL TABLE command. An optional parameter 
is specified using its name, followed by an "=" character, followed by an
optionally quoted value. Whitespace may separate the name, "=" character 
and value. For example:

CREATE VIRTUAL TABLE temp.sv USING swarmvtab (
  'SELECT ...',                -- the SELECT statement
  maxopen = 20,                -- An optional parameter
  missing='missing_udf'        -- Another optional parameter
);


The following sections describe the supported parameters. Specifying
an unrecognized parameter name is an error.


3.1. SQL Parameters

If a parameter name begins with a ":", then it is assumed to be a
value to bind to the SQL statement before executing it. The value is always
bound as text. It is an error if the specified SQL parameter does not
exist. For example:

CREATE VIRTUAL TABLE temp.x1 USING swarmvtab (
  "SELECT :dir || local_filename, tbl, min, max FROM components",
  :dir = '/home/user/app/databases/'
);


When the above CREATE VIRTUAL TABLE statement is executed, swarmvtab binds
the text value "/home/user/app/databases/" to the :dir parameter of the
SQL statement before executing it.

A single CREATE VIRTUAL TABLE statement may contain any number of SQL
parameters.


3.2. The "maxopen" Parameter

By default, swarmvtab attempts to limit the number of simultaneously
open databases to nine. This parameter allows that limit to be changed.
For example, to create a swarmvtab table that may hold up to 30 databases
open simultaneously:

CREATE VIRTUAL TABLE temp.x1 USING swarmvtab (
  "SELECT ...",
  maxopen=30
);


Raising the number of open databases may improve performance in some
scenarios.


3.3. The "openclose" Callback

The "openclose" parameter allows the user to specify the name of a
application-defined SQL function that will be invoked just before
swarmvtab opens a database, and again just after it closes one. The first
argument passed to the open close function is the filename or URI
identifying the database to be opened or just recently closed (the same
value returned in the leftmost column of the SQL statement provided to
the CREATE VIRTUAL TABLE command). The second argument is integer value
0 when the function is invoked before opening a database, and 1 when it
is invoked after one is closed. For example, if:

CREATE VIRTUAL TABLE temp.x1 USING swarmvtab (
  "SELECT ...",
  openclose = 'openclose_udf'
);


then before each database containing a component table is opened, 
swarmvtab effectively executes:

SELECT openclose_udf(&lt;database-name&gt;, 0);


After a database is closed, swarmvtab runs the equivalent of:

SELECT openclose_udf(&lt;database-name&gt;, 1);


Any value returned by the openclose function is ignored. If an invocation
made before opening a database returns an error, then the database file is
not opened and the error returned to the user. This is the only scenario
in which swarmvtab will issue an "open" invocation without also eventually
issuing a corresponding "close" call. If there are still databases open,
"close" calls may be issued from within the eventual sqlite3_close() call
on the applications database that deletes the temp schema in which the
swarmvtab table resides.

Errors returned by "close" invocations are always ignored.


3.4. The "missing" Callback

The "missing" parameter allows the user to specify the name of a
application-defined SQL function that will be invoked just before
swarmvtab opens a database if it finds that the required database file
is not present on disk. This provides the application with an opportunity
to retrieve the required database from a remote source before swarmvtab
attempts to open it. The only argument passed to the "missing" function
is the name or URI that identifies the database being opened. Assuming:

CREATE VIRTUAL TABLE temp.x1 USING swarmvtab (
  "SELECT ...",
  openclose = 'openclose_udf',
  missing='missing_udf'
);


then the missing function is invoked as follows:

SELECT missing_udf(&lt;database-name&gt;);


If the missing function returns an error, then the database is not 
opened and the error returned to the user. If an openclose function is
configured, then a "close" invocation is issued at this point to match
the earlier "open". The following pseudo-code illustrates the procedure used
by a swarmvtab instance with both missing and openclose functions configured
when a component database is opened.

SELECT openclose_udf(&lt;database-name&gt;, 0);
if( error ) return error;
if( db does not exist ){
  SELECT missing_udf(&lt;database-name&gt;);
  if( error ){
    SELECT openclose_udf(&lt;database-name&gt;, 1);
    return error;
  }
}
sqlite3_open_v2(&lt;database-name&gt;);
if( error ){
  SELECT openclose_udf(&lt;database-name&gt;, 1);
  return error;
}
// db successfully opened!



3.5. Component table "context" values

 If the SELECT statement specified as part of the CREATE VIRTUAL 
TABLE command returns five columns, then the final column is used
for application context only. Swarmvtab does not use this value at
all, except that it is passed after &lt;database-name&gt; to both
the openclose and missing functions, if specified. In other words,
instead of invoking the functions as described above, if the "context"
column is present swarmvtab instead invokes:

SELECT missing_udf(&lt;database-name&gt;, &lt;context&gt;);
SELECT openclose_udf(&lt;database-name&gt;, &lt;context&gt;, 0);
SELECT openclose_udf(&lt;database-name&gt;, &lt;context&gt;, 1);


as required.


c2: SQLite In 5 Minutes Or Less
c5: quickstart.html
c4:
Here is what you do to start experimenting with SQLite without having
to do a lot of tedious reading and configuration:

Download The Code


Get a copy of the prebuilt binaries for your machine, or get a copy
of the sources and compile them yourself.  Visit
the download page for more information.


Create A New Database


At a shell or DOS prompt, enter: "sqlite3 test.db".  This will
create a new database named "test.db".  (You can use a different name if
you like.)
Enter SQL commands at the prompt to create and populate the
new database.
Additional documentation is available here.


Write Programs That Use SQLite


Below is a simple 
TCL program that demonstrates how to use
the TCL interface to SQLite.  The program executes the SQL statements
given as the second argument on the database defined by the first
argument.  The commands to watch for are the sqlite3 command
on line 7 which opens an SQLite database and creates
a new object named "db" to access that database, the
use of the eval method on the db object on line 8 to run
SQL commands against the database, and the closing of the database connection
on the last line of the script.


01  #!/usr/bin/tclsh
02  if {$argc!=2} {
03    puts stderr "Usage: %s DATABASE SQL-STATEMENT"
04    exit 1
05  }
06  package require sqlite3
07  sqlite3 db &#91;lindex $argv 0]
08  db eval &#91;lindex $argv 1] x {
09    foreach v $x(*) {
10      puts "$v = $x($v)"
11    }
12    puts ""
13  }
14  db close



Below is a simple C program that demonstrates how to use
the C/C++ interface to SQLite.  The name of a database is given by
the first argument and the second argument is one or more SQL statements
to execute against the database.  The function calls to pay attention
to here are the call to sqlite3_open() on line 22 which opens
the database, sqlite3_exec() on line 28 that executes SQL
commands against the database, and sqlite3_close() on line 33
that closes the database connection.

See also the Introduction To The SQLite C/C++ Interface for
an introductory overview and roadmap to the dozens of SQLite interface
functions.


01  #include &lt;stdio.h&gt;
02  #include &lt;sqlite3.h&gt;
03  
04  static int callback(void *NotUsed, int argc, char **argv, char **azColName){
05    int i;
06    for(i=0; i&lt;argc; i++){
07      printf("%s = %s\n", azColName&#91;i], argv&#91;i] ? argv&#91;i] : "NULL");
08    }
09    printf("\n");
10    return 0;
11  }
12  
13  int main(int argc, char **argv){
14    sqlite3 *db;
15    char *zErrMsg = 0;
16    int rc;
17  
18    if( argc!=3 ){
19      fprintf(stderr, "Usage: %s DATABASE SQL-STATEMENT\n", argv&#91;0]);
20      return(1);
21    }
22    rc = sqlite3_open(argv&#91;1], &db);
23    if( rc ){
24      fprintf(stderr, "Can't open database: %s\n", sqlite3_errmsg(db));
25      sqlite3_close(db);
26      return(1);
27    }
28    rc = sqlite3_exec(db, argv&#91;2], callback, 0, &zErrMsg);
29    if( rc!=SQLITE_OK ){
30      fprintf(stderr, "SQL error: %s\n", zErrMsg);
31      sqlite3_free(zErrMsg);
32    }
33    sqlite3_close(db);
34    return 0;
35  }

See the How To Compile SQLite document for instructions and hints on
how to compile the program shown above.


c2: STRICT Tables
c5: stricttables.html
c4:
1. Introduction

SQLite strives to be flexible regarding the datatype of
the content that it stores.  For example, if a table column has a type of
"INTEGER", then SQLite tries to convert anything inserted into that column
into an integer.  So an attempt to insert the string '123' results
in an integer 123 being inserted.  But if the content cannot be losslessly
converted into an integer, for example if the input is 'xyz', then
the original string is inserted instead.
See the Datatypes In SQLite document for additional information.

Some developers appreciate the freedom that SQLite's flexible typing
rules provide and use that freedom to advantage.
But other developers are aghast at SQLite's
flagrant rule-breaking and prefer the traditional rigid type
system found in all other SQL database engines and in the
SQL standard.  For this latter group, SQLite supports a strict typing
mode, as of version 3.37.0 (2021-11-27), that is enabled
separately for each table.

2. STRICT Tables

In a CREATE TABLE statement, if the "STRICT" table-option keyword is
added to the end, after the closing ")", then strict typing rules apply
to that table. 
The STRICT keyword causes the following differences:



Every column definition must specify a datatype for that column.
The freedom to specify a column without a datatype is removed.


The datatype must be one of following:

 INT
 INTEGER
 REAL
 TEXT
 BLOB
 ANY

No other datatype names are allowed, though new types might be added in
future releases of SQLite.


Content inserted into the column with a datatype other than ANY
must be either a NULL (assuming there
is no NOT NULL constraint on the column) or the type specified.
SQLite attempts to coerce the data into the appropriate type using the usual
affinity rules, as PostgreSQL, MySQL, SQL Server,
and Oracle all do.  If the value cannot be
losslessly converted in the specified datatype, then an
SQLITE_CONSTRAINT_DATATYPE error is raised.


Columns with datatype ANY can accept any kind of data (except they will
reject NULL values if they have a NOT NULL constraint, of course).  No
type coercion occurs for a column of type ANY in a STRICT table.


Columns that are part of the PRIMARY KEY are implicitly NOT NULL.
However, even though the PRIMARY KEY has an implicit NOT NULL constraint,
when a NULL value is inserted into an INTEGER PRIMARY KEY column, the
NULL is automatically converted into a unique integer, using the same
rules for INTEGER PRIMARY KEY on ordinary, non-strict tables.


The PRAGMA integrity_check and PRAGMA quick_check commands check the
type of the content of all columns in STRICT tables and show errors if
anything is amiss.



Everything else about a STRICT table works the same as it does in an
ordinary non-strict table:


 CHECK constraints work the same.
 NOT NULL constraints work the same.
 FOREIGN KEY constraints work the same.
 UNIQUE constraints work the same.
 DEFAULT clauses work the same.
 COLLATE clauses work the same.
 Generated columns work the same.
 ON CONFLICT clauses work the same.
 Indexes work the same.
 AUTOINCREMENT works the same.
 An INTEGER PRIMARY KEY column is an alias for the rowid, but an
INT PRIMARY KEY column is not.
 The on-disk format for the table data is the same.


3. The ANY datatype

The ability to host any type of data in a single column has proven to
be remarkably useful over the years.  In order to continue supporting this
ability, even in STRICT tables, the new ANY datatype name is introduced.
When the datatype of a column is "ANY", that means that any kind of data - 
integers, floating point values, strings, or binary blobs, can be inserted
into that table and its value and datatype will be preserved exactly as
it is inserted.  As far as we know, SQLite is the only SQL database engine
that supports this advanced capability.

The behavior of ANY is slightly different in a
STRICT table versus an ordinary non-strict table.  In a STRICT table,
a column of type ANY always preserves the data exactly as it is received.
For an ordinary non-strict table, a column of type ANY will attempt to
convert strings that look like numbers into a numeric value, and if
successful will store the numeric value rather than the original string.
For example:



STRICTordinary non-strict
CREATE TABLE t1(a ANY) STRICT;
INSERT INTO t1 VALUES('000123');
SELECT typeof(a), quote(a) FROM t1;
-- result: text '000123'

CREATE TABLE t1(a ANY);
INSERT INTO t1 VALUES('000123');
SELECT typeof(a), quote(a) FROM t1;
-- result: integer 123



4. Backwards Compatibility

The STRICT keyword at the end of a CREATE TABLE statement is only
recognized by SQLite version 3.37.0 (2021-11-27) and later.  If
you try to open a database containing the STRICT keyword in an earlier
version of SQLite, it will not recognize the keyword and will report
an error (except as noted below).  But apart from the extra STRICT keyword,
the underlying file format of the database is identical.

Thus, in general, a database file that contains one or more STRICT
tables can only be read and written by SQLite version 3.37.0 or later.
However, a database created by SQLite 3.37.0 or later can still be
read and written by earlier versions of SQLite, going all the way back
to version 3.0.0 (2004-06-18) as long as the database does not contain
any STRICT tables or other features that were introduced after the older
version of SQLite.

The STRICT keyword may still be used as an identifier.
(It is only treated as a keyword in a certain part of the syntax,
and sqlite3_keyword_check(..) does not recognize it as a regular keyword.)

4.1. Accessing STRICT tables in earlier versions of SQLite

Because of a quirk in the SQL language parser, versions of SQLite prior
to 3.37.0 can still read and write STRICT tables if they set
"PRAGMA writable_schema=ON" immediately after opening the database
file, prior to doing anything else that requires knowing the schema.
One of the features of PRAGMA writable_schema=ON is that it disables
errors in the schema parser.  This is intentional, because a big reason for
having PRAGMA writable_schema=ON is to facilitate recovery of database files
with corrupt schemas.  So with writable_schema=ON, when the schema
parser reaches the STRICT keyword, it says to itself "I don't know what
to do with this, but everything up to this point seems like a valid
table definition so I'll just use what I have."  Hence, the STRICT
keyword is effectively ignored.  Because nothing else about the file
format changes for STRICT tables, everything else will work normally.
Of course, rigid type enforcement will not occur because the earlier
versions of SQLite do not know how to do that.

The .dump command in the CLI sets PRAGMA writable_schema=ON, because
.dump is designed to extract as much content as it can even from a corrupt
database file.  Hence, if you are using an older version of SQLite and
you open a database with STRICT tables in the CLI and issue the ".dump"
command before doing anything else, you will be able to read and write
to the STRICT tables without rigid type enforcement.  This could, potentially,
corrupt the database, by allowing incorrect types into STRICT tables.
Reopening the database with a newer version of SQLite and running
"PRAGMA quick_check" will detect and report all such corruption.

5. Other Table Options

The SQLite parser accepts a comma-separated list of table options after
the final close parenthesis in a CREATE TABLE statement.  As of this
writing (2021-08-23) only two options are recognized:


 STRICT
 WITHOUT ROWID


If there are multiple options, they can be specified in any order.
To keep things simple, the current parser accepts duplicate options without
complaining, but that might change in future releases, so applications
should not rely on it.


c2: The dbhash.exe Utility Program
c5: dbhash.html
c4:
1. Overview

The dbhash (or dbhash.exe on Windows) utility is a
command-line program that computes the SHA1 hash of the schema and content 
for an SQLite database.

Dbhash ignores extraneous formatting details and hashes only the database
schema and content.  Hence the hash is constant even if the database file
is modified by:


 VACUUM
 PRAGMA page_size
 PRAGMA journal_mode
 REINDEX
 ANALYZE
 copied via the backup API
 ... and so forth


The operations above can potentially cause vast changes the raw database file,
and hence cause very different SHA1 hashes at the file level.
But since the content represented in the database file is unchanged by these
operations, the hash computed by dbhash is also unchanged.

Dbhash can be used to compare two databases to confirm that they
are equivalent, even though their representation on disk is quite different.
Dbhash might also be used to verify the content of a remote database without having
to transmit the entire content of the remote database over a slow link.

2. Usage

Dbhash is a command-line utility.
To run it, type "dbhash" on a command-line prompt followed by the names of
one or more SQLite database files that are to be hashed.
The database hashes will be displayed on standard output.
For example:

drh@bella:~/sqlite/bld$ dbhash ~/Fossils/sqlite.fossil
8d3da9ff87196312aaa33076627ccb7943ef79e3 /home/drh/Fossils/sqlite.fossil


Dbhash supports command-line options that can restrict the tables of the
database file that are hashed, or restrict the hash to only content or only
the schema.  Run "dbhash --help" for further information.

3. Building

To build a copy of the dbhash utility program on unix, get a copy of the
canonical SQLite source code and enter:

./configure
make dbhash


On Windows, enter:

nmake /f makefile.msc dbhash.exe


The dbhash program is implemented by a single file of C-code
called dbhash.c.
To build the dbhash program manually, simply compile the dbhash.c source file
and link it against the SQLite library.


c2: SQLite Frequently Asked Questions
c5: faq.html
c4:
Frequently Asked QuestionsHow do I create an AUTOINCREMENT field?What datatypes does SQLite support?SQLite lets me insert a string into a database column of type integer!Why doesn't SQLite allow me to use '0' and '0.0' as the primary
  key on two different rows of the same table?Can multiple applications or multiple instances of the same
  application access a single database file at the same time?Is SQLite threadsafe?How do I list all tables/indices contained in an SQLite databaseAre there any known size limits to SQLite databases?What is the maximum size of a VARCHAR in SQLite?Does SQLite support a BLOB type?How do I add, delete or rename columns from an existing table in SQLite?I deleted a lot of data but the database file did not get any
  smaller.  Is this a bug?Can I use SQLite in my commercial product without paying royalties?How do I use a string literal that contains an embedded single-quote (')
  character?What is an SQLITE_SCHEMA error, and why am I getting one?I get some compiler warnings when I compile SQLite.
  Isn't this a problem?  Doesn't it indicate poor code quality?Case-insensitive matching of Unicode characters does not work.INSERT is really slow - I can only do few dozen INSERTs per secondI accidentally deleted some important information from my SQLite database.
  How can I recover it?What is an SQLITE_CORRUPT error?  What does it mean for the database
  to be "malformed"? Why am I getting this error?Does SQLite support foreign keys?I get a compiler error if I use the SQLITE_OMIT_... 
  compile-time options when building SQLite.My WHERE clause expression column1="column1" does not work.
  It causes every row of the table to be returned, not just the rows
  where column1 has the value "column1".How are the syntax diagrams (a.k.a. "railroad" diagrams) for
  SQLite generated?The SQL standard requires that a UNIQUE constraint be enforced even if
  one or more of the columns in the constraint are NULL, but SQLite does
  not do this.  Isn't that a bug?What is the Export Control Classification Number (ECCN) for SQLite?My query does not return the column name that I expect.  Is this a bug?
(1) How do I create an AUTOINCREMENT field?
Short answer: A column declared INTEGER PRIMARY KEY will
  autoincrement.

  Longer answer:
  If you declare a column of a table to be INTEGER PRIMARY KEY, then
  whenever you insert a NULL
  into that column of the table, the NULL is automatically converted
  into an integer which is one greater than the largest value of that
  column over all other rows in the table, or 1 if the table is empty.
  Or, if the largest existing integer key 9223372036854775807 is in use then an
  unused key value is chosen at random.
  For example, suppose you have a table like this:

CREATE TABLE t1(
  a INTEGER PRIMARY KEY,
  b INTEGER
);

  With this table, the statement

INSERT INTO t1 VALUES(NULL,123);

  is logically equivalent to saying:

INSERT INTO t1 VALUES((SELECT max(a) FROM t1)+1,123);


  There is a function named
  sqlite3_last_insert_rowid() which will return the integer key
  for the most recent insert operation.

  Note that the integer key is one greater than the largest
  key that was in the table just prior to the insert.  The new key
  will be unique over all keys currently in the table, but it might
  overlap with keys that have been previously deleted from the
  table.  To create keys that are unique over the lifetime of the
  table, add the AUTOINCREMENT keyword to the INTEGER PRIMARY KEY
  declaration.  Then the key chosen will be one more than the
  largest key that has ever existed in that table.  If the largest
  possible key has previously existed in that table, then the INSERT
  will fail with an SQLITE_FULL error code.

(2) What datatypes does SQLite support?
SQLite uses dynamic typing.  Content can be stored as INTEGER,
  REAL, TEXT, BLOB, or as NULL.

(3) SQLite lets me insert a string into a database column of type integer!
This is a feature, not a bug.  SQLite uses dynamic typing. 
  It does not enforce data type  constraints.  Data of any type can
  (usually) be inserted into any column.  You can put arbitrary length
  strings into integer columns, floating point numbers in boolean columns,
  or dates in character columns.  The datatype you assign to a column in the
  CREATE TABLE command does not restrict what data can be put into
  that column.  Every column is able to hold
  an arbitrary length string.  (There is one exception: Columns of
  type INTEGER PRIMARY KEY may only hold a 64-bit signed integer.
  An error will result
  if you try to put anything other than an integer into an
  INTEGER PRIMARY KEY column.)

  But SQLite does use the declared type of a column as a hint
  that you prefer values in that format.  So, for example, if a
  column is of type INTEGER and you try to insert a string into
  that column, SQLite will attempt to convert the string into an
  integer.  If it can, it inserts the integer instead.  If not,
  it inserts the string.  This feature is called type affinity.
  

(4) Why doesn't SQLite allow me to use '0' and '0.0' as the primary
  key on two different rows of the same table?
This problem occurs when your primary key is a numeric type.  Change the
  datatype of your primary key to TEXT and it should work.

  Every row must have a unique primary key.  For a column with a
  numeric type, SQLite thinks that '0' and '0.0' are the
  same value because they compare equal to one another numerically.
  (See the previous question.)  Hence the values are not unique.

(5) Can multiple applications or multiple instances of the same
  application access a single database file at the same time?
Multiple processes can have the same database open at the same
  time.  Multiple processes can be doing a SELECT
  at the same time.  But only one process can be making changes to
  the database at any moment in time, however.

  SQLite uses reader/writer locks to control access to the database.
  (Under Win95/98/ME which lacks support for reader/writer locks, a
  probabilistic simulation is used instead.)
  But use caution: this locking mechanism might
  not work correctly if the database file is kept on an NFS filesystem.
  This is because fcntl() file locking is broken on many NFS implementations.
  You should avoid putting SQLite database files on NFS if multiple
  processes might try to access the file at the same time.  On Windows,
  Microsoft's documentation says that locking may not work under FAT
  filesystems if you are not running the Share.exe daemon.  People who
  have a lot of experience with Windows tell me that file locking of
  network files is very buggy and is not dependable.  If what they
  say is true, sharing an SQLite database between two or more Windows
  machines might cause unexpected problems.

  We are aware of no other embedded SQL database engine that
  supports as much concurrency as SQLite.  SQLite allows multiple processes
  to have the database file open at once, and for multiple processes to
  read the database at once.  When any process wants to write, it must
  lock the entire database file for the duration of its update.  But that
  normally only takes a few milliseconds.  Other processes just wait on
  the writer to finish then continue about their business.  Other embedded
  SQL database engines typically only allow a single process to connect to
  the database at once.

  However, client/server database engines (such as PostgreSQL, MySQL,
  or Oracle) usually support a higher level of concurrency and allow
  multiple processes to be writing to the same database at the same time.
  This is possible in a client/server database because there is always a
  single well-controlled server process available to coordinate access.
  If your application has a need for a lot of concurrency, then you should
  consider using a client/server database.  But experience suggests that
  most applications need much less concurrency than their designers imagine.
  

  When SQLite tries to access a file that is locked by another
  process, the default behavior is to return SQLITE_BUSY.  You can
  adjust this behavior from C code using the 
  sqlite3_busy_handler() or sqlite3_busy_timeout()
  API functions.

(6) Is SQLite threadsafe?
Threads are evil.
  Avoid them.

  SQLite is threadsafe.  We make this concession since many users choose
  to ignore the advice given in the previous paragraph.
  But in order to be thread-safe, SQLite must be compiled
  with the SQLITE_THREADSAFE preprocessor macro set to 1.  Both the Windows
  and Linux precompiled binaries in the distribution are compiled this way.
  If you are unsure if the SQLite library you are linking against is compiled
  to be threadsafe you can call the sqlite3_threadsafe()
  interface to find out.
  

  SQLite is threadsafe because it uses mutexes to serialize
  access to common data structures.  However, the work of acquiring and
  releasing these mutexes will slow SQLite down slightly.  Hence, if you
  do not need SQLite to be threadsafe, you should disable the mutexes
  for maximum performance.  See the threading mode documentation for
  additional information.

  Under Unix, you should not carry an open SQLite database across
  a fork() system call into the child process.

(7) How do I list all tables/indices contained in an SQLite database
If you are running the sqlite3 command-line access program
  you can type ".tables" to get a list of all tables.  Or you
  can type ".schema" to see the complete database schema including
  all tables and indices.  Either of these commands can be followed by
  a LIKE pattern that will restrict the tables that are displayed.

  From within a C/C++ program (or a script using Tcl/Ruby/Perl/Python
  bindings) you can get access to table and index names by doing a SELECT
  on a special table named "SQLITE_SCHEMA".  Every SQLite database
  has an SQLITE_SCHEMA table that defines the schema for the database.
  The SQLITE_SCHEMA table looks like this:

CREATE TABLE sqlite_schema (
  type TEXT,
  name TEXT,
  tbl_name TEXT,
  rootpage INTEGER,
  sql TEXT
);

  For tables, the type field will always be 'table' and the
  name field will be the name of the table.  So to get a list of
  all tables in the database, use the following SELECT command:

SELECT name FROM sqlite_schema
WHERE type='table'
ORDER BY name;

  For indices, type is equal to 'index', name is the
  name of the index and tbl_name is the name of the table to which
  the index belongs.  For both tables and indices, the sql field is
  the text of the original CREATE TABLE or CREATE INDEX statement that
  created the table or index.  For automatically created indices (used
  to implement the PRIMARY KEY or UNIQUE constraints) the sql field
  is NULL.

  The SQLITE_SCHEMA table cannot be modified using UPDATE, INSERT, 
  or DELETE (except under
  extraordinary conditions).  
  The SQLITE_SCHEMA table is automatically updated by commands like
  CREATE TABLE, CREATE INDEX, DROP TABLE, and DROP INDEX.

  Temporary tables do not appear in the SQLITE_SCHEMA table.  Temporary
  tables and their indices and triggers occur in another special table
  named SQLITE_TEMP_SCHEMA.  SQLITE_TEMP_SCHEMA works just like SQLITE_SCHEMA
  except that it is only visible to the application that created the 
  temporary tables.  To get a list of all tables, both permanent and
  temporary, one can use a command similar to the following:

SELECT name FROM 
   (SELECT * FROM sqlite_schema UNION ALL
    SELECT * FROM sqlite_temp_schema)
WHERE type='table'
ORDER BY name


(8) Are there any known size limits to SQLite databases?
See limits.html for a full discussion of
  the limits of SQLite.

(9) What is the maximum size of a VARCHAR in SQLite?
SQLite does not enforce the length of a VARCHAR.  You can declare
  a VARCHAR(10) and SQLite will be happy to store a 500-million character
  string there.  And it will keep all 500-million characters intact.
  Your content is never truncated.  SQLite understands the column type
  of "VARCHAR(N)" to be the same as "TEXT", regardless of the value
  of N.
  

(10) Does SQLite support a BLOB type?
SQLite allows you to store BLOB data in any 
  column, even columns that are declared to hold some other type.
  BLOBs can even be used as PRIMARY KEYs.

(11) How do I add, delete or rename columns from an existing table in SQLite?
SQLite has limited ALTER TABLE support that you can use to
  add, rename or drop columns or to change the name of a table
  as detailed at ALTER TABLE.

  If you want to make more complex changes in the structure or
  constraints of a table or its columns, you will have to recreate it.
  You can save existing data to a temporary table, drop the
  old table, create the new table, then copy the data back in from
  the temporary table. See 
  Making Other Kinds Of Table Schema Changes for procedure.

(12) I deleted a lot of data but the database file did not get any
  smaller.  Is this a bug?
No.  When you delete information from an SQLite database, the
  unused disk space is added to an internal "free-list" and is reused
  the next time you insert data.  The disk space is not lost.  But
  neither is it returned to the operating system.

  If you delete a lot of data and want to shrink the database file,
  run the VACUUM command.
  VACUUM will reconstruct
  the database from scratch.  This will leave the database with an empty
  free-list and a file that is minimal in size.  Note, however, that the
  VACUUM can take some time to run and it can use up to twice
  as much temporary disk space as the original file while it is running.
  

  An alternative to using the VACUUM command
  is auto-vacuum mode, enabled using the 
  auto_vacuum pragma.

(13) Can I use SQLite in my commercial product without paying royalties?
Yes.  SQLite is in the 
  public domain.  No claim of ownership is made
  to any part of the code.  You can do anything you want with it.

(14) How do I use a string literal that contains an embedded single-quote (')
  character?
The SQL standard specifies that single-quotes in strings are escaped
  by putting two single quotes in a row.  SQL works like the Pascal programming
  language in this regard. Example:
  

  
    INSERT INTO xyz VALUES('5 O''clock');
  

(15) What is an SQLITE_SCHEMA error, and why am I getting one?
An SQLITE_SCHEMA error is returned when a 
  prepared SQL statement is no longer valid and cannot be executed.
  When this occurs, the statement must be recompiled from SQL using 
  the sqlite3_prepare() API.
  An SQLITE_SCHEMA error can only occur when using the sqlite3_prepare(),
  and sqlite3_step() interfaces to run SQL.
  You will never receive an SQLITE_SCHEMA error from
  sqlite3_exec().  Nor will you receive an error if you
  prepare statements using sqlite3_prepare_v2() instead of
  sqlite3_prepare().

  The sqlite3_prepare_v2() interface creates a
  prepared statement that will automatically recompile itself if
  the schema changes.  The easiest way to deal with
  SQLITE_SCHEMA errors is to always use sqlite3_prepare_v2()
  instead of sqlite3_prepare().

(17) I get some compiler warnings when I compile SQLite.
  Isn't this a problem?  Doesn't it indicate poor code quality?
Quality assurance in SQLite is done using 
  full-coverage testing,
  not by compiler warnings or other static code analysis tools.
  In other words, we verify that SQLite actually gets the
  correct answer, not that it merely satisfies stylistic constraints.
  Most of the SQLite code base is devoted purely to testing.
  The SQLite test suite runs tens of thousands of separate test cases and
  many of those test cases are parameterized so that hundreds of millions
  of tests involving billions of SQL statements are run and evaluated
  for correctness prior to every release.  The developers use code
  coverage tools to verify that all paths through the code are tested.
  Whenever a bug is found in SQLite, new test cases are written to
  exhibit the bug so that the bug cannot recur undetected in the future.

  During testing, the SQLite library is compiled with special
  instrumentation that allows the test scripts to simulate a wide
  variety of failures in order to verify that SQLite recovers
  correctly.  Memory allocation is carefully tracked and no memory
  leaks occur, even following memory allocation failures.  A custom
  VFS layer is used to simulate operating system crashes and power
  failures in order to ensure that transactions are atomic across
  these events.  A mechanism for deliberately injecting I/O errors
  shows that SQLite is resilient to such malfunctions.  (As an
  experiment, try inducing these kinds of errors on other SQL database
  engines and see what happens!)

  We also run SQLite using Valgrind
  on Linux and verify that it detects no problems.

  Some people say that we should eliminate all warnings because
  benign warnings mask real warnings that might arise in future changes.
  This is true enough.  But in reply, the developers observe that all
  warnings have already been fixed in the builds
  used for SQLite development (various versions of GCC, MSVC,
  and clang).
  Compiler warnings usually only arise from compilers or compile-time 
  options that the SQLite developers do not use themselves.

(18) Case-insensitive matching of Unicode characters does not work.
The default configuration of SQLite only supports case-insensitive
  comparisons of ASCII characters.  The reason for this is that doing
  full Unicode case-insensitive comparisons and case conversions 
  requires tables and logic that would nearly double the size of
  the SQLite library.  The
  SQLite developers reason that any application that needs full
  Unicode case support probably already has the necessary tables and
  functions and so SQLite should not take up space to 
  duplicate this ability.

  Instead of providing full Unicode case support by default, 
  SQLite provides the ability to link against external
  Unicode comparison and conversion routines.
  The application can overload the built-in NOCASE collating
  sequence (using sqlite3_create_collation()) and the built-in
  like(), upper(), and lower() functions
  (using sqlite3_create_function()).  
  The SQLite source code includes an "ICU" extension that does 
  these overloads.  Or, developers can write their own overloads
  based on their own Unicode-aware comparison routines already
  contained within their project.

(19) INSERT is really slow - I can only do few dozen INSERTs per second
Actually, SQLite will easily do 50,000 or more INSERT statements per second
  on an average desktop computer.  But it will only do a few dozen transactions
  per second.  Transaction speed is limited by the rotational speed of
  your disk drive.  A transaction normally requires two complete rotations
  of the disk platter, which on a 7200RPM disk drive limits you to about
  60 transactions per second.

  Transaction speed is limited by disk drive speed because (by default)
  SQLite actually waits until the data really is safely stored on the disk
  surface before the transaction is complete.  That way, if you suddenly lose
  power or if your OS crashes, your data is still safe.  For details, 
  read about atomic commit in SQLite..

  By default, each INSERT statement is its own transaction.  But if you
  surround multiple INSERT statements with BEGIN...COMMIT then all the
  inserts are grouped into a single transaction.  The time needed to commit
  the transaction is amortized over all the enclosed insert statements and
  so the time per insert statement is greatly reduced.

  Another option is to run PRAGMA synchronous=OFF.  This command will
  cause SQLite to not wait on data to reach the disk surface, which will make
  write operations appear to be much faster.  But if you lose power in the
  middle of a transaction, your database file might go corrupt.

(20) I accidentally deleted some important information from my SQLite database.
  How can I recover it?
If you have a backup copy of your database file, recover the information
  from your backup.

  If you do not have a backup, recovery is very difficult.  You might
  be able to find partial string data in a binary dump of the raw database
  file.  Recovering numeric data might also be possible given special tools,
  though to our knowledge no such tools exist.  SQLite is sometimes compiled
  with the SQLITE_SECURE_DELETE option which overwrites all deleted content
  with zeros.  If that is the case then recovery is clearly impossible.
  Recovery is also impossible if you have run VACUUM since the data was
  deleted.  If SQLITE_SECURE_DELETE is not used and VACUUM has not been run,
  then some of the deleted content might still be in the database file, in
  areas marked for reuse.  But, again, there exist no procedures or tools
  that we know of to help you recover that data.

(21) What is an SQLITE_CORRUPT error?  What does it mean for the database
  to be "malformed"? Why am I getting this error?
An SQLITE_CORRUPT error is returned when SQLite detects an error
  in the structure, format, or other control elements of the
  database file.

  SQLite does not corrupt database files without external help.
  If your application crashes in the middle of an
  update, your data is safe.  The database is safe even if your OS
  crashes or takes a power loss.  The crash-resistance of SQLite has
  been extensively studied and tested and is attested by years of real-world 
  experience by billions of users.

  That said, there are a number of things that external programs or bugs
  in your hardware or OS can do to corrupt a database file.  See
  How To Corrupt An SQLite Database File for
  further information.

  You can use PRAGMA integrity_check 
  to do a thorough but time intensive test of the database integrity.

  You can use PRAGMA quick_check to do a faster 
  but less thorough test of the database integrity.

  Depending how badly your database is corrupted, you may be able to 
  recover some of the data by using the CLI to dump the schema and contents
  to a file and then recreate.  Unfortunately, once humpty-dumpty falls off 
  the wall, it is generally not possible to put him back together again.

(22) Does SQLite support foreign keys?

  As of version 3.6.19 (2009-10-14), 
  SQLite supports foreign key constraints.  But enforcement
  of foreign key constraints is turned off by default (for backwards compatibility).
  To enable foreign key constraint enforcement, run 
  PRAGMA foreign_keys=ON or compile with
  -DSQLITE_DEFAULT_FOREIGN_KEYS=1.

(23) I get a compiler error if I use the SQLITE_OMIT_... 
  compile-time options when building SQLite.
The SQLITE_OMIT_... compile-time options only work
  when building from canonical source files.  They do not work
  when you build from the SQLite amalgamation or from the pre-processed
  source files.

  It is possible to build a special amalgamation that will work with
  a predetermined set of SQLITE_OMIT_... options.  Instructions for doing
  so can be found with the SQLITE_OMIT_... documentation.

(24) My WHERE clause expression column1="column1" does not work.
  It causes every row of the table to be returned, not just the rows
  where column1 has the value "column1".
Use single-quotes, not double-quotes, around string literals in SQL.
  This is what the SQL standard requires.  Your WHERE clause expression
  should read: column1='column1'

  SQL uses double-quotes around identifiers (column or table names) that
  contains special characters or which are keywords.  So double-quotes are
  a way of escaping identifier names.  Hence, when you say
  column1="column1" that is equivalent to 
  column1=column1 which is obviously always true.

(25) How are the syntax diagrams (a.k.a. "railroad" diagrams) for
  SQLite generated?
Each diagram is hand-written using the Pikchr
  diagramming language.  These hand-written specifications are converted into
  SVG and inserted inline in the HTML files as part of the documentation build process.
  
  Many historical versions of the SQLite documentation used a different process for
  generating the syntax diagrams.  The historical process was based on Tcl/Tk and is
  described  at http://wiki.tcl-lang.org/21708.  The newer Pikchr-based syntax diagrams
  first landed on trunk on 2020-09-26.

(26) The SQL standard requires that a UNIQUE constraint be enforced even if
  one or more of the columns in the constraint are NULL, but SQLite does
  not do this.  Isn't that a bug?
Perhaps you are referring to the following statement from SQL92:

  
  A unique constraint is satisfied if and only if no two rows in a
  table have the same non-null values in the unique columns. 
  

  That statement is ambiguous, having at least two possible interpretations:

  
  A unique constraint is satisfied if and only if no two rows in a
  table have the same values and have non-null values in the unique columns.
  A unique constraint is satisfied if and only if no two rows in a 
  table have the same values in the subset of unique columns that are not null.
  

  SQLite follows interpretation (1), as does PostgreSQL, MySQL, Oracle,
  and Firebird.  It is true that Informix and Microsoft SQL Server use
  interpretation (2), however we the SQLite developers hold that 
  interpretation (1) is the most natural reading 
  of the requirement and we also want to maximize compatibility with other
  SQL database engines, and most other database engines also go with (1),
  so that is what SQLite does.

(27) What is the Export Control Classification Number (ECCN) for SQLite?
After careful review of the Commerce Control List (CCL), we are convinced
  that the core public-domain SQLite source code is not described by any ECCN,
  hence the ECCN should be reported as EAR99.

  The above is true for the core public-domain SQLite.  If you extend
  SQLite by adding new code, or if you statically link SQLite with your
  application, that might change the ECCN in your particular case.

(28) My query does not return the column name that I expect.  Is this a bug?
If the columns of your result set are named by AS clauses, then SQLite
  is guaranteed to use the identifier to the right of the AS keyword as the
  column name.  If the result set does not use an AS clause, then SQLite
  is free to name the column anything it wants.
  See the sqlite3_column_name() documentation for further information.


c2: SQLite's Built-in printf()
c5: printf.html
c4:
1. Overview

SQLite contains its own implementation of the string formatting routine "printf()",
accessible via the following interfaces:


 format() &rarr; an SQL function returning the formatted string
 sqlite3_mprintf() &rarr; Store the formatted string in memory obtained
     sqlite3_malloc64().
 sqlite3_snprintf() &rarr; Store the formatted string in a static buffer
 sqlite3_str_appendf() &rarr; Append formatted text to a dynamic string
 sqlite3_vmprintf() &rarr; Varargs version of sqlite3_mprintf()
 sqlite3_vsnprintf() &rarr; Varargs version of sqlite3_snprintf()
 sqlite3_str_vappendf() &rarr; Varargs version of sqlite3_str_appendf()


The same core string formatter is also used internally by SQLite.

1.1. Advantages

Why does SQLite have its own private built-in printf() implementation?
Why not use the printf() implementation from the standard C library?
Several reasons:




By using its own built-in implementation, SQLite guarantees that the
output will be the same on all platforms and in all LOCALEs.
This is important for consistency and for testing.  It would be problematic
if one machine gave an answer of "5.25e+08" and another gave an answer
of "5.250e+008".  Both answers are correct, but it is better when SQLite
always gives the same answer.


We know of no way to use the standard library printf() C interface to
implement the format() SQL function feature of SQLite.  The built-in
printf() implementation can be easily adapted to that task, however.


The printf() in SQLite supports new non-standard substitution
types (%q, %Q, %w, and %z), and enhanced substitution
behavior (%s and %z) that are useful both internally to SQLite
and to applications using SQLite.
Standard library printf()s cannot normally be extended in this way.


Via the sqlite3_mprintf() and sqlite3_vmprintf() interfaces,
the built-in printf() implementation supports the ability to render an
arbitrary-length string into a memory buffer obtained from sqlite3_malloc64().
This is safer and less error prone than trying to precompute an upper size
limit on the result string, allocate an appropriately sized buffer, and
then calling snprintf().


The SQLite-specific printf() supports a new flag (!) called the
"alternate-form-2" flag.  The alternate-form-2 flag changes the processing
of floating-point conversions in subtle ways so that the output is always
an SQL-compatible text representation of a floating-point number - something
that is not possible to achieve with standard-library printf().  For
string substitutions, the alternate-form-2 flag causes the width and
precision to be measured in characters instead of bytes, which simplifies
processing of strings containing multi-byte UTF8 characters.


The built-in SQLite has compile-time options such as
SQLITE_PRINTF_PRECISION_LIMIT that provide defense against 
denial-of-service attacks for application that expose the
printf() functionality to untrusted users.


Using a built-in printf() implementation means that SQLite has one
fewer dependency on the host environment, making it more portable.


1.2. Disadvantages


In fairness, having a built-in implementation of printf() also comes with
some disadvantages.  To wit:



The built-in printf() implementation uses extra code space 
(about 7800 bytes on GCC 5.4 with -Os).


The floating-point to text conversion subfunction for the built-in printf()
is limited in precision to 16 significant digits or 26 significant digits
if the "!" alternate-form-2 flag is used.
Every IEEE-754 double can be represented exactly as a decimal floating-point
value, but some doubles require more than 16 or 26 significant digits.


The order of the buffer pointer and buffer size parameters in the built-in
snprintf() implementation is reversed from the order used in standard-library
implementations.


The built-in printf() implementation does not handle posix positional referencing
modifiers that allow the order of arguments to printf() to be different from the
order of the %-substitutions.  In the built-in printf(), the order of the arguments
must exactly match the order of the %-substitutions.



In spite of the disadvantages, the developers believe that having a built-in
printf() implementation inside of SQLite is a net positive.

2. Formatting Details

The format string for printf() is a template for the generated
string.  Substitutions are made whenever a "%" character appears in
the format string.  The "%" is followed by one or more additional
characters that describe the substitution.  Each substitution has
the following format:


%&#91;flags&#93;&#91;width&#93;&#91;.precision&#93;&#91;length&#93;type


All substitutions begin with a single "%" and end with a single type character.
The other elements of the substitution are optional.

To include a single "%" character in the output, put two consecutive
"%" characters in the template.

2.1. Substitution Types

The following chart shows the substitution types supported by SQLite:




Substitution TypeMeaning

%
Two "%" characters in a row are translated into a single "%" in the output,
    without substituting any values.

d, i
The argument is a signed integer which is displayed in decimal.

u
The argument is an unsigned integer which is displayed in decimal.

f
The argument is a double which is displayed in decimal.

e, E
The argument is a double which is displayed in exponential notation.
    The exponent character is 'e' or 'E' depending on the type.

g, G
The argument is a double which is displayed in either normal decimal
    notation or if the exponent is not close to zero, in exponential
    notation.

x, X
The argument is an integer which is displayed in hexadecimal.
    Lower-case hexadecimal is used for %x and upper-case is used
    for %X

o
The argument is an integer which is displayed in octal.

s, z



    The argument is either a zero-terminated string that is displayed,
    or a null pointer which is treated as an empty string.  For
    the %z type in C-language interfaces, sqlite3_free() is invoked
    on the string after it has been copied into the output. The %s and %z
    substitutions are identical for the SQL printf() function, with
    a NULL argument treated as an empty string.
    The %s substitution is universal among printf functions, but
    the %z substitution and safe treatment of null pointers
    are SQLite enhancements, not found in other
    printf() implementations.

c
For the C-language interfaces, the argument is an integer which
    is interpreted as a character.  For the format() SQL function the
    argument is a string from which the first character is extracted and
    displayed.

p
The argument is a pointer which is displayed as a hexadecimal address.
    Since the SQL language has no concept of a pointer, the %p substitution
    for the format() SQL function works like %x.

n
The argument is a pointer to an integer.  Nothing is displayed for
    this substitution type.  Instead, the integer to which the argument
    points is overwritten with the number of characters in the generated
    string that result from all format symbols to the left of the %n.

q, Q


    The argument is a zero-terminated string.  The string is printed with
    all single quote (') characters doubled so that the string can safely
    appear inside an SQL string literal.  The %Q substitution type also
    puts single-quotes on both ends of the substituted string.
    If the argument
    to %Q is a null pointer then the output is an unquoted "NULL".  In other
    words, a null pointer generates an SQL NULL, and a non-null pointer generates
    a valid SQL string literal.  If the argument to %q is a null pointer
    then no output is generated.  Thus a null-pointer to %q is the same as
    an empty string.
    For these  substitutions, the precision is the number of bytes or
    characters taken from the argument, not the number of bytes or characters that
    are written into the output.
    
    The %q and %Q substitutions are SQLite enhancements, not found in
    most other printf() implementations.

w


    This substitution works like %q except that it doubles all double-quote
    characters (") instead of single-quotes, making the result suitable for
    using with a double-quoted identifier name in an SQL statement.
    
    The %w substitution is an SQLite enhancements, not found in
    most other printf() implementations.



2.2. The Optional Length Field

The length of the argument value can be specified by one or more letters
that occur just prior to the substitution type letter.  In SQLite, the
length only matter for integer types.  The length is ignored for the
format() SQL function which always uses 64-bit values.  The following
table shows the length specifiers allowed by SQLite:




Length Specifier
Meaning

(default)
An "int" or "unsigned int".  32-bits on all modern systems.

l
A "long int" or "long unsigned int".  Also 32-bits on all modern systems.

ll
A "long long int" or "long long unsigned" or an "sqlite3_int64" or
    "sqlite3_uint64" value.  These are 64-bit integers on all modern systems.



Only the "ll" length modifier ever makes a difference for SQLite.  And
it only makes a difference when using the C-language interfaces.

2.3. The Optional Width Field

The width field specifies the minimum width of the substituted value in
the output.  If the string or number that is written into the output is shorter
than the width, then the value is padded.  Padding is on the left (the
value is right-justified) by default.  If the "-" flag is used, then the
padding is on the right and the value is left-justified.

The width is measured in bytes by default.  However, if the "!" flag is
present then the width is in characters.  This only makes a difference for
multi-byte utf-8 characters, and those only occur on string substitutions.

If the width is a single "*" character instead of a number, then the
actual width value is read as an integer from the argument list.  If the
value read is negative, then the absolute value is used for the width and
the value is left-justified as if the "-" flag were present.

If the value being substituted is larger than the width, then full value
is added to the output.  In other words, the width is the minimum width of
the value as it is rendered in the output.

2.4. The Optional Precision Field

The precision field, if it is present, must follow the width separated
by a single "." character.  If there is no width, then the "." that introduces
the precision immediately follows either the flags (if there are any) or
the initial "%".

For string substitutions (%s, %z, %q, %Q, or %w) the precision is the number
of byte or character used from the argument.  The number is bytes by default but
is characters if the "!" flag is present.  If there is no precision, then the
entire string is substituted.  Examples:  "%.3s" substitutes the first 3 bytes
of the argument string.  "%!.3s" substitutes the first three characters of the
argument string.

For integer substitutions (%d, %i, %x, %X, %o, and %p) the precision specifies
minimum number of digits to display.  Leading zeros are added if necessary, to
expand the output to the minimum number of digits.

For floating-point substitutions (%e, %E, %f, %g, %G) the precision specifies 
the number of digits to display to the right of the decimal point.

For the character substitution (%c) a precision N greater than 1 causes the
character to be repeated N times.  This is a non-standard extension found only
in SQLite.

If the precision is a single "*" character instead of a number, then the
actual precision value is read as an integer from the argument list.

2.5. The Options Flags Field

Flags consist of zero or more characters that immediately follow the
"%" that introduces the substitution.  The various flags and their meanings
are as follows:




Flag
Meaning

-
Left-justify the value in the output.  The default is to right-justify.
If the width is zero or is otherwise less than the length of the value being
substituted, then there is no padding and the "-" flag is a no-op.

+
For signed numeric substitutions, include a "+" sign before positive numbers.
A "-" sign always appears before negative numbers regardless of flag settings.

(space)
For signed numeric substitutions, prepend a single space before positive
numbers.

0
Prepend as many "0" characters to numeric substitutions as necessary to
expand the value out to the specified width.  If the width field is omitted,
then this flag is a no-op.

#
This is the "alternate-form-1" flag.
For %g and %G substitutions, this causes trailing zeros to be removed.
This flag forces a decimal point to appear for all floating-point substitutions.
For %o, %x, and %X substitutions, the alternate-form-1 flag cause the value
to be prepended with "0", "0x", or "0X", respectively.

,
This flag causes comma-separators to be added to the output of %d and %i
substitutions, between every 3 digits from the left.  This can help humans
to more easily discern the magnitude of large integer values.  For example,
the value 2147483647 would be rendered as "2147483647" using "%d" but would
appear as "2,147,483,647" with "%,d".  This flag is a non-standard extension.

!
This is the "alternate-form-2 flag.
For string substitutions, this flag causes the width and precision to be understand
in terms of characters rather than bytes.
For floating point substitutions, the alternate-form-2 flag increases the 
maximum number of significant digits displayed from 16 to 26,
forces the display of the decimal point and causes at least one digit
to appear after the decimal point.
The alternate-form-2 flag is a non-standard extension that appears in no
other printf() implementations, as far as we know.



3. Implementation And History


The core string formatting routine is the sqlite3VXPrintf() function found in the
printf.c source file.  All the
various interfaces invoke (sometimes indirectly) this one core function.
The sqlite3VXPrintf() function began as code written by the first author
of SQLite (Hipp) when he was a graduate student at Duke University in the
late 1980s.  Hipp kept this printf() implementation in his personal toolbox until
he started working on SQLite in 2000.  The code was incorporated into the
SQLite source tree on 2000-10-08
for SQLite version 1.0.9.


The Fossil Version Control System uses its own
printf() implementation that is derived from an early version of the SQLite
printf() implementation, but those two implementations have since diverged.


The sqlite3_snprintf() function has its buffer pointer and buffer size
arguments reversed from what is found in the standard C library snprintf()
routine. This is because there was no snprintf() routine in the
standard C library
when Hipp was first implementing his version, and he chose a different order
than the designers of the standard C library.


c2: Version Numbers in SQLite
c5: versionnumbers.html
c4:
1. SQLite Version Numbers


Beginning with version 3.9.0 (2015-10-14) SQLite uses 
semantic versioning.
Prior to that time, SQLite employed a version identifier that
contained between two and four numbers.

1.1. The New Version Numbering System (After 2015-10-14)


All SQLite releases starting with 3.9.0 use a three-number
"semantic version" of the form X.Y.Z.
The first number X is only increased when there is a change that
breaks backward compatibility.  The
current value for X is 3, and the SQLite developers plan to support
the current SQLite database file format, SQL syntax, and C interface
through at least the year 2050.  Hence, one
can expect that all future versions of SQLite for the next several
decades will begin with "3.".


The second number Y is incremented for any change that breaks forward
compatibility by adding new features.
Most future SQLite releases are expected
to increment the second number Y.  The Z is reset to zero whenever Y
is increased.


The third number Z is incremented for releases consisting of only
small changes that implement performance enhancements and/or bug fixes.


The rate of enhancement for SQLite over the previous five years
(2010-2015) is approximately 6 increments of Y per year.  The
numbering format used by for SQLITE_VERSION_NUMBER and
sqlite3_libversion_number() allows versions up to 3.999.999, which is
more than enough for the planned end-of-support date for SQLite
in 2050.  However, the current tarball naming conventions only
reserve two digits for the Y and so the naming format for downloads
will need to be revised in about 2030.

1.2. The Historical Numbering System (Before 2015-10-14)

This historical version numbering system used a two-, three-,
or four-number version:  W.X, W.X.Y, or W.X.Y.Z.
W was the file format: 1 or 2 or 3.
X was the major version.
Y was the minor version.
Z was used only for patch releases to fix bugs.


There have been three historical file formats for SQLite.
SQLite 1.0 through 1.0.32 used the
gdbm library as its storage
engine.
SQLite 2.0.0 through 2.8.17 used a custom b-tree storage engine that
supported only text keys and data.
All modern versions of SQLite (3.0.0 to present) use a b-tree storage
engine that has full support for binary data and Unicode.


This major version number X was historically incremented only for
large and important changes to the code.  What constituted "large
and important" was subjective.  The 3.6.23 to 3.7.0 change
was a result of adding support for WAL mode.
The 3.7.17 to 3.8.0 change was a result of rewrite known as the
next generation query planner.


The minor version number Y was historically incremented for new
features and/or new interfaces that did not significantly change
the structure of the code.  The addition of common table expressions,
partial indexes, and indexes on expressions are all examples of
"minor" changes.  Again, the distinction between "major" and "minor"
is subjective.


The patch level Z was historically only used for bug-fix releases
that changed only a small number of code lines.

1.3. Version History


Chronology
Change log


c2: The generate_series Table-Valued Function
c5: series.html
c4:
1. Overview

The generate_series(START,END,STEP) table-valued function is a
loadable extension included in the SQLite source tree, and compiled into
the command-line shell.  The generate_series() table has a single
result column named "value" holding integer values
and a number of rows determined by the
parameters START, END, and STEP.  The first row of the table has
a value of START.  Subsequent rows increase by STEP up to END.

Omitted parameters take on default values.  STEP defaults to 1.
END defaults to 9223372036854775807.  The START parameter is required
as of version 3.37.0 (2021-11-27) and later and an error will
be raised if START is omitted or has a self-referential or otherwise
uncomputable value.  Older versions used a default of 0 for START.
The legacy behavior can be obtained from recent code by compiling
with -DZERO_ARGUMENT_GENERATE_SERIES.

1.1. Equivalent Recursive Common Table Expression

The generate_series table can be simulated using a
recursive common table expression.  If the three parameters
are $start, $end, and $step, then the equivalent common table
expression is:

WITH RECURSIVE generate_series(value) AS (
  SELECT $start
  UNION ALL
  SELECT value+$step FROM generate_series
   WHERE value+$step&lt;=$end
) ...


The common table expression works without having to load an
extension.  On the other hand, the extension is easier to program
and faster.

2. Usage Examples

Generate all multiples of 5 less than or equal to 100:

SELECT value FROM generate_series(5,100,5);


Generate the 20 random integer values:

SELECT random() FROM generate_series(1,20);


Find the name of every customer whose account number
   is an even multiple of 100 between 10000 and 20000.

SELECT customer.name
  FROM customer, generate_series(10000,20000,100)
 WHERE customer.id=value;
/* or */
SELECT name FROM customer
 WHERE id IN (SELECT value
                FROM generate_series(10000,20000,200));


c2: SQLite Is Serverless
c5: serverless.html
c4:
1. SQLite Is Serverless


Most SQL database engines are implemented as a separate server process.
Programs that want to access the database communicate with the server
using some kind of interprocess communication (typically TCP/IP) to send 
requests to the server and to receive back results. 
SQLite does not work this way. 
With SQLite, the process that wants to access the database reads and 
writes directly from the database files on disk. 
There is no intermediary server process.



There are advantages and disadvantages to being serverless.
The main advantage is that there is no separate server process
to install, setup, configure, initialize, manage, and troubleshoot. 
This is one reason why SQLite is a 
"zero-configuration" database engine. 
Programs that use SQLite require no administrative support for 
setting up the database engine before they are run.
Any program that is able to access the disk is able to use an SQLite database.



On the other hand, a database engine that uses a server can 
provide better protection from bugs in the client 
application - stray pointers in a client cannot corrupt memory 
on the server. 
And because a server is a single persistent process,
it is able to control database access with more precision, 
allowing for finer-grained locking and better concurrency.



Most SQL database engines are client/server based. 
Of those that are serverless, SQLite is the only one
known to this author that allows multiple applications
to access the same database at the same time. 


2. Classic Serverless Vs. Neo-Serverless


(This section was added on 2018-04-02)


Recently, folks have begun to use the
word "serverless" to mean something subtly different from its intended
meaning in this document.  Here are two possible definitions of "serverless":



Classic Serverless:
The database engine runs within the same process, thread, and address space
as the application.  There is no message passing or network activity.


Neo-Serverless:
The database engine runs in a separate namespace from the application,
probably on a separate machine, but the database is provided as a
turn-key service by the hosting provider, requires no management or
administration by the application owners, and is so easy to use
that the developers can think of the database as being serverless
even if it really does use a server under the covers.



SQLite is an example of a classic serverless database engine.
With SQLite, there are no other processes, threads, machines, or
other mechanisms (apart from host computer OS and filesystem)
to help provide database services or implementation.  There really
is no server.


Microsoft Azure Cosmo DB
and
Amazon S3
are examples of a neo-serverless databases.
These database are implemented by server processes running separately
in the cloud.
But the servers are maintained and administered by the ISP, not by
the application developer.
Application developers just use the service.  Developers do not have to
provision, configure, or manage database server instances, as all of that
work is handled automatically by the service provider.  Database servers
do in fact exist, they are just hidden from the developers.


It is important to understand these two different definitions for
"serverless".
When a database claims to be "serverless",
be sure to discern whether they mean "classic serverless"
or "neo-serverless".


c2: Features Of SQLite
c5: features.html
c4:
Features Of SQLite


Transactions
    are atomic, consistent, isolated, and durable (ACID)
    even after system crashes and power failures.
Zero-configuration
     - no setup or administration needed.
Full-featured SQL implementation
    with advanced capabilities like partial indexes,
    indexes on expressions, JSON,
    common table expressions, and window functions.
    (Omitted features)
A complete database is stored in a 
    single cross-platform disk file.
    Great for use as an application file format.
Supports terabyte-sized databases and gigabyte-sized strings
    and blobs.  (See limits.html.)
Small code footprint: 
    less than 750KiB fully configured or much less
    with optional features omitted.
Simple, easy to use API.
Fast:  In some cases, SQLite is 
    faster than direct filesystem I/O
Written in ANSI-C.  TCL bindings included.
    Bindings for dozens of other languages available separately.
Well-commented source code with
    100% branch test coverage.
Available as a 
    single ANSI-C source-code file 
    that is easy to compile and hence is easy
    to add into a larger project.
Self-contained:
    no external dependencies.
Cross-platform: Android, *BSD, iOS, Linux, Mac, Solaris, VxWorks, 
    and Windows (Win32, WinCE, WinRT)
    are supported out of the box.  Easy to port to other systems.
Sources are in the public domain.
    Use for any purpose.
Comes with a standalone command-line interface
    (CLI) client that can be used to administer SQLite databases.



Suggested Uses For SQLite:


Database For The Internet Of Things.
SQLite is popular choice for the database engine in cellphones,
PDAs, MP3 players, set-top boxes, and other electronic gadgets.
SQLite has a small code footprint, makes efficient use of memory,
disk space, and disk bandwidth, is highly reliable, and requires
no maintenance from a Database Administrator.

Application File Format.
Rather than using fopen() to write XML, JSON, CSV,
or some proprietary format into
disk files used by your application, use an SQLite database.
You'll avoid having to write and troubleshoot a parser, your data
will be more easily accessible and cross-platform, and your updates
will be transactional.  (more...)

Website Database.
Because it requires no configuration and stores information in ordinary
disk files, SQLite is a popular choice as the database to back small
to medium-sized websites.

Stand-in For An Enterprise RDBMS.
SQLite is often used as a surrogate for an enterprise RDBMS for
demonstration purposes or for testing.  SQLite is fast and requires
no setup, which takes a lot of the hassle out of testing and which
makes demos perky and easy to launch.

More suggestions...


c2: The SQLite Zipfile Module
c5: zipfile.html
c4:
1. Overview

 The zipfile module provides read/write access to simple 
ZIP archives.
The current implementation has the following restrictions:


   Does not support encryption.
   Does not support ZIP archives that span multiple files.
   Does not support zip64 extensions.
   The only compression algorithm supported is
       "deflate".


 Some or all of these restrictions may be removed in the future.

2. Obtaining and Compiling Zipfile

The code for the zipfile module is found in the 
ext/misc/zipfile.c
file of the
main SQLite source tree.
It may be compiled into an SQLite 
loadable extension using a command like:

gcc -g -fPIC -shared zipfile.c -o zipfile.so


Alternatively, the zipfile.c file may be compiled into the application. 
In this case, the following function should be invoked to register the
extension with each new database connection:

int sqlite3_zipfile_init(sqlite3 *db, void*, void*);


 The first argument passed should be the database handle to register the
extension with. The second and third arguments should both be passed 0.

 Zipfile is included in most builds of the command-line shell.

3. Using Zipfile

The zipfile module provides three similar interfaces for accessing, updating
and creating zip file archives:


   A table-valued function, which provides read-only access to existing
       archives, either from the file-system or in-memory.
   A virtual table, which provides read and write access to archives
       stored in the file-system.
   An SQL aggregate function, which can be used to create new archives
       in memory.


The zipfile module provides two similar interfaces for accessing zip
archives. A table-valued function, which provides read-only access to
existing archives, and a virtual table interface, which provides both
read and write access.

3.1. Table-Valued Function (read-only access)

For reading existing zip archives, the Zipfile module provides a
table-valued function that accepts a single argument. If the argument
is a text value, then it is a path to a zip archive to read from the
file-system. Or, if the argument is an SQL blob, then it is the zip
archive data itself.

For example, to inspect the contents of zip archive "test.zip" from 
the current directory:

SELECT * FROM zipfile('test.zip');


Or, from the SQLite shell tool (the readfile() 
function reads the contents of a file from the file-system and returns it as a
blob):

SELECT * FROM zipfile( readfile('test.zip') );


The table-valued function returns one row for each record (file, 
directory or symbolic link) in the zip archive. Each row has the 
following columns:


Column&nbsp;NameContents
name  File name/path for the zip file record.
mode  UNIX mode, as returned by stat(2) for the zip file record (an
                  integer). This identifies the type of record (file, directory
                  or symbolic link), and the associated user/group/all 
                  permissions.
mtime  UTC timestamp, in seconds since the UNIX epoch (an integer).
sz     Size of associated data in bytes after it has been 
                   uncompressed (an integer).
rawdata  Raw (possibly compressed) data associated with zip file
                     entry (a blob).
data  If the compression method for the record is either 0 or 8
                  (see below), then the uncompressed data associated with the
                  zip file entry. Or, if the compression method is not 0 or 8, 
                  this column contains a NULL value.
method  The compression method used to compress the data (an
                    integer). The value 0 indicates that the data is stored
                    in the zip archive without compression. 8 means the
                    raw deflate algorithm. 


3.2. Virtual Table Interface (read/write access)

In order to create or modify an existing zip file, a "zipfile" virtual 
table must be created in the database schema. The CREATE VIRTUAL TABLE
statement expects a path to the zip file as its only argument. For example, to
write to zip file "test.zip" in the current directory, a zipfile table may be
created using:

CREATE VIRTUAL TABLE temp.zip USING zipfile('test.zip');


Such a virtual table has the same columns as the table-valued function
described in the previous section. It may be read from using a SELECT 
statement in the same way as the table-valued function can. 

Using the virtual table interface, new entries may be added to a zip
archive by inserting new rows into the virtual table. Entries may be
removed by deleting rows or modified by updating them.


3.2.1. Adding Entries to a Zip Archive

Entries may be added to a zip archive by inserting new rows. The easiest
way to do this is to specify values for the "name" and "data" columns only and
have zipfile fill in sensible defaults for other fields.  To insert a directory
into the archive, set the "data" column to NULL.  For example, to add the
directory "dir1" and the file "m.txt" containing the text "abcdefghi" to zip
archive "test.zip":

INSERT INTO temp.zip(name, data) VALUES('dir1', NULL);           -- Add directory 
INSERT INTO temp.zip(name, data) VALUES('m.txt', 'abcdefghi');   -- Add regular file 


When a directory is inserted, if the "name" value does not end with
a '/' character, the zipfile module appends one. This is necessary for
compatibility with other programs (most notably "info-zip") that 
manipulate zip archives.

To insert a symbolic link, the user must also supply a "mode" value.
For example, to add a symbolic link from "link.txt" to "m.txt":

INSERT INTO temp.zip(name, mode, data) VALUES('link.txt', 'lrwxrw-rw-', 'abcdefghi');


The following rules and caveats apply to the values specified as part of
each INSERT statement:


Columns  Notes
 name
   A non-NULL text value must be specified for the name column. 
       It is an error if the specified name already exists in the
       archive. 

 mode
   If NULL is inserted into the mode column, then the mode of the
       new archive entry is automatically set to either 33188 (-rw-r--r--)
       or 16877 (drwxr-xr-x), depending on whether or not the values 
       specified for columns "sz", "data" and "rawdata" indicate that
       the new entry is a directory.
       
       If the specified value is an integer (or text that looks like
       an integer), it is inserted verbatim. If the value is not a valid UNIX
       mode, some programs may behave unexpectedly when extracting files
       from the archive.

       Finally, if the value specified for this column is not an integer
       or a NULL, then it is assumed to be a UNIX permissions string similar
       to those output by the "ls -l" command (e.g. "-rw-r--r--", "drwxr-xr-x"
       etc.). In this case, if the string cannot be parsed it is an error.

 mtime
   If NULL is inserted into the mtime column, then the timestamp
       of the new entry is set to the current time. Otherwise, the specified
       value is interpreted as an integer and used as is.

 sz
   This column must be set to NULL. If a non-NULL value is inserted into
       this column, or if a new non-NULL value is provided using an UPDATE
       statement, it is an error.

 rawdata
   This column must be set to NULL. If a non-NULL value is inserted into
       this column, or if a new non-NULL value is provided using an UPDATE
       statement, it is an error.

 data

       To insert a directory into the archive, this field must be set to 
       NULL. In this case if a value was explicitly specified for the "mode"
       column, then it must be consistent with a directory (i.e. it must be
       true that (mode & 0040000)=0040000). 

       Otherwise, the value inserted into this field is the file contents
       for a regular file, or the target of a symbolic link.
 method

       This field must be set one of integer values 0 and 8, or else to
       NULL. 

       For a directory entry, any value inserted into this field is ignored.
       Otherwise, if it is set to 0, then the file data or symbolic link
       target is stored as is in the zip archive and the compression method
       set to 0. If it is set to 8, then the file data or link target is
       compressed using deflate compression before it is stored and the
       compression method set to 8. Finally, if a NULL value is written
       to this field, the zipfile module automatically decides whether
       or not to compress the data before storing it.


 Specifying an explicit value for the rowid field as part of an INSERT
statement is not supported. Any value supplied is ignored.

3.2.2.  Deleting Zip Archive Entries 

Records may be removed from an existing zip archive by deleting the
corresponding rows. For example, to remove file "m.txt" from zip archive
"test.zip" using the virtual table created above:

DELETE FROM temp.zip WHERE name = 'm.txt';


Note that deleting records from a zip archive does not reclaim the 
space used within the archive - it merely removes an entry from the
archives "Central Directory Structure", making the entry inaccessible.
One way to work around this inefficiency is to create a new zip 
archive based on the contents of the edited archive. For example, after
editing the archive accessed via virtual table temp.zzz:

-- Create a new, empty, archive: 
CREATE VIRTUAL TABLE temp.newzip USING zipfile('new.zip');

-- Copy the contents of the existing archive into the new archive
INSERT INTO temp.newzip(name, mode, mtime, data, method)
    SELECT name, mode, mtime, data, method FROM temp.zzz;


3.2.3.  Updating Existing Zip Archive Entries 

Existing zip archive entries may be modified using UPDATE statements.

The three leftmost columns of a zipfile virtual table, "name", "mode" 
and "mtime", may each be set to any value that may be inserted into the same
column (see above). If either "mode" or "mtime" is set to NULL, the final 
value is determined as described for an INSERT of a NULL value - the current
time for "mtime" and either 33188 or 16877 for "mode", depending on whether 
or not the values specified for the next four columns of the zipfile table
indicate that the entry is a directory or a file.

It is an error to attempt to set the sz or rawdata field to any value
other than NULL. 

The data and method columns may also be set as described for an INSERT
above.  

3.3.  The zipfile() Aggregate Function 

 New zip archives may be constructed entirely within memory using the
zipfile() aggregate function. Each row visited by the aggregate function
adds an entry to the zip archive. The value returned is a blob containing
the entire archive image.

 The zipfile() aggregate function may be called with 2, 4 or 5 
arguments. If it is called with 5 arguments, then the entry added to
the archive is equivalent to inserting the same values into the "name", 
"mode", "mtime", "data" and "method" columns of a zipfile virtual table.

 If zipfile() is invoked with 2 arguments, then the entry added to
the archive is equivalent to that added by inserting the same two values into
the "name" and "data" columns of a zipfile virtual table, with all
other values set to NULL. If invoked with 4 arguments, it is equivalent
to inserting the 4 values into the "name", "mode", "mtime" and "data"
columns. In other words, the following pairs of queries are equivalent:

SELECT zipfile(name, data) ...
SELECT zipfile(name, NULL, NULL, data, NULL) ...

SELECT zipfile(name, mode, mtime, data) ...
SELECT zipfile(name, mode, mtime, data, NULL) ...


 For example, to create an archive containing two text files, "a.txt" and
"b.txt", containing the text "abc" and "123" respectively:

WITH contents(name, data) AS (
  VALUES('a.txt', 'abc') UNION ALL
  VALUES('b.txt', '123')
)
SELECT zipfile(name, data) FROM contents;


c2: The Bytecode() And Tables_Used() Table-Valued Functions
c5: bytecodevtab.html
c4:
1. Overview


Bytecode and tables_used are virtual tables built into SQLite that
access information about prepared statements.
Both bytecode and tables_used operate as table-valued functions.
They take a single required argument which is either the text of
an SQL statement, or a pointer to an existing prepared statement.
The bytecode function returns one row of result for each bytecode
operation in the prepared statement.  The tables_used function returns
one row for each persistent btree (either a table or an index) accessed
by the prepared statement.

2. Usage


The bytecode and tables_used tables are only available if SQLite has
been compiled with the -DSQLITE_ENABLE_BYTECODE_VTAB compile-time option.
The CLI has been compiled that way, and so you can use the standard
CLI as a test platform to experiement.


Both virtual tables are read-only eponymous-only virtual tables.  You use them
by mentioning them directly in the FROM clause of a SELECT statement.
They both require a single argument which is the SQL statement to be
analyzed.  For example:

SELECT * FROM bytecode('SELECT * FROM bytecode(?1)');



The argument can be either the text of an SQL statement, in which case
the bytecode (or tables_used) for that statement is returned, or the
argument can be a parameter such as ?1 or $stmt that is later bound
to a prepared statement object using the
sqlite3_bind_pointer() interface.  Use a pointer type of
"stmt-pointer" for the sqlite3_bind_pointer() interface.

2.1. Schema


The schema of the bytecode table is:

CREATE TABLE bytecode(
  addr INT,
  opcode TEXT,
  p1 INT,
  p2 INT,
  p3 INT,
  p4 TEXT,
  p5 INT,
  comment TEXT,
  subprog TEXT,
  stmt HIDDEN
);



The first eight columns are the address, opcode, and operands for a
single bytecode in the virtual machine that implements the statement.
These columns are the same columns output when using EXPLAIN.  The
bytecode virtual tables shows all operations in the prepared statement,
both the main body of the prepared statement and in subprograms used
to implement triggers or foreign key actions.  The "subprog" field
is NULL for the main body of the prepared statement, or is the trigger
name or the string "(FK)" for triggers and foreign key actions.


The schema for the tables_used table is:

CREATE TABLE tables_used(
  type TEXT,
  schema TEXT,
  name TEXT,
  wr INT,
  subprog TEXT,
  stmt HIDDEN
);



The tables_used table is intended to show which btrees of the database file
are read or written by a prepared statement, both by the main statement
itself but also by related triggers and foreign key actions.  The columns
are as follows:


type &rarr; Either "table" or "index", depending on what role
the btree is serving.

schema &rarr; Which database file the btree is located in.
This will be "main" for the main database (the usual case), or "temp" for
TEMP tables and indexes, or the name assigned to attached databases by
the ATTACH statement.

name &rarr; The name of the table or index

wr &rarr; 0 if the object is read, 1 if the object is written

subprog &rarr;  The sub-program in which the object is
accessed.  NULL means the main body of the prepared statement.  Otherwise
this field is the name of a trigger or "(FK)" for a foreign key action.


c2: NULL Handling in SQLite
c5: nulls.html
c4:
NULL Handling in SQLite Versus Other Database Engines


The goal is
to make SQLite handle NULLs in a standards-compliant way.
But the descriptions in the SQL standards on how to handle
NULLs seem ambiguous. 
It is not clear from the standards documents exactly how NULLs should
be handled in all circumstances.



So instead of going by the standards documents, various popular
SQL engines were tested to see how they handle NULLs.  The idea
was to make SQLite work like all the other engines.
An SQL test script was developed and run by volunteers on various
SQL RDBMSes and the results of those tests were used to deduce
how each engine processed NULL values.
The original tests were run in May of 2002.
A copy of the test script is found at the end of this document.



SQLite was originally coded in such a way that the answer to
all questions in the chart below would be "Yes".  But the
experiments run on other SQL engines showed that none of them
worked this way.  So SQLite was modified to work the same as
Oracle, PostgreSQL, and DB2.  This involved making NULLs
indistinct for the purposes of the SELECT DISTINCT statement and
for the UNION operator in a SELECT.  NULLs are still distinct
in a UNIQUE column.  This seems somewhat arbitrary, but the desire
to be compatible with other engines outweighed that objection.



It is possible to make SQLite treat NULLs as distinct for the
purposes of the SELECT DISTINCT and UNION.  To do so, one should
change the value of the NULL_ALWAYS_DISTINCT #define in the
sqliteInt.h source file and recompile.




Update 2003-07-13:
Since this document was originally written some of the database engines
tested have been updated and users have been kind enough to send in
corrections to the chart below.  The original data showed a wide variety
of behaviors, but over time the range of behaviors has converged toward
the PostgreSQL/Oracle model.  The only significant difference 
is that Informix and MS-SQL both treat NULLs as
indistinct in a UNIQUE column.



The fact that NULLs are distinct for UNIQUE columns but are indistinct for
SELECT DISTINCT and UNION continues to be puzzling.  It seems that NULLs
should be either distinct everywhere or nowhere.  And the SQL standards
documents suggest that NULLs should be distinct everywhere.  Yet as of
this writing, no SQL engine tested treats NULLs as distinct in a SELECT
DISTINCT statement or in a UNION.





The following table shows the results of the NULL handling experiments.



&nbsp;&nbsp;
SQLite
PostgreSQL
Oracle
Informix
DB2
MS-SQL
OCELOT


Adding anything to null gives null
Yes
Yes
Yes
Yes
Yes
Yes
Yes

Multiplying null by zero gives null
Yes
Yes
Yes
Yes
Yes
Yes
Yes

nulls are distinct in a UNIQUE column
Yes
Yes
Yes
No
(Note 4)
No
Yes

nulls are distinct in SELECT DISTINCT
No
No
No
No
No
No
No

nulls are distinct in a UNION
No
No
No
No
No
No
No

"CASE WHEN null THEN 1 ELSE 0 END" is 0?
Yes
Yes
Yes
Yes
Yes
Yes
Yes

"null OR true" is true
Yes
Yes
Yes
Yes
Yes
Yes
Yes

"not (null AND false)" is true
Yes
Yes
Yes
Yes
Yes
Yes
Yes




&nbsp;&nbsp;
MySQL3.23.41
MySQL4.0.16
Firebird
SQLAnywhere
BorlandInterbase


Adding anything to null gives null
Yes
Yes
Yes
Yes
Yes

Multiplying null by zero gives null
Yes
Yes
Yes
Yes
Yes

nulls are distinct in a UNIQUE column
Yes
Yes
Yes
(Note 4)
(Note 4)

nulls are distinct in SELECT DISTINCT
No
No
No (Note 1)
No
No

nulls are distinct in a UNION
(Note 3)
No
No (Note 1)
No
No

"CASE WHEN null THEN 1 ELSE 0 END" is 0?
Yes
Yes
Yes
Yes
(Note 5)

"null OR true" is true
Yes
Yes
Yes
Yes
Yes

"not (null AND false)" is true
No
Yes
Yes
Yes
Yes





Notes:&nbsp;&nbsp;
1.&nbsp;
Older versions of firebird omit all NULLs from SELECT DISTINCT
and from UNION.

2.&nbsp;
Test data unavailable.

3.&nbsp;
MySQL version 3.23.41 does not support UNION.

4.&nbsp;
DB2, SQL Anywhere, and Borland Interbase 
do not allow NULLs in a UNIQUE column.

5.&nbsp;
Borland Interbase does not support CASE expressions.




&nbsp;

The following script was used to gather information for the table
above.



-- I have about decided that SQL's treatment of NULLs is capricious and cannot be
-- deduced by logic.  It must be discovered by experiment.  To that end, I have 
-- prepared the following script to test how various SQL databases deal with NULL.
-- My aim is to use the information gathered from this script to make SQLite as
-- much like other databases as possible.
--
-- If you could please run this script in your database engine and mail the results
-- to me at drh@hwaci.com, that will be a big help.  Please be sure to identify the
-- database engine you use for this test.  Thanks.
--
-- If you have to change anything to get this script to run with your database
-- engine, please send your revised script together with your results.
--

-- Create a test table with data
create table t1(a int, b int, c int);
insert into t1 values(1,0,0);
insert into t1 values(2,0,1);
insert into t1 values(3,1,0);
insert into t1 values(4,1,1);
insert into t1 values(5,null,0);
insert into t1 values(6,null,1);
insert into t1 values(7,null,null);

-- Check to see what CASE does with NULLs in its test expressions
select a, case when b0 then 1 else 0 end from t1;
select a+10, case when not b0 then 1 else 0 end from t1;
select a+20, case when b0 and c0 then 1 else 0 end from t1;
select a+30, case when not (b0 and c0) then 1 else 0 end from t1;
select a+40, case when b0 or c0 then 1 else 0 end from t1;
select a+50, case when not (b0 or c0) then 1 else 0 end from t1;
select a+60, case b when c then 1 else 0 end from t1;
select a+70, case c when b then 1 else 0 end from t1;

-- What happens when you multiply a NULL by zero?
select a+80, b*0 from t1;
select a+90, b*c from t1;

-- What happens to NULL for other operators?
select a+100, b+c from t1;

-- Test the treatment of aggregate operators
select count(*), count(b), sum(b), avg(b), min(b), max(b) from t1;

-- Check the behavior of NULLs in WHERE clauses
select a+110 from t1 where b10;
select a+130 from t1 where b


c2: Compile-time Options
c5: compile.html
c4:
1. Overview


For most purposes, SQLite can be built just fine using the default
compilation options. However, if required, the compile-time options
documented below can be used to 
omit SQLite features (resulting in
a smaller compiled library size) or to change the
default values of some parameters.



Every effort has been made to ensure that the various combinations
of compilation options work harmoniously and produce a working library.
Nevertheless, it is strongly recommended that the SQLite test-suite
be executed to check for errors before using an SQLite library built
with non-standard compilation options.





2. Recommended Compile-time Options

The following compile-time options are recommended for applications that
are able to use them, in order to minimized the number of CPU cycles and
the bytes of memory used by SQLite.
Not all of these compile-time options are usable by every application.
For example, the SQLITE_THREADSAFE=0 option is only usable by applications
that never access SQLite from more than one thread at a time.  And the
SQLITE_OMIT_PROGRESS_CALLBACK option is only usable by applications that
do not use the sqlite3_progress_handler() interface.  And so forth.

It is impossible to test every possible combination of compile-time
options for SQLite.  But the following set of compile-time options is
one configuration that is always fully tested.


SQLITE_DQS=0.
This setting disables the double-quoted string literal misfeature.

SQLITE_THREADSAFE=0.
Setting -DSQLITE_THREADSAFE=0 causes all of the mutex and thread-safety logic
in SQLite to be omitted.  This is the single compile-time option causes SQLite
to run about 2% faster and also reduces the size of the library by about 2%.
But the downside is that using the compile-time option means that SQLite can never
be used by more than a single thread at a time, even if each thread has its own
database connection.

SQLITE_DEFAULT_MEMSTATUS=0.
This setting causes the sqlite3_status() interfaces that track memory usage
to be disabled.  This helps the sqlite3_malloc() routines run much faster,
and since SQLite uses sqlite3_malloc() internally, this helps to make the
entire library faster.

SQLITE_DEFAULT_WAL_SYNCHRONOUS=1.
For maximum database safety following a power loss, the setting of
PRAGMA synchronous=FULL is recommended.  However, in WAL mode, complete
database integrity is guaranteed with PRAGMA synchronous=NORMAL.  With
PRAGMA synchronous=NORMAL in WAL mode, recent changes to the database might
be rolled back by a power loss, but the database will not be corrupted.
Furthermore, transaction commit is much faster in WAL mode using
synchronous=NORMAL than with the default synchronous=FULL.  For these
reasons, it is recommended that the synchronous setting be changed from
FULL to NORMAL when switching to WAL mode.  This compile-time option will
accomplish that.

SQLITE_LIKE_DOESNT_MATCH_BLOBS.
Historically, SQLite has allowed BLOB operands to the LIKE and GLOB
operators.  But having a BLOB as an operand of LIKE or GLOB complicates
and slows the LIKE optimization.  When this option is set, it means that
the LIKE and GLOB operators always return FALSE if either operand is a BLOB.
That simplifies the implementation of the LIKE optimization and allows
queries that use the LIKE optimization to run faster.

SQLITE_MAX_EXPR_DEPTH=0.
Setting the maximum expression parse-tree depth to zero disables all checking
of the expression parse-tree depth, which simplifies the code resulting in
faster execution, and helps the parse tree to use less memory.

SQLITE_OMIT_DECLTYPE.
By omitting the (seldom-needed) ability to return the declared type of
columns from the result set of query, prepared statements can be made
to consume less memory.

SQLITE_OMIT_DEPRECATED.
Omitting deprecated interfaces and features will not help SQLite to
run any faster.  It will reduce the library footprint, however.  And
it is the right thing to do.

SQLITE_OMIT_PROGRESS_CALLBACK.
The progress handler callback counter must be checked in the inner loop
of the bytecode engine.  By omitting this interface, a single conditional
is removed from the inner loop of the bytecode engine, helping SQL statements
to run slightly faster.

SQLITE_OMIT_SHARED_CACHE.
Omitting the possibility of using shared cache allows many conditionals
in performance-critical sections of the code to be eliminated.  This can
give a noticeable improvement in performance.

SQLITE_USE_ALLOCA.
Make use of alloca() for dynamically allocating temporary stack space for 
use within a single function, on systems that support alloca().  Without
this option, temporary space is allocated from the heap.

SQLITE_OMIT_AUTOINIT.
The SQLite library needs to be initialized using a call to
sqlite3_initialize() before certain interfaces are used.  
This initialization normally happens automatically the first time
it is needed.  However, with the SQLITE_OMIT_AUTOINIT option, the automatic
initialization is omitted.  This helps many API calls to run a little faster
(since they do not have to check to see if initialization has already occurred
and then run initialization if it has not previously been invoked) but it
also means that the application must call sqlite3_initialize() manually.
If SQLite is compiled with -DSQLITE_OMIT_AUTOINIT and a routine like
sqlite3_malloc() or sqlite3_vfs_find() or sqlite3_open() is invoked
without first calling sqlite3_initialize(), the likely result will be
a segfault.


When all of the recommended compile-time options above are used,
the SQLite library will be approximately 3% smaller and use about 5% fewer
CPU cycles.  So these options do not make a huge difference.  But in 
some design situations, every little bit helps.


3.  Platform Configuration


_HAVE_SQLITE_CONFIG_H
  If the _HAVE_SQLITE_CONFIG_H macro is defined
  then the SQLite source code will attempt to #include a file named "config.h".
  The "config.h" file usually contains other configuration options, especially
  "HAVE_INTERFACE" type options generated by autoconf scripts.

HAVE_FDATASYNC
  If the HAVE_FDATASYNC compile-time option is true, then the default VFS
  for unix systems will attempt to use fdatasync() instead of fsync() where
  appropriate.  If this flag is missing or false, then fsync() is always used.

HAVE_GMTIME_R
  If the HAVE_GMTIME_R option is true and if SQLITE_OMIT_DATETIME_FUNCS is true,
  then the CURRENT_TIME, CURRENT_DATE, and CURRENT_TIMESTAMP keywords will use
  the threadsafe "gmtime_r()" interface rather than "gmtime()".  In the usual case
  where SQLITE_OMIT_DATETIME_FUNCS is not defined or is false, then the
  built-in date and time functions are used to implement the CURRENT_TIME,
  CURRENT_DATE, and CURRENT_TIMESTAMP keywords and neither gmtime_r() nor
  gmtime() is ever called.

HAVE_ISNAN
  If the HAVE_ISNAN option is true, then SQLite invokes the system library isnan()
  function to determine if a double-precision floating point value is a NaN.
  If HAVE_ISNAN is undefined or false, then SQLite substitutes its own home-grown
  implementation of isnan().

HAVE_LOCALTIME_R
  If the HAVE_LOCALTIME_R option is true, then SQLite uses the threadsafe
  localtime_r() library routine instead of localtime()
  to help implement the localtime modifier
  to the built-in date and time functions.

HAVE_LOCALTIME_S
  If the HAVE_LOCALTIME_S option is true, then SQLite uses the threadsafe
  localtime_s() library routine instead of localtime()
  to help implement the localtime modifier
  to the built-in date and time functions.

HAVE_MALLOC_USABLE_SIZE
  If the HAVE_MALLOC_USABLE_SIZE option is true, then SQLite tries uses the
  malloc_usable_size() interface to find the size of a memory allocation obtained
  from the standard-library malloc() or realloc() routines.  This option is only
  applicable if the standard-library malloc() is used.  On Apple systems,
  "zone malloc" is used instead, and so this option is not applicable.  And, of
  course, if the application supplies its own malloc implementation using
  SQLITE_CONFIG_MALLOC then this option has no effect.
  
  If the HAVE_MALLOC_USABLE_SIZE option is omitted or is false, then SQLite 
  uses a wrapper around system malloc() and realloc() that enlarges each allocation
  by 8 bytes and writes the size of the allocation in the initial 8 bytes, and
  then SQLite also implements its own home-grown version of malloc_usable_size()
  that consults that 8-byte prefix to find the allocation size.  This approach
  works but it is suboptimal.  Applications are encouraged to use
  HAVE_MALLOC_USABLE_SIZE whenever possible.

HAVE_STRCHRNUL
  If the HAVE_STRCHRNUL option is true, then SQLite uses the strchrnul() library
  function.  If this option is missing or false, then SQLite substitutes its own
  home-grown implementation of strchrnul().

HAVE_USLEEP
  If the HAVE_USLEEP option is true, then the default unix VFS uses the
  usleep() system call to implement the xSleep method.  If this option is
  undefined or false, then xSleep on unix is implemented using sleep() which
  means that sqlite3_sleep() will have a minimum wait interval of 1000
  milliseconds regardless of its argument.  

HAVE_UTIME
  If the HAVE_UTIME option is true, then the built-in but non-standard
  "unix-dotfile" VFS will use the utime() system call, instead of utimes(),
  to set the last access time on the lock file.

SQLITE_BYTEORDER=(0|1234|4321)
  SQLite needs to know if the native byte order of the target CPU is
  big-endian or little-ending.  The SQLITE_BYTEORDER preprocessor is set
  to 4321 for big-endian machines and 1234 for little-endian machines, or
  it can be 0 to mean that the byte order must be determined at run-time.
  There are #ifdefs in the code that set SQLITE_BYTEORDER automatically
  for all common platforms and compilers.  However, it may be advantageous
  to set SQLITE_BYTEORDER appropriately when compiling SQLite for obscure
  targets.  If the target byte order cannot be determined at compile-time,
  then SQLite falls back to doing run-time checks, which always work, though
  with a small performance penalty.



4.  Options To Set Default Parameter Values


SQLITE_DEFAULT_AUTOMATIC_INDEX=&lt;0 or 1&gt;
  This macro determines the initial setting for PRAGMA automatic_index
  for newly opened database connections.
  For all versions of SQLite through 3.7.17,
  automatic indices are normally enabled for new database connections if
  this compile-time option is omitted.
  However, that might change in future releases of SQLite.
  See also: SQLITE_OMIT_AUTOMATIC_INDEX

SQLITE_DEFAULT_AUTOVACUUM=&lt;0 or 1 or 2&gt;
  This macro determines if SQLite creates databases with the 
  auto_vacuum flag set by default to OFF (0), FULL (1), or
  INCREMENTAL (2). The default value is 0 meaning that databases
  are created with auto-vacuum turned off.
  In any case the compile-time default may be overridden by the 
  PRAGMA auto_vacuum command.

SQLITE_DEFAULT_CACHE_SIZE=&lt;N&gt;
  This macro sets the default maximum size of the page-cache for each attached
  database.  A positive value means that the limit is N page.  If N is negative
  that means to limit the cache size to -N*1024 bytes.
  The suggested maximum cache size can be overridden by the 
  PRAGMA cache_size command. The default value is -2000, which translates
  into a maximum of 2048000 bytes per cache.

SQLITE_DEFAULT_FILE_FORMAT=&lt;1 or 4&gt;
  The default schema format number used by SQLite when creating
  new database files is set by this macro.  The schema formats are all
  very similar.  The difference between formats 1 and 4 is that format
  4 understands descending indices and has a tighter encoding for
  boolean values.

  All versions of SQLite since 3.3.0 (2006-01-10)
  can read and write any schema format
  between 1 and 4.  But older versions of SQLite might not be able to
  read formats greater than 1.  So that older versions of SQLite will
  be able to read and write database files created by newer versions
  of SQLite, the default schema format was set to 1 for SQLite versions
  through 3.7.9 (2011-11-01).  Beginning with 
  version 3.7.10 (2012-01-16), the default
  schema format is 4.

  The schema format number for a new database can be set at runtime using
  the PRAGMA legacy_file_format command.

SQLITE_DEFAULT_FILE_PERMISSIONS=N
  The default numeric file permissions for newly created database files
  under unix.  If not specified, the default is 0644 which means that
  the files is globally readable but only writable by the creator.

SQLITE_DEFAULT_FOREIGN_KEYS=&lt;0 or 1&gt;
  This macro determines whether enforcement of 
  foreign key constraints is enabled or disabled by default for
  new database connections.  Each database connection can always turn
  enforcement of foreign key constraints on and off and run-time using
  the foreign_keys pragma.  Enforcement of foreign key constraints
  is normally off by default, but if this compile-time parameter is
  set to 1, enforcement of foreign key constraints will be on by default.

SQLITE_DEFAULT_MMAP_SIZE=N
  This macro sets the default limit on the amount of memory that
  will be used for memory-mapped I/O
  for each open database file.  If the N
  is zero, then memory mapped I/O is disabled by default.  This
  compile-time limit and the SQLITE_MAX_MMAP_SIZE can be modified 
  at start-time using the
  sqlite3_config(SQLITE_CONFIG_MMAP_SIZE) call, or at run-time
  using the mmap_size pragma.

SQLITE_DEFAULT_JOURNAL_SIZE_LIMIT=&lt;bytes&gt;
  This option sets the size limit on rollback journal files in
  persistent journal mode and
  exclusive locking mode and on the size of the
  write-ahead log file in WAL mode. When this 
  compile-time option is omitted there is no upper bound on the
  size of the rollback journals or write-ahead logs.  
  The journal file size limit
  can be changed at run-time using the journal_size_limit pragma.

SQLITE_DEFAULT_LOCKING_MODE=&lt;1 or 0&gt;
  If set to 1, then the default locking_mode is set to EXCLUSIVE.
  If omitted or set to 0 then the default locking_mode is NORMAL.

SQLITE_DEFAULT_LOOKASIDE=SZ,N
  Sets the default size of the lookaside memory allocator memory pool
  to N entries of SZ bytes each.  This setting can be modified at
  start-time using sqlite3_config(SQLITE_CONFIG_LOOKASIDE) and/or
  as each database connection is opened using
  sqlite3_db_config(db, SQLITE_DBCONFIG_LOOKASIDE).

SQLITE_DEFAULT_MEMSTATUS=&lt;1 or 0&gt;
  This macro is used to determine whether or not the features enabled and
  disabled using the SQLITE_CONFIG_MEMSTATUS argument to sqlite3_config()
  are available by default. The default value is 1 (SQLITE_CONFIG_MEMSTATUS
  related features enabled).
  
  The sqlite3_memory_used() and sqlite3_memory_highwater() interfaces,
  the sqlite3_status64(SQLITE_STATUS_MEMORY_USED) interface,
  and the SQLITE_MAX_MEMORY compile-time option are all non-functional
  when memory usage tracking is disabled.

SQLITE_DEFAULT_PCACHE_INITSZ=N
  This macro determines the number of pages initially allocated by the 
  page cache module when SQLITE_CONFIG_PAGECACHE configuration option is
  not use and memory for the page cache is obtained from sqlite3_malloc()
  instead.  The number of pages set by this macro are allocated in a single
  allocation, which reduces the load on the memory allocator.

SQLITE_DEFAULT_PAGE_SIZE=&lt;bytes&gt;
  This macro is used to set the default page-size used when a
  database is created. The value assigned must be a power of 2. The
  default value is 4096. The compile-time default may be overridden at 
  runtime by the PRAGMA page_size command.

SQLITE_DEFAULT_SYNCHRONOUS=&lt;0-3&gt;
  This macro determines the default value of the
  PRAGMA synchronous setting.  If not overridden at compile-time,
  the default setting is 2 (FULL).

SQLITE_DEFAULT_WAL_SYNCHRONOUS=&lt;0-3&gt;
  This macro determines the default value of the
  PRAGMA synchronous setting for database files that open in
  WAL mode.  If not overridden at compile-time, this value is the
  same as SQLITE_DEFAULT_SYNCHRONOUS.
  
  If SQLITE_DEFAULT_WAL_SYNCHRONOUS differs from SQLITE_DEFAULT_SYNCHRONOUS,
  and if the application has not modified the synchronous setting for
  the database file using the PRAGMA synchronous statement, then
  the synchronous setting is changed to value defined by
  SQLITE_DEFAULT_WAL_SYNCHRONOUS when the database connection switches
  into WAL mode for the first time.
  If the SQLITE_DEFAULT_WAL_SYNCHRONOUS value is not overridden at
  compile-time, then it will always be the same as
  SQLITE_DEFAULT_SYNCHRONOUS and so no automatic synchronous setting
  changes will ever occur.

SQLITE_DEFAULT_WAL_AUTOCHECKPOINT=&lt;pages&gt;
  This macro sets the default page count for the WAL
  automatic checkpointing feature.  If unspecified,
  the default page count is 1000.

SQLITE_DEFAULT_WORKER_THREADS=N
  This macro sets the default value for
  the SQLITE_LIMIT_WORKER_THREADS parameter.  The SQLITE_LIMIT_WORKER_THREADS
  parameter sets the maximum number of auxiliary threads that a single
  prepared statement will launch to assist it with a query.  If not specified,
  the default maximum is 0.
  The value set here cannot be more than SQLITE_MAX_WORKER_THREADS.

SQLITE_DQS=N
  This macro determines the default values for
  SQLITE_DBCONFIG_DQS_DDL and SQLITE_DBCONFIG_DQS_DML, which
  in turn how SQLite handles each double-quoted string literal.
  The "DQS" name stands for 
  "Double-Quoted String".
  The N argument should be an integer 0, 1, 2, or 3.
  
  SQLITE_DQSDouble-Quoted Strings Allowed
      Remarks
  In DDLIn DML
  3yesyesdefault
  2yesno&nbsp;
  1noyes&nbsp;
  0nonorecommended
  
  The recommended setting is 0, meaning that double-quoted
  strings are disallowed in all contexts.  However, the default
  setting is 3 for maximum compatibility with legacy applications.

SQLITE_EXTRA_DURABLE
  The SQLITE_EXTRA_DURABLE compile-time option that used to cause the default
  PRAGMA synchronous setting to be EXTRA, rather than FULL.  This option
  is no longer supported.  Use
  SQLITE_DEFAULT_SYNCHRONOUS=3 instead.

SQLITE_FTS3_MAX_EXPR_DEPTH=N
  This macro sets the maximum depth of the search tree that corresponds to
  the right-hand side of the MATCH operator in an FTS3 or FTS4 full-text
  index.  The full-text search uses a recursive algorithm, so the depth of
  the tree is limited to prevent using too much stack space.  The default
  limit is 12.  This limit is sufficient for up to 4095 search terms on the
  right-hand side of the MATCH operator and it holds stack space usage to 
  less than 2000 bytes.
  
  For ordinary FTS3/FTS4 queries, the search tree depth is approximately
  the base-2 logarithm of the number of terms in the right-hand side of the
  MATCH operator.  However, for phrase queries and NEAR queries the
  search tree depth is linear in the number of right-hand side terms.
  So the default depth limit of 12 is sufficient for up to 4095 ordinary
  terms on a MATCH, it is only sufficient for 11 or 12 phrase or NEAR
  terms.  Even so, the default is more than enough for most application.

SQLITE_LIKE_DOESNT_MATCH_BLOBS
  This compile-time option causes the LIKE operator to always return 
  False if either operand is a BLOB.  The default behavior of LIKE 
  is that BLOB operands are cast to TEXT before the comparison is done.
  
  This compile-time option makes SQLite run more efficiently when processing
  queries that use the LIKE operator, at the expense of breaking backwards
  compatibility.  However, the backwards compatibility break may be only
  a technicality.  There was a long-standing bug in the LIKE processing logic
  (see https://www.sqlite.org/src/info/05f43be8fdda9f) that caused it to
  misbehavior for BLOB operands and nobody observed that bug in nearly
  10 years of active use.  So for more users, it is probably safe to
  enable this compile-time option and thereby save a little CPU time
  on LIKE queries.
  
  This compile-time option affects the SQL LIKE operator only and has
  no impact on the sqlite3_strlike() C-language interface.

SQLITE_MAX_MEMORY=N
  This option limits the total amount of memory that SQLite will request
  from malloc() to N bytes.  Any attempt by SQLite to allocate
  new memory that would cause the sum of all allocations held by SQLite to exceed
  N bytes will result in an out-of-memory error.
  This is a hard upper limit.  See also the sqlite3_soft_heap_limit()
  interface.
  
  This limit is only functional if memory usage statistics are available via
  the sqlite3_memory_used() and sqlite3_status64(SQLITE_STATUS_MEMORY_USED)
  interfaces.  Without that memory usage information, SQLite has no way of
  knowing when it is about to go over the limit, and thus is unable to prevent
  the excess memory allocation.  Memory usage tracking is turned on by default,
  but can be disabled at compile-time using the SQLITE_DEFAULT_MEMSTATUS option,
  or at start-time using sqlite3_config(SQLITE_CONFIG_MEMSTATUS).

SQLITE_MAX_MMAP_SIZE=N
  This macro sets a hard upper bound on the amount of address space that
  can be used by any single database for memory-mapped I/O.
  Setting this value to 0 completely disables memory-mapped I/O and
  causes logic associated with memory-mapped I/O to be omitted from the
  build.  This option does change the default memory-mapped I/O address
  space size (set by SQLITE_DEFAULT_MMAP_SIZE or
  sqlite3_config(SQLITE_CONFIG_MMAP_SIZE) or the
  run-time memory-mapped I/O address space size (set by
  sqlite3_file_control(SQLITE_FCNTL_MMAP_SIZE) or
  PRAGMA mmap_size) as long as those other settings are less than the
  maximum value defined here.

SQLITE_MAX_SCHEMA_RETRY=N
  Whenever the database schema changes, prepared statements are automatically
  reprepared to accommodate the new schema.  There is a race condition here
  in that if one thread is constantly changing the schema, another thread
  might spin on reparses and repreparations of a prepared statement and
  never get any real work done.  This parameter prevents an infinite loop
  by forcing the spinning thread to give up after a fixed number of attempts
  at recompiling the prepared statement.  The default setting is 50 which is
  more than adequate for most applications.

SQLITE_MAX_WORKER_THREADS=N
  Set an upper bound on the sqlite3_limit(db,SQLITE_LIMIT_WORKER_THREADS,N)
  setting that determines the maximum number of auxiliary threads that a single
  prepared statement will use to aid with CPU-intensive computations
  (mostly sorting).  See also the SQLITE_DEFAULT_WORKER_THREADS options.

SQLITE_MEMDB_DEFAULT_MAXSIZE=N
  Set the default size limit (in bytes) for in-memory databases created using
  sqlite3_deserialize().  This is just the default.  The limit can be
  changed at start-time using
  sqlite3_config(SQLITE_CONFIG_MEMDB_MAXSIZE,N)
  or at run-time for individual databases using the
  SQLITE_FCNTL_SIZE_LIMIT file-control.
  If no default is specified, 1073741824 is used.

SQLITE_MINIMUM_FILE_DESCRIPTOR=N
  The unix VFS will never use a file descriptor less than N.  The
  default value of N is 3.
  
  Avoiding the use of low-numbered file descriptors is a defense against
  accidental database corruption.  If a database file was opened using
  file descriptor 2, for example, and then an assert() failed and invoked
  write(2,...), that would likely cause database corruption by overwriting
  part of the database file with the assertion error message.  Using only
  higher-valued file descriptors avoids this potential problem.  The 
  protection against
  using low-numbered file descriptors can be disabled by setting this
  compile-time option to 0.

SQLITE_POWERSAFE_OVERWRITE=&lt;0 or 1&gt;
  This option changes the default assumption about powersafe overwrite
  for the underlying filesystems for the unix and windows VFSes.
  Setting SQLITE_POWERSAFE_OVERWRITE to 1 causes SQLite to assume that
  application-level writes cannot changes bytes outside the range of
  bytes written even if the write occurs just before a power loss.
  With SQLITE_POWERSAFE_OVERWRITE set to 0, SQLite assumes that other
  bytes in the same sector with a written byte might be changed or 
  damaged by a power loss.

SQLITE_PRINTF_PRECISION_LIMIT=N
  This option limits the maximum width and precision of substitutions
  for the printf() SQL function and the other C-language string
  formatting functions such as sqlite3_mprintf() and
  sqlite3_str_appendf().  This is turn can prevent a hostile or
  malfunctioning script from using excessive memory by invoking
  a format such as: "printf('%*s',2147483647,'hi')".
  A value for N of around 100000 is normally sufficient.
  
  The printf() SQL function is subject to the SQLITE_LIMIT_LENGTH
  limit of sqlite3_limit().  Hence any printf() result with a
  width or precision more than the SQLITE_LIMIT_LENGTH will cause
  an SQLITE_TOOBIG error.  However, the low-level formatting
  for the printf() function is done by a subroutine that does not
  have access to SQLITE_LIMIT_LENGTH.  So the low-level formatting
  is done into a memory allocation that might be considerably larger
  than SQLITE_LIMIT_LENGTH and the SQLITE_LIMIT_LENGTH check is only
  performed after all formatting is complete.  Thus there might be a
  transient buffer that exceeds SQLITE_LIMIT_LENGTH.  The
  SQLITE_PRINTF_PRECISION_LIMIT option is an additional check
  that prevents excess sizes for the transient buffer used inside
  the low-level formatting subroutine, prior to the
  SQLITE_LIMIT_LENGTH check.
  
  Be careful not to set SQLITE_PRINTF_PRECISION_LIMIT too low.
  SQLite uses its built-in printf() functionality to format the text
  of CREATE statements stored in the sqlite_schema table.  So
  SQLITE_PRINTF_PRECISION_LIMIT should be at least as big as the largest
  table, index, view, or trigger definition that you are likely to
  encounter.
  
  No error is raised if a width or precision exceeds
  SQLITE_PRINTF_PRECISION_LIMIT.  Instead, the large width or
  precision is silently truncated.
  
  The default value for SQLITE_PRINTF_PRECISION_LIMIT is 2147483647
  (0x7fffffff).


SQLITE_QUERY_PLANNER_LIMIT=N
  As part of the query planning process, SQLite enumerates all usable
  combinations of indexes and WHERE-clause constraints.  For certain
  pathological queries, the number of these index-and-constraint combinations
  can be very large, resulting in slow performance by the query planner.
  The SQLITE_QUERY_PLANNER_LIMIT value (in conjunction with the
  related SQLITE_QUERY_PLANNER_LIMIT_INCR setting) limits the number 
  of index-and-constraint combinations that the query planner will 
  consider, in order to prevent the query planner from using excess
  CPU time.  The default value for SQLITE_QUERY_PLANNER_LIMIT is set
  high enough so that is never reached for real-world queries.  The
  query planner search limit only applies to queries that are deliberately
  crafted to use excess planning time.

SQLITE_QUERY_PLANNER_LIMIT_INCR=N
  The SQLITE_QUERY_PLANNER_LIMIT option sets an initial baseline value
  for the maximum number of index-and-constraint combinations that the
  query planner consider.  The baseline query planner limit is increased
  by SQLITE_QUERY_PLANNER_LIMIT_INCR prior to processing each table of a
  join so that each table is guaranteed to be able to propose at least
  some index-and-constraint combinations to the optimizer even if prior
  tables of the join have exhausted the baseline limit.  The default
  value for both this compile-time option and the
  SQLITE_QUERY_PLANNER_LIMIT option are set high enough so that they should
  never be reached for real-world queries.

SQLITE_REVERSE_UNORDERED_SELECTS
  This option causes the PRAGMA reverse_unordered_selects setting to be
  enabled by default.  When enabled, SELECT statements that lack an
  ORDER BY clause will run in reverse order.
  This option is useful for detecting when applications (incorrectly)
  assume that the order of rows in a SELECT without an ORDER BY clause
  will always be the same.

SQLITE_SORTER_PMASZ=N
  If multi-threaded processing is enabled via the
  PRAGMA threads setting, then sort operations will
  attempt to start helper threads when the amount of content
  to be sorted exceeds the minimum of the cache_size and PMA Size
  determined by the SQLITE_CONFIG_PMASZ start-time option.
  This compile-time option sets the default value for the
  SQLITE_CONFIG_PMASZ start-time option.
  The default value is 250.

SQLITE_STMTJRNL_SPILL=N
  The SQLITE_STMTJRNL_SPILL compile-time option determines the
  default setting of the SQLITE_CONFIG_STMTJRNL_SPILL start-time
  setting.  That setting determines the size threshold above which
  statement journals are moved from memory to disk.

SQLITE_WIN32_MALLOC
  This option enables the use of the Windows Heap API functions for memory
  allocation instead of the standard library malloc() and free() routines.

YYSTACKDEPTH=&lt;max_depth&gt;
  This macro sets the maximum depth of the LALR(1) stack used by
  the SQL parser within SQLite.  The default value is 100.  A typical
  application will use less than about 20 levels of the stack.
  Developers whose applications contain SQL statements that 
  need more than 100 LALR(1) stack entries should seriously
  consider refactoring their SQL as it is likely to be well beyond
  the ability of any human to comprehend.


5.  Options To Set Size Limits

There are compile-time options that will set upper bounds
on the sizes of various structures in SQLite.  The compile-time
options normally set a hard upper bound that can be changed
at run-time on individual database connections using the
sqlite3_limit() interface.

The compile-time options for setting upper bounds are
documented separately.  The following is a list of
the available settings:


 SQLITE_MAX_ATTACHED  
 SQLITE_MAX_COLUMN  
 SQLITE_MAX_COMPOUND_SELECT  
 SQLITE_MAX_EXPR_DEPTH  
 SQLITE_MAX_FUNCTION_ARG  
 SQLITE_MAX_LENGTH  
 SQLITE_MAX_LIKE_PATTERN_LENGTH  
 SQLITE_MAX_PAGE_COUNT  
 SQLITE_MAX_SQL_LENGTH  
 SQLITE_MAX_VARIABLE_NUMBER  



6.  Options To Control Operating Characteristics


SQLITE_4_BYTE_ALIGNED_MALLOC
  On most systems, the malloc() system call returns a buffer that is
  aligned to an 8-byte boundary.  But on some systems (ex: windows) malloc()
  returns 4-byte aligned pointer.  This compile-time option must be used
  on systems that return 4-byte aligned pointers from malloc().

SQLITE_CASE_SENSITIVE_LIKE
  If this option is present, then the built-in LIKE operator will be
  case sensitive.  This same effect can be achieved at run-time using
  the case_sensitive_like pragma.

SQLITE_DIRECT_OVERFLOW_READ
  When this option is present, content contained in
  overflow pages of the database file is read directly from disk,
  bypassing the page cache, during read transactions.  In applications
  that do a lot of reads of large BLOBs, this option might improve read
  performance.

SQLITE_HAVE_ISNAN
  If this option is present, then SQLite will use the isnan() function from
  the system math library.  This is an alias for the HAVE_ISNAN configuration
  option.

SQLITE_OS_OTHER=&lt;0 or 1&gt;
  The option causes SQLite to omit its built-in operating system interfaces
  for Unix, Windows, and OS/2.  The resulting library will have no default
  operating system interface.  Applications must use
  sqlite3_vfs_register() to register an appropriate interface before
  using SQLite.  Applications must also supply implementations for the
  sqlite3_os_init() and sqlite3_os_end() interfaces.  The usual practice
  is for the supplied sqlite3_os_init() to invoke sqlite3_vfs_register().
  SQLite will automatically invoke sqlite3_os_init() when it initializes.

  This option is typically used when building SQLite for an embedded
  platform with a custom operating system.

SQLITE_SECURE_DELETE
  This compile-time option changes the default setting of the
  secure_delete pragma.  When this option is not used, secure_delete defaults
  to off.  When this option is present, secure_delete defaults to on.

  The secure_delete setting causes deleted content to be overwritten with
  zeros.  There is a small performance penalty since additional I/O
  must occur.  On the other hand, secure_delete can prevent fragments of 
  sensitive information from lingering in unused parts of the database file 
  after it has been deleted.  See the documentation on the
  secure_delete pragma for additional information.

SQLITE_THREADSAFE=&lt;0 or 1 or 2&gt;
  This option controls whether or not code is included in SQLite to
  enable it to operate safely in a multithreaded environment.  The
  default is SQLITE_THREADSAFE=1 which is safe for use in a multithreaded
  environment.  When compiled with SQLITE_THREADSAFE=0 all mutexing code
  is omitted and it is unsafe to use SQLite in a multithreaded program.
  When compiled with SQLITE_THREADSAFE=2, SQLite can be used in a multithreaded
  program so long as no two threads attempt to use the same
  database connection (or any prepared statements derived from
  that database connection) at the same time.

  To put it another way, SQLITE_THREADSAFE=1 sets the default
  threading mode to Serialized.  SQLITE_THREADSAFE=2 sets the default
  threading mode to Multi-threaded.  And SQLITE_THREADSAFE=0 sets the
  threading mode to Single-threaded.

  The value of SQLITE_THREADSAFE can be determined at run-time
  using the sqlite3_threadsafe() interface.

  When SQLite has been compiled with SQLITE_THREADSAFE=1 or
  SQLITE_THREADSAFE=2 then the threading mode
  can be altered at run-time using the sqlite3_config() interface together
  with one of these verbs:

  
  SQLITE_CONFIG_SINGLETHREAD
  SQLITE_CONFIG_MULTITHREAD
  SQLITE_CONFIG_SERIALIZED
  

  The SQLITE_OPEN_NOMUTEX and
  SQLITE_OPEN_FULLMUTEX flags to sqlite3_open_v2() can also be used
  to adjust the threading mode of individual database connections
  at run-time.

  Note that when SQLite is compiled with SQLITE_THREADSAFE=0, the code
  to make SQLite threadsafe is omitted from the build.  When this occurs,
  it is impossible to change the threading mode at start-time or run-time.

  See the threading mode documentation for additional information
  on aspects of using SQLite in a multithreaded environment.

SQLITE_TEMP_STORE=&lt;0 through 3&gt;
  This option controls whether temporary files are stored on disk or
  in memory.  The meanings for various settings of this compile-time
  option are as follows:

  
  SQLITE_TEMP_STOREMeaning
  0Always use temporary files
  1Use files by default but allow the
  PRAGMA temp_store command to override
  2Use memory by default but allow the
  PRAGMA temp_store command to override
  3Always use memory
  

  The default setting is 1.  
  Additional information can be found in tempfiles.html.

SQLITE_TRACE_SIZE_LIMIT=N
  If this macro is defined to a positive integer N, then the length of
  strings and BLOB that are expanded into parameters in the output of
  sqlite3_trace() is limited to N bytes.  

SQLITE_TRUSTED_SCHEMA=&lt;0 or 1&gt;
  This macro determines the default value for the
  SQLITE_DBCONFIG_TRUSTED_SCHEMA and PRAGMA trusted_schema setting.
  If no alternative is specified, the trusted-schema setting defaults
  to ON (a value of 1) for legacy compatibility.  However, for best
  security, systems that implement
  application-defined SQL functions and/or virtual tables should
  consider changing the default to OFF.

SQLITE_USE_URI
  This option causes the URI filename process logic to be enabled by 
  default.  



7.  Options To Enable Features Normally Turned Off


SQLITE_ALLOW_URI_AUTHORITY
  URI filenames normally throws an error if the authority section is
  not either empty or "localhost".  However, if SQLite is compiled with
  the SQLITE_ALLOW_URI_AUTHORITY compile-time option, then the URI is
  converted into a Uniform Naming Convention (UNC) filename and passed
  down to the underlying operating system that way.  
  
  Some future versions of SQLite may change to enable this feature
  by default.

SQLITE_ALLOW_COVERING_INDEX_SCAN=&lt;0 or 1&gt;
  This C-preprocess macro determines the default setting of the
  SQLITE_CONFIG_COVERING_INDEX_SCAN configuration setting.  It defaults
  to 1 (on) which means that covering indices are used for full table
  scans where possible, in order to reduce I/O and improve performance.
  However, the use of a covering index for a full scan will cause results
  to appear in a different order from legacy, which could cause some
  (incorrectly-coded) legacy applications to break.  Hence, the covering
  index scan option can be disabled at compile-time on systems that what
  to minimize their risk of exposing errors in legacy applications.

SQLITE_ENABLE_8_3_NAMES=&lt;1 or 2&gt;
  If this C-preprocessor macro is defined, then extra code is
  included that allows SQLite to function on a filesystem that
  only support 8+3 filenames.  If the value of this macro is 1,
  then the default behavior is to continue to use long filenames and
  to only use 8+3 filenames if the 
  database connection is opened using URI filenames with
  the "8_3_names=1" query parameter.  If the value of
  this macro is 2, then the use of 8+3 filenames becomes the default
  but may be disabled on using the 8_3_names=0 query parameter.

SQLITE_ENABLE_API_ARMOR
  When defined, this C-preprocessor macro activates extra code that
  attempts to detect misuse of the SQLite API, such as passing in NULL
  pointers to required parameters or using objects after they have been
  destroyed.

SQLITE_ENABLE_ATOMIC_WRITE
  If this C-preprocessor macro is defined and if the
  xDeviceCharacteristics method of sqlite3_io_methods object for
  a database file reports (via one of the SQLITE_IOCAP_ATOMIC bits)
  that the filesystem supports atomic writes and if a transaction
  involves a change to only a single page of the database file,
  then the transaction commits with just a single write request of
  a single page of the database and no rollback journal is created
  or written.  On filesystems that support atomic writes, this
  optimization can result in significant speed improvements for
  small updates.  However, few filesystems support this capability
  and the code paths that check for this capability slow down write
  performance on systems that lack atomic write capability, so this
  feature is disabled by default.

SQLITE_ENABLE_BATCH_ATOMIC_WRITE
  This compile-time option enables SQLite to take advantage batch
  atomic write capabilities in the underlying filesystem.  As of
  SQLite version 3.21.0 (2017-10-24) this is only supported on
  F2FS.  However, the interface
  is implemented generically, using sqlite3_file_control() with
  SQLITE_FCNTL_BEGIN_ATOMIC_WRITE and SQLITE_FCNTL_COMMIT_ATOMIC_WRITE
  so the capability can be added to other filesystem times in the
  future.  When this option is enabled, SQLite automatically detects
  that the underlying filesystem supports batch atomic writes, and
  when it does so it avoids writing the rollback journal for transaction
  control.  This can make transactions over twice as fast, while
  simultaneously reducing wear on SSD storage devices.

  Future versions of SQLite might enable the batch-atomic-write
  capability by default, at which point this compile-time option
  will become superfluous.

SQLITE_ENABLE_BYTECODE_VTAB
  This option enables the bytecode and tables_used virtual tables.

SQLITE_ENABLE_COLUMN_METADATA
  When this C-preprocessor macro is defined, SQLite includes some
  additional APIs that provide convenient access to meta-data about
  tables and queries.  The APIs that are enabled by this option are:

  
   sqlite3_column_database_name() 
   sqlite3_column_database_name16() 
   sqlite3_column_table_name() 
   sqlite3_column_table_name16() 
   sqlite3_column_origin_name() 
   sqlite3_column_origin_name16() 
  

SQLITE_ENABLE_DBPAGE_VTAB
  This option enables the SQLITE_DBPAGE virtual table.

SQLITE_ENABLE_DBSTAT_VTAB
  This option enables the dbstat virtual table.

SQLITE_ENABLE_DESERIALIZE
  This option was formerly used to enable
  the sqlite3_serialize() and sqlite3_deserialize()
  interfaces.  However, as of SQLite 3.36.0 (2021-06-18)
  those interfaces are enabled by default and a new
  compile-time option SQLITE_OMIT_DESERIALIZE is added
  to omit them.

SQLITE_ENABLE_EXPLAIN_COMMENTS
  This option adds extra logic to SQLite that inserts comment text into the
  output of EXPLAIN.  These extra comments use extra memory, thus
  making prepared statements larger and very slightly slower, and so they are
  turned off by default and in most application.  But some applications, such
  as the command-line shell for SQLite, value clarity of EXPLAIN output
  over raw performance and so this compile-time option is available to them.
  The SQLITE_ENABLE_EXPLAIN_COMMENTS compile-time option is also enabled
  automatically if SQLITE_DEBUG is enabled.

SQLITE_ENABLE_FTS3
  When this option is defined in the amalgamation, version 3
  of the full-text search engine is added to the build automatically.

SQLITE_ENABLE_FTS3_PARENTHESIS
  This option modifies the query pattern parser in FTS3 such that it
  supports operators AND and NOT (in addition to the usual OR and NEAR)
  and also allows query expressions to contain nested parenthesis.

SQLITE_ENABLE_FTS3_TOKENIZER
  This option enables the two-argument version of the fts3_tokenizer()
  interface.  The second argument to fts3_tokenizer() is suppose to be a
  pointer to a function (encoded as a BLOB) that implements an
  application defined tokenizer.  If hostile actors are able to run
  the two-argument version of fts3_tokenizer() with an arbitrary second
  argument, they could use crash or take control of the process. 
  
  Because of security concerns, the two-argument fts3_tokenizer() feature 
  was disabled beginning with Version 3.11.0 (2016-02-15)
  unless this compile-time option is used.
  Version 3.12.0 (2016-03-29) added the 
  sqlite3_db_config(db,SQLITE_DBCONFIG_ENABLE_FTS3_TOKENIZER,1,0) interface
  that activates the two-argument version of fts3_tokenizer()
  for a specific database connection at run-time.

SQLITE_ENABLE_FTS4
  When this option is defined in the amalgamation, versions 3 and 4
  of the full-text search engine is added to the build automatically.

SQLITE_ENABLE_FTS5
  When this option is defined in the amalgamation, versions 5
  of the full-text search engine (fts5) is added to the build automatically.

SQLITE_ENABLE_GEOPOLY
  When this option is defined in the amalgamation, the Geopoly extension
  is included in the build.

SQLITE_ENABLE_ICU
  This option causes the 
  International Components for Unicode
  or "ICU" extension to SQLite to be added to the build.  

SQLITE_ENABLE_IOTRACE
  When both the SQLite core and the Command Line Interface (CLI) are both 
  compiled with this option, then the CLI provides an extra command
  named ".iotrace" that provides a low-level log of I/O activity.
  This option is experimental and may be discontinued in a future release.

SQLITE_ENABLE_MATH_FUNCTIONS
  This macro enables the built-in SQL math functions.  This option
  is automatically added to the Makefile by the configure script on unix platforms,
  unless the --disable-math option is used.
  This option is also included on Windows builds using the
  "Makefile.msc" makefile for nmake.

SQLITE_ENABLE_JSON1
  This compile-time option is a no-op.  Prior to SQLite version 3.38.0
  (2022-02-22), it was necessary to compile with this option in order
  to include the JSON SQL functions in the build.  However, beginning
  with SQLite version 3.38.0, those functions are included by default.
  Use the -DSQLITE_OMIT_JSON option to omit them.

SQLITE_ENABLE_LOCKING_STYLE
  This option enables additional logic in the OS interface layer for
  Mac OS X. The additional logic attempts to determine the type of the
  underlying filesystem and choose and alternative locking strategy
  that works correctly for that filesystem type. Five locking strategies 
  are available:

  
     POSIX locking style. This is the default locking style and the
         style used by other (non Mac OS X) Unixes. Locks are obtained and 
         released using the fcntl() system call.

     AFP locking style. This locking style is used for network file 
         systems that use the AFP (Apple Filing Protocol) protocol. Locks
         are obtained by calling the library function _AFPFSSetLock().

     Flock locking style. This is used for file-systems that do not
         support POSIX locking style. Locks are obtained and released using
         the flock() system call.

     Dot-file locking style. This locking style is used when neither
         flock nor POSIX locking styles are supported by the file system.
         Database locks are obtained by creating and entry in the file-system
         at a well-known location relative to the database file (a "dot-file")
         and relinquished by deleting the same file.

     No locking style. If none of the above can be supported, this 
         locking style is used. No database locking mechanism is used. When
         this system is used it is not safe for a single database to be
         accessed by multiple clients.
  

  Additionally, five extra VFS implementations are provided as well as the
  default. By specifying one of the extra VFS implementations 
  when calling sqlite3_open_v2(), an application may bypass the file-system
  detection logic and explicitly select one of the above locking styles. The
  five extra VFS implementations are called "unix-posix", "unix-afp",
  "unix-flock", "unix-dotfile" and "unix-none".

SQLITE_ENABLE_MEMORY_MANAGEMENT
  This option adds extra logic to SQLite that allows it to release unused
  memory upon request.  This option must be enabled in order for the
  sqlite3_release_memory() interface to work.  If this compile-time
  option is not used, the sqlite3_release_memory() interface is a 
  no-op.

SQLITE_ENABLE_MEMSYS3
  This option includes code in SQLite that implements an alternative
  memory allocator.  This alternative memory allocator is only engaged
  when the SQLITE_CONFIG_HEAP option to sqlite3_config() is used to
  supply a large chunk of memory from which all memory allocations are
  taken.
  The MEMSYS3 memory allocator uses a hybrid allocation algorithm 
  patterned after dlmalloc().   Only one of SQLITE_ENABLE_MEMSYS3 and 
  SQLITE_ENABLE_MEMSYS5 may be enabled at once.

SQLITE_ENABLE_MEMSYS5
  This option includes code in SQLite that implements an alternative
  memory allocator.  This alternative memory allocator is only engaged
  when the SQLITE_CONFIG_HEAP option to sqlite3_config() is used to
  supply a large chunk of memory from which all memory allocations are
  taken.
  The MEMSYS5 module rounds all allocations up to the next power
  of two and uses a first-fit, buddy-allocator algorithm
  that provides strong guarantees against fragmentation and breakdown
  subject to certain operating constraints.

SQLITE_ENABLE_NORMALIZE
  This option includes the sqlite3_normalized_sql() API.

SQLITE_ENABLE_NULL_TRIM
  This option enables an optimization that omits NULL columns at
  the ends of rows, for a space savings on disk.
  
  Databases generated with this option enabled are not readable
  by SQLite version 3.1.6 (2005-03-17) and earlier.  Also,
  databases generated with this option enabled are prone to
  triggering the
  e6e962d6b0f06f46
  bug in the sqlite3_blob_reopen() interface.  For those reasons,
  this optimization is disabled by default.  However, this optimization
  may be enabled by default in a future release of SQLite.

SQLITE_ENABLE_OFFSET_SQL_FUNC
  This option enables support for the sqlite_offset(X) SQL function.
  
  The sqlite_offset(X) SQL function requires a new interface on the
  B-tree storage engine, a new opcode in the virtual machine that
  runs SQL statements, and a new conditional in a critical path of the
  code generator.  To avoid that overhead in applications that do not
  need the utility of sqlite_offset(X), the function is disabled by
  default.

SQLITE_ENABLE_PREUPDATE_HOOK
  This option enables 
  several new APIs that provide callbacks
  prior to any change to a rowid table.  The callbacks can be used
  to record the state of the row before the change occurs.
  The action of the preupdate hook is similar to the
  update hook except that the callback is
  invoked before the change, not afterwards, and the preupdate
  hook interfaces are omitted unless this compile-time option is
  used.
  The preupdate hook interfaces were originally added to
  support the session extension.

SQLITE_ENABLE_QPSG
  This option causes the query planner stability guarantee (QPSG) to
  be on by default.  Normally the QPSG is off and must be activated
  at run-time using the SQLITE_DBCONFIG_ENABLE_QPSG option to the
  sqlite3_db_config() interface.

SQLITE_ENABLE_RBU
  Enable the code the implements the RBU extension.

SQLITE_ENABLE_RTREE
  This option causes SQLite to include support for the
  R*Tree index extension.

SQLITE_ENABLE_SESSION
  This option enables the session extension.

SQLITE_ENABLE_SNAPSHOT
  This option enables the code to support the sqlite3_snapshot object
  and its related interfaces:
  
   sqlite3_snapshot_get() (constructor)
   sqlite3_snapshot_free() (destructor)
   sqlite3_snapshot_open()
   sqlite3_snapshot_cmp()
   sqlite3_snapshot_recover()
  

SQLITE_ENABLE_SORTER_REFERENCES
  This option activates an optimization that reduces the memory required
  by the sorter at the cost of doing additional B-tree lookups after
  the sort has occurred.
  
  The default sorting procedure is to gather all information that will
  ultimately be output into a "record" and pass that complete record
  to the sorter.  But in some cases, for example if some of the output
  columns consists of large BLOB values, the size of the each record
  can be large, which means that the sorter has to either use more memory,
  and/or write more content to temporary storage.
  
  When SQLITE_ENABLE_SORTER_REFERENCES is enabled, the records passed to
  the sorter often contain only a ROWID value.  Such records are much
  smaller.  This means the sorter has much less "payload" to deal with and
  can run faster.  After sorting has occurred, the ROWID is used to look up 
  the output column values in the original table.  That requires another
  search into the table, and could potentially result in a slowdown.  Or,
  it might be a performance win, depending on how large the values are.
  
  Even when the SQLITE_ENABLE_SORTER_REFERENCES compile-time option is on,
  sorter references are still disabled by default.  To use sorter references,
  the application must set a sorter reference size threshold using the
  sqlite3_config(SQLITE_CONFIG_SORTERREF_SIZE) interface at start-time.
  
  Because the SQLite developers do not know whether the 
  SQLITE_ENABLE_SORTER_REFERENCES option will help or hurt performance,
  it is disabled by default at this time (2018-05-04).  It might be enabled
  by default in some future release, depending on what is learned about its
  impact on performance.

SQLITE_ENABLE_STMT_SCANSTATUS
  This option enables the sqlite3_stmt_scanstatus() interface.  The
  sqlite3_stmt_scanstatus() interface is normally omitted from the build
  because it imposes a small performance penalty, even on statements that
  do not use the feature.

SQLITE_ENABLE_STMTVTAB
  This compile-time option enables the SQLITE_STMT virtual table logic.

SQLITE_RTREE_INT_ONLY
  This compile-time option is deprecated and untested.

SQLITE_ENABLE_SQLLOG
  This option enables extra code (especially the SQLITE_CONFIG_SQLLOG
  option to sqlite3_config()) that can be used to create logs of all
  SQLite processing performed by an application.  These logs can be useful
  in doing off-line analysis of the behavior of an application, and especially
  for performance analysis.  In order for the SQLITE_ENABLE_SQLLOG option to 
  be useful, some extra code is required.  The 
  "test_sqllog.c"
  source code
  file in the SQLite source tree is a working example of the required extra
  code.  On unix and windows systems, a developer can append the text of the
  "test_sqllog.c" source code file to the end of an "sqlite3.c" amalgamation,
  recompile the application using the -DSQLITE_ENABLE_SQLLOG option, then 
  control logging using environment variables.  See the header comment on 
  the "test_sqllog.c" source file for additional detail.  

SQLITE_ENABLE_STAT2
  This option used to cause the ANALYZE command to collect
  index histogram data in the sqlite_stat2 table.  But that
  functionality was superceded by SQLITE_ENABLE_STAT3 as of
  SQLite version 3.7.9 (2011-11-01).  
  The SQLITE_ENABLE_STAT2 compile-time option
  is now a no-op.

SQLITE_ENABLE_STAT3
  This option used to cause the ANALYZE command to collect
  index histogram data in the sqlite_stat3 table.  But that
  functionality was superceded by SQLITE_ENABLE_STAT4 as of
  SQLite version 3.8.1 (2013-10-17).  
  The SQLITE_ENABLE_STAT3 compile-time option
  continued to be supported through version 3.29.0 (2019-07-10)
  but has now become a no-op.
  

SQLITE_ENABLE_STAT4
  This option adds additional logic to the ANALYZE command and to
  the query planner that can help SQLite to chose a better query plan
  under certain situations.  The ANALYZE command is enhanced to collect
  histogram data from all columns of every index and store that data
  in the sqlite_stat4 table.  The query planner will then use the
  histogram data to help it make better index choices.  The downside of
  this compile-time option is that it violates the
  query planner stability guarantee making it more difficult to ensure
  consistent performance in mass-produced applications.
  
  SQLITE_ENABLE_STAT4 is an enhancement of SQLITE_ENABLE_STAT3.  STAT3
  only recorded histogram data for the left-most column of each index
  whereas the STAT4 enhancement records histogram data from all columns
  of each index.
  The SQLITE_ENABLE_STAT3 compile-time option has become a no-op.

SQLITE_ENABLE_TREE_EXPLAIN
  This compile-time option is no longer used.

SQLITE_ENABLE_UPDATE_DELETE_LIMIT
  This option enables an optional ORDER BY and LIMIT clause on 
  UPDATE and DELETE statements.

  If this option is defined, then it must also be 
  defined when using the Lemon parser generator tool to generate a parse.c
  file. Because of this, this option may only be used when the library is built
  from source, not from the amalgamation or from the collection of
  pre-packaged C files provided for non-Unix like platforms on the website.
  

SQLITE_ENABLE_UNKNOWN_SQL_FUNCTION
  When the SQLITE_ENABLE_UNKNOWN_SQL_FUNCTION compile-time option is
  activated, SQLite will suppress "unknown function" errors when running
  an EXPLAIN or EXPLAIN QUERY PLAN.  Instead of throwing an error,
  SQLite will insert a substitute no-op function named "unknown()".
  The substitution of "unknown()" in place of unrecognized functions
  only occurs on EXPLAIN and EXPLAIN QUERY PLAN, not on ordinary
  statements.
  
  When used in the command-line shell, the
  SQLITE_ENABLE_UNKNOWN_SQL_FUNCTION feature allows SQL text that contains
  application-defined functions to be pasted into the shell for 
  analysis and debugging without having to create and load an
  extension that implements the application-defined functions.

SQLITE_ENABLE_UNLOCK_NOTIFY
  This option enables the sqlite3_unlock_notify() interface and
  its associated functionality.  See the documentation titled
  Using the SQLite Unlock Notification Feature for additional
  information.

SQLITE_INTROSPECTION_PRAGMAS
  This option is obsolete.  It used to enable some extra
  some extra PRAGMA statements such as
  PRAGMA function_list, PRAGMA module_list, and
  PRAGMA pragma_list, but those pragmas are now all
  enabled by default.  See SQLITE_OMIT_INTROSPECTION_PRAGMAS.

SQLITE_SOUNDEX
  This option enables the soundex() SQL function.

SQLITE_USE_ALLOCA
  If this option is enabled, then the alloca() memory allocator will be
  used in a few situations where it is appropriate.  This results in a slightly
  smaller and faster binary.  The SQLITE_USE_ALLOCA compile-time only
  works, of course, on systems that support alloca().

SQLITE_USE_FCNTL_TRACE
  This option causes SQLite to issue extra SQLITE_FCNTL_TRACE file controls
  to provide supplementary information to the VFS.  The "vfslog.c" extension
  makes use of this to provide enhanced logs of VFS activity.

SQLITE_HAVE_ZLIB
  This option causes some extensions to link against the 
  zlib compression library.
  
  This option has no effect on the SQLite core.  It is only used by extensions.
  This is option is necessary for the compression and decompression
  functions that are part of SQL Archive support in the
  command-line shell.
  
  When compiling with this option, it will normally
  be necessary to add a linker option to include the zlib library in the
  build.  Normal this option is "-lz" but might be different on different
  systems.
  
  When building with MSVC on Windows systems, one can put the zlib source
  code in the compat/zlib subdirectory of the source tree and then add
  the USE_ZLIB=1 option to the nmake command to cause the The Makefile.msc
  to automatically build and use an appropriate zlib library implementation.

YYTRACKMAXSTACKDEPTH
  This option causes the LALR(1) parser stack depth to be tracked
  and reported using the sqlite3_status(SQLITE_STATUS_PARSER_STACK,...)
  interface.  SQLite's LALR(1) parser has a fixed stack depth
  (determined at compile-time using the YYSTACKDEPTH options).
  This option can be used to help determine if an application is
  getting close to exceeding the maximum LALR(1) stack depth.



8.  Options To Disable Features Normally Turned On


SQLITE_DISABLE_LFS
  If this C-preprocessor macro is defined, large file support
  is disabled.

SQLITE_DISABLE_DIRSYNC
  If this C-preprocessor macro is defined, directory syncs
  are disabled.  SQLite typically attempts to sync the parent
  directory when a file is deleted to ensure the directory
  entries are updated immediately on disk.

SQLITE_DISABLE_FTS3_UNICODE
  If this C-preprocessor macro is defined, the unicode61 tokenizer
  in FTS3 is omitted from the build and is unavailable to 
  applications.

SQLITE_DISABLE_FTS4_DEFERRED
  If this C-preprocessor macro disables the "deferred token" optimization
  in FTS4.  The "deferred token" optimization avoids loading massive
  posting lists for terms that are in most documents of the collection
  and instead simply scans for those tokens in the document source.  FTS4
  should get exactly the same answer both with and without this optimization.

SQLITE_DISABLE_INTRINSIC
  This option disables the use of compiler-specific built-in functions
  such as __builtin_bswap32() and __builtin_add_overflow() in GCC and Clang, 
  or _byteswap_ulong() and _ReadWriteBarrier() with MSVC.  




9.  Options To Omit Features

The following options can be used to 
reduce the size of the compiled library
by omitting unused features. This is probably only useful
in embedded systems where space is especially tight, as even with all
features included the SQLite library is relatively small. Don't forget
to tell your compiler to optimize for binary size! (the -Os option if
using GCC).  Telling your compiler to optimize for size usually has
a much larger impact on library footprint than employing any of these
compile-time options.  You should also verify that 
debugging options are disabled.

The macros in this section do not require values. The following 
compilation switches all have the same effect:
-DSQLITE_OMIT_ALTERTABLE
-DSQLITE_OMIT_ALTERTABLE=1
-DSQLITE_OMIT_ALTERTABLE=0


If any of these options are defined, then the same set of SQLITE_OMIT_*
options must also be defined when using the Lemon parser generator
tool to generate the
parse.c file and when compiling the 'mkkeywordhash' tool which generates 
the keywordhash.h file.
Because of this, these options may only be used when the library is built
from canonical source, not from the amalgamation.
Some SQLITE_OMIT_* options might work, or appear to work, when used with
the amalgamation.  But this is not guaranteed.  In general, always compile
from canonical sources in order to take advantage of SQLITE_OMIT_* options.



Important Note: The SQLITE_OMIT_* options may not work with the
amalgamation.  SQLITE_OMIT_* compile-time
options usually work correctly only when SQLite is built from canonical 
source files.




Special versions of the SQLite amalgamation that do work with a
predetermined set of SQLITE_OMIT_* options can be generated.  To do so,
make a copy of the Makefile.linux-gcc makefile template in the canonical
source code distribution.  Change the name of your copy to simply "Makefile".
Then edit "Makefile" to set up appropriate compile-time options.  Then
type:
make clean; make sqlite3.c

The resulting "sqlite3.c" amalgamation code file (and its associated
header file "sqlite3.h") can then be moved to a non-unix platform
for final compilation using a native compiler.

The SQLITE_OMIT_* options are unsupported.  By this we mean that
an SQLITE_OMIT_* option that omits code from the build in the current
release might become a no-op in the next release.  Or the other way around:
an SQLITE_OMIT_* that is a no-op in the current release might cause code
to be excluded in the next release.  Also, not all SQLITE_OMIT_* options
are tested.  Some SQLITE_OMIT_* options might cause SQLite to malfunction
and/or provide incorrect answers.


Important Note:
The SQLITE_OMIT_* compile-time options are mostly unsupported.


The following are the available OMIT options:


SQLITE_OMIT_ALTERTABLE
  When this option is defined, the 
  ALTER TABLE command is not included in the 
  library. Executing an ALTER TABLE statement causes a parse error.

SQLITE_OMIT_ANALYZE
  When this option is defined, the ANALYZE command is omitted from
  the build.

SQLITE_OMIT_ATTACH
  When this option is defined, the ATTACH and DETACH commands are
  omitted from the build.

SQLITE_OMIT_AUTHORIZATION
  Defining this option omits the authorization callback feature from the
  library. The sqlite3_set_authorizer() API function is not present
  in the library.

SQLITE_OMIT_AUTOINCREMENT
  This option is omits the AUTOINCREMENT feature. 
  When this is macro is defined, columns declared as 
  "INTEGER PRIMARY KEY AUTOINCREMENT"
  behave in the same way as columns declared as "INTEGER PRIMARY KEY" when a 
  NULL is inserted. The sqlite_sequence system table is neither created, nor
  respected if it already exists.

SQLITE_OMIT_AUTOINIT
  For backwards compatibility with older versions of SQLite that lack
  the sqlite3_initialize() interface, the sqlite3_initialize() interface
  is called automatically upon entry to certain key interfaces such as
  sqlite3_open(), sqlite3_vfs_register(), and sqlite3_mprintf().
  The overhead of invoking sqlite3_initialize() automatically in this
  way may be omitted by building SQLite with the SQLITE_OMIT_AUTOINIT
  C-preprocessor macro.  When built using SQLITE_OMIT_AUTOINIT, SQLite
  will not automatically initialize itself and the application is required
  to invoke sqlite3_initialize() directly prior to beginning use of the
  SQLite library.

SQLITE_OMIT_AUTOMATIC_INDEX
  This option is used to omit the 
  automatic indexing functionality.
  See also: SQLITE_DEFAULT_AUTOMATIC_INDEX.

SQLITE_OMIT_AUTORESET
  By default, the sqlite3_step() interface will automatically invoke
  sqlite3_reset() to reset the prepared statement if necessary.  This
  compile-time option changes that behavior so that sqlite3_step() will
  return SQLITE_MISUSE if it called again after returning anything other
  than SQLITE_ROW, SQLITE_BUSY, or SQLITE_LOCKED unless there was an
  intervening call to sqlite3_reset().

  In SQLite version 3.6.23.1 (2010-03-26)
  and earlier, sqlite3_step() used to always
  return SQLITE_MISUSE if it was invoked again after returning anything
  other than SQLITE_ROW without an intervening call to sqlite3_reset().
  This caused problems on some poorly written smartphone applications which
  did not correctly handle the SQLITE_LOCKED and SQLITE_BUSY error 
  returns.  Rather than fix the many defective smartphone applications, 
  the behavior of SQLite was changed in 3.6.23.2 to automatically reset
  the prepared statement.  But that changed caused issues in other 
  improperly implemented applications that were actually looking
  for an SQLITE_MISUSE return to terminate their query loops.  (Anytime
  an application gets an SQLITE_MISUSE error code from SQLite, that means the
  application is misusing the SQLite interface and is thus incorrectly
  implemented.)  The SQLITE_OMIT_AUTORESET interface was added to SQLite
  version 3.7.5 (2011-02-01) in an effort to get all of the (broken)
  applications to work again without having to actually fix the applications.

SQLITE_OMIT_AUTOVACUUM
  If this option is defined, the library cannot create or write to 
  databases that support auto_vacuum.
  Executing a PRAGMA auto_vacuum statement is not an error
  (since unknown PRAGMAs are silently ignored), but does not return a value
  or modify the auto-vacuum flag in the database file. If a database that
  supports auto-vacuum is opened by a library compiled with this option, it
  is automatically opened in read-only mode.

SQLITE_OMIT_BETWEEN_OPTIMIZATION
  This option disables the use of indices with WHERE clause terms
  that employ the BETWEEN operator.

SQLITE_OMIT_BLOB_LITERAL
  When this option is defined, it is not possible to specify a blob in
  an SQL statement using the X'ABCD' syntax.

SQLITE_OMIT_BTREECOUNT
  This option is no longer used for anything.  It is a no-op.

SQLITE_OMIT_BUILTIN_TEST
  This compile-time option has been renamed to SQLITE_UNTESTABLE.

SQLITE_OMIT_CASE_SENSITIVE_LIKE_PRAGMA
  This compile-time option disables the PRAGMA case_sensitive_like
  command.

SQLITE_OMIT_CAST
  This option causes SQLite to omit support for the CAST operator.

SQLITE_OMIT_CHECK
  This option causes SQLite to omit support for CHECK constraints.
  The parser will still accept CHECK constraints in SQL statements,
  they will just not be enforced.

SQLITE_OMIT_COMPILEOPTION_DIAGS
  This option is used to omit the compile-time option diagnostics available
  in SQLite, including the sqlite3_compileoption_used() and
  sqlite3_compileoption_get() C/C++ functions, the
  sqlite_compileoption_used() and sqlite_compileoption_get() SQL functions,
  and the compile_options pragma.

SQLITE_OMIT_COMPLETE
  This option causes the sqlite3_complete() and sqlite3_complete16()
  interfaces to be omitted.

SQLITE_OMIT_COMPOUND_SELECT
  This option is used to omit the compound SELECT functionality. 
  SELECT statements that use the 
  UNION, UNION ALL, INTERSECT or EXCEPT compound SELECT operators will 
  cause a parse error.

  An INSERT statement with multiple values in the VALUES clause is
  implemented internally as a compound SELECT.  Hence, this option also
  disables the ability to insert more than a single row using an
  INSERT INTO ... VALUES ... statement.

SQLITE_OMIT_CTE
  This option causes support for common table expressions to be omitted.

SQLITE_OMIT_DATETIME_FUNCS
  If this option is defined, SQLite's built-in date and time manipulation
  functions are omitted. Specifically, the SQL functions julianday(), date(),
  time(), datetime() and strftime() are not available. The default column
  values CURRENT_TIME, CURRENT_DATE and CURRENT_TIMESTAMP are still available.

SQLITE_OMIT_DECLTYPE
  This option causes SQLite to omit support for the
  sqlite3_column_decltype() and sqlite3_column_decltype16()
  interfaces.

SQLITE_OMIT_DEPRECATED
  This option causes SQLite to omit support for interfaces
  marked as deprecated.  This includes 
  sqlite3_aggregate_count(),
  sqlite3_expired(),
  sqlite3_transfer_bindings(),
  sqlite3_global_recover(),
  sqlite3_thread_cleanup() and
  sqlite3_memory_alarm() interfaces and
  PRAGMA statements PRAGMA count_changes,
  PRAGMA data_store_directory,
  PRAGMA default_cache_size,
  PRAGMA empty_result_callbacks,
  PRAGMA full_column_names,
  PRAGMA short_column_names, and
  PRAGMA temp_store_directory.

SQLITE_OMIT_DESERIALIZE
  This option causes the
  sqlite3_serialize() and sqlite3_deserialize()
  interfaces to be omitted from the build.

SQLITE_OMIT_DISKIO
  This option omits all support for writing to the disk and forces
  databases to exist in memory only.  This option has not been 
  maintained and probably does not work with newer versions of SQLite.

SQLITE_OMIT_EXPLAIN
  Defining this option causes the EXPLAIN command to be omitted from the
  library. Attempting to execute an EXPLAIN statement will cause a parse
  error.

SQLITE_OMIT_FLAG_PRAGMAS
  This option omits support for a subset of PRAGMA commands that
  query and set boolean properties.

SQLITE_OMIT_FLOATING_POINT
  This option is used to omit floating-point number support from the SQLite
  library. When specified, specifying a floating point number as a literal 
  (i.e. "1.01") results in a parse error.

  In the future, this option may also disable other floating point 
  functionality, for example the sqlite3_result_double(), 
  sqlite3_bind_double(), sqlite3_value_double() and
  sqlite3_column_double() API functions.
  

SQLITE_OMIT_FOREIGN_KEY
  If this option is defined, then foreign key constraint syntax is
  not recognized.

SQLITE_OMIT_GENERATED_COLUMNS
  If this option is defined, then generated column syntax is
  not recognized.

SQLITE_OMIT_GET_TABLE
  This option causes support for sqlite3_get_table() and
  sqlite3_free_table() to be omitted.

SQLITE_OMIT_HEX_INTEGER
  This option omits support for hexadecimal integer literals.

SQLITE_OMIT_INCRBLOB
  This option causes support for incremental BLOB I/O
  to be omitted.

SQLITE_OMIT_INTEGRITY_CHECK
  This option omits support for the integrity_check pragma.

SQLITE_OMIT_INTROSPECTION_PRAGMAS
  This option omits support for
  PRAGMA function_list, PRAGMA module_list, and
  PRAGMA pragma_list.

SQLITE_OMIT_JSON
  This option omits the JSON SQL functions from the build.

SQLITE_OMIT_LIKE_OPTIMIZATION
  This option disables the ability of SQLite to use indices to help
  resolve LIKE and GLOB operators in a WHERE clause.

SQLITE_OMIT_LOAD_EXTENSION
  This option omits the entire extension loading mechanism from
  SQLite, including sqlite3_enable_load_extension() and
  sqlite3_load_extension() interfaces.

SQLITE_OMIT_LOCALTIME
  This option omits the "localtime" modifier from the date and time
  functions.  This option is sometimes useful when trying to compile
  the date and time functions on a platform that does not support the
  concept of local time.

SQLITE_OMIT_LOOKASIDE
  This option omits the lookaside memory allocator.

SQLITE_OMIT_MEMORYDB
  When this is defined, the library does not respect the special database
  name ":memory:" (normally used to create an in-memory database). If 
  ":memory:" is passed to sqlite3_open(), sqlite3_open16(), or
  sqlite3_open_v2(), a file with this name will be 
  opened or created.

SQLITE_OMIT_OR_OPTIMIZATION
  This option disables the ability of SQLite to use an index together
  with terms of a WHERE clause connected by the OR operator.

SQLITE_OMIT_PAGER_PRAGMAS
  Defining this option omits pragmas related to the pager subsystem from 
  the build.

SQLITE_OMIT_PRAGMA
  This option is used to omit the PRAGMA command
  from the library. Note that it is useful to define the macros that omit
  specific pragmas in addition to this, as they may also remove supporting code
  in other sub-systems. This macro removes the PRAGMA command only.

SQLITE_OMIT_PROGRESS_CALLBACK
  This option may be defined to omit the capability to issue "progress" 
  callbacks during long-running SQL statements. The 
  sqlite3_progress_handler()
  API function is not present in the library.

SQLITE_OMIT_QUICKBALANCE
  This option omits an alternative, faster B-Tree balancing routine.
  Using this option makes SQLite slightly smaller at the expense of
  making it run slightly slower.

SQLITE_OMIT_REINDEX
  When this option is defined, the REINDEX
  command is not included in the library.
  Executing a REINDEX statement causes 
  a parse error.

SQLITE_OMIT_SCHEMA_PRAGMAS
  Defining this option omits pragmas for querying the database schema from 
  the build.

SQLITE_OMIT_SCHEMA_VERSION_PRAGMAS
  Defining this option omits pragmas for querying and modifying the 
  database schema version and user version from the build. Specifically, the 
  schema_version and user_version PRAGMAs are omitted.

SQLITE_OMIT_SHARED_CACHE
  This option builds SQLite without support for shared cache mode.
  The sqlite3_enable_shared_cache() is omitted along with a fair
  amount of logic within the B-Tree subsystem associated with shared
  cache management.

  This compile-time option is recommended most applications as it
  results in improved performance and reduced library footprint.

SQLITE_OMIT_SUBQUERY
  If defined, support for sub-selects and the IN() operator are omitted.

SQLITE_OMIT_TCL_VARIABLE
  If this macro is defined, then the special "$" syntax
  used to automatically bind SQL variables to TCL variables is omitted.

SQLITE_OMIT_TEMPDB
  This option omits support for TEMP or TEMPORARY tables.

SQLITE_OMIT_TRACE
  This option omits support for the sqlite3_profile() and
  sqlite3_trace() interfaces and their associated logic.

SQLITE_OMIT_TRIGGER
  Defining this option omits support for TRIGGER objects. Neither the 
  CREATE TRIGGER or DROP TRIGGER
  commands are available in this case, and attempting to execute
  either will result in a parse error.
  This option also disables enforcement of foreign key constraints,
  since the code that implements triggers and which is omitted by this
  option is also used to implement foreign key actions.

SQLITE_OMIT_TRUNCATE_OPTIMIZATION
  A default build of SQLite, if a DELETE statement has no WHERE clause
  and operates on a table with no triggers, an optimization occurs that
  causes the DELETE to occur by dropping and recreating the table.  
  Dropping and recreating a table is usually much faster than deleting
  the table content row by row.  This is the "truncate optimization".

SQLITE_OMIT_UTF16
  This macro is used to omit support for UTF16 text encoding. When this is
  defined all API functions that return or accept UTF16 encoded text are
  unavailable. These functions can be identified by the fact that they end
  with '16', for example sqlite3_prepare16(), sqlite3_column_text16() and
  sqlite3_bind_text16().

SQLITE_OMIT_VACUUM
  When this option is defined, the VACUUM
  command is not included in the library.
  Executing a VACUUM statement causes 
  a parse error.

SQLITE_OMIT_VIEW
  Defining this option omits support for VIEW objects. Neither the 
  CREATE VIEW nor the DROP VIEW
  commands are available in this case, and
  attempting to execute either will result in a parse error.

  WARNING: If this macro is defined, it will not be possible to open a database
  for which the schema contains VIEW objects. 

SQLITE_OMIT_VIRTUALTABLE
  This option omits support for the Virtual Table
  mechanism in SQLite.

SQLITE_OMIT_WAL
  This option omits the "write-ahead log" (a.k.a. "WAL") capability.

SQLITE_OMIT_WINDOWFUNC
  This option omits window functions from the build.

SQLITE_OMIT_WSD
  This option builds a version of the SQLite library that contains no
  Writable Static Data (WSD).  WSD is global variables and/or static
  variables.  Some platforms do not support WSD, and this option is necessary
  in order for SQLite to work those platforms.  

  Unlike other OMIT options which make the SQLite library smaller,
  this option actually increases the size of SQLite and makes it run
  a little slower.  Only use this option if SQLite is being built for an
  embedded target that does not support WSD.

SQLITE_OMIT_XFER_OPT
  This option omits support for optimizations that help statements
  of the form "INSERT INTO ... SELECT ..." run faster.

SQLITE_UNTESTABLE
  A standard SQLite build includes a small amount of logic associated
  with sqlite3_test_control() to exercise
  parts of the SQLite core that are otherwise difficult to validate.
  This compile-time option omits that extra testing logic.  This
  compile-time option was called "SQLITE_OMIT_BUILTIN_TEST" prior
  to SQLite version 3.16.0 (2017-01-02).  The name was changed
  to better describe the implications of using it.
  
  Setting this compile-time option prevents SQLite from being fully
  testable.  Branch test coverage drops from 100% down to about 95%.
  
  SQLite developers follow the NASA principle of
  "fly what you test and test what you fly".  This principle is violated
  if this option is enabled for delivery but disabled for testing.
  But if this option is enabled during testing, not all branches are 
  reachable.  Therefore, the use of this compile-time option is discouraged.

SQLITE_ZERO_MALLOC
  This option omits both the default memory allocator and the
  debugging memory allocator from the build and substitutes a stub
  memory allocator that always fails.  SQLite will not run with this
  stub memory allocator since it will be unable to allocate memory.  But
  this stub can be replaced at start-time using
  sqlite3_config(SQLITE_CONFIG_MALLOC,...) or
  sqlite3_config(SQLITE_CONFIG_HEAP,...).
  So the net effect of this compile-time option is that it allows SQLite
  to be compiled and linked against a system library that does not support
  malloc(), free(), and/or realloc().


10.  Analysis and Debugging Options

SQLITE_DEBUG
  The SQLite source code contains literally thousands of assert() statements
  used to verify internal assumptions and subroutine preconditions and
  postconditions.  These assert() statements are normally turned off
  (they generate no code) since turning them on makes SQLite run approximately
  three times slower.  But for testing and analysis, it is useful to turn
  the assert() statements on.  The SQLITE_DEBUG compile-time option does this.
  SQLITE_DEBUG also enables some other debugging features, such as
  special PRAGMA statements that turn on tracing and listing features
  used for troubleshooting and analysis of the VDBE and code generator.

SQLITE_MEMDEBUG
  The SQLITE_MEMDEBUG option causes an instrumented 
  debugging memory allocator
  to be used as the default memory allocator within SQLite.  The
  instrumented memory allocator checks for misuse of dynamically allocated
  memory.  Examples of misuse include using memory after it is freed,
  writing off the ends of a memory allocation, freeing memory not previously
  obtained from the memory allocator, or failing to initialize newly
  allocated memory.


11.  Windows-Specific Options

SQLITE_WIN32_HEAP_CREATE
  This option forces the Win32 native memory allocator, when enabled, to
  create a private heap to hold all memory allocations.

SQLITE_WIN32_MALLOC_VALIDATE
  This option forces the Win32 native memory allocator, when enabled, to
  make strategic calls into the HeapValidate() function if assert() is also
  enabled.


12. Compiler Linkage and Calling Convention Control

The following macros specify interface details
for certain kinds of SQLite builds.  The Makefiles will normally
handle setting these macros automatically.  Application developers should
not need to worry with these macros.  The following documentation about these 
macros is included for completeness.


SQLITE_API
  This macro identifies an externally visible interface for SQLite.
  This macro is sometimes set to "extern".  But the definition is
  compiler-specific.

SQLITE_APICALL
  This macro identifies the calling convention used by public interface
  routines in SQLite which accept a fixed number of arguments.
  This macro is normally defined to be nothing,
  though on Windows builds it can sometimes be set to "__cdecl" or "__stdcall".
  The "__cdecl" setting is the default, but "__stdcall" is used when SQLite
  is intended to be compiled as a Windows system library.
  
  A single function declaration should contain no more than one of
  the following:  SQLITE_APICALL, SQLITE_CDECL, or SQLITE_SYSAPI.

SQLITE_CALLBACK
  This macro specifies the calling convention used with callback pointers
  in SQLite.  This macro is normally defined to be nothing, though on Windows
  builds it can sometimes be set to "__cdecl" or "__stdcall".  The
  "__cdecl" setting is the default, but "__stdcall" is used when SQLite
  is intended to be compiled as a Windows system library.

SQLITE_CDECL
  This macro specifies the calling convention used by varargs interface
  routines in SQLite.  This macro is normally defined to be nothing,
  though on Windows builds it can sometimes be set to "__cdecl".  This
  macro is used on varargs routines and so cannot be set to "__stdcall"
  since the __stdcall calling convention does not support varargs functions.
  
  A single function declaration should contain no more than one of
  the following:  SQLITE_APICALL, SQLITE_CDECL, or SQLITE_SYSAPI.

SQLITE_EXTERN
  This macro specifies linkage for public interface variables in SQLite.
  It should normally be allowed to default to "extern".

SQLITE_STDCALL
  This macro is no longer used and is now deprecated.

SQLITE_SYSAPI
  This macro identifies the calling convention used by operating system
  interfaces for the target platform for an SQLite build.
  This macro is normally defined to be nothing,
  though on Windows builds it can sometimes be set to "__stdcall".
  
  A single function declaration should contain no more than one of
  the following:  SQLITE_APICALL, SQLITE_CDECL, or SQLITE_SYSAPI.

SQLITE_TCLAPI
  This macro specifies the calling convention used by the 
  TCL library interface routines.
  This macro is not used by the SQLite core, but only by the TCL Interface
  and TCL test suite.
  This macro is normally defined to be nothing,
  though on Windows builds it can sometimes be set to "__cdecl".  This
  macro is used on TCL library interface routines which are always compiled
  as __cdecl, even on platforms that prefer to use __stdcall, so this
  macro should not be set to __stdcall unless the platform has a custom
  TCL library build that supports __stdcall.
  
  This macro may not be used in combination with any of SQLITE_APICALL,
  SQLITE_CALLBACK, SQLITE_CDECL or SQLITE_SYSAPI.


c2: SQLite: Single File Database
c5: onefile.html
c4:
Single-file Cross-platform Database


A database in SQLite is a single disk file&sup1;.
Furthermore, the file format is cross-platform.
A database that is created on one machine can be
copied and used on a different machine with
a different architecture.  SQLite databases
are portable across 32-bit and 64-bit machines
and between 
big-endian and 
little-endian
architectures.



The SQLite database file format is also stable.
All releases of SQLite version 3 can read and write database
files created by the very first SQLite 3 release (version 3.0.0)
going back to 2004-06-18.  This is "backwards compatibility".
The developers promise to maintain backwards compatibility of
the database file format for all future releases of SQLite 3.
"Forwards compatibility" means that older releases
of SQLite can also read and write databases created by newer
releases.  SQLite is usually, but not completely forwards
compatible. 



The stability of the SQLite database file format and the fact
that the file format is cross-platform combine to make SQLite
database files an excellent choice as an 
Application File Format.
The US Library Of Congress acknowledges this by listing SQLite
as a recommended storage format for long-term preservation of
digital content.





Notes:

Temporary journal files are created as part of transaction control,
but those extra files are not part of the steady-state database.


c2: Indexes On Expressions
c5: expridx.html
c4:
Normally, an SQL index references columns of a table.  But an index
can also be formed on expressions involving table columns.


As an example, consider the following table that tracks
dollar-amount changes on various "accounts":

CREATE TABLE account_change(
  chng_id INTEGER PRIMARY KEY,
  acct_no INTEGER REFERENCES account,
  location INTEGER REFERENCES locations,
  amt INTEGER,  -- in cents
  authority TEXT,
  comment TEXT
);
CREATE INDEX acctchng_magnitude ON account_change(acct_no, abs(amt));



Each entry in the account_change table records a deposit or a withdrawal
into an account.  Deposits have a positive "amt" and withdrawals have
a negative "amt".


The acctchng_magnitude index is over the account number ("acct_no") and
on the absolute value of the amount.  This index allows one to do 
efficient queries over the magnitude of a change to the account.
For example, to list all changes to account number $xyz that are
more than $100.00, one can say:


SELECT * FROM account_change WHERE acct_no=$xyz AND abs(amt)>=10000;



Or, to list all changes to one particular account ($xyz) in order of
decreasing magnitude, one can write:

SELECT * FROM account_change WHERE acct_no=$xyz
 ORDER BY abs(amt) DESC;



Both of the above example queries would work fine without the
acctchng_magnitude index.
The acctchng_magnitude index merely helps the queries to run
faster, especially on databases where there are many entries in
the table for each account.

1. How To Use Indexes On Expressions


Use a CREATE INDEX statement to create a new index on one or more
expressions just like you would to create an index on columns.  The only
difference is that expressions are listed as the elements to be indexed
rather than column names.


The SQLite query planner will consider using an index on an expression
when the expression that is indexed appears in the WHERE clause or in
the ORDER BY clause of a query, exactly as it is written in the
CREATE INDEX statement.  The query planner does not do algebra.  In order
to match WHERE clause constraints and ORDER BY terms to indexes, SQLite
requires that the expressions be the same, except for minor syntactic
differences such as white-space changes.  So if you have:

CREATE TABLE t2(x,y,z);
CREATE INDEX t2xy ON t2(x+y);



And then you run the query:

SELECT * FROM t2 WHERE y+x=22;



Then the index will not be used because 
the expression on the CREATE INDEX
statement (x+y) is not the same as the expression as it appears in the 
query (y+x).  The two expressions might be mathematically equivalent, but
the SQLite query planner insists that they be the same, not merely
equivalent.  Consider rewriting the query thusly:

SELECT * FROM t2 WHERE x+y=22;



This second query will likely use the index because now the expression
in the WHERE clause (x+y) matches the expression in the index exactly.


2. Restrictions


There are certain reasonable restrictions on expressions that appear in
CREATE INDEX statements:



Expressions in CREATE INDEX statements
may only refer to columns of the table being indexed, not to
columns in other tables.


Expressions in CREATE INDEX statements
may contain function calls, but only to functions whose output
is always determined completely by its input parameters (a.k.a.:
deterministic functions).  Obviously, functions like random() will not
work well in an index.  But also functions like sqlite_version(), though
they are constant across any one database connection, are not constant
across the life of the underlying database file, and hence may not be
used in a CREATE INDEX statement.


Note that application-defined SQL functions are by default considered
non-deterministic and may not be used in a CREATE INDEX statement unless
the SQLITE_DETERMINISTIC flag is used when the function is registered.


Expressions in CREATE INDEX statements may not use subqueries.


Expressions may only be used in CREATE INDEX statements, not within
UNIQUE or PRIMARY KEY constraints within the CREATE TABLE statement.



3. Compatibility


The ability to index expressions was added to SQLite with 
version 3.9.0 (2015-10-14).  A database that uses an index on
expressions will not be usable by earlier versions of SQLite.


c2: Using SQLite In Multi-Threaded Applications
c5: threadsafe.html
c4:
1. Overview

SQLite supports three different threading modes:


Single-thread.
In this mode, all mutexes are disabled and SQLite is unsafe to use in
more than a single thread at once.

Multi-thread.
In this mode, SQLite can be safely used by multiple threads provided that
no single database connection is used simultaneously in two or more threads.


Serialized.
In serialized mode, SQLite can be safely used by multiple threads with no
restriction.



The threading mode can be selected at compile-time (when the SQLite
library is being compiled from source code) or at start-time (when the
application that intends to use SQLite is initializing) or at
run-time (when a new SQLite database connection is being created).
Generally speaking, run-time overrides start-time and start-time
overrides compile-time.  Except, single-thread mode cannot be 
overridden once selected.



The default mode is serialized.


2. Compile-time selection of threading mode


Use the SQLITE_THREADSAFE compile-time parameter to select the
threading mode.  If no SQLITE_THREADSAFE compile-time parameter is
present, then serialized mode is used.
This can be made explicit with 
-DSQLITE_THREADSAFE=1.
With
-DSQLITE_THREADSAFE=0 the threading mode is
single-thread.  With
-DSQLITE_THREADSAFE=2 the threading mode is
multi-thread.



The return value of the sqlite3_threadsafe() interface is the value
of SQLITE_THREADSAFE set at compile-time. It does not reflect changes
to the threading mode made at runtime via the sqlite3_config()
interface or by flags given as the third argument to sqlite3_open_v2().



If single-thread mode is selected at compile-time, then critical
mutexing logic is omitted from the build and it is impossible to
enable either multi-thread or serialized modes at start-time or
run-time.


3. Start-time selection of threading mode


Assuming that the compile-time threading mode is not single-thread, then
the threading mode can be changed during initialization using the
sqlite3_config() interface.  The SQLITE_CONFIG_SINGLETHREAD verb
puts SQLite into single-thread mode, the SQLITE_CONFIG_MULTITHREAD
verb sets multi-thread mode, and the SQLITE_CONFIG_SERIALIZED verb
sets serialized mode.


4. Run-time selection of threading mode

If single-thread mode has not been selected at compile-time or start-time,
then individual database connections can be created as either multi-thread
or serialized.  It is not possible to downgrade an individual database
connection to single-thread mode.  Nor is it possible to escalate an
individual database connection if the compile-time or start-time mode
is single-thread.

The threading mode for an individual database connection is determined
by flags given as the third argument to sqlite3_open_v2().  The
SQLITE_OPEN_NOMUTEX flag causes the database connection to be in the
multi-thread mode and the SQLITE_OPEN_FULLMUTEX flag causes the connection
to be in serialized mode.  If neither flag is specified or if
sqlite3_open() or sqlite3_open16() are used instead of 
sqlite3_open_v2(), then the default
mode determined by the compile-time and start-time settings is used.


c2: The SQLite R*Tree Module
c5: rtree.html
c4:
1. Overview


An R-Tree is a special
index that is designed for doing range queries.  R-Trees are most commonly
used in geospatial systems where each entry is a rectangle with minimum and
maximum X and Y coordinates.  Given a query rectangle, an R-Tree is able
to quickly find all entries that are contained within the query rectangle
or which overlap the query rectangle.  This idea is easily extended to
three dimensions for use in CAD systems.  R-Trees also find use in time-domain
range look-ups.  For example, suppose a database records the starting and
ending times for a large number of events.  A R-Tree is able to quickly
find all events that were active at any time during a given
time interval, or all events that started during a particular time interval,
or all events that both started and ended within a given time interval.
And so forth.



The R-Tree concept originated with 
Toni Guttman: 
R-Trees: A Dynamic Index Structure for Spatial Searching,
Proc. 1984 ACM SIGMOD International Conference on Management of Data,
pp. 47-57.
The implementation found in SQLite is a refinement of Guttman's original
idea, commonly called "R*Trees", that was described by
Norbert Beckmann, Hans-Peter Kriegel, Ralf Schneider, Bernhard Seeger:
The R*-Tree: An Efficient and Robust Access Method for Points
and Rectangles. SIGMOD Conference 1990: 322-331.


2. Compiling The R*Tree Module


The source code to the SQLite R*Tree module is included as part
of the amalgamation but is disabled by default.  To enable the
R*Tree module, simply compile with the SQLITE_ENABLE_RTREE 
C-preprocessor macro defined.  With many compilers, this is accomplished
by adding the option "-DSQLITE_ENABLE_RTREE=1" to the compiler
command-line.


3. Using the R*Tree Module


The SQLite R*Tree module is implemented as a
virtual table.  Each R*Tree index is a
virtual table with an odd number of columns between 3 and 11.
The first column is always a 64-bit signed integer primary key.
The other columns are pairs, one pair per dimension, containing the
minimum and maximum values for that dimension, respectively.
A 1-dimensional R*Tree thus has 3 columns.  
A 2-dimensional R*Tree has 5 columns.
A 3-dimensional R*Tree has 7 columns.
A 4-dimensional R*Tree has 9 columns.
And a 5-dimensional R*Tree has 11 columns.  The SQLite R*Tree implementation
does not support R*Trees wider than 5 dimensions.



The first column of an SQLite R*Tree is similar to an integer primary 
key column of a normal SQLite table. It may only store a 64-bit signed
integer value. Inserting a NULL value into this column causes SQLite
to automatically generate a new unique primary key value. If an attempt
is made to insert any other non-integer value into this column,
the r-tree module silently converts it to an integer before writing it
into the database.

The min/max-value pair columns are stored as 32-bit floating point values for
"rtree" virtual tables or as 32-bit signed integers in "rtree_i32" virtual
tables.  Unlike regular SQLite tables which can store data in a variety of
datatypes and formats, the R*Tree rigidly enforce these storage types. 
If any other type of value is inserted into such a column, the r-tree
module silently converts it to the required type before writing the
new record to the database.

3.1. Creating An R*Tree Index


A new R*Tree index is created as follows:


CREATE VIRTUAL TABLE &lt;name&gt; USING rtree(&lt;column-names&gt;);



The &lt;name&gt; is the name your application chooses for the
R*Tree index and &lt;column-names&gt; is a comma separated list
of between 3 and 11 columns.
The virtual &lt;name&gt; table creates three shadow tables to actually
store its content.  The names of these shadow tables are:


&lt;name&gt;_node
&lt;name&gt;_rowid
&lt;name&gt;_parent




The shadow tables are ordinary SQLite data tables.  You can query them 
directly if you like, though this unlikely to reveal anything particularly
useful. 
And you can UPDATE, DELETE, INSERT or even DROP 
the shadow tables, though doing so will corrupt your R*Tree index.
So it is best to simply ignore the shadow tables.  Recognize that they
hold your R*Tree index information and let it go as that.



As an example, consider creating a two-dimensional R*Tree index for use in 
spatial queries:


CREATE VIRTUAL TABLE demo_index USING rtree(
   id,              -- Integer primary key
   minX, maxX,      -- Minimum and maximum X coordinate
   minY, maxY       -- Minimum and maximum Y coordinate
);



3.1.1. Column naming details


In the argments to "rtree" in the CREATE VIRTUAL TABLE statement, the
names of the columns are taken from the first token of each argument.
All subsequent tokens within each argument are silently ignored.
This means, for example, that if you try to give a column a
type affinity or add a constraint such as UNIQUE or NOT NULL or DEFAULT to
a column, those extra tokens are accepted as valid, but they do not change
the behavior of the rtree.
In an RTREE virtual table, the first column always has a
type affinity of INTEGER and all other data columns have a
type affinity of REAL.
In an RTREE_I32 virtual table, all columns have type affinity of INTEGER.


Recommended practice is to omit any extra tokens in the rtree specification.
Let each argument to "rtree" be a single ordinary label that is the name of
the corresponding column, and omit all other tokens from the argument list.

3.2. Populating An R*Tree Index


The usual INSERT, UPDATE, and DELETE commands work on an R*Tree
index just like on regular tables.  So to insert some data into our sample
R*Tree index, we can do something like this:


INSERT INTO demo_index VALUES
  (28215, -80.781227, -80.604706, 35.208813, 35.297367),
  (28216, -80.957283, -80.840599, 35.235920, 35.367825),
  (28217, -80.960869, -80.869431, 35.133682, 35.208233),
  (28226, -80.878983, -80.778275, 35.060287, 35.154446),
  (28227, -80.745544, -80.555382, 35.130215, 35.236916),
  (28244, -80.844208, -80.841988, 35.223728, 35.225471),
  (28262, -80.809074, -80.682938, 35.276207, 35.377747),
  (28269, -80.851471, -80.735718, 35.272560, 35.407925),
  (28270, -80.794983, -80.728966, 35.059872, 35.161823),
  (28273, -80.994766, -80.875259, 35.074734, 35.172836),
  (28277, -80.876793, -80.767586, 35.001709, 35.101063),
  (28278, -81.058029, -80.956375, 35.044701, 35.223812),
  (28280, -80.844208, -80.841972, 35.225468, 35.227203),
  (28282, -80.846382, -80.844193, 35.223972, 35.225655);




The entries above are bounding boxes (longitude and latitude) for 14
zipcodes near Charlotte, NC.  A real database would have many thousands,
millions, or billions of such entries, but this small 14-row sample will
be sufficient to illustrate the ideas.


3.3. Querying An R*Tree Index


Any valid query will work against an R*Tree index.  The R*Tree
implementation just makes some kinds of queries especially
efficient.  Queries against the primary key are efficient:


SELECT * FROM demo_index WHERE id=28269;




Of course, an ordinary SQLite table will also do a query against its
integer primary key efficiently, so the previous is not important.
The big reason for using an R*Tree is so that
you can efficiently do range queries against the coordinate
ranges.  For example, the main office of the SQLite project is
located at 35.37785, -80.77470. 
To find which zipcodes might service that office, one could right:


SELECT id FROM demo_index
 WHERE minX&lt;=-80.77470 AND maxX&gt;=-80.77470
   AND minY&lt;=35.37785  AND maxY&gt;=35.37785;




The query above will quickly locate all zipcodes that contain
the SQLite main office in their bounding box, even if the
R*Tree contains many entries.  The previous is an example
of a "contained-within" query.  The R*Tree also supports "overlapping"
queries.  For example, to find all zipcode bounding boxes that overlap
with the 28269 zipcode:


SELECT A.id FROM demo_index AS A, demo_index AS B
 WHERE A.maxX&gt;=B.minX AND A.minX&lt;=B.maxX
   AND A.maxY&gt;=B.minY AND A.minY&lt;=B.maxY
   AND B.id=28269;




This second query will find both 28269 entry (since every bounding box
overlaps with itself) and also other zipcode that is close enough to
28269 that their bounding boxes overlap.



Note that it is not necessary for all coordinates in an R*Tree index
to be constrained in order for the index search to be efficient.
One might, for example, want to query all objects that overlap with
the 35th parallel:


SELECT id FROM demo_index
 WHERE maxY&gt;=35.0  AND minY&lt;=35.0;




But, generally speaking, the more constraints that the R*Tree module
has to work with, and the smaller the bounding box, the faster the
results will come back.


3.4. Roundoff Error


By default, coordinates are stored in an R*Tree using 32-bit floating
point values.  When a coordinate cannot be exactly represented by a
32-bit floating point number, the lower-bound coordinates are rounded down
and the upper-bound coordinates are rounded up.  Thus, bounding boxes might
be slightly larger than specified, but will never be any smaller.  This
is exactly what is desired for doing the more common "overlapping" queries
where the application wants to find every entry in the R*Tree that overlaps
a query bounding box.  Rounding the entry bounding boxes outward might cause a
few extra entries to appears in an overlapping query if the edge of the
entry bounding box corresponds to an edge of the query bounding box.  But
the overlapping query will never miss a valid table entry.  

However, for a "contained-within" style query, rounding the bounding
boxes outward might cause some entries to be excluded from the result set
if the edge of the entry bounding box corresponds to the edge of the query
bounding box.  To guard against this, applications should expand their
contained-within query boxes slightly (by 0.000012%) by rounding down the
lower coordinates and rounding up the top coordinates, in each dimension.



3.5. Reading And Writing At The Same Time


It is the nature of the Guttman R-Tree algorithm that any write might
radically restructure the tree, and in the process change the scan order
of the nodes.  For this reason, it is not generally possible to modify
the R-Tree in the middle of a query of the R-Tree.  Attempts to do so
will fail with a SQLITE_LOCKED "database table is locked" error.


So, for example, suppose an application runs one query against an R-Tree like
this:

SELECT id FROM demo_index
 WHERE maxY&gt;=35.0  AND minY&lt;=35.0;



Then for each "id" value returned, suppose the application creates an 
UPDATE statement like the following and binds the "id" value returned against
the "?1" parameter:

UPDATE demo_index SET maxY=maxY+0.5 WHERE id=?1;



Then the UPDATE might fail with an SQLITE_LOCKED error.  The reason is that
the initial query has not run to completion.  It is remembering its place
in the middle of a scan of the R-Tree.  So an update to the R-Tree cannot
be tolerated as this would disrupt the scan.


This is a limitation of the R-Tree extension only.  Ordinary tables in
SQLite are able to read and write at the same time.  Other virtual tables
might (or might not) also that capability.  And R-Tree can appear to read
and write at the same time in some circumstances, if it can figure out how
to reliably run the query to completion before starting the update.  But
you shouldn't count on that for every query.  Generally speaking, it is
best to avoid running queries and updates to the same R-Tree at the same
time.


If you really need to update an R-Tree based on complex queries against
the same R-Tree, it is best to run the complex queries first and store
the results in a temporary table, then update the R-Tree based on the values
stored in the temporary table.

4. Using R*Trees Effectively


For SQLite versions prior to 3.24.0 (2018-06-04),
the only information that an R*Tree index stores about an object is
its integer ID and its bounding box.  Additional information needs to
be stored in separate tables and related to the R*Tree index using
the primary key.  For the example above, one might create an auxiliary
table as follows:


CREATE TABLE demo_data(
  id INTEGER PRIMARY KEY,  -- primary key
  objname TEXT,            -- name of the object
  objtype TEXT,            -- object type
  boundary BLOB            -- detailed boundary of object
);



In this example, the demo_data.boundary field is intended to hold some
kind of binary representation of the precise boundaries of the object.
The R*Tree index only holds an axis-aligned rectangular boundary for the
object.  The R*Tree boundary is just an approximation of the true object
boundary.  So what typically happens is that the R*Tree index is used to
narrow a search down to a list of candidate objects and then more detailed
and expensive computations are done on each candidate to find if the
candidate truly meets the search criteria.



Key Point:
An R*Tree index does not normally provide the exact answer but merely
reduces the set of potential answers from millions to dozens.



Suppose the demo_data.boundary field holds some proprietary data description
of a complex two-dimensional boundary for a zipcode and suppose that the
application has used the sqlite3_create_function() interface to
created an application-defined function "contained_in(boundary,lat,long)"
that accepts the demo_data.boundary object and a latitute and longitude
and returns return true or false if the lat/long is contained within
the boundary.
One may assume that "contained_in()" is a relatively slow
functions that we do not want to invoke too frequently.
Then an efficient way to find the specific ZIP code for the main
SQLite office would be to run a query like this:



SELECT objname FROM demo_data, demo_index
 WHERE demo_data.id=demo_index.id
   AND contained_in(demo_data.boundary, 35.37785, -80.77470)
   AND minX&lt;=-80.77470 AND maxX&gt;=-80.77470
   AND minY&lt;=35.37785  AND maxY&gt;=35.37785;


Notice how the query above works:  The R*Tree index runs in the outer
loop to find entries that contain the SQLite main office in their
boundary box.
For each row found, SQLite looks up
the corresponding entry in the demo_data table.  It then uses the boundary
field from the demo_data table as a parameter to the contained_in()
function and if that function returns true, then we know the sought after
coordinate is in that ZIP code boundary.

One would get the same answer without the use of the R*Tree index
using the following simpler query:

SELECT objname FROM demo_data
 WHERE contained_in(demo_data.boundary, 35.37785, -80.77470);


The problem with this latter query is that it must apply the
contained_in() function to all entries in the demo_data table.
The use of the R*Tree in the penultimate query reduces the number of
calls to contained_in() function to a small subset of the entire table.
The R*Tree index did not find the exact answer itself, it merely
limited the search space.



4.1. Auxiliary Columns


Beginning with SQLite version 3.24.0 (2018-06-04), r-tree tables
can have auxiliary columns that store arbitrary data.
Auxiliary columns can be used in place of
secondary tables such as "demo_data".


Auxiliary columns are marked with a "+" symbol before the column name.
Auxiliary columns must come after all of the coordinate boundary columns.
An RTREE table can have no more than 100 columns total.  In other words,
the count of columns including the integer primary key column, 
the coordinate boundary columns, and all auxiliary columns must be 100 or less.
The following example shows an r-tree table with auxiliary columns that
is equivalent to the two tables "demo_index" and "demo_data" above:

CREATE VIRTUAL TABLE demo_index2 USING rtree(
   id,              -- Integer primary key
   minX, maxX,      -- Minimum and maximum X coordinate
   minY, maxY,      -- Minimum and maximum Y coordinate
   +objname TEXT,   -- name of the object
   +objtype TEXT,   -- object type
   +boundary BLOB   -- detailed boundary of object
);



By combining location data and related information into the same
table, auxiliary columns can provide a cleaner model
and reduce the need to joins.
For example, the earlier
join between demo_index and demo_data can now
be written as a simple query, like this:

SELECT objname FROM demo_index2
 WHERE contained_in(boundary, 35.37785, -80.77470)
   AND minX&lt;=-80.77470 AND maxX&gt;=-80.77470
   AND minY&lt;=35.37785  AND maxY&gt;=35.37785;


4.1.1. Limitations


For auxiliary columns, only the name of the column matters.
The type affinity is ignored.
Constraints such as NOT NULL, UNIQUE, REFERENCES, or CHECK
are also ignored.  However, future versions
of SQLite might start paying attention to the type affinity and
constraints, so users of auxiliary columns are advised to leave
both blank, to avoid future compatibility problems.



5. Integer-Valued R-Trees


The default virtual table ("rtree") stores coordinates as
single-precision (4-byte) floating point numbers.  If integer coordinates
are desired, declare the table using "rtree_i32" instead:

CREATE VIRTUAL TABLE intrtree USING rtree_i32(id,x0,x1,y0,y1,z0,z1);



An rtree_i32 stores coordinates as 32-bit signed integers.  
Even though it stores values using integer, the rtree_i32 virtual
table still uses floating point computations internally as part of
the r-tree algorithm.



6. Custom R-Tree Queries

By using standard SQL expressions in the WHERE clause of a SELECT query,
a programmer can query for all R*Tree entries that 
intersect with or are contained within a particular bounding-box.
Custom R*Tree queries, using the MATCH
operator in the WHERE clause of a SELECT, allow the programmer to query for 
the set of R*Tree entries that intersect any arbitrary region or shape, not 
just a box.  This capability is useful, for example, in computing the 
subset of objects in the R*Tree that are visible from a camera positioned 
in 3-D space.

Regions for custom R*Tree queries are defined by R*Tree geometry callbacks
implemented by the application and registered with SQLite via a call to one
of the following two APIs:

int sqlite3_rtree_query_callback(
  sqlite3 *db,
  const char *zQueryFunc,
  int (*xQueryFunc)(sqlite3_rtree_query_info*),
  void *pContext,
  void (*xDestructor)(void*)
);
int sqlite3_rtree_geometry_callback(
  sqlite3 *db,
  const char *zGeom,
  int (*xGeom)(sqlite3_rtree_geometry *, int nCoord, double *aCoord, int *pRes),
  void *pContext
);


The sqlite3_rtree_query_callback() became available with SQLite
version 3.8.5 (2014-06-04) and is the preferred interface.
The sqlite3_rtree_geometry_callback() is an older and less flexible
interface that is supported for backwards compatibility.

A call to one of the above APIs creates a new SQL function named by the
second parameter (zQueryFunc or zGeom).  When that SQL function appears
on the right-hand side of the MATCH operator and the left-hand side of the
MATCH operator is any column in the R*Tree virtual table, then the callback 
defined by the third argument (xQueryFunc or xGeom) is invoked to determine
if a particular object or subtree overlaps the desired region.

For example, a query like the following might be used to find all
R*Tree entries that overlap with a circle centered a 45.3,22.9 with a
radius of 5.0:

SELECT id FROM demo_index WHERE id MATCH circle(45.3, 22.9, 5.0)


The SQL syntax for custom queries is the same regardless of which
interface, sqlite3_rtree_geometry_callback() or sqlite3_rtree_query_callback(),
is used to register the SQL function.  However, the newer query-style
callbacks give the application greater control over how the query proceeds.

6.1. The Legacy xGeom Callback

The legacy xGeom callback is invoked with four arguments.  The first
argument is a pointer to an sqlite3_rtree_geometry structure which provides
information about how the SQL function was invoked.  The second argument
is the number of coordinates in each r-tree entry, and is always the same
for any given R*Tree.  The number of coordinates is 2 for a 1-dimensional R*Tree,
4 for a 2-dimensional R*Tree, 6 for a 3-dimensional R*Tree, and so forth.
The third argument, aCoord[], is an array of nCoord coordinates that defines
a bounding box to be tested.  The last argument is a pointer into which 
the callback result should be written.  The result is zero
if the bounding-box defined by aCoord[] is completely outside
the region defined by the xGeom callback and the result is non-zero if
the bounding-box is inside or overlaps with the xGeom region.  The xGeom
callback should normally return SQLITE_OK.  If xGeom returns anything other
than SQLITE_OK, then the r-tree query will abort with an error.

The sqlite3_rtree_geometry structure that the first argument to the
xGeom callback points to has a structure shown below.  The exact same
sqlite3_rtree_geometry
structure is used for every callback for same MATCH operator in the same
query.  The contents of the sqlite3_rtree_geometry
structure are initialized by SQLite but are
not subsequently modified.  The callback is free to make changes to the
pUser and xDelUser elements of the structure if desired.

typedef struct sqlite3_rtree_geometry sqlite3_rtree_geometry;
struct sqlite3_rtree_geometry {
  void *pContext;                 /* Copy of pContext passed to s_r_g_c() */
  int nParam;                     /* Size of array aParam */
  double *aParam;                 /* Parameters passed to SQL geom function */
  void *pUser;                    /* Callback implementation user data */
  void (*xDelUser)(void *);       /* Called by SQLite to clean up pUser */
};


The pContext member of the sqlite3_rtree_geometry
structure is always set to a copy of the pContext
argument passed to sqlite3_rtree_geometry_callback() when the
callback is registered. The aParam[] array (size nParam) contains the parameter
values passed to the SQL function on the right-hand side of the MATCH operator.
In the example "circle" query above, nParam would be set to 3 and the aParam[]
array would contain the three values 45.3, 22.9 and 5.0.

The pUser and xDelUser members of the sqlite3_rtree_geometry structure are
initially set to NULL. The pUser variable may be set by the callback
implementation to any arbitrary value that may be useful to subsequent
invocations of the callback within the same query (for example, a
pointer to a complicated data structure used to test for region intersection).
If the xDelUser variable is set to a non-NULL value, then after the
query has finished running SQLite automatically invokes it with the
value of the pUser variable as the only argument. In other words, xDelUser
may be set to a destructor function for the pUser value.

The xGeom callback always does a depth-first search of the r-tree.



6.2. The New xQueryFunc Callback

The newer xQueryFunc callback receives more information from the r-tree
query engine on each call, and it sends more information back to the query engine
before it returns.
To help keep the interface manageable, the xQueryFunc callback sends and receives
information from the query engine as fields in the
sqlite3_rtree_query_info structure:

struct sqlite3_rtree_query_info {
  void *pContext;                   /* pContext from when function registered */
  int nParam;                       /* Number of function parameters */
  sqlite3_rtree_dbl *aParam;        /* value of function parameters */
  void *pUser;                      /* callback can use this, if desired */
  void (*xDelUser)(void*);          /* function to free pUser */
  sqlite3_rtree_dbl *aCoord;        /* Coordinates of node or entry to check */
  unsigned int *anQueue;            /* Number of pending entries in the queue */
  int nCoord;                       /* Number of coordinates */
  int iLevel;                       /* Level of current node or entry */
  int mxLevel;                      /* The largest iLevel value in the tree */
  sqlite3_int64 iRowid;             /* Rowid for current entry */
  sqlite3_rtree_dbl rParentScore;   /* Score of parent node */
  int eParentWithin;                /* Visibility of parent node */
  int eWithin;                      /* OUT: Visiblity */
  sqlite3_rtree_dbl rScore;         /* OUT: Write the score here */
  /* The following fields are only available in 3.8.11 and later */
  sqlite3_value **apSqlParam;       /* Original SQL values of parameters */
};


The first five fields of the sqlite3_rtree_query_info structure are identical
to the sqlite3_rtree_geometry structure, and have exactly the same meaning.
The sqlite3_rtree_query_info structure also contains nCoord and aCoord fields 
which have the same meaning as the parameter of the same name in the xGeom callback.

The xQueryFunc must set the eWithin field of sqlite3_rtree_query_info to
one of the values NOT_WITHIN, PARTLY_WITHIN, or FULLY_WITHIN depending on whether
or not the bounding box defined by aCoord[] is completely outside the region,
overlaps the region, or is completely inside the region, respectively.  In
addition, the xQueryFunc must set the rScore field to a non-negative value that
indicates the order in which subtrees and entries of the query should be analyzed
and returned.  Smaller scores are processed first.

As its name implies, an R*Tree is organized as a tree.  Each node of the
tree is a bounding box.  The root of the tree is a bounding box that encapsulates
all elements of the tree.  Beneath the root are a number of subtrees (typically
20 or more) each with their own smaller bounding boxes and each containing some
subset of the R*Tree entries.  The subtrees may have sub-subtrees, and so forth
until finally one reaches the leaves of the tree which are the actual R*Tree
entries.

An R*Tree query is initialized by making the root node the only entry
in a priority queue sorted by rScore.
The query proceeds by extracting the entry from the priority queue that has
the lowest score.  If that entry is a leaf (meaning that it is an actual
R*Tree entry and not a subtree) then that entry
is returned as one row of the query result.  
If the extracted priority queue entry is a node (a subtree),
then the next child of that node is passed to the xQueryFunc callback.
If the node has more children then it is returned to the priority queue.
Otherwise it is discarded. Those subelements for which the xQueryFunc
callback sets eWithin to PARTLY_WITHIN or FULLY_WITHIN are added to the
priority queue using the score supplied by the callback.  Subelements that
return NOT_WITHIN are discarded.  The query runs until the priority queue is
empty.

Every leaf entry and node (subtree) within the R*Tree has an integer "level".
The leaves have a level of 0.  The first containing subtree of the leaves has
a level of 1.  The root of the R*Tree has the largest level value.  The
mxLevel entry in the sqlite3_rtree_query_info structure is the level value for
the root of the R*Tree.  The iLevel entry in sqlite3_rtree_query_info gives the
level for the object being interrogated.

Most R*Tree queries use a depth-first search.  This is accomplished by setting
the rScore equal to iLevel.  A depth-first search is usually preferred since it
minimizes the number of elements in the priority queue, which reduces memory
requirements and speeds processing.  However, some application may prefer a
breadth-first search, which can be accomplished by setting rScore to mxLevel-iLevel.
By creating more complex formulas for rScore, applications can exercise
detailed control over the order in which subtree are searched and leaf
R*Tree entries are returned.  For example, in an application with many
millions of R*Tree entries, the rScore might be arranged so that the
largest or most significant entries are returned first, allowing the
application to display the most important information quickly, and
filling in smaller and less important details as they become available.

Other information fields of the sqlite3_rtree_query_info structure are
available for use by the xQueryFunc callback, if desired.  The iRowid field
is the rowid (the first of the 3 to 11 columns in the R*Tree) for the element
being considered.  iRowid is only valid for leaves.  The eParentWithin and
rParentScore values are copies of the eWithin and rScore values from the
containing subtree of the current row.  The anQueue field is an array
of mxLevel+1 unsigned integers that tell the current number of elements in
the priority queue at each level.

6.3. Additional Considerations for Custom Queries


The MATCH operator of a custom R*Tree query function must be a top-level
AND-connected term of the WHERE clause, or else it will not be usable
by the R*Tree query optimizer and the query will not be runnable.
If the MATCH operator is connected to other terms of the WHERE clause 
via an OR operator, for example, the query will fail with an error.


Two or more MATCH operators are allowed in the same WHERE clause, as long
as they are connected by AND operators.  However,
the R*Tree query engine only contains a single priority queue.  The priority
assigned to each node in the search is the lowest priority returned by any
of the MATCH operators.

7. Implementation Details


The following sections describe some low-level details of the R*Tree implementation,
that might be useful for trouble-shooting or performance analysis.



7.1. Shadow Tables

The content of an R*Tree index is actually stored in three ordinary
SQLite tables with names derived from the name of the R*Tree.  These
three tables are called "shadow tables".  This is their schema:

CREATE TABLE %_node(nodeno INTEGER PRIMARY KEY, data)
CREATE TABLE %_parent(nodeno INTEGER PRIMARY KEY, parentnode)
CREATE TABLE %_rowid(rowid INTEGER PRIMARY KEY, nodeno)


The "%" in the name of each shadow table is replaced by the name of the
R*Tree virtual table.  So, if the name of the R*Tree table is "xyz" then
the three shadow tables would be "xyz_node", "xyz_parent", and "xyz_rowid".

There is one entry in the %_node table for each R*Tree node.  An
R*Tree node consists of one or more entries that are proximate to one another.
The nodes of an R*Tree for a tree.  All nodes other than the root have an
entry in the %_parent shadow table that identifies the parent node.
Each entry in an R*Tree has a rowid.  The %_rowid shadow table maps entry
rowids to the node that contains that entry.

Extra columns appended to the %_rowid table hold the 
content of auxiliary columns.  The names of these extra
%_rowid columns are probably not the same as the
actual auxiliary column names.



7.2. Integrity Check using the rtreecheck() SQL function

The scalar SQL function rtreecheck(R) or rtreecheck(S,R) runs an
integrity check on the rtree table named R contained within database S.
The function returns a human-language description of any problems found,
or the string 'ok' if everything is ok.  Running rtreecheck() on an R*Tree
virtual table is similar to running PRAGMA integrity_check on a
database.

Example: To verify that an R*Tree named "demo_index" is well-formed
and internally consistent, run:

SELECT rtreecheck('demo_index');


The rtreecheck() function performs the following checks:



For each cell in the r-tree structure (%_node table), that:

 for each dimension, (coord1 &lt;= coord2).
 unless the cell is on the root node, that the cell is bounded
        by the parent cell on the parent node.
 for leaf nodes, that there is an entry in the %_rowid 
        table corresponding to the cell's rowid value that 
        points to the correct node.
 for cells on non-leaf nodes, that there is an entry in the 
        %_parent table mapping from the cell's child node to the
        node that it resides on.


That there are the same number of entries in the %_rowid table
as there are leaf cells in the r-tree structure, and that there
is a leaf cell that corresponds to each entry in the %_rowid table.

That there are the same number of entries in the %_parent table
as there are non-leaf cells in the r-tree structure, and that 
there is a non-leaf cell that corresponds to each entry in the 
%_parent table.


c2: How To Corrupt An SQLite Database File
c5: howtocorrupt.html
c4:
Overview 

An SQLite database is highly resistant to corruption.
If an application crash, or an operating-system crash, or even
a power failure occurs in the middle of a transaction, the partially
written transaction should be automatically rolled back the next time
the database file is accessed.  The recovery process is fully
automatic and does not require any action on the part of the user
or the application.


Though SQLite is resistant to database corruption, it is not immune.
This document describes the various ways that an SQLite database might
go corrupt.

1.  File overwrite by a rogue thread or process

SQLite database files are ordinary disk files.
That means that any process can open the file and 
overwrite it with garbage.  There is nothing that the SQLite
library can do to defend against this.



1.1.  Continuing to use a file descriptor after it has been closed

We have seen multiple cases where a file descriptor was open on a file,
then that file descriptor was closed and reopened on an SQLite database.
Later, some other thread continued to write into the
old file descriptor, not realizing that the original file had been closed
already.  But because the file descriptor had been reopened by SQLite,
the information that was intended to go into the original file ended up
overwriting parts of the SQLite database, leading to corruption of the
database.

One example of this occurred circa 2013-08-30 on the canonical repository
for the Fossil DVCS.  In that event,
file descriptor 2 (standard error) was being erroneously closed (by
stunnel, we suspect) prior to 
sqlite3_open_v2() so that the file descriptor used for the
repository database file was 2.  Later, an application 
bug caused an assert() statement to emit
an error message by invoking write(2,...).  But since file descriptor 2 was 
now connected to a database file, the error message
overwrote part of the database.  To guard against this kind of problem,
SQLite version 3.8.1 (2013-10-17)
and later refuse to use low-numbered file descriptors
for database files. 
(See SQLITE_MINIMUM_FILE_DESCRIPTOR for additional information.)

Another example of corruption caused by using a closed file
descriptor was 
reported by facebook engineers in a blog post on 2014-08-12.

Another example of this error was reported against
Fossil on 2019-07-11.  A file descriptor would
be opened for debugging output, but then closed and reopened by SQLite.
But the debugging logic continued to write into the original file
descriptor. See the
forum discussion
for the bug report and a link to the fix.

1.2.  Backup or restore while a transaction is active

Systems that run automatic backups in the background might try to
make a backup copy of an SQLite database file while it is in the middle
of a transaction.  The backup copy then might contain some old and some
new content, and thus be corrupt.

The best approach to make reliable backup copies of an SQLite database
is to make use of the backup API that is part of the SQLite library.
Failing that, it is safe to make a copy of an SQLite database file as long
as there are no transactions in progress by any process.  If the previous
transaction failed, then it is important that any rollback journal
(the *-journal file) or write-ahead log (the *-wal file)
be copied together with the database file itself.



1.3.  Deleting a hot journal

SQLite normally stores all content in a single disk file.  However,
while performing a transaction, information necessary to recover the
database following a crash or power failure is stored in auxiliary
journal files.  Such journal files are described as "hot".
The journal files have the same name as the
original database file with the addition
of -journal or -wal suffix.

SQLite must see the journal files in order to recover from a crash
or power failure.  If the hot journal files are moved, deleted, or renamed
after a crash or power failure, then automatic recovery will not work
and the database may go corrupt.

Another manifestation of this problem is
database corruption caused by inconsistent use of 8+3 filenames.



1.4.  Mispairing database files and hot journals 

 The previous example is a specific case of a more general problem:
The state of an SQLite database is controlled by both the
database file and the journal file.  In a quiescent state, the journal
file does not exist and only the database file matters.
But if the journal file does exist, it must be
kept together with the database to avoid corruption.  The following
actions are all likely to lead to corruption:

 Swapping journal files between two different databases.
 Overwritting a journal file with a different journal file.
 Moving a journal file from one database to another.
 Copying a database file without also copying its journal.
 Overwriting a database file with another without also
     deleting any hot journal associated with the original database.


2.  File locking problems

SQLite uses file locks on the database file, and on the 
write-ahead log or WAL file, to coordinate access between concurrent
processes.  Without coordination, two threads or processes might try
to make incompatible changes to a database file at the same time,
resulting in database corruption.

2.1.  Filesystems with broken or missing lock implementations

SQLite depends on the underlying filesystem to do locking as the
documentation says it will.  But some filesystems contain bugs in their
locking logic such that the locks do not always behave as advertised.
This is especially true of network filesystems and NFS in particular.
If SQLite is used on a filesystem where the locking primitives contain
bugs, and if two or more threads or processes try to access the same
database at the same time, then database corruption might result.



2.2.  Posix advisory locks canceled by a separate thread doing close()

The default locking mechanism used by SQLite on unix platforms is
POSIX advisory locking.  Unfortunately, POSIX advisory locking has design
quirks that make it prone to misuse and failure. In particular, any
thread in the same process with a file descriptor that is holding a POSIX
advisory lock can override that lock using a different file descriptor.
One particularly pernicious problem is that the close() system
call will cancel all POSIX advisory locks on the same file for all
threads and all file descriptors in the process.

So, for example, suppose a multi-thread process has
two or more threads with separate SQLite database connections to the
same database file.  Then a third thread comes along and wants to read
something out of that same database file on its own, without using the 
SQLite library.
The third thread does an open(), a read() and then
a close().
One would think this would be harmless.
But the close() system call caused the
locks held on the database by all the other threads to be dropped.  Those
other threads have no way of knowing that their locks have just been
trashed (POSIX does not provide any mechanism to determine this) and so
they keep on running under the assumption that their locks are still valid.
This can lead to two or more threads or processes trying to write to the
database at the same time, resulting in database corruption.

Note that it is perfectly safe for two or more threads to access the
same SQLite database file using the SQLite library.  The unix drivers for
SQLite know about the POSIX advisory locking quirks and work around them.
This problem only arises when a thread tries to bypass the SQLite library
and read the database file directly.

2.2.1. Multiple copies of SQLite linked into the same application

As pointed out in the previous paragraph, SQLite takes steps to work
around the quirks of POSIX advisory locking.  Part of that work-around involves
keeping a global list (mutex protected) of open SQLite database files.
But, if multiple copies of SQLite are linked into the same application,
then there will be multiple instances of this global list.
Database connections opened using one copy of the SQLite library
will be unaware of database connections opened using the other copy,
and will be unable to work around the POSIX advisory locking quirks.
A close() operation on one connection might unknowingly 
clear the locks on a different database connection, leading to database
corruption.

The scenario above sounds far-fetched.
But the SQLite developers are aware of at 
least one commercial product that was released
with exactly this bug.  The vendor came to the SQLite developers seeking
help in tracking down some infrequent database corruption issues they were
seeing on Linux and Mac.  The problem was eventually traced to the
fact that the application was linking against two separate copies of SQLite.
The solution was to change the application build procedures to link against
just one copy of SQLite instead of two.

2.3.  Two processes using different locking protocols

The default locking mechanism used by SQLite on unix platforms is
POSIX advisory locking, but there are other options.  By selecting an
alternative sqlite3_vfs using the sqlite3_open_v2() interface, an
application can make use of other locking protocols that might be more
appropriate to certain filesystems.  For example, dot-file locking might
be select for use in an application that has to run on an NFS filesystem
that does not support POSIX advisory locking.

It is important that all connections to the same database file use 
the same locking protocol.
If one application is using POSIX advisory locks and another application
is using dot-file locking, then the two applications will not see each
other's locks and will not be able to coordinate database access, possibly
leading to database corruption.



2.4.  Unlinking or renaming a database file while in use

If two processes have open connections to the same database file and
one process closes its connection, unlinks the file, then creates a new
database file in its place with the same name and reopens the new file,
then the two processes will be talking to different database files with 
the same name.  (Note that this is only possible on Posix and Posix-like
systems that permit a file to be unlinked while it is still open for
reading and writing.  Windows does not allow this to occur.)
Since rollback journals and WAL files are based on the name of the database
file, the two different database files will share the same rollback
journal or WAL file.  A rollback or recovery for one of the databases
might use content from the other database, resulting in corruption.
A similar problem occurs if a database file is renamed while it is
opened and a new file is created with the old name.

In other words, unlinking or renaming an open database file 
results in behavior that is undefined and probably undesirable.

Beginning with SQLite version 3.7.17 (2013-05-20),
the unix OS interface will
send SQLITE_WARNING messages to the error log if a database file is unlinked
while it is still in use.



2.5.  Multiple links to the same file

If a single database file has multiple links (either hard or soft links)
then that is just another way of saying that the file has multiple names.
If two or more processes open the database using different names, then
they will use different rollback journals and WAL files.  That means that
if one process crashes, the other process will be unable to recover the
transaction in progress because it will be looking in the wrong place
for the appropriate journal.

In other words, opening and using a database file that has two or
more names results in behavior that is undefined and probably undesirable.

Beginning with SQLite version 3.7.17 (2013-05-20),
the unix OS interface will
send SQLITE_WARNING messages to the error log if a database file has 
multiple hard links.

Beginning with SQLite version 3.10.0 (2016-01-06), 
the unix OS interface will
attempt to resolve symbolic links and open the database file by its
canonical name.  Prior to version 3.10.0, opening a database file 
through a symbolic link was similar to opening a database file
that had multiple hard links and resulted in undefined behavior.



2.6.  Carrying an open database connection across a fork() 

 Do not open an SQLite database connection, then fork(), then try
to use that database connection in the child process.  All kinds of
locking problems will result and you can easily end up with a corrupt
database.  SQLite is not designed to support that kind of behavior.
Any database connection that is used in a child process must be opened
in the child process, not inherited from the parent.

 Do not even call sqlite3_close() on a database connection from a
child process if the connection was opened in the parent.  It is safe
to close the underlying file descriptor, but the sqlite3_close()
interface might invoke cleanup activities that will delete content out
from under the parent, leading to errors and perhaps even database
corruption.

3.  Failure to sync

In order to guarantee that database files are always consistent, SQLite
will occasionally ask the operating system to flush all pending writes to
persistent storage then wait for that flush to complete.  This is 
accomplished using the fsync() system call under unix and
FlushFileBuffers() under Windows.  We call this flush of
pending writes a "sync".

Actually, if one is only concerned with atomic and consistent writes and
is willing to forego durable writes, the sync operation does not need
to wait until the content is completely stored on persistent media.  Instead,
the sync operation can be thought of as an I/O barrier.  As long as all
writes that occur before the sync are completed before any write that happens
after the sync, no database corruption will occur.  If sync is operating as
an I/O barrier and not as a true sync, then a power failure or system crash
might cause one or more previously committed transactions to roll back
(in violation of the "durable" property of "ACID") but the database will at
least continue to be consistent, and that is what most people care about.

3.1.  Disk drives that do not honor sync requests

Unfortunately, most consumer-grade mass storage devices lie about
syncing.  Disk drives will report that content is safely on persistent
media as soon as it reaches the track buffer and before actually being
written to oxide.  This makes the disk drives seem to operate faster
(which is vitally important to the manufacturer so that they can show
good benchmark numbers in trade magazines).  And in fairness, the lie
normally causes no harm, as long as there is no power loss or hard reset
prior to the track buffer actually being written to oxide.  But if a
power loss or hard reset does occur, and if that results in content that
was written after a sync reaching oxide while content written before
the sync is still in a track buffer, then database corruption can occur.

USB flash memory sticks seem to be especially pernicious liars 
regarding sync requests.  One can easily see this by committing a large
transaction to an SQLite database on a USB memory stick.  The COMMIT
command will return relatively quickly, indicating that the memory stick
has told the operating system and the operating system has told SQLite that
all content is safely in persistent storage, and yet the LED on the end
of the memory stick will continue flashing for several more seconds. 
Pulling out the memory stick while the LED is still flashing will frequently
result in database corruption.

Note that SQLite must believe whatever the operating system and hardware
tell it about the status of sync requests.  There is no way for SQLite to
detect that either is lying and that writes might be occurring out-of-order.
However, SQLite in WAL mode is far more forgiving of
out-of-order writes than in the default rollback journal modes.  In WAL
mode, the only time that a failed sync operation can cause database corruption
is during a checkpoint operation.  A sync failure during a COMMIT might
result in loss of durability but not in a corrupt database file.  Hence,
one line of defense against database corruption due to failed sync operations
is to use SQLite in WAL mode and to checkpoint as infrequently as possible.

3.2.  Disabling sync using PRAGMAs

The sync operations that SQLite performs to help ensure integrity
can be disabled at run-time using the synchronous pragma.  By setting
PRAGMA synchronous=OFF, all sync operations are omitted.  This makes
SQLite seem to run faster, but it also allows the operating system to freely
reorder writes, which could result in database corruption if a power failure
or hard reset occurs prior to all content reaching persistent storage.

For maximum reliability and for robustness against database corruption,
SQLite should always be run with its default synchronous setting of FULL.



4.  Disk Drive and Flash Memory Failures

An SQLite database can become corrupt if the file content changes 
due to a disk drive or flash memory failure.  It is very rare, but disks 
will occasionally flip a bit in the middle of a sector.

4.1.  Non-powersafe flash memory controllers

We are told that in some flash memory controllers the wear-leveling logic
can cause random filesystem damage if power is interrupted during a write.
This can manifest, for example, as random changes in the middle of a file
that was not even open at the time of the power loss.  So, for example,
a device would be writing content into an MP3 file in flash memory when a
power loss occurs, and that could result in an SQLite database being
corrupted even though the database was not even in use at the time of the
power loss.



4.2.  Fake capacity USB sticks

There are many fraudulent USB sticks in circulation that report to have
a high capacity (ex: 8GB) but are really only capable of storing a much
smaller amount (ex: 1GB).   Attempts to write on these devices will
often result in unrelated files being overwritten.  Any use of a fraudulent
flash memory device can easily lead to database corruption, therefore.
Internet searches such as "fake capacity usb" will turn up lots of
disturbing information about this problem.

5.  Memory corruption

SQLite is a C-library that runs in the same address space as the 
application that it serves.  That means that stray pointers, buffer
overruns, heap corruption, or other malfunctions in the application can
corrupt internal SQLite data structure and ultimately result in a
corrupt database file.  Normally these kinds of problems manifest themselves
as segfaults prior to any database corruption occurring, but there have
been instances where application code errors have caused SQLite to
malfunction subtly so as to corrupt the database file rather than
panicking.

The memory corruption problem becomes more acute when
using memory-mapped I/O.
When all or part of the database file is mapped into the application's
address space, then a stray pointer that overwrites any part of that
mapped space will immediately corrupt the database file, without
requiring the application to do a subsequent write() system call.

6.  Other operating system problems

Sometimes operating systems will exhibit non-standard behavior which
can lead to problems.  Sometimes this non-standard behavior is deliberate,
and sometimes it is a mistake in the implementation.  But in any event,
if the operating performs differently from they way SQLite expects it to
perform, the possibility of database corruption exists.

6.1.  Linux Threads

Some older versions of Linux used the LinuxThreads library for thread
support.  LinuxThreads is similar to Pthreads, but is subtly different
with respect to handling of POSIX advisory locks.  SQLite versions
2.2.3 through 3.6.23 recognized that LinuxThreads were being used at
runtime and took appropriate action to work around the non-standard
behavior of LinuxThreads.  But most modern Linux implementations make
use of the newer, and correct, NPTL implementation of Pthreads.  Beginning
with SQLite version 3.7.0 (2010-07-21), the use of NPTL is assumed.
No checks are 
made.  Hence, recent versions of SQLite will subtly malfunction and may 
corrupt database files if used in multi-threaded application that run
on older linux systems that make use of LinuxThreads.

6.2.  Failures of mmap() on QNX

There exists some subtle problem with mmap() on QNX such that making
a second mmap() call against a single file descriptor can cause
the memory obtained from the first mmap() call to be zeroed.  SQLite on
unix uses mmap() to create a shared memory region for transaction 
coordination in WAL mode, and it will call mmap() multiple times
for large transactions.  The QNX mmap() has been demonstrated to corrupt
database file under that scenario.  QNX engineers are aware of this problem
and are working on a solution; the problem may have already been fixed by
the time you read this.

When running on QNX, it is recommended that memory-mapped I/O never
be used.  Furthermore, to use WAL mode, it is recommended that applications
employ the exclusive locking mode in order to 
use WAL without shared memory.




6.3.  Filesystem Corruption

Since SQLite databases are ordinary disk files, any malfunction in the
filesystem can corrupt the database.  Filesystems in modern operating systems
are very reliable, but errors do still occur.  For example, on 2013-10-01
the SQLite database that holds the
Wiki for Tcl/Tk went corrupt a few days
after the host computer was moved to a dodgy build of the (linux) kernel
that had issues in the filesystem layer.  In that event, the filesystem
eventually became so badly corrupted that the machine was unusable, but
the earliest symptom of trouble was the corrupted SQLite database.



7. SQLite Configuration Errors

SQLite has many built-in protections against database corruption.
But many of these protections can be disabled by configuration options.
If protections are disabled, database corruption may occur.

The following are examples of disabling the built-in protection
mechanisms of SQLite:


Setting PRAGMA synchronous=OFF can cause the database to
go corrupt if there is an operating-system crash or power failure,
though this setting is safe from damage due to application crashes.

Changing the PRAGMA schema_version while other database
connections are open.

Using PRAGMA journal_mode=OFF or PRAGMA journal_mode=MEMORY
and taking an application crash in the middle of a write transaction.

Setting PRAGMA writable_schema=ON and then changing the
database schema using DML statements can render the database completely
unreadable, if not done carefully.


8.  Bugs in SQLite

SQLite is very carefully tested to help ensure that it is
as bug-free as possible.  Among the many tests that are carried out for
every SQLite version are tests that simulate power failures, I/O errors,
and out-of-memory (OOM) errors and verify that no database corruption occurs
during any of these events.  SQLite is also field-proven with approximately
two billion active deployments with no serious problems.

Nevertheless, no software is 100% perfect.  There have been a few
historical bugs in SQLite (now fixed) that could cause database corruption.
And there may be yet a few more that remain undiscovered.  Because of the
extensive testing and widespread use of SQLite, bugs that result in
database corruption tend to be very obscure.  The likelihood
of an application encountering an SQLite bug is small.  To illustrate this,
an account is given below 
of all database-corruption bugs found in SQLite during the
four-year period from 2009-04-01 to 2013-04-15.
This account should give the reader an intuitive sense of the
kinds of bugs in SQLite that manage to slip through testing procedures
and make it into a release.


8.1.  False corruption reports due to database shrinkage

If a database is written by SQLite version 3.7.0 or later and then
written again by SQLite version 3.6.23 or earlier in such a way as to
make the size of the database file decrease, then the next time that
SQLite version 3.7.0 access the database file, it might report that the
database file is corrupt.  The database file is not really corrupt, however.
Version 3.7.0 was simply being overly zealous in its corruption detection.

The problem was fixed on 2011-02-20.  The fix first appears in
SQLite version 3.7.6 (2011-04-12).

8.2.  Corruption following switches between rollback and WAL modes

Repeatedly switching an SQLite database in and out of WAL mode
and running the VACUUM command in between switches, in one process or
thread, can cause another process or thread that has the database file
open to miss the fact that the database has changed.  That second process
or thread might then try to modify the database using a stale cache and
cause database corruption.

This problem was discovered during internal testing and has never been
observed in the wild.  The problem was fixed on 2011-01-27 and in version
3.7.5.

8.3.  I/O error while obtaining a lock leads to corruption

If the operating system returns an I/O error while attempting to obtain
a certain lock on shared memory in WAL mode then SQLite might fail 
to reset its cache,
which could lead to database corruption if subsequent writes are attempted.

Note that this problem only occurs if the attempt to acquire the lock
resulted in an I/O error.  If the lock is simply not granted (because some
other thread or process is already holding a conflicting lock) then no
corruption will ever occur.  We are not aware of any operating systems that
will fail with an I/O error while attempting to get a file lock on shared
memory.  So this is a theoretical problem rather than a real problem.
Needless to say, this problem has never been observed in the wild.  The
problem was discovered while doing stress testing of SQLite in a test
harness that simulates I/O errors.

This problem was fixed on 2010-09-20 for SQLite version 3.7.3.

8.4.  Database pages leak from the free page list

When content is deleted from an SQLite database, pages that are no
longer used are added to a free list and are reused to hold content
added by subsequent inserts.  A bug in SQLite that was present in
version 3.6.16 through 3.7.2 might cause pages to go missing out of
the free list when incremental_vacuum was used.  This would not cause
data loss.  But it would result in the database file being larger than
necessary.  And it would cause the integrity_check pragma to report
pages missing from the free list.

This problem was fixed on 2010-08-23 for SQLite version 3.7.2.

8.5.  Corruption following alternating writes from 3.6 and 3.7.

SQLite version 3.7.0 introduced a number of new enhancements to
the SQLite database file format (such as but not limited to WAL).
The 3.7.0 release was a shake-out release for these new features. 
We expected to find problems and were not disappointed.

If a database were originally created using SQLite version 3.7.0,
then written by SQLite version 3.6.23.1 such that the size of the database
file increased, then written again by SQLite version 3.7.0, the database
file could go corrupt.

This problem was fixed on 2010-08-04 for SQLite version 3.7.1.

8.6.  Race condition in recovery on windows system.

SQLite version 3.7.16.2 fixes a subtle race condition in the locking
logic on Windows systems.  When a database file is in need
of recovery because the previous process writing to it crashed in the
middle of a transaction and two or more processes try to open the 
that database at the same time, then the race condition might cause
one of those processes to get a false indication that the recovery 
has already completed, allowing that process to continue using the
database file without running recovery first.  If that process writes
to the file, then the file might go corrupt.  This race condition
had apparently existed in all prior versions of SQLite for Windows going
back to 2004.  But the race was very tight.  Practically speaking, you
need a fast multi-core machine in which you launch two processes to run
recovery at the same moment on two separate cores.  This defect was
on Windows systems only and did not affect the posix OS interface.



8.7.  Boundary value error in the secondary journals used by nested transactions.

When a nested transaction is started using SAVEPOINT, SQLite uses
a secondary rollback journal to track the changes for the nested
transaction, in case the inner transaction needs to be rolled back.  Secondary
journals are not involved in protecting the database from corruption due
to program crashes or power outages.  The secondary journals only come into
play when rolling back an inner transaction of a nested transaction.

These secondary journals can be held either in memory or as temporary
files on disk.  The default behavior is to store them on disk.  But that 
can be changed using the -DSQLITE_TEMP_STORE compile-time option,
or at run-time using the PRAGMA temp_store statement.  The bug
only arises when secondary journals are held in memory.

In SQLite version 3.35.0 (2021-03-12), a new optimization was
added so that when SQLite is holding secondary journals in memory,
less memory will be used. Unfortunately, an boundary check in
the new logic was coded incorrectly.
What should have been a "&lt;" operator was coded as "&lt;=".  This
error might cause the secondary journal to enter an inconsistent state
if it is ever rolled back.  If additional changes are made and the
outer transaction eventually commits, the database might be left in
an inconsistent state.

This problem was discovered by an 
independent researcher
who was attempting to find bugs in SQLite using a fuzzer.  The fuzzer found a
failure in an assert() statement that is used
to help verify the internal state of the secondary journal.  The bug was a
sufficiently obscure corner-case that it might have gone unnoticed for many
years, had it not been for the intensive use of assert() statements in SQLite,
the persistence and tenacity of the security researchers, and their
customized state-of-the-art fuzzer.

This problem was fixed
in version 3.37.2 (2022-01-06).


