# page_content

c2: Compile-time Options
c3: 8. Options To Disable Features Normally Turned On
c5: compile.html#_options_to_disable_features_normally_turned_on
c4:
SQLITE_DISABLE_LFS
  If this C-preprocessor macro is defined, large file support
  is disabled.

SQLITE_DISABLE_DIRSYNC
  If this C-preprocessor macro is defined, directory syncs
  are disabled.  SQLite typically attempts to sync the parent
  directory when a file is deleted to ensure the directory
  entries are updated immediately on disk.

SQLITE_DISABLE_FTS3_UNICODE
  If this C-preprocessor macro is defined, the unicode61 tokenizer
  in FTS3 is omitted from the build and is unavailable to
  applications.

SQLITE_DISABLE_FTS4_DEFERRED
  If this C-preprocessor macro disables the "deferred token" optimization
  in FTS4.  The "deferred token" optimization avoids loading massive
  posting lists for terms that are in most documents of the collection
  and instead simply scans for those tokens in the document source.  FTS4
  should get exactly the same answer both with and without this optimization.

SQLITE_DISABLE_INTRINSIC
  This option disables the use of compiler-specific built-in functions
  such as __builtin_bswap32() and __builtin_add_overflow() in GCC and Clang,
  or _byteswap_ulong() and _ReadWriteBarrier() with MSVC.

SQLITE_DISABLE_PAGECACHE_OVERFLOW_STATS
  This option disables the collection of the sqlite3_status() 
  SQLITE_STATUS_PAGECACHE_OVERFLOW and SQLITE_STATUS_PAGECACHE_SIZE
  statistics.  Setting this option has been shown to increase performance in
  high concurrency multi-threaded applications.


c2: Compile-time Options
c3: 9. Options To Omit Features
c5: compile.html#_options_to_omit_features
c4:
The following options can be used to
reduce the size of the compiled library
by omitting unused features. This is probably only useful
in embedded systems where space is especially tight, as even with all
features included the SQLite library is relatively small. Don't forget
to tell your compiler to optimize for binary size! (the -Os option if
using GCC).  Telling your compiler to optimize for size usually has
a much larger impact on library footprint than employing any of these
compile-time options.  You should also verify that
debugging options are disabled.

The macros in this section do not require values. The following
compilation switches all have the same effect:
-DSQLITE_OMIT_ALTERTABLE
-DSQLITE_OMIT_ALTERTABLE=1
-DSQLITE_OMIT_ALTERTABLE=0


If any of these options are defined, then the same set of SQLITE_OMIT_*
options must also be defined when using the Lemon parser generator
tool to generate the
parse.c file and when compiling the 'mkkeywordhash' tool which generates
the keywordhash.h file.
Because of this, these options may only be used when the library is built
from canonical source, not from the amalgamation.
Some SQLITE_OMIT_* options might work, or appear to work, when used with
the amalgamation.  But this is not guaranteed.  In general, always compile
from canonical sources in order to take advantage of SQLITE_OMIT_* options.



Important Note: The SQLITE_OMIT_* options may not work with the
amalgamation.  SQLITE_OMIT_* compile-time
options usually work correctly only when SQLite is built from canonical
source files.




Special versions of the SQLite amalgamation that do work with a
predetermined set of SQLITE_OMIT_* options can be generated.  To do so,
make a copy of the Makefile.linux-gcc makefile template in the canonical
source code distribution.  Change the name of your copy to simply "Makefile".
Then edit "Makefile" to set up appropriate compile-time options.  Then
type:
make clean; make sqlite3.c

The resulting "sqlite3.c" amalgamation code file (and its associated
header file "sqlite3.h") can then be moved to a non-unix platform
for final compilation using a native compiler.

The SQLITE_OMIT_* options are unsupported.  By this we mean that
an SQLITE_OMIT_* option that omits code from the build in the current
release might become a no-op in the next release.  Or the other way around:
an SQLITE_OMIT_* that is a no-op in the current release might cause code
to be excluded in the next release.  Also, not all SQLITE_OMIT_* options
are tested.  Some SQLITE_OMIT_* options might cause SQLite to malfunction
and/or provide incorrect answers.


Important Note:
The SQLITE_OMIT_* compile-time options are mostly unsupported.


The following are the available OMIT options:


SQLITE_OMIT_ALTERTABLE
  When this option is defined, the
  ALTER TABLE command is not included in the
  library. Executing an ALTER TABLE statement causes a parse error.

SQLITE_OMIT_ANALYZE
  When this option is defined, the ANALYZE command is omitted from
  the build.

SQLITE_OMIT_ATTACH
  When this option is defined, the ATTACH and DETACH commands are
  omitted from the build.

SQLITE_OMIT_AUTHORIZATION
  Defining this option omits the authorization callback feature from the
  library. The sqlite3_set_authorizer() API function is not present
  in the library.

SQLITE_OMIT_AUTOINCREMENT
  This option is omits the AUTOINCREMENT feature.
  When this is macro is defined, columns declared as
  "INTEGER PRIMARY KEY AUTOINCREMENT"
  behave in the same way as columns declared as "INTEGER PRIMARY KEY" when a
  NULL is inserted. The sqlite_sequence system table is neither created, nor
  respected if it already exists.

SQLITE_OMIT_AUTOINIT
  For backwards compatibility with older versions of SQLite that lack
  the sqlite3_initialize() interface, the sqlite3_initialize() interface
  is called automatically upon entry to certain key interfaces such as
  sqlite3_open(), sqlite3_vfs_register(), and sqlite3_mprintf().
  The overhead of invoking sqlite3_initialize() automatically in this
  way may be omitted by building SQLite with the SQLITE_OMIT_AUTOINIT
  C-preprocessor macro.  When built using SQLITE_OMIT_AUTOINIT, SQLite
  will not automatically initialize itself and the application is required
  to invoke sqlite3_initialize() directly prior to beginning use of the
  SQLite library.

SQLITE_OMIT_AUTOMATIC_INDEX
  This option is used to omit the
  automatic indexing functionality.
  See also: SQLITE_DEFAULT_AUTOMATIC_INDEX.

SQLITE_OMIT_AUTORESET
  By default, the sqlite3_step() interface will automatically invoke
  sqlite3_reset() to reset the prepared statement if necessary.  This
  compile-time option changes that behavior so that sqlite3_step() will
  return SQLITE_MISUSE if it called again after returning anything other
  than SQLITE_ROW, SQLITE_BUSY, or SQLITE_LOCKED unless there was an
  intervening call to sqlite3_reset().

  In SQLite version 3.6.23.1 (2010-03-26)
  and earlier, sqlite3_step() used to always
  return SQLITE_MISUSE if it was invoked again after returning anything
  other than SQLITE_ROW without an intervening call to sqlite3_reset().
  This caused problems on some poorly written smartphone applications which
  did not correctly handle the SQLITE_LOCKED and SQLITE_BUSY error
  returns.  Rather than fix the many defective smartphone applications,
  the behavior of SQLite was changed in 3.6.23.2 to automatically reset
  the prepared statement.  But that changed caused issues in other
  improperly implemented applications that were actually looking
  for an SQLITE_MISUSE return to terminate their query loops.  (Anytime
  an application gets an SQLITE_MISUSE error code from SQLite, that means the
  application is misusing the SQLite interface and is thus incorrectly
  implemented.)  The SQLITE_OMIT_AUTORESET interface was added to SQLite
  version 3.7.5 (2011-02-01) in an effort to get all of the (broken)
  applications to work again without having to actually fix the applications.

SQLITE_OMIT_AUTOVACUUM
  If this option is defined, the library cannot create or write to
  databases that support auto_vacuum.
  Executing a PRAGMA auto_vacuum statement is not an error
  (since unknown PRAGMAs are silently ignored), but does not return a value
  or modify the auto-vacuum flag in the database file. If a database that
  supports auto-vacuum is opened by a library compiled with this option, it
  is automatically opened in read-only mode.

SQLITE_OMIT_BETWEEN_OPTIMIZATION
  This option disables the use of indices with WHERE clause terms
  that employ the BETWEEN operator.

SQLITE_OMIT_BLOB_LITERAL
  When this option is defined, it is not possible to specify a blob in
  an SQL statement using the X'ABCD' syntax.

SQLITE_OMIT_BTREECOUNT
  This option is no longer used for anything.  It is a no-op.

SQLITE_OMIT_BUILTIN_TEST
  This compile-time option has been renamed to SQLITE_UNTESTABLE.

SQLITE_OMIT_CASE_SENSITIVE_LIKE_PRAGMA
  This compile-time option disables the PRAGMA case_sensitive_like
  command.

SQLITE_OMIT_CAST
  This option causes SQLite to omit support for the CAST operator.

SQLITE_OMIT_CHECK
  This option causes SQLite to omit support for CHECK constraints.
  The parser will still accept CHECK constraints in SQL statements,
  they will just not be enforced.

SQLITE_OMIT_COMPILEOPTION_DIAGS
  This option is used to omit the compile-time option diagnostics available
  in SQLite, including the sqlite3_compileoption_used() and
  sqlite3_compileoption_get() C/C++ functions, the
  sqlite_compileoption_used() and sqlite_compileoption_get() SQL functions,
  and the compile_options pragma.

SQLITE_OMIT_COMPLETE
  This option causes the sqlite3_complete() and sqlite3_complete16()
  interfaces to be omitted.

SQLITE_OMIT_COMPOUND_SELECT
  This option is used to omit the compound SELECT functionality.
  SELECT statements that use the
  UNION, UNION ALL, INTERSECT or EXCEPT compound SELECT operators will
  cause a parse error.

  An INSERT statement with multiple values in the VALUES clause is
  implemented internally as a compound SELECT.  Hence, this option also
  disables the ability to insert more than a single row using an
  INSERT INTO ... VALUES ... statement.

SQLITE_OMIT_CTE
  This option causes support for common table expressions to be omitted.

SQLITE_OMIT_DATETIME_FUNCS
  If this option is defined, SQLite's built-in date and time manipulation
  functions are omitted. Specifically, the SQL functions julianday(), date(),
  time(), datetime() and strftime() are not available. The default column
  values CURRENT_TIME, CURRENT_DATE and CURRENT_TIMESTAMP are still available.

SQLITE_OMIT_DECLTYPE
  This option causes SQLite to omit support for the
  sqlite3_column_decltype() and sqlite3_column_decltype16()
  interfaces.

SQLITE_OMIT_DEPRECATED
  This option causes SQLite to omit support for interfaces
  marked as deprecated.  This includes
  sqlite3_aggregate_count(),
  sqlite3_expired(),
  sqlite3_transfer_bindings(),
  sqlite3_global_recover(),
  sqlite3_thread_cleanup() and
  sqlite3_memory_alarm() interfaces and
  PRAGMA statements PRAGMA count_changes,
  PRAGMA data_store_directory,
  PRAGMA default_cache_size,
  PRAGMA empty_result_callbacks,
  PRAGMA full_column_names,
  PRAGMA short_column_names, and
  PRAGMA temp_store_directory.

SQLITE_OMIT_DESERIALIZE
  This option causes the
  sqlite3_serialize() and sqlite3_deserialize()
  interfaces to be omitted from the build.

SQLITE_OMIT_DISKIO
  This option omits all support for writing to the disk and forces
  databases to exist in memory only.  This option has not been
  maintained and probably does not work with newer versions of SQLite.

SQLITE_OMIT_EXPLAIN
  Defining this option causes the EXPLAIN command to be omitted from the
  library. Attempting to execute an EXPLAIN statement will cause a parse
  error.

SQLITE_OMIT_FLAG_PRAGMAS
  This option omits support for a subset of PRAGMA commands that
  query and set boolean properties.

SQLITE_OMIT_FLOATING_POINT
  This option is used to omit floating-point number support from the SQLite
  library. When specified, specifying a floating point number as a literal
  (i.e. "1.01") results in a parse error.

  In the future, this option may also disable other floating point
  functionality, for example the sqlite3_result_double(),
  sqlite3_bind_double(), sqlite3_value_double() and
  sqlite3_column_double() API functions.
  

SQLITE_OMIT_FOREIGN_KEY
  If this option is defined, then foreign key constraint syntax is
  not recognized.

SQLITE_OMIT_GENERATED_COLUMNS
  If this option is defined, then generated column syntax is
  not recognized.

SQLITE_OMIT_GET_TABLE
  This option causes support for sqlite3_get_table() and
  sqlite3_free_table() to be omitted.

SQLITE_OMIT_HEX_INTEGER
  This option omits support for hexadecimal integer literals.

SQLITE_OMIT_INCRBLOB
  This option causes support for incremental BLOB I/O
  to be omitted.

SQLITE_OMIT_INTEGRITY_CHECK
  This option omits support for the integrity_check pragma.

SQLITE_OMIT_INTROSPECTION_PRAGMAS
  This option omits support for
  PRAGMA function_list, PRAGMA module_list, and
  PRAGMA pragma_list.

SQLITE_OMIT_JSON
  This option omits the JSON SQL functions from the build.

SQLITE_OMIT_LIKE_OPTIMIZATION
  This option disables the ability of SQLite to use indices to help
  resolve LIKE and GLOB operators in a WHERE clause.

SQLITE_OMIT_LOAD_EXTENSION
  This option omits the entire extension loading mechanism from
  SQLite, including sqlite3_enable_load_extension() and
  sqlite3_load_extension() interfaces.

SQLITE_OMIT_LOCALTIME
  This option omits the "localtime" modifier from the date and time
  functions.  This option is sometimes useful when trying to compile
  the date and time functions on a platform that does not support the
  concept of local time.

SQLITE_OMIT_LOOKASIDE
  This option omits the lookaside memory allocator.

SQLITE_OMIT_MEMORYDB
  When this is defined, the library does not respect the special database
  name ":memory:" (normally used to create an in-memory database). If
  ":memory:" is passed to sqlite3_open(), sqlite3_open16(), or
  sqlite3_open_v2(), a file with this name will be
  opened or created.

SQLITE_OMIT_OR_OPTIMIZATION
  This option disables the ability of SQLite to use an index together
  with terms of a WHERE clause connected by the OR operator.

SQLITE_OMIT_PAGER_PRAGMAS
  Defining this option omits pragmas related to the pager subsystem from
  the build.

SQLITE_OMIT_PRAGMA
  This option is used to omit the PRAGMA command
  from the library. Note that it is useful to define the macros that omit
  specific pragmas in addition to this, as they may also remove supporting code
  in other sub-systems. This macro removes the PRAGMA command only.

SQLITE_OMIT_PROGRESS_CALLBACK
  This option may be defined to omit the capability to issue "progress"
  callbacks during long-running SQL statements. The
  sqlite3_progress_handler()
  API function is not present in the library.

SQLITE_OMIT_QUICKBALANCE
  This option omits an alternative, faster B-Tree balancing routine.
  Using this option makes SQLite slightly smaller at the expense of
  making it run slightly slower.

SQLITE_OMIT_REINDEX
  When this option is defined, the REINDEX
  command is not included in the library.
  Executing a REINDEX statement causes
  a parse error.

SQLITE_OMIT_SCHEMA_PRAGMAS
  Defining this option omits pragmas for querying the database schema from
  the build.

SQLITE_OMIT_SCHEMA_VERSION_PRAGMAS
  Defining this option omits pragmas for querying and modifying the
  database schema version and user version from the build. Specifically, the
  schema_version and user_version PRAGMAs are omitted.

SQLITE_OMIT_SHARED_CACHE
  This option builds SQLite without support for shared cache mode.
  The sqlite3_enable_shared_cache() is omitted along with a fair
  amount of logic within the B-Tree subsystem associated with shared
  cache management.

  This compile-time option is recommended most applications as it
  results in improved performance and reduced library footprint.

SQLITE_OMIT_SUBQUERY
  If defined, support for sub-selects and the IN() operator are omitted.

SQLITE_OMIT_TCL_VARIABLE
  If this macro is defined, then the special "$" syntax
  used to automatically bind SQL variables to TCL variables is omitted.

SQLITE_OMIT_TEMPDB
  This option omits support for TEMP or TEMPORARY tables.

SQLITE_OMIT_TRACE
  This option omits support for the sqlite3_profile() and
  sqlite3_trace() interfaces and their associated logic.

SQLITE_OMIT_TRIGGER
  Defining this option omits support for TRIGGER objects. Neither the
  CREATE TRIGGER or DROP TRIGGER
  commands are available in this case, and attempting to execute
  either will result in a parse error.
  This option also disables enforcement of foreign key constraints,
  since the code that implements triggers and which is omitted by this
  option is also used to implement foreign key actions.

SQLITE_OMIT_TRUNCATE_OPTIMIZATION
  A default build of SQLite, if a DELETE statement has no WHERE clause
  and operates on a table with no triggers, an optimization occurs that
  causes the DELETE to occur by dropping and recreating the table.
  Dropping and recreating a table is usually much faster than deleting
  the table content row by row.  This is the "truncate optimization".

SQLITE_OMIT_UTF16
  This macro is used to omit support for UTF16 text encoding. When this is
  defined all API functions that return or accept UTF16 encoded text are
  unavailable. These functions can be identified by the fact that they end
  with '16', for example sqlite3_prepare16(), sqlite3_column_text16() and
  sqlite3_bind_text16().

SQLITE_OMIT_VACUUM
  When this option is defined, the VACUUM
  command is not included in the library.
  Executing a VACUUM statement causes
  a parse error.

SQLITE_OMIT_VIEW
  Defining this option omits support for VIEW objects. Neither the
  CREATE VIEW nor the DROP VIEW
  commands are available in this case, and
  attempting to execute either will result in a parse error.

  WARNING: If this macro is defined, it will not be possible to open a database
  for which the schema contains VIEW objects.

SQLITE_OMIT_VIRTUALTABLE
  This option omits support for the Virtual Table
  mechanism in SQLite.

SQLITE_OMIT_WAL
  This option omits the "write-ahead log" (a.k.a. "WAL") capability.

SQLITE_OMIT_WINDOWFUNC
  This option omits window functions from the build.

SQLITE_OMIT_WSD
  This option builds a version of the SQLite library that contains no
  Writable Static Data (WSD).  WSD is global variables and/or static
  variables.  Some platforms do not support WSD, and this option is necessary
  in order for SQLite to work those platforms.

  Unlike other OMIT options which make the SQLite library smaller,
  this option actually increases the size of SQLite and makes it run
  a little slower.  Only use this option if SQLite is being built for an
  embedded target that does not support WSD.

SQLITE_OMIT_XFER_OPT
  This option omits support for optimizations that help statements
  of the form "INSERT INTO ... SELECT ..." run faster.

SQLITE_UNTESTABLE
  A standard SQLite build includes a small amount of logic associated
  with sqlite3_test_control() to exercise
  parts of the SQLite core that are otherwise difficult to validate.
  This compile-time option omits that extra testing logic.  This
  compile-time option was called "SQLITE_OMIT_BUILTIN_TEST" prior
  to SQLite version 3.16.0 (2017-01-02).  The name was changed
  to better describe the implications of using it.
  
  Setting this compile-time option prevents SQLite from being fully
  testable.  Branch test coverage drops from 100% down to about 95%.
  
  SQLite developers follow the NASA principle of
  "fly what you test and test what you fly".  This principle is violated
  if this option is enabled for delivery but disabled for testing.
  But if this option is enabled during testing, not all branches are
  reachable.  Therefore, the use of this compile-time option is discouraged.

SQLITE_ZERO_MALLOC
  This option omits both the default memory allocator and the
  debugging memory allocator from the build and substitutes a stub
  memory allocator that always fails.  SQLite will not run with this
  stub memory allocator since it will be unable to allocate memory.  But
  this stub can be replaced at start-time using
  sqlite3_config(SQLITE_CONFIG_MALLOC,...) or
  sqlite3_config(SQLITE_CONFIG_HEAP,...).
  So the net effect of this compile-time option is that it allows SQLite
  to be compiled and linked against a system library that does not support
  malloc(), free(), and/or realloc().


c2: Compile-time Options
c3: 10. Analysis and Debugging Options
c5: compile.html#_analysis_and_debugging_options
c4:
SQLITE_DEBUG
  The SQLite source code contains literally thousands of assert() statements
  used to verify internal assumptions and subroutine preconditions and
  postconditions.  These assert() statements are normally turned off
  (they generate no code) since turning them on makes SQLite run approximately
  three times slower.  But for testing and analysis, it is useful to turn
  the assert() statements on.  The SQLITE_DEBUG compile-time option does this.
  SQLITE_DEBUG also enables some other debugging features, such as
  special PRAGMA statements that turn on tracing and listing features
  used for troubleshooting and analysis of the VDBE and code generator.

SQLITE_MEMDEBUG
  The SQLITE_MEMDEBUG option causes an instrumented
  debugging memory allocator
  to be used as the default memory allocator within SQLite.  The
  instrumented memory allocator checks for misuse of dynamically allocated
  memory.  Examples of misuse include using memory after it is freed,
  writing off the ends of a memory allocation, freeing memory not previously
  obtained from the memory allocator, or failing to initialize newly
  allocated memory.


c2: Compile-time Options
c3: 11. Windows-Specific Options
c5: compile.html#_windows_specific_options
c4:
SQLITE_WIN32_HEAP_CREATE
  This option forces the Win32 native memory allocator, when enabled, to
  create a private heap to hold all memory allocations.

SQLITE_WIN32_MALLOC_VALIDATE
  This option forces the Win32 native memory allocator, when enabled, to
  make strategic calls into the HeapValidate() function if assert() is also
  enabled.


c2: Compile-time Options
c3: 12. Compiler Linkage and Calling Convention Control
c5: compile.html#compiler_linkage_and_calling_convention_control
c4:
The following macros specify interface details
for certain kinds of SQLite builds.  The Makefiles will normally
handle setting these macros automatically.  Application developers should
not need to worry with these macros.  The following documentation about these
macros is included for completeness.


SQLITE_API
  This macro identifies an externally visible interface for SQLite.
  This macro is sometimes set to "extern".  But the definition is
  compiler-specific.

SQLITE_APICALL
  This macro identifies the calling convention used by public interface
  routines in SQLite which accept a fixed number of arguments.
  This macro is normally defined to be nothing,
  though on Windows builds it can sometimes be set to "__cdecl" or "__stdcall".
  The "__cdecl" setting is the default, but "__stdcall" is used when SQLite
  is intended to be compiled as a Windows system library.
  
  A single function declaration should contain no more than one of
  the following:  SQLITE_APICALL, SQLITE_CDECL, or SQLITE_SYSAPI.

SQLITE_CALLBACK
  This macro specifies the calling convention used with callback pointers
  in SQLite.  This macro is normally defined to be nothing, though on Windows
  builds it can sometimes be set to "__cdecl" or "__stdcall".  The
  "__cdecl" setting is the default, but "__stdcall" is used when SQLite
  is intended to be compiled as a Windows system library.

SQLITE_CDECL
  This macro specifies the calling convention used by varargs interface
  routines in SQLite.  This macro is normally defined to be nothing,
  though on Windows builds it can sometimes be set to "__cdecl".  This
  macro is used on varargs routines and so cannot be set to "__stdcall"
  since the __stdcall calling convention does not support varargs functions.
  
  A single function declaration should contain no more than one of
  the following:  SQLITE_APICALL, SQLITE_CDECL, or SQLITE_SYSAPI.

SQLITE_EXTERN
  This macro specifies linkage for public interface variables in SQLite.
  It should normally be allowed to default to "extern".

SQLITE_STDCALL
  This macro is no longer used and is now deprecated.

SQLITE_SYSAPI
  This macro identifies the calling convention used by operating system
  interfaces for the target platform for an SQLite build.
  This macro is normally defined to be nothing,
  though on Windows builds it can sometimes be set to "__stdcall".
  
  A single function declaration should contain no more than one of
  the following:  SQLITE_APICALL, SQLITE_CDECL, or SQLITE_SYSAPI.

SQLITE_TCLAPI
  This macro specifies the calling convention used by the
  TCL library interface routines.
  This macro is not used by the SQLite core, but only by the TCL Interface
  and TCL test suite.
  This macro is normally defined to be nothing,
  though on Windows builds it can sometimes be set to "__cdecl".  This
  macro is used on TCL library interface routines which are always compiled
  as __cdecl, even on platforms that prefer to use __stdcall, so this
  macro should not be set to __stdcall unless the platform has a custom
  TCL library build that supports __stdcall.
  
  This macro may not be used in combination with any of SQLITE_APICALL,
  SQLITE_CALLBACK, SQLITE_CDECL or SQLITE_SYSAPI.

This page last modified on  2024-05-09 08:10:19 UTC


c2: Profiling SQL Queries
c3: 1. Overview
c5: profile.html#_overview_
c4:
SQLite contains built-in support for profiling SQL queries, but it is not
enabled by default. In order to enable support for query profiling, SQLite must
be compiled with the following option:

-DSQLITE_ENABLE_STMT_SCANSTATUS


Building SQLite with this option enables the sqlite3_stmt_scanstatus_v2()
API, which provides access to the various profiling metrics. The remainder of
this page discusses the profiling reports generated by the SQLite 
command-line shell using these metrics, rather than the API directly. 

The profiling reports generated by the shell are very similar to the query
plan reports generated by the EXPLAIN QUERY PLAN command. This page assumes 
that the reader is familiar with this format.

Within a command-line shell compiled with the option above, query profiling
is enabled using the ".scanstats on" command:

sqlite> .scanstats on


Once enabled, the shell automatically outputs a query profile after each 
SQL query executed. Query profiling can be disabled using ".scanstats off".
For example:

sqlite> .scanstats on
sqlite> SELECT a FROM t1, t2 WHERE a IN (1,2,3) AND a=d+e;
QUERY PLAN (cycles=255831538 &#91;100%&#93;)
|--SEARCH t1 USING INTEGER PRIMARY KEY (rowid=?)     (cycles=60048488 &#91;23%&#93; loops=1 rows=3)
`--SCAN t2                                           (cycles=133558052 &#91;52%&#93; loops=3 rows=150000)


c2: Profiling SQL Queries
c3: 2. Simple Cases - Rows, Loops and Cycles
c5: profile.html#simple_cases_rows_loops_and_cycles
c4:
Consider a database with the following schema:

CREATE VIRTUAL TABLE ft USING fts5(text);
CREATE TABLE t1(a, b);
CREATE TABLE t2(c INTEGER PRIMARY KEY, d);


Then, after first executing ".scanstats on":

sqlite3> SELECT * FROM t1, t2 WHERE t2.c=t1.a;
&lt;...query results...&gt;
QUERY PLAN (cycles=1140768 &#91;100%&#93;)
|--SCAN t1                                           (cycles=455974 &#91;40%&#93; loops=1 rows=500)
`--SEARCH t2 USING INTEGER PRIMARY KEY (rowid=?)     (cycles=619820 &#91;54%&#93; loops=500 rows=250)


The text in the example above following the snipped 
"&lt;...query results...&gt;" is the profile report for the join query just
executed. The parts of the profile report that are similar to the 
EXPLAIN QUERY PLAN output indicate that the query is implemented by doing a
full table-scan of table "t1", with a lookup by INTEGER PRIMARY KEY on table
"t2" for each row visited.

The "loops=1" notation on the "SCAN t1" line indicates that this loop - the
full-table scan of table "t1" - ran exactly once. "rows=500" indicates that
that single scan visited 500 rows.

The "SEARCH t2 USING ..." line contains the annotation "loops=500" to indicate
that this "loop" (really a lookup by INTEGER PRIMARY KEY) ran 500 times.
Which makes sense - it ran once for each row visited by the full-table scan
of "t1". "rows=250" means that, altogether, those 500 loops visited 250 rows.
In other words, only half of the INTEGER PRIMARY KEY lookups on table t2 were
successful, for the other half of the lookups there was no row to find.

The loop-count for a SEARCH or SCAN entry is not necessarily the same as the
number of rows output by the outer loop. For example, if the query above
were modified as follows:

sqlite3> SELECT * FROM t1, t2 WHERE t1.b&lt;=100 AND t2.c=t1.a;
&lt;...query results...&gt;
QUERY PLAN (cycles=561002 &#91;100%&#93;)
|--SCAN t1                                           (cycles=345950 &#91;62%&#93; loops=1 rows=500)
`--SEARCH t2 USING INTEGER PRIMARY KEY (rowid=?)     (cycles=128690 &#91;23%&#93; loops=100 rows=50)


This time, even though the "SCAN t1" loop still visits 500 rows, the 
"SEARCH t2" lookup is only performed 100 times. This is because SQLite
was able to discard rows from t1 that did not match the "t1.b&lt;=100" 
constraint.

The "cycles" measurements are based on the 
CPU time-stamp counter
, and so are a good proxy for wall-clock time. For the query above, the
total number of cycles was 561002. For each of the two loops ("SCAN t1..." and
"SEARCH t2..."), the cycles count represents the time spent in operations that
can be directly attributed to that loop only. Specifically, this is the time
spent navigating and extracting data from the database b-trees for that loop.
These values never quite add up to the total cycles for the query, as there are
other internal operations performed by SQLite that are not directly
attributable to either loop.

The cycles count for the "SCAN t1" loop was 345950 - 62% of the total for the
query. The 100 lookups peformed by the "SEARCH t1" loop took 128690 cycles, 23%
of the total.

When a virtual table is used, the "rows" and "loops" metrics have the same 
meaning as for loops on regular SQLite tables. The "cycles" meaurement is the
total cycles consumed within virtual table methods associated with the loop.
For example:

sqlite3> SELECT * FROM ft('sqlite'), t2 WHERE t2.c=ft.rowid;
&lt;...query results...&gt;
QUERY PLAN (cycles=836434 &#91;100%&#93;
|--SCAN ft VIRTUAL TABLE INDEX 0:M1                  (cycles=739602 &#91;88%&#93; loops=1 rows=48)
`--SEARCH t2 USING INTEGER PRIMARY KEY (rowid=?)     (cycles=62866 &#91;8%&#93; loops=48 rows=25)


In this case the single query (loops=1) on fts5 table "ft" returned 48 rows
(rows=48) and consumed 739602 cycles (cycles=739602), which was roughly 88% of
the total query time.


c2: Profiling SQL Queries
c3: 3. Complex Cases - Rows, Loops and Cycles
c5: profile.html#complex_cases_rows_loops_and_cycles
c4:
Using the same schema as in the previous section, consider this more
complicated example:

sqlite3&gt; WITH cnt(i) AS (
  SELECT 1 UNION SELECT i+1 FROM cnt WHERE i&lt;100
)
SELECT
  *, (SELECT d FROM t2 WHERE c=ft.rowid)
FROM
  (SELECT count(*), a FROM t1 GROUP BY a) AS v1 CROSS JOIN
  ft('sqlite'),
  cnt
WHERE cnt.i=ft.rowid AND v1.a=ft.rowid;
&lt;...query results...&gt;
QUERY PLAN (cycles=177665334 &#91;100%&#93;)
|--CO-ROUTINE v1                                        (cycles=4500444 &#91;3%&#93;)
|  |--SCAN t1                                           (cycles=397052 &#91;0%&#93; loops=1 rows=500)
|  `--USE TEMP B-TREE FOR GROUP BY
|--MATERIALIZE cnt                                      (cycles=1275068 &#91;1%&#93;)
|  |--SETUP
|  |  `--SCAN CONSTANT ROW
|  `--RECURSIVE STEP
|     `--SCAN cnt                                       (cycles=129166 &#91;0%&#93; loops=100 rows=100)
|--SCAN v1                                              (loops=1 rows=500)
|--SCAN ft VIRTUAL TABLE INDEX 0:M1=                    (cycles=161874340 &#91;91%&#93; loops=500 rows=271)
|--SCAN cnt                                             (cycles=7336350 &#91;4%&#93; loops=95 rows=9500)
`--CORRELATED SCALAR SUBQUERY 3                         (cycles=168538 &#91;0%&#93; loops=37)
   `--SEARCH t2 USING INTEGER PRIMARY KEY (rowid=?)     (cycles=94724 &#91;0%&#93; loops=37 rows=21)


The most complicated part of the example above is understanding the query 
plan - the portion of the report that would also be generated by an
EXPLAIN QUERY PLAN command. Other points of interest are:


   Sub-query "v1" is implemented as a co-routine.
  In this case the sub-query is reported on separately, and a "cycles"
  count is available for the entire sub-query. There is also a "SCAN v1"
  line - this represents the invocation of the sub-query co-routine
  from the main query. This entry has no cycles associated with it, as
  the entire cost of the sub-query is attributed to the co-routine.
  It does have "loops" and "rows" values - the sub-query is scanned
  once and returns 500 rows.

  Recursive sub-query "cnt" is materialized (cached in a temp
  table) before the main query is run. The entire cost of the materialization
  is attributed to the "MATERIALIZE cnt" element. There is also a 
  "SCAN cnt" item representing the scans of the materialized sub-query.
  The cycles value associated with this item represents the time taken
  to scan the temp-table containing the materialized sub-query, which is 
  separate from the cycles used to populate it.

  There are cycles and loops measurements for the scalar sub-query
  as well. These represent the total cycles consumed while executing the
  sub-query and the number of times it was executed, respectively.

  Where one item is a parent of another, as in "CORRELATED SCALAR
  SUBQUERY 3" and "SEARCH t2 USING...", then the cycles value associated
  with the parent includes those cycles associated with all child elements.
  In all cases, the percentage values relate to the total cycles used by
  the query, not the cycles used by the parent.


The following query uses an automatic index and
an external sort:

sqlite> SELECT * FROM
  t2,
  (SELECT count(*) AS cnt, d FROM t2 GROUP BY d) AS v2
WHERE v2.d=t2.d AND t2.d>100
ORDER BY v2.cnt;
&lt;...query results...&gt;
QUERY PLAN (cycles=6234376 &#91;100%&#93;)
|--MATERIALIZE v2                                     (cycles=2351916 &#91;38%&#93;)
|  |--SCAN t2                                         (cycles=188428 &#91;3%&#93; loops=1 rows=250)
|  `--USE TEMP B-TREE FOR GROUP BY
|--SCAN t2                                            (cycles=455736 &#91;7%&#93; loops=1 rows=250)
|--CREATE AUTOMATIC INDEX ON v2(d, cnt)               (cycles=1668380 &#91;27%&#93; loops=1 rows=250)
|--SEARCH v2 USING AUTOMATIC COVERING INDEX (d=?)     (cycles=932824 &#91;15%&#93; loops=200 rows=200)
`--USE TEMP B-TREE FOR ORDER BY                       (cycles=662456 &#91;11%&#93; loops=1 rows=200)


Points of interest are:


   This query materializes the sub-query into a temp table, then creates
  an automatic (i.e. transient) index on it, then uses that index to optimize
  the join. All three of these steps - "MATERIALIZE v2", "CREATE AUTOMATIC
  INDEX" and "SEARCH ... USING AUTOMATIC INDEX" have separate cycle counts.
  The "rows" associated with the "CREATE AUTOMATIC INDEX" line represents the
  total number of rows included in the index. The "loops" and "rows" associated
  with the "SEARCH ... USING AUTOMATIC INDEX" line represent the number of
  lookups the index was used for and the total number of rows found by those
  lookups.

   The external sort "USE TEMP B-TREE FOR ORDER BY" is also accounted
  for separately. The cycles count represents the extra cycles consumed by
  sorting the returned rows, above those that would have been used if the rows
  were returned in arbitrary order. The rows count represents the number of
  rows sorted.


c2: Profiling SQL Queries
c3: 4. Planner Estimates
c5: profile.html#planner_estimates
c4:
As well as ".scanstats on" to enable profiling and ".scanstats off" to
disable it, the shell also accepts ".scanstats est":

sqlite> .scanstats est


This enables a special kind of profiling report that includes two extra
values associated with each "SCAN..." and "SEARCH..." element of a query
profile:


   rpl (rows per loop): This is the value of the "rows" metric
       divided by that of the "loops" metric.
   est (estimate): This is the number of rows per loop that the
       query planner estimated would be returned by the current query
       element. If this value is radically different from the actual
       rows per loop value and query performance is unsatisfactory, 
       the quality of this estimate may be improved by running ANALYZE.

    
sqlite> SELECT a FROM t1, t2 WHERE a IN (1,2,3) AND a=d+e ORDER BY a;
&lt;query results...&gt;
QUERY PLAN (cycles=264725190 &#91;100%&#93;
|--SEARCH t1 USING INTEGER PRIMARY KEY (rowid=?)     (cycles=60511568 &#91;23%&#93; loops=1 rows=3 rpl=3.0 est=3.0)
`--SCAN t2                                           (cycles=139461608 &#91;53%&#93; loops=3 rows=150000 rpl=50000.0 est=1048576.0)


c2: Profiling SQL Queries
c3: 5. Low-Level Profiling Data
c5: profile.html#low_level_profiling_data
c4:
Also supported is the ".scanstats vm" command. This enables a lower-level
profiling report showing the number of times each VM instruction was executed
and the percentage of clock-cycles that passed while it was executing:

sqlite> .scanstats vm


Then:

sqlite> SELECT count(*) FROM t2 WHERE (d % 5) = 0;
&lt;query results...&gt;
addr  cycles  nexec   opcode         p1    p2    p3    p4             p5  comment      
----  ------  ------  -------------  ----  ----  ----  -------------  --  -------------
0     0.0%    1       Init           1     18    0                    0   Start at 18
1     0.0%    1       Null           0     1     1                    0   r&#91;1..1]=NULL
2     0.0%    1       OpenRead       0     2     0     2              0   root=2 iDb=0; t2
3     0.0%    1       ColumnsUsed    0     0     0     2              0   
4     0.0%    1       Explain        4     0     0     SCAN t2        0   
5     0.0%    1       CursorHint     0     0     0     EQ(REM(c1,5),0) 0   
6     0.0%    1       Rewind         0     14    0                    0   
7     46.86%  150000    Column         0     1     3                    0   r&#91;3]= cursor 0 column 1
8     18.94%  150000    Remainder      4     3     2                    0   r&#91;2]=r&#91;3]%r&#91;4]
9     5.41%   150000    ReleaseReg     3     1     0                    0   release r&#91;3] mask 0
10    12.09%  150000    Ne             5     13    2                    80  if r&#91;2]!=r&#91;5] goto 13
11    1.02%   30000     ReleaseReg     2     1     0                    0   release r&#91;2] mask 0
12    2.95%   30000     AggStep1       0     0     1     count(0)       0   accum=r&#91;1] step(r&#91;0])
13    12.72%  150000  Next           0     7     0                    1   
14    0.0%    1       AggFinal       1     0     0     count(0)       0   accum=r&#91;1] N=0
15    0.0%    1       Copy           1     6     0                    0   r&#91;6]=r&#91;1]
16    0.0%    1       ResultRow      6     1     0                    0   output=r&#91;6]
17    0.01%   1       Halt           0     0     0                    0   
18    0.0%    1       Transaction    0     0     1     0              1   usesStmtJournal=0
19    0.0%    1       Integer        5     4     0                    0   r&#91;4]=5
20    0.0%    1       Integer        0     5     0                    0   r&#91;5]=0
21    0.0%    1       Goto           0     1     0                    0


c2: How SQLite Is Tested
c3: 1. Introduction
c5: testing.html#introduction
c4:
The reliability and robustness of SQLite is achieved in part
by thorough and careful testing.

As of version 3.42.0 (2023-05-16),
the SQLite library consists of approximately
155.8 KSLOC of C code.
(KSLOC means thousands of "Source Lines Of Code" or, in other words,
lines of code excluding blank lines and comments.)
By comparison, the project has
590 times as much
test code and test scripts -
92053.1 KSLOC.


c2: How SQLite Is Tested
c3: 1.1. Executive Summary
c5: testing.html#executive_summary
c4:
Four independently developed test harnesses
 100% branch test coverage in an as-deployed configuration
 Millions and millions of test cases
 Out-of-memory tests
 I/O error tests
 Crash and power loss tests
 Fuzz tests
 Boundary value tests
 Disabled optimization tests
 Regression tests
 Malformed database tests
 Extensive use of assert() and run-time checks
 Valgrind analysis
 Undefined behavior checks
 Checklists


c2: How SQLite Is Tested
c3: 2. Test Harnesses
c5: testing.html#test_harnesses
c4:
There are four independent test harnesses used for testing the
core SQLite library.
Each test harness is designed, maintained, and managed separately
from the others.






The TCL Tests are the original tests for SQLite.
They are contained in the same source tree as the
SQLite core and like the SQLite core are in the public domain.  The
TCL tests are the primary tests used during development.
The TCL tests are written using the
TCL scripting language.
The TCL test harness itself consists of 27.2 KSLOC
of C code used to create the TCL interface.  The test scripts are contained
in 1390 files totaling
23.2MB in size.  There are
51445 distinct test cases, but many of the test
cases are parameterized and run multiple times (with different parameters)
so that on a full test run millions of
separate tests are performed.




The TH3 test harness is a set of proprietary tests, written in
C that provide 100% branch test coverage
(and 100% MC/DC test coverage) to
the core SQLite library.  The TH3 tests are designed to run
on embedded and specialized platforms that would not easily support
TCL or other workstation services.  TH3 tests use only the published
SQLite interfaces. TH3 consists of about
76.9 MB or 1055.4 KSLOC
of C code implementing 50362 distinct test cases.
TH3 tests are heavily parameterized, though, so a full-coverage test runs
about 2.4 million different test
instances.
The cases that provide 100% branch test coverage constitute
a subset of the total TH3 test suite.  A soak test
prior to release does about
248.5 million tests.
Additional information on TH3 is available separately.




The SQL Logic Test
or SLT test harness is used to run huge numbers
of SQL statements against both SQLite and several other SQL database engines
and verify that they all get the same answers.  SLT currently compares
SQLite against PostgreSQL, MySQL, Microsoft SQL Server, and Oracle 10g.
SLT runs 7.2 million queries comprising
1.12GB of test data.



The dbsqlfuzz engine is a
proprietary fuzz tester.  Other fuzzers for SQLite
mutate either the SQL inputs or the database file.  Dbsqlfuzz mutates
both the SQL and the database file at the same time, and is thus able
to reach new error states.  Dbsqlfuzz is built using the
libFuzzer framework of LLVM
with a custom mutator.  There are
336 seed files. The dbsqlfuzz fuzzer
runs about one billion test mutations per day.
Dbsqlfuzz helps ensure
that SQLite is robust against attack via malicious SQL or database
inputs.


In addition to the four main test harnesses, there many other
small programs that implement specialized tests.  Here are a few
examples:

The "speedtest1.c" program
estimates the performance of SQLite under a typical workload.
The "mptester.c" program is a stress test for multiple processes
concurrently reading and writing a single database.
The "threadtest3.c" program is a stress test for multiple threads using
SQLite simultaneously.
The "fuzzershell.c" program is used to
run some fuzz tests.
The "jfuzz" program is a libfuzzer-based fuzzer for
JSONB inputs to the JSON SQL functions.



All of the tests above must run successfully, on multiple platforms
and under multiple compile-time configurations,
before each release of SQLite.

Prior to each check-in to the SQLite source tree, developers
typically run a subset (called "veryquick") of the Tcl tests
consisting of about
304.7 thousand test cases.
The veryquick tests include most tests other than the anomaly, fuzz, and
soak tests.  The idea behind the veryquick tests are that they are
sufficient to catch most errors, but also run in only a few minutes
instead of a few hours.


c2: How SQLite Is Tested
c3: 3. Anomaly Testing
c5: testing.html#anomaly_testing
c4:
Anomaly tests are tests designed to verify the correct behavior
of SQLite when something goes wrong.  It is (relatively) easy to build
an SQL database engine that behaves correctly on well-formed inputs
on a fully functional computer.  It is more difficult to build a system
that responds sanely to invalid inputs and continues to function following
system malfunctions.  The anomaly tests are designed to verify the latter
behavior.


c2: How SQLite Is Tested
c3: 3.1. Out-Of-Memory Testing
c5: testing.html#out_of_memory_testing
c4:
SQLite, like all SQL database engines, makes extensive use of
malloc()  (See the separate report on
dynamic memory allocation in SQLite for
additional detail.)
On servers and workstations, malloc() never fails in practice and so correct
handling of out-of-memory (OOM) errors is not particularly important.
But on embedded devices, OOM errors are frighteningly common and since
SQLite is frequently used on embedded devices, it is important that
SQLite be able to gracefully handle OOM errors.

OOM testing is accomplished by simulating OOM errors.
SQLite allows an application to substitute an alternative malloc()
implementation using the sqlite3_config(SQLITE_CONFIG_MALLOC,...)
interface.  The TCL and TH3 test harnesses are both capable of
inserting a modified version of malloc() that can be rigged to fail
after a certain number of allocations.  These instrumented mallocs
can be set to fail only once and then start working again, or to
continue failing after the first failure.  OOM tests are done in a
loop.  On the first iteration of the loop, the instrumented malloc
is rigged to fail on the first allocation.  Then some SQLite operation
is carried out and checks are done to make sure SQLite handled the
OOM error correctly.  Then the time-to-failure counter
on the instrumented malloc is increased by one and the test is
repeated.  The loop continues until the entire operation runs to
completion without ever encountering a simulated OOM failure.
Tests like this are run twice, once with the instrumented malloc
set to fail only once, and again with the instrumented malloc set
to fail continuously after the first failure.


c2: How SQLite Is Tested
c3: 3.2. I/O Error Testing
c5: testing.html#i_o_error_testing
c4:
I/O error testing seeks to verify that SQLite responds sanely
to failed I/O operations.  I/O errors might result from a full disk drive,
malfunctioning disk hardware, network outages when using a network
file system, system configuration or permission changes that occur in the
middle of an SQL operation, or other hardware or operating system
malfunctions.  Whatever the cause, it is important that SQLite be able
to respond correctly to these errors and I/O error testing seeks to
verify that it does.

I/O error testing is similar in concept to OOM testing; I/O errors
are simulated and checks are made to verify that SQLite responds
correctly to the simulated errors.  I/O errors are simulated in both
the TCL and TH3 test harnesses by inserting a new
Virtual File System object that is specially rigged
to simulate an I/O error after a set number of I/O operations.
As with OOM error testing, the I/O error simulators can be set to
fail just once, or to fail continuously after the first failure.
Tests are run in a loop, slowly increasing the point of failure until
the test case runs to completion without error.  The loop is run twice,
once with the I/O error simulator set to simulate only a single failure
and a second time with it set to fail all I/O operations after the first
failure.

In I/O error tests, after the I/O error simulation failure mechanism
is disabled, the database is examined using
PRAGMA integrity_check to make sure that the I/O error has not
introduced database corruption.


c2: How SQLite Is Tested
c3: 3.3. Crash Testing
c5: testing.html#crash_testing
c4:
Crash testing seeks to demonstrate that an SQLite database will not
go corrupt if the application or operating system crashes or if there
is a power failure in the middle of a database update.  A separate
white-paper titled
Atomic Commit in SQLite describes the
defensive measure SQLite takes to prevent database corruption following
a crash.  Crash tests strive to verify that those defensive measures
are working correctly.

It is impractical to do crash testing using real power failures, of
course, and so crash testing is done in simulation.  An alternative
Virtual File System is inserted that allows the test
harness to simulate the state of the database file following a crash.

In the TCL test harness, the crash simulation is done in a separate
process.  The main testing process spawns a child process which runs
some SQLite operation and randomly crashes somewhere in the middle of
a write operation.  A special VFS randomly reorders and corrupts
the unsynchronized
write operations to simulate the effect of buffered filesystems.  After
the child dies, the original test process opens and reads the test
database and verifies that the changes attempted by the child either
completed successfully or else were completely rolled back.  The
integrity_check PRAGMA is used to make sure no database corruption
occurs.

The TH3 test harness needs to run on embedded systems that do not
necessarily have the ability to spawn child processes, so it uses
an in-memory VFS to simulate crashes.  The in-memory VFS can be rigged
to make a snapshot of the entire filesystem after a set number of I/O
operations.  Crash tests run in a loop.  On each iteration of the loop,
the point at which a snapshot is made is advanced until the SQLite
operations being tested run to completion without ever hitting a
snapshot.  Within the loop, after the SQLite operation under test has
completed, the filesystem is reverted to the snapshot and random file
damage is introduced that is characteristic of the kinds of damage
one expects to see following a power loss.  Then the database is opened
and checks are made to ensure that it is well-formed and that the
transaction either ran to completion or was completely rolled back.
The interior of the loop is repeated multiple times for each
snapshot with different random damage each time.


c2: How SQLite Is Tested
c3: 3.4. Compound failure tests
c5: testing.html#compound_failure_tests
c4:
The test suites for SQLite also explore the result of stacking
multiple failures.  For example, tests are run to ensure correct behavior
when an I/O error or OOM fault occurs while trying to recover from a
prior crash.


c2: How SQLite Is Tested
c3: 4. Fuzz Testing
c5: testing.html#fuzz_testing
c4:
Fuzz testing
seeks to establish that SQLite responds correctly to invalid, out-of-range,
or malformed inputs.


c2: How SQLite Is Tested
c3: 4.1. SQL Fuzz
c5: testing.html#sql_fuzz
c4:
SQL fuzz testing consists of creating syntactically correct yet
wildly nonsensical SQL statements and feeding them to SQLite to see
what it will do with them.  Usually some kind of error is returned
(such as "no such table").  Sometimes, purely by chance, the SQL
statement also happens to be semantically correct.  In that case, the
resulting prepared statement is run to make sure it gives a reasonable
result.


c2: How SQLite Is Tested
c3: 4.1.1. SQL Fuzz Using The American Fuzzy Lop Fuzzer
c5: testing.html#sql_fuzz_using_the_american_fuzzy_lop_fuzzer
c4:
The concept of fuzz testing has been around for decades, but fuzz
testing was not an effective way to find bugs until 2014 when
Michal Zalewski invented the first practical profile-guided fuzzer,
American Fuzzy Lop or "AFL".
Unlike prior fuzzers that blindly generate random inputs, AFL
instruments the program being tested (by modifying the assembly-language
output from the C compiler) and uses that instrumentation to detect when
an input causes the program to do something different - to follow
a new control path or loop a different number of times.  Inputs that provoke
new behavior are retained and further mutated.  In this way, AFL is able
to "discover" new behaviors of the program under test, including behaviors
that were never envisioned by the designers.

AFL proved adept at finding arcane bugs in SQLite.
Most of the findings have been assert() statements where the conditional
was false under obscure circumstances.  But AFL has also found
a fair number of crash bugs in SQLite, and even a few cases where SQLite
computed incorrect results.

Because of its past success, AFL became a standard part of the testing
strategy for SQLite beginning with version 3.8.10 (2015-05-07) until
it was superseded by better fuzzers in version 3.29.0 (2019-07-10).


c2: How SQLite Is Tested
c3: 4.1.2. Google OSS Fuzz
c5: testing.html#google_oss_fuzz
c4:
Beginning in 2016, a team of engineers at Google started the
OSS Fuzz project.
OSS Fuzz uses a AFL-style guided fuzzer running on Google's infrastructure.
The Fuzzer automatically downloads the latest check-ins for participating
projects, fuzzes them, and sends email to the developers reporting any
problems.  When a fix is checked in, the fuzzer automatically detects this
and emails a confirmation to the developers.

SQLite is one of many open-source projects that OSS Fuzz tests. The
test/ossfuzz.c source file
in the SQLite repository is SQLite's interface to OSS fuzz.

OSS Fuzz no longer finds historical bugs in SQLite.  But it is still
running and does occasionally find issues in new development check-ins.
Examples:
&#91;1&#93;
&#91;2&#93;
&#91;3&#93;.


c2: How SQLite Is Tested
c3: 4.1.3. The dbsqlfuzz and jfuzz fuzzers
c5: testing.html#the_dbsqlfuzz_and_jfuzz_fuzzers
c4:
Beginning in late 2018, SQLite has been fuzzed using a proprietary
fuzzer called "dbsqlfuzz".  Dbsqlfuzz is built using the
libFuzzer framework of LLVM.

The dbsqlfuzz fuzzer mutates both the SQL input and the database file
at the same time.  Dbsqlfuzz uses a custom
Structure-Aware Mutator
on a specialized input file that defines both an input database and SQL
text to be run against that database. Because it mutates both the input
database and the input SQL at the same time, dbsqlfuzz has been able to
find some obscure faults in SQLite that were missed by prior fuzzers that
mutated only SQL inputs or only the database file.
The SQLite developers keep dbsqlfuzz running against trunk in about
16 cores at all times.  Each instance of dbsqlfuzz program is able to
evalutes about 400 test cases per second, meaning that about 500 million
cases are checked every day.

The dbsqlfuzz fuzzer has been very successful at hardening the
SQLite code base against malicious attack.  Since dbsqlfuzz has been
added to the SQLite internal test suite, bug reports from external
fuzzers such as OSSFuzz have all but stopped.

Note that dbsqlfuzz is not the Protobuf-based structure-aware
fuzzer for SQLite that is used by Chromium and described in the
Structure-Aware Mutator article.
There is no connection between these two fuzzers, other than the fact that they
are both based on libFuzzer
The Protobuf fuzzer for SQLite is written and maintained by the Chromium
team at Google, whereas dbsqlfuzz is written and maintained by the original
SQLite developers.  Having multiple independently-developed fuzzers for SQLite
is good, as it means that obscure issues are more likely to be uncovered.

Near the end of January 2024, a second libFuzzer-based tool called
"jfuzz" came into use.  Jfuzz generates corrupt JSONB blobs and feeds
them into the JSON SQL functions to verify that the JSON functions
are able to safely and efficiently deal with corrupt binary inputs.


c2: How SQLite Is Tested
c3: 4.1.4. Other third-party fuzzers
c5: testing.html#other_third_party_fuzzers
c4:
SQLite seems to be a popular target for third-parties to fuzz.
The developers hear about many attempts to fuzz SQLite
and they do occasionally get bug reports found by independent
fuzzers.  All such reports are promptly fixed, so the product is
improved and that the entire SQLite user community benefits.
This mechanism of having many independent testers is similar to
Linus's law:
"given enough eyeballs, all bugs are shallow".

One fuzzing researcher of particular note is
Manuel Rigger.
Most fuzzers only look for assertion faults, crashes, undefined behavior (UB),
or other easily detected anomalies.  Dr. Rigger's fuzzers, on the other hand,
are able to find cases where SQLite computes an incorrect answer.
Rigger has found
many such cases.
Most of these finds are obscure corner cases involving type
conversions and affinity transformations, and a good number of the finds
are against unreleased features.  Nevertheless, his finds are still important
as they are real bugs,
and the SQLite developers are grateful to be able to identify and fix
the underlying problems.


c2: How SQLite Is Tested
c3: 4.1.5. The fuzzcheck test harness
c5: testing.html#the_fuzzcheck_test_harness
c4:
Historical test cases from AFL, OSS Fuzz, and dbsqlfuzz are
collected in a set of database files in the main SQLite source tree
and then rerun by the "fuzzcheck" utility program whenever one runs
"make test".  Fuzzcheck only runs a few thousand "interesting" cases
out of the billions of cases that the various fuzzers have
examined over the years.  "Interesting" cases are cases that exhibit
previously unseen behavior.  Actual bugs found by fuzzers are always
included among the interesting test cases, but most of the cases run
by fuzzcheck were never actual bugs.


c2: How SQLite Is Tested
c3: 4.1.6. Tension Between Fuzz Testing And 100% MC/DC Testing
c5: testing.html#tension_between_fuzz_testing_and_100_mc_dc_testing
c4:
Fuzz testing and 100% MC/DC testing are in tension with
one another.
That is to say, code tested to 100% MC/DC will tend to be
more vulnerable to problems found by fuzzing and code that performs
well during fuzz testing will tend to have (much) less than
100% MC/DC.
This is because MC/DC testing discourages defensive code with
unreachable branches, but without defensive code, a fuzzer is
more likely to find a path that causes problems.  MC/DC testing
seems to work well for building code that is robust during
normal use, whereas fuzz testing is good for building code that is
robust against malicious attack.

Of course, users would prefer code that is both robust in normal
use and resistant to malicious attack.  The SQLite developers are
dedicated to providing that.  The purpose of this section is merely
to point out that doing both at the same time is difficult.

For much of its history SQLite has been focused on 100% MC/DC testing.
Resistance to fuzzing attacks only became a concern with the introduction
of AFL in 2014.  For a while there, fuzzers were finding many problems
in SQLite.  In more recent years, the testing strategy of SQLite has
evolved to place more emphasis on fuzz testing.  We still maintain
100% MC/DC of the core SQLite code, but most testing CPU cycles are
now devoted to fuzzing.

While fuzz testing and 100% MC/DC testing are in tension, they
are not completely at cross-purposes.  The fact that the SQlite test
suite does test to 100% MC/DC means that when fuzzers do find problems,
those problems can be fixed quickly and with little risk of introducing
new errors.


c2: How SQLite Is Tested
c3: 4.2. Malformed Database Files
c5: testing.html#malformed_database_files
c4:
There are numerous test cases that verify that SQLite is able to
deal with malformed database files.
These tests first build a well-formed database file, then add
corruption by changing one or more bytes in the file by some means
other than SQLite.  Then SQLite is used to read the database.
In some cases, the bytes changes are in the middle of data.
This causes the content of the database to change while keeping the
database well-formed.
In other cases, unused bytes of the file are modified, which has
no effect on the integrity of the database.
The interesting cases are when bytes of the file that
define database structure get changed.  The malformed database tests
verify that SQLite finds the file format errors and reports them
using the SQLITE_CORRUPT return code without overflowing
buffers, dereferencing NULL pointers, or performing other
unwholesome actions.

The dbsqlfuzz fuzzer also does an excellent job of verifying
that SQLite responds sanely to malformed database files.


c2: How SQLite Is Tested
c3: 4.3. Boundary Value Tests
c5: testing.html#boundary_value_tests
c4:
SQLite defines certain limits on its operation, such as the
maximum number of columns in a table, the maximum length of an
SQL statement, or the maximum value of an integer.  The TCL and TH3 test
suites both contains numerous tests that push SQLite right to the edge
of its defined limits and verify that it performs correctly for
all allowed values.  Additional tests go beyond the defined limits
and verify that SQLite correctly returns errors.  The source code
contains testcase macros to verify that both sides of each boundary
have been tested.


c2: How SQLite Is Tested
c3: 5. Regression Testing
c5: testing.html#regression_testing
c4:
Whenever a bug is reported against SQLite, that bug is not considered
fixed until new test cases that would exhibit the bug have been added
to either the TCL or TH3 test suites.
Over the years,
this has resulted in thousands and thousands of new tests.
These regression tests ensure that bugs that have
been fixed in the past are not reintroduced into future versions of
SQLite.


c2: How SQLite Is Tested
c3: 6. Automatic Resource Leak Detection
c5: testing.html#automatic_resource_leak_detection
c4:
Resource leak occurs when system resources
are allocated and never freed.  The most troublesome resource leaks
in many applications are memory leaks - when memory is allocated using
malloc() but never released using free().  But other kinds of resources
can also be leaked:  file descriptors, threads, mutexes, etc.

Both the TCL and TH3 test harnesses automatically track system
resources and report resource leaks on every test run.
No special configuration or setup is required.   The test harnesses
are especially vigilant with regard to memory leaks.  If a change
causes a memory leak, the test harnesses will recognize this
quickly.  SQLite is designed to never leak memory, even after
an exception such as an OOM error or disk I/O error.  The test
harnesses are zealous to enforce this.


c2: How SQLite Is Tested
c3: 7. Test Coverage
c5: testing.html#test_coverage
c4:
The SQLite core, including the unix VFS,
has 100% branch test coverage under TH3 in
its default configuration as measured by
gcov.
Extensions such as FTS3 and RTree are excluded from this
analysis.


c2: How SQLite Is Tested
c3: 7.1. Statement versus branch coverage
c5: testing.html#statement_versus_branch_coverage
c4:
There are many ways to measure test coverage.  The most popular
metric is "statement coverage".  When you hear someone say that their
program as "XX% test coverage" without further explanation, they usually
mean statement coverage.  Statement coverage measures what percentage
of lines of code are executed at least once by the test suite.

Branch coverage is more rigorous than statement coverage.  Branch
coverage measures the number of machine-code branch instructions that
are evaluated at least once on both directions.

To illustrate the difference between statement coverage and
branch coverage, consider the following hypothetical
line of C code:

if( a>b && c!=25 ){ d++; }


Such a line of C code might generate a dozen separate machine code
instructions.  If any one of those instructions is ever evaluated, then
we say that the statement has been tested.  So, for example, it might
be the case that the conditional expression is
always false and the "d" variable is
never incremented.  Even so, statement coverage counts this line of
code as having been tested.

Branch coverage is more strict.  With branch coverage, each test and
each subblock within the statement is considered separately.  In order
to achieve 100% branch coverage in the example above, there must be at
least three test cases:


 a&lt;=b
 a&gt;b && c==25
 a&gt;b && c!=25


Any one of the above test cases would provide 100% statement coverage
but all three are required for 100% branch coverage.  Generally speaking,
100% branch coverage implies 100% statement coverage, but the converse is
not true.  To reemphasize, the
TH3 test harness for SQLite provides the stronger form of
test coverage - 100% branch test coverage.


c2: How SQLite Is Tested
c3: 7.2. Coverage testing of defensive code
c5: testing.html#coverage_testing_of_defensive_code
c4:
A well-written C program will typically contain some defensive
conditionals which in practice are always true or always false.
This leads to a
programming dilemma:  Does one remove defensive code in order to obtain
100% branch coverage?

In SQLite, the answer to the previous question is "no".
For testing purposes, the SQLite source code defines
macros called ALWAYS() and NEVER().   The ALWAYS() macro
surrounds conditions
which are expected to always evaluate as true and NEVER() surrounds
conditions that are always evaluated to false.  These macros serve as
comments to indicate that the conditions are defensive code.
In release builds, these macros are pass-throughs:

#define ALWAYS(X)  (X)
#define NEVER(X)   (X)


During most testing, however, these macros will throw an assertion
fault if their argument does not have the expected truth value.  This
alerts the developers quickly to incorrect design assumptions.

#define ALWAYS(X)  ((X)?1:assert(0),0)
#define NEVER(X)   ((X)?assert(0),1:0)


When measuring test coverage, these macros are defined to be constant
truth values so that they do not generate assembly language branch
instructions, and hence do not come into play when calculating the
branch coverage:

#define ALWAYS(X)  (1)
#define NEVER(X)   (0)


The test suite is designed to be run three times, once for each of
the ALWAYS() and NEVER() definitions shown above.  All three test runs
should yield exactly the same result.  There is a run-time test using
the sqlite3_test_control(SQLITE_TESTCTRL_ALWAYS, ...) interface that
can be used to verify that the macros are correctly set to the first
form (the pass-through form) for deployment.


c2: How SQLite Is Tested
c3: 7.3. Forcing coverage of boundary values and boolean vector tests
c5: testing.html#forcing_coverage_of_boundary_values_and_boolean_vector_tests
c4:
Another macro used in conjunction with test coverage measurement is
the testcase() macro.  The argument is a condition for which
we want test cases that evaluate to both true and false.
In non-coverage builds (that is to say, in release builds) the
testcase() macro is a no-op:

#define testcase(X)


But in a coverage measuring build, the testcase() macro
generates code that evaluates the conditional expression in its argument.
Then during analysis, a check
is made to ensure tests exist that evaluate the conditional to both true
and false.  Testcase() macros are used, for example, to help verify
that boundary values are tested.  For example:

testcase( a==b );
testcase( a==b+1 );
if( a>b && c!=25 ){ d++; }


Testcase macros are also used when two or more cases of a switch
statement go to the same block of code, to make sure that the code was
reached for all cases:

switch( op ){
  case OP_Add:
  case OP_Subtract: {
    testcase( op==OP_Add );
    testcase( op==OP_Subtract );
    /* ... */
    break;
  }
  /* ... */
}


For bitmask tests, testcase() macros are used to verify that every
bit of the bitmask affects the outcome.  For example, in the following block
of code, the condition is true if the mask contains either of two bits
indicating either a MAIN_DB or a TEMP_DB is being opened.
The testcase()
macros that precede the if statement verify that both cases are tested:

testcase( mask & SQLITE_OPEN_MAIN_DB );
testcase( mask & SQLITE_OPEN_TEMP_DB );
if( (mask & (SQLITE_OPEN_MAIN_DB|SQLITE_OPEN_TEMP_DB))!=0 ){ ... }


The SQLite source code contains 1184
uses of the testcase() macro.


c2: How SQLite Is Tested
c3: 7.4. Branch coverage versus MC/DC
c5: testing.html#branch_coverage_versus_mc_dc
c4:
Two methods of measuring test coverage were described above:
"statement" and "branch" coverage.  There are many other test coverage
metrics besides these two.  Another popular metric is "Modified
Condition/Decision Coverage" or MC/DC.
Wikipedia
defines MC/DC as follows:


 Each decision tries every possible outcome.
 Each condition in a decision takes on every possible outcome.
 Each entry and exit point is invoked.
 Each condition in a decision is shown to independently
     affect the outcome of the decision.


In the C programming language
where &amp;&amp; and ||
are "short-circuit" operators, MC/DC and branch coverage are very nearly
the same thing.  The primary difference is in boolean vector tests.
One can test for any of several bits in bit-vector and still obtain
100% branch test coverage even though the second element of MC/DC - the
requirement that each condition in a decision take on every possible outcome -
might not be satisfied.

SQLite uses testcase() macros as described in the previous
subsection to make sure that every condition in a bit-vector decision takes
on every possible outcome.  In this way, SQLite also achieves 100% MC/DC
in addition to 100% branch coverage.


c2: How SQLite Is Tested
c3: 7.5. Measuring branch coverage
c5: testing.html#measuring_branch_coverage
c4:
Branch coverage in SQLite is currently measured
using gcov with the "-b"
option.  First the test program is compiled using options
"-g -fprofile-arcs -ftest-coverage" and then the test program is run.
Then "gcov -b" is run to generate a coverage report.
The coverage report is verbose and inconvenient to read,
so the gcov-generated report is processed using
some simple scripts to put it into a more human-friendly format.
This entire process is automated using scripts, of course.

Note that running SQLite with gcov is not a test of SQLite &mdash;
it is a test of the test suite.  The gcov run does not test SQLite because
the -fprofile-args and -ftest-coverage options cause the compiler to
generate different code.
The gcov run merely verifies that the test suite provides 100% branch test
coverage.  The gcov run is a test of the test - a meta-test.

After gcov has been run to verify 100% branch test coverage,
then the test program is recompiled using delivery compiler options
(without the special -fprofile-arcs and -ftest-coverage options)
and the test program is rerun.
This second run is the actual test of SQLite.

It is important to verify that the gcov test run
and the second real test run both give the same output.  Any
differences in output indicate either the use of undefined or
indeterminate behavior in the SQLite code (and hence a bug),
or a bug in the compiler.
Note that SQLite has, over the previous decade, encountered bugs
in each of GCC, Clang, and MSVC.  Compiler bugs, while rare, do happen,
which is why it is so important to test the code in an as-delivered
configuration.


c2: How SQLite Is Tested
c3: 7.6. Mutation testing
c5: testing.html#mutation_testing
c4:
Using gcov (or similar) to show that every branch instruction is taken
at least once in both directions is good measure of test suite quality.
But even better is showing that every branch instruction makes
a difference in the output.  In other words, we want to show
not only that every branch instruction both jumps and falls through but also
that every branch is doing useful work and that the test suite is able
to detect and verify that work.  When a branch is found that does not
make a difference in the output, that suggests that code associated with
the branch can be removed (reducing the size of the library and perhaps
making it run faster) or that the test suite is inadequately testing the
feature that the branch implements.

SQLite strives to verify that every branch instruction makes a difference
using mutation testing.
A script
first compiles the SQLite source code into assembly language
(using, for example, the -S option to gcc).  Then the script steps through
the generated assembly language and, one by one, changes each branch
instruction into either an unconditional jump or a no-op, compiles the
result, and verifies that the test suite catches the mutation.


Unfortunately, SQLite contains many branch instructions that
help the code run faster without changing the output.
Such branches generate false-positives during mutation testing.
As an example, consider the following
hash function
used to accelerate table-name lookup:

55  static unsigned int strHash(const char *z){
56    unsigned int h = 0;
57    unsigned char c;
58    while( (c = (unsigned char)*z++)!=0 ){     /*OPTIMIZATION-IF-TRUE*/
59      h = (h&lt;&lt;3) &#94; h &#94; sqlite3UpperToLower&#91;c&#93;;
60    }
61    return h;
62  }



If the branch instruction that implements the "c!=0" test on line 58
is changed into a no-op, then the while-loop will loop forever and the
test suite will fail with a time-out.  But if that branch is changed
into an unconditional jump, then the hash function will always return 0.
The problem is that 0 is a valid hash.  A hash function that always
returns 0 still works in the sense that SQLite still always gets the correct
answer.  The table-name hash table degenerates into a linked-list
and so the table-name lookups that occur while parsing SQL statements
might be a little slower, but the end result will be the same.


To work around this problem, comments of the form
"/*OPTIMIZATION-IF-TRUE*/" and
"/*OPTIMIZATION-IF-FALSE*/" are inserted into the SQLite
source code to tell the mutation testing script to ignore some branch
instructions.


c2: How SQLite Is Tested
c3: 7.7. Experience with full test coverage
c5: testing.html#experience_with_full_test_coverage
c4:
The developers of SQLite have found that full coverage testing is an
extremely effective method for locating and preventing bugs.
Because every single branch
instruction in SQLite core code is covered by test cases, the developers
can be confident that changes made in one part of the code
do not have unintended consequences in other parts of the code.
The many new features and performance improvements that have been
added to SQLite in recent years would not have been possible without
the availability of full-coverage testing.

Maintaining 100% MC/DC is laborious and time-consuming.
The level of effort needed to maintain full-coverage testing
is probably not cost effective for a typical application.
However, we think that full-coverage testing is justified for a
very widely deployed infrastructure library
like SQLite, and especially for a database library which by its very
nature "remembers" past mistakes.


c2: How SQLite Is Tested
c3: 8. Dynamic Analysis
c5: testing.html#dynamic_analysis
c4:
Dynamic analysis refers to internal and external checks on the
SQLite code which are performed while the code is live and running.
Dynamic analysis has proven to be a great help in maintaining the
quality of SQLite.


c2: How SQLite Is Tested
c3: 8.1. Assert
c5: testing.html#assert
c4:
The SQLite core contains 6754 assert()
statements that verify function preconditions and postconditions and
loop invariants.  Assert() is a macro which is a standard part of
ANSI-C.  The argument is a boolean value that is assumed to always be
true.  If the assertion is false, the program prints an error message
and halts.

Assert() macros are disabled by compiling with the NDEBUG macro defined.
In most systems, asserts are enabled by default.  But in SQLite, the
asserts are so numerous and are in such performance critical places, that
the database engine runs about three times slower when asserts are enabled.
Hence, the default (production) build of SQLite disables asserts.
Assert statements are only enabled when SQLite is compiled with the
SQLITE_DEBUG preprocessor macro defined.

See the Use Of assert in SQLite document
for additional information about how SQLite uses assert().


c2: How SQLite Is Tested
c3: 8.2. Valgrind
c5: testing.html#valgrind
c4:
Valgrind is perhaps the most amazing
and useful developer tool in the world.  Valgrind is a simulator - it simulates
an x86 running a Linux binary.  (Ports of Valgrind for platforms other
than Linux are in development, but as of this writing, Valgrind only
works reliably on Linux, which in the opinion of the SQLite developers
means that Linux should be the preferred platform for all software development.)
As Valgrind runs a Linux binary, it looks for all kinds of interesting
errors such as array overruns, reading from uninitialized memory,
stack overflows, memory leaks, and so forth.  Valgrind finds problems
that can easily slip through all of the other tests run against SQLite.
And, when Valgrind does find an error, it can dump the developer directly
into a symbolic debugger at the exact point where the error occur, to
facilitate a quick fix.

Because it is a simulator, running a binary in Valgrind is slower than
running it on native hardware.  (To a first approximation, an application
running in Valgrind on a workstation will perform about the same as it
would running natively on a smartphone.)  So it is impractical to run the full
SQLite test suite through Valgrind.  However, the veryquick tests and
the coverage of the TH3 tests are run through Valgrind prior to every
release.


c2: How SQLite Is Tested
c3: 8.3. Memsys2
c5: testing.html#memsys2
c4:
SQLite contains a pluggable
memory allocation subsystem.
The default implementation uses system malloc() and free().
However, if SQLite is compiled with SQLITE_MEMDEBUG, an alternative
memory allocation wrapper (memsys2)
is inserted that looks for memory allocation
errors at run-time.  The memsys2 wrapper checks for memory leaks, of
course, but also looks for buffer overruns, uses of uninitialized memory,
and attempts to use memory after it has been freed.  These same checks
are also done by valgrind (and, indeed, Valgrind does them better)
but memsys2 has the advantage of being much faster than Valgrind, which
means the checks can be done more often and for longer tests.


c2: How SQLite Is Tested
c3: 8.4. Mutex Asserts
c5: testing.html#mutex_asserts
c4:
SQLite contains a pluggable mutex subsystem.  Depending on
compile-time options, the default mutex system contains interfaces
sqlite3_mutex_held() and sqlite3_mutex_notheld() that detect
whether or not a particular mutex is held by the calling thread.
These two interfaces are used extensively within assert() statements
in SQLite to verify mutexes are held and released at all the right
moments, in order to double-check that SQLite does work correctly
in multi-threaded applications.


c2: How SQLite Is Tested
c3: 8.5. Journal Tests
c5: testing.html#journal_tests
c4:
One of the things that SQLite does to ensure that transactions
are atomic across system crashes and power failures is to write
all changes into the rollback journal file prior to changing the
database.  The TCL test harness contains an alternative
OS backend implementation that helps to
verify this is occurring correctly.  The "journal-test VFS" monitors
all disk I/O traffic between the database file and rollback journal,
checking to make sure that nothing is written into the database
file which has not first been written and synced to the rollback journal.
If any discrepancies are found, an assertion fault is raised.

The journal tests are an additional double-check over and above
the crash tests to make sure that SQLite transactions will be atomic
across system crashes and power failures.


c2: How SQLite Is Tested
c3: 8.6. Undefined Behavior Checks
c5: testing.html#undefined_behavior_checks
c4:
In the C programming language, it is very easy to write code that
has "undefined" or "implementation defined" behavior.
That means that the code might work during development, but then give
a different answer on a different system, or when recompiled using different
compiler options.
Examples of undefined and implementation-defined behavior in
ANSI C include:

Signed integer overflow.  (Signed integer overflow does not
necessarily wrap around, as most people expect.)
Shifting an N-bit integer by more than N bits.
Shifting by a negative amount.
Shifting a negative number.
Using the memcpy() function on overlapping buffers.
The order of evaluation of function arguments.
Whether or not "char" variables are signed or unsigned.
And so forth....


Since undefined and implementation-defined behavior is non-portable
and can easily lead to incorrect answers, SQLite works very hard to avoid it.
For example,
when adding two integer column values together as part of an SQL statement,
SQLite does not simply add them together using the C-language "+" operator.
Instead, it first checks to make sure the
addition will not overflow, and if it will, it does the addition using
floating point instead.

To help ensure that SQLite does not make use of undefined or
implementation defined behavior, the test suites are rerun using
instrumented builds that try to detect undefined behavior.  For example,
test suites are run using the "-ftrapv" option of GCC.  And they
are run again using the "-fsanitize=undefined" option on Clang.  And
again using the "/RTC1" option in MSVC.  Then the test suites are rerun
using options like "-funsigned-char" and "-fsigned-char" to make sure
that implementation differences do not matter either.  Tests are then repeated
on 32-bit and 64-bit systems and on big-endian and little-endian systems,
using a variety of CPU architectures.
Furthermore, the test suites are augmented with many test cases that are
deliberately designed to provoke undefined behavior.  For example:
"SELECT -1*(-9223372036854775808);".


c2: How SQLite Is Tested
c3: 9. Disabled Optimization Tests
c5: testing.html#disabled_optimization_tests
c4:
The sqlite3_test_control(SQLITE_TESTCTRL_OPTIMIZATIONS, ...) interface
allows selected SQL statement optimizations to be disabled at run-time.
SQLite should always generate exactly the same answer with optimizations
enabled and with optimizations disabled; the answer simply arrives quicker
with the optimizations turned on.  So in a production environment, one always
leaves the optimizations turned on (the default setting).

One verification technique used on SQLite is to run an entire test suite
twice, once with optimizations left on and a second time with optimizations
turned off, and verify that the same output is obtained both times.  This
shows that the optimizations do not introduce errors.

Not all test cases can be handled this way.  Some test cases check
to verify that the optimizations really are reducing the amount of
computation by counting the number of disk accesses, sort operations,
full-scan steps, or other processing steps that occur during queries.
Those test cases will appear to fail when optimizations are disabled.
But the majority of test cases simply check that the correct answer
was obtained, and all of those cases can be run successfully with and
without the optimizations, in order to show that the optimizations do not
cause malfunctions.


c2: How SQLite Is Tested
c3: 10. Checklists
c5: testing.html#checklists
c4:
The SQLite developers use an on-line checklist to coordinate testing
activity and to verify that all tests pass prior each SQLite release.
Past checklists
are retained for historical reference.
(The checklists are read-only for anonymous internet viewers, but
developers can log in and update checklist items in their web
browsers.)
The use of checklists for SQLite testing and other development activities
is inspired by 
The Checklist Manifesto
.

The latest checklists contain approximately 200 items that are
individually verified for each release.  Some checklist items only take
a few seconds to verify and mark off.  Others involve test suites
that run for many hours.

The release checklist is not automated: developers run each item on
the checklist manually.  We find that it is important to keep a human in
the loop.  Sometimes problems are found while running a checklist item
even though the test itself passed.  It is important to have a human
reviewing the test output at the highest level, and constantly asking
"Is this really right?"

The release checklist is continuously evolving.  As new problems or
potential problems are discovered, new checklist items are added to
make sure those problems do not appear in subsequent releases.  The
release checklist has proven to be an invaluable tool in helping to
ensure that nothing is overlooked during the release process.


c2: How SQLite Is Tested
c3: 11. Static Analysis
c5: testing.html#static_analysis
c4:
Static analysis means analyzing source code at compile-time to
check for correctness.  Static analysis includes compiler
warning messages and more in-depth analysis engines such as the
Clang Static Analyzer.
SQLite compiles without warnings on GCC and Clang using
the -Wall and -Wextra flags on Linux and Mac and on MSVC on Windows.
No valid warnings are generated by the Clang Static Analyzer tool "scan-build"
either (though recent versions of clang seem to generate many false-positives.)
Nevertheless, some warnings might be generated by other
static analyzers.  Users are encouraged not to stress over these
warnings and to instead take solace in the intense testing of SQLite
described above.


Static analysis has not been helpful in finding
bugs in SQLite.  Static analysis has found a few bugs in SQLite, but
those are the exceptions.  More bugs have been
introduced into SQLite while trying to get it to compile without
warnings than have been found by static analysis.


c2: How SQLite Is Tested
c3: 12. Summary
c5: testing.html#summary
c4:
SQLite is open source.  This gives many people the idea that
it is not well tested as commercial software and is perhaps unreliable.
But that impression is false.
SQLite has exhibited very high reliability in the field and
a very low defect rate, especially considering how rapidly it is evolving.
The quality of SQLite is achieved in part by careful code design and
implementation.  But extensive testing also plays a vital role in
maintaining and improving the quality of SQLite.  This document has
summarized the testing procedures that every release of SQLite undergoes
with the hope of inspiring confidence that SQLite is
suitable for use in mission-critical applications.


c2: Version Numbers in SQLite
c3: 1. SQLite Version Numbers
c5: versionnumbers.html#sqlite_version_numbers
c4:
Beginning with version 3.9.0 (2015-10-14) SQLite uses 
semantic versioning.
Prior to that time, SQLite employed a version identifier that
contained between two and four numbers.


c2: Version Numbers in SQLite
c3: 1.1. The New Version Numbering System (After 2015-10-14)
c5: versionnumbers.html#the_new_version_numbering_system_after_dateof_3_9_0_
c4:
All SQLite releases starting with 3.9.0 use a three-number
"semantic version" of the form X.Y.Z.
The first number X is only increased when there is a change that
breaks backward compatibility.  The
current value for X is 3, and the SQLite developers plan to support
the current SQLite database file format, SQL syntax, and C interface
through at least the year 2050.  Hence, one
can expect that all future versions of SQLite for the next several
decades will begin with "3.".


The second number Y is incremented for any change that breaks forward
compatibility by adding new features.
Most future SQLite releases are expected
to increment the second number Y.  The Z is reset to zero whenever Y
is increased.


The third number Z is incremented for releases consisting of only
small changes that implement performance enhancements and/or bug fixes.


The rate of enhancement for SQLite over the previous five years
(2010-2015) is approximately 6 increments of Y per year.  The
numbering format used by for SQLITE_VERSION_NUMBER and
sqlite3_libversion_number() allows versions up to 3.999.999, which is
more than enough for the planned end-of-support date for SQLite
in 2050.  However, the current tarball naming conventions only
reserve two digits for the Y and so the naming format for downloads
will need to be revised in about 2030.


c2: Version Numbers in SQLite
c3: 1.2. The Historical Numbering System (Before 2015-10-14)
c5: versionnumbers.html#the_historical_numbering_system_before_dateof_3_9_0_
c4:
This historical version numbering system used a two-, three-,
or four-number version:  W.X, W.X.Y, or W.X.Y.Z.
W was the file format: 1 or 2 or 3.
X was the major version.
Y was the minor version.
Z was used only for patch releases to fix bugs.


There have been three historical file formats for SQLite.
SQLite 1.0 through 1.0.32 used the
gdbm library as its storage
engine.
SQLite 2.0.0 through 2.8.17 used a custom b-tree storage engine that
supported only text keys and data.
All modern versions of SQLite (3.0.0 to present) use a b-tree storage
engine that has full support for binary data and Unicode.


This major version number X was historically incremented only for
large and important changes to the code.  What constituted "large
and important" was subjective.  The 3.6.23 to 3.7.0 change
was a result of adding support for WAL mode.
The 3.7.17 to 3.8.0 change was a result of rewrite known as the
next generation query planner.


The minor version number Y was historically incremented for new
features and/or new interfaces that did not significantly change
the structure of the code.  The addition of common table expressions,
partial indexes, and indexes on expressions are all examples of
"minor" changes.  Again, the distinction between "major" and "minor"
is subjective.


The patch level Z was historically only used for bug-fix releases
that changed only a small number of code lines.


c2: Vulnerabilities
c3: 1. Executive Summary
c5: cves.html#executive_summary
c4:
CVEs about SQLite probably do not apply to your use of SQLite.


All historical vulnerabilities reported against SQLite require at least
one of these preconditions:


The attacker can submit and run arbitrary SQL statements.

The attacker can submit a maliciously crafted database file to the
application that the application will then open and query.



Few real-world applications meet either of these preconditions, and hence
few real-world applications are vulnerable, even if they use older
and unpatched versions of SQLite.


The SQLite development team fixes bugs promptly,
usually within hours of discovery.  New releases of SQLite
are issued if the bug seems likely to impact real-world
applications.


Grey-hat hackers are rewarded based on the number and severity of 
CVEs that they write.  This results in a proliferation of CVEs that
have minor impact, or no impact at all, but which make
exaggerated impact claims.


Very few CVEs written about SQLite are real vulnerabilities in the 
sense that they do not give any new capabilities to an attacker.
Consider:


    Almost all CVEs written against SQLite require the ability to
    inject and run arbitrary SQL.

    The advertised consequence of most CVEs is "denial of service",
    typically by causing a crash through a NULL pointer dereference or
    a division by zero, or similar.

    But if an attacker can already run
    arbitrary SQL, they do not need a bug to cause a denial of service.
    There are plenty of perfectly legal and valid SQL statements
    that will consume unlimited CPU, memory, and disk I/O in order
    to create a denial-of-service without requiring help from bugs.

    Hence, the mere fact that an attacker has a way to inject and run
    arbitrary SQL is in and of itself a denial-of-service attack.  That
    the arbitrary SQL might also tickle a bug in SQLite and cause a
    crash is not a new vulnerability.



The SQLite developers do not write CVEs.  Any CVEs you find on
SQLite are generated by third-parties, often without any input from the
core developers.  A common scenario is that someone will report a bug in
SQLite, which will promptly be fixed, then weeks later a CVE for that bug will
appear, unbeknownst to the developers.


You should not assume that a CVE about
SQLite contains authoritative information.
CVEs often contain inaccuracies.
The SQLite developers have attempted to add clarifications and
corrections to CVEs about SQLite.


c2: Vulnerabilities
c3: 2. About CVEs
c5: cves.html#about_cves
c4:
CVEs ("Common Vulnerabilities and Exposures") are reports of software
bugs that might allow a system to be hacked.  The idea
behind CVEs is sound.  They provide a common naming scheme whereby 
software bugs that might compromise information security can be easily
tracked.

While the original idea being CVEs is sound, the current processes for
creating and managing CVEs are inadequate.  There are countless grey-hat
hackers running fuzzers against a wide-variety of open-source software
products (SQLite as well as many others) and writing up CVEs against
any problems they find.  The grey-hats are rewarded, sometimes with
prestige and sometimes financially, by the number and severity of
the CVEs they write.  This incentive results in a proliferation
of CVEs which are often not well-vetted and which can have exaggerated
impact claims.  The quality-control procedures for CVEs are unable
to cope with this flood of inputs, making it difficult to correct
exaggerated, misleading, omitted, or inaccurate claims.

This is not to say that CVEs are useless.  CVEs do still (mostly)
report actual bugs.  But in most cases the bugs are not true vulnerabilities,
in the sense that they do not contribute to data loss or compromise
in and of themselves.
It is good that bugs are reported and fixed.  But not every bug is
accessible from every application.  In the case of SQLite, most of the
bugs reported by CVEs are inaccessible in most applications.  Upgrading
to the latest version of SQLite is always a good plan, but it need not
be an emergency just because an anonymous grey-hat on the internet
wrote up a CVE.


c2: Vulnerabilities
c3: 2.1. A separate SQL injection vulnerability is usually required
c5: cves.html#a_separate_sql_injection_vulnerability_is_usually_required
c4:
Other C-libraries that process complex structured inputs will
routinely be asked to deal with unvetted inputs from untrusted
sources.  Libraries like libjpeg, or libzip, or OpenSSL are
handed input streams that come directly from potentially hostile
agents.


But database engines like SQLite are usually not this way.
The SQL scripts that are passed into SQLite come from the
(trusted) application itself, not from an attacker.  Sometimes
applications contain bugs by which an external attacker can
trick the application into sending SQL of the attackers design
into the database engine.  This is a separate bug in the
application called an
SQL Injection
vulnerability.  Since SQL text is executable code, an
SQL Injection vulnerability is actually a special case of a
Remote
Code Execution (RCE) vulnerability.  An SQL Injection is perhaps not
quite as bad as other kinds of RCEs because,
while SQL is a powerful language, it is not as convenient
for crafting an exploit as Python or shell script or raw machine code.
Nevertheless, an SQL Injection is a serious problem.


Most CVEs written about SQLite assume that the attacker is
able to run arbitrary SQL scripts in SQLite.  In most applications,
this means that there must first be an SQL Injection vulnerability
that allows the attacker to inject the malicious SQL.


A few applications do allow untrusted SQL scripts received from
potentially hostile agents to be run direct in SQLite.  The main
example of this is the Chrome and Safari web browsers, which allow
an anonymous web page to run SQL using the WebSQL feature of Javascript.
This is done inside a sandbox with tightly controlled constraints on
resources, lest the SQL script try to soak up all available memory
or CPU cycles in a denial-of-service attack.  Chrome and Safari
have the infrastructure in place to allow a hostile agent to run
code which does not harm or compromise the rest of the machine.
They have to, as they also run Javascript which could, if not
tightly controlled, do even more damage than unrestrained SQL.
Apart from Chrome and Safari, no applications known to the
SQLite developers deliberately allows an anonymous remote agent
to run arbitrary SQL text.

However, most CVEs written against SQLite flippantly assume
that an attacker is free to run any arbitrary SQL in the database
engine.  So to a good approximation, this means most CVEs
written against SQLite really only apply to SQLite as it is
used in Chrome and Safari.  Or, in other words, most CVEs
for SQLite do not apply to you unless you are one of the
developers of Chrome or Safari.


c2: Vulnerabilities
c3: 2.2. Defense Against Dark Arts
c5: cves.html#defense_against_dark_arts
c4:
Most applications can use SQLite without having to worry about
bugs in obscure SQL inputs.  If the application controls
the SQL, and the application is not deliberately trying to break
SQLite, then everything should just work.
It is not necessary to have the latest patched version of SQLite.
Any older version should work just fine.


However, there are some occasions where an application does need
to be able to safely run untrusted SQL. The SQLite developers work hard
to make SQLite safe for this purpose, though there are occasional
slip-ups.  It is good to keep up-to-date with the latest patches
in this case.  The separate defense against dark arts document
contains additional suggestions that can help prevent zero-day
attacks in cases where SQLite is given inputs that come directly
from untrusted sources.


c2: Vulnerabilities
c3: 2.3. The SQLite Developer Policy Toward CVEs
c5: cves.html#the_sqlite_developer_policy_toward_cves
c4:
SQLite developers fix all bugs in SQLite as soon as they are reported,
usually within a few hours.  The fixes are immediately available on the
public SQLite source tree.
If a bug seems like it might cause problems for existing applications,
a new patch release for SQLite will be issued.

However, the SQLite developers do not track CVEs.  There are 
various reasons for this:



The developers often do not find out about CVEs until long after the
bug is fixed.  You can see this by the fact that many CVEs reference the
bug fix in their initial report.


CVEs are a low-quality source of information about bugs in SQLite
that are likely to affect most applications.


Almost all bugs reported by CVEs are just bugs and not
true vulnerabilities.  Claiming that they are vulnerabilities is
stretching the meaning of the word "vulnerability" and the SQLite
developers do not wish to participate in that deception.


The developers have no editorial influence on the content of CVEs,
and they do not like to be controlled by groups in which they have
no voice.


c2: Vulnerabilities
c3: 3. Status Of Recent SQLite CVEs
c5: cves.html#status_of_recent_sqlite_cves
c4:
Though the SQLite developers do not consider CVEs to be a reliable
source of information about bugs in SQLite, they recognize that many
groups, and especially small teams working at the bottom of tall
bureaucracies, sometimes need to track CVEs, whether they are useful
or not.  To aid in this chore, the following table of recent CVEs
affecting SQLite is provided.

If you notice new CVEs associated with SQLite that are not in
the table below, please bring them to the attention of the developers
on the SQLite Forum so they can
be added.




CVE Number
Fix
Comments





CVE-2024-0232

3.43.2(2023-10-10)
An attacker that can inject arbitrary SQL statements into an application
  might be able to provoke a use-after-free bug in SQLite's JSON parser that
  can (in theory) lead to an application crash and denial of service. See
  forum thread b25edc1d4662
  for the bug report.



CVE-2023-32697

Not a bug in SQLite
This is a bug in the SQLite JDBC library, which is a wrapper library that
  provides access to SQLite from Java.  SQLite JDBC is created and maintained
  independently from SQLite.  Despite the use of "SQLite" in the name, the
  SQLite JDBC library is not affiliated with the SQLite project in any way.
  The vulnerability identified by this CVE is in the separate SQLite JDBC
  wrapper library and does not affect SQLite itself.



CVE-2023-39939

Not a bug in SQLite
This is not a bug in SQLite.  This is an 
  SQL injection bug in an application
  (LuxCal Web Calendar) that links against SQLite.
  Even though this CVE is not about SQLite, "SQLite" is
  mentioned in the description and so we list it here.



CVE-2023-39543

Not a bug in SQLite
This is not a bug in SQLite.  This is an
  XSS vulnerability
  in a separate application (LuxCal Web Calendar) that links against
  SQLite.  The bug is in the application, not in SQLite.  However "SQLite" is
  mentioned in the description and so we list it here.



CVE-2023-7104

3.43.1(2023-09-11)
This is a bug in the session extension of SQLite, not in the SQLite core.
  This bug is only reachable by applications that recompile SQLite using the
  -DSQLITE_ENABLE_SESSION compile-time option and then use the Session
  C-language APIs to process a changeset that has been subtly corrupted by an
  adversary.  So this bug probably does not apply to you.  See
  forum post f935c4708dd528d9
  for additional information.



CVE-2022-46908

Not a bug in the core SQLite library
This is a bug in the --safe command-line option of the command-line shell
  program that is available for accessing SQLite database files.  The bug does
  not exist in the SQLite library.  Nor is it an issue for the CLI as long as
  the user does not depend on the --safe option.  It is not serious.  It is
  debatable whether or not this is a security issue.



CVE-2022-38627

Not a bug in SQLite
This is not a bug in SQLite.  This is an 
  SQL injection bug in a specific
  PHP application.  In other words, the bug is in the PHP application code, not
  in SQLite.  Even though this CVE is not about SQLite, "SQLite" is
  mentioned in the publicity about the bug and so we list it here.



CVE-2022-35737

3.39.2(2022-07-21)
This bug is an array-bounds overflow.  The bug is only accessible when using some
  of the C-language APIs provided by SQLite.  The bug cannot be reached using SQL
  nor can it be reached by providing SQLite with a corrupt database file.
  The bug only comes up when very long string inputs (greater than 2 billion bytes
  in length) are provided as arguments to a few specific C-language interfaces,
  and even then only under special circumstances.



CVE-2022-24854

Not a bug in SQLite
This CVE describes a bug in an application that uses SQLite, not in SQLite itself.
  SQLite is doing everything correctly.  The application grants users the ability to
  run SQL statements, using SQLite, that can leak or change information that those users
  should not normally have access to.  This is purely an application bug.  It does not
  describe a malfunction or vulnerability in SQLite.



CVE-2022-21227

Not a bug in SQLite
This CVE describes a bug in a third-party packages that provides a binding
  for SQLite to Node.js.  The bug reported is in the third-party Node.js binding,
  not in SQLite itself.  Do not be confused by the use of the word "SQLite" in the
  ambiguously-worded CVE description.



CVE-2021-45346

Not a bug in SQLite
This CVE is misinformation.  See the discussion around
  SQLite forum post 53de8864ba114bf.



CVE-2021-42169

Not a bug in SQLite
This CVE has nothing whatsoever to do with SQLite.  It is about a bug in
  application that happens to use SQLite.  Since SQLite is mentioned in the 
  CVE description, the CVE is included here to emphasize that
  this is not an SQLite bug.



CVE-2021-36690

Not a bug in the core SQLite library
This bug is not in the SQLite core library, but rather in an
  
  experimental extension that is used to implement the
  .expert command in the CLI.  The code that contains the bug
  does not appear in standard SQLite builds, though it
  is included in the sqlite3.exe command-line tool.
  Applications must link against the extra source code files that
  implement the extension and take other deliberate actions to
  activate the extension before the troublesome code can be run.
  For the rare application that uses the troublesome extension,
  the consequence of this bug is that malicious SQL can cause a
  NULL pointer deference and denial of service.
(details)



CVE-2021-28305

Not a bug in SQLite
This is not a bug in SQLite.  The bug is in a third-party application that
  uses SQLite.  SQLite is mentioned by name in the CVE description,
  however, so we have included the CVE in the list.



CVE-2021-23404

Not a bug in SQLite
This is not a bug in SQLite.  The bug is in a third-party application that
  uses SQLite and includes "sqlite" in its name.  This CVE is included on the
  list because it mentions SQLite even though the bug has nothing to do
  with SQLite.



CVE-2021-20227

3.34.1(2021-01-20)
Malicious SQL statement causes read-after-free.  No harm can come of this
  particular read-after-free instance, as far as anyone knows.  The bug is
  undetectable without a memory sanitizer. The CVE
  claims that this bug is an RCE - a Remote Code Execution
  vulnerability, but that claim is incorrect.
  The RCE claim is misinformation.
(details)



CVE-2021-20223

3.34.0(2020-12-01)
The problem identified by this CVE is not a vulnerability. 
  It is a malfunction. A coding error causes FTS5
  to sometimes return inconsistent and incorrect results under obscure circumstances,
  but no memory errors occur.
(details)



CVE-2020-15358

3.32.3(2020-06-18)
Malicious SQL statement causes a read past the end of a heap buffer.
(details)



CVE-2020-13871

3.32.3(2020-06-18)
Malicious SQL statement causes a read-only use-after-free memory error.
(details)



CVE-2020-13632

3.32.0(2020-05-22)
Malicious SQL statement causes a read of a NULL pointer in the
  matchinfo() SQL function of the FTS3 extension, resulting in
  denial of service.
(details)



CVE-2020-13631

3.32.0(2020-05-22)
Malicious SQL statement (an ALTER TABLE that tries to rename a
  virtual table into one of its own shadow tables)
  causes an infinite loop and denial of service.
(details)



CVE-2020-13630

3.32.0(2020-05-22)
Malicious SQL statement causes a read-only use-after-free,
  possibly resulting in an incorrect output from the snippet()
  SQL function of the FTS3 extension.  There is no known
  way to exfiltrate data or crash the application using this bug.
(details)



CVE-2020-13435

3.32.1(2020-05-25)
Malicious SQL statement causes a read access to a NULL pointer and
  denial of service.
(details)



CVE-2020-13434

3.32.1(2020-05-25)
Malicious SQL statement involving the printf() SQL function results
  in an integer overflow which can overwrite the stack with over 2
  billion bytes of 0x30 or 0x20 (ASCII '0' or ' ').
  Even though this is a stack overwrite, there is no known way to
  redirect control or otherwise escalate the level of harm.
  This is a denial-of-service attack only.
(details)



CVE-2020-11656

3.32.0(2020-05-22)
Malicious SQL statement causes read-only use-after-free of memory allocation
  if SQLite is compile with -DSQLITE_DEBUG.  Does not affect release
  builds.
(details)



CVE-2020-11655

3.32.0(2020-05-22)
Malicious SQL statement causes a read using an uninitialized pointer
  and denial-of-service.
(details)



CVE-2020-9327

3.32.0(2020-05-22)
Malicious SQL statement causes a read using an uninitialized pointer
  and denial-of-service
(details)



CVE-2020-6405

3.31.0(2020-01-22)
Malicious SQL statement causes a NULL pointer dereference and
  denial-of-service
(details)



CVE-2019-20218

3.31.0(2020-01-22)
Malicious SQL statement causes an uninitialized pointer read and
  denial-of-service.
(details)



CVE-2019-19959

3.31.0(2020-01-22)
Malicious SQL statement causes a NULL pointer dereference
  in the Zipfile virtual table extension and
  denial-of-service.  This is only possible when the optional
  Zipfile virtual table extension is deployed, which is not
  the case in default builds.
(details)



CVE-2019-19926

3.31.0(2020-01-22)
Malicious SQL statement causes an uninitialized pointer read and
  denial-of-service.
(details)



CVE-2019-19925

3.31.0(2020-01-22)
Malicious SQL statement causes a NULL pointer dereference and
  in the Zipfile virtual table extension and
  denial-of-service.  This is only possible when the optional
  Zipfile virtual table extension is deployed, which is not
  the case in default builds.
(details)



CVE-2019-19924

3.31.0(2020-01-22)
Malicious SQL statement causes an uninitialized pointer reference and
  denial-of-service.
(details)



CVE-2019-19923

3.31.0(2020-01-22)
Malicious SQL statement causes a NULL pointer dereference and
  denial-of-service.
(details)



CVE-2019-19646

3.31.0(2020-01-22)
The PRAGMA integrity_check command might cause the byte-code for a prepared
  statement to loop indefinitely.  This might enable a denial-of-service, if the 
  application has not taken appropriate and prudent steps
  to limit the run-time of SQL statements.  This is not a vulnerability, as there
  are countless perfectly valid SQL queries, especially queries involving
  recursive common table expressions, that also run essentially forever.
(details)



CVE-2019-19317

3.31.0(2020-01-22)
This CVE identifies a bug in a development check-in of
  SQLite.  The bug never appeared in any official SQLite release.
(details)


c2: LoC Recommended Storage Format
c3: 1. Overview
c5: locrsf.html#overview
c4:
SQLite is a
Recommended Storage Format
for datasets according to the
US Library of Congress.
Further information:




https://www.loc.gov/preservation/digital/formats/fdd/fdd000461.shtml#local

https://www.loc.gov/preservation/resources/rfs/data.html



As of this writing (2018-05-29) the only other recommended storage formats
for datasets are XML, JSON, and CSV.


c2: LoC Recommended Storage Format
c3: 1.1. What Is A Recommended Storage Format?
c5: locrsf.html#what_is_a_recommended_storage_format_
c4:
Recommended storage formats are formats which, in the opinion of the
preservationists at the Library of Congress, maximizes the chance of
survival and continued accessibility of digital content.
When selecting recommended storage formats, the following criteria are
considered (quoting from the LOC website):


Disclosure.
Degree to which complete specifications and tools for validating 
technical integrity exist and are accessible to those creating and
sustaining digital content.
A spectrum of disclosure levels can be observed for digital formats.
What is most significant is not approval by a recognized standards body,
but the existence of complete documentation.
Adoption.
Degree to which the format is already used by the primary creators,
disseminators, or users of information resources. 
This includes use as a master format, for delivery to end users, 
and as a means of interchange between systems.
Transparency.
Degree to which the digital representation is open to direct analysis 
with basic tools, such as human readability using a text-only editor.
Self-documentation.
Self-documenting digital objects contain basic descriptive, 
technical, and other administrative metadata.
External Dependencies.
Degree to which a particular format depends on particular hardware,
operating system, or software for rendering or use and the predicted
complexity of dealing with those dependencies in future technical 
environments.
Impact of Patents.
Degree to which the ability of archival institutions to sustain content 
in a format will be inhibited by patents.
Technical Protection Mechanisms.
Implementation of mechanisms such as encryption that prevent the 
preservation of content by a trusted repository. 

This page last modified on  2022-01-08 05:02:57 UTC


c2: The Error And Warning Log
c3: Overview
c5: errlog.html#overview
c4:
SQLite can be configured to invoke a callback function containing
an error code and a terse error message whenever anomalies occur.
This mechanism is very helpful in tracking obscure problems that
occur rarely and in the field.  Application developers are encouraged
to take advantage of the error logging facility of SQLite in their
products, as it is very low CPU and memory cost but can be a
huge aid for debugging.


c2: The Error And Warning Log
c3: 1. Setting Up The Error Logging Callback
c5: errlog.html#setting_up_the_error_logging_callback
c4:
There can only be a single error logging callback per process.
The error logging callback is registered at start-time using C-code
similar to the following:


sqlite3_config(SQLITE_CONFIG_LOG, errorLogCallback, pData);


The error logger callback function might look something like this:


void errorLogCallback(void *pArg, int iErrCode, const char *zMsg){
  fprintf(stderr, "(%d) %s\n", iErrCode, zMsg);
}


The example above illustrates the signature of the error logger callback.
However, in an embedded application, one usually does not print
messages on stderr.  Instead, one might store the messages in a
preallocated circular buffer where they can be accessed when diagnostic
information is needed during debugging.  Or perhaps the messages can be
sent to Syslog.  Somehow, the
messages need to be stored where they are accessible to developers,
not displayed to end users.

Do not misunderstand: There is nothing technically wrong with displaying 
the error logger messages to end users.  The messages do not contain
sensitive or private information that must be protected from unauthorized
viewing.  Rather the messages are technical in nature and are not useful
or meaningful to the typical end user.  The messages coming from the
error logger are intended for database geeks.  Display them accordingly.


c2: The Error And Warning Log
c3: 2. Interface Details
c5: errlog.html#interface_details
c4:
The third argument to the sqlite3_config(SQLITE_CONFIG_LOG,...) 
interface (the "pData" argument in the example above) is a pointer to arbitrary
data.  SQLite passes this pointer through to the first argument of the
error logger callback.  The pointer can be used to pass application-specific 
setup or state information, if desired.  Or it can simply be a NULL 
pointer which is ignored by the callback.

The second argument to the error logger callback is an integer
extended error code.  The third argument to the error logger is the
text of the error message.  The error message text is stored in a fixed-length
stack buffer in the calling function and so will only be valid for the
duration of the error logger callback function.  The error logger should
make a copy of this message into persistent storage if retention of the
message is needed.

The error logger callback should be treated like a signal handler.
The application should save off or otherwise process the error, then return
as soon as possible.  No other SQLite APIs should be invoked, directly or
indirectly, from the error logger.  SQLite is not reentrant through
the error logger callback.  In particular, the error logger callback
is invoked when a memory allocation fails, so it is generally a bad idea
to try to allocate memory inside the error logger.  Do not even think
about trying to store the error message in another SQLite database.

Applications can use the sqlite3_log(E,F,..) API to send new messages
to the log, if desired, but this is discouraged.  The sqlite3_log()
interface is intended for use by extensions only, not by applications.


c2: The Error And Warning Log
c3: 3. Variety of Error Messages
c5: errlog.html#variety_of_error_messages
c4:
The error messages that might be sent to the error logger and their
exact format is subject to changes from one release to the next.  So
applications should not depend on any particular error message text formats or
error codes.  Things do not change capriciously, but they do sometimes
changes.

The following is a partial list of the kinds of messages that might
appear in the error logger callback.



Any time there is an error either compiling an SQL statement 
(using sqlite3_prepare_v2() or its siblings) or running an SQL
statement (using sqlite3_step()) that error is logged.



When a schema change occurs that requires a prepared statement to be reparsed
and reprepared, that event is logged with the error code SQLITE_SCHEMA.
The reparse and reprepare is normally automatic (assuming that
sqlite3_prepare_v2() has been used to prepare the statements originally,
which is recommended) and so these logging events are normally the only
way to know that reprepares are taking place.


SQLITE_NOTICE messages are logged whenever a database has to be recovered
because the previous writer crashed without completing its transaction.
The error code is SQLITE_NOTICE_RECOVER_ROLLBACK when recovering a
rollback journal and SQLITE_NOTICE_RECOVER_WAL when recovering a 
write-ahead log.



SQLITE_WARNING messages are logged when database files are renamed or
aliased in ways that can lead to database corruption.
(See 1 and 2 for
additional information.)



Out of memory (OOM) error conditions generate error logging events
with the SQLITE_NOMEM error code and a message that says how many bytes
of memory were requested by the failed allocation.


I/O errors in the OS-interface generate error logging events.
The message to these events gives the line number in the source code where
the error originated and the filename associated with the event when
there is a corresponding file. 

When database corruption is detected, an SQLITE_CORRUPT error
logger callback is invoked.  As with I/O errors, the error message text
contains the line number in the original source code where the error
was first detected.


An error logger callback is invoked on SQLITE_MISUSE errors.
This is useful in detecting application design issues when return codes
are not consistently checked in the application code.


SQLite strives to keep error logger traffic low and only send messages
to the error logger when there really is something wrong.  Applications
might further cull the error message traffic 
by deliberately ignoring certain classes of error
messages that they do not care about.  For example, an application that
makes frequent database schema changes might want to ignore all
SQLITE_SCHEMA errors.


c2: The Error And Warning Log
c3: 4. Summary
c5: errlog.html#summary
c4:
The use of the error logger callback is highly recommended.
The debugging information that the error logger provides has proven
very useful in tracking down obscure problems that occur with applications
after they get into the field.  The error logger callback has also 
proven useful in catching occasional errors that the application
misses because of inconsistent checking of API return codes.
Developers are encouraged to implement an error logger callback early
in the development cycle in order to spot unexpected behavior quickly,
and to leave the error logger callback turned on through deployment.
If the error logger never finds a problem, then no harm is done.  
But failure to set up an appropriate error logger might compromise
diagnostic capabilities later on.
This page last modified on  2022-04-18 02:55:50 UTC


c2: Using SQLite In Multi-Threaded Applications
c3: 1. Overview
c5: threadsafe.html#overview
c4:
SQLite supports three different threading modes:


Single-thread.
In this mode, all mutexes are disabled and SQLite is unsafe to use in
more than a single thread at once.

Multi-thread.
In this mode, SQLite can be safely used by multiple threads provided that
no single database connection nor any object derived from database connection,
such as a prepared statement,
is used in two or more threads at the same time.


Serialized.
In serialized mode, API calls to affect or use any SQLite database connection
or any object derived from such a database connection
can be made safely from multiple threads.
The effect on an individual object is the same as if the API calls had all
been made in the same order from a single thread.  The name "serialized"
arises from the fact that SQLite uses mutexes to serialize access to each
object.



The threading mode can be selected at compile-time (when the SQLite
library is being compiled from source code) or at start-time (when the
application that intends to use SQLite is initializing) or at
run-time (when a new SQLite database connection is being created).
Generally speaking, run-time overrides start-time and start-time
overrides compile-time.  Except, single-thread mode cannot be
overridden once selected.



The default mode is serialized.


c2: Using SQLite In Multi-Threaded Applications
c3: 2. Compile-time selection of threading mode
c5: threadsafe.html#compile_time_selection_of_threading_mode
c4:
Use the SQLITE_THREADSAFE compile-time parameter to select the
threading mode.  If no SQLITE_THREADSAFE compile-time parameter is
present, then serialized mode is used.
This can be made explicit with 
-DSQLITE_THREADSAFE=1.
With
-DSQLITE_THREADSAFE=0 the threading mode is
single-thread.  With
-DSQLITE_THREADSAFE=2 the threading mode is
multi-thread.



The return value of the sqlite3_threadsafe() interface is the value
of SQLITE_THREADSAFE set at compile-time. It does not reflect changes
to the threading mode made at runtime via the sqlite3_config()
interface or by flags given as the third argument to sqlite3_open_v2().



If single-thread mode is selected at compile-time, then critical
mutexing logic is omitted from the build and it is impossible to
enable either multi-thread or serialized modes at start-time or
run-time.


c2: Using SQLite In Multi-Threaded Applications
c3: 3. Start-time selection of threading mode
c5: threadsafe.html#start_time_selection_of_threading_mode
c4:
Assuming that the compile-time threading mode is not single-thread, then
the threading mode can be changed during initialization using the
sqlite3_config() interface.  The SQLITE_CONFIG_SINGLETHREAD verb
puts SQLite into single-thread mode, the SQLITE_CONFIG_MULTITHREAD
verb sets multi-thread mode, and the SQLITE_CONFIG_SERIALIZED verb
sets serialized mode.


c2: Using SQLite In Multi-Threaded Applications
c3: 4. Run-time selection of threading mode
c5: threadsafe.html#run_time_selection_of_threading_mode
c4:
If single-thread mode has not been selected at compile-time or start-time,
then individual database connections can be created as either multi-thread
or serialized.  It is not possible to downgrade an individual database
connection to single-thread mode.  Nor is it possible to escalate an
individual database connection if the compile-time or start-time mode
is single-thread.

The threading mode for an individual database connection is determined
by flags given as the third argument to sqlite3_open_v2().  The
SQLITE_OPEN_NOMUTEX flag causes the database connection to be in the
multi-thread mode and the SQLITE_OPEN_FULLMUTEX flag causes the connection
to be in serialized mode.  If neither flag is specified or if
sqlite3_open() or sqlite3_open16() are used instead of 
sqlite3_open_v2(), then the default
mode determined by the compile-time and start-time settings is used.

This page last modified on  2023-12-05 14:43:20 UTC


c2: Floating Point Numbers
c3: 1. How SQLite Stores Numbers
c5: floatingpoint.html#how_sqlite_stores_numbers
c4:
SQLite stores integer values in the 64-bit 
twos-complement
format&sup1.
This gives a storage range of -9223372036854775808 to +9223372036854775807,
inclusive.  Integers within this range are exact.




So-called "REAL" or floating point values are stored in the
IEEE 754
Binary-64
format&sup1.
This gives a range of positive values between approximately
1.7976931348623157e+308 and 4.9406564584124654e-324 with an equivalent
range of negative values.  A binary64 can also be 0.0 (and -0.0), positive
and negative infinity and "NaN" or "Not-a-Number".  Floating point
values are approximate.


Pay close attention to the last sentence in the previous paragraph:

Floating point values are approximate.



If you need an exact answer, you should not use binary64 floating-point
values, in SQLite or in any other product.  This is not an SQLite limitation.
It is a mathematical limitation inherent in the design of floating-point numbers.

&mdash;&sup1;
Exception:  The R-Tree extension stores information as 32-bit floating
point or integer values.


c2: Floating Point Numbers
c3: 1.1. Floating-Point Accuracy
c5: floatingpoint.html#floating_point_accuracy
c4:
SQLite promises to preserve the 15 most significant digits of a floating
point value.  However, it makes no guarantees about the accuracy of
computations on floating point values, as no such guarantees are possible.
Performing math on floating-point values introduces error.
For example, consider what happens if you attempt to subtract two floating-point
numbers of similar magnitude:



1152693165.1106291898
-1152693165.1106280772

0.0000011126



The result shown above (0.0000011126) is the correct answer.  But if you
do this computation using binary64 floating-point, the answer you get is
0.00000095367431640625 - an error of about 14%.  If you do many similar
computations as part of your program, the errors add up so that your final
result might be completely meaningless.

The error arises because only about the first 15 significant digits of
each number are stored accurately, and the first difference between the two numbers
being subtracted is in the 16th digit.


c2: Floating Point Numbers
c3: 1.2. Floating Point Numbers
c5: floatingpoint.html#floating_point_numbers
c4:
The binary64 floating-point format uses 64 bits per number.  Hence there
are 1.845e+19 different possible floating point values.  On the other hand
there are infinitely many real numbers in the range of 
1.7977e+308 and 4.9407e-324.  It follows then that binary64 cannot possibly
represent all possible real numbers within that range.  Approximations are
required.


An IEEE 754 floating-point value is an integer multiplied by a power
of two:


M &times 2E


The M value is the "mantissa" and E is the "exponent".  Both
M and E are integers.

For Binary64, M is a 53-bit integer and E is an 11-bit integer that is
offset so that represents a range of values between -1074 and +972, inclusive.

(NB:  The usual description of IEEE 754 is more complex, and it is important
to understand the added complexity if you really want to appreciate the details,
merits, and limitations of IEEE 754.  However, the integer description shown
here, while not exactly right, is easier to understand and is sufficient for
the purposes of this article.)


c2: Floating Point Numbers
c3: 1.2.1. Unrepresentable numbers
c5: floatingpoint.html#unrepresentable_numbers
c4:
Not every decimal number with fewer than 16 significant digits can be
represented exactly as a binary64 number.  In fact, most decimal numbers
with digits to the right of the decimal point lack an exact binary64
equivalent.  For example, if you have a database column that is intended
to hold an item price in dollars and cents, the only cents value that
can be exactly represented are 0.00, 0.25, 0.50, and 0.75.  Any other
numbers to the right of the decimal point result in an approximation.
If you provide a "price" value of 47.49, that number will be represented
in binary64 as:


6683623321994527 &times; 2-47


Which works out to be:


47.49000000000000198951966012828052043914794921875


That number is very close to 47.49, but it is not exact.  It is a little
too big.  If we reduce M by one to 6683623321994526 so that we have the
next smaller possible binary64 value, we get:


47.4899999999999948840923025272786617279052734375




This second number is too small.
The first number is closer to the desired value of 47.49, so that is the
one that gets used.  But it is not exact.  Most decimal values work this
way in IEEE 754.  Remember the key point we made above:


Floating point values are approximate.


If you remember nothing else about floating-point values, 
please don't forget this one key idea.


c2: Floating Point Numbers
c3: 1.2.2. Is it close enough?
c5: floatingpoint.html#is_it_close_enough_
c4:
The precision provided by IEEE 754 Binary64 is sufficient for most computations.
For example, if "47.49" represents a price and inflation is running
at 2% per year, then the price is going up by about 0.0000000301 dollars per
second.  The error in the recorded value of 47.49 represents about 66 nanoseconds
worth of inflation.  So if the 47.49 price is exact
when you enter it, then the effects of inflation will cause the true value to
exactly equal the value actually stored
(47.4900000000000019895196601282805204391479492187) in less than 
one ten-millionth of a second.
Surely that level of precision is sufficient for most purposes?


c2: Floating Point Numbers
c3: 2.1. The ieee754.c Extension
c5: floatingpoint.html#the_ieee754_c_extension
c4:
The ieee754 extension converts a floating point number between its
binary64 representation and the M&times;2E format.
In other words in the expression:


F = M &times 2E


The ieee754 extension converts between F and (M,E) and back again.

The ieee754 extension is not part of the amalgamation, but it is included
by default in the CLI.  If you want to include the ieee754 extension in your
application, you will need to compile and load it separately.


c2: Floating Point Numbers
c3: 2.1.1. The ieee754() function
c5: floatingpoint.html#the_ieee754_function
c4:
The ieee754(F) SQL function takes a single floating-point argument
as its input and returns a string that looks like this:


'ieee754(M,E)'


Except that the M and E are replaced by the mantissa and exponent of the
floating point number.  For example:

sqlite> .mode box
sqlite> SELECT ieee754(47.49) AS x;
┌───────────────────────────────┐
│               x               │
├───────────────────────────────┤
│ ieee754(6683623321994527,-47) │
└───────────────────────────────┘



Going in the other direction, the 2-argument version of ieee754() takes
the M and E values and converts them into the corresponding F value:

sqlite> select ieee754(6683623321994527,-47) as x;
┌───────┐
│   x   │
├───────┤
│ 47.49 │
└───────┘


c2: Floating Point Numbers
c3: 2.1.2. The ieee754_mantissa() and ieee754_exponent() functions
c5: floatingpoint.html#the_ieee754_mantissa_and_ieee754_exponent_functions
c4:
The text output of the one-argument form of ieee754() is great for human
readability, but it is awkward to use as part of a larger expression.  Hence
the ieee754_mantissa() and ieee754_exponent() routines were added to return
the M and E values corresponding to their single argument F
value.
For example:

sqlite> .mode box
sqlite> SELECT ieee754_mantissa(47.49) AS M, ieee754_exponent(47.49) AS E;
┌──────────────────┬─────┐
│        M         │  E  │
├──────────────────┼─────┤
│ 6683623321994527 │ -47 │
└──────────────────┴─────┘


c2: Floating Point Numbers
c3: 2.1.3. The ieee754_from_blob() and ieee754_to_blob() functions
c5: floatingpoint.html#the_ieee754_from_blob_and_ieee754_to_blob_functions
c4:
The ieee754_to_blob(F) SQL function converts the floating point number F
into an 8-byte BLOB that is the big-endian binary64 encoding of that number.
The ieee754_from_blob(B) function goes the other way, converting an 8-byte
blob into the floating-point value that the binary64 encoding represents.

So, for example, if you read
on
Wikipedia that the encoding for the minimum positive binary64 value is
0x0000000000000001, then you can find the corresponding floating point value
like this:

sqlite> .mode box
sqlite> SELECT ieee754_from_blob(x'0000000000000001') AS F;
┌───────────────────────┐
│           F           │
├───────────────────────┤
│ 4.94065645841247e-324 │
└───────────────────────┘


Or go the other way:

sqlite> .mode box
sqlite> SELECT quote(ieee754_to_blob(4.94065645841247e-324)) AS binary64;
┌─────────────────────┐
│      binary64       │
├─────────────────────┤
│ X'0000000000000001' │
└─────────────────────┘


c2: Floating Point Numbers
c3: 2.2. The decimal.c Extension
c5: floatingpoint.html#the_decimal_c_extension
c4:
The decimal extension provides arbitrary-precision decimal arithmetic on
numbers stored as text strings.  Because the numbers are stored to arbitrary
precision and as text, no approximations are needed.  Computations can be
done exactly.

The decimal extension is not (currently) part of the SQLite amalgamation.
However, it is included in the CLI.

There are three math functions available:



 decimal_add(A,B)
 decimal_sub(A,B)
 decimal_mul(A,B)
 decimal_pow2(N)



The first three functions respectively add, subtract, and multiply their arguments
and return a new text string that is the decimal representation of the result. The
argument is interpreted as text.
There is no division operator because decimal division often does not generate
a finite decimal result.

The decimal_pow2(N) function returns 2.0 raised to the N-th power where N is an
integer between -20000 and +20000.

Use the decimal_cmp(A,B) to compare two decimal values.  The result will
be negative, zero, or positive if A is less than, equal to, or greater than B,
respectively.

The decimal_sum(X) function is an aggregate, like the built-in
sum() aggregate function, except that decimal_sum() computes its result
to arbitrary precision and is therefore precise.

The decimal extension provides the "decimal" collating sequences
that compares decimal text strings in numeric order.

The decimal(X) and decimal_exp(X) generate a decimal representation for input X.
The decimal_exp(X) function returns the result in exponential notation (with a "e+NN"
at the end) and decimal(X) returns a pure decimal (without the "e+NN").  If the input
X is a floating point value, it is expanded to its exact decimal equivalent.  For
example:

sqlite> .mode qbox
sqlite> select decimal(47.49);
┌──────────────────────────────────────────────────────┐
│                    decimal(47.49)                    │
├──────────────────────────────────────────────────────┤
│ '47.49000000000000198951966012828052043914794921875' │
└──────────────────────────────────────────────────────┘


c2: Indexes On Expressions
c5: expridx.html#
c4:
Normally, an SQL index references columns of a table.  But an index
can also be formed on expressions involving table columns.


As an example, consider the following table that tracks
dollar-amount changes on various "accounts":

CREATE TABLE account_change(
  chng_id INTEGER PRIMARY KEY,
  acct_no INTEGER REFERENCES account,
  location INTEGER REFERENCES locations,
  amt INTEGER,  -- in cents
  authority TEXT,
  comment TEXT
);
CREATE INDEX acctchng_magnitude ON account_change(acct_no, abs(amt));



Each entry in the account_change table records a deposit or a withdrawal
into an account.  Deposits have a positive "amt" and withdrawals have
a negative "amt".


The acctchng_magnitude index is over the account number ("acct_no") and
on the absolute value of the amount.  This index allows one to do 
efficient queries over the magnitude of a change to the account.
For example, to list all changes to account number $xyz that are
more than $100.00, one can say:


SELECT * FROM account_change WHERE acct_no=$xyz AND abs(amt)>=10000;



Or, to list all changes to one particular account ($xyz) in order of
decreasing magnitude, one can write:

SELECT * FROM account_change WHERE acct_no=$xyz
 ORDER BY abs(amt) DESC;



Both of the above example queries would work fine without the
acctchng_magnitude index.
The acctchng_magnitude index merely helps the queries to run
faster, especially on databases where there are many entries in
the table for each account.


c2: Indexes On Expressions
c3: 1. How To Use Indexes On Expressions
c5: expridx.html#how_to_use_indexes_on_expressions
c4:
Use a CREATE INDEX statement to create a new index on one or more
expressions just like you would to create an index on columns.  The only
difference is that expressions are listed as the elements to be indexed
rather than column names.


The SQLite query planner will consider using an index on an expression
when the expression that is indexed appears in the WHERE clause or in
the ORDER BY clause of a query, exactly as it is written in the
CREATE INDEX statement.  The query planner does not do algebra.  In order
to match WHERE clause constraints and ORDER BY terms to indexes, SQLite
requires that the expressions be the same, except for minor syntactic
differences such as white-space changes.  So if you have:

CREATE TABLE t2(x,y,z);
CREATE INDEX t2xy ON t2(x+y);



And then you run the query:

SELECT * FROM t2 WHERE y+x=22;



Then the index will not be used because 
the expression on the CREATE INDEX
statement (x+y) is not the same as the expression as it appears in the 
query (y+x).  The two expressions might be mathematically equivalent, but
the SQLite query planner insists that they be the same, not merely
equivalent.  Consider rewriting the query thusly:

SELECT * FROM t2 WHERE x+y=22;



This second query will likely use the index because now the expression
in the WHERE clause (x+y) matches the expression in the index exactly.


c2: Indexes On Expressions
c3: 2. Restrictions
c5: expridx.html#restrictions
c4:
There are certain reasonable restrictions on expressions that appear in
CREATE INDEX statements:



Expressions in CREATE INDEX statements
may only refer to columns of the table being indexed, not to
columns in other tables.


Expressions in CREATE INDEX statements
may contain function calls, but only to functions whose output
is always determined completely by its input parameters (a.k.a.:
deterministic functions).  Obviously, functions like random() will not
work well in an index.  But also functions like sqlite_version(), though
they are constant across any one database connection, are not constant
across the life of the underlying database file, and hence may not be
used in a CREATE INDEX statement.


Note that application-defined SQL functions are by default considered
non-deterministic and may not be used in a CREATE INDEX statement unless
the SQLITE_DETERMINISTIC flag is used when the function is registered.


Expressions in CREATE INDEX statements may not use subqueries.


Expressions may only be used in CREATE INDEX statements, not within
UNIQUE or PRIMARY KEY constraints within the CREATE TABLE statement.


c2: Indexes On Expressions
c3: 3. Compatibility
c5: expridx.html#compatibility
c4:
The ability to index expressions was added to SQLite with 
version 3.9.0 (2015-10-14).  A database that uses an index on
expressions will not be usable by earlier versions of SQLite.
This page last modified on  2023-02-11 20:57:33 UTC


c2: SQLite Archive Files
c3: 1. Introduction
c5: sqlar.html#introduction
c4:
An "SQLite Archive" is a file container similar to a
ZIP archive or
Tarball but
based on an SQLite database.


An SQLite Archive is an ordinary SQLite database file that contains the
following table as part of its schema:

CREATE TABLE sqlar(
  name TEXT PRIMARY KEY,  -- name of the file
  mode INT,               -- access permissions
  mtime INT,              -- last modification time
  sz INT,                 -- original file size
  data BLOB               -- compressed content
);



Each row of the SQLAR table holds the content of a single file.
The filename (the full pathname relative to the root of the archive)
is in the "name" field.
The "mode" field is an integer which is the unix-style access permissions
for the file.  "mtime" is the modification time of the file in seconds
since 1970.  "sz" is the original uncompressed size of the file.
The "data" field contains the file content.  The content is usually
compressed using Deflate, though not always.  If the
"sz" field is equal to the size of the "data" field, then the content
is stored uncompressed.


c2: SQLite Archive Files
c3: 1.1. Database As Container Object
c5: sqlar.html#database_as_container_object
c4:
An SQLite Archive is one example of a more general
idea that an SQLite database can behave as a container object holding
lots of smaller data components.


With client/server databases like PostgreSQL or Oracle, users and
developers tend to think of the database as a service or a "node", not
as an object.  This is because the database content is spread out across
multiple files on the server, or possibly across multiple servers in a
service cluster.  One cannot point to a single file or even a single
directory and say "this is the database".


SQLite, in contrast, stores all content in a single file on disk.
That single file is something you can point to and say
"this is the database".  It behaves as an object.
An SQLite database file can be copied, renamed, sent as an
email attachment, passed as the argument a POST HTTP request,
or otherwise treated as other data object such as an image,
document, or media file.


Studies show that many applications already use
SQLite as a container object.  For example,
Kennedy
(no relation to the SQLite developer) reports that 14% of
Android applications never write to their SQLite databases.  It is
believed that these applications are downloading entire databases
from the cloud and then using the information locally as needed.  In other
words, the applications are using SQLite not so much as a database but as
a queryable wire-transfer format.


c2: SQLite Archive Files
c3: 1.2. Applications Using SQLite Archives
c5: sqlar.html#applications_using_sqlite_archives
c4:
The Fossil Distributed Version Control system
provides users with the option to download check-ins as either Tarballs,
ZIP Archives, or SQLite Archives.


c2: SQLite Archive Files
c3: 2. Advantages Of SQLite Archives
c5: sqlar.html#advantages_of_sqlite_archives
c4:
An SQLite Archive is flexible.
ZIP Archives and Tarballs are limited to storing only files.  An
SQLite Archive stores files plus whatever other tabular
and/or relational data seems useful to the application.


An SQLite Archive is transactional.
Updates are atomic and durable, even if there are crashes
or power losses in the middle of the update.
Readers see a consistent and unchanging version of the content even
is some other process is simultaneously updating the archive.


An SQLite Archive can be updated incrementally.
Individual files can be added or removed or replaced without having
to rewrite the entire archive.


An SQLite Archive can be queried using a high-level query language (SQL).
Some examples:

 What is the total size of all files in the archive whose names
     end in ".h" or ".cpp"?
 What percentage of the files are compressed by less than 25%?
 How many executable files are in the archive?

Questions like these (and countless others) can be answered without
having to uncompress or extract any content.


Applications that already use SQLite for other purposes can easily
add support for SQLite Archives using a small extension
(https://sqlite.org/src/file/ext/misc/sqlar.c) to handle the compression
and decompression of content.  Even this tiny extension can be omitted
if the files in the archive are uncompressed.  In contrast, supporting
ZIP Archives and/or Tarballs requires either separate libraries or
lots of extra custom code, or sometimes both.


An SQLite Archive can work around firewall-imposed censorship.
For example, certain file types that are considered "dangerous"
(examples: DLLs) will be
blocked by Gmail
and probably many other email services and firewalls, even if the
files are wrapped inside a ZIP Archive or Tarball.
But these firewalls usually do not (yet) know about SQLite Archives and
so content can be put inside an SQLite Archive to evade censorship.


c2: SQLite Archive Files
c3: 3. Disadvantages Of SQLite Archives
c5: sqlar.html#disadvantages_of_sqlite_archives
c4:
The SQLite Archive is a relatively new format.  It was first described in
in 2014.  ZIP Archives and Tarballs, on the other hand, have been around
for decades and are well-entrenched as standard formats.  Most programmers
know what a ZIP Archive or Tarball is, but if you say "SQLite Archive" you
are more likely to get a reply of "What?"  Tooling to process ZIP Archives
and Tarballs is more likely to be installed on stock computers.


Since an SQLite database is a more general format (it is designed to do
much more than simply store a bunch of files) it is not as compact as either
the ZIP Archive or Tarball formats.  An SQLite Archive is usually about 1%
larger than the equivalent ZIP Archive.  Tarballs are compressed as a single
unit rather than compressing each file separately as is done by both
SQLite and ZIP Archives.  For these reason, Tarballs tend to be smaller
than either ZIP or SQLite Archives.

As an example, the following table show the relative sizes for an
SQLite Archive, a ZIP Archive, and a Tarball of the 1,743 files
in the SQLite 3.22.0 source tree:

  SQLite Archive10,754,048
  ZIP Archive (using Info-ZIP 3.0)10,662,365
  ZIP Archive (using zipfile)10,390,215
  Tarball&#x2007;9,781,109



An SQLite Archive supports only the Deflate compression
method.  Tarballs and ZIP Archive support a wider assortment of
compression methods.


c2: SQLite Archive Files
c3: 4. Managing An SQLite Archive From The Command-Line
c5: sqlar.html#managing_an_sqlite_archive_from_the_command_line
c4:
The recommended way of creating, updating, listing, and extracting
an SQLite Archive is to use the sqlite3.exe command-line shell
for SQLite version 3.23.0 (2018-04-02) or later.  This CLI
supports the -A command-line option that allows easy management
of SQLite Archives.
The CLI for SQLite version 3.22.0 (2018-01-22) has the
.archive command for managing SQLite Archives, but that requires
interacting with the shell.


To list all of the files in an SQLite Archive named "example.sqlar"
using one of these commands:

sqlite3 example.sqlar -At
sqlite3 example.sqlar -Atv



To extract all files from an SQLite Archive named "example.sqlar":

sqlite3 example.sqlar -Ax



To create a new SQLite Archive named "alltxt.sqlar" containing all *.txt
files in the current directory:

sqlite3 alltxt.sqlar -Ac *.txt



To add or update files in an existing SQLite Archive:

sqlite3 example.sqlar -Au *.md



For usage hints and a summary of all options, simply give the CLI
the -A option with no additional arguments:

sqlite3 -A



All of these commands work the same way if the filename argument is
is a ZIP Archive instead of an SQLite database.


c2: SQLite Archive Files
c3: 4.1. Other command-line tools
c5: sqlar.html#other_command_line_tools
c4:
Just as there is the "zip" program to manage ZIP Archives, and the
"tar" program to manage Tarballs, the
"sqlar" program exists to manage SQL Archives.
The "sqlar" program is able to create a new SQLite Archive, list the
content of an existing archive, add or remove files from the archive,
and/or extract files from the archive.
A separate "sqlarfs" program is able to mount the SQLite Archive as
a Fuse Filesystem.


c2: SQLite Archive Files
c3: 5. Managing SQLite Archives From Application Code
c5: sqlar.html#managing_sqlite_archives_from_application_code
c4:
Applications can easily read or write SQLite Archives by linking against
SQLite and including the
ext/misc/sqlar.c extension
to handle the compression and decompression.  The sqlar.c extension
creates two new SQL functions.


sqlar_compress(X)

The sqlar_compress(X) function attempts to compress a copy of the
blob X using the Default algorithm and returns the
result as a blob.  If the input X is not a compressible blob, then
a copy of X is returned.  This routine is used when inserting content
into an SQLite Archive.
sqlar_uncompress(Y,SZ)

The sqlar_uncompress(Y,SZ) function will undo the compression accomplished
by sqlar_compress(X).  The Y parameter is the compressed content (the output
from a prior call to sqlar_compress()) and SZ is the original uncompressed
size of the input X that generated Y.  If SZ is less than or equal to the
size of Y, that indicates that no compression occurred, and so
sqlar_uncompress(Y,SZ) returns a copy of Y.  Otherwise, sqlar_uncompress(Y,SZ)
runs the Inflate algorithm on Y to uncompress it and restore it to its
original form and returns the uncompressed content.
This routine is used when extracting content from an SQLite Archive.



Using the two routines above, it is simple for applications to insert
new records into or extract existing records from an SQLite Archive.
Insert a new into an SQLite Archive using code like this:

INSERT INTO sqlar(name,mode,mtime,sz,data)
 VALUES ($name,$mode,strftime('%s',$mtime),
         length($content),sqlar_compress($content));



Extract an entry from the SQLite Archive using code like this:

SELECT name, mode, datetime(mtime,'unixepoch'), sqlar_uncompress(data,sz)
  FROM sqlar
 WHERE ...;



The code above is for the general case.  For the special case of an
SQLite Archive that only stores uncompressed or uncompressible content
(this might come up, for example, in an SQLite Archive that stores only
JPEG, GIF, and/or PNG images) then the content can be inserted into
and extracted from the database without using the sqlar_compress()
and sqlar_uncompress() functions, and the sqlar.c extension is not
required.


c2: The Lemon LALR(1) Parser Generator
c3: 1. Overview
c5: lemon.html#overview
c4:
The SQL language parser for SQLite is generated using a code-generator
program called "Lemon".  The Lemon program reads a grammar of the input
language and emits C-code to implement a parser for that language.


c2: The Lemon LALR(1) Parser Generator
c3: 1.1. Lemon Source Files And Documentation
c5: lemon.html#lemon_source_files_and_documentation
c4:
Lemon does not have its own source repository.  Rather, Lemon consists
of a few files in the SQLite source tree:



     lemon.html &rarr;
     The original detailed usage documentation and programmers reference
     for Lemon.

     lemon.c &rarr; The source code
     for the utility program that reads a grammar file and generates 
     corresponding parser C-code.

     lempar.c &rarr; A template
     for the generated parser C-code.  The "lemon" utility program reads this
     template and inserts additional code in order to generate a parser.


c2: The Lemon LALR(1) Parser Generator
c3: 2. Advantages of Lemon
c5: lemon.html#advantages_of_lemon
c4:
Lemon generates an LALR(1) parser.  Its operation is similar to the
more familiar tools Yacc and
Bison, but Lemon adds important
improvements, including:



     The grammar syntax is less error prone - using symbolic names for
     semantic values rather that the "$1"-style positional notation
     of Yacc.

     In Lemon, the tokenizer calls the parser.  Yacc operates the other
     way around, with the parser calling the tokenizer.  The Lemon
     approach is reentrant and threadsafe, whereas Yacc uses global 
     variables and is therefore neither.  Reentrancy is especially
     important for SQLite since some SQL statements make recursive calls
     to the parser.  For example, when parsing a CREATE TABLE statement,
     SQLite invokes the parser recursively to generate an INSERT statement
     to make a new entry in the sqlite_schema table.

     Lemon has the concept of a non-terminal destructor that can be
     used to reclaim memory or other resources following a syntax error
     or other aborted parse.


c2: The Lemon LALR(1) Parser Generator
c3: 2.1. Use of Lemon Within SQLite
c5: lemon.html#use_of_lemon_within_sqlite
c4:
Lemon is used in two places in SQLite.

The primary use of Lemon is to create the SQL language parser.
A grammar file (parse.y) is
compiled by Lemon into parse.c and parse.h.  The parse.c file is
incorporated into the amalgamation without further modification.

Lemon is also used to generate the parser for the query pattern
expressions in the FTS5 extension.  In this case, the input grammar
file is fts5parse.y.


c2: The Lemon LALR(1) Parser Generator
c3: 2.2. Lemon Customizations Especially For SQLite
c5: lemon.html#lemon_customizations_especially_for_sqlite
c4:
One of the advantages of hosting code generator tools as part of
the project is that the tools can be optimized to serve specific needs of
the overall project.  Lemon has benefited from this effect. Over the years,
the Lemon parser generator has been extended and enhanced to provide
new capabilities and improved performance to SQLite.  A few of the
specific enhancements to Lemon that are specifically designed for use
by SQLite include:



Lemon has the concept of a "fallback" token.
The SQL language contains a large number of keywords and these keywords
have the potential to collide with identifier names.
Lemon has the ability to designate some keywords as being able to
"fallback" to an identifier.  If the keyword appears in the input token
stream in a context that would otherwise be a syntax error, the token
is automatically transformed into its fallback before the syntax error
is raised.  This feature allows the parser to be very forgiving of
reserved words used as identifiers, which is a problem that comes up
frequently in the SQL language.


In support of the 100% MC/DC testing goal for SQLite, 
the parser code generated by Lemon has no unreachable branches,
and contains extra (compile-time selected) instrumentation useful
for measuring test coverage.


Lemon supports conditional compilation of grammar file rules, so that
a different parser can be generated depending on compile-time options.


As a performance optimization, reduce actions in the Lemon input grammar
are allowed to contain comments of the form "/*A-overwrites-Z*/" to indicate
that the semantic value "A" on the right-hand side of the rule is allowed
to directly overwrite the semantic value "Z" on the left-hand side.
This simple optimization reduces the number of stack operations in the
push-down automaton used to parse the input grammar, and thus improve
performance of the parser.  It also makes the generated code a little smaller.


The parsing of SQL statements is a significant consumer of CPU cycles 
in any SQL database engine.  On-going efforts to optimize SQLite have caused
the developers to spend a lot of time tweaking Lemon to generate faster
parsers.  These efforts have benefited all users of the Lemon parser generator,
not just SQLite.  But if Lemon had been a separately maintained tool, it
would have been more difficult to make coordinated changes to both SQLite
and Lemon, and as a result not as much optimization would have been
accomplished.  Hence, the fact that the parser generator tool is included
in the source tree for SQLite has turned out to be a net benefit for both
the tool itself and for SQLite.


c2: The Lemon LALR(1) Parser Generator
c3: 3. History Of Lemon
c5: lemon.html#history_of_lemon
c4:
Lemon was originally written by D. Richard Hipp (also the creator of SQLite)
while he was in graduate school at Duke University between 1987 and 1992.
The original creation date of Lemon has been lost, but was probably sometime
around 1990.  Lemon generates an LALR(1) parser.  There was a companion 
LL(1) parser generator tool named "Lime", but the source code for Lime
has been lost.

The Lemon source code was originally written as separate source files,
and only later merged into a single "lemon.c" source file.

The author of Lemon and SQLite (Hipp) reports that his C programming
skills were greatly enhanced by studying John Ousterhout's original
source code to Tcl.  Hipp discovered and studied Tcl in 1993.  Lemon
was written before then, and SQLite afterwards.  There is a clear
difference in the coding styles of these two products, with SQLite seeming
to be cleaner, more readable, and easier to maintain.
This page last modified on  2022-04-18 02:55:50 UTC


c2: Appropriate Uses For SQLite
c5: whentouse.html#
c4:
SQLite is not directly comparable to client/server SQL database engines such
as MySQL, Oracle, PostgreSQL, or SQL Server since SQLite is trying to
solve a different problem.



Client/server SQL database engines strive to implement a shared repository
of enterprise data.  They emphasize scalability, concurrency, centralization,
and control.
SQLite strives to provide local data storage for
individual applications and devices.  SQLite emphasizes economy,
efficiency, reliability, independence, and simplicity.




SQLite does not compete with client/server databases.
SQLite competes with fopen().


c2: Appropriate Uses For SQLite
c3: 1. Situations Where SQLite Works Well
c5: whentouse.html#situations_where_sqlite_works_well
c4:
Embedded devices and the internet of things

Because an SQLite database requires no administration,
it works well in devices that must operate without expert human support.
SQLite is a good fit for use in 
cellphones, set-top boxes, televisions, game consoles,
cameras, watches, kitchen appliances, thermostats, automobiles, 
machine tools, airplanes, remote sensors, drones, medical devices,
and robots:  the "internet of things".


Client/server database engines are designed to live inside a
lovingly-attended datacenter at the core of the network.
SQLite works there too, but SQLite also thrives at the edge of the network,
fending for itself while providing fast and
reliable data services to applications that would otherwise
have dodgy connectivity.


Application file format


SQLite is often used as the on-disk file format
for desktop applications such as version control systems,
financial analysis tools, media cataloging and editing suites, CAD
packages, record keeping programs, and so forth.  The traditional
File/Open operation calls sqlite3_open() to attach to the database
file.  Updates happen automatically as application content is revised
so the File/Save menu option becomes superfluous.  The File/Save_As
menu option can be implemented using the backup API.


There are many benefits to this approach, including improved
performance, reduced cost and complexity, and
improved reliability.  See technical notes
"aff_short.html" and 
"appfileformat.html" and
"fasterthanfs.html" for more information.
This use case is closely related to the
data transfer format and
data container use cases below.




Websites

SQLite works great as the database engine for most low to
medium traffic websites (which is to say, most websites).
The amount of web traffic that SQLite can handle depends
on how heavily the website uses its database.  Generally
speaking, any site that gets fewer than 100K hits/day should work
fine with SQLite.
The 100K hits/day figure is a conservative estimate, not a
hard upper bound.
SQLite has been demonstrated to work with 10 times that amount
of traffic.

The SQLite website (https://www.sqlite.org/) uses SQLite itself,
of course, and as of this writing (2015) it handles about 400K to 500K
HTTP requests per day, about 15-20% of which are dynamic pages touching
the database.  Dynamic content uses about 200 SQL statements per webpage.
This setup runs on a single VM that shares a physical server with 23 others
and yet still keeps the load average below 0.1 most of the time.

See also: Hacker New discussion from 2022-12-13.


Data analysis


People who understand SQL can employ the 
sqlite3 command-line shell (or various third-party
SQLite access programs) to analyze large
datasets. Raw data can be imported from CSV files, then that
data can be sliced and diced to generate a myriad of summary
reports.  More complex analysis can be done using simple scripts written
in Tcl or Python (both of which come with SQLite built-in) or in R or
other languages using readily available adaptors.
Possible uses include website log analysis, sports
statistics analysis, compilation of programming metrics, and
analysis of experimental results.  Many bioinformatics researchers
use SQLite in this way.



The same thing can be done with an enterprise client/server
database, of course.  The advantage of SQLite is
that it is easier to install and use and the resulting database 
is a single file that can be written to a USB memory stick
or emailed to a colleague.



Cache for enterprise data


Many applications use SQLite as a cache of relevant content from
an enterprise RDBMS.
This reduces latency, since most queries now occur against the local
cache and avoid a network round-trip.  It also reduces the load 
on the network and on the central database server.  And in many cases, 
it means that the client-side application can continue operating during
network outages.





Server-side database


Systems designers
report success using SQLite as a data store on server applications
running in the datacenter, or in other words, using SQLite as the underlying
storage engine for an application-specific database server.

With this pattern, the overall system is still client/server:
clients send requests to the server and get back replies over the network.
But instead of sending generic SQL and getting back raw table content, 
the client requests and server responses are high-level and 
application-specific.
The server translates requests into multiple SQL queries, gathers the
results, does post-processing, filtering, and analysis, then constructs
a high-level reply containing only the essential information.

Developers report that SQLite is often faster than a client/server
SQL database engine in this scenario.
Database requests are serialized by the server, so concurrency is not
an issue.  Concurrency is also improved by "database sharding":
using separate database files for different subdomains.  For
example, the server might have a separate SQLite database for each
user, so that the server can handle hundreds or thousands of simultaneous
connections, but each SQLite database is only used by one connection.




Data transfer format

Because an SQLite database is a single compact file in a
well-defined cross-platform format, it is often used
as a container for transferring content from one system to another.
The sender gathers content into an SQLite database file, transfers
that one file to the receiver, then the receiver uses SQL to extract
the content as needed.

An SQLite database facilitates data transfer between systems even
when the endpoints have different word sizes and/or byte orders.
The data can be a complex mix of large binary blobs, text, and small
numeric or boolean values.  The data format can be easily extended
by adding new tables and/or columns, without breaking legacy receivers.
The SQL query language means that receivers are not required to parse
the entire transfer all at once, but can instead query the
received content as needed.  The data format is "transparent" in the
sense that it is easily decoded for human viewing using 
a variety of universally available, open-source tools, from multiple
vendors.




File archive and/or data container


The SQLite Archive idea shows how
SQLite can be used as a substitute for ZIP archives or Tarballs.
An archive of files stored in SQLite is only very slightly larger, and
in some cases actually smaller, than the equivalent ZIP archive.
And an SQLite archive features incremental and atomic updating
and the ability to store much richer metadata.


Fossil version 2.5 and later offers
SQLite Archive files as a download format, in addition
to traditional tarball and ZIP archive.
The sqlite3.exe command-line shell version 3.22.0 and later will create,
list, or unpack an SQL archiving using the 
.archive command.


SQLite is a good solution for any situation that requires bundling
diverse content into a self-contained and self-describing package 
for shipment across a network.
Content is encoded in a 
well-defined, cross-platform, and stable file format.
The encoding is efficient, and receivers can extract small subsets
of the content without having to read and parse the entire file.


SQL archives are useful as the distribution format for software
or content updates that are broadcast to many clients.  Variations
on this idea are used, for example, to transmit TV programming guides
to set-top boxes and to send over-the-air updates to vehicle navigation
systems.


Replacement for ad hoc disk files

Many programs use 
fopen(),
fread(), and 
fwrite() to create and
manage files of data in home-grown formats.  SQLite works 
particularly well as a
replacement for these ad hoc data files.
Contrary to intuition, SQLite can be faster than the filesystem
for reading and writing content to disk.


Internal or temporary databases


For programs that have a lot of data that must be sifted and sorted
in diverse ways, it is often easier and quicker to load the data into
an in-memory SQLite database and use queries with joins and ORDER BY
clauses to extract the data in the form and order needed rather than
to try to code the same operations manually.
Using an SQL database internally in this way also gives the program
greater flexibility since new columns and indices can be added without
having to recode every query.



Stand-in for an enterprise database during demos or testing


Client applications typically use a generic database interface that allows
connections to various SQL database engines.  It makes good sense to 
include SQLite in the mix of supported databases and to statically
link the SQLite engine in with the client.  That way the client program
can be used standalone with an SQLite data file for testing or for
demonstrations.



Education and Training


Because it is simple to setup and use (installation is trivial: just
copy the sqlite3 or sqlite3.exe executable to the target machine
and run it) SQLite makes a good database engine for use in teaching SQL.
Students can easily create as many databases as they like and can
email databases to the instructor for comments or grading.  For more
advanced students who are interested in studying how an RDBMS is
implemented, the modular and well-commented and documented SQLite code
can serve as a good basis.



Experimental SQL language extensions

The simple, modular design of SQLite makes it a good platform for
prototyping new, experimental database language features or ideas.


c2: Appropriate Uses For SQLite
c3: 2. Situations Where A Client/Server RDBMS May Work Better
c5: whentouse.html#situations_where_a_client_server_rdbms_may_work_better
c4:
Client/Server Applications

If there are many client programs sending SQL to the same 
database over a network, then use a client/server database
engine instead of SQLite.  SQLite will work over a network filesystem,
but because of the latency associated with most network filesystems,
performance will not be great.  Also, file locking logic is buggy in
many network filesystem implementations (on both Unix and Windows).
If file locking does not work correctly,
two or more clients might try to modify the
same part of the same database at the same time, resulting in 
corruption.  Because this problem results from bugs in
the underlying filesystem implementation, there is nothing SQLite
can do to prevent it.

A good rule of thumb is to avoid using SQLite
in situations where the same database will be accessed directly
(without an intervening application server) and simultaneously
from many computers over a network.


High-volume Websites

SQLite will normally work fine as the database backend to a website.
But if the website is write-intensive or is so busy that it requires
multiple servers, then consider using an enterprise-class client/server 
database engine instead of SQLite.


Very large datasets

An SQLite database is limited in size to 281 terabytes 
(248 bytes, 256 tibibytes).
And even if it could handle larger databases, SQLite stores the entire
database in a single disk file and many filesystems limit the maximum
size of files to something less than this.  So if you are contemplating
databases of this magnitude, you would do well to consider using a
client/server database engine that spreads its content across multiple
disk files, and perhaps across multiple volumes.



High Concurrency


SQLite supports an unlimited number of simultaneous readers, but it 
will only allow one writer at any instant in time.
For many situations, this is not a problem.  Writers queue up. Each application
does its database work quickly and moves on, and no lock lasts for more
than a few dozen milliseconds. But there are some applications that require
more concurrency, and those applications may need to seek a different
solution.


c2: Appropriate Uses For SQLite
c3: 3. Checklist For Choosing The Right Database Engine
c5: whentouse.html#checklist_for_choosing_the_right_database_engine
c4:
Is the data separated from the application by a network?
       &rarr; choose client/server

Relational database engines act as bandwidth-reducing data filters.
So it is best to keep the database engine and the data on
the same physical device so that the high-bandwidth engine-to-disk
link does not have to traverse the network, only the lower-bandwidth
application-to-engine link.

But SQLite is built into the application.  So if the data is on a
separate device from the application, it is required that the higher
bandwidth engine-to-disk link be across the network.  This works, but
it is suboptimal.  Hence, it is usually better to select a client/server
database engine when the data is on a separate device from the
application.

Nota Bene:
In this rule, "application" means the code that issues SQL statements.
If the "application" is an application server and
if the content resides on the same physical machine as the application server,
then SQLite might still be appropriate even though the end user is
another network hop away.


Many concurrent writers? &rarr; choose client/server

If many threads and/or processes need to write the
database at the same instant (and they cannot queue up and take turns)
then it is best to select a database engine that supports that
capability, which always means a client/server database engine.

SQLite only supports one writer at a time per database file.
But in most cases, a write transaction only takes milliseconds and
so multiple writers can simply take turns.  SQLite will handle
more write concurrency than many people suspect.  Nevertheless,
client/server database systems, because they have a long-running
server process at hand to coordinate access, can usually handle 
far more write concurrency than SQLite ever will.


Big data? &rarr; choose client/server

If your data will grow to a size that you are uncomfortable
or unable to fit into a single disk file, then you should select
a solution other than SQLite.  SQLite supports databases up to
281 terabytes in size, assuming you can find a disk drive and filesystem
that will support 281-terabyte files.  Even so, when the size of the
content looks like it might creep into the terabyte range, it would
be good to consider a centralized client/server database.


Otherwise &rarr; choose SQLite!

For device-local storage with low writer concurrency and less than a
terabyte of content, SQLite is almost always a better solution.  SQLite
is fast and reliable and it requires no configuration or maintenance.
It keeps things simple.  SQLite "just works".


This page last modified on  2024-04-03 17:48:26 UTC


c2: NUL Characters In Strings
c3: 1. Introduction
c5: nulinstr.html#introduction
c4:
SQLite allows NUL characters (ASCII 0x00, Unicode \u0000) in the middle
of string values stored in the database.  However, the use of NUL within
strings can lead to surprising behaviors:



The length() SQL function only counts characters up to and excluding
the first NUL.



The quote() SQL function only shows characters up to and excluding
the first NUL.


The .dump command in the CLI omits the first NUL character and all
subsequent text in the SQL output that it generates.  In fact, the
CLI omits everything past the first NUL character in all contexts.



The use of NUL characters in SQL text strings is not recommended.


c2: NUL Characters In Strings
c3: 2. Unexpected Behavior
c5: nulinstr.html#unexpected_behavior
c4:
Consider the following SQL:

CREATE TABLE t1(
  a INTEGER PRIMARY KEY,
  b TEXT
);
INSERT INTO t1(a,b) VALUES(1, 'abc'||char(0)||'xyz');

SELECT a, b, length(b) FROM t1;



The SELECT statement above shows output of:

1,'abc',3



(Through this document, we assume that the CLI has ".mode quote" set.)
But if you run:

SELECT * FROM t1 WHERE b='abc';



Then no rows are returned.  SQLite knows that the t1.b column actually
holds a 7-character string, and the 7-character string 'abc'||char(0)||'xyz'
is not equal to the 3-character string 'abc', and so no rows are returned.
But a user might be easily confused by this because the CLI output
seems to show that the string has only 3 characters.  This seems like
a bug.  But it is how SQLite works.


c2: NUL Characters In Strings
c3: 3. How To Tell If You Have NUL Characters In Your Strings
c5: nulinstr.html#how_to_tell_if_you_have_nul_characters_in_your_strings
c4:
If you CAST a string into a BLOB, then the entire length of the
string is shown.  For example:

SELECT a, CAST(b AS BLOB) FROM t1;



Gives this result:

1,X'6162630078797a'



In the BLOB output, you can clearly see the NUL character as the 4th
character in the 7-character string.


Another, more automated, way
to tell if a string value X contains embedded NUL characters is to
use an expression like this:

instr(X,char(0))



If this expression returns a non-zero value N, then there exists an 
embedded NUL at the N-th character position.  Thus to count the number
of rows that contain embedded NUL characters:

SELECT count(*) FROM t1 WHERE instr(b,char(0))>0;


c2: NUL Characters In Strings
c3: 4. Removing NUL Characters From A Text Field
c5: nulinstr.html#removing_nul_characters_from_a_text_field
c4:
The following example shows how to remove NUL character, and all text
that follows, from a column of a table.  So if you have a database file
that contains embedded NULs and you would like to remove them, running
UPDATE statements similar to the following might help:

UPDATE t1 SET b=substr(b,1,instr(b,char(0)))
 WHERE instr(b,char(0));

This page last modified on  2022-05-23 22:21:54 UTC


c2: The SQLITE_STMT Virtual Table
c3: 1. Overview
c5: stmt.html#overview
c4:
The SQLITE_STMT extension implements an eponymous-only virtual table that
provides information about all prepared statements associated with
the database connection.



The SQLITE_STMT extension is included in the amalgamation though 
it is disabled
by default.  Use the SQLITE_ENABLE_STMTVTAB compile-time option to enable
the SQLITE_STMT extension.  The SQLITE_STMT extension can also be 
loaded at run-time
by compiling the extension into a shared library or DLL using the source
code at https://sqlite.org/src/file/ext/misc/stmt.c and following the
instructions for how to compile loadable extensions.



The SQLITE_STMT extension is enabled in default builds
of the command-line shell.


c2: The SQLITE_STMT Virtual Table
c3: 2. Usage
c5: stmt.html#usage
c4:
The SQLITE_STMT virtual table is a read-only table that can be directly
queried to access information about all prepared statements on the 
current database connection.  For example:

SELECT * FROM sqlite_stmt;



A statement such as the above can be run immediately prior to invoking
sqlite3_close() to confirm that all prepared statements have been 
finalized and to help identify and track down prepared
statements that have "leaked" and missed finalization.


The SQLITE_STMT virtual table can also be used to access performance
information about prepared statements, to aid in optimization an application.
For example,
to find out how much memory is being used by prepared statements that have
never been used, one could run:

SELECT sum(mem) FROM sqlite_stmt WHERE run=0;


c2: The SQLITE_STMT Virtual Table
c3: 2.1. Columns
c5: stmt.html#columns
c4:
The columns are provided by the SQLITE_STMT virtual table are summarized by
the hypothetical CREATE TABLE statement show here:

CREATE TABLE sqlite_stmt(
  sql    TEXT,    -- Original SQL text
  ncol   INT,     -- Number of output columns
  ro     BOOLEAN, -- True for "read only" statements
  busy   BOOLEAN, -- True if the statement is current running
  nscan  INT,     -- Number of full-scan steps
  nsort  INT,     -- Number of sort operations
  naidx  INT,     -- Number of automatic index inserts
  nstep  INT,     -- Number of byte-code engine steps
  reprep INT,     -- Number of reprepare operations
  run    INT,     -- Number of times this statement has been run
  mem    INT      -- Heap memory used by this statement
);


Future releases may add new output columns and may change the order
of legacy columns.
Further detail about the meaning of each column in the SQLITE_STMT virtual
table is provided below:


sql:
The original SQL text of the prepared statement.  If the prepared
statement is compiled using the sqlite3_prepare() interface, then
the SQL text might not have been saved, in which case this column
will be NULL.

ncol:
The number of columns in the result set of a query.
For DML statements, this column has a value of 0.

ro:
The "read only" column.  This column is true (non-zero) if the
SQL statement is a query and false (zero) if it is a DML statement.

busy:
This field is true if the prepared statement is currently running.
In other words, this field is true if sqlite3_step() has been called
on the prepared statement at least once but sqlite3_reset() has
not yet been called to reset it.

nscan:
This field is the number of times that the bytecode engine has stepped
through a table as part of a full-table scan.  A large number if this
field may indicate an opportunity to improve performance by adding an
index.  This field is equivalent to the SQLITE_STMTSTATUS_FULLSCAN_STEP
value.

nsort:
This field is the number of times that the bytecode engine had to sort.
A positive value in this field may indicate an opportunity to improve
performance by adding an index that will cause the query results to
appear naturally in the desired order.  
This field is equivalent to the SQLITE_STMTSTATUS_SORT value.

naidx:
This field is the number of rows that have been inserted into
automatic indexes.  A positive value in this field may indicate 
an opportunity to improve performance by adding a named index that
take the place of the automatic index.
This field is equivalent to the SQLITE_STMTSTATUS_AUTOINDEX value.

nstep:
This field is the number of bytecode engine operations that have
been performed for the prepared statement.  This field can be used
as a proxy for how much CPU time a statement has used.
This field is equivalent to the SQLITE_STMTSTATUS_VM_STEP value.

reprep:
This field is the number of times that the statement has had to be
reprepared due to schema changes or changes to parameter bindings.
This field is equivalent to the SQLITE_STMTSTATUS_REPREPARE value.

run:
This field is the number of times that the statement has been run.
This field is equivalent to the SQLITE_STMTSTATUS_RUN value.

mem:
This field is the number of bytes of heap storage used by the
prepared statement.
This field is equivalent to the SQLITE_STMTSTATUS_MEMUSED value.

This page last modified on  2022-01-08 05:02:57 UTC


c2: The COMPLETION() Table-Valued Function
c3: 1. Overview
c5: completion.html#overview
c4:
The COMPLETION extension implements a table-valued function named
"completion" that can be used to suggest completions of partially entered
words during interactive SQL input.  The completion table can be
used to help implement tab-completion, for example.


c2: The COMPLETION() Table-Valued Function
c3: 2. Details
c5: completion.html#details
c4:
The designed query interface is:

SELECT DISTINCT candidate COLLATE nocase
  FROM completion($prefix, $wholeline)
 ORDER BY 1;


The query above will return suggestions for the whole input word that
begins with $prefix.  The $wholeline parameter is all text from the beginning
of the line up to the insertion point.  The $wholeline parameter is used
for context.

The $prefix parameter may be NULL, in which case the prefix is deduced
from $wholeline.  Or, the $wholeline parameter may be NULL or omitted if 
context information is unavailable or if context-aware completion is not
desired.

The completion table might return the same candidate more than once, and
it will return candidates in an arbitrary order.  The DISTINCT keyword and
the ORDER BY in the sample query above are added to make the answers unique
and in lexicographical order.


c2: The COMPLETION() Table-Valued Function
c3: 2.1. Example Usage
c5: completion.html#example_usage
c4:
The completion table is used to implement tab-completion in the
command-line shell in conjunction with either the readline or linenoise
input line editing packages for unix.  See the
https://sqlite.org/src/file/src/shell.c.in source file for example
code.  Search for "FROM completion" to find the relevant code sections.

Because the completion table is built into the command-line shell in order
to provide for tab-completions, you can run test queries against the
completion table directly in the command-line shell.  Simply type a
query such as the example shown above, filling in appropriate values
for $prefix and $wholeline, and observe the output.


c2: The COMPLETION() Table-Valued Function
c3: 3. Limitations
c5: completion.html#limitations
c4:
The completion table is designed for interactive use.  It will return
answers at a speed appropriate for human typing.  No effort is made to
be unusually efficient, so long as the response time is nearly instantaneous
in a user interface.

As of this writing (2017-07-13), the completion virtual table only
looks for SQL keywords, and schema, table, and column names.  The
context contained in $wholeline is completely ignored.  Future enhancements
will try to return new completions taken from function and pragma names
and other sources, as well as consider more context.  The completion
table should be considered a work-in-progress.
This page last modified on  2022-01-08 05:02:57 UTC


c2: Imposter Tables
c3: 1. Introduction
c5: imposter.html#introduction
c4:
An imposter table is a table that is attached to the same b-tree
as an index.
An imposter table allows the content of an index to be queried or
modified as if the index were an ordinary table.



Imposter tables are intended for analysis and debugging only.
This is not a feature that most application developers should understand
or even know about.
Imposter tables are for experts only.



Improper use of imposter tables can cause index corruption, though
any corruption created this way can be fixed by running REINDEX.


c2: Imposter Tables
c3: 2. Details
c5: imposter.html#details
c4:
Each table and each index in SQLite is stored in a separate b-tree
in the database file.  Each b-tree is identified by its root page
number.  The root page number for any index or table can be found
by querying the "rootpage" column of the sqlite_schema table.
See the indexing tutorial and the file format documentation
for further background on this design.



Usually the b-trees for tables and indexes are slightly different.
A table b-tree contains a 64-bit integer key and arbitrary data.
The 64-bit integer key is the ROWID.  Index b-trees contain
an arbitrary binary key and no data.  So table b-trees and index
b-trees are not directly compatible.



However, the b-tree for a WITHOUT ROWID table is in the same format
as an index b-tree.  Thus, an index b-tree can be accessed as if it
were a WITHOUT ROWID table.


c2: Imposter Tables
c3: 2.1. Manually Created Imposter Tables
c5: imposter.html#manually_created_imposter_tables
c4:
One way to create an imposter table is to directly edit the sqlite_schema
table to insert a new row that describes the table.
For example, suppose the schema is like this:


CREATE TABLE t1(a INTEGER PRIMARY KEY,b TEXT,c INT, d INT);
CREATE INDEX t1bc ON t1(b,c);



The WITHOUT ROWID table that has the same structure as the t1bc index
would look like this:


CREATE TABLE t2(b TEXT,c INT,a INT, PRIMARY KEY(b,c,a)) WITHOUT ROWID;



To create a permanent imposter table "t2" against index "t1bc" one
should first enable editing of the sqlite_schema table by running
"PRAGMA writable_schema=ON".
(Be careful to observe the warnings that accompany this PRAGMA.
A mistake can cause severe database corruption.)
Then insert a new entry into the sqlite_schema table like this:


INSERT INTO sqlite_schema(type,name,tbl_name,rootpage,sql)
 SELECT 'table','t2','t2',rootpage,
   'CREATE TABLE t2(b,c,a,PRIMARY KEY(b,c,a))WITHOUT ROWID'
   FROM sqlite_schema
  WHERE name='t1bc';



The INSERT statement above adds a new row to the sqlite_schema table that
defines a table "t2" having the same on-disk format as index "t1bc" and 
pointing to the same b-tree.
After adding this sqlite_schema table entry, it is necessary to close and
reopen the database in order to get SQLite to reread the schema.
Then the "t2" table can be queried to see the content of the "t1bc" index.


c2: Imposter Tables
c3: 2.1.1. Corrupted Database
c5: imposter.html#corrupted_database
c4:
A serious problem with the manual imposter table approach described above
is that after adding the new "t2" entry to the "sqlite_schema" table, the
database file will technically be corrupt.  Both the "t1bc" index and the
"t2" table will point to the same b-tree.  This will not cause
any immediate problems, though one should avoid running VACUUM.



It is possible to write into the "t2" table, thus changing the content
of the index.
But doing so will get the "t1bc" index out of synchronization with its
parent table "t1".  An out-of-sync index can result in incorrect query
results.



Since the "t2" imposter table is a form of database corruption, the
manual approach to creating imposter tables is not recommended.
Actually, any use of imposter tables is discouraged for all but
expert developers, but manually created imposter tables are
especially discouraged because they are permanent.


c2: Imposter Tables
c3: 2.2. Transient Imposter Tables
c5: imposter.html#transient_imposter_tables
c4:
Another (safer) approach to creating an imposter table is to add an
entry for the imposter table to SQLite's internal symbol table without
updating the "sqlite_schema" table on disk.
That way, the imposter table exists in only a single database connection
and is automatically removed whenever the schema is reloaded.



Creation of a transient imposter table involves a special
sqlite3_test_control() call.  Unlike all other SQLite APIs,
sqlite3_test_control() interface is subject to incompatible changes
from one release to the next, and so the mechanism described below
is not guaranteed to work in future releases of SQLite.  The
SQLite developers do not consider this a problem because imposter
tables should not be used in applications.  Imposter tables are for
analysis and testing use only.



To create a transient imposter table, first call sqlite3_test_control()
as follows:


sqlite3_test_control(SQLITE_TESTCTRL_IMPOSTER, db, "main", 1, tnum);



The "db" parameter is a pointer to the database connection.
The "main" argument is the name of the schema in which the imposter
table is to be created.  The "1" argument enables the imposter table
mechanism.  "tnum" is the root page of the index that the imposter
table should mirror.



After the sqlite3_test_control() call above, then run a CREATE TABLE
statement the defines the imposter table.
With the imposter mechanism enabled, this CREATE TABLE statement does
not create a real table but instead merely adds an entry in SQLite's
internal symbol table.  Note that the CREATE TABLE statement must
be in the correct format for the index.  If the imposter table has the
wrong number of columns or is not a WITHOUT ROWID table or is otherwise
incompatible with the index b-tree, then SQLITE_CORRUPT errors will result
when the imposter table is used.



After running the CREATE TABLE statement, disable the imposter mechanism
as follows:


sqlite3_test_control(SQLITE_TESTCTRL_IMPOSTER, db, "main", 0, 0);



In other words, make the same sqlite3_test_control() call except change
the last two parameters to zero.



After the imposter table is loaded into SQLite's internal schema as
described above, the imposter table can be used as any other table.
But the imposter table will only be visible to the one database
connection that created it.  No changes are made to the database file
on disk.  And the imposter table will disappear the next time the schema 
is loaded.


c2: Imposter Tables
c3: 2.3. The .imposter Shell Command
c5: imposter.html#the_imposter_shell_command
c4:
As of SQLite 3.16.0 (2017-01-02), the command-line shell contains
a dot-command ".imposter" that does all of the work of setting up a
transient imposter table.
Instead of making multiple calls to sqlite3_test_control() and figuring
out and invoking a compatible CREATE TABLE statement, a transient
imposter table can be constructed as follows:


.imposter t1bc t2



Of course, substitute the desired index and imposter table names in
place of the "t1bc" and "t2" shown in the example.
The ".imposter" command reads the schema of the "t1bc" index, uses
that information to construct a compatible CREATE TABLE statement for
the imposter table, then makes all the necessary calls to create the
transient imposter table automatically.


c2: Imposter Tables
c3: 3. Summary And Final Warning
c5: imposter.html#summary_and_final_warning
c4:
The imposter table mechanism is a power analysis and debugging tool
for SQLite.  But as with all sharp tools, it can also be dangerous and
can result in corrupt database files if misused.  Do not attempt to
use imposter tables in an application.  Imposter tables are intended
for use in the laboratory by experts.

This page last modified on  2022-01-08 05:02:57 UTC


c2: The CSV Virtual Table
c3: 1. Overview
c5: csv.html#overview
c4:
The CSV virtual table reads
RFC 4180 formatted comma-separated
values, and returns that content as if it were rows and columns of an SQL
table.



The CSV virtual table is useful to applications that need to bulk-load
large amounts of comma-separated value content.
The CSV virtual table is also useful as a template source file for
implementing other virtual tables.




The CSV virtual table is not built into the SQLite amalgamation.
It is available as a
separate source file
that can be compiled into a loadable extension.
Typical usage of the CSV virtual table from the
command-line shell would be something like this:

.load ./csv
CREATE VIRTUAL TABLE temp.t1 USING csv(filename='thefile.csv');
SELECT * FROM t1;



The first line of the script above causes the command-line shell to
read and activate the run-time loadable extension for CSV.  For an
application, the equivalent C-language API is
sqlite3_load_extension().
Observe that the filename extension (ex: ".dll" or ".so" or ".dylib") is
omitted from the extension filename.  Omitting the filename extension is
not required, but it helps in making the script cross-platform.  SQLite
will automatically append the appropriate extension.


The second line above creates a virtual table named "t1" that reads
the content of the file named in the argument.  The number and names of
the columns is determined automatically by reading the first line of
content.  Other options to the CSV virtual table provide the ability to
take the CSV content from a string rather than a separate file, and give 
the programmer more control over the number and names of the columns.
The options are detailed below.  The CSV virtual table is usually
created as a TEMP table so that it exists only for the current database
connection and does not become a permanent part of the database schema.
Note that there is no "CREATE TEMP VIRTUAL TABLE" command in SQLite.
Instead, prepend the "temp." schema prefix to the name of the virtual
table.


The third line of the example shows the virtual table being used, to read
all content of the CSV file.  This is perhaps the simplest possible use
of the virtual table.  The CSV virtual table can be used anywhere an ordinary
virtual table can be used.  One can use the CSV virtual table inside subqueries,
or common table expressions or add WHERE, GROUP BY, HAVING, ORDER BY,
and LIMIT clauses as required.


c2: The CSV Virtual Table
c3: 2. Arguments
c5: csv.html#arguments
c4:
The example above showed a single filename='thefile.csv' argument
for the CSV virtual table.  But other arguments are also possible.


filename=FILENAME
The filename= argument specifies an external file from which
CSV content is read.  Every CSV virtual table must have either a 
filename= argument or a data= argument and not both.

data=TEXT
The data= argument specifies that TEXT is the literal
content of the CSV file.

schema=SCHEMA
 The schema= argument specifies a CREATE TABLE statement that
the CSV virtual table passes to the sqlite3_declare_vtab() interface in
order to define the names of the columns in the virtual table.

columns=N
The columns=N argument specifies the number of columns
in the CSV file.
If the input data contains more columns than this,
then the excess columns are ignored.  If the input data contains fewer columns,
then extra columns are filled with NULL.
If the columns=N argument is omitted, the first line of the
CSV file is read to determine the number of columns.

header=BOOLEAN
or just
header
If the header argument is true then the first row of the CSV file
to be treated as a header rather than as data.  The second line of the CSV
file becomes the first row of content.
If the schema= options is omitted, then the first line of the CSV
file determines the names of the columns.


c2: The CSV Virtual Table
c3: 3. Column Names
c5: csv.html#column_names
c4:
The column names of the virtual table are determined primarily by the
schema= argument.
If the schema= argument is omitted, but header is true, then
the values found in the first line of the CSV file become the column names.
If the schema= argument is omitted and header is false, then
the columns are named "c0", "c1", "c2", and so forth.
This page last modified on  2022-01-08 05:02:57 UTC


c2: Recovering Data From A Corrupt SQLite Database
c3: 1. Recovering (Some) Data From A Corrupt SQLite Database
c5: recovery.html#recovering_some_data_from_a_corrupt_sqlite_database
c4:
SQLite databases are remarkably rebust.  Application faults and
power failures typically leave the content of the database intact.
However, it is possible to corrupt an SQLite database.
For example, hardware malfunctions can damage the database file, or a
rogue process can open the database and overwrite parts of it.


Given a corrupt database file, it is sometimes desirable to try to
salvage as much data from the file as possible.  The recovery API
is designed to facilitate this.


c2: Recovering Data From A Corrupt SQLite Database
c3: 1.1. Limitations
c5: recovery.html#limitations
c4:
It is sometimes possible to perfectly restore a database that has
gone corrupt, but that is the exception.  Usually
the recovered database will be defective in a number of ways:



Some content might be permanently deleted and unrecoverable.
This can happen, for example, if a rogue process overwrites part
of the database file.


Previously deleted content might reappear.  Normally when SQLite
does a DELETE operation, it does not actually overwrite the old content
but instead remembers that space is available for reuse during the next
INSERT.  If such deleted content is still in the file when a recovery
is attempted, it might be extracted and "resurrected".


Recovered content might be altered.
For example, the value stored in a particular row
might change from 48 to 49.  Or it might change from an integer into
a string or blob.  A value that was NULL might become an integer.
A string value might become a BLOB.  And so forth.


Constraints may not be valid after recovery.  CHECK constraints, 
FOREIGN KEY constraints, UNIQUE constraints, type constraints on
STRICT tables - any of these might be violated in the recovered
database.


Content might be moved from one table into another.



The recovery API does as good of a job as it can at restoring a database,
but the results will always be suspect.  Sometimes (for example if the
corruption is restricted to indexes) the recovery will perfectly restore
the database content.  However in other cases, the recovery will be imperfect.
The impact of this imperfection depends on the application.  A database that
holds a list of bookmarks is still a list of bookmarks after recovery.
A few bookmarks might be missing or added or altered after recovery, but
the list is "fuzzy" and imperfect to begin with so adding a bit more
uncertainty will not be fatal to the application.  But if an accounting
database goes corrupt and is subsequently recovered, the books might be
out of balance.


It is best to think of the recovery API as a salvage undertaking.
Recovery will extract as much usable data as it can from the wreck
of the old database, but some parts may be damaged beyond repair and
some rework and testing should be performed prior to returning the
recovered database to service.


c2: Recovering Data From A Corrupt SQLite Database
c3: 2. Recovery Using The ".recover" Command In The CLI
c5: recovery.html#recovery_using_the_recover_command_in_the_cli
c4:
The easiest way to manually recover a corrupt database is using
the Command Line Interface or "CLI" for SQLite.  The CLI is a program
named "sqlite3".  Use it to recover a corrupt database file using
a command similar to the following:

sqlite3 corrupt.db .recover &gt;data.sql



This will generate SQL text in the file named "data.sql" that can be used
to reconstruct the original database:

sqlite3 recovered.db &lt;data.sql



The ".recover" option is actually a command that is issued to the
CLI.  That command can accept arguments.  For example, by running:

sqlite3 corruptdb ".recover --ignore-freelist" &gt;data.sql



Notice that the ".recover" command and its arguments must be contained
in quotes.  The following options are supported:



--ignore-freelist

Ignore pages of the database that appear to be part of the
freelist.  Normally the freelist is scanned, and if it contains
pages that look like they have content, that content is output.
But if the page really is on the freelist, that can mean that
previously deleted information is reintroduced into the database.


--lost-and-found TABLE

If content is found during recovery that cannot be associated
with a particular table, it is put into the "lost_and_found"
table.  Use this option to change the name of the
"lost_and_found" table to "TABLE".


--no-rowids

If this option is provided, then rowid values that are not also
INTEGER PRIMARY KEY values are not extracted from the
corrupt database.


c2: Recovering Data From A Corrupt SQLite Database
c3: 3.1. Source Code Files
c5: recovery.html#source_code_files
c4:
If you want to build the recovery API into your application, you will
need to add some source files to your build, above and beyond the usual
"sqlite3.c" and "sqlite3.h" source files.  You will need:






sqlite3recover.c

This is the main source file that implements the recovery API.





sqlite3recover.h

This is the header file that goes with sqlite3recover.h.





dbdata.c

This file implements two virtual tables name "sqlite_dbdata" and
"sqlite_dbptr" that required by sqlite3recover.c.






The two C source file above need to be linked into your application in the
same way as "sqlite3.c" is linked in.  And the header file needs to be
accessible to the compiler when the C files are being compiled.


Additionally, the application, or more specifically the sqlite3.c linked
into the application, must be compiled with the following option:

        -DSQLITE_ENABLE_DBPAGE_VTAB


c2: Recovering Data From A Corrupt SQLite Database
c3: 3.2. How To Implement Recovery
c5: recovery.html#how_to_implement_recovery
c4:
These are the basic steps needed to recover content from a corrupt
Database:



Creates an sqlite3_recover handle by calling either
sqlite3_recover_init() or sqlite3_recover_init_sql().
Use sqlite3_recover_init() to store the recovered content
in a separate database and use sqlite3_recover_init_sql()
to generate SQL text that will reconstruct the database.


Make zero or more calls to sqlite3_recover_config() to set
options on the new sqlite3_recovery handle.


Invoke sqlite3_recover_step() repeatedly
until it returns something other than SQLITE_OK. If it
returns SQLITE_DONE, then the recovery operation completed without 
error. If it returns some other non-SQLITE_OK value, then an error 
has occurred.  The sqlite3_recover_run() interface is also
available as a convenience wrapper that simply invokes
sqlite3_recover_step() repeatedly until it returns something other
than SQLITE_DONE.


Retrieves any error code and English language error message using the
sqlite3_recover_errcode() and sqlite3_recover_errmsg() interfaces,
respectively.


Invoke sqlite3_recover_finish() to destroy the sqlite3_recover object.



Details of the interface are described in comments in the
sqlite3_recover.h header file.


c2: Recovering Data From A Corrupt SQLite Database
c3: 3.3. Example Implementations
c5: recovery.html#example_implementations
c4:
Examples of how the recovery extension is used by SQLite itself
can be seen at the following links:


https://sqlite.org/src/info/30475c820dc5ab8a8?ln=999,1026

An example of the recovery extension found in the 
"fuzzcheck" testing utility in the SQLite tree.

https://sqlite.org/src/info/84bb08d8762920285f08f1c0?ln=7299,7361

The code that implements the ".recover" command in the CLI.

This page last modified on  2023-02-28 11:22:37 UTC


c2: Clustered Indexes and the WITHOUT ROWID Optimization
c3: 1. Introduction
c5: withoutrowid.html#introduction
c4:
By default, every row in SQLite has a special column, usually called the
"rowid", that uniquely identifies that row within the table.  However
if the phrase "WITHOUT ROWID" is added to the end of a CREATE TABLE statement,
then the special "rowid" column is omitted.  There are sometimes
space and performance advantages to omitting the rowid.

A WITHOUT ROWID table is a table that uses a
Clustered Index
as the primary key.


c2: Clustered Indexes and the WITHOUT ROWID Optimization
c3: 1.1. Syntax
c5: withoutrowid.html#syntax
c4:
To create a WITHOUT ROWID table, simply add the keywords "WITHOUT ROWID"
to the end of the CREATE TABLE statement.  For example:


CREATE TABLE IF NOT EXISTS wordcount(
  word TEXT PRIMARY KEY,
  cnt INTEGER
) WITHOUT ROWID;


As with all SQL syntax, the case of the keywords does not matter.
One can write "WITHOUT rowid" or "without rowid" or "WiThOuT rOwId" and
it will mean the same thing.

Every WITHOUT ROWID table must have a PRIMARY KEY.  An error is raised
if a CREATE TABLE statement with the WITHOUT ROWID clause lacks a PRIMARY KEY.

In most contexts, the special "rowid" column of normal tables can
also be called "oid" or "_rowid_".  However, only "rowid" works as
the keyword in the CREATE TABLE statement.


c2: Clustered Indexes and the WITHOUT ROWID Optimization
c3: 1.2. Compatibility
c5: withoutrowid.html#compatibility
c4:
SQLite version 3.8.2 (2013-12-06) or later
is necessary in order to use a WITHOUT
ROWID table.  An attempt to open a database that contains one or more WITHOUT
ROWID tables using an earlier version of SQLite will result in a
"malformed database schema" error.


c2: Clustered Indexes and the WITHOUT ROWID Optimization
c3: 1.3. Quirks
c5: withoutrowid.html#quirks
c4:
WITHOUT ROWID is found only in SQLite and is not compatible
with any other SQL database engine, as far as we know.
In an elegant system, all tables would behave as WITHOUT ROWID
tables even without the WITHOUT ROWID keyword.  However, when SQLite was
first designed, it used only integer rowids for row keys
to simplify the implementation.
This approach worked well for many years.  But as the demands on
SQLite grew, the need for tables in which the PRIMARY KEY really did
correspond to the underlying row key grew more acute.  The WITHOUT ROWID
concept was added
in order to meet that need without breaking backwards
compatibility with the billions of SQLite databases already in use at
the time (circa 2013).


c2: Clustered Indexes and the WITHOUT ROWID Optimization
c3: 2. Differences From Ordinary Rowid Tables
c5: withoutrowid.html#differences_from_ordinary_rowid_tables
c4:
The WITHOUT ROWID syntax is an optimization.  It provides no new
capabilities.  Anything that can be done using a WITHOUT ROWID table
can also be done in exactly the same way, and exactly the same syntax,
using an ordinary rowid table.  The only advantage of a WITHOUT ROWID
table is that it can sometimes use less disk space and/or perform a little
faster than an ordinary rowid table.

For the most part, ordinary rowid tables and WITHOUT ROWID tables
are interchangeable.  But there are some additional restrictions on
WITHOUT ROWID tables that do not apply to ordinary rowid tables:



Every WITHOUT ROWID table must have a PRIMARY KEY.
An attempt to create a WITHOUT ROWID table without a PRIMARY KEY results
in an error.


The special behaviors associated "INTEGER PRIMARY KEY" do not apply
on WITHOUT ROWID tables.
In an ordinary table, "INTEGER PRIMARY KEY" means that the column is an
alias for the rowid.  But since there is no rowid in a WITHOUT ROWID
table, that special meaning no longer applies.  An "INTEGER PRIMARY KEY"
column in a WITHOUT ROWID table works
like an "INT PRIMARY KEY" column in an ordinary table: It is a PRIMARY KEY
that has integer affinity.


AUTOINCREMENT does not work on WITHOUT ROWID tables.
The AUTOINCREMENT mechanism assumes the presence of a rowid and so it
does not work on a WITHOUT ROWID table.  An error is raised if the
 "AUTOINCREMENT" keyword is used in the CREATE TABLE statement for
a WITHOUT ROWID table.


NOT NULL is enforced on every column of the PRIMARY KEY in a WITHOUT
ROWID table.
This is in accordance with the SQL standard.  Each column of a PRIMARY KEY
is supposed to be individually NOT NULL.  However, NOT NULL was not enforced
on PRIMARY KEY columns by early versions of SQLite due to a bug.  By the
time that this bug was discovered, so many SQLite databases were already
in circulation that the decision was made not to fix this bug for fear of
breaking compatibility.  So, ordinary rowid tables in SQLite violate the
SQL standard and allow NULL values in PRIMARY KEY fields.  But WITHOUT ROWID
tables do follow the standard and will throw an error on any attempt to
insert a NULL into a PRIMARY KEY column.


The sqlite3_last_insert_rowid() function
does not work for WITHOUT ROWID tables.
Inserts into a WITHOUT ROWID do not change the value returned by the
sqlite3_last_insert_rowid() function.  The last_insert_rowid() SQL
function is also unaffected since it is just a wrapper around
sqlite3_last_insert_rowid().


The incremental blob I/O mechanism does not work
for WITHOUT ROWID tables.
Incremental BLOB I/O uses the rowid to create an sqlite3_blob object for
doing the direct I/O.  However, WITHOUT ROWID tables do not have a rowid,
and so there is no way to create an sqlite3_blob object for a WITHOUT
ROWID table.


The sqlite3_update_hook() interface does not fire callbacks for changes
to a WITHOUT ROWID table.
Part of the callback from sqlite3_update_hook() is the rowid of the table
row that has changed.  However, WITHOUT ROWID tables do not have a rowid.
Hence, the update hook is not invoked when a WITHOUT ROWID table changes.


c2: Clustered Indexes and the WITHOUT ROWID Optimization
c3: 3. Benefits Of WITHOUT ROWID Tables
c5: withoutrowid.html#benefits_of_without_rowid_tables
c4:
A WITHOUT ROWID table is an optimization that can reduce storage and
processing requirements.

In an ordinary SQLite table, the PRIMARY KEY is really just a
UNIQUE index.  The key used to look up records on disk
is the rowid.
The special "INTEGER PRIMARY KEY" column type in ordinary SQLite tables
causes the column to be an alias for the rowid, and so an INTEGER PRIMARY
KEY is a true PRIMARY KEY.  But any other kind of PRIMARY KEYs, including
"INT PRIMARY KEY" are just unique indexes in an ordinary rowid table.

Consider a table (shown below) intended to store a
vocabulary of words together with a count of the number of occurrences of
each word in some text corpus:


CREATE TABLE IF NOT EXISTS wordcount(
  word TEXT PRIMARY KEY,
  cnt INTEGER
);


As an ordinary SQLite table, "wordcount" is implemented as two
separate B-Trees.  The main table uses the hidden rowid value as the key
and stores the "word" and "cnt" columns as data.  The "TEXT PRIMARY KEY"
phrase of the CREATE TABLE statement
causes the creation of an unique index on the "word" column.  This index is a
separate B-Tree that uses "word" and the "rowid" as the key and stores no
data at all.  Note that the complete text of every "word" is stored twice:
once in the main table and again in the index.

Consider querying this table to find the number of occurrences of the
word "xsync".:


SELECT cnt FROM wordcount WHERE word='xsync';


This query first has to search the index B-Tree looking for any entry
that contains the matching value for "word".  When an entry is found in
the index, the rowid is extracted and used to search the main table.
Then the "cnt" value is read out of the main table and returned.  Hence, two
separate binary searches are required to fulfill the request.

A WITHOUT ROWID table uses a different data design for the equivalent
table.


CREATE TABLE IF NOT EXISTS wordcount(
  word TEXT PRIMARY KEY,
  cnt INTEGER
) WITHOUT ROWID;


In this latter table, there is only a single B-Tree which uses the "word"
column as its key and the "cnt" column as its data.  (Technicality:  the
low-level implementation actually stores both "word" and "cnt" in the "key"
area of the B-Tree.  But unless you are looking at the low-level byte encoding
of the database file, that fact is unimportant.)  Because there is only
a single B-Tree, the text of the "word" column is only stored once in the
database.  Furthermore, querying the "cnt" value for a specific "word"
only involves a single binary search into the main B-Tree, since the "cnt"
value can be retrieved directly from the record found by that first search
and without the need to do a second binary search on the rowid.

Thus, in some cases, a WITHOUT ROWID table can use about half the amount
of disk space and can operate nearly twice as fast.  Of course, in a
real-world schema, there will typically be secondary indices and/or
UNIQUE constraints, and the situation is more complicated.  But even then,
there can often be space and performance advantages to using WITHOUT ROWID
on tables that have non-integer or composite PRIMARY KEYs.


c2: Clustered Indexes and the WITHOUT ROWID Optimization
c3: 4. When To Use WITHOUT ROWID
c5: withoutrowid.html#when_to_use_without_rowid
c4:
The WITHOUT ROWID optimization is likely to be helpful for tables
that have non-integer or composite (multi-column) PRIMARY KEYs and that do
not store large strings or BLOBs.

WITHOUT ROWID tables will work correctly (that is to say, they
provide the correct answer) for tables with a single INTEGER PRIMARY KEY.
However, ordinary rowid tables will run faster in that case.
Hence, it is good design
to avoid creating WITHOUT ROWID tables with single-column PRIMARY KEYs
of type INTEGER.

WITHOUT ROWID tables work best when individual rows are not too large.
A good rule-of-thumb is that the average size of a single row in a
WITHOUT ROWID table should be less than about 1/20th the size of
a database page.  That means that rows should not contain more than about
50 bytes each for a 1KiB page size or about 200 bytes each for 4KiB
page size.  WITHOUT ROWID tables will work (in the sense that
they get the correct answer) for arbitrarily large rows - up to 2GB in size -
but traditional rowid tables tend to work faster for large row sizes.
This is because rowid tables are implemented as B*-Trees where
all content is stored in the leaves of the tree, whereas WITHOUT ROWID
tables are implemented using ordinary B-Trees with content stored on both
leaves and intermediate nodes.  Storing content in
intermediate nodes causes each intermediate node entry to take up more
space on the page and thus reduces the fan-out, increasing the search cost.

The "sqlite3_analyzer.exe" utility program, available as source code
in the SQLite source tree or as a precompiled binary on the
SQLite Download page, can be
used to measure the average sizes of table rows in an existing SQLite
database.

Note that except for a few corner-case differences detailed above,
WITHOUT ROWID tables and rowid tables work the same.  They both generate
the same answers given the same SQL statements.  So it is a simple matter
to run experiments on an application, late in the development cycle,
to test whether or not the use of WITHOUT ROWID tables will be helpful.
A good strategy is to simply not worry about WITHOUT ROWID until near
the end of product development, then go back and run tests to see
if adding WITHOUT ROWID to tables with non-integer PRIMARY KEYs helps
or hurts performance, and retaining the WITHOUT ROWID only in those cases
where it helps.


c2: Clustered Indexes and the WITHOUT ROWID Optimization
c3: 5. Determining If An Existing Table Is WITHOUT ROWID
c5: withoutrowid.html#determining_if_an_existing_table_is_without_rowid
c4:
A WITHOUT ROWID table returns the same content for
PRAGMA table_info and PRAGMA table_xinfo as does an ordinary
table.  But unlike an ordinary table, a WITHOUT ROWID also
responds to the PRAGMA index_info command.  The PRAGMA index_info
on a WITHOUT ROWID table returns information about the PRIMARY KEY
for the table.  In this way, the PRAGMA index_info command can be
used to unambiguously determine whether a particular table is a
WITHOUT ROWID table or an ordinary table - an ordinary table will
always return no rows but a WITHOUT ROWID table will always return
one or more rows.


c2: Why Is SQLite Coded In C
c3: 1. C Is Best
c5: whyc.html#c_is_best
c4:
Note: Sections 2.0 and 3.0 of this article were added in response
to comments on 
Hacker News and
Reddit.



Since its inception on 2000-05-29, SQLite has been implemented in generic C.
C was and continues to be the best language for implementing a software
library like SQLite.  There are no plans to recode SQLite in any other
programming language at this time.


The reasons why C is the best language to implement SQLite include:



 Performance
 Compatibility
 Low-dependency
 Stability


c2: Why Is SQLite Coded In C
c3: 1.1. Performance
c5: whyc.html#performance
c4:
An intensively used low-level library like SQLite needs to be fast.
(And SQLite is fast, see Internal Versus External BLOBs and
35% Faster Than The Filesystem for example.)

C is a great language for writing fast code.  C is sometimes
described as "portable assembly language".  It enables to developers
to code as close to the underlying hardware as possible while still
remaining portable across platforms.

Other programming languages sometimes claim to be "as fast as C".
But no other language claims to be faster than C for general-purpose
programming, because none are.


c2: Why Is SQLite Coded In C
c3: 1.2. Compatibility
c5: whyc.html#compatibility
c4:
Nearly all systems have the ability to call libraries
written in C.  This is not true of other implementation languages.

So, for example, Android applications written in Java are able to
invoke SQLite (through an adaptor).  Maybe it would have been more
convenient for Android if SQLite had been coded in Java as that would
make the interface simpler.  However, on iPhone applications are coded
in Objective-C or Swift, neither of which have the ability to call
libraries written in Java.  Thus, SQLite would be unusable on iPhones
had it been written in Java.


c2: Why Is SQLite Coded In C
c3: 1.3. Low-Dependency
c5: whyc.html#low_dependency
c4:
Libraries written in C do not have a huge run-time dependency.
In its minimum configuration, SQLite requires only the following
routines from the standard C library:






 memcmp()
 memcpy()
 memmove()
 memset()


&nbsp;&nbsp;&nbsp;


 strcmp()
 strlen()
 strncmp()







In a more complete build, SQLite also uses library routines like
malloc() and free() and operating system interfaces for opening, reading,
writing, and closing files.  But even then, the number of dependencies
is very small.  Other "modern" language, in contrast, often require
multi-megabyte runtimes loaded with thousands and thousands of interfaces.


c2: Why Is SQLite Coded In C
c3: 1.4. Stability
c5: whyc.html#stability
c4:
The C language is old and boring.
It is a well-known and well-understood language.
This is exactly what one wants when developing a module like SQLite.
Writing a small, fast, and reliable database engine is hard enough as it
is without the implementation language changing out from under you with
each update to the implementation language specification.


c2: Why Is SQLite Coded In C
c3: 2. Why Isn't SQLite Coded In An Object-Oriented Language?
c5: whyc.html#why_isn_t_sqlite_coded_in_an_object_oriented_language_
c4:
Some programmers cannot imagine developing a complex system like
SQLite in a language that is not "object oriented".  So why is
SQLite not coded in C++ or Java?



Libraries written in C++ or Java can generally only be used by
applications written in the same language. It is difficult to
get an application written in Haskell or Java to invoke a library
written in C++.  On the other hand, libraries written in C are
callable from any programming language.


Object-Oriented is a design pattern, not a programming language.
You can do object-oriented programming in any language you want,
including assembly language.  Some languages (ex: C++ or Java) make
object-oriented easier.  But you can still do object-oriented programming
in languages like C.


Object-oriented is not the only valid design pattern.
Many programmers have been taught to think purely in terms of
objects.  And, to be fair, objects are often a good way to
decompose a problem.  But objects are not the only way, and are
not always the best way to decompose a problem.  Sometimes good old
procedural code is easier to write, easier to maintain and understand,
and faster than object-oriented code.


When SQLite was first being developed, Java was a young and immature
language.  C++ was older, but was undergoing such growing pains that
it was difficult to find any two C++ compilers that worked the same
way.  So C was definitely a better choice back when SQLite was first
being developed.  The situation is less stark now, but there is little
to no benefit in recoding SQLite at this point.


c2: Why Is SQLite Coded In C
c3: 3. Why Isn't SQLite Coded In A "Safe" Language?
c5: whyc.html#why_isn_t_sqlite_coded_in_a_safe_language_
c4:
There has lately been a lot of interest in "safe" programming languages
like Rust or Go in which it is impossible, or is at least difficult, to make
common programming errors like memory leaks or array overruns.  So the
question often arises as to why SQLite is not coded in a "safe" language.



None of the safe programming languages existed for the first 10 years
of SQLite's existence.  SQLite could be recoded in Go or Rust, but doing
so would probably introduce far more bugs than would be fixed, and it
seems also likely to result in slower code.


Safe languages insert additional machine branches to do things like
verify that array accesses are in-bounds.  In correct code, those
branches are never taken.  That means that the machine code cannot
be 100% branch tested, which is an important component of SQLite's
quality strategy.


Safe languages usually want to abort if they encounter an out-of-memory
(OOM) situation.  SQLite is designed to recover gracefully from an OOM.
It is unclear how this could be accomplished in the current crop of
safe languages.


All of the existing safe languages are new.  The developers of SQLite
applaud the efforts of computer language researchers in trying to
develop languages that are easier to program safely.  We encourage these
efforts to continue.  But we ourselves are more interested in old and
boring languages when it comes to implementing SQLite.



All that said, it is possible that SQLite might
one day be recoded in Rust.  Recoding SQLite in Go is unlikely
since Go hates assert().  But Rust is a possibility.  Some
preconditions that must occur before SQLite is recoded in Rust
include:



 Rust needs to mature a little more, stop changing so fast, and
     move further toward being old and boring.
 Rust needs to demonstrate that it can be used to create general-purpose
     libraries that are callable from all other programming languages.
 Rust needs to demonstrate that it can produce object code that
     works on obscure embedded devices, including devices that lack
     an operating system.
 Rust needs to pick up the necessary tooling that enables one to
     do 100% branch coverage testing of the compiled binaries.
 Rust needs a mechanism to recover gracefully from OOM errors.
 Rust needs to demonstrate that it can do the kinds of work that
     C does in SQLite without a significant speed penalty.



If you are a "rustacean" and feel that Rust already meets the
preconditions listed above, and that SQLite should be recoded in
Rust, then you are welcomed and encouraged
to contact the SQLite developers privately
and argue your case.
This page last modified on  2022-07-29 00:41:26 UTC


c2: Application-Defined SQL Functions
c3: 1. Executive Summary
c5: appfunc.html#executive_summary
c4:
Applications that use SQLite can define custom SQL functions that call
back into application code to compute their results.  The custom SQL
function implementations can be embedded in the application code itself,
or can be loadable extensions.

Application-defined or custom SQL functions are created using the
sqlite3_create_function() family of interfaces.
Custom SQL functions can be scalar functions, aggregate functions,
or window functions.
Custom SQL functions can have any number of arguments from 0 up to
SQLITE_MAX_FUNCTION_ARG.
The sqlite3_create_function() interface specifies callbacks that are
invoked to carry out the processing for the new SQL function.

SQLite also supports custom table-valued functions, but they are
implemented by a different mechanism that is not covered in this document.


c2: Application-Defined SQL Functions
c3: 2. Defining New SQL Functions
c5: appfunc.html#defining_new_sql_functions
c4:
The sqlite3_create_function() family of interfaces is used to create
new custom SQL functions.  Each member of this family is a wrapper around
a common core.  All family members accomplish the same thing; they merely
have different calling signatures.


sqlite3_create_function() &rarr;
The original version of sqlite3_create_function() allows the application
to create a single new SQL function that can be either a scalar or an
aggregate.  The name of the function is specified using UTF8.

sqlite3_create_function16() &rarr;
This variant works exactly like the sqlite3_create_function() original
except that the name of the function itself is specified as a UTF16
string rather than as a UTF8 string.

sqlite3_create_function_v2() &rarr;
This variant works like the original sqlite3_create_function() except
that it includes an additional parameter that is a pointer to a
destructor for the sqlite3_user_data() pointer that is passed in
as the 5th argument to all of the sqlite3_create_function() variants.
That destructor function (if it is non-NULL) is called when the
custom function is deleted - usually when the database connection is
closing.

sqlite3_create_window_function() &rarr;
This variant works like the original sqlite3_create_function() except
that it accepts a different set of callback pointers - the callback
pointers used by window function definitions.


c2: Application-Defined SQL Functions
c3: 2.1. Common Parameters
c5: appfunc.html#common_parameters
c4:
Many of the parameters passed to the sqlite3_create_function()
family of interfaces are common across the entire family.


db &rarr;
The 1st parameter is always a pointer to the database connection
on which the custom SQL function will work.  Custom SQL functions are
created separately for each database connection.  There is no short-hand
mechanism for creating SQL functions that work across all database
connections.

zFunctionName &rarr;
The 2nd parameter is the name of the SQL function that is being
created.  The name is usually in UTF8, except that the name should
be in UTF16 in the native byte order for sqlite3_create_function16().

The maximum length of a SQL function name is 255 bytes of UTF8.
Any attempt to create a function with a longer name will result in
an SQLITE_MISUSE error.

The SQL function creation interfaces may be called multiple
times with the same function name.
If two calls have the same function number but a different number of
arguments, for example, then two variants of the SQL function will
be registered, each taking a different number of arguments.

nArg &rarr;
The 3rd parameter is always the number of arguments that the function
accepts.  The value must be an integer between -1 and 
SQLITE_MAX_FUNCTION_ARG (default value: 127).  A value of -1 means
that the SQL function is a variadic function that can take any number
of arguments between 0 and SQLITE_MAX_FUNCTION_ARG.

eTextRep &rarr;
The 4th parameter is a 32-bit integer flag whose bits convey various
properties about the new function.  The original purpose of this
parameter was to specify the preferred text encoding for the function,
using one of the following constants:

 SQLITE_UTF8
 SQLITE_UTF16BE
 SQLITE_UTF16LE

All custom SQL functions will accept text in any encoding.  Encoding
conversions will happen automatically.  The preferred encoding merely
specifies the encoding for which the function implementation is optimized.
It is possible to specify multiple functions with the same name and the
same number of arguments, but different preferred encodings and different
callbacks used to implement the function, and SQLite will chose the
set of callbacks for which the input encodings most closely match the
preferred encoding.

The 4th parameter as more recently be extended with additional flag bits
to convey additional information about the function.  The additional
bits include:

 SQLITE_DETERMINISTIC
 SQLITE_DIRECTONLY
 SQLITE_INNOCUOUS
 SQLITE_SUBTYPE


Additional bits may be added in future versions of SQLite.

pApp &rarr;
The 5th parameter is an arbitrary pointer that is passed through
into the callback routines.  SQLite itself does nothing with this
pointer, except to make it available to the callbacks, and to pass
it into the destructor when the function is unregistered.


c2: Application-Defined SQL Functions
c3: 2.2. Multiple Calls To sqlite3_create_function() For The Same Function
c5: appfunc.html#multiple_calls_to_sqlite3_create_function_for_the_same_function
c4:
It is common for an application to invoke sqlite3_create_function() multiple
times for the same SQL function.  For example, if an SQL function can take
either 2 or 3 arguments, then sqlite3_create_function() would be invoked
once for the 2-argument version and a second time for the 3-argument version.
The underlying implementation (the callbacks) can be different for both
variants.


An application can also register multiple SQL functions with the same name
and same number of arguments, but a different preferred text encoding.
In that case, SQLite will invoke the function using the callbacks for
the version whose preferred text encoding most closely matches the database
text encoding.  In this way, multiple implementations of the same function
can be provided that are optimized for UTF8 or UTF16.


If multiple calls to sqlite3_create_function() specify the same function name,
and the same number of arguments, and the same preferred text encoding, then
the callbacks and other parameters of the second call overwrite the first,
and the destructor callback from the first call (if it exists) is invoked.


c2: Application-Defined SQL Functions
c3: 2.3.1. The Scalar Function Callback
c5: appfunc.html#the_scalar_function_callback
c4:
Scalar SQL functions are implemented by a single callback in the
xFunc parameter to sqlite3_create_function().
The following code demonstrations the implementation of a "noop(X)"
scalar SQL function that merely returns its argument:

static void noopfunc(
  sqlite3_context *context,
  int argc,
  sqlite3_value **argv
){
  assert( argc==1 );
  sqlite3_result_value(context, argv&#91;0&#93;);
}



The 1st parameter, context, is a pointer to an opaque object
that describes the content from which the SQL function was invoked.  This
context point becomes the first parameter to many other routines that
the function implement might to invoke, including:



sqlite3_aggregate_context
sqlite3_context_db_handle
sqlite3_get_auxdata
sqlite3_result_blob
sqlite3_result_blob64
sqlite3_result_double
sqlite3_result_error
sqlite3_result_error16
sqlite3_result_error_code
sqlite3_result_error_nomem
sqlite3_result_error_toobig
sqlite3_result_int
sqlite3_result_int64
sqlite3_result_null
sqlite3_result_pointer
sqlite3_result_subtype
sqlite3_result_text
sqlite3_result_text16
sqlite3_result_text16be
sqlite3_result_text16le
sqlite3_result_text64
sqlite3_result_value
sqlite3_result_zeroblob
sqlite3_result_zeroblob64
sqlite3_set_auxdata
sqlite3_user_data




The sqlite3_result() family of functions are
used to specify the result of the scalar SQL function.  One or more of
these should be invoked by the callback to set the function return value.
If none of these routines are invoked for a specific callback, then the
return value will be NULL.

The sqlite3_user_data() routine returns a copy of the pArg
pointer that was given to sqlite3_create_function() when the SQL
function was created.

The sqlite3_context_db_handle() routine returns a pointer to the
database connection object.

The sqlite3_aggregate_context() routine is used only in the
implementations of aggregate and window functions.  Scalar functions
may not use sqlite3_aggregate_context().  The sqlite3_aggregate_context()
function is included in the interface list only for completeness.


The 2nd and 3rd arguments to the scalar SQL function implemenetation,
argc and argv, are
the number of arguments to the SQL function itself and the values for
each argument of the SQL function.
Argument values can be of any datatype and are thus stored in
instances of the sqlite3_value object.
Specific C-language values can be extracted from this object using
the sqlite3_value() family of interfaces.


c2: Application-Defined SQL Functions
c3: 2.3.2. The Aggregate Function Callbacks
c5: appfunc.html#the_aggregate_function_callbacks
c4:
Aggregate SQL functions are implemented by using two callback
functions, xStep and xFinal.  The xStep() function 
is called for each row of the aggregate and the xFinal() function
is invoked to compute the final answer at the end.
The following (slightly simplified) version of the built-in
count() function illustrates:

typedef struct CountCtx CountCtx;
struct CountCtx {
  i64 n;
};
static void countStep(sqlite3_context *context, int argc, sqlite3_value **argv){
  CountCtx *p;
  p = sqlite3_aggregate_context(context, sizeof(*p));
  if( (argc==0 || SQLITE_NULL!=sqlite3_value_type(argv&#91;0&#93;)) && p ){
    p->n++;
  }
}   
static void countFinalize(sqlite3_context *context){
  CountCtx *p;
  p = sqlite3_aggregate_context(context, 0);
  sqlite3_result_int64(context, p ? p->n : 0);
}


Recall that there are two versions of the count() aggregate.
With zero arguments, count() returns a count of the number of rows.
With one argument, count() returns the number of times that the
argument was non-NULL.

The countStep() callback is invoked once for each row in the aggregate.
As you can see, the count is incremented if either there are no arguments,
or if the one argument is not NULL.

The step function for an aggregate should always begin with a call
to the sqlite3_aggregate_context() routine to fetch the persistent
state of the aggregate function.  On the first invocation of the step()
function, the aggregate context is initialized to a block of memory
that is N bytes in size, where N is the second parameter to
sqlite3_aggregate_context() and that memory is zeroed.  On all subsequent
calls to the step() function, the same block of memory is returned.
Except, sqlite3_aggregate_context() might return NULL in the case of
an out-of-memory error, so aggregate functions should be prepared to
deal with that case.

After all rows are processed the countFinalize() routine is called
exactly once.  This routine computes the final result and invokes
one of the sqlite3_result() family of functions
to set the final result.  The aggregate context will be freed automatically
by SQLite, though the xFinalize() routine must clean up any substructure
associated with the aggregate context before it returns.  If the xStep()
method is called one or more times, then SQLite guarantees thta the
xFinal() method will be called at once, even if the query aborts.


c2: Application-Defined SQL Functions
c3: 2.3.3. The Window Function Callbacks
c5: appfunc.html#the_window_function_callbacks
c4:
Window functions use the same xStep() and xFinal() callbacks that
aggregate functions use, plus two others: xValue and xInverse.
See the documentation on
application-defined window functions for further details.


c2: Application-Defined SQL Functions
c3: 2.3.4. Examples
c5: appfunc.html#examples
c4:
There are dozens and dozens of SQL function implementations scattered
throughout the SQLite source code that can be used as example applications.
The built-in SQL functions use the same interface as application-defined
SQL functions, so built-in functions can be used as examples too.
Search for "sqlite3_context" in the SQLite source code to find examples.


c2: Application-Defined SQL Functions
c3: 3. Security Implications
c5: appfunc.html#security_implications
c4:
Application-defined SQL functions can become security vulnerabilities if
not carefully managed.  Suppose, for example, an application defines
a new "system(X)" SQL function that runs its argument X as a command and
returns the integer result code.  Perhaps the implementation is like this:

static void systemFunc(
  sqlite3_context *context,
  int argc,
  sqlite3_value **argv
){
  const char *zCmd = (const char*)sqlite3_value_text(argv&#91;0&#93;);
  if( zCmd!=0 ){
    int rc = system(zCmd);
    sqlite3_result_int(context, rc);
  }
}



This is a function with powerful side-effects.  Most programmers would
be naturally cautious about using it, but probably would not see the
harm in merely having it available.  But there is great risk in merely
defining such a function, even if the application itself never invokes
it!


Suppose the application normally does a query against table TAB1
when it starts up.  If an attacker can gain access to the database
file and modify the schema like this:

ALTER TABLE tab1 RENAME TO tab1_real;
CREATE VIEW tab1 AS SELECT * FROM tab1 WHERE system('rm -rf *') IS NOT NULL;



Then, when the application attempts to open the database, register the
system() function, then run an innocent query against the "tab1" table,
it instead deletes all the files in its working directory.  Yikes!


To prevent this kind of mischief, applications that create their own
custom SQL functions should take one or more of the following safety
precautions.  The more precautions taken the better:



Invoke sqlite3_db_config(db,SQLITE_DBCONFIG_TRUSTED_SCHEMA,0,0)
on each database connection as soon as it is opened.
This prevents application-defined functions from being used in places
where an attacker might be able to surreptiously invoke them by modifying
a database schema:

 In VIEWs.
 In TRIGGERs.
 In CHECK constraints of a table definition.
 In DEFAULT constraints of a table definition.
 In the definitions of generated columns.
 In the expression part of an index on an expression.
 In the WHERE clause of a partial index.


To put it another way, this setting requires that application-defined
functions only be run directly by top-level SQL invoked from the application
itself, not as a consequence of doing some other innocent-looking query.


Use the PRAGMA trusted_schema=OFF SQL statement to disable trusted
schema.  This has the same effect as the previous bullet, but does not
require the use of C-code and hence can be performed in programs written
in another programming language and that do not have access SQLite
C-language APIs.


Compile SQLite using the -DSQLITE_TRUSTED_SCHEMA=0 compile-time option.
This make SQLite distrust application-defined functions inside of
the schema by default.


If any application-defined SQL functions have potentially dangerous
side-effects, or if they could potentially leak sensitive information
to an attacker if misused, then tag those functions using the
SQLITE_DIRECTONLY option on the "enc" parameter.  This means
that the function can never be run from schema-code even if the
trusted-schema option is on.


Never tag an application-defined SQL function with SQLITE_INNOCUOUS
unless you really need to and you have checked the implementation closely
and are certain that it can do no harm even if it falls under the
control of an attacker.

This page last modified on  2024-04-16 17:22:18 UTC


c2: Defense Against The Dark Arts
c3: 1. SQLite Always Validates Its Inputs
c5: security.html#sqlite_always_validates_its_inputs
c4:
SQLite should never crash, overflow a buffer, leak memory,
or exhibit any other harmful behavior, even when presented with
maliciously malformed SQL inputs or database files.  SQLite should
always detect erroneous inputs and raise an error, not crash or
corrupt memory.
Any malfunction caused by an SQL input or database file
is considered a serious bug and will be promptly addressed when
brought to the attention of the SQLite developers.  SQLite is
extensively fuzz-tested to help ensure that it is resistant
to these kinds of errors.


Nevertheless, bugs happen.
If you are writing an application that sends untrusted SQL inputs
or database files to SQLite, there are additional steps you can take
to help reduce the attack surface and
prevent zero-day exploits caused by undetected bugs.


c2: Defense Against The Dark Arts
c3: 1.1. Untrusted SQL Inputs
c5: security.html#untrusted_sql_inputs
c4:
Applications that accept untrusted SQL inputs should take the following
precautions:



Set the SQLITE_DBCONFIG_DEFENSIVE flag.
This prevents ordinary SQL statements from deliberately corrupting the 
database file.  SQLite should be proof against attacks that involve both
malicious SQL inputs and a maliciously corrupted database file at the
same time.  Nevertheless, denying a script-only attacker access to 
corrupt database inputs provides an extra layer of defense.


Reduce the limits that SQLite imposes on inputs.  This can help prevent
denial of service attacks and other kinds of mischief that can occur
as a result of unusually large inputs.  You can do this either at compile-time
using -DSQLITE_MAX_... options, or at run-time using the
sqlite3_limit() interface.  Most applications can reduce limits
dramatically without impacting functionality.  The table below
provides some suggestions, though exact values will vary depending
on the application:


Limit SettingDefault ValueHigh-security Value
LIMIT_LENGTH1,000,000,0001,000,000
LIMIT_SQL_LENGTH1,000,000,000100,000
LIMIT_COLUMN2,000100
LIMIT_EXPR_DEPTH1,00010
LIMIT_COMPOUND_SELECT5003
LIMIT_VDBE_OP250,000,00025,000
LIMIT_FUNCTION_ARG1278
LIMIT_ATTACH100
LIMIT_LIKE_PATTERN_LENGTH50,00050
LIMIT_VARIABLE_NUMBER99910
LIMIT_TRIGGER_DEPTH1,00010



Consider using the sqlite3_set_authorizer() interface to limit
the scope of SQL that will be processed.  For example, an application
that does not need to change the database schema might add an
sqlite3_set_authorizer() callback that causes any CREATE or DROP
statement to fail.


The SQL language is very powerful, and so it is always possible for
malicious SQL inputs (or erroneous SQL inputs caused by an application
bug) to submit SQL that runs for a very long time.  To prevent this
from becoming a denial-of-service attack, consider using the
sqlite3_progress_handler() interface to invoke a callback periodically
as each SQL statement runs, and have that callback return non-zero to
abort the statement if the statement runs for too long.  Alternatively,
set a timer in a separate thread and invoke sqlite3_interrupt() when
the timer goes off to prevent the SQL statement from running forever.


Limit the maximum amount of memory that SQLite will allocate using
the sqlite3_hard_heap_limit64() interface.  This helps prevent
denial-of-service attacks.  To find out how much heap space an
application actually needs, run the it against typical inputs and
then measure the maximum instantaneous memory usage with the 
sqlite3_memory_highwater() interface.  Set the hard heap limit
to the maximum observed instantaneous memory usage plus some margin.


Consider setting the SQLITE_MAX_ALLOCATION_SIZE compile-time option
to something smaller than its default value of 2147483391 (0x7ffffeff).
A value of 100000000 (100 million) or even smaller would not be unreasonable,
depending on the application.


For embedded systems, consider compiling SQLite with the
-DSQLITE_ENABLE_MEMSYS5 option and then providing SQLite with
a fixed chunk of memory to use as its heap via the
sqlite3_config(SQLITE_CONFIG_HEAP) interface. This will
prevent malicious SQL from executing a denial-of-service attack
by using an excessive amount of memory. If (say) 5 MB of memory
is provided for SQLite to use, once that much has been consumed,
SQLite will start returning SQLITE_NOMEM errors rather than
soaking up memory needed by other parts of the application.
This also sandboxes SQLite's memory so that a write-after-free
error in some other part of the application will not cause
problems for SQLite, or vice versa.



To control memory usage in the printf() SQL function, compile
with "-DSQLITE_PRINTF_PRECISION_LIMIT=100000" or some similarly
reasonable value.
This #define limits the width and precision for %-substitutions in the
printf() function, and thus prevents a hostile SQL statement from
consuming large amounts of RAM via constructs such as
"printf('%1000000000s','hi')".

Note that SQLite uses its built-in printf() internally to help it
format the sql column in the sqlite_schema table.  For that reason,
no table, index, view, or trigger definition can be much larger than the
precision limit.  You can set a precision limit of less than 100000,
but be careful that whatever precision limit you use is at least as
long as the longest CREATE statement in your schema.


c2: Defense Against The Dark Arts
c3: 1.2. Untrusted SQLite Database Files
c5: security.html#untrusted_sqlite_database_files
c4:
Applications that read or write SQLite database files of uncertain
provenance should take precautions enumerated below.

Even if the application does not deliberately accept database files 
from untrusted sources, beware of attacks in which a local 
database file is altered.  For best security, any database file which 
might have ever been writable by an agent in a different security domain
should be treated as suspect.



If the application includes any custom SQL functions or 
custom virtual tables that have side effects or that might leak
privileged information, then the application should use one or more
of the techniques below to prevent a maliciously crafted database
schema from surreptitiously running those SQL functions and/or
virtual tables for nefarious purposes:

 Invoke sqlite3_db_config(db,SQLITE_DBCONFIG_TRUSTED_SCHEMA,0,0)
     on each database connection as soon as it is opened.
 Run the PRAGMA trusted_schema=OFF statement on each database connection
     as soon as it is opened.
 Compile SQLite using the -DSQLITE_TRUSTED_SCHEMA=0 compile-time option.
 Disable the surreptitious use of custom SQL functions and virtual tables
     by setting the SQLITE_DIRECTONLY flag on all custom SQL functions and
     the SQLITE_VTAB_DIRECTONLY flag on all custom virtual tables.



If the application does not use triggers or views, consider disabling the
unused capabilities with:

sqlite3_db_config(db,SQLITE_DBCONFIG_ENABLE_TRIGGER,0,0);
sqlite3_db_config(db,SQLITE_DBCONFIG_ENABLE_VIEW,0,0);





For reading database files that are unusually high-risk, such as database
files that are received from remote machines, and possibly from anonymous
contributors, the following extra precautions
might be justified.  These added defenses come with performance costs,
however, and so may not be appropriate in every situation:



Run PRAGMA integrity_check or PRAGMA quick_check on the database
as the first SQL statement after opening the database files and
prior to running any other SQL statements.  Reject and refuse to
process any database file containing errors.


Enable the PRAGMA cell_size_check=ON setting.



Do not enable memory-mapped I/O.
In other words, make sure that PRAGMA mmap_size=0.


c2: Defense Against The Dark Arts
c3: 2. Summary
c5: security.html#summary
c4:
The precautions above are not required in order to use SQLite safely
with potentially hostile inputs.
However, they do provide an extra layer of defense against zero-day
exploits and are encouraged for applications that pass data from
untrusted sources into SQLite.
This page last modified on  2024-01-16 14:06:27 UTC


c2: Query Planning
c3: Overview
c5: queryplanner.html#overview
c4:
The best feature of SQL (in all its implementations, not just SQLite)
is that it is a declarative language, not a procedural
language.  When programming in SQL you tell the system what you
want to compute, not how to compute it.  The task of figuring out
the how is delegated to the query planner subsystem within 
the SQL database engine.

For any given SQL statement, there might be hundreds or thousands or
even millions of different algorithms of performing the operation.  All
of these algorithms will get the correct answer, though some will run
faster than others.
The query planner is an 
AI that 
tries to pick the fastest and most efficient algorithm for each SQL
statement.



Most of the time, the query planner in SQLite does a good job.
However, the query planner needs indices to
work with.  
These indices must normally be added by programmers.
Rarely, the query planner AI will make a suboptimal algorithm
choice.
In those cases, programmers may want to provide additional
hints to help the query planner do a better job.



This document provides background information about how the
SQLite query planner and query engine work.
Programmers can use this information to help create better
indexes, and provide hints to help the query planner when
needed.



Additional information is provided in the
SQLite query planner and 
next generation query planner documents.


c2: Query Planning
c3: 1.1. Tables Without Indices
c5: queryplanner.html#_tables_without_indices
c4:
Most tables in SQLite consist of zero or more rows with a unique integer
key (the rowid or INTEGER PRIMARY KEY) followed by content.  
(The exception is WITHOUT ROWID tables.)
The rows
are logically stored in order of increasing rowid.  As an example, this
article uses a table named "FruitsForSale" which relates various fruits 
to the state
where they are grown and their unit price at market.  The schema is this:



CREATE TABLE FruitsForSale(
  Fruit TEXT,
  State TEXT,
  Price REAL
);




With some (arbitrary) data, such a table might be logically stored on disk
as shown in figure 1:





Figure 1: Logical Layout Of Table "FruitsForSale"




In this example, the rowids are not
consecutive but they are ordered.  SQLite usually creates rowids beginning
with one and increasing by one with each added row.  But if rows are 
deleted, gaps can appear in the sequence.  And the application can control
the rowid assigned if desired, so that rows are not necessarily inserted 
at the bottom.  But regardless of what happens, the rowids are always 
unique and in strictly ascending order.



Suppose you want to look up the price of peaches.  The query would
be as follows:



SELECT price FROM fruitsforsale WHERE fruit='Peach';




To satisfy this query, SQLite reads every row out of the
table, checks to see if the "fruit" column has the value of "Peach" and if
so, outputs the "price" column from that row.  The process is illustrated
by figure 2 below.
This is algorithm is called a full table scan 
since the entire content of the
table must be read and examined in order to find the one row of interest.
With a table of only 7 rows, a full table scan is acceptable, 
but if the table contained 7 million rows, a full table scan might read 
megabytes of content in order to find a single 8-byte number.  
For that reason, one normally tries to avoid full table scans.





Figure 2: Full Table Scan


c2: Query Planning
c3: 1.2. Lookup By Rowid
c5: queryplanner.html#_lookup_by_rowid
c4:
One technique for avoiding a full table scan is to do lookups by
rowid (or by the equivalent INTEGER PRIMARY KEY).   To lookup the
price of peaches, one would query for the entry with a rowid of 4:



SELECT price FROM fruitsforsale WHERE rowid=4;




Since the information is stored in the table in rowid order, SQLite
can find the correct row using a binary search.
If the table contains N elements, the time required to look up the
desired row is proportional to logN rather than being proportional
to N as in a full table scan.  If the table contains 10 million elements,
that means the query will be on the order of N/logN or about 1 million
times faster.





Figure 3: Lookup By Rowid


c2: Query Planning
c3: 1.3. Lookup By Index
c5: queryplanner.html#_lookup_by_index
c4:
The problem with looking up information by rowid is that you probably
do not care what the price of "item 4" is - you want to know the price
of peaches.  And so a rowid lookup is not helpful.



To make the original query more efficient, we can add an index on the
"fruit" column of the "fruitsforsale" table like this:



CREATE INDEX Idx1 ON fruitsforsale(fruit);




An index is another table similar to the original "fruitsforsale" table
but with the content (the fruit column in this case) stored in front of the
rowid and with all rows in content order.
Figure 4 gives a logical view of the Idx1 index.
The "fruit" column is the primary key used to order the elements of the
table and the "rowid" is the secondary key used to break the tie when
two or more rows have the same "fruit".  In the example, the rowid
has to be used as a tie-breaker for the "Orange" rows.
Notice that since the rowid
is always unique over all elements of the original table, the composite key
of "fruit" followed by "rowid" will be unique over all elements of the index.





Figure 4: An Index On The Fruit Column




This new index can be used to implement a faster algorithm for the
original "Price of Peaches" query.



SELECT price FROM fruitsforsale WHERE fruit='Peach';




The query starts by doing a binary search on the Idx1 index for entries
that have fruit='Peach'.  SQLite can do this binary search on the Idx1 index
but not on the original FruitsForSale table because the rows in Idx1 are sorted
by the "fruit" column.  Having found a row in the Idx1 index that has
fruit='Peach', the database engine can extract the rowid for that row.
Then the database engines does a second binary search
on the original FruitsForSale table to find the
original row that contains fruit='Peach'.  
From the row in the FruitsForSale table,
SQLite can then extract the value of the price column.
This procedure is illustrated by figure 5.





Figure 5: Indexed Lookup For The Price Of Peaches




SQLite has to do two binary searches to find the price of peaches using
the method show above.  But for a table with a large number of rows, this
is still much faster than doing a full table scan.


c2: Query Planning
c3: 1.4. Multiple Result Rows
c5: queryplanner.html#_multiple_result_rows
c4:
In the previous query the fruit='Peach' constraint narrowed the result
down to a single row.  But the same technique works even if multiple
rows are obtained.  Suppose we looked up the price of Oranges instead of
Peaches:



SELECT price FROM fruitsforsale WHERE fruit='Orange'




Figure 6: Indexed Lookup For The Price Of Oranges




In this case, SQLite still does a single binary search to find the first
entry of the index where fruit='Orange'.  Then it extracts the rowid from
the index and uses that rowid to lookup the original table entry via
binary search and output the price from the original table.  But instead
of quitting, the database engine then advances to the next row of index
to repeat the process for next fruit='Orange' entry.  Advancing to the
next row of an index (or table) is much less costly than doing a binary
search since the next row is often located on the same database page as
the current row.  In fact, the cost of advancing to the next row is so
cheap in comparison to a binary search that we usually ignore it.  So
our estimate for the total cost of this query is 3 binary searches.
If the number of rows of output is K and the number of rows in the table
is N, then in general the cost of doing the query is proportional
to (K+1)*logN.


c2: Query Planning
c3: 1.5. Multiple AND-Connected WHERE-Clause Terms
c5: queryplanner.html#_multiple_and_connected_where_clause_terms
c4:
Next, suppose that you want to look up the price of not just any orange,
but specifically California-grown oranges.  The appropriate query would
be as follows:



SELECT price FROM fruitsforsale WHERE fruit='Orange' AND state='CA'




Figure 7: Indexed Lookup Of California Oranges




One approach to this query is to use the fruit='Orange' term of the WHERE
clause to find all rows dealing with oranges, then filter those rows
by rejecting any that are from states other than California.  This
process is shown by figure 7 above.  This is a perfectly
reasonable approach in most cases.  Yes, the database engine did have
to do an extra binary search for the Florida orange row that was
later rejected, so it was not as efficient as we might hope, though
for many applications it is efficient enough.  



Suppose that in addition to the index on "fruit" there was also
an index on "state".



CREATE INDEX Idx2 ON fruitsforsale(state);




Figure 8: Index On The State Column




The "state" index works just like the "fruit" index in that it is a
new table with an extra column in front of the rowid and sorted by
that extra column as the primary key.  The only difference is that
in Idx2, the first column is "state" instead of "fruit" as it is with
Idx1.  In our example data set, there is more redundancy in the "state"
column and so they are more duplicate entries.  The ties are still
resolved using the rowid.



Using the new Idx2 index on "state", SQLite has another option for
lookup up the price of California oranges:  it can look up every row
that contains fruit from California and filter out those rows that
are not oranges.





Figure 9: Indexed Lookup Of California Oranges




Using Idx2 instead of Idx1 causes SQLite to examine a different set of
rows, but it gets the same answer in the end (which is very important -
remember that indices should never change the answer, only help SQLite to
get to the answer more quickly) and it does the same amount of work.
So the Idx2 index did not help performance in this case.



The last two queries take the same amount of time, in our example.
So which index, Idx1 or Idx2, will SQLite choose?  If the
ANALYZE command has been run on the database, so that SQLite has
had an opportunity to gather statistics about the available indices,
then SQLite will know that the Idx1 index usually narrows the search
down to a single item (our example of fruit='Orange' is the exception
to this rule) whereas the Idx2 index will normally only narrow the 
search down to two rows.  So, if all else is equal, SQLite will
choose Idx1 with the hope of narrowing the search to as small
a number of rows as possible.  This choice is only possible because
of the statistics provided by ANALYZE.  If ANALYZE has not been
run then the choice of which index to use is arbitrary.


c2: Query Planning
c3: 1.6. Multi-Column Indices
c5: queryplanner.html#_multi_column_indices
c4:
To get the maximum performance out of a query with multiple AND-connected
terms in the WHERE clause, you really want a multi-column index with
columns for each of the AND terms.  In this case we create a new index
on the "fruit" and "state" columns of FruitsForSale:



CREATE INDEX Idx3 ON FruitsForSale(fruit, state);




Figure 1: A Two-Column Index




A multi-column index follows the same pattern as a single-column index;
the indexed columns are added in front of the rowid.  The only difference
is that now multiple columns are added.  The left-most column is the
primary key used for ordering the rows in the index.  The second column is
used to break ties in the left-most column.  If there were a third column,
it would be used to break ties for the first two columns.  And so forth for
all columns in the index.  Because rowid is guaranteed
to be unique, every row of the index will be unique even if all of the
content columns for two rows are the same.  That case does not happen
in our sample data, but there is one case (fruit='Orange') where there
is a tie on the first column which must be broken by the second column.



Given the new multi-column Idx3 index, it is now possible for SQLite
to find the price of California oranges using only 2 binary searches:



SELECT price FROM fruitsforsale WHERE fruit='Orange' AND state='CA'




Figure 11: Lookup Using A Two-Column Index




With the Idx3 index on both columns that are constrained by the WHERE clause,
SQLite can do a single binary search against Idx3 to find the one rowid
for California oranges, then do a single binary search to find the price
for that item in the original table.  There are no dead-ends and no
wasted binary searches.  This is a more efficient query.



Note that Idx3 contains all the same information as the original 
Idx1.  And so if we have Idx3, we do not really need Idx1
any more.  The "price of peaches" query can be satisfied using Idx3
by simply ignoring the "state" column of Idx3:



SELECT price FROM fruitsforsale WHERE fruit='Peach'




Figure 12: Single-Column Lookup On A Multi-Column Index




Hence, a good rule of thumb is that your database schema should never
contain two indices where one index is a prefix of the other.  Drop the
index with fewer columns.  SQLite will still be able to do efficient
lookups with the longer index.


c2: Query Planning
c3: 1.7. Covering Indexes
c5: queryplanner.html#_covering_indexes
c4:
The "price of California oranges" query was made more efficient through
the use of a two-column index.  But SQLite can do even better with a
three-column index that also includes the "price" column:



CREATE INDEX Idx4 ON FruitsForSale(fruit, state, price);




Figure 13: A Covering Index




This new index contains all the columns of the original FruitsForSale table that
are used by the query - both the search terms and the output.  We call
this a "covering index".  Because all of the information needed is in
the covering index, SQLite never needs to consult the original table
in order to find the price.



SELECT price FROM fruitsforsale WHERE fruit='Orange' AND state='CA';




Figure 14: Query Using A Covering Index




Hence, by adding extra "output" columns onto the end of an index, one
can avoid having to reference the original table and thereby
cut the number of binary searches for a query in half.  This is a
constant-factor improvement in performance (roughly a doubling of
the speed).  But on the other hand, it is also just a refinement;
A two-fold performance increase is not nearly as dramatic as the
one-million-fold increase seen when the table was first indexed.
And for most queries, the difference between 1 microsecond and
2 microseconds is unlikely to be noticed.


c2: Query Planning
c3: 1.8. OR-Connected Terms In The WHERE Clause
c5: queryplanner.html#_or_connected_terms_in_the_where_clause
c4:
Multi-column indices only work if the constraint terms in the WHERE
clause of the query are connected by AND.
So Idx3 and Idx4 are helpful when the search is for items that
are both Oranges and grown in California, but neither index would
be that useful if we wanted all items that were either oranges
or are grown in California.



SELECT price FROM FruitsForSale WHERE fruit='Orange' OR state='CA';




When confronted with OR-connected terms in a WHERE clause, SQLite 
examines each OR term separately and tries to use an index to
find the rowids associated with each term.
It then takes the union of the resulting rowid sets to find
the end result.  The following figure illustrates this process:





Figure 15: Query With OR Constraints




The diagram above implies that SQLite computes all of the rowids first
and then combines them with a union operation before starting to do
rowid lookups on the original table.  In reality, the rowid lookups
are interspersed with rowid computations.  SQLite uses one index at
a time to find rowids while remembering which rowids it has seen
before so as to avoid duplicates.  That is just an implementation
detail, though.  The diagram, while not 100% accurate, provides a good
overview of what is happening.



In order for the OR-by-UNION technique shown above to be useful, there
must be an index available that helps resolve every OR-connected term
in the WHERE clause.  If even a single OR-connected term is not indexed,
then a full table scan would have to be done in order to find the rowids
generated by the one term, and if SQLite has to do a full table scan, it
might as well do it on the original table and get all of the results in
a single pass without having to mess with union operations and follow-on
binary searches.



One can see how the OR-by-UNION technique could also be leveraged to
use multiple indices on queries where the WHERE clause has terms connected
by AND, by using an intersect operator in place of union.  Many SQL
database engines will do just that.  But the performance gain over using
just a single index is slight and so SQLite does not implement that technique
at this time.  However, a future version SQLite might be enhanced to support
AND-by-INTERSECT.


c2: Query Planning
c3: 2. Sorting
c5: queryplanner.html#_sorting
c4:
SQLite (like all other SQL database engines) can also use indices to
satisfy the ORDER BY clauses in a query, in addition to expediting
lookup.  In other words, indices can be used to speed up sorting as
well as searching.



When no appropriate indices are available, a query with an ORDER BY
clause must be sorted as a separate step.  Consider this query:



SELECT * FROM fruitsforsale ORDER BY fruit;




SQLite processes this by gathering all the output of query and then
running that output through a sorter.





Figure 16: Sorting Without An Index




If the number of output rows is K, then the time needed to sort is
proportional to KlogK.  If K is small, the sorting time is usually
not a factor, but in a query such as the above where K==N, the time
needed to sort can be much greater than the time needed to do a
full table scan.  Furthermore, the entire output is accumulated in
temporary storage (which might be either in main memory or on disk,
depending on various compile-time and run-time settings)
which can mean that a lot of temporary storage is required to complete
the query.


c2: Query Planning
c3: 2.1. Sorting By Rowid
c5: queryplanner.html#_sorting_by_rowid
c4:
Because sorting can be expensive, SQLite works hard to convert ORDER BY
clauses into no-ops.  If SQLite determines that output will
naturally appear in the order specified, then no sorting is done.
So, for example, if you request the output in rowid order, no sorting
will be done:



SELECT * FROM fruitsforsale ORDER BY rowid;




Figure 17: Sorting By Rowid




You can also request a reverse-order sort like this:



SELECT * FROM fruitsforsale ORDER BY rowid DESC;




SQLite will still omit the sorting step.  But in order for output to
appear in the correct order, SQLite will do the table scan starting at
the end and working toward the beginning, rather than starting at the
beginning and working toward the end as shown in 
figure 17.


c2: Query Planning
c3: 2.2. Sorting By Index
c5: queryplanner.html#_sorting_by_index
c4:
Of course, ordering the output of a query by rowid is seldom useful.
Usually one wants to order the output by some other column.



If an index is available on the ORDER BY column, that index can be used
for sorting.  Consider the request for all items sorted by "fruit":



SELECT * FROM fruitsforsale ORDER BY fruit;






Figure 18: Sorting With An Index




The Idx1 index is scanned from top to bottom (or from bottom to top if
"ORDER BY fruit DESC" is used) in order to find the rowids for each item
in order by fruit.  Then for each rowid, a binary search is done to lookup
and output that row.  In this way, the output appears in the requested order
without the need to gather the entire output and sort it using a separate step.



But does this really save time?  The number of steps in the 
original indexless sort is proportional to NlogN since
that is how much time it takes to sort N rows.  But when we use Idx1 as
shown here, we have to do N rowid lookups which take logN time each, so
the total time of NlogN is the same!



SQLite uses a cost-based query planner.  When there are two or more ways
of solving the same query, SQLite tries to estimate the total amount of
time needed to run the query using each plan, and then uses the plan with
the lowest estimated cost.  A cost is computed mostly from the estimated
time, and so this case could go either way depending on the table size and
what WHERE clause constraints were available, and so forth.  But generally
speaking, the indexed sort would probably be chosen, if for no other
reason, because it does not need to accumulate the entire result set in
temporary storage before sorting and thus uses much less temporary storage.


c2: Query Planning
c3: 2.3. Sorting By Covering Index
c5: queryplanner.html#_sorting_by_covering_index
c4:
If a covering index can be used for a query, then the multiple rowid lookups
can be avoided and the cost of the query drops dramatically.





Figure 19: Sorting With A Covering Index




With a covering index, SQLite can simply walk the index from one end to the
other and deliver the output in time proportional to N and without having
allocate a large buffer to hold the result set.


c2: Query Planning
c3: 3. Searching And Sorting At The Same Time
c5: queryplanner.html#_searching_and_sorting_at_the_same_time
c4:
The previous discussion has treated searching and sorting as separate
topics.  But in practice, it is often the case that one wants to search
and sort at the same time.  Fortunately, it is possible to do this
using a single index.


c2: Query Planning
c3: 3.1. Searching And Sorting With A Multi-Column Index
c5: queryplanner.html#_searching_and_sorting_with_a_multi_column_index
c4:
Suppose we want to find the prices of all kinds of oranges sorted in
order of the state where they are grown.  The query is this:



SELECT price FROM fruitforsale WHERE fruit='Orange' ORDER BY state




The query contains both a search restriction in the WHERE clause
and a sort order in the ORDER BY clause.  Both the search and the sort
can be accomplished at the same time using the two-column index Idx3.





Figure 20: Search And Sort By Multi-Column Index




The query does a binary search on the index to find the subset of rows
that have fruit='Orange'.  (Because the fruit column is the left-most column
of the index and the rows of the index are in sorted order, all such 
rows will be adjacent.)  Then it scans the matching index rows from top to
bottom to get the rowids for the original table, and for each rowid does
a binary search on the original table to find the price.



You will notice that there is no "sort" box anywhere in the above diagram.
The ORDER BY clause of the query has become a no-op.  No sorting has to be
done here because the output order is by the state column and the state
column also happens to be the first column after the fruit column in the
index.  So, if we scan entries of the index that have the same value for
the fruit column from top to bottom, those index entries are guaranteed to
be ordered by the state column.


c2: Query Planning
c3: 3.2. Searching And Sorting With A Covering Index
c5: queryplanner.html#_searching_and_sorting_with_a_covering_index
c4:
A covering index can also be used to search and sort at the same time.
Consider the following:



SELECT * FROM fruitforsale WHERE fruit='Orange' ORDER BY state




Figure 21: Search And Sort By Covering Index




As before, SQLite does single binary search
for the range of rows in the covering
index that satisfy the WHERE clause, the scans that range from top to 
bottom to get the desired results.  
The rows that satisfy the WHERE clause are guaranteed to be adjacent
since the WHERE clause is an equality constraint on the left-most
column of the index.  And by scanning the matching index rows from
top to bottom, the output is guaranteed to be ordered by state since the
state column is the very next column to the right of the fruit column.
And so the resulting query is very efficient.



SQLite can pull a similar trick for a descending ORDER BY:



SELECT * FROM fruitforsale WHERE fruit='Orange' ORDER BY state DESC




The same basic algorithm is followed, except this time the matching rows
of the index are scanned from bottom to top instead of from top to bottom,
so that the states will appear in descending order.


c2: Query Planning
c3: 3.3. Partial Sorting Using An Index (a.k.a. Block Sorting)
c5: queryplanner.html#_partial_sorting_using_an_index_a_k_a_block_sorting_
c4:
Sometimes only part of an ORDER BY clause can be satisfied using indexes.
Consider, for example, the following query:



SELECT * FROM fruitforsale ORDER BY fruit, price




If the covering index is used for the scan, the "fruit" column will appear
naturally in the correct order, but when there are two or more rows with
the same fruit, the price might be out of order.  When this occurs, SQLite
does many small sorts, one sort for each distinct value of fruit, rather
than one large sort.  Figure 22 below illustrates the concept.





Figure 22: Partial Sort By Index




In the example, instead of a single sort of 7 elements, there
are 5 sorts of one-element each and 1 sort of 2 elements for the
case of fruit=='Orange'.


The advantages of doing many smaller sorts instead of a single large sort
are:

Multiple small sorts collectively use fewer CPU cycles than a single
    large sort.
Each small sort is run independently, meaning that much less information
    needs to be kept in temporary storage at any one time.
Those columns of the ORDER BY that are already in the correct order
    due to indexes can be omitted from the sort key, further reducing
    storage requirements and CPU time.
Output rows can be returned to the application as each small sort
    completes, and well before the table scan is complete.
If a LIMIT clause is present, it might be possible to avoid scanning
    the entire table.


Because of these advantages, SQLite always tries to do a partial sort using an
index even if a complete sort by index is not possible.


c2: Query Planning
c3: 4. WITHOUT ROWID tables
c5: queryplanner.html#_without_rowid_tables
c4:
The basic principals described above apply to both ordinary rowid tables
and WITHOUT ROWID tables.
The only difference is that the rowid column that serves as the key for
tables and that appears as the right-most term in indexes is replaced by
the PRIMARY KEY.

This page last modified on  2022-10-26 13:30:36 UTC


c2: Atomic Commit In SQLite
c3: 1. Introduction
c5: atomiccommit.html#_introduction
c4:
An important feature of transactional databases like SQLite
is "atomic commit".
Atomic commit means that either all database changes within a single
transaction occur or none of them occur.  With atomic commit, it
is as if many different writes to different sections of the database
file occur instantaneously and simultaneously.
Real hardware serializes writes to mass storage, and writing
a single sector takes a finite amount of time.
So it is impossible to truly write many different sectors of a
database file simultaneously and/or instantaneously.
But the atomic commit logic within
SQLite makes it appear as if the changes for a transaction
are all written instantaneously and simultaneously.

SQLite has the important property that transactions appear
to be atomic even if the transaction is interrupted by an
operating system crash or power failure.

This article describes the techniques used by SQLite to create the
illusion of atomic commit.

The information in this article applies only when SQLite is operating
in "rollback mode", or in other words when SQLite is not
using a write-ahead log.  SQLite still supports atomic commit when
write-ahead logging is enabled, but it accomplishes atomic commit by
a different mechanism from the one described in this article.  See
the write-ahead log documentation for additional information on how
SQLite supports atomic commit in that context.


c2: Atomic Commit In SQLite
c3: 2. Hardware Assumptions
c5: atomiccommit.html#_hardware_assumptions
c4:
Throughout this article, we will call the mass storage device "disk"
even though the mass storage device might really be flash memory.

We assume that disk is written in chunks which we call a "sector".
It is not possible to modify any part of the disk smaller than a sector.
To change a part of the disk smaller than a sector, you have to read in
the full sector that contains the part you want to change, make the
change, then write back out the complete sector.

On a traditional spinning disk, a sector is the minimum unit of transfer
in both directions, both reading and writing.  On flash memory, however,
the minimum size of a read is typically much smaller than a minimum write.
SQLite is only concerned with the minimum write amount and so for the
purposes of this article, when we say "sector" we mean the minimum amount
of data that can be written to mass storage in a single go.


  Prior to SQLite version 3.3.14, a sector size of 512 bytes was
  assumed in all cases.  There was a compile-time option to change
  this but the code had never been tested with a larger value.  The
  512 byte sector assumption seemed reasonable since until very recently
  all disk drives used a 512 byte sector internally.  However, there
  has recently been a push to increase the sector size of disks to
  4096 bytes.  Also the sector size
  for flash memory is usually larger than 512 bytes.  For these reasons,
  versions of SQLite beginning with 3.3.14 have a method in the OS
  interface layer that interrogates the underlying filesystem to find
  the true sector size.  As currently implemented (version 3.5.0) this
  method still returns a hard-coded value of 512 bytes, since there
  is no standard way of discovering the true sector size on either
  Unix or Windows.  But the method is available for embedded device
  manufacturers to tweak according to their own needs.  And we have
  left open the possibility of filling in a more meaningful implementation
  on Unix and Windows in the future.

SQLite has traditionally assumed that a sector write is not atomic.
However, SQLite does always assume that a sector write is linear.  By "linear"
we mean that SQLite assumes that when writing a sector, the hardware begins
at one end of the data and writes byte by byte until it gets to
the other end.  The write might go from beginning to end or from
end to beginning.  If a power failure occurs in the middle of a
sector write it might be that part of the sector was modified
and another part was left unchanged.  The key assumption by SQLite
is that if any part of the sector gets changed, then either the
first or the last bytes will be changed.  So the hardware will
never start writing a sector in the middle and work towards the
ends.  We do not know if this assumption is always true but it
seems reasonable.

The previous paragraph states that SQLite does not assume that
sector writes are atomic.  This is true by default.  But as of
SQLite version 3.5.0, there is a new interface called the
Virtual File System (VFS) interface.  The VFS is the only means
by which SQLite communicates to the underlying filesystem.  The
code comes with default VFS implementations for Unix and Windows
and there is a mechanism for creating new custom VFS implementations
at runtime.  In this new VFS interface there is a method called
xDeviceCharacteristics.  This method interrogates the underlying
filesystem to discover various properties and behaviors that the
filesystem may or may not exhibit.  The xDeviceCharacteristics
method might indicate that sector writes are atomic, and if it does
so indicate, SQLite will try to take advantage of that fact.  But
the default xDeviceCharacteristics method for both Unix and Windows
does not indicate atomic sector writes and so these optimizations
are normally omitted.

SQLite assumes that the operating system will buffer writes and
that a write request will return before data has actually been stored
in the mass storage device.
SQLite further assumes that write operations will be reordered by
the operating system.
For this reason, SQLite does a "flush" or "fsync" operation at key
points.  SQLite assumes that the flush or fsync will not return until
all pending write operations for the file that is being flushed have
completed.  We are told that the flush and fsync primitives
are broken on some versions of Windows and Linux.  This is unfortunate.
It opens SQLite up to the possibility of database corruption following
a power loss in the middle of a commit.  However, there is nothing
that SQLite can do to test for or remedy the situation.  SQLite
assumes that the operating system that it is running on works as
advertised.  If that is not quite the case, well then hopefully you
will not lose power too often.

SQLite assumes that when a file grows in length that the new
file space originally contains garbage and then later is filled in
with the data actually written.  In other words, SQLite assumes that
the file size is updated before the file content.  This is a
pessimistic assumption and SQLite has to do some extra work to make
sure that it does not cause database corruption if power is lost
between the time when the file size is increased and when the
new content is written.  The xDeviceCharacteristics method of
the VFS might indicate that the filesystem will always write the
data before updating the file size.  (This is the
SQLITE_IOCAP_SAFE_APPEND property for those readers who are looking
at the code.)  When the xDeviceCharacteristics method indicates
that files content is written before the file size is increased,
SQLite can forego some of its pedantic database protection steps
and thereby decrease the amount of disk I/O needed to perform a
commit.  The current implementation, however, makes no such assumptions
for the default VFSes for Windows and Unix.

SQLite assumes that a file deletion is atomic from the
point of view of a user process.  By this we mean that if SQLite
requests that a file be deleted and the power is lost during the
delete operation, once power is restored either the file will
exist completely with all if its original content unaltered, or
else the file will not be seen in the filesystem at all.  If
after power is restored the file is only partially deleted,
if some of its data has been altered or erased,
or the file has been truncated but not completely removed, then
database corruption will likely result.

SQLite assumes that the detection and/or correction of
bit errors caused by cosmic rays, thermal noise, quantum
fluctuations, device driver bugs, or other mechanisms, is the
responsibility of the underlying hardware and operating system.
SQLite does not add any redundancy to the database file for
the purpose of detecting corruption or I/O errors.
SQLite assumes that the data it reads is exactly the same data
that it previously wrote.

By default, SQLite assumes that an operating system call to write
a range of bytes will not damage or alter any bytes outside of that range
even if a power loss or OS crash occurs during that write.  We
call this the "powersafe overwrite" property.
Prior to version 3.7.9 (2011-11-01),
SQLite did not assume powersafe overwrite.  But with the standard
sector size increasing from 512 to 4096 bytes on most disk drives, it
has become necessary to assume powersafe overwrite in order to maintain
historical performance levels and so powersafe overwrite is assumed by
default in recent versions of SQLite.  The assumption of powersafe
overwrite property can be disabled at compile-time or a run-time if
desired.  See the powersafe overwrite documentation for further
details.


c2: Atomic Commit In SQLite
c3: 3. Single File Commit
c5: atomiccommit.html#_single_file_commit
c4:
We begin with an overview of the steps SQLite takes in order to
perform an atomic commit of a transaction against a single database
file.  The details of file formats used to guard against damage from
power failures and techniques for performing an atomic commit across
multiple databases are discussed in later sections.


c2: Atomic Commit In SQLite
c3: 3.1. Initial State
c5: atomiccommit.html#_initial_state
c4:
The state of the computer when a database connection is
first opened is shown conceptually by the diagram at the
right.
The area of the diagram on the extreme right (labeled "Disk") represents
information stored on the mass storage device.  Each rectangle is
a sector.  The blue color represents that the sectors contain
original data.
The middle area is the operating systems disk cache.  At the
onset of our example, the cache is cold and this is represented
by leaving the rectangles of the disk cache empty.
The left area of the diagram shows the content of memory for
the process that is using SQLite.  The database connection has
just been opened and no information has been read yet, so the
user space is empty.


c2: Atomic Commit In SQLite
c3: 3.2. Acquiring A Read Lock
c5: atomiccommit.html#_acquiring_a_read_lock
c4:
Before SQLite can write to a database, it must first read
the database to see what is there already.  Even if it is just
appending new data, SQLite still has to read in the database
schema from the "sqlite_schema" table so that it can know
how to parse the INSERT statements and discover where in the
database file the new information should be stored.

The first step toward reading from the database file
is obtaining a shared lock on the database file.  A "shared"
lock allows two or more database connections to read from the
database file at the same time.  But a shared lock prevents
another database connection from writing to the database file
while we are reading it.  This is necessary because if another
database connection were writing to the database file at the
same time we are reading from the database file, we might read
some data before the change and other data after the change.
This would make it appear as if the change made by the other
process is not atomic.

Notice that the shared lock is on the operating system
disk cache, not on the disk itself.  File locks
really are just flags within the operating system kernel,
usually.  (The details depend on the specific OS layer
interface.)  Hence, the lock will instantly vanish if the
operating system crashes or if there is a power loss.  It
is usually also the case that the lock will vanish if the
process that created the lock exits.


c2: Atomic Commit In SQLite
c3: 3.3. Reading Information Out Of The Database
c5: atomiccommit.html#_reading_information_out_of_the_database
c4:
After the shared lock is acquired, we can begin reading
information from the database file.  In this scenario, we
are assuming a cold cache, so information must first be
read from mass storage into the operating system cache then
transferred from operating system cache into user space.
On subsequent reads, some or all of the information might
already be found in the operating system cache and so only
the transfer to user space would be required.

Usually only a subset of the pages in the database file
are read.  In this example we are showing three
pages out of eight being read.  In a typical application, a
database will have thousands of pages and a query will normally
only touch a small percentage of those pages.


c2: Atomic Commit In SQLite
c3: 3.4. Obtaining A Reserved Lock
c5: atomiccommit.html#_obtaining_a_reserved_lock
c4:
Before making changes to the database, SQLite first
obtains a "reserved" lock on the database file.  A reserved
lock is similar to a shared lock in that both a reserved lock
and shared lock allow other processes to read from the database
file.  A single reserve lock can coexist with multiple shared
locks from other processes.  However, there can only be a
single reserved lock on the database file.  Hence only a
single process can be attempting to write to the database
at one time.

The idea behind a reserved lock is that it signals that
a process intends to modify the database file in the near
future but has not yet started to make the modifications.
And because the modifications have not yet started, other
processes can continue to read from the database.  However,
no other process should also begin trying to write to the
database.


c2: Atomic Commit In SQLite
c3: 3.5. Creating A Rollback Journal File
c5: atomiccommit.html#_creating_a_rollback_journal_file
c4:
Prior to making any changes to the database file, SQLite first
creates a separate rollback journal file and writes into the
rollback journal the original
content of the database pages that are to be altered.
The idea behind the rollback journal is that it contains
all information needed to restore the database back to
its original state.

The rollback journal contains a small header (shown in green
in the diagram) that records the original size of the database
file.  So if a change causes the database file to grow, we
will still know the original size of the database.  The page
number is stored together with each database page that is
written into the rollback journal.


  When a new file is created, most desktop operating systems
  (Windows, Linux, Mac OS X) will not actually write anything to
  disk.  The new file is created in the operating systems disk
  cache only.  The file is not created on mass storage until sometime
  later, when the operating system has a spare moment.  This creates
  the impression to users that I/O is happening much faster than
  is possible when doing real disk I/O.  We illustrate this idea in
  the diagram to the right by showing that the new rollback journal
  appears in the operating system disk cache only and not on the
  disk itself.


c2: Atomic Commit In SQLite
c3: 3.6. Changing Database Pages In User Space
c5: atomiccommit.html#_changing_database_pages_in_user_space
c4:
After the original page content has been saved in the rollback
journal, the pages can be modified in user memory.  Each database
connection has its own private copy of user space, so the changes
that are made in user space are only visible to the database connection
that is making the changes.  Other database connections still see
the information in operating system disk cache buffers which have
not yet been changed.  And so even though one process is busy
modifying the database, other processes can continue to read their
own copies of the original database content.


c2: Atomic Commit In SQLite
c3: 3.7. Flushing The Rollback Journal File To Mass Storage
c5: atomiccommit.html#_flushing_the_rollback_journal_file_to_mass_storage
c4:
The next step is to flush the content of the rollback journal
file to nonvolatile storage.
As we will see later,
this is a critical step in insuring that the database can survive
an unexpected power loss.
This step also takes a lot of time, since writing to nonvolatile
storage is normally a slow operation.

This step is usually more complicated than simply flushing
the rollback journal to the disk.  On most platforms two separate
flush (or fsync()) operations are required.  The first flush writes
out the base rollback journal content.  Then the header of the
rollback journal is modified to show the number of pages in the
rollback journal.  Then the header is flushed to disk.  The details
on why we do this header modification and extra flush are provided
in a later section of this paper.


c2: Atomic Commit In SQLite
c3: 3.8. Obtaining An Exclusive Lock
c5: atomiccommit.html#_obtaining_an_exclusive_lock
c4:
Prior to making changes to the database file itself, we must
obtain an exclusive lock on the database file.  Obtaining an
exclusive lock is really a two-step process.  First SQLite obtains
a "pending" lock.  Then it escalates the pending lock to an
exclusive lock.

A pending lock allows other processes that already have a
shared lock to continue reading the database file.  But it
prevents new shared locks from being established.  The idea
behind a pending lock is to prevent writer starvation caused
by a large pool of readers.  There might be dozens, even hundreds,
of other processes trying to read the database file.  Each process
acquires a shared lock before it starts reading, reads what it
needs, then releases the shared lock.  If, however, there are
many different processes all reading from the same database, it
might happen that a new process always acquires its shared lock before
the previous process releases its shared lock.  And so there is
never an instant when there are no shared locks on the database
file and hence there is never an opportunity for the writer to
seize the exclusive lock.  A pending lock is designed to prevent
that cycle by allowing existing shared locks to proceed but
blocking new shared locks from being established.  Eventually
all shared locks will clear and the pending lock will then be
able to escalate into an exclusive lock.


c2: Atomic Commit In SQLite
c3: 3.9. Writing Changes To The Database File
c5: atomiccommit.html#_writing_changes_to_the_database_file
c4:
Once an exclusive lock is held, we know that no other
processes are reading from the database file and it is
safe to write changes into the database file.  Usually
those changes only go as far as the operating systems disk
cache and do not make it all the way to mass storage.


c2: Atomic Commit In SQLite
c3: 3.10. 0 Flushing Changes To Mass Storage
c5: atomiccommit.html#0_flushing_changes_to_mass_storage
c4:
Another flush must occur to make sure that all the
database changes are written into nonvolatile storage.
This is a critical step to ensure that the database will
survive a power loss without damage.  However, because
of the inherent slowness of writing to disk or flash memory,
this step together with the rollback journal file flush in section
3.7 above takes up most of the time required to complete a
transaction commit in SQLite.


c2: Atomic Commit In SQLite
c3: 3.11. 1 Deleting The Rollback Journal
c5: atomiccommit.html#1_deleting_the_rollback_journal
c4:
After the database changes are all safely on the mass
storage device, the rollback journal file is deleted.
This is the instant where the transaction commits.
If a power failure or system crash occurs prior to this
point, then recovery processes to be described later make
it appear as if no changes were ever made to the database
file.  If a power failure or system crash occurs after
the rollback journal is deleted, then it appears as if
all changes have been written to disk.  Thus, SQLite gives
the appearance of having made no changes to the database
file or having made the complete set of changes to the
database file depending on whether or not the rollback
journal file exists.

Deleting a file is not really an atomic operation, but
it appears to be from the point of view of a user process.
A process is always able to ask the operating system "does
this file exist?" and the process will get back a yes or no
answer.  After a power failure that occurs during a
transaction commit, SQLite will ask the operating system
whether or not the rollback journal file exists.  If the
answer is "yes" then the transaction is incomplete and is
rolled back.  If the answer is "no" then it means the transaction
did commit.

The existence of a transaction depends on whether or
not the rollback journal file exists and the deletion
of a file appears to be an atomic operation from the point of
view of a user-space process.  Therefore,
a transaction appears to be an atomic operation.

The act of deleting a file is expensive on many systems.
As an optimization, SQLite can be configured to truncate
the journal file to zero bytes in length
or overwrite the journal file header with zeros.  In either
case, the resulting journal file is no longer capable of rolling
back and so the transaction still commits.  Truncating a file
to zero length, like deleting a file, is assumed to be an atomic
operation from the point of view of a user process.  Overwriting
the header of the journal with zeros is not atomic, but if any
part of the header is malformed the journal will not roll back.
Hence, one can say that the commit occurs as soon as the header
is sufficiently changed to make it invalid.  Typically this happens
as soon as the first byte of the header is zeroed.


c2: Atomic Commit In SQLite
c3: 3.12. 2 Releasing The Lock
c5: atomiccommit.html#2_releasing_the_lock
c4:
The last step in the commit process is to release the
exclusive lock so that other processes can once again
start accessing the database file.

In the diagram at the right, we show that the information
that was held in user space is cleared when the lock is released.
This used to be literally true for older versions of SQLite.  But
more recent versions of SQLite keep the user space information
in memory in case it might be needed again at the start of the
next transaction.  It is cheaper to reuse information that is
already in local memory than to transfer the information back
from the operating system disk cache or to read it off of the
disk drive again.  Prior to reusing the information in user space,
we must first reacquire the shared lock and then we have to check
to make sure that no other process modified the database file while
we were not holding a lock.  There is a counter in the first page
of the database that is incremented every time the database file
is modified.  We can find out if another process has modified the
database by checking that counter.  If the database was modified,
then the user space cache must be cleared and reread.  But it is
commonly the case that no changes have been made and the user
space cache can be reused for a significant performance savings.


c2: Atomic Commit In SQLite
c3: 4. Rollback
c5: atomiccommit.html#_rollback
c4:
An atomic commit is supposed to happen instantaneously.  But the processing
described above clearly takes a finite amount of time.
Suppose the power to the computer were cut
part way through the commit operation described above.  In order
to maintain the illusion that the changes were instantaneous, we
have to "rollback" any partial changes and restore the database to
the state it was in prior to the beginning of the transaction.


c2: Atomic Commit In SQLite
c3: 4.1. When Something Goes Wrong...
c5: atomiccommit.html#_when_something_goes_wrong_
c4:
Suppose the power loss occurred
during step 3.10 above,
while the database changes were being written to disk.
After power is restored, the situation might be something
like what is shown to the right.  We were trying to change
three pages of the database file but only one page was
successfully written.  Another page was partially written
and a third page was not written at all.

The rollback journal is complete and intact on disk when
the power is restored.  This is a key point.  The reason for
the flush operation in step 3.7
is to make absolutely sure that
all of the rollback journal is safely on nonvolatile storage
prior to making any changes to the database file itself.


c2: Atomic Commit In SQLite
c3: 4.2. Hot Rollback Journals
c5: atomiccommit.html#_hot_rollback_journals
c4:
The first time that any SQLite process attempts to access
the database file, it obtains a shared lock as described in
section 3.2 above.
But then it notices that there is a
rollback journal file present.  SQLite then checks to see if
the rollback journal is a "hot journal".   A hot journal is
a rollback journal that needs to be played back in order to
restore the database to a sane state.  A hot journal only
exists when an earlier process was in the middle of committing
a transaction when it crashed or lost power.

A rollback journal is a "hot" journal if all of the following
are true:


The rollback journal exists.
The rollback journal is not an empty file.
There is no reserved lock on the main database file.
The header of the rollback journal is well-formed and in particular
    has not been zeroed out.
The rollback journal does not
contain the name of a super-journal file (see
section 5.5 below) or if does
contain the name of a super-journal, then that super-journal
file exists.


The presence of a hot journal is our indication
that a previous process was trying to commit a transaction but
it aborted for some reason prior to the completion of the
commit.  A hot journal means that
the database file is in an inconsistent state and needs to
be repaired (by rollback) prior to being used.


c2: Atomic Commit In SQLite
c3: 4.3. Obtaining An Exclusive Lock On The Database
c5: atomiccommit.html#_obtaining_an_exclusive_lock_on_the_database
c4:
The first step toward dealing with a hot journal is to
obtain an exclusive lock on the database file.  This prevents two
or more processes from trying to rollback the same hot journal
at the same time.


c2: Atomic Commit In SQLite
c3: 4.4. Rolling Back Incomplete Changes
c5: atomiccommit.html#_rolling_back_incomplete_changes
c4:
Once a process obtains an exclusive lock, it is permitted
to write to the database file.  It then proceeds to read the
original content of pages out of the rollback journal and write
that content back to where it came from in the database file.
Recall that the header of the rollback journal records the original
size of the database file prior to the start of the aborted
transaction.  SQLite uses this information to truncate the
database file back to its original size in cases where the
incomplete transaction caused the database to grow.  At the
end of this step, the database should be the same size and
contain the same information as it did before the start of
the aborted transaction.


c2: Atomic Commit In SQLite
c3: 4.5. Deleting The Hot Journal
c5: atomiccommit.html#_deleting_the_hot_journal
c4:
After all information in the rollback journal has been
played back into the database file (and flushed to disk in case
we encounter yet another power failure), the hot rollback journal
can be deleted.

As in section 3.11, the journal
file might be truncated to zero length or its header might
be overwritten with zeros as an optimization on systems where
deleting a file is expensive.  Either way, the journal is no
longer hot after this step.


c2: Atomic Commit In SQLite
c3: 4.6. Continue As If The Uncompleted Writes Had Never Happened
c5: atomiccommit.html#_continue_as_if_the_uncompleted_writes_had_never_happened
c4:
The final recovery step is to reduce the exclusive lock back
to a shared lock.  Once this happens, the database is back in the
state that it would have been if the aborted transaction had never
started.  Since all of this recovery activity happens completely
automatically and transparently, it appears to the program using
SQLite as if the aborted transaction had never begun.


c2: Atomic Commit In SQLite
c3: 5. Multi-file Commit
c5: atomiccommit.html#_multi_file_commit
c4:
SQLite allows a single
database connection to talk to
two or more database files simultaneously through the use of
the ATTACH DATABASE command.
When multiple database files are modified within a single
transaction, all files are updated atomically.
In other words, either all of the database files are updated or
else none of them are.
Achieving an atomic commit across multiple database files is
more complex that doing so for a single file.  This section
describes how SQLite works that bit of magic.


c2: Atomic Commit In SQLite
c3: 5.1. Separate Rollback Journals For Each Database
c5: atomiccommit.html#_separate_rollback_journals_for_each_database
c4:
When multiple database files are involved in a transaction,
each database has its own rollback journal and each database
is locked separately.  The diagram at the right shows a scenario
where three different database files have been modified within
one transaction.  The situation at this step is analogous to
the single-file transaction scenario at
step 3.6.  Each database file has
a reserved lock.  For each database, the original content of pages
that are being changed have been written into the rollback journal
for that database, but the content of the journals have not yet
been flushed to disk.  No changes have been made to the database
file itself yet, though presumably there are changes being held
in user memory.

For brevity, the diagrams in this section are simplified from
those that came before.  Blue color still signifies original content
and pink still signifies new content.  But the individual pages
in the rollback journal and the database file are not shown and
we are not making the distinction between information in the
operating system cache and information that is on disk.  All of
these factors still apply in a multi-file commit scenario.  They
just take up a lot of space in the diagrams and they do not add
any new information, so they are omitted here.


c2: Atomic Commit In SQLite
c3: 5.2. The Super-Journal File
c5: atomiccommit.html#_the_super_journal_file
c4:
The next step in a multi-file commit is the creation of a
"super-journal" file.  The name of the super-journal file is
the same name as the original database filename (the database
that was opened using the
sqlite3_open() interface,
not one of the ATTACHed auxiliary
databases) with the text "-mjHHHHHHHH" appended where
HHHHHHHH is a random 32-bit hexadecimal number.  The
random HHHHHHHH suffix changes for every new super-journal.

(Nota bene: The formula for computing the super-journal filename
given in the previous paragraph corresponds to the implementation as
of SQLite version 3.5.0.  But this formula is not part of the SQLite
specification and is subject to change in future releases.)

Unlike the rollback journals, the super-journal does not contain
any original database page content.  Instead, the super-journal contains
the full pathnames for rollback journals for every database that is
participating in the transaction.

After the super-journal is constructed, its content is flushed
to disk before any further actions are taken.  On Unix, the directory
that contains the super-journal is also synced in order to make sure
the super-journal file will appear in the directory following a
power failure.

The purpose of the super-journal is to ensure that multi-file
transactions are atomic across a power-loss.  But if the database files
have other settings that compromise integrity across a power-loss event
(such as PRAGMA synchronous=OFF or PRAGMA journal_mode=MEMORY) then
the creation of the super-journal is omitted, as an optimization.


c2: Atomic Commit In SQLite
c3: 5.3. Updating Rollback Journal Headers
c5: atomiccommit.html#_updating_rollback_journal_headers
c4:
The next step is to record the full pathname of the super-journal file
in the header of every rollback journal.  Space to hold the
super-journal filename was reserved at the beginning of each rollback journal
as the rollback journals were created.

The content of each rollback journal is flushed to disk both before
and after the super-journal filename is written into the rollback
journal header.  It is important to do both of these flushes.  Fortunately,
the second flush is usually inexpensive since typically only a single
page of the journal file (the first page) has changed.

This step is analogous to
step 3.7 in the single-file commit
scenario described above.


c2: Atomic Commit In SQLite
c3: 5.4. Updating The Database Files
c5: atomiccommit.html#_updating_the_database_files
c4:
Once all rollback journal files have been flushed to disk, it
is safe to begin updating database files.  We have to obtain an
exclusive lock on all database files before writing the changes.
After all the changes are written, it is important to flush the
changes to disk so that they will be preserved in the event of
a power failure or operating system crash.

This step corresponds to steps
3.8,
3.9, and
3.10 in the single-file commit
scenario described previously.


c2: Atomic Commit In SQLite
c3: 5.5. Delete The Super-Journal File
c5: atomiccommit.html#_delete_the_super_journal_file
c4:
The next step is to delete the super-journal file.
This is the point where the multi-file transaction commits.
This step corresponds to
step 3.11 in the single-file
commit scenario where the rollback journal is deleted.

If a power failure or operating system crash occurs at this
point, the transaction will not rollback when the system reboots
even though there are rollback journals present.  The
difference is the super-journal pathname in the header of the
rollback journal.  Upon restart, SQLite only considers a journal
to be hot and will only playback the journal if there is no
super-journal filename in the header (which is the case for
a single-file commit) or if the super-journal file still
exists on disk.


c2: Atomic Commit In SQLite
c3: 5.6. Clean Up The Rollback Journals
c5: atomiccommit.html#_clean_up_the_rollback_journals
c4:
The final step in a multi-file commit is to delete the
individual rollback journals and drop the exclusive locks on
the database files so that other processes can see the changes.
This corresponds to
step 3.12 in the single-file
commit sequence.

The transaction has already committed at this point so timing
is not critical in the deletion of the rollback journals.
The current implementation deletes a single rollback journal
then unlocks the corresponding database file before proceeding
to the next rollback journal.  But in the future we might change
this so that all rollback journals are deleted before any database
files are unlocked.  As long as the rollback journal is deleted before
its corresponding database file is unlocked it does not matter in what
order the rollback journals are deleted or the database files are
unlocked.


c2: Atomic Commit In SQLite
c3: 6. Additional Details Of The Commit Process
c5: atomiccommit.html#_additional_details_of_the_commit_process
c4:
Section 3.0 above provides an overview of
how atomic commit works in SQLite.  But it glosses over a number of
important details.  The following subsections will attempt to fill
in the gaps.


c2: Atomic Commit In SQLite
c3: 6.1. Always Journal Complete Sectors
c5: atomiccommit.html#_always_journal_complete_sectors
c4:
When the original content of a database page is written into
the rollback journal (as shown in section 3.5),
SQLite always writes a complete sector of data, even if the
page size of the database is smaller than the sector size.
Historically, the sector size in SQLite has been hard coded to 512
bytes and since the minimum page size is also 512 bytes, this has never
been an issue.  But beginning with SQLite version 3.3.14, it is possible
for SQLite to use mass storage devices with a sector size larger than 512
bytes.  So, beginning with version 3.3.14, whenever any page within a
sector is written into the journal file, all pages in that same sector
are stored with it.

It is important to store all pages of a sector in the rollback
journal in order to prevent database corruption following a power
loss while writing the sector.  Suppose that pages 1, 2, 3, and 4 are
all stored in sector 1 and that page 2 is modified.  In order to write
the changes to page 2, the underlying hardware must also rewrite the
content of pages 1, 3, and 4 since the hardware must write the complete
sector.  If this write operation is interrupted by a power outage,
one or more of the pages 1, 3, or 4 might be left with incorrect data.
Hence, to avoid lasting corruption to the database, the original content
of all of those pages must be contained in the rollback journal.


c2: Atomic Commit In SQLite
c3: 6.2. Dealing With Garbage Written Into Journal Files
c5: atomiccommit.html#_dealing_with_garbage_written_into_journal_files
c4:
When data is appended to the end of the rollback journal,
SQLite normally makes the pessimistic assumption that the file
is first extended with invalid "garbage" data and that afterwards
the correct data replaces the garbage.  In other words, SQLite assumes
that the file size is increased first and then afterwards the content
is written into the file.  If a power failure occurs after the file
size has been increased but before the file content has been written,
the rollback journal can be left containing garbage data.  If after
power is restored, another SQLite process sees the rollback journal
containing the garbage data and tries to roll it back into the original
database file, it might copy some of the garbage into the database file
and thus corrupt the database file.

SQLite uses two defenses against this problem.  In the first place,
SQLite records the number of pages in the rollback journal in the header
of the rollback journal.  This number is initially zero.  So during an
attempt to rollback an incomplete (and possibly corrupt) rollback
journal, the process doing the rollback will see that the journal
contains zero pages and will thus make no changes to the database.  Prior
to a commit, the rollback journal is flushed to disk to ensure that
all content has been synced to disk and there is no "garbage" left
in the file, and only then is the page count in the header changed from
zero to true number of pages in the rollback journal.  The rollback journal
header is always kept in a separate sector from any page data so that
it can be overwritten and flushed without risking damage to a data
page if a power outage occurs.  Notice that the rollback journal
is flushed to disk twice: once to write the page content and a second
time to write the page count in the header.

The previous paragraph describes what happens when the
synchronous pragma setting is "full".


PRAGMA synchronous=FULL;


The default synchronous setting is full so the above is what usually
happens.  However, if the synchronous setting is lowered to "normal",
SQLite only flushes the rollback journal once, after the page count has
been written.
This carries a risk of corruption because it might happen that the
modified (non-zero) page count reaches the disk surface before all
of the data does.  The data will have been written first, but SQLite
assumes that the underlying filesystem can reorder write requests and
that the page count can be burned into oxide first even though its
write request occurred last.  So as a second line of defense, SQLite
also uses a 32-bit checksum on every page of data in the rollback
journal.  This checksum is evaluated for each page during rollback
while rolling back a journal as described in
section 4.4.  If an incorrect checksum
is seen, the rollback is abandoned.  Note that the checksum does
not guarantee that the page data is correct since there is a small
but finite probability that the checksum might be right even if the data is
corrupt.  But the checksum does at least make such an error unlikely.


Note that the checksums in the rollback journal are not necessary
if the synchronous setting is FULL.  We only depend on the checksums
when synchronous is lowered to NORMAL.  Nevertheless, the checksums
never hurt and so they are included in the rollback journal regardless
of the synchronous setting.


c2: Atomic Commit In SQLite
c3: 6.3. Cache Spill Prior To Commit
c5: atomiccommit.html#_cache_spill_prior_to_commit
c4:
The commit process shown in section 3.0
assumes that all database changes fit in memory until it is time to
commit.  This is the common case.  But sometimes a larger change will
overflow the user-space cache prior to transaction commit.  In those
cases, the cache must spill to the database before the transaction
is complete.

At the beginning of a cache spill, the status of the database
connection is as shown in step 3.6.
Original page content has been saved in the rollback journal and
modifications of the pages exist in user memory.  To spill the cache,
SQLite executes steps 3.7 through
3.9.  In other words, the rollback journal
is flushed to disk, an exclusive lock is acquired, and changes are
written into the database.  But the remaining steps are deferred
until the transaction really commits.  A new journal header is
appended to the end of the rollback journal (in its own sector)
and the exclusive database lock is retained, but otherwise processing
returns to step 3.6.  When the transaction
commits, or if another cache spill occurs, steps
3.7 and 3.9 are
repeated.  (Step 3.8 is omitted on second
and subsequent passes since an exclusive database lock is already held
due to the first pass.)

A cache spill causes the lock on the database file to
escalate from reserved to exclusive.  This reduces concurrency.
A cache spill also causes extra disk flush or fsync operations to
occur and these operations are slow, hence a cache spill can
seriously reduce performance.
For these reasons a cache spill is avoided whenever possible.


c2: Atomic Commit In SQLite
c3: 7. Optimizations
c5: atomiccommit.html#_optimizations
c4:
Profiling indicates that for most systems and in most circumstances
SQLite spends most of its time doing disk I/O.  It follows then that
anything we can do to reduce the amount of disk I/O will likely have a
large positive impact on the performance of SQLite.  This section
describes some of the techniques used by SQLite to try to reduce the
amount of disk I/O to a minimum while still preserving atomic commit.


c2: Atomic Commit In SQLite
c3: 7.1. Cache Retained Between Transactions
c5: atomiccommit.html#_cache_retained_between_transactions
c4:
Step 3.12 of the commit process shows
that once the shared lock has been released, all user-space cache
images of database content must be discarded.  This is done because
without a shared lock, other processes are free to modify the database
file content and so any user-space image of that content might become
obsolete.  Consequently, each new transaction would begin by rereading
data which had previously been read.  This is not as bad as it sounds
at first since the data being read is still likely in the operating
systems file cache.  So the "read" is really just a copy of data
from kernel space into user space.  But even so, it still takes time.

Beginning with SQLite version 3.3.14 a mechanism has been added
to try to reduce the needless rereading of data.  In newer versions
of SQLite, the data in the user-space pager cache is retained when
the lock on the database file is released.  Later, after the
shared lock is acquired at the beginning of the next transaction,
SQLite checks to see if any other process has modified the database
file.  If the database has been changed in any way since the lock
was last released, the user-space cache is erased at that point.
But commonly the database file is unchanged and the user-space cache
can be retained, and some unnecessary read operations can be avoided.

In order to determine whether or not the database file has changed,
SQLite uses a counter in the database header (in bytes 24 through 27)
which is incremented during every change operation.  SQLite saves a copy
of this counter prior to releasing its database lock.  Then after
acquiring the next database lock it compares the saved counter value
against the current counter value and erases the cache if the values
are different, or reuses the cache if they are the same.


c2: Atomic Commit In SQLite
c3: 7.2. Exclusive Access Mode
c5: atomiccommit.html#_exclusive_access_mode
c4:
SQLite version 3.3.14 adds the concept of "Exclusive Access Mode".
In exclusive access mode, SQLite retains the exclusive
database lock at the conclusion of each transaction.  This prevents
other processes from accessing the database, but in many deployments
only a single process is using a database so this is not a
serious problem.  The advantage of exclusive access mode is that
disk I/O can be reduced in three ways:


It is not necessary to increment the change counter in the
database header for transactions after the first transaction.  This
will often save a write of page one to both the rollback
journal and the main database file.

No other processes can change the database so there is never
a need to check the change counter and clear the user-space cache
at the beginning of a transaction.

Each transaction can be committed by overwriting the rollback
journal header with zeros rather than deleting the journal file.
This avoids having to modify the directory entry for the journal file
and it avoids having to deallocate disk sectors associated with the
journal.  Furthermore, the next transaction will overwrite existing
journal file content rather than append new content and on most systems
overwriting is much faster than appending.


The third optimization, zeroing the journal file header rather than
deleting the rollback journal file,
does not depend on holding an exclusive lock at all times.
This optimization can be set independently of exclusive lock mode
using the journal_mode pragma
as described in section 7.6 below.


c2: Atomic Commit In SQLite
c3: 7.3. Do Not Journal Freelist Pages
c5: atomiccommit.html#_do_not_journal_freelist_pages
c4:
When information is deleted from an SQLite database, the pages used
to hold the deleted information are added to a "freelist".  Subsequent
inserts will draw pages off of this freelist rather than expanding the
database file.

Some freelist pages contain critical data; specifically the locations
of other freelist pages.  But most freelist pages contain nothing useful.
These latter freelist pages are called "leaf" pages.  We are free to
modify the content of a leaf freelist page in the database without
changing the meaning of the database in any way.

Because the content of leaf freelist pages is unimportant, SQLite
avoids storing leaf freelist page content in the rollback journal
in step 3.5 of the commit process.
If a leaf freelist page is changed and that change does not get rolled back
during a transaction recovery, the database is not harmed by the omission.
Similarly, the content of a new freelist page is never written back
into the database at step 3.9 nor
read from the database at step 3.3.
These optimizations can greatly reduce the amount of I/O that occurs
when making changes to a database file that contains free space.


c2: Atomic Commit In SQLite
c3: 7.4. Single Page Updates And Atomic Sector Writes
c5: atomiccommit.html#_single_page_updates_and_atomic_sector_writes
c4:
Beginning in SQLite version 3.5.0, the new Virtual File System (VFS)
interface contains a method named xDeviceCharacteristics which reports
on special properties that the underlying mass storage device
might have.  Among the special properties that
xDeviceCharacteristics might report is the ability of to do an
atomic sector write.

Recall that by default SQLite assumes that sector writes are
linear but not atomic.  A linear write starts at one end of the
sector and changes information byte by byte until it gets to the
other end of the sector.  If a power loss occurs in the middle of
a linear write then part of the sector might be modified while the
other end is unchanged.  In an atomic sector write, either the entire
sector is overwritten or else nothing in the sector is changed.

We believe that most modern disk drives implement atomic sector
writes.  When power is lost, the drive uses energy stored in capacitors
and/or the angular momentum of the disk platter to provide power to
complete any operation in progress.  Nevertheless, there are so many
layers in between the write system call and the on-board disk drive
electronics that we take the safe approach in both Unix and w32 VFS
implementations and assume that sector writes are not atomic.  On the
other hand, device
manufacturers with more control over their filesystems might want
to consider enabling the atomic write property of xDeviceCharacteristics
if their hardware really does do atomic writes.

When sector writes are atomic and the page size of a database is
the same as a sector size, and when there is a database change that
only touches a single database page, then SQLite skips the whole
journaling and syncing process and simply writes the modified page
directly into the database file.  The change counter in the first
page of the database file is modified separately since no harm is
done if power is lost before the change counter can be updated.


c2: Atomic Commit In SQLite
c3: 7.5. Filesystems With Safe Append Semantics
c5: atomiccommit.html#_filesystems_with_safe_append_semantics
c4:
Another optimization introduced in SQLite version 3.5.0 makes
use of "safe append" behavior of the underlying disk.
Recall that SQLite assumes that when data is appended to a file
(specifically to the rollback journal) that the size of the file
is increased first and that the content is written second.  So
if power is lost after the file size is increased but before the
content is written, the file is left containing invalid "garbage"
data.  The xDeviceCharacteristics method of the VFS might, however,
indicate that the filesystem implements "safe append" semantics.
This means that the content is written before the file size is
increased so that it is impossible for garbage to be introduced
into the rollback journal by a power loss or system crash.

When safe append semantics are indicated for a filesystem,
SQLite always stores the special value of -1 for the page count
in the header of the rollback journal.  The -1 page count value
tells any process attempting to rollback the journal that the
number of pages in the journal should be computed from the journal
size.  This -1 value is never changed.  So that when a commit
occurs, we save a single flush operation and a sector write of
the first page of the journal file.  Furthermore, when a cache
spill occurs we no longer need to append a new journal header
to the end of the journal; we can simply continue appending
new pages to the end of the existing journal.


c2: Atomic Commit In SQLite
c3: 7.6. Persistent Rollback Journals
c5: atomiccommit.html#_persistent_rollback_journals
c4:
Deleting a file is an expensive operation on many systems.
So as an optimization, SQLite can be configured to avoid the
delete operation of section 3.11.
Instead of deleting the journal file in order to commit a transaction,
the file is either truncated to zero bytes in length or its
header is overwritten with zeros.  Truncating the file to zero
length saves having to make modifications to the directory containing
the file since the file is not removed from the directory.
Overwriting the header has the additional savings of not having
to update the length of the file (in the "inode" on many systems)
and not having to deal with newly freed disk sectors.  Furthermore,
at the next transaction the journal will be created by overwriting
existing content rather than appending new content onto the end
of a file, and overwriting is often much faster than appending.

SQLite can be configured to commit transactions by overwriting
the journal header with zeros instead of deleting the journal file
by setting the "PERSIST" journaling mode using the
journal_mode PRAGMA.
For example:


PRAGMA journal_mode=PERSIST;


The use of persistent journal mode provides a noticeable performance
improvement on many systems.  Of course, the drawback is that the
journal files remain on the disk, using disk space and cluttering
directories, long after the transaction commits.  The only safe way
to delete a persistent journal file is to commit a transaction
with journaling mode set to DELETE:


PRAGMA journal_mode=DELETE;
BEGIN EXCLUSIVE;
COMMIT;


Beware of deleting persistent journal files by any other means
since the journal file might be hot, in which case deleting it will
corrupt the corresponding database file.

Beginning in SQLite version 3.6.4 (2008-10-15),
the TRUNCATE journal mode is
also supported:


PRAGMA journal_mode=TRUNCATE;


In truncate journal mode, the transaction is committed by truncating
the journal file to zero length rather than deleting the journal file
(as in DELETE mode) or by zeroing the header (as in PERSIST mode).
TRUNCATE mode shares the advantage of PERSIST mode that the directory
that contains the journal file and database does not need to be updated.
Hence truncating a file is often faster than deleting it.  TRUNCATE has
the additional advantage that it is not followed by a
system call (ex: fsync()) to synchronize the change to disk.  It might
be safer if it did.
But on many modern filesystems, a truncate is an atomic and
synchronous operation and so we think that TRUNCATE will usually be safe
in the face of power failures.  If you are uncertain about whether or
not TRUNCATE will be synchronous and atomic on your filesystem and it is
important to you that your database survive a power loss or operating
system crash that occurs during the truncation operation, then you might
consider using a different journaling mode.

On embedded systems with synchronous filesystems, TRUNCATE results
in slower behavior than PERSIST.  The commit operation is the same speed.
But subsequent transactions are slower following a TRUNCATE because it is
faster to overwrite existing content than to append to the end of a file.
New journal file entries will always be appended following a TRUNCATE but
will usually overwrite with PERSIST.


c2: Atomic Commit In SQLite
c3: 8. Testing Atomic Commit Behavior
c5: atomiccommit.html#_testing_atomic_commit_behavior
c4:
The developers of SQLite are confident that it is robust
in the face of power failures and system crashes because the
automatic test procedures do extensive checks on
the ability of SQLite to recover from simulated power loss.
We call these the "crash tests".

Crash tests in SQLite use a modified VFS that can simulate
the kinds of filesystem damage that occur during a power
loss or operating system crash.  The crash-test VFS can simulate
incomplete sector writes, pages filled with garbage data because
a write has not completed, and out of order writes, all occurring
at varying points during a test scenario.  Crash tests execute
transactions over and over, varying the time at which a simulated
power loss occurs and the properties of the damage inflicted.
Each test then reopens the database after the simulated crash and
verifies that the transaction either occurred completely
or not at all and that the database is in a completely
consistent state.

The crash tests in SQLite have discovered a number of very
subtle bugs (now fixed) in the recovery mechanism.  Some of
these bugs were very obscure and unlikely to have been found
using only code inspection and analysis techniques.  From this
experience, the developers of SQLite feel confident that any other
database system that does not use a similar crash test system
likely contains undetected bugs that will lead to database
corruption following a system crash or power failure.


c2: Atomic Commit In SQLite
c3: 9. Things That Can Go Wrong
c5: atomiccommit.html#_things_that_can_go_wrong
c4:
The atomic commit mechanism in SQLite has proven to be robust,
but it can be circumvented by a sufficiently creative
adversary or a sufficiently broken operating system implementation.
This section describes a few of the ways in which an SQLite database
might be corrupted by a power failure or system crash.
(See also: How To Corrupt Your Database Files.)


c2: Atomic Commit In SQLite
c3: 9.1. Broken Locking Implementations
c5: atomiccommit.html#_broken_locking_implementations
c4:
SQLite uses filesystem locks to make sure that only one
process and database connection is trying to modify the database
at a time.  The filesystem locking mechanism is implemented
in the VFS layer and is different for every operating system.
SQLite depends on this implementation being correct.  If something
goes wrong and two or more processes are able to write the same
database file at the same time, severe damage can result.

We have received reports of implementations of both
Windows network filesystems and NFS in which locking was
subtly broken.  We can not verify these reports, but as
locking is difficult to get right on a network filesystem
we have no reason to doubt them.  You are advised to
avoid using SQLite on a network filesystem in the first place,
since performance will be slow.  But if you must use a
network filesystem to store SQLite database files, consider
using a secondary locking mechanism to prevent simultaneous
writes to the same database even if the native filesystem
locking mechanism malfunctions.

The versions of SQLite that come preinstalled on Apple
Mac OS X computers contain a version of SQLite that has been
extended to use alternative locking strategies that work on
all network filesystems that Apple supports.  These extensions
used by Apple work great as long as all processes are accessing
the database file in the same way.  Unfortunately, the locking
mechanisms do not exclude one another, so if one process is
accessing a file using (for example) AFP locking and another
process (perhaps on a different machine) is using dot-file locks,
the two processes might collide because AFP locks do not exclude
dot-file locks or vice versa.


c2: Atomic Commit In SQLite
c3: 9.2. Incomplete Disk Flushes
c5: atomiccommit.html#_incomplete_disk_flushes
c4:
SQLite uses the fsync() system call on Unix and the FlushFileBuffers()
system call on w32 in order to sync the file system buffers onto disk
oxide as shown in step 3.7 and
step 3.10.  Unfortunately, we have received
reports that neither of these interfaces works as advertised on many
systems.  We hear that FlushFileBuffers() can be completely disabled
using registry settings on some Windows versions.  Some historical
versions of Linux contain versions of fsync() which are no-ops on
some filesystems, we are told.  Even on systems where
FlushFileBuffers() and fsync() are said to be working, often
the IDE disk control lies and says that data has reached oxide
while it is still held only in the volatile control cache.

On the Mac, you can set this pragma:


PRAGMA fullfsync=ON;


Setting fullfsync on a Mac will guarantee that data really does
get pushed out to the disk platter on a flush.  But the implementation
of fullfsync involves resetting the disk controller.  And so not only
is it profoundly slow, it also slows down other unrelated disk I/O.
So its use is not recommended.


c2: Atomic Commit In SQLite
c3: 9.3. Partial File Deletions
c5: atomiccommit.html#_partial_file_deletions
c4:
SQLite assumes that file deletion is an atomic operation from the
point of view of a user process.  If power fails in the middle of
a file deletion, then after power is restored SQLite expects to see
either the entire file with all of its original data intact, or it
expects not to find the file at all.  Transactions may not be atomic
on systems that do not work this way.


c2: Atomic Commit In SQLite
c3: 9.4. Garbage Written Into Files
c5: atomiccommit.html#_garbage_written_into_files
c4:
SQLite database files are ordinary disk files that can be
opened and written by ordinary user processes.  A rogue process
can open an SQLite database and fill it with corrupt data.
Corrupt data might also be introduced into an SQLite database
by bugs in the operating system or disk controller; especially
bugs triggered by a power failure.  There is nothing SQLite can
do to defend against these kinds of problems.


c2: Atomic Commit In SQLite
c3: 9.5. Deleting Or Renaming A Hot Journal
c5: atomiccommit.html#_deleting_or_renaming_a_hot_journal
c4:
If a crash or power loss does occur and a hot journal is left on
the disk, it is essential that the original database file and the hot
journal remain on disk with their original names until the database
file is opened by another SQLite process and rolled back.
During recovery at step 4.2 SQLite locates
the hot journal by looking for a file in the same directory as the
database being opened and whose name is derived from the name of the
file being opened.  If either the original database file or the
hot journal have been moved or renamed, then the hot journal will
not be seen and the database will not be rolled back.

We suspect that a common failure mode for SQLite recovery happens
like this:  A power failure occurs.  After power is restored, a well-meaning
user or system administrator begins looking around on the disk for
damage.  They see their database file named "important.data".  This file
is perhaps familiar to them.  But after the crash, there is also a
hot journal named "important.data-journal".  The user then deletes
the hot journal, thinking that they are helping to cleanup the system.
We know of no way to prevent this other than user education.

If there are multiple (hard or symbolic) links to a database file,
the journal will be created using the name of the link through which
the file was opened.  If a crash occurs and the database is opened again
using a different link, the hot journal will not be located and no
rollback will occur.

Sometimes a power failure will cause a filesystem to be corrupted
such that recently changed filenames are forgotten and the file is
moved into a "/lost+found" directory.  When that happens, the hot
journal will not be found and recovery will not occur.
SQLite tries to prevent this
by opening and syncing the directory containing the rollback journal
at the same time it syncs the journal file itself.  However, the
movement of files into /lost+found can be caused by unrelated processes
creating unrelated files in the same directory as the main database file.
And since this is out from under the control of SQLite, there is nothing
that SQLite can do to prevent it.  If you are running on a system that
is vulnerable to this kind of filesystem namespace corruption (most
modern journalling filesystems are immune, we believe) then you might
want to consider putting each SQLite database file in its own private
subdirectory.


c2: Atomic Commit In SQLite
c3: 10. Future Directions And Conclusion
c5: atomiccommit.html#_future_directions_and_conclusion
c4:
Every now and then someone discovers a new failure mode for
the atomic commit mechanism in SQLite and the developers have to
put in a patch.  This is happening less and less and the
failure modes are becoming more and more obscure.  But it would
still be foolish to suppose that the atomic commit logic of
SQLite is entirely bug-free.  The developers are committed to fixing
these bugs as quickly as they might be found.


The developers are also on the lookout for new ways to
optimize the commit mechanism.  The current VFS implementations
for Unix (Linux and Mac OS X) and Windows make pessimistic assumptions about
the behavior of those systems.  After consultation with experts
on how these systems work, we might be able to relax some of the
assumptions on these systems and allow them to run faster.  In
particular, we suspect that most modern filesystems exhibit the
safe append property and that many of them might support atomic
sector writes.  But until this is known for certain, SQLite will
take the conservative approach and assume the worst.
This page last modified on  2022-12-31 21:51:03 UTC


c2: The generate_series Table-Valued Function
c3: 1. Overview
c5: series.html#overview
c4:
The generate_series(START,STOP,STEP) table-valued function is a
loadable extension included in the SQLite source tree, and compiled into
the command-line shell.  The generate_series() table has one visible
result column named "value" holding integer values
and a number of rows determined by the
parameters START, STOP, and STEP.  The first row of the table has
a value of START.  Subsequent rows increment by STEP to a value
not exceeding STOP.

The generate_series() table has additional, hidden columns
named "start", "stop", and "step" whose values are the effective
values of START, STOP and STEP as provided or defaulted.
It also has a rowid, accessible by its usual names.

Omitted parameters take on default values.  STEP defaults to 1.
STOP defaults to 4294967295.  The START parameter is required
as of version 3.37.0 (2021-11-27) and later and an error will
be raised if START is omitted or has a self-referential or otherwise
uncomputable value.  Older versions used a default of 0 for START.
The legacy behavior can be obtained from recent code by compiling
with -DZERO_ARGUMENT_GENERATE_SERIES.


c2: The generate_series Table-Valued Function
c3: 1.1. Equivalent Recursive Common Table Expression
c5: series.html#equivalent_recursive_common_table_expression
c4:
The generate_series table can be simulated for positive step values
using a recursive common table expression.  If the three parameters
are $start, $end, and $step, then the equivalent common table
expression is:

WITH RECURSIVE generate_series(value) AS (
  SELECT $start
  UNION ALL
  SELECT value+$step FROM generate_series
   WHERE value+$step&lt;=$end
) ...


The common table expression works without having to load an
extension.  On the other hand, the extension is easier to program
and faster.


c2: The generate_series Table-Valued Function
c3: 2. Usage Examples
c5: series.html#usage_examples
c4:
Generate all multiples of 5 less than or equal to 100:

SELECT value FROM generate_series(5,100,5);


Generate the 20 random integer values:

SELECT random() FROM generate_series(1,20);


Find the name of every customer whose account number
   is an even multiple of 100 between 10000 and 20000.

SELECT customer.name
  FROM customer, generate_series(10000,20000,100)
 WHERE customer.id=value;
/* or */
SELECT name FROM customer
 WHERE id IN (SELECT value
                FROM generate_series(10000,20000,200));

This page last modified on  2023-05-01 21:49:55 UTC


c2: Swarmvtab Virtual Table
c3: 1. Overview
c5: swarmvtab.html#overview
c4:
The "swarmvtab" virtual table allows the user to query a large number 
of tables (hereafter "component" tables) with similar schemas but distinct
ranges of rowid values as if they were a single database table. The tables may
be (and usually are) located in different databases. Swarmvtab tables are
read-only.

Component tables must not be declared WITHOUT ROWID, and must all have
the same schema, but may have different names within their databases. In
this context, "the same schema" means that:


  All component tables must have the same set of columns, in the same 
      order.
  The types and default collation sequences attached to each column
      must be the same for all component tables.
  All component tables must have the same PRIMARY KEY declaration (if any).


A swarmvtab table has the same schema as each of its component tables.

A swarmvtab virtual table is created as follows:

CREATE VIRTUAL TABLE temp.&lt;name&gt; USING swarmvtab(&lt;sql-statement&gt;);


Swarmvtab virtual tables must be created in the temp schema. Attempting
to create a swarmvtab in the main or an attached database is an error.

The SQL statement supplied as the argument to the CREATE VIRTUAL TABLE
statement is executed when the table is created. It must return either four
or five columns. Each row returned describes one of the component tables. The
first four columns are interpreted, from first to last, as:


   Database URI. A filename or URI that can be used to open the
  database containing the component table.

   Table name. The name of the component table within its database.

   Minimum rowid. The smallest rowid value that the component
  table may contain.

   Maximum rowid. The smallest rowid value that the component
  table may contain.


The interpretation of the final column, if it is present, is 
described here.

For example, say the SQL statement returns the following data when 
executed:


Database URITable nameMinimum rowidMaximum rowid
test.db1 t1 0 10
test.db2 t2 11 20
test.db3 t1 21 30
test.db4 t1 31 40


and the user queries the swarmvtab table for the row with rowid value
25. The swarmvtab table will open database file "test.db3" and read the
data to return from table "t1" (as 25 falls within the range of rowids
assigned to table "t1" in "test.db3").

Swarmvtab efficiently handles range and equality constraints on the
rowid (or other INTEGER PRIMARY KEY) field only. If a query does not 
contain such a constraint, then swarmvtab finds the results by opening
each database in turn and linearly scanning the component table. Which 
generates a correct result, but is often slow.

There must be no overlapping rowid ranges in the rows returned by
the SQL statement. It is an error if there are.

The swarmvtab implementation may open or close databases at any 
point. By default, it attempts to limit the maximum number of 
simultaneously open database files to nine. This is not a hard limit -
it is possible to construct a scenario that will cause swarmvtab to 
exceed it.


c2: Swarmvtab Virtual Table
c3: 2. Compiling and Using Swarmvtab
c5: swarmvtab.html#compiling_and_using_swarmvtab
c4:
The code for the swarmvtab virtual table is found in the
ext/misc/unionvtab.c file of the main SQLite source tree. It may be compiled
into an SQLite loadable extension using a command like:

gcc -g -fPIC -shared unionvtab.c -o unionvtab.so


Alternatively, the unionvtab.c file may be compiled into the application. 
In this case, the following function should be invoked to register the
extension with each new database connection:

int sqlite3_unionvtab_init(sqlite3 *db, void*, void*);


 The first argument passed should be the database handle to register the
extension with. The second and third arguments should both be passed 0.

 The source file and entry point are named for "unionvtab" instead of
"swarmvtab". Unionvtab is a separately documented virtual table 
that is bundled with swarmvtab.


c2: Swarmvtab Virtual Table
c3: 3. Advanced Usage
c5: swarmvtab.html#advanced_usage
c4:
Most users of swarmvtab will only use the features described above. 
This section describes features designed for more esoteric use cases. These
features all involve specifying extra optional parameters following the SQL
statement as part of the CREATE VIRTUAL TABLE command. An optional parameter 
is specified using its name, followed by an "=" character, followed by an
optionally quoted value. Whitespace may separate the name, "=" character 
and value. For example:

CREATE VIRTUAL TABLE temp.sv USING swarmvtab (
  'SELECT ...',                -- the SELECT statement
  maxopen = 20,                -- An optional parameter
  missing='missing_udf'        -- Another optional parameter
);


The following sections describe the supported parameters. Specifying
an unrecognized parameter name is an error.


c2: Swarmvtab Virtual Table
c3: 3.1. SQL Parameters
c5: swarmvtab.html#sql_parameters
c4:
If a parameter name begins with a ":", then it is assumed to be a
value to bind to the SQL statement before executing it. The value is always
bound as text. It is an error if the specified SQL parameter does not
exist. For example:

CREATE VIRTUAL TABLE temp.x1 USING swarmvtab (
  "SELECT :dir || local_filename, tbl, min, max FROM components",
  :dir = '/home/user/app/databases/'
);


When the above CREATE VIRTUAL TABLE statement is executed, swarmvtab binds
the text value "/home/user/app/databases/" to the :dir parameter of the
SQL statement before executing it.

A single CREATE VIRTUAL TABLE statement may contain any number of SQL
parameters.


c2: Swarmvtab Virtual Table
c3: 3.2. The "maxopen" Parameter
c5: swarmvtab.html#the_maxopen_parameter
c4:
By default, swarmvtab attempts to limit the number of simultaneously
open databases to nine. This parameter allows that limit to be changed.
For example, to create a swarmvtab table that may hold up to 30 databases
open simultaneously:

CREATE VIRTUAL TABLE temp.x1 USING swarmvtab (
  "SELECT ...",
  maxopen=30
);


Raising the number of open databases may improve performance in some
scenarios.


c2: Swarmvtab Virtual Table
c3: 3.3. The "openclose" Callback
c5: swarmvtab.html#the_openclose_callback
c4:
The "openclose" parameter allows the user to specify the name of a
application-defined SQL function that will be invoked just before
swarmvtab opens a database, and again just after it closes one. The first
argument passed to the open close function is the filename or URI
identifying the database to be opened or just recently closed (the same
value returned in the leftmost column of the SQL statement provided to
the CREATE VIRTUAL TABLE command). The second argument is integer value
0 when the function is invoked before opening a database, and 1 when it
is invoked after one is closed. For example, if:

CREATE VIRTUAL TABLE temp.x1 USING swarmvtab (
  "SELECT ...",
  openclose = 'openclose_udf'
);


then before each database containing a component table is opened, 
swarmvtab effectively executes:

SELECT openclose_udf(&lt;database-name&gt;, 0);


After a database is closed, swarmvtab runs the equivalent of:

SELECT openclose_udf(&lt;database-name&gt;, 1);


Any value returned by the openclose function is ignored. If an invocation
made before opening a database returns an error, then the database file is
not opened and the error returned to the user. This is the only scenario
in which swarmvtab will issue an "open" invocation without also eventually
issuing a corresponding "close" call. If there are still databases open,
"close" calls may be issued from within the eventual sqlite3_close() call
on the applications database that deletes the temp schema in which the
swarmvtab table resides.

Errors returned by "close" invocations are always ignored.


c2: Swarmvtab Virtual Table
c3: 3.4. The "missing" Callback
c5: swarmvtab.html#the_missing_callback
c4:
The "missing" parameter allows the user to specify the name of a
application-defined SQL function that will be invoked just before
swarmvtab opens a database if it finds that the required database file
is not present on disk. This provides the application with an opportunity
to retrieve the required database from a remote source before swarmvtab
attempts to open it. The only argument passed to the "missing" function
is the name or URI that identifies the database being opened. Assuming:

CREATE VIRTUAL TABLE temp.x1 USING swarmvtab (
  "SELECT ...",
  openclose = 'openclose_udf',
  missing='missing_udf'
);


then the missing function is invoked as follows:

SELECT missing_udf(&lt;database-name&gt;);


If the missing function returns an error, then the database is not 
opened and the error returned to the user. If an openclose function is
configured, then a "close" invocation is issued at this point to match
the earlier "open". The following pseudo-code illustrates the procedure used
by a swarmvtab instance with both missing and openclose functions configured
when a component database is opened.

SELECT openclose_udf(&lt;database-name&gt;, 0);
if( error ) return error;
if( db does not exist ){
  SELECT missing_udf(&lt;database-name&gt;);
  if( error ){
    SELECT openclose_udf(&lt;database-name&gt;, 1);
    return error;
  }
}
sqlite3_open_v2(&lt;database-name&gt;);
if( error ){
  SELECT openclose_udf(&lt;database-name&gt;, 1);
  return error;
}
// db successfully opened!


c2: Swarmvtab Virtual Table
c3: 3.5. Component table "context" values
c5: swarmvtab.html#component_table_context_values
c4:
If the SELECT statement specified as part of the CREATE VIRTUAL 
TABLE command returns five columns, then the final column is used
for application context only. Swarmvtab does not use this value at
all, except that it is passed after &lt;database-name&gt; to both
the openclose and missing functions, if specified. In other words,
instead of invoking the functions as described above, if the "context"
column is present swarmvtab instead invokes:

SELECT missing_udf(&lt;database-name&gt;, &lt;context&gt;);
SELECT openclose_udf(&lt;database-name&gt;, &lt;context&gt;, 0);
SELECT openclose_udf(&lt;database-name&gt;, &lt;context&gt;, 1);


as required.
This page last modified on  2022-01-08 05:02:57 UTC


c2: Why SQLite Uses Bytecode
c3: 1. Introduction
c5: whybytecode.html#introduction
c4:
Every SQL database engine works in roughly the same way:  It first 
translates the input SQL text into a "prepared statement".  Then it "executes"
the prepared statement to generate a result.

A prepared statement is an object that represents the steps needed
to accomplish the input SQL.  Or, to think of it in another way,
the prepared statement is the SQL statement translated into a form that is
more easily understood by the computer.

In SQLite, a prepared statement is an instance of the
sqlite3_stmt object.  In other systems, the prepared
statement is usually an internal data structure that is not directly visible to
the application programmer.  Developers of other SQL database engines
do not necessarily call these objects "prepared statements".
But such objects exists, whatever they might be called.
This paper will use the term "prepared statement".

There are countless ways of implementing a prepared statement.  This
paper will look at the two most common methods:



Bytecode &rarr;  The input SQL is translated into a virtual machine language
that is then run by a virtual machine interpreter.  This is the technique
used by SQLite.


Tree-Of-Objects &rarr; The input SQL is translated in a tree of objects
that represent the processing to be done.  The SQL is executed by walking this
tree.  This is the technique used by MySQL and PostgreSQL.



There are advantages and disadvantages to each of these representations of
a prepared statement.  The purpose of this paper is to articulate some of those
advantages and disadvantages.


c2: Why SQLite Uses Bytecode
c3: 1.1. How To Provide Feedback
c5: whybytecode.html#how_to_provide_feedback
c4:
This document is written from the perspective of the original author of SQLite.
If you disagree with any of the opinions offered in this document, you are
welcomed to offer corrections and/or contrary views on the
SQLite Forum.  Or you can email the author
directly.


c2: Why SQLite Uses Bytecode
c3: 1.2. Definition Of "Bytecode"
c5: whybytecode.html#definition_of_bytecode_
c4:
The bytecode generated by SQLite might be a little different from what
many readers think of as bytecode.  The bytecode used (for example) by the
Java virtual machine or
by WebAssembly consists almost
entirely of low-level operations, similar to what physical CPUs implement:
basic math operators, comparisons, conditional jumps, and
instructions to move content between different memory locations.  SQLite bytecode
has these kinds of low-level instructions, too.  But SQLite bytecode also contains
some high-level operations that are specific to the needs of a database engine.
Here are just a few examples:



OP_Column &rarr; Extract the value from the N-th column of the
database row that a particular cursor is currently pointing at.
OP_CreateBtree &rarr; Allocate space for a new B-Tree in the
database file.
OP_ParseSchema &rarr; Reread and reparse all or part of the
sqlite_schema table and refresh internal symbol tables accordingly.
OP_SeekGE &rarr; Move a cursor on a particular B-Tree to the first
entry that is greater than or equal to a given key.
OP_Next &rarr; Advance a cursor on a particular B-Tree to the next
entry in the B-Tree and jump, or fall through if there are no more entries in that
B-Tree.


In other words, the "bytecode" used by SQLite is not so much a set of CPU
instructions as it is a list of database primitives that are to be run
in a particular order.


c2: Why SQLite Uses Bytecode
c3: 1.3. Definition Of "Abstract Syntax Tree" or "AST"
c5: whybytecode.html#definition_of_abstract_syntax_tree_or_ast_
c4:
An "Abstract Syntax Tree" or AST
is a data structure that describes a program or statement in some kind of formal
language.  In our context, the formal language is SQL.  An AST is typically
implemented as a tree of objects where each object represents one small part of
the overall SQL statement.  ASTs emerge naturally from parsers for formal languages.
The usual technique is to use an 
LALR(1) parser.
With such a parser, each terminal
symbol holds metadata that will become a leaf of the AST, and each non-terminal 
symbol holds metadata that will become a sub-branch of the overall AST.
As rules of the grammar are "reduced" by the parser, new nodes of the AST are
allocated and connected to subnodes.
After the parse completes, the start-symbol of the grammar is left holding the
root of the AST.

An AST is a tree of objects.  But an AST is not a suitable form for
a prepared statement.  After being generated, an AST first needs to be
transformed in various ways before it can executed.  Symbols need to be
resolved.  Semantic rules need to be checked.  Optimizations need to be
applied that transform input SQL statement into different forms that
execute more quickly.  Finally, the AST needs to be translated into an
alternative representation that is more amenable to execution.

Some people refer to the tree of objects that is used as the
executable form for MySQL and PostgreSQL as an AST.  This is probably a
misuse of the term "AST", because by the time the tree of objects is
ready to be executed, it has been changed so much that it has little
resemblance to the original SQL text.  The confusion arises in part because
both the final prepared statement object and the original AST are both
trees of objects.  The usual technique is for the original AST that comes
directly out of the parser to be transformed little by little, in multiple
passes, until at the end it is fully converted into an tree of objects
that is no longer strictly an AST but that can be evaluated to
generate a result.  There is not necessarily a clear point during this
process when the tree-of-objects ceases to be an AST and becomes a
prepared statement instead.  And because there is no clear boundary between an
AST and a prepared statement, people often refer to a prepared statement
that is represented as a tree of objects as an "AST", even though that
description is not precise.


c2: Why SQLite Uses Bytecode
c3: 1.4. Dataflow Programming
c5: whybytecode.html#dataflow_programming
c4:
Dataflow programming
is a style of programming in which individual nodes specialize in doing
one small part of the overall computation.  Each node receives inputs from 
other nodes and sends its output to other nodes.  Thus the nodes form a
directed graph that carry inputs into outputs.


A "dataflow program" is perhaps a better description than "AST" for
the tree of objects that an SQL database engine uses as a prepared statement.


c2: Why SQLite Uses Bytecode
c3: 2. Advantages To Compiling Into Bytecode
c5: whybytecode.html#advantages_to_compiling_into_bytecode
c4:
SQLite compiles to bytecode, and the SQLite developers are very happy
with this approach.  Here is why:


c2: Why SQLite Uses Bytecode
c3: 2.1. Bytecode Is Easier To Understand
c5: whybytecode.html#bytecode_is_easier_to_understand
c4:
A flat list of opcodes can be easily printed to see exactly
how an SQL statement is being implemented.  This is what happens in SQLite
when you preface an SQL statement with the "EXPLAIN" keyword:  Instead of
actually running the SQL, the result is a listing of the bytecode
that would have been used to implement that SQL.

Bytecode lends itself to this because a bytecode program is easily
represented as a table.  In SQLite bytecode, each instruction
has one opcode and five operands.  Thus a prepared statement can be
rendered as if it were a query against a six-column table.

A tree-of-objects representation is more difficult to publish in
a human-readable form.  The objects that comprise the tree tend to
all be very different, and thus it is tricky to come up with a 
consistent and simple table representation with which to display
the objects.  Any such table representation that you do come up
with would almost certainly have more than six columns, probably many more.
The problem of rendering a tree-of-objects as a table is sufficiently
difficult that nobody does it, as far as I know.  Hence, no
tree-of-objects database engine provides the level
of detail in their "EXPLAIN" output that SQLite provides.


c2: Why SQLite Uses Bytecode
c3: 2.2. Bytecode Is Easier To Debug
c5: whybytecode.html#bytecode_is_easier_to_debug
c4:
Bytecode provides a clear separation between front-end parsing and
analysis and back-end evaluation of an SQL statement.  When problems arise
(incorrect answers and/or poor performance), the developers can examine
the bytecode to quickly determine if the source of the trouble is either the
front-end analysis or the back-end data storage section of the product.

In debugging builds of SQLite, the PRAGMA vdbe_trace=ON; command will
cause a trace of the bytecode execution to appear on the console.


c2: Why SQLite Uses Bytecode
c3: 2.3. Bytecode Can Be Run Incrementally
c5: whybytecode.html#bytecode_can_be_run_incrementally
c4:
SQL statements written in bytecode can be evaluated incrementally.
For example, a statement can be run until it generates just its first
row of output.  The statement then pauses until it is stepped again.
It is not necessary to run the statement to completion before examining
the first row of output.


This is more difficult to achieve in a tree-of-objects design.  When
the prepared statement is a tree-of-objects, execution is normally
accomplished by walking the tree.  To pause the statement in the middle
of a computation means unwinding the stack back up to the caller, all
the while saving enough state to resume evaluation where it last left
off.  This is not impossible to do, but is sufficiently difficult that
I have never seen it actually done.


Most SQL database engines do not really need to do incremental
execution of prepared statements because most SQL database engines
are client/server.  In client/server engines, a single SQL statement is sent
to the server, then the complete reply comes back over the wire
all at once.  Thus each statement runs to completion in a single go.
But SQLite is not client/server.  SQLite is a library
that runs in the same address space and using the same stack as the
application.  Being able to easily and reliably perform incremental
execution of an SQL statement is important to SQLite.


c2: Why SQLite Uses Bytecode
c3: 2.4. Bytecode Is Smaller
c5: whybytecode.html#bytecode_is_smaller
c4:
The bytecode generated by SQLite is usually smaller than the corresponding
AST coming out of the parser.  During initial processing of SQL text
(during the call to sqlite3_prepare() and similar) both the AST and the
bytecode exist in memory at the same time, so more memory is used then.
But that is a transient state.  The AST
is quickly discarded and its memory recycled, even before the call to
sqlite3_prepare() returns, so the resulting prepared statement ends
up consuming less memory in its bytecode representation than it did as an AST.
This is important because calls to sqlite3_prepare() are transient, but
prepared statements are often cached for possible reuse and persist in memory
for a long time.


c2: Why SQLite Uses Bytecode
c3: 2.5. Bytecode Is Faster
c5: whybytecode.html#bytecode_is_faster
c4:
I believe that a bytecode representation of a prepared statement
runs faster, because fewer decisions need to be made for each step of
the computation.  Emphasis on "believe" in the previous sentence
&rarr; it is difficult to verify
this claim experimentally since nobody has ever put in the multiple years
of effort necessary to generate equivalent bytecode and tree-of-object
representations of a prepared statement to see which one actually runs faster.
We do know that SQLite is very fast, but we
do not have good side-by-side comparisons with other SQL databases since
the other databases spend a lot of time doing client/server message processing,
and it is difficult to untangle the message round-trip overhead from the
actual processing time.


c2: Why SQLite Uses Bytecode
c3: 3. Advantages Of Compiling Into A Tree Of Objects
c5: whybytecode.html#advantages_of_compiling_into_a_tree_of_objects
c4:
The SQLite developers think that the bytecode approach is
best, at least for the use cases the SQLite tries to fill, but the
tree-of-objects approach to processing SQL does have some advantages over 
bytecode. There are always tradeoffs.


c2: Why SQLite Uses Bytecode
c3: 3.1. Query Planning Decisions Can Be Deferred Until Runtime
c5: whybytecode.html#query_planning_decisions_can_be_deferred_until_runtime
c4:
When a prepared statement is bytecode, once the bytecode has been generated,
the algorithm is fixed and cannot be subsequently changed without completely
rewriting the bytecode.
This is not the case with a tree-of-objects prepared
statement.  A tree-of-objects is easier to modify on-the-fly.  The query
plan is mutable and can be tweaked as it is running, based on the progress
of the query.  Thus a query can be dynamically self-tuning.


c2: Why SQLite Uses Bytecode
c3: 3.2. Dataflow Programs Are Easy To Parallelize
c5: whybytecode.html#dataflow_programs_are_easy_to_parallelize
c4:
In a dataflow program, each processing node can be assigned to a
different thread.  There needs to be some kind of threadsafe queuing
mechanism for transferring intermediate results from one node to the
next.  But no synchronization primitives are typically needed within
each node of the program.  Node schedule is trivial:  A node becomes
eligible to run when it has data available and there is space in its
output queue.

This is an important consideration for database engines that are
designed to run large analytic queries 
(OLAP)
on large multi-core servers.
The primary focus of SQLite is transaction processing 
(OLTP) 
on the internet-of-things, so there is less need to
represent prepared statements as dataflow programs in SQLite.
This page last modified on  2024-05-09 17:38:03 UTC


c2: Deterministic SQL Functions
c3: 1. Overview
c5: deterministic.html#overview
c4:
SQL functions in SQLite can be either "deterministic" or "non-deterministic".


A deterministic function always gives the same answer when it has
the same inputs.  Most built-in SQL functions in SQLite are
deterministic.  For example, the abs(X) function always returns
the same answer as long as its input X is the same.


Non-deterministic functions might give different answers on each
invocation, even if the arguments are always the same.  The following
are examples of non-deterministic functions:


 random()
 changes()
 last_insert_rowid()
 sqlite3_version()



The random() function is obviously non-deterministic because it gives
a different answer every time it is invoked.  The answers from changes()
and last_insert_rowid() depend on prior SQL statements, and so they
are also non-deterministic. The
sqlite3_version() function is mostly constant, but it can change when
SQLite is upgraded, and so even though it always returns the same answer
for any particular session, because it can change answers across sessions
it is still considered non-deterministic.


c2: Deterministic SQL Functions
c3: 2. Restrictions on the use of non-deterministic functions
c5: deterministic.html#restrictions_on_the_use_of_non_deterministic_functions
c4:
There are some contexts in SQLite that do not allow the use of
non-deterministic functions:


In the expression of a CHECK constraint.
In the WHERE clause of a partial index.
In an expression used as part of an expression index.
In the expression of a generated column.



In the cases above, the values returned by the function affects the
information stored in the database file.  The values of functions
in CHECK constraints determines which entries are valid for a table,
and functions in the WHERE clause of a partial index or in an index on
an expression compute values stored in the index b-tree.
If any of these functions later returns a different
value, then the database might no longer be well-formed.  
Hence, to avoid database corruption,
only deterministic functions can be used in the contexts
above.


c2: Deterministic SQL Functions
c3: 3. Special-case Processing For Date/Time Functions
c5: deterministic.html#special_case_processing_for_date_time_functions
c4:
The built-in date and time functions of SQLite are a special case.
These functions are usually considered deterministic.  However, if
these functions use the string "now" as the date, or if they use
the localtime modifier or the utc modifier, then they are
considered non-deterministic.  Because the function inputs are
not necessarily known until run-time, the date/time functions will
throw an exception if they encounter any of the non-deterministic
features in a context where only deterministic functions are allowed.


Prior to SQLite 3.20.0 (2017-08-01) all date/time functions were
always considered non-deterministic.  The ability for date/time functions
to be deterministic sometimes and non-deterministic at other times,
depending on their arguments, was added for the 3.20.0 release.


c2: Deterministic SQL Functions
c3: 3.1. Bug fix in version 3.35.2
c5: deterministic.html#bug_fix_in_version_3_35_2
c4:
When the enhancement was made to SQLite 3.20.0 such that date/time
functions would be considered deterministic as they do not depend
on the current time, one case was overlooked:
Many of the date/time functions can be called
with no arguments at all.  These no-argument date/time functions
behave as if they had a single "'now'" argument.
Thus "datetime()" and
"datetime('now')" both yield the current date and time.
However, only the second form was recognized as non-deterministic.
This meant that developers could sneak the non-deterministic
"datetime()" form into CHECK constraints, index
expressions, generated column expressions, and similar places
where non-deterministic functions make no sense.
This oversight was fixed in version 3.35.2 (2021-03-17).
However, there may be legacy databases in circulation that were created
by SQLite version 3.20.0 through 3.35.1 that have non-deterministic
date/time functions in their schemas.


c2: Deterministic SQL Functions
c3: 4. Application-defined deterministic functions
c5: deterministic.html#application_defined_deterministic_functions
c4:
By default, application-defined SQL functions are considered to
be non-deterministic.  However, if the 4th parameter to
sqlite3_create_function_v2() is OR-ed with 
SQLITE_DETERMINISTIC, then SQLite will treat that function as if it
were deterministic.


Note that if a non-deterministic function is tagged with
SQLITE_DETERMINISTIC and if that function ends up being used in
the WHERE clause of a partial index or in an
expression index, then when the function begins to return different
answers, the associated index may become corrupt.  If an SQL function
is nearly deterministic (which is to say, if it only rarely changes,
like sqlite_version()) and it is used in an index that becomes
corrupt, the corruption can be fixed by running REINDEX.


The interfaces necessary to construct a function that is sometimes
deterministic and sometimes non-deterministic depending on their
inputs, such as the built-in date/time functions, are not published.
Generic application-defined SQL functions must
be always deterministic or always non-deterministic.
This page last modified on  2022-01-08 05:02:57 UTC


c2: The Geopoly Interface To The SQLite R*Tree Module
c3: 1. Overview
c5: geopoly.html#overview
c4:
The Geopoly module is an alternative interface to the R-Tree extension that uses
the GeoJSON notation
(RFC-7946) to describe two-dimensional
polygons.  Geopoly includes functions for detecting when one polygon is
contained within or overlaps with another, for computing the
area enclosed by a polygon, for doing linear transformations of polygons,
for rendering polygons as
SVG, and other
similar operations.


The source code for Geopoly is included in the amalgamation.  However,
depending on configuration options and the a particular version of SQLite
you are using, the Geopoly extension may or may not be enabled by default.
To ensure that Geopoly is enabled for your build, 
add the -DSQLITE_ENABLE_GEOPOLY=1 compile-time option.


Geopoly operates on "simple" polygons - that is, polygons for which
the boundary does not intersect itself.  Geopoly thus extends the capabilities
of the R-Tree extension which can only deal with rectangular areas.
On the other hand, the R-Tree extension is
able to handle between 1 and 5 coordinate dimensions, whereas Geopoly is restricted
to 2-dimensional shapes only.


Each polygon in the Geopoly module can be associated with an arbitrary
number of auxiliary data fields.


c2: The Geopoly Interface To The SQLite R*Tree Module
c3: 1.1. GeoJSON
c5: geopoly.html#geojson
c4:
The GeoJSON standard is syntax for
exchanging geospatial information using JSON.  GeoJSON is a rich standard
that can describe nearly any kind of geospatial content.

The Geopoly module only understands
a small subset of GeoJSON, but a critical subset.  
In particular, GeoJSON understands
the JSON array of vertexes that describes a simple polygon.

A polygon is defined by its vertexes.
Each vertex is a JSON array of two numeric values which are the
X and Y coordinates of the vertex.
A polygon is a JSON array of at least four of these vertexes, 
and hence is an array of arrays.
The first and last vertex in the array must be the same.
The polygon follows the right-hand rule:  When tracing a line from
one vertex to the next, the area to the right of the line is outside
of the polygon and the area to the left is inside the polygon.
In other words, the net rotation of the vertexes is counter-clockwise.


For example, the following JSON describes an isosceles triangle, sitting
on the X axis and with an area of 0.5:

&#91;&#91;0,0],&#91;1,0],&#91;0.5,1],&#91;0,0]]



A triangle has three vertexes, but the GeoJSON description of the triangle
has 4 vertexes because the first and last vertex are duplicates.


c2: The Geopoly Interface To The SQLite R*Tree Module
c3: 1.2. Binary storage format
c5: geopoly.html#binary_storage_format
c4:
Internally, Geopoly stores polygons in a binary format - an SQL BLOB.
Details of the binary format are given below.
All of the Geopoly interfaces are able to accept polygons in either the
GeoJSON format or in the binary format.


c2: The Geopoly Interface To The SQLite R*Tree Module
c3: 2. Using The Geopoly Extension
c5: geopoly.html#using_the_geopoly_extension
c4:
A geopoly table is created as follows:

CREATE VIRTUAL TABLE newtab USING geopoly(a,b,c);



The statement above creates a new geopoly table named "newtab".
Every geopoly table contains a built-in integer "rowid" column
and a "_shape" column that contains
the polygon associated with that row of the table.
The example above also defines three auxiliary data columns 
named "a", "b", and "c" that can store whatever additional
information the application needs to associate
with each polygon.  If there is no need to store auxiliary
information, the list of auxiliary columns can be omitted.


Store new polygons in the table using ordinary INSERT statements:

INSERT INTO newtab(_shape) VALUES('&#91;&#91;0,0],&#91;1,0],&#91;0.5,1],&#91;0,0]]');



UPDATE and DELETE statements work similarly.


c2: The Geopoly Interface To The SQLite R*Tree Module
c3: 2.1. Queries
c5: geopoly.html#queries
c4:
To query the geopoly table using an indexed geospatial search, 
use one of the functions geopoly_overlap()
or geopoly_within() as a boolean function in the WHERE clause,
with the "_shape" column as the first argument to the function.
For example:

SELECT * FROM newtab WHERE geopoly_overlap(_shape, $query_polygon);



The previous example will return every row for which the _shape
overlaps the polygon in the $query_polygon parameter.  The
geopoly_within() function works similarly, but only returns rows for
which the _shape is completely contained within $query_polygon.


Queries (and also DELETE and UPDATE statements) in which the WHERE
clause contains a bare geopoly_overlap() or geopoly_within() function
make use of the underlying R*Tree data structures for a fast lookup that
only has to examine a subset of the rows in the table.  The number of
rows examines depends, of course, on the size of the $query_polygon.
Large $query_polygons will normally need to look at more rows than small
ones.


Queries against the rowid of a geopoly table are also very quick, even
for tables with a vast number of rows.
However, none of the auxiliary data columns are indexes, and so queries
against the auxiliary data columns will involve a full table scan.


c2: The Geopoly Interface To The SQLite R*Tree Module
c3: 3. Special Functions
c5: geopoly.html#special_functions
c4:
The geopoly module defines several new SQL functions that are useful for
dealing with polygons.  All polygon arguments to these functions can be
either the GeoJSON format or the internal binary format.


c2: The Geopoly Interface To The SQLite R*Tree Module
c3: 3.1. The geopoly_overlap(P1,P2) Function
c5: geopoly.html#the_geopoly_overlap_p1_p2_function
c4:
If P1 and P2 are both polygons, then the geopoly_overlap(P1,P2) function returns
a non-zero integer if there is any overlap between P1 and P2, or it returns
zero if P1 and P2 completely disjoint.
If either P1 or P2 is not a polygon, this routine returns NULL.


The geopoly_overlap(P1,P2) function is special in that the geopoly virtual
table knows how to use R*Tree indexes to optimize queries in which the 
WHERE clause uses geopoly_overlap() as a boolean function.  Only the
geopoly_overlap(P1,P2) and geopoly_within(P1,P2) functions have this
capability.


c2: The Geopoly Interface To The SQLite R*Tree Module
c3: 3.2. The geopoly_within(P1,P2) Function
c5: geopoly.html#the_geopoly_within_p1_p2_function
c4:
If P1 and P2 are both polygons, then the geopoly_within(P1,P2) function returns
a non-zero integer if P1 is completely contained within P2, or it returns zero
if any part of P1 is outside of P2.  If P1 and P2 are the same polygon, this routine
returns non-zero.
If either P1 or P2 is not a polygon, this routine returns NULL.


The geopoly_within(P1,P2) function is special in that the geopoly virtual
table knows how to use R*Tree indexes to optimize queries in which the 
WHERE clause uses geopoly_within() as a boolean function.  Only the
geopoly_within(P1,P2) and geopoly_overlap(P1,P2) functions have this
capability.


c2: The Geopoly Interface To The SQLite R*Tree Module
c3: 3.3. The geopoly_area(P) Function
c5: geopoly.html#the_geopoly_area_p_function
c4:
If P is a polygon, then geopoly_area(P) returns the area enclosed by
that polygon.  If P is not a polygon, geopoly_area(P) returns NULL.


c2: The Geopoly Interface To The SQLite R*Tree Module
c3: 3.4. The geopoly_blob(P) Function
c5: geopoly.html#the_geopoly_blob_p_function
c4:
If P is a polygon, then geopoly_blob(P) returns the binary encoding
of that polygon as a BLOB.
If P is not a polygon, geopoly_blob(P) returns NULL.


c2: The Geopoly Interface To The SQLite R*Tree Module
c3: 3.5. The geopoly_json(P) Function
c5: geopoly.html#the_geopoly_json_p_function
c4:
If P is a polygon, then geopoly_json(P) returns the GeoJSON representation
of that polygon as a TEXT string.
If P is not a polygon, geopoly_json(P) returns NULL.


c2: The Geopoly Interface To The SQLite R*Tree Module
c3: 3.6. The geopoly_svg(P,...) Function
c5: geopoly.html#the_geopoly_svg_p_function
c4:
If P is a polygon, then geopoly_svg(P,...) returns a text string which is a
Scalable Vector Graphics (SVG)
representation of that polygon.  If there is more one argument, then second
and subsequent arguments are added as attributes to each SVG glyph.  For example:

SELECT geopoly_svg($polygon,'class="poly"','style="fill:blue;"');



If P is not a polygon, geopoly_svg(P,...) returns NULL.


Note that geopoly uses a traditional right-handed cartesian coordinate system
with the origin at the lower left, whereas SVG uses a left-handed coordinate
system with the origin at the upper left.  The geopoly_svg() routine makes no
attempt to transform the coordinate system, so the displayed images are shown
in mirror image and rotated.  If that is undesirable, the geopoly_xform() routine
can be used to transform the output from cartesian to SVG coordinates prior to
passing the polygons into geopoly_svg().


c2: The Geopoly Interface To The SQLite R*Tree Module
c3: 3.7. The geopoly_bbox(P) and geopoly_group_bbox(P) Functions
c5: geopoly.html#the_geopoly_bbox_p_and_geopoly_group_bbox_p_functions
c4:
If P is a polygon, then geopoly_bbox(P) returns a new polygon that is
the smallest (axis-aligned) rectangle completely enclosing P.
If P is not a polygon, geopoly_bbox(P) returns NULL.


The geopoly_group_bbox(P) function is an aggregate version of geopoly_bbox(P).
The geopoly_group_bbox(P) function returns the smallest rectangle that will
enclose all P values seen during aggregation.


c2: The Geopoly Interface To The SQLite R*Tree Module
c3: 3.8. The geopoly_contains_point(P,X,Y) Function
c5: geopoly.html#the_geopoly_contains_point_p_x_y_function
c4:
If P is a polygon, then geopoly_contains_point(P,X,Y) returns a 
non-zero integer if and only
if the coordinate X,Y is inside or on the boundary of the polygon P.
If P is not a polygon, geopoly_contains_point(P,X,Y) returns NULL.


c2: The Geopoly Interface To The SQLite R*Tree Module
c3: 3.9. The geopoly_xform(P,A,B,C,D,E,F) Function
c5: geopoly.html#the_geopoly_xform_p_a_b_c_d_e_f_function
c4:
The geopoly_xform(P,A,B,C,D,E,F) function returns a new polygon that is an
affine transformation of the polygon P and where the transformation
is defined by values A,B,C,D,E,F. If P is not a valid polygon, this
routine returns NULL.


The transformation converts each vertex of the polygon according to the
following formula:

x1 = A*x0 + B*y0 + E
y1 = C*x0 + D*y0 + F



So, for example, to move a polygon by some amount DX, DY without changing
its shape, use:

geopoly_xform($polygon, 1, 0, 0, 1, $DX, $DY)



To rotate a polygon by R radians around the point 0, 0:

geopoly_xform($polygon, cos($R), sin($R), -sin($R), cos($R), 0, 0)



Note that a transformation that flips the polygon might cause the
order of vertexes to be reversed.  In other words, the transformation
might cause the vertexes to circulate in clockwise order instead of
counter-clockwise.  This can be corrected by sending the result
through the geopoly_ccw() function after transformation.


c2: The Geopoly Interface To The SQLite R*Tree Module
c3: 3.10. The geopoly_regular(X,Y,R,N) Function
c5: geopoly.html#the_geopoly_regular_x_y_r_n_function
c4:
The geopoly_regular(X,Y,R,N) function returns a convex, simple, regular,
equilateral, equiangular polygon with N sides, centered at X,Y, and with
a circumradius of R.  Or, if R is negative or if N is less than 3, the
function returns NULL.  The N value is capped at 1000 so that the routine
will never render a polygon with more than 1000 sides even if the N value
is larger than 1000.


As an example, the following graphic:


<svg width="600" height="300" style="border:1px solid black">
<polyline points="140.003,100 80.0019,134.644 80.0019,65.3565 140.003,100" style="fill:none;stroke:red;stroke-width:2"></polyline> <text x="100" y="106" alignment-baseline="central" text-anchor="middle">3</text>
<polyline points="240.003,100 200,140.003 159.997,100 200,59.9973 240.003,100" style="fill:none;stroke:orange;stroke-width:2"></polyline> <text x="200" y="106" alignment-baseline="central" text-anchor="middle">4</text>
<polyline points="340.003,100 312.358,138.042 267.637,123.511 267.637,76.4893 312.358,61.9583 340.003,100" style="fill:none;stroke:green;stroke-width:2"></polyline> <text x="300" y="106" alignment-baseline="central" text-anchor="middle">5</text>
<polyline points="440.003,100 419.998,134.644 380.002,134.644 359.997,100 380.002,65.3565 419.998,65.3565 440.003,100" style="fill:none;stroke:blue;stroke-width:2"></polyline> <text x="400" y="106" alignment-baseline="central" text-anchor="middle">6</text>
<polyline points="540.003,100 524.94,131.276 491.101,138.995 463.959,117.353 463.959,82.6471 491.101,61.005 524.94,68.7243 540.003,100" style="fill:none;stroke:purple;stroke-width:2"></polyline> <text x="500" y="106" alignment-baseline="central" text-anchor="middle">7</text>
<polyline points="140.003,200 128.286,228.286 100,240.003 71.7143,228.286 59.9973,200 71.7143,171.714 100,159.997 128.286,171.714 140.003,200" style="fill:none;stroke:red;stroke-width:2"></polyline> <text x="100" y="206" alignment-baseline="central" text-anchor="middle">8</text>
<polyline points="240.003,200 232.363,223.511 212.358,238.042 187.642,238.042 167.637,223.511 159.997,200 167.637,176.489 187.642,161.958 212.358,161.958 232.363,176.489 240.003,200" style="fill:none;stroke:orange;stroke-width:2"></polyline> <text x="200" y="206" alignment-baseline="central" text-anchor="middle">10</text>
<polyline points="340.003,200 334.644,219.998 319.998,234.644 300,240.003 280.002,234.644 265.356,219.998 259.997,200 265.356,180.002 280.002,165.356 300,159.997 319.998,165.356 334.644,180.002 340.003,200" style="fill:none;stroke:green;stroke-width:2"></polyline> <text x="300" y="206" alignment-baseline="central" text-anchor="middle">12</text>
<polyline points="440.003,200 436.956,215.305 428.286,228.286 415.305,236.956 400,240.003 384.695,236.956 371.714,228.286 363.044,215.305 359.997,200 363.044,184.695 371.714,171.714 384.695,163.044 400,159.997 415.305,163.044 428.286,171.714 436.956,184.695 440.003,200" style="fill:none;stroke:blue;stroke-width:2"></polyline> <text x="400" y="206" alignment-baseline="central" text-anchor="middle">16</text>
<polyline points="540.003,200 538.042,212.358 532.363,223.511 523.511,232.363 512.358,238.042 500,240.003 487.642,238.042 476.489,232.363 467.637,223.511 461.958,212.358 459.997,200 461.958,187.642 467.637,176.489 476.489,167.637 487.642,161.958 500,159.997 512.358,161.958 523.511,167.637 532.363,176.489 538.042,187.642 540.003,200" style="fill:none;stroke:purple;stroke-width:2"></polyline> <text x="500" y="206" alignment-baseline="central" text-anchor="middle">20</text>
</svg>


Was generated by this script:

SELECT '&lt;svg width="600" height="300">';
WITH t1(x,y,n,color) AS (VALUES
   (100,100,3,'red'),
   (200,100,4,'orange'),
   (300,100,5,'green'),
   (400,100,6,'blue'),
   (500,100,7,'purple'),
   (100,200,8,'red'),
   (200,200,10,'orange'),
   (300,200,12,'green'),
   (400,200,16,'blue'),
   (500,200,20,'purple')
)
SELECT
   geopoly_svg(geopoly_regular(x,y,40,n),
        printf('style="fill:none;stroke:%s;stroke-width:2"',color))
   || printf(' &lt;text x="%d" y="%d" alignment-baseline="central" text-anchor="middle">%d&lt;/text>',x,y+6,n)
  FROM t1;
SELECT '&lt;/svg>';


c2: The Geopoly Interface To The SQLite R*Tree Module
c3: 3.11. The geopoly_ccw(J) Function
c5: geopoly.html#the_geopoly_ccw_j_function
c4:
The geopoly_ccw(J) function returns the polygon J with counter-clockwise (CCW) rotation.


RFC-7946 requires that polygons use CCW rotation.
But the spec also observes that many legacy GeoJSON files do not following the spec and
contain polygons with clockwise (CW) rotation.  The geopoly_ccw() function is useful for
applications that are reading legacy GeoJSON scripts.  If the input to geopoly_ccw() is
a correctly-formatted polygon, then no changes are made.  However, if the circulation of
the input polygon is backwards, then geopoly_ccw() reverses the circulation order so that
it conforms to the spec and so that it will work correctly with the Geopoly module.


c2: The Geopoly Interface To The SQLite R*Tree Module
c3: 4. Implementation Details
c5: geopoly.html#implementation_details
c4:
The geopoly module is an extension to the R-Tree extension.  Geopoly
uses the same underlying logic and shadow tables as the R-Tree extension.
Geopoly merely presents a different interface, and provides some extra logic
to compute polygon decoding, overlap, and containment.


c2: The Geopoly Interface To The SQLite R*Tree Module
c3: 4.1. Binary Encoding of Polygons
c5: geopoly.html#binary_encoding_of_polygons
c4:
Geopoly stores all polygons internally using a binary format.  A binary
polygon consists of a 4-byte header following by an array of coordinate
pairs in which each dimension of each coordinate is a 32-bit floating point
number.


The first byte of the header is a flag byte.  The least significant bit
of the flag byte determines whether the coordinate pairs that follow the
header are stored big-endian or little-endian.  A value of 0 for the least
significant bit means big-endian and a value of 1 means little endian.
Other bits of the first byte in the header are reserved for future expansion.


The next three bytes in the header record the number of vertexes in the polygon
as a big-endian integer.  Thus there is an upper bound of about 16 million
vertexes per polygon.


Following the header is the array of coordinate pairs.  Each coordinate is
a 32-bit floating point number.  The use of 32-bit floating point values for
coordinates means that any point on the earth's surface can be mapped with
a resolution of approximately 2.5 meters.  Higher resolutions are of course
possible if the map is restricted to a single continent or country.
Note that the resolution of coordinates in the geopoly module is similar
in magnitude to daily movement of points on the earth's surface due to
tidal forces.


The list of coordinates in the binary format contains no redundancy.  
The last coordinate is not a repeat of the first as it is with GeoJSON.  
Hence, there is always one fewer coordinate pair in the binary representation of
a polygon compared to the GeoJSON representation.


c2: The Geopoly Interface To The SQLite R*Tree Module
c3: 4.2. Shadow Tables
c5: geopoly.html#shadow_tables
c4:
The geopoly module is built on top of the R-Tree extension and uses the
same underlying shadow tables and algorithms.  For indexing purposes, each
polygon is represented in the shadow tables as a rectangular bounding box.
The underlying R-Tree implementation uses bounding boxes to limit the search
space.  Then the geoploy_overlap() and/or geopoly_within() routines further
refine the search to the exact answer.
This page last modified on  2023-12-05 14:43:20 UTC


c2: How To Download Canonical SQLite Source Code
c3: 1. Introduction
c5: getthecode.html#introduction
c4:
Most programmers compile SQLite into their applications using
the amalgamation.  The amalgamation is C-code but it is not
"source code".  The amalgamation is generated from source code
by scripts.

This document describes how to obtain the canonical source code
for SQLite - the raw source files from which the amalgamation is
built.  See the How To Compile SQLite page for additional information
on what to do with the canonical source code once it is obtained.


c2: How To Download Canonical SQLite Source Code
c3: 2. Direct Downloads
c5: getthecode.html#direct_downloads
c4:
Snapshots of official releases of SQLite source code can often
be obtained directly from the download page of the SQLite website.
Even if the specific version desired is not listed on the download page,
the naming conventions are fairly clear and so programmers can often
guess the name of an historical release and download it that way.


c2: How To Download Canonical SQLite Source Code
c3: 3. Obtaining Code Directly From the Version Control System
c5: getthecode.html#obtaining_code_directly_from_the_version_control_system
c4:
For any historical version of SQLite, the source tree can be obtained
from the Fossil version control system,
either downloading a tarball or ZIP archive for a specific version, or
by cloning the entire project history.

SQLite sources are maintained on three geographically dispersed
servers:


https://www.sqlite.org/src (Dallas)
https://www2.sqlite.org/src (Newark)
https://www3.sqlite.org/src (San Francisco)


The documentation is maintained in separate source repositories on
those same servers:


https://www.sqlite.org/docsrc (Dallas)
https://www2.sqlite.org/docsrc (Newark)
https://www3.sqlite.org/docsrc (San Francisco)


To download a specific historical version, first locate the specific
version desired by visiting the timeline page on one of these servers
(for example: https://www.sqlite.org/src/timeline).  If
you know the approximate date of the version you want to download, you
can add a query parameter like "c=YYYY-MM-DD" to the "timeline" URL to
see a timeline centered on that date.  For example, to see all the check-ins
that occurred around August 26, 2013, visit
https://www.sqlite.org/src/timeline?c=2013-08-26.
If you are looking for an official release, visit the
chronology page, click on the date to the left of the release
you are looking for, and that will take you immediately to the
check-in corresponding to the release.

Once you locate a specific version, click on the hyperlink for that
version to see the "Check-in Information Page".
Then click on either the "Tarball" link or the
"ZIP archive" link to download the complete source tree.


c2: How To Download Canonical SQLite Source Code
c3: 4. Verifying That The Code Is Unmodified
c5: getthecode.html#verifying_that_the_code_is_unmodified
c4:
The "manifest" file at the root directory of the source tree
contains either a SHA3-256 hash or a SHA1 hash for every source file
in the repository. The name of the version of the entire source tree
is just the SHA3-256 hash of the "manifest" file itself, possibly with
the last line of that file omitted if the last line begins with 
"# Remove this line". The "manifest.uuid" file should contain
the SHA3-256 hash of the "manifest" file. If all of the above hash 
comparisons are correct, then you can be confident that your source
tree is authentic and unadulterated. Details on the format of
manifest files are available on the 

Fossil website.


The process of checking source code authenticity is automated by the makefile:

make verify-source



Or on windows:

nmake /f Makefile.msc verify-source



Using the makefile to verify source integrity is good for detecting
accidental changes to the source tree, but malicious changes could
be hidden by also modifying the makefiles.


c2: How To Download Canonical SQLite Source Code
c3: 5. Cloning The Complete Development History
c5: getthecode.html#cloning_the_complete_development_history
c4:
To clone the entire history of SQLite, first go to the
https://www.fossil-scm.org/download.html page and grab a precompiled binary
for the Fossil version control program.  Or get the source code on the
same page and compile it yourself.

As of 2017-03-12, you must use Fossil version
2.0 or later for the following instructions to work.  
The SQLite repository started using
artifacts named using SHA3 hashes instead of SHA1 hashes on that date,
and Fossil 2.0 or later is needed in order to understand the new SHA3
hashes.  To find out what version of Fossil you are running, 
type "fossil -v".

Fossil is a completely stand-alone
program, so install it simply by putting the "fossil" or "fossil.exe"
executable someplace on your $PATH or %PATH%.  After you have Fossil
installed, do this:

fossil clone https://www.sqlite.org/src sqlite.fossil


The command above
will make a copy of the complete development history of
SQLite into the "sqlite.fossil" file on your computer.  Making this copy
takes about a minute and uses about 32 megabytes of transfer.  After
making the copy, "open" the repository by typing:

fossil open sqlite.fossil


This second command will "checkout" the latest check-in from the SQLite
source tree into your current directory.  Subsequently, you can easily switch
to a different version by typing:

fossil update VERSION


Where VERSION can be a branch name (like "trunk" or "session") to get the
latest check-in on a specific branch, or VERSION can be a SHA1 hash or a
prefix of a SHA1 hash for a specific check-in, or VERSION can be a tag
such as "version-3.8.8".  Every time you run "fossil update" it will
automatically reach out to the original repository at
https://www.sqlite.org/src to obtain new check-ins that might have been
made by others since your previous update.


c2: The SQLITE_MEMSTAT Virtual Table
c3: 1. Overview
c5: memstat.html#overview
c4:
The SQLITE_MEMSTAT extension implements an eponymous-only virtual table that
provides SQL access to the sqlite3_status64() and
sqlite3_db_status() interfaces.



The SQLITE_STMT extension can also be loaded at run-time
by compiling the extension into a shared library or DLL using the source
code at https://sqlite.org/src/file/ext/misc/memstat.c and following the
instructions for how to compile loadable extensions.


c2: The SQLITE_MEMSTAT Virtual Table
c3: 2. Usage
c5: memstat.html#usage
c4:
The SQLITE_MEMSTAT virtual table is a read-only table that can be
queried to determine performance characteristics (primarily the
amount of memory being used) of the current instance of SQLite.
The SQLITE_MEMSTATE table is essentially a wrapper around the
C-language APIs sqlite3_status64() and sqlite3_db_status().
If the 
memstat.c source
file is compiled with the -DSQLITE_ENABLE_ZIPVFS option, then SQLITE_MEMSTAT
will also do some file-control calls to extract
memory usage information about the 
ZIPVFS subsystem, 
if that subsystem as been licensed, installed, and is in use.


The SQLITE_MEMSTAT table appears to have the following schema:

CREATE TABLE sqlite_memstat(
  name TEXT,
  schema TEXT,
  value INT,
  hiwtr INT
);



Each row of the SQLITE_MEMSTAT table corresponds to a single call to
one of the sqlite3_status64() or sqlite3_db_status() interfaces.
The NAME column of the row identifies which "verb" was passed to those
interfaces.  For example, if sqlite3_status64() is invoked with
SQLITE_STATUS_MEMORY_USED, then the NAME column is 'MEMORY_USED'.
Or if sqlite3_db_status() is invoked with SQLITE_DBSTATUS_CACHE_USED,
then the NAME column is "DB_CACHE_USED".


The SCHEMA column is NULL, except for cases when the sqlite3_file_control()
interface is used to interrogate the ZIPVFS backend.  As this only happens
when the memstat.c module is compiled with -DSQLITE_ENABLE_ZIPVFS and when
ZIPVFS is in use, 
SCHEMA is usually NULL.


The VALUE and HIWTR columns report the current value of the measure and
its "high-water mark".  The high-water mark is the highest value ever seen
for the measurement, at least since the last reset.  The SQLITE_MEMSTAT
virtual table does not provide a mechanism for resetting the high-water mark.


Depending on which parameter is being interrogated, one of the VALUE
or HIWTR mark measurements might be undefined.  For example, only the
high-water mark is meaningful for SQLITE_STATUS_MALLOC_SIZE, and
only the current value is meaningful for SQLITE_DBSTATUS_CACHE_USED.
For rows where one or the other of VALUE or HIWTR is not meaningful,
that value is returned as NULL.
the 

interfaces, with the initial
This page last modified on  2022-01-08 05:02:57 UTC


c2: The SQLite R*Tree Module
c3: 1. Overview
c5: rtree.html#overview
c4:
An R-Tree is a special
index that is designed for doing range queries.  R-Trees are most commonly
used in geospatial systems where each entry is a rectangle with minimum and
maximum X and Y coordinates.  Given a query rectangle, an R-Tree is able
to quickly find all entries that are contained within the query rectangle
or which overlap the query rectangle.  This idea is easily extended to
three dimensions for use in CAD systems.  R-Trees also find use in time-domain
range look-ups.  For example, suppose a database records the starting and
ending times for a large number of events.  A R-Tree is able to quickly
find all events that were active at any time during a given
time interval, or all events that started during a particular time interval,
or all events that both started and ended within a given time interval.
And so forth.



The R-Tree concept originated with
Toni Guttman:
R-Trees: A Dynamic Index Structure for Spatial Searching,
Proc. 1984 ACM SIGMOD International Conference on Management of Data,
pp. 47-57.
The implementation found in SQLite is a refinement of Guttman's original
idea, commonly called "R*Trees", that was described by
Norbert Beckmann, Hans-Peter Kriegel, Ralf Schneider, Bernhard Seeger:
The R*-Tree: An Efficient and Robust Access Method for Points
and Rectangles. SIGMOD Conference 1990: 322-331.


c2: The SQLite R*Tree Module
c3: 2. Compiling The R*Tree Module
c5: rtree.html#compiling_the_r_tree_module
c4:
The source code to the SQLite R*Tree module is included as part
of the amalgamation.  However, depending on configuration options
and the particular version of SQLite you are using, it may or may not
be enabled by default.  To ensure that the R*Tree module is enabled,
simply compile with the SQLITE_ENABLE_RTREE 
C-preprocessor macro defined.  With many compilers, this is accomplished
by adding the option "-DSQLITE_ENABLE_RTREE=1" to the compiler
command-line.


c2: The SQLite R*Tree Module
c3: 3. Using the R*Tree Module
c5: rtree.html#using_the_r_tree_module
c4:
The SQLite R*Tree module is implemented as a
virtual table.  Each R*Tree index is a
virtual table with an odd number of columns between 3 and 11.
The first column is always a 64-bit signed integer primary key.
The other columns are pairs, one pair per dimension, containing the
minimum and maximum values for that dimension, respectively.
A 1-dimensional R*Tree thus has 3 columns.
A 2-dimensional R*Tree has 5 columns.
A 3-dimensional R*Tree has 7 columns.
A 4-dimensional R*Tree has 9 columns.
And a 5-dimensional R*Tree has 11 columns.  The SQLite R*Tree implementation
does not support R*Trees wider than 5 dimensions.



The first column of an SQLite R*Tree is similar to an integer primary
key column of a normal SQLite table. It may only store a 64-bit signed
integer value. Inserting a NULL value into this column causes SQLite
to automatically generate a new unique primary key value. If an attempt
is made to insert any other non-integer value into this column,
the r-tree module silently converts it to an integer before writing it
into the database.

The min/max-value pair columns are stored as 32-bit floating point values for
"rtree" virtual tables or as 32-bit signed integers in "rtree_i32" virtual
tables.  Unlike regular SQLite tables which can store data in a variety of
datatypes and formats, the R*Tree rigidly enforce these storage types.
If any other type of value is inserted into such a column, the r-tree
module silently converts it to the required type before writing the
new record to the database.


