# page_content

c2: Powersafe Overwrite
c5: psow.html
c4:
Powersafe Overwrite

"Powersafe overwrite" is a term used by the SQLite team to describe
a behavior of some filesystems and disk-controllers related to
data preservation during a power loss.  Powersafe overwrite
is a boolean property: either the storage system has it or it does not.

We say that a system has the powersafe overwrite property if the following
statement is true:


  When an application writes a range of bytes in a file, no
  bytes outside of that range will change, even if the write occurs
  just before a crash or power failure.


The powersafe overwrite property says nothing about the state of the
bytes that were written.  Those bytes might contain their old values,
their new values, random values, or some combination of these.  The powersafe
overwrite property merely states that writes cannot change bytes outside
of the range of bytes written.

In other words, powersafe overwrite means that there is no "collateral
damage" when a power loss occurs while writing.  Only those bytes actually
being written might be damaged.

In practical terms, what the powersafe write property means is that when
the disk controller detects an impending power loss, it finishes writing
whatever sector it is working on prior to parking the heads.  It means that
individual sector writes will complete once started, even if
there is a power loss.

Consider what would happen if disk sector writes are interrupted
by a power loss.  If an application writes two or three bytes in the middle
of some file, the operating system will implement this by first reading
the entire sector containing those bytes, making the change to the
sector in memory, then writing the entire sector back to the disk.  If a power
loss occurs during the writeback and the sector was not completely written,
then on the next read after reboot, error correcting codes
in the sector will probably detect irreparable damage and the disk 
controller will read out the sector as all zeros or all ones.  Thus
values will have changed outside of the range of the two or three bytes 
that were written at the application level - a violation of the powersafe
overwrite property.

SQLite Assumptions About Powersafe Overwrite

All versions of SQLite up to and including version 3.7.9
(2011-11-01) assume that
the filesystem does not provide powersafe overwrite.  SQLite 
has traditionally assumed that when any one byte of a file changes, all
other bytes within the same sector of that byte have the potential of
being corrupted on a power loss.  When writing, SQLite has made sure
to journal all bytes in the same sector of any modifications
and it pads journal files out to the next sector boundary so that
subsequent appends to that journal cannot damage prior records.
SQLite understands the sector size to be the value returned by the
xSectorSize method in the VFS.  The SQLite team has often referred
to the value returned by xSectorSize as the "blast radius" of a write,
since it expresses the range of bytes that might be damaged if a power
loss occurs during the write.
The default VFSes for unix and windows have always returned 512 as 
the sector size (or blast radius) for all versions of SQLite up to
and including version 3.7.9.

Newer disk drives have begun using 4096 byte sectors however.  Beginning
with SQLite version 3.7.10 (2012-01-16), 
the SQLite development team experimented with 
changes xSectorSize to report 4096 bytes as the blast radius.
This had the effect of increasing write overhead on
many databases.  For a database with a PRAGMA page_size of 1024
(a very common choice) making a change to a single page in the database
now requires SQLite to backup three other adjacent pages to the rollback
journal, whereas formerly it only had to backup the one page that was
changing.  In WAL mode, each transaction had to be padded out to the
next 4096-byte boundary in the WAL file, rather than the next 512-byte
boundary, resulting in thousands of extra bytes being written
per transaction.

The extra write overhead prompted a reexamination of assumptions about
powersafe overwrite.  With modern disk drives, the capacity has become
so large and the data density so great that a single sector is very
small and writing a single sector takes very little time.  We know that
disk drives can detect an impending power loss and continue
to operate for some small amount of time on residual energy because those
drives are able to park their heads before spinning down.  And
so if an impending power loss is detectable by the disk controller, it
seems reasonable that the controller will finish writing
whatever sector it is current working on when the imminent power loss 
is first detected, prior to parking the heads, as long as doing so
does not take too long, which it should not with
small and dense sectors.  Hence it seems reasonable
to assume powersafe overwrite for modern disks.  Indeed, BerkeleyDB has
made this assumption for decades, we are told.  Caution is advised
though. As Roger Binns noted on the SQLite developers mailing list:
"'poorly written' should be the main assumption about drive firmware."



Torn Pages

A torn page occurs when a database page is larger than a disk sector,
the database page is written to disk, but a power loss occurs prior to
all sectors of the database page being written.  Then, upon recovery, part of
the database page will have the old content while some other parts of the
page will have the new content.  Some database engines assume that 
page writes are atomic and hence a torn page is an unrecoverable error.


SQLite never assumes that database page writes are atomic,
regardless of the PSOW setting.(1)
And hence SQLite is always able to automatically recover from torn pages
induced by a crash.  Enabling PSOW does not decrease SQLite's ability
to recover from a torn page.

Changes In SQLite Version 3.7.10

The VFS for SQLite version 3.7.10 (2012-01-16)
adds a new device characteristic 
named SQLITE_IOCAP_POWERSAFE_OVERWRITE.  Database files that report this
characteristic are assumed to reside on storage systems that have the
powersafe overwrite property.
The default unix and windows VFSes now report
SQLITE_IOCAP_POWERSAFE_OVERWRITE if SQLite is compiled with
-DSQLITE_POWERSAFE_OVERWRITE=1 or they
make the legacy assumption that storage does not have the powersafe
overwrite property if compiled with
-DSQLITE_POWERSAFE_OVERWRITE=0.
For now, the default is for powersafe overwrite to be turned on, though
we may revisit this in the future and default it off.

The powersafe overwrite property for individual databases can be
specified as the database is opened using the "psow" query parameter
with a URI filename.  For example, to always assume powersafe
overwrite for a file (perhaps to ensure maximum write performance), 
open it as


   file:somefile.db?psow=1


Or to be extra safe with a database and to force SQLite to assume the
database lacks powersafe overwrite, open it using


   file:somefile.db?psow=0


There is also a new SQLITE_FCNTL_POWERSAFE_OVERWRITE opcode for
the sqlite3_file_control() that allows
an application to query the powersafe overwrite property for a database
file.


Notes:

SQLite never assumes atomic page writes in its default configurations.
But a custom VFS can set one of the 
SQLITE_IOCAP_ATOMIC bits in the result of the xDeviceCharacteristic()
method and then SQLite will assume that page writes are atomic.  The
application must supply a custom VFS to accomplish this, however, since
none of the standard VFSes will ever set any of the atomic bits in the
xDeviceCharacteristics() vector.


c2: Vulnerabilities
c5: cves.html
c4:
1. Executive Summary



CVEs about SQLite probably do not apply to your use of SQLite.


All historical vulnerabilities reported against SQLite require at least
one of these preconditions:


The attacker can submit and run arbitrary SQL statements.

The attacker can submit a maliciously crafted database file to the
application that the application will then open and query.



Few real-world applications meet either of these preconditions, and hence
few real-world applications are vulnerable, even if they use older
and unpatched versions of SQLite.


The SQLite development team fixes bugs promptly,
usually within hours of discovery.  New releases of SQLite
are issued if the bug seems likely to impact real-world
applications.


Grey-hat hackers are rewarded based on the number and severity of 
CVEs that they write.  This results in a proliferation of CVEs that
have minor impact, or no impact at all, but which make
exaggerated impact claims.


Very few CVEs written about SQLite are real vulnerabilities in the 
sense that they do not give any new capabilities to an attacker.
Consider:


    Almost all CVEs written against SQLite require the ability to
    inject and run arbitrary SQL.

    The advertised consequence of most CVEs is "denial of service",
    typically by causing a crash through a NULL pointer dereference or
    a division by zero, or similar.

    But if an attacker can already run
    arbitrary SQL, they do not need a bug to cause a denial of service.
    There are plenty of perfectly legal and valid SQL statements
    that will consume unlimited CPU, memory, and disk I/O in order
    to create a denial-of-service without requiring help from bugs.

    Hence, the mere fact that an attacker has a way to inject and run
    arbitrary SQL is in and of itself a denial-of-service attack.  That
    the arbitrary SQL might also tickle a bug in SQLite and cause a
    crash is not a new vulnerability.



The SQLite developers do not write CVEs.  Any CVEs you find on
SQLite are generated by third-parties, often without any input from the
core developers.  A common scenario is that someone will report a bug in
SQLite, which will promptly be fixed, then weeks later a CVE for that bug will
appear, unbeknownst to the developers.


You should not assume that a CVE about
SQLite contains authoritative information.
CVEs often contain inaccuracies.
The SQLite developers have attempted to add clarifications and
corrections to CVEs about SQLite.  



2. About CVEs

CVEs ("Common Vulnerabilities and Exposures") are reports of software
bugs that might allow a system to be hacked.  The idea
behind CVEs is sound.  They provide a common naming scheme whereby 
software bugs that might compromise information security can be easily
tracked.

While the original idea being CVEs is sound, the current processes for
creating and managing CVEs are inadequate.  There are countless grey-hat
hackers running fuzzers against a wide-variety of open-source software
products (SQLite as well as many others) and writing up CVEs against
any problems they find.  The grey-hats are rewarded, sometimes with
prestige and sometimes financially, by the number and severity of
the CVEs they write.  This incentive results in a proliferation
of CVEs which are often not well-vetted and which can have exaggerated
impact claims.  The quality-control procedures for CVEs are unable
to cope with this flood of inputs, making it difficult to correct
exaggerated, misleading, omitted, or inaccurate claims.

This is not to say that CVEs are useless.  CVEs do still (mostly)
report actual bugs.  But in most cases the bugs are not true vulnerabilities,
in the sense that they do not contribute to data loss or compromise
in and of themselves.
It is good that bugs are reported and fixed.  But not every bug is
accessible from every application.  In the case of SQLite, most of the
bugs reported by CVEs are inaccessible in most applications.  Upgrading
to the latest version of SQLite is always a good plan, but it need not
be an emergency just because an anonymous grey-hat on the internet
wrote up a CVE.

2.1. A separate SQL injection vulnerability is usually required


Other C-libraries that process complex structured inputs will
routinely be asked to deal with unvetted inputs from untrusted
sources.  Libraries like libjpeg, or libzip, or OpenSSL are
handed input streams that come directly from potentially hostile
agents.


But database engines like SQLite are usually not this way.
The SQL scripts that are passed into SQLite come from the
(trusted) application itself, not from an attacker.  Sometimes
applications contain bugs by which an external attacker can
trick the application into sending SQL of the attackers design
into the database engine.  This is a separate bug in the
application called an
SQL Injection
vulnerability.  Since SQL text is executable code, an
SQL Injection vulnerability is actually a special case of a
Remote
Code Execution (RCE) vulnerability.  An SQL Injection is perhaps not
quite as bad as other kinds of RCEs because,
while SQL is a powerful language, it is not as convenient
for crafting an exploit as Python or shell script or raw machine code.
Nevertheless, an SQL Injection is a serious problem.


Most CVEs written about SQLite assume that the attacker is
able to run arbitrary SQL scripts in SQLite.  In most applications,
this means that there must first be an SQL Injection vulnerability
that allows the attacker to inject the malicious SQL.


A few applications do allow untrusted SQL scripts received from
potentially hostile agents to be run direct in SQLite.  The main
example of this is the Chrome and Safari web browsers, which allow
an anonymous web page to run SQL using the WebSQL feature of Javascript.
This is done inside a sandbox with tightly controlled constraints on
resources, lest the SQL script try to soak up all available memory
or CPU cycles in a denial-of-service attack.  Chrome and Safari
have the infrastructure in place to allow a hostile agent to run
code which does not harm or compromise the rest of the machine.
They have to, as they also run Javascript which could, if not
tightly controlled, do even more damage than unrestrained SQL.
Apart from Chrome and Safari, no applications known to the
SQLite developers deliberately allows an anonymous remote agent
to run arbitrary SQL text.

However, most CVEs written against SQLite flippantly assume
that an attacker is free to run any arbitrary SQL in the database
engine.  So to a good approximation, this means most CVEs
written against SQLite really only apply to SQLite as it is
used in Chrome and Safari.  Or, in other words, most CVEs
for SQLite do not apply to you unless you are one of the
developers of Chrome or Safari.

2.2. Defense Against Dark Arts


Most applications can use SQLite without having to worry about
bugs in obscure SQL inputs.  If the application controls
the SQL, and the application is not deliberately trying to break
SQLite, then everything should just work.
It is not necessary to have the latest patched version of SQLite.
Any older version should work just fine.


However, there are some occasions where an application does need
to be able to safely run untrusted SQL. The SQLite developers work hard
to make SQLite safe for this purpose, though there are occasional
slip-ups.  It is good to keep up-to-date with the latest patches
in this case.  The separate defense against dark arts document
contains additional suggestions that can help prevent zero-day
attacks in cases where SQLite is given inputs that come directly
from untrusted sources. 

2.3. The SQLite Developer Policy Toward CVEs

SQLite developers fix all bugs in SQLite as soon as they are reported,
usually within a few hours.  The fixes are immediately available on the
public SQLite source tree.
If a bug seems like it might cause problems for existing applications,
a new patch release for SQLite will be issued.

However, the SQLite developers do not track CVEs.  There are 
various reasons for this:



The developers often do not find out about CVEs until long after the
bug is fixed.  You can see this by the fact that many CVEs reference the
bug fix in their initial report.


CVEs are a low-quality source of information about bugs in SQLite
that are likely to affect most applications.


Almost all bugs reported by CVEs are just bugs and not
true vulnerabilities.  Claiming that they are vulnerabilities is
stretching the meaning of the word "vulnerability" and the SQLite
developers do not wish to participate in that deception.


The developers have no editorial influence on the content of CVEs,
and they do not like to be controlled by groups in which they have
no voice.





3. Status Of Recent SQLite CVEs

Though the SQLite developers do not consider CVEs to be a reliable
source of information about bugs in SQLite, they recognize that many
groups, and especially small teams working at the bottom of tall
bureaucracies, sometimes need to track CVEs, whether they are useful
or not.  To aid in this chore, the following table of recent CVEs
affecting SQLite is provided.

If you notice new CVEs associated with SQLite that are not in
the table below, please bring them to the attention of the developers
on the SQLite Forum so they can
be added.




CVE Number
Fix
Comments





CVE-2022-35737

3.39.2(2022-07-21)
This bug is an array-bounds overflow.  The bug is only accessible when using some
  of the C-language APIs provided by SQLite.  The bug cannot be reached using SQL
  nor can it be reached by providing SQLite with a corrupt database file.
  The bug only comes up when very long string inputs (greater than 2 billion bytes
  in length) are provided as arguments to a few specific C-language interfaces,
  and even then only under special circumstances.



CVE-2022-24854

Not a bug
This CVE describes a bug in an application that uses SQLite, not in SQLite itself.
  SQLite is doing everything correctly.  The application grants users the ability to
  run SQL statements, using SQLite, that can leak or change information that those users
  should not normally have access to.  This is purely an application bug.  It does not
  describe a malfunction or vulnerability in SQLite.



CVE-2022-21227

Not a bug
This CVE describes a bug in a third-party packages that provides a binding
  for SQLite to Node.js.  The bug reported is in the third-party Node.js binding,
  not in SQLite itself.  Do not be confused by the use of the word "SQLite" in the
  ambiguously-worded CVE description.



CVE-2021-45346

Not a bug
This CVE is misinformation.  See the discussion around
  SQLite forum post 53de8864ba114bf.



CVE-2021-42169

Not a bug
This CVE has nothing whatsoever to do with SQLite.  It is about a bug in
  application that happens to use SQLite.  Since SQLite is mentioned in the 
  CVE description, the CVE is included here to emphasize that
  this is not an SQLite bug.



CVE-2021-36690

Bug not in the SQLite library
This bug is not in the SQLite core library, but rather in an
  
  experimental extension that is used to implement the
  .expert command in the CLI.  The code that contains the bug
  does not appear in standard SQLite builds, though it
  is included in the sqlite3.exe command-line tool.
  Applications must link against the extra source code files that
  implement the extension and take other deliberate actions to
  activate the extension before the troublesome code can be run.
  For the rare application that uses the troublesome extension,
  the consequence of this bug is that malicious SQL can cause a
  NULL pointer deference and denial of service.
(details)



CVE-2021-28305

Not a bug
This is not a bug in SQLite.  The bug is in a third-party application that
  uses SQLite.  SQLite is mentioned by name in the CVE description,
  however, so we have included the CVE in the list.



CVE-2021-23404

Not a bug
This is not a bug in SQLite.  The bug is in a third-party application that
  uses SQLite and includes "sqlite" in its name.  This CVE is included on the
  list because it mentions SQLite even though the bug has nothing to do
  with SQLite.



CVE-2021-20227

3.34.1(2021-01-20)
Malicious SQL statement causes read-after-free.  No harm can come of this
  particular read-after-free instance, as far as anyone knows.  The bug is
  undetectable without a memory sanitizer. The CVE
  claims that this bug is an RCE - a Remote Code Execution
  vulnerability, but that claim is incorrect.
  The RCE claim is misinformation.
(details)



CVE-2021-20223

3.34.0(2020-12-01)
The problem identified by this CVE is not a vulnerability. 
  It is a malfunction. A coding error causes FTS5
  to sometimes return inconsistent and incorrect results under obscure circumstances,
  but no memory errors occur.
(details)



CVE-2020-15358

3.32.3(2020-06-18)
Malicious SQL statement causes an read past the end of a heap buffer.
(details)



CVE-2020-13871

3.32.3(2020-06-18)
Malicious SQL statement causes a read-only use-after-free memory error.
(details)



CVE-2020-13632

3.32.0(2020-05-22)
Malicious SQL statement causes a read of a NULL pointer in the
  matchinfo() SQL function of the FTS3 extension, resulting in
  denial of service.
(details)



CVE-2020-13631

3.32.0(2020-05-22)
Malicious SQL statement (an ALTER TABLE that tries to rename a
  virtual table into one of its own shadow tables)
  causes an infinite loop and denial of service.
(details)



CVE-2020-13630

3.32.0(2020-05-22)
Malicious SQL statement causes a read-only use-after-free,
  possibly resulting in a incorrect output from the snippet()
  SQL function of the FTS3 extension.  There is no known
  way to exfiltrate data or crash the application using this bug.
(details)



CVE-2020-13435

3.32.1(2020-05-25)
Malicious SQL statement causes a read access to a NULL pointer and
  denial of service.
(details)



CVE-2020-13434

3.32.1(2020-05-25)
Malicious SQL statement involving the printf() SQL function results
  in an integer overflow which can overwrite the stack with over 2
  billion bytes of 0x30 or 0x20 (ASCII '0' or ' ').
  Even though this is a stack overwrite, there is no known way to
  redirect control or otherwise escalate the level of harm.
  This is a denial-of-service attack only.
(details)



CVE-2020-11656

3.32.0(2020-05-22)
Malicious SQL statement causes read-only use-after-free of memory allocation
  if SQLite is compile with -DSQLITE_DEBUG.  Does not affect release
  builds.
(details)



CVE-2020-11655

3.32.0(2020-05-22)
Malicious SQL statement causes a read using an uninitialized pointer
  and denial-of-service.
(details)



CVE-2020-9327

3.32.0(2020-05-22)
Malicious SQL statement causes a read using an uninitialized pointer
  and denial-of-service
(details)



CVE-2020-6405

3.31.0(2020-01-22)
Malicious SQL statement causes a NULL pointer dereference and
  denial-of-service
(details)



CVE-2019-20218

3.31.0(2020-01-22)
Malicious SQL statement causes an uninitialized pointer read and
  denial-of-service.
(details)



CVE-2019-19959

3.31.0(2020-01-22)
Malicious SQL statement causes a NULL pointer dereference
  in the Zipfile virtual table extension and
  denial-of-service.  This is only possible when the optional
  Zipfile virtual table extension is deployed, which is not
  the case in default builds.
(details)



CVE-2019-19926

3.31.0(2020-01-22)
Malicious SQL statement causes an uninitialized pointer read and
  denial-of-service.
(details)



CVE-2019-19925

3.31.0(2020-01-22)
Malicious SQL statement causes a NULL pointer dereference and
  in the Zipfile virtual table extension and
  denial-of-service.  This is only possible when the optional
  Zipfile virtual table extension is deployed, which is not
  the case in default builds.
(details)



CVE-2019-19924

3.31.0(2020-01-22)
Malicious SQL statement causes a uninitialized pointer reference and
  denial-of-service.
(details)



CVE-2019-19923

3.31.0(2020-01-22)
Malicious SQL statement causes a NULL pointer dereference and
  denial-of-service.
(details)



CVE-2019-19317

3.31.0(2020-01-22)
This CVE identifies a bug in a development check-in of
  SQLite.  The bug never appeared in any official SQLite release.
(details)


c2: Pointer Passing Interfaces
c5: bindptr.html
c4:
1. Overview


Three new "_pointer()" interfaces were added to SQLite 3.20.0 (2017-08-01):

 sqlite3_bind_pointer(),
 sqlite3_result_pointer(), and
 sqlite3_value_pointer().


Questions and confusion quickly arose
on the mailing lists about the purpose behind these new interfaces,
why they were introduced, and what problem they solve.  This essay attempts
to answer those questions and clear up the confusion.

2. A Brief History Of Pointer Passing In SQLite


It is sometimes convenient for SQLite extensions to
communicate non-SQL values between subcomponents or between the extension
and the application.  Some examples:



In the FTS3 extension, the 
MATCH operator (which does the full-text search)
needs to communicate details of matching entries 
to the snippet(), offsets(),
and matchinfo() functions so that those functions can convert the details
of the match into useful output.


In order for an application to 
add new extensions to FTS5, such as new tokenizers, 
the application needs a pointer to the "fts5_api" object.


In the CARRAY extension, the application needs to tell the 
extension the
location of a C-language array that contains the data for the table-valued
function that the extension implements.



The traditional way of communicating this information was to transform a
C-language pointer into a BLOB or a 64-bit integer, then move that BLOB or
integer through SQLite using the usual interfaces like
sqlite3_bind_blob(), sqlite3_result_blob(), sqlite3_value_blob() or
the integer equivalents.

2.1. Upping The Threat Level


Passing around pointers as if they were integers or BLOBs is easy,
effective, and works well in an environment where the application
components are all friendly toward one another.  However, passing pointers
as integers and BLOBs allows hostile SQL text to forge invalid pointers that
can carry out mischief.


For example, the first argument to the snippet() function is supposed to
be a special column of the FTS3 table that contains a pointer to an fts3cursor
object that contains information about the current full text search match.
That pointer was formerly passed as a BLOB.  
For example, if the FTS3 table is named "t1" and has a column named "cx",
one might write:

SELECT snippet(t1) FROM t1 WHERE cx MATCH $pattern;



But if a hacker is able to run arbitrary SQL, he might run a slightly
different query, like this:

SELECT hex(t1) FROM t1 WHERE cx MATCH $pattern;



Because the pointer is passed in the t1 column of the t1
table as a BLOB (in older versions of SQLite), such a query would have 
shown the value of the
pointer in hex.  The attacker could then modify that pointer to try to
get the snippet() function to modify memory in some other part of 
the application address space instead of the fts3cursor object it 
was supposed to be operating on:

SELECT snippet(x'6092310100000000') FROM t1 WHERE cx MATCH $pattern;



Historically, this was not considered a threat.  The argument was that if
a hostile agent is able to inject arbitrary SQL text into the application,
then that agent is already in full control of the application, so
letting the hostile agent forge a pointer does not give the agent
any new capability.


For most cases, it is true that potential attackers have no way of injecting
arbitrary SQL, and so most uses of SQLite are immune to the attack above.
But there are some notable exceptions.  To wit:



The WebSQL interface
to webkit allowed any webpage to run arbitrary SQL in the browser
for Chrome and Safari.  That arbitrary SQL was supposed to be run inside
a sandbox where it could do no harm even if exploited, but that sandbox
turned out to be less secure than people supposed.  In the spring of 2017, 
one team of hackers was able to root an iMac using a long sequence of 
exploits, one of which involved corrupting the pointers passed as BLOB 
values to the snippet() FTS3 function of an SQLite database running via
the WebSQL interface inside of Safari.


On Android, we are told, there are many services that will blindly 
run arbitrary SQL that is passed to them by untrustworthy apps
that have been downloaded from dodgy corners of the internet.
Android services are suppose to be more guarded about running SQL
from unvetted sources.  This author does not have any specific examples
to the contrary, but he has heard rumors that they exist.  Even if
all Android services are more careful and properly vet all the SQL
they run, it would be difficult to audit them
all in order to verify that they are safe.  Hence, security-minded people
are keen to ensure that no exploits are possible by passing arbitrary
SQL text.


The Fossil version control system (designed
and written for the purpose of supporting SQLite development) allows
mildly trusted users to enter arbitrary SQL for generating trouble-ticket
reports.  That SQL is sanitized using the
sqlite3_set_authorizer() interface, and no exploits have ever been
found.  But this is an example of potentially hostile agents being able to
inject arbitrary SQL into the system.


2.2. Preventing Forged Pointers


The first attempt at closing security gaps in pointer passing was to
prevent pointer values from being forged.  This was accomplished by
having the sender attach a subtype to each pointer using
sqlite3_result_subtype() and having the receiver verify that subtype
using sqlite3_value_subtype() and reject pointers that had an incorrect
subtype.  Since there is no way to attach a subtype to a result using
pure SQL, this prevents pointers from being forged using SQL.  The only
way to send a pointer is to use C code.  If an attacker can set a subtype,
then he is also able to forge a pointer without the help of SQLite.


Using subtypes to identify valid pointers prevented the WebSQL exploit.
But it turned out to be an incomplete solution.



2.3. Pointer Leaks


The use of subtypes on pointers prevented pointer forgery using
pure SQL.  But subtypes do nothing to prevent an attacker from reading
the values of pointers.  In other words, subtypes on pointer values
prevent attacks using SQL statements like this:

SELECT snippet(x'6092310100000000') FROM t1 WHERE cx MATCH $pattern;



The BLOB argument to snippet() does not have the correct subtype, so the
snippet function ignores it, makes no changes to any data structures,
and harmlessly returns NULL.


But the use of subtypes does nothing to prevent the value of a
pointer from being read using SQL code like this:

SELECT hex(t1) FROM t1 WHERE cx MATCH $pattern;



What harm could come of that, you ask?  The SQLite developers (including
this author) wondered the same thing.  But then security researchers
pointed out that knowledge of pointers can help attackers to circumvent
address-space randomization defenses.  This is called a "pointer leak".
A pointer leak is not itself a vulnerability, but it can aid an attacker
in effectively exploiting other vulnerabilities.

3. The New Pointer-Passing Interfaces


Allowing extension components to pass private information to one another
securely and without introducing pointer leaks requires new interfaces:


sqlite3_bind_pointer(S,I,P,T,D) &rarr;
Bind pointer P of type T to the I-th parameter of prepared statement S.
D is an optional destructor function for P.
sqlite3_result_pointer(C,P,T,D) &rarr;
Return pointer P of type T as the argument of function C.
D is an optional destructor function for P.
sqlite3_value_pointer(V,T) &rarr;
Return the pointer of type T associated with value V, or if V has no
associated pointer, or if the pointer on V is of a type different from
T, then return NULL.



To SQL, the values created by sqlite3_bind_pointer() and 
sqlite3_result_pointer() are indistinguishable from NULL.  An
SQL statement that tries to use the hex() function to read the
value of a pointer will get an SQL NULL answer.  The only way to
discover whether or not a value has an associated pointer is to
use the sqlite3_value_pointer() interface with the appropriate
type string T. 


Pointer values read by sqlite3_value_pointer()
cannot be generated by pure SQL.  Hence, it is not possible for SQL to
forge pointers.


Pointer values generated by sqlite3_bind_pointer() and
sqlite3_result_pointer() cannot be read by pure SQL.
Hence, it is not possible for SQL to leak the value of pointers.


In this way the new pointer-passing interface seems to solve all of the
security problems associated with passing pointer values from one
extension to another in SQLite.



3.1. Pointer Types


The "pointer type" in the last parameter to sqlite3_bind_pointer(),
sqlite3_result_pointer(), and sqlite3_value_pointer() is used to prevent
pointers intended for one extension from being redirected to a different
extension.  For example, without the use of pointer types, an attacker 
could still get access to pointer information in a system that included 
both the FTS3 and the CARRAY extension using SQL like this:

SELECT ca.value FROM t1, carray(t1,10) AS ca WHERE cx MATCH $pattern



In the statement above, the FTS3 cursor pointer generated by the
MATCH operator is send into the carray() table-valued function instead
of its intended recipient snippet().  The carray() function treats the
pointer as a pointer to an array of integers and returns each integer
one by one, thus leaking the content of the FTS3 cursor object.  Since
the FTS3 cursor object contains pointers to other objects, the statement
above would be a pointer leak.


Except, the statement above does not work, thanks to pointer types.
The pointer generated by the MATCH operator has a type of "fts3cursor"
but the carray() function expects to receives a pointer of type "carray".
Because the pointer type on the sqlite3_result_pointer() does not match
the pointer type on the sqlite3_value_pointer() call, 
sqlite3_value_pointer() returns NULL in carray() and thus signals
the CARRAY extension that it has been passed an invalid pointer.

3.1.1. Pointer types are static strings


Pointer types are static strings, which ideally should be string literals
embedded directly in the SQLite API call, not parameters passed in from
other functions.  Consideration was given to using integer values as
the pointer type, but static strings provides a much larger name space
which reduces the chance of accidental type-name collisions between
unrelated extensions.


By "static string", we mean a zero-terminated array of bytes that is
fixed and unchanging for the life of the program.  In other words, the
pointer type string should be a string constant.
In contrast, a "dynamic string" is a zero-terminated array of bytes
that is held in memory allocated
from the heap, and which must be freed to avoid a memory leak.
Do not use dynamic strings as the pointer type string.


Multiple commentators have expressed a desire to use dynamic strings
for the pointer type, and to have SQLite take ownership of the type strings
and to automatically free the type string
when it has finished using it.  That design is rejected for the
following reasons:



The pointer type is not intended to be flexible and dynamic.  The
pointer type is intended to be a design-time constant.  Applications
should not synthesize pointer type strings at run-time.  Providing
support for dynamic pointer type strings would lead developers
to misuse the pointer-passing interfaces by creating run-time
synthesized pointer type strings.  Requiring the pointer type strings
to be static encourages developers to do the right thing by choosing
fixed pointer type names at design-time and encoding those names
as constant strings.


All string values at the SQL level in SQLite are dynamic strings.  
Requiring type strings to be static makes it difficult to 
create an application-defined SQL function that
can synthesize a pointer of an arbitrary type.  We do not want users
to create such SQL functions, since such functions would compromise the
security of the system.  Thus, the requirement to use static strings
helps to defend that the integrity of the pointer-passing interfaces against
ill-designed SQL functions.  The static string requirement is not
a perfect defense, since a sophisticated programmer can code around
it, and a novice program can simply take the memory leak.  But by
stating that the pointer type string must be static, we hope to encourage
developers who might otherwise use a dynamic string for the pointer type
to think more carefully about the problem and avoid introducing
security issues.


Having SQLite take ownership of the type strings would impose a performance
cost on all applications, even applications that do not use the 
pointer-passing interfaces.  SQLite passes values around as instances
of the sqlite3_value object.  That object has a destructor, which because
of the fact that sqlite3_value objects are used for nearly everything, is
invoked frequently.  If the destructor needs to check to see if there is
a pointer type string that needs to be freed, that is a few extra CPU
cycles that need to be burned on each call to the destructor.  Those
cycles add up.  We would be willing to bear the cost of the extra CPU
cycles if pointer-passing was a commonly used programming paradigm, but
pointer-passing is rare, and so it seems unwise to impose a run-time cost
on billions and billions of applications that do not use pointer passing
just for convenience of a few applications that do.



If you feel that you need dynamic pointer type strings in your application,
that is a strong indicator that you are misusing the pointer-passing interface.
Your intended use may be unsafe.
Please rethink your design.  Determine if you really need to be passing
pointers through SQL in the first place.  Or perhaps find a different
mechanism other than the pointer-passing interfaces described by this
article.

3.2. Destructor Functions


The last parameter to the sqlite3_bind_pointer() and
sqlite3_result_pointer() routines is a pointer to a procedure
used to dispose of the P pointer once SQLite has finished with it.
This pointer can be NULL, in which case no destructor is called.


When the D parameter is not NULL, that means that ownership of the
pointer is being transferred to SQLite.  SQLite will take responsibility
for freeing resources associated with the pointer when it has finished
using the pointer.  If the D parameter is NULL, that means that ownership
of the pointer remains with the caller and the caller is responsible for
disposing of the pointer.


Note that the destructor function D is for the pointer value P, not for
the type string T.  The type string T should be a static string with an
infinite lifetime.


If ownership of the pointer is passed into SQLite by providing a
non-NULL D parameter to sqlite3_bind_pointer() or sqlite3_result_pointer()
then the ownership remains with SQLite until the object is destroyed.
There is no way to transfer ownership out of SQLite and back into the
application again.

4. Restrictions On The Use of Pointer Values


The pointers that piggy-back on SQL NULL values using the
sqlite3_bind_pointer(), sqlite3_result_pointer(), and
sqlite3_value_pointer() interface are transient and ephemeral.
The pointers are never written into the database.  The pointers
will not survive sorting.  The latter fact is why there is no
sqlite3_column_pointer() interface, since it is impossible to
predict whether or not the query planner will insert a sort operation
prior to returning a value from a query, so it would be impossible to
know if a pointer value inserted into a query by
sqlite3_bind_pointer() or sqlite3_result_pointer() would survive
through to the result set.


Pointer values must flow directly from their producer into their
consumer, with no intermediate operators or functions.  Any transformation
of a pointer value destroys the pointer and transforms the value into
an ordinary SQL NULL.


Both the pointer and the pointer type parameter to the
sqlite3_bind_pointer() and sqlite3_result_pointer() interfaces
are "owned" by the caller.  In other words, the caller is responsible
for ensuring that both values remain valid until after the last access
via sqlite3_value_pointer().

5. Summary

Key take-aways from this essay:



The internet is an increasingly hostile place.  These day, developers
should assume that attackers will find a way to execute arbitrary SQL 
in an application.
Applications should be designed to prevent the execution of arbitrary
SQL from escalating into a more severe exploit.


A few SQLite extensions benefit from passing pointers:

The FTS3 MATCH operator passes pointers into snippet(),
    offsets(), and matchinfo().
The carray table-valued function needs to accept a pointer to
    an array of C-language values from the application.
The remember() extension
    needs a pointer to a C-language integer variable in which to remember
    the value it passes.
Applications need to receive a pointer to the "fts5_api" object in order
    to add extensions, such as custom tokenizers, to the FTS5 extension.



Pointers should never be exchanged by encoding them as some other
SQL datatype, such as integers or BLOBs.  Instead, use the interfaces
designed to facilitate secure pointer passing:
sqlite3_bind_pointer(), sqlite3_result_pointer(), and
sqlite3_value_pointer().


The use of pointer-passing is an advanced technique that should be used
infrequently and cautiously.  Pointer-passing should not be
used haphazardly or carelessly.  Pointer-passing is a sharp tool 
that can leave deep scars if misused.


The "pointer type" string which is the last parameter to each of the
pointer-passing interfaces should be a distinct, application-specific
string literal that appears directly in the API call.  The pointer type
should not be a parameter passed in from a higher-level function.


c2: SQLite Archive Files
c5: sqlar.html
c4:
1. Introduction


An "SQLite Archive" is a file container similar to a 
ZIP archive or
Tarball but
based on an SQLite database.


An SQLite Archive is an ordinary SQLite database file that contains the
following table as part of its schema:

CREATE TABLE sqlar(
  name TEXT PRIMARY KEY,  -- name of the file
  mode INT,               -- access permissions
  mtime INT,              -- last modification time
  sz INT,                 -- original file size
  data BLOB               -- compressed content
);



Each row of the SQLAR table holds the content of a single file.
The filename (the full pathname relative to the root of the archive)
is in the "name" field.
The "mode" field is an integer which is the unix-style access permissions
for the file.  "mtime" is the modification time of the file in seconds
since 1970.  "sz" is the original uncompressed size of the file.
The "data" field contains the file content.  The content is usually
compressed using Deflate, though not always.  If the
"sz" field is equal to the size of the "data" field, then the content
is stored uncompressed.



1.1. Database As Container Object


An SQLite Archive is one example of a more general
idea that an SQLite database can behave as a container object holding
lots of smaller data components.


With client/server databases like PostgreSQL or Oracle, users and
developers tend to think of the database as a service or a "node", not
as an object.  This is because the database content is spread out across
multiple files on the server, or possibly across multiple servers in a
service cluster.  One cannot point to a single file or even a single
directory and say "this is the database".


SQLite, in contrast, stores all content in a single file on disk.
That single file is something you can point to and say
"this is the database".  It behaves as an object.
An SQLite database file can be copied, renamed, sent as an
email attachment, passed as the argument a POST HTTP request,
or otherwise treated as other data object such as an image,
document, or media file.


Studies show that many applications already use
SQLite as a container object.  For example,
Kennedy
(no relation to the SQLite developer) reports that 14% of
Android applications never write to their SQLite databases.  It is
believed that these applications are downloading entire databases
from the cloud and then using the information locally as needed.  In other
words, the applications are using SQLite not so much as a database but as
a queryable wire-transfer format.

1.2. Applications Using SQLite Archives


The Fossil Distributed Version Control system
provides users with the option to download check-ins as either Tarballs,
ZIP Archives, or SQLite Archives.
2. Advantages Of SQLite Archives



An SQLite Archive is flexible.
ZIP Archives and Tarballs are limited to storing only files.  An
SQLite Archive stores files plus whatever other tabular
and/or relational data seems useful to the application.


An SQLite Archive is transactional.
Updates are atomic and durable, even if there are crashes
or power losses in the middle of the update.
Readers see a consistent and unchanging version of the content even
is some other process is simultaneously updating the archive.


An SQLite Archive can be updated incrementally.
Individual files can be added or removed or replaced without having
to rewrite the entire archive.


An SQLite Archive can be queried using a high-level query language (SQL).
Some examples:

 What is the total size of all files in the archive whose names
     end in ".h" or ".cpp"?
 What percentage of the files are compressed by less than 25%?
 How many executable files are in the archive?

Questions like these (and countless others) can be answered without
having to uncompress or extract any content.


Applications that already use SQLite for other purposes can easily
add support for SQLite Archives using a small extension
(https://sqlite.org/src/file/ext/misc/sqlar.c) to handle the compression
and decompression of content.  Even this tiny extension can be omitted
if the files in the archive are uncompressed.  In contrast, supporting
ZIP Archives and/or Tarballs requires either separate libraries or
lots of extra custom code, or sometimes both.


An SQLite Archive can work around firewall-imposed censorship.
For example, certain file types that are considered "dangerous" 
(examples: DLLs) will be
blocked by Gmail
and probably many other email services and firewalls, even if the
files are wrapped inside a ZIP Archive or Tarball.
But these firewalls usually do not (yet) know about SQLite Archives and
so content can be put inside an SQLite Archive to evade censorship.


3. Disadvantages Of SQLite Archives



The SQLite Archive is a relatively new format.  It was first described in
in 2014.  ZIP Archives and Tarballs, on the other hand, have been around
for decades and are well-entrenched as standard formats.  Most programmers
know what a ZIP Archive or Tarball is, but if you say "SQLite Archive" you
are more likely to get a reply of "What?"  Tooling to process ZIP Archives
and Tarballs is more likely to be installed on stock computers.


Since an SQLite database is a more general format (it is designed to do
much more than simply store a bunch of files) it is not as compact as either
the ZIP Archive or Tarball formats.  An SQLite Archive is usually about 1%
larger than the equivalent ZIP Archive.  Tarballs are compressed as a single
unit rather than compressing each file separately as is done by both
SQLite and ZIP Archives.  For these reason, Tarballs tend to be smaller
than either ZIP or SQLite Archives.

As an example, the following table show the relative sizes for an
SQLite Archive, a ZIP Archive, and a Tarball of the 1,743 files
in the SQLite 3.22.0 source tree:

  SQLite Archive10,754,048
  ZIP Archive (using Info-ZIP 3.0)10,662,365
  ZIP Archive (using zipfile)10,390,215
  Tarball&#x2007;9,781,109



An SQLite Archive supports only the Deflate compression
method.  Tarballs and ZIP Archive support a wider assortment of
compression methods.




4. Managing An SQLite Archive From The Command-Line


The recommended way of creating, updating, listing, and extracting
an SQLite Archive is to use the sqlite3.exe command-line shell 
for SQLite version 3.23.0 (2018-04-02) or later.  This CLI
supports the -A command-line option that allows easy management
of SQLite Archives.
The CLI for SQLite version 3.22.0 (2018-01-22) has the
.archive command for managing SQLite Archives, but that requires
interacting with the shell.


To list all of the files in an SQLite Archive named "example.sqlar"
using one of these commands:

sqlite3 example.sqlar -At
sqlite3 example.sqlar -Atv



To extract all files from an SQLite Archive named "example.sqlar":

sqlite3 example.sqlar -Ax



To create a new SQLite Archive named "alltxt.sqlar" containing all *.txt
files in the current directory:

sqlite3 alltxt.sqlar -Ac *.txt



To add or update files in an existing SQLite Archive:

sqlite3 example.sqlar -Au *.md



For usage hints and a summary of all options, simply give the CLI
the -A option with no additional arguments:

sqlite3 -A



All of these commands work the same way if the filename argument is
is a ZIP Archive instead of an SQLite database.

4.1. Other command-line tools


Just as there is the "zip" program to manage ZIP Archives, and the
"tar" program to manage Tarballs, the 
"sqlar" program exists to manage SQL Archives.
The "sqlar" program is able to create a new SQLite Archive, list the
content of an existing archive, add or remove files from the archive,
and/or extract files from the archive.
A separate "sqlarfs" program is able to mount the SQLite Archive as
a Fuse Filesystem.

5. Managing SQLite Archives From Application Code


Applications can easily read or write SQLite Archives by linking against
SQLite and including the 
ext/misc/sqlar.c extension
to handle the compression and decompression.  The sqlar.c extension
creates two new SQL functions.


sqlar_compress(X)

The sqlar_compress(X) function attempts to compress a copy of the
string or blob X using the Default algorithm and
returns the result as a blob.  If the input X is incompressible, then
a copy of X is returned.  This routine is used when inserting content
into an SQLite Archive.
sqlar_uncompress(Y,SZ)

The sqlar_uncompress(Y,SZ) function will undo the compression accomplished
by sqlar_compress(X).  The Y parameter is the compressed content (the output
from a prior call to sqlar_compress()) and SZ is the original uncompressed
size of the input X that generated Y.  If SZ is less than or equal to the
size of Y, that indicates that no compression occurred, and so
sqlar_uncompress(Y,SZ) returns a copy of Y.  Otherwise, sqlar_uncompress(Y,SZ)
runs the Inflate algorithm on Y to uncompress it and restore it to its
original form and returns the uncompressed content.
This routine is used when extracting content from an SQLite Archive.



Using the two routines above, it is simple for applications to insert
new records into or extract existing records from an SQLite Archive.
Insert a new into an SQLite Archive using code like this:

INSERT INTO sqlar(name,mode,mtime,sz,data)
 VALUES ($name,$mode,strftime('%s',$mtime),
         length($content),sqlar_compress($content));



Extract an entry from the SQLite Archive using code like this:

SELECT name, mode, datetime(mtime,'unixepoch'), sqlar_uncompress(data,sz)
  FROM sqlar
 WHERE ...;



The code above is for the general case.  For the special case of an
SQLite Archive that only stores uncompressed or uncompressible content
(this might come up, for example, in an SQLite Archive that stores only
JPEG, GIF, and/or PNG images) then the content can be inserted into
and extracted from the database without using the sqlar_compress()
and sqlar_uncompress() functions, and the sqlar.c extension is not
required.


c2: LoC Recommended Storage Format
c5: locrsf.html
c4:
1. Overview








SQLite is a
Recommended Storage Format
for datasets according to the
US Library of Congress.
Further information:




https://www.loc.gov/preservation/digital/formats/fdd/fdd000461.shtml#local

https://www.loc.gov/preservation/resources/rfs/data.html



As of this writing (2018-05-29) the only other recommended storage formats
for datasets are XML, JSON, and CSV.

1.1. What Is A Recommended Storage Format?


Recommended storage formats are formats which, in the opinion of the
preservationists at the Library of Congress, maximizes the chance of
survival and continued accessibility of digital content.
When selecting recommended storage formats, the following criteria are
considered (quoting from the LOC website):


Disclosure.
Degree to which complete specifications and tools for validating 
technical integrity exist and are accessible to those creating and
sustaining digital content.
A spectrum of disclosure levels can be observed for digital formats.
What is most significant is not approval by a recognized standards body,
but the existence of complete documentation.
Adoption.
Degree to which the format is already used by the primary creators,
disseminators, or users of information resources. 
This includes use as a master format, for delivery to end users, 
and as a means of interchange between systems.
Transparency.
Degree to which the digital representation is open to direct analysis 
with basic tools, such as human readability using a text-only editor.
Self-documentation.
Self-documenting digital objects contain basic descriptive, 
technical, and other administrative metadata.
External Dependencies.
Degree to which a particular format depends on particular hardware,
operating system, or software for rendering or use and the predicted
complexity of dealing with those dependencies in future technical 
environments.
Impact of Patents.
Degree to which the ability of archival institutions to sustain content 
in a format will be inhibited by patents.
Technical Protection Mechanisms.
Implementation of mechanisms such as encryption that prevent the 
preservation of content by a trusted repository.


c2: Measuring and Reducing CPU Usage in SQLite
c5: cpu.html
c4:
1. Overview

The graph below shows the number of CPU cycles used by SQLite on a
standard workload, for versions of SQLite going back about 10 years.
Recent versions of SQLite use about one third as many the CPU cycles
compared to older versions.


This article describes how the SQLite developers measure CPU usage,
what those measurements actually mean, and the techniques used by
SQLite developers on their continuing quest to further reduce the
CPU usage of the SQLite library.






Measured using cachegrind on Ubuntu 16.04 on x64 with gcc 5.4.0 and -Os.



2. Measuring Performance

In brief, the CPU performance of SQLite is measured as follows:


 Compile SQLite in an as-delivered configuration, without any special
     telemetry or debugging options.
 Link SQLite against a test program that runs approximately 30,000
     SQL statements representing a typical workload.
 Count the number of CPU cycles consumed using
     cachegrind.


2.1. Compile Options

For performance measurement, SQLite is compiled in approximately the same
way as it would be for use in production systems.  The compile-time configuration
is "approximate" in the sense that every production use of SQLite is 
different. Compile-time options used by one system are not necessarily
the same as those used by others.  The key point is that options that 
significantly impact the generated machine code are avoided.  For example,
the -DSQLITE_DEBUG option is omitted because that option inserts thousands
of assert() statements in the middle of performance critical sections of the
SQLite library.  The -pg option (on GCC) is omitted because it causes the
compiler to emit extra probabilistic performance measuring code which interferes
with actual performance measurements.


For performance measurements,
the -Os option is used (optimize for size) rather than -O2 because the
-O2 option creates so much code movement that it is difficult to associate
specific CPU instructions to C source code lines.

2.2. Workload


The "typical" workload is generated by the
speedtest1.c
program in the canonical SQLite source tree.  This program strives to
exercise the SQLite library in a way that is typical of real-world
applications.  Of course, every application is different, and so
no test program can exactly mirror the behavior of all applications.


The speedtest1.c program is updated from time to time as the SQLite
developers' understanding of what constitutes "typical" usage evolves.


The 
speed-check.sh shell
script, also in the canonical source tree, is used to run the speedtest1.c
program.  To replicate the performance measurements, collect the following
files into a single directory:

 the "speed-check.sh" script,
 the "speedtest1.c" test program, and
 the SQLite amalgamation source files "sqlite3.c" and
     "sqlite3.h"


Then run "sh speed-check.sh trunk".


2.3. Performance Measurement


Cachegrind is used to
measure performance because it gives answers that are repeatable to 
7 or more significant digits.  In comparison, actual (wall-clock)
run times are scarcely repeatable beyond one significant digit.



2.4. Microoptimizations


The high repeatability of cachegrind allows the SQLite developers to
implement and measure "microoptimizations".  A microoptimization is
a change to the code that results in a very small performance increase.
Typical micro-optimizations reduce the number of CPU cycles by 0.1% or
0.05% or even less.  Such improvements are impossible to measure with
real-world timings.  But hundreds or thousands of microoptimizations
add up, resulting in measurable real-world performance gains.

3. Performance Measurement Workflow


As SQLite developers edit the SQLite source code, they run the
speed-check.sh
shell script to track the performance impact of changes.  This
script compiles the speedtest1.c program, runs it under cachegrind,
processes the cachegrind output using the
cg_anno.tcl TCL
script, then saves the results in a series of text files.
Typical output from the speed-check.sh script looks like this:


==8683== 
==8683== I   refs:      1,060,925,768
==8683== I1  misses:       23,731,246
==8683== LLi misses:            5,176
==8683== I1  miss rate:          2.24%
==8683== LLi miss rate:          0.00%
==8683== 
==8683== D   refs:        557,686,925  (361,828,925 rd   + 195,858,000 wr)
==8683== D1  misses:        5,067,063  (  3,544,278 rd   +   1,522,785 wr)
==8683== LLd misses:           57,958  (     16,067 rd   +      41,891 wr)
==8683== D1  miss rate:           0.9% (        1.0%     +         0.8%  )
==8683== LLd miss rate:           0.0% (        0.0%     +         0.0%  )
==8683== 
==8683== LL refs:          28,798,309  ( 27,275,524 rd   +   1,522,785 wr)
==8683== LL misses:            63,134  (     21,243 rd   +      41,891 wr)
==8683== LL miss rate:            0.0% (        0.0%     +         0.0%  )
   text	   data	    bss	    dec	    hex	filename
 523044	   8240	   1976	 533260	  8230c	sqlite3.o
 220507 1007870 7769352 sqlite3.c


The important parts of the output (the parts that the developers pay
the most attention to) are shown in red.
Basically, the developers want to know the size of the compiled SQLite
library and how many CPU cycles were needed to run the performance test.

The output from the 
cg_anno.tcl script
shows the number of CPU cycles spent on each line of code.
The report is approximately 80,000 lines long.  The following is a brief
snippet taken from the middle of the report to show what it looks like:


         .  SQLITE_PRIVATE int sqlite3BtreeNext(BtCursor *pCur, int *pRes){
         .    MemPage *pPage;
         .    assert( cursorOwnsBtShared(pCur) );
         .    assert( pRes!=0 );
         .    assert( *pRes==0 || *pRes==1 );
         .    assert( pCur-&gt;skipNext==0 || pCur-&gt;eState!=CURSOR_VALID );
   369,648    pCur-&gt;info.nSize = 0;
   369,648    pCur-&gt;curFlags &amp;= ~(BTCF_ValidNKey|BTCF_ValidOvfl);
   369,648    *pRes = 0;
   739,296    if( pCur-&gt;eState!=CURSOR_VALID ) return btreeNext(pCur, pRes);
 1,473,580    pPage = pCur-&gt;apPage&#91;pCur-&gt;iPage&#93;;
 1,841,975    if( (++pCur-&gt;aiIdx&#91;pCur-&gt;iPage&#93;)&gt;=pPage-&gt;nCell ){
     4,340      pCur-&gt;aiIdx&#91;pCur-&gt;iPage&#93;--;
     5,593      return btreeNext(pCur, pRes);
         .    }
   728,110    if( pPage-&gt;leaf ){
         .      return SQLITE_OK;
         .    }else{
     3,117      return moveToLeftmost(pCur);
         .    }
   721,876  }



The numbers on the left are the CPU cycle counts for that line of code,
of course.


The cg_anno.tcl script removes extraneous details from the default 
cachegrind annotation
output so that before-and-after reports can be compared using a 
side-by-side diff to view specific details of how a
micro-optimization attempt affected performance.


4. Limitations

The use of the standardized speedtest1.c workload and cachegrind has
enabled significant performance improvement.
However, it is important to recognize the limitations of this approach:



Performance measurements are done with a single compiler (gcc 5.4.0),
optimization setting (-Os), and
on a single platform (Ubuntu 16.04 LTS on x64).  The performance of
other compilers and processors may vary.


The speedtest1.c workload that is being measured tries to be representative
of a wide range of typical uses of SQLite.  But every application is
different.  The speedtest1.c workload might not be a good proxy for the
kinds of activities performed by some applications.  The SQLite developers
are constantly working to improve the speedtest1.c program, to make it
a better proxy for actual SQLite usage.  Community feedback is welcomed.


The cycle counts provided by cachegrind are a good proxy for actual
performance, but they are not 100% accurate.


Only CPU cycle counts are being measured here. 
CPU cycle counts are a good proxy for energy consumption,
but do not necessary correlate well with real-world timings.
Time spent doing I/O is not reflected in the CPU cycle counts,
and I/O time predominates in many SQLite usage scenarios.


c2: SQLite Backup API
c5: backup.html
c4:
Using the SQLite Online Backup API


  Historically, backups (copies) of SQLite databases have been created
  using the following method:


   Establish a shared lock on the database file using the SQLite API (i.e.
       the shell tool).
   Copy the database file using an external tool (for example the unix 'cp'
       utility or the DOS 'copy' command).
   Relinquish the shared lock on the database file obtained in step 1.



  This procedure works well in many scenarios and is usually very
  fast. However, this technique has the following shortcomings:


   Any database clients wishing to write to the database file while a
       backup is being created must wait until the shared lock is
       relinquished.

   It cannot be used to copy data to or from in-memory databases.

   If a power failure or operating system failure occurs while copying
       the database file the backup database may be corrupted following
       system recovery.



  The Online Backup API was created to 
  address these concerns. The online backup API allows the contents of
  one database to be copied into another database file, replacing any 
  original contents of the target database. The copy operation may be 
  done incrementally, in which case the source database does not need
  to be locked for the duration of the copy, only for the brief periods
  of time when it is actually being read from. This allows other database
  users to continue without excessive delays while a backup of an online
  database is made.

  The effect of completing the backup call sequence is to make the
  destination a bit-wise identical copy of the source database as it
  was when the copying commenced. (The destination becomes a "snapshot.")


  The online backup API is documented here.
  The remainder of this page contains two C language examples illustrating 
  common uses of the API and discussions thereof. Reading these examples
  is no substitute for reading the API documentation!


  Update: The VACUUM INTO command introduced in 
  SQLite version 3.27.0 (2019-02-07) can serve as an
  alternative to the backup API.

Example 1: Loading and Saving In-Memory Databases


/*
** This function is used to load the contents of a database file on disk 
** into the "main" database of open database connection pInMemory, or
** to save the current contents of the database opened by pInMemory into
** a database file on disk. pInMemory is probably an in-memory database, 
** but this function will also work fine if it is not.
**
** Parameter zFilename points to a nul-terminated string containing the
** name of the database file on disk to load from or save to. If parameter
** isSave is non-zero, then the contents of the file zFilename are 
** overwritten with the contents of the database opened by pInMemory. If
** parameter isSave is zero, then the contents of the database opened by
** pInMemory are replaced by data loaded from the file zFilename.
**
** If the operation is successful, SQLITE_OK is returned. Otherwise, if
** an error occurs, an SQLite error code is returned.
*/
int loadOrSaveDb(sqlite3 *pInMemory, const char *zFilename, int isSave){
  int rc;                   /* Function return code */
  sqlite3 *pFile;           /* Database connection opened on zFilename */
  sqlite3_backup *pBackup;  /* Backup object used to copy data */
  sqlite3 *pTo;             /* Database to copy to (pFile or pInMemory) */
  sqlite3 *pFrom;           /* Database to copy from (pFile or pInMemory) */

  /* Open the database file identified by zFilename. Exit early if this fails
  ** for any reason. */
  rc = sqlite3_open(zFilename, &pFile);
  if( rc==SQLITE_OK ){

    /* If this is a 'load' operation (isSave==0), then data is copied
    ** from the database file just opened to database pInMemory. 
    ** Otherwise, if this is a 'save' operation (isSave==1), then data
    ** is copied from pInMemory to pFile.  Set the variables pFrom and
    ** pTo accordingly. */
    pFrom = (isSave ? pInMemory : pFile);
    pTo   = (isSave ? pFile     : pInMemory);

    /* Set up the backup procedure to copy from the "main" database of 
    ** connection pFile to the main database of connection pInMemory.
    ** If something goes wrong, pBackup will be set to NULL and an error
    ** code and message left in connection pTo.
    **
    ** If the backup object is successfully created, call backup_step()
    ** to copy data from pFile to pInMemory. Then call backup_finish()
    ** to release resources associated with the pBackup object.  If an
    ** error occurred, then an error code and message will be left in
    ** connection pTo. If no error occurred, then the error code belonging
    ** to pTo is set to SQLITE_OK.
    */
    pBackup = sqlite3_backup_init(pTo, "main", pFrom, "main");
    if( pBackup ){
      (void)sqlite3_backup_step(pBackup, -1);
      (void)sqlite3_backup_finish(pBackup);
    }
    rc = sqlite3_errcode(pTo);
  }

  /* Close the database connection opened on database file zFilename
  ** and return the result of this function. */
  (void)sqlite3_close(pFile);
  return rc;
}




  The C function to the right demonstrates one of the simplest,
  and most common, uses of the backup API: loading and saving the contents
  of an in-memory database to a file on disk. The backup API is used as
  follows in this example:

  
    Function sqlite3_backup_init() is called to create an sqlite3_backup
        object to copy data between the two databases (either from a file and
        into the in-memory database, or vice-versa).
    Function sqlite3_backup_step() is called with a parameter of 
        -1 to copy the entire source database to the destination.
    Function sqlite3_backup_finish() is called to clean up resources
        allocated by sqlite3_backup_init().
  

Error handling


  If an error occurs in any of the three main backup API routines
  then the error code and message are attached to
  the destination database connection.
  Additionally, if
  sqlite3_backup_step() encounters an error, then the error code is returned
  by both the sqlite3_backup_step() call itself, and by the subsequent call
  to sqlite3_backup_finish(). So a call to sqlite3_backup_finish()
  does not overwrite an error code stored in the destination
  database connection by sqlite3_backup_step().  This feature
  is used in the example code to reduce amount of error handling required.
  The return values of the sqlite3_backup_step() and sqlite3_backup_finish()
  calls are ignored and the error code indicating the success or failure of
  the copy operation collected from the destination database connection
  afterward.

Possible Enhancements


  The implementation of this function could be enhanced in at least two ways:

  
     Failing to obtain the lock on database file zFilename (an SQLITE_BUSY
         error) could be handled, and
     Cases where the page-sizes of database pInMemory and zFilename are
         different could be handled better.
  


  Since database zFilename is a file on disk, then it may be accessed 
  externally by another process. This means that when the call to
  sqlite3_backup_step() attempts to read from or write data to it, it may
  fail to obtain the required file lock. If this happens, this implementation
  will fail, returning SQLITE_BUSY immediately. The solution would be to
  register a busy-handler callback or 
  timeout with database connection pFile 
  using sqlite3_busy_handler() or sqlite3_busy_timeout()
  as soon as it is opened. If it fails to obtain a required lock immediately,
  sqlite3_backup_step() uses any registered busy-handler callback or timeout
  in the same way as sqlite3_step() or sqlite3_exec() does.


  Usually, it does not matter if the page-sizes of the source database and the
  destination database are different before the contents of the destination
  are overwritten. The page-size of the destination database is simply changed
  as part of the backup operation. The exception is if the destination database
  happens to be an in-memory database. In this case, if the page sizes
  are not the same at the start of the backup operation, then the operation
  fails with an SQLITE_READONLY error. Unfortunately, this could occur when
  loading a database image from a file into an in-memory database using
  function loadOrSaveDb().


  However, if in-memory database pInMemory has just been opened (and is 
  therefore completely empty) before being passed to function loadOrSaveDb(), 
  then it is still possible to change its page size using an SQLite "PRAGMA
  page_size" command. Function loadOrSaveDb() could detect this case, and
  attempt to set the page-size of the in-memory database to the page-size
  of database zFilename before invoking the online backup API functions.
  
Example 2: Online Backup of a Running Database


/*
** Perform an online backup of database pDb to the database file named
** by zFilename. This function copies 5 database pages from pDb to
** zFilename, then unlocks pDb and sleeps for 250 ms, then repeats the
** process until the entire database is backed up.
** 
** The third argument passed to this function must be a pointer to a progress
** function. After each set of 5 pages is backed up, the progress function
** is invoked with two integer parameters: the number of pages left to
** copy, and the total number of pages in the source file. This information
** may be used, for example, to update a GUI progress bar.
**
** While this function is running, another thread may use the database pDb, or
** another process may access the underlying database file via a separate 
** connection.
**
** If the backup process is successfully completed, SQLITE_OK is returned.
** Otherwise, if an error occurs, an SQLite error code is returned.
*/
int backupDb(
  sqlite3 *pDb,               /* Database to back up */
  const char *zFilename,      /* Name of file to back up to */
  void(*xProgress)(int, int)  /* Progress function to invoke */     
){
  int rc;                     /* Function return code */
  sqlite3 *pFile;             /* Database connection opened on zFilename */
  sqlite3_backup *pBackup;    /* Backup handle used to copy data */

  /* Open the database file identified by zFilename. */
  rc = sqlite3_open(zFilename, &pFile);
  if( rc==SQLITE_OK ){

    /* Open the sqlite3_backup object used to accomplish the transfer */
    pBackup = sqlite3_backup_init(pFile, "main", pDb, "main");
    if( pBackup ){

      /* Each iteration of this loop copies 5 database pages from database
      ** pDb to the backup database. If the return value of backup_step()
      ** indicates that there are still further pages to copy, sleep for
      ** 250 ms before repeating. */
      do {
        rc = sqlite3_backup_step(pBackup, 5);
        xProgress(
            sqlite3_backup_remaining(pBackup),
            sqlite3_backup_pagecount(pBackup)
        );
        if( rc==SQLITE_OK || rc==SQLITE_BUSY || rc==SQLITE_LOCKED ){
          sqlite3_sleep(250);
        }
      } while( rc==SQLITE_OK || rc==SQLITE_BUSY || rc==SQLITE_LOCKED );

      /* Release resources allocated by backup_init(). */
      (void)sqlite3_backup_finish(pBackup);
    }
    rc = sqlite3_errcode(pFile);
  }
  
  /* Close the database connection opened on database file zFilename
  ** and return the result of this function. */
  (void)sqlite3_close(pFile);
  return rc;
}




  The function presented in the previous example copies the entire source
  database in one call to sqlite3_backup_step(). This requires holding a 
  read-lock on the source database file for the duration of the operation,
  preventing any other database user from writing to the database. It also
  holds the mutex associated with database pInMemory throughout the copy,
  preventing any other thread from using it. The C function in this section,
  designed to be called by a background thread or process for creating a 
  backup of an online database, avoids these problems using the following 
  approach:

  
    Function sqlite3_backup_init() is called to create an sqlite3_backup
        object to copy data from database pDb to the backup database file 
        identified by zFilename.
    Function sqlite3_backup_step() is called with a parameter of 5 to
        copy 5 pages of database pDb to the backup database (file zFilename).
    If there are still more pages to copy from database pDb, then the
        function sleeps for 250 milliseconds (using the sqlite3_sleep()
        utility) and then returns to step 2.
    Function sqlite3_backup_finish() is called to clean up resources
        allocated by sqlite3_backup_init().
  

File and Database Connection Locking


  During the 250 ms sleep in step 3 above, no read-lock is held on the database
  file and the mutex associated with pDb is not held. This allows other threads
  to use database connection pDb and other connections to write to the
  underlying database file. 


  If another thread or process writes to the source database while this 
  function is sleeping, then SQLite detects this and usually restarts the 
  backup process when sqlite3_backup_step() is next called. There is one 
  exception to this rule: If the source database is not an in-memory database,
  and the write is performed from within the same process as the backup
  operation and uses the same database handle (pDb), then the destination
  database (the one opened using connection pFile) is automatically updated
  along with the source. The backup process may then be continued after the 
  sqlite3_sleep() call returns as if nothing had happened. 


  Whether or not the backup process is restarted as a result of writes to
  the source database mid-backup, the user can be sure that when the backup
  operation is completed the backup database contains a consistent and 
  up-to-date snapshot of the original. However:

  
     Writes to an in-memory source database, or writes to a file-based 
         source database by an external process or thread using a 
         database connection other than pDb are significantly more expensive 
         than writes made to a file-based source database using pDb (as the
         entire backup operation must be restarted in the former two cases).

     If the backup process is restarted frequently enough it may never
         run to completion and the backupDb() function may never return.
  
 
backup_remaining() and backup_pagecount()


  The backupDb() function uses the sqlite3_backup_remaining() and
  sqlite3_backup_pagecount() functions to report its progress via the
  user-supplied xProgress() callback. Function sqlite3_backup_remaining()
  returns the number of pages left to copy and sqlite3_backup_pagecount()
  returns the total number of pages in the source database (in this case 
  the database opened by pDb). So the percentage completion of the process
  may be calculated as:


  Completion = 100% * (pagecount() - remaining()) / pagecount()


  The sqlite3_backup_remaining() and sqlite3_backup_pagecount() APIs report
  values stored by the previous call to sqlite3_backup_step(), they do not
  actually inspect the source database file. This means that if the source
  database is written to by another thread or process after the call to
  sqlite3_backup_step() returns but before the values returned by
  sqlite3_backup_remaining() and sqlite3_backup_pagecount() are used, the 
  values may be technically incorrect. This is not usually a problem.


c2: Query Planning
c5: queryplanner.html
c4:
Overview 


The best feature of SQL (in all its implementations, not just SQLite)
is that it is a declarative language, not a procedural
language.  When programming in SQL you tell the system what you
want to compute, not how to compute it.  The task of figuring out
the how is delegated to the query planner subsystem within 
the SQL database engine.

For any given SQL statement, there might be hundreds or thousands or
even millions of different algorithms of performing the operation.  All
of these algorithms will get the correct answer, though some will run
faster than others.
The query planner is an 
AI that 
tries to pick the fastest and most efficient algorithm for each SQL
statement.



Most of the time, the query planner in SQLite does a good job.
However, the query planner needs indices to
work with.  
These indices must normally be added by programmers.
Rarely, the query planner AI will make a suboptimal algorithm
choice.
In those cases, programmers may want to provide additional
hints to help the query planner do a better job.



This document provides background information about how the
SQLite query planner and query engine work.
Programmers can use this information to help create better
indexes, and provide hints to help the query planner when
needed.



Additional information is provided in the
SQLite query planner and 
next generation query planner documents.




1.  Searching

1.1.  Tables Without Indices


Most tables in SQLite consist of zero or more rows with a unique integer
key (the rowid or INTEGER PRIMARY KEY) followed by content.  
(The exception is WITHOUT ROWID tables.)
The rows
are logically stored in order of increasing rowid.  As an example, this
article uses a table named "FruitsForSale" which relates various fruits 
to the state
where they are grown and their unit price at market.  The schema is this:



CREATE TABLE FruitsForSale(
  Fruit TEXT,
  State TEXT,
  Price REAL
);




With some (arbitrary) data, such a table might be logically stored on disk
as shown in figure 1:





Figure 1: Logical Layout Of Table "FruitsForSale"




In this example, the rowids are not
consecutive but they are ordered.  SQLite usually creates rowids beginning
with one and increasing by one with each added row.  But if rows are 
deleted, gaps can appear in the sequence.  And the application can control
the rowid assigned if desired, so that rows are not necessarily inserted 
at the bottom.  But regardless of what happens, the rowids are always 
unique and in strictly ascending order.



Suppose you want to look up the price of peaches.  The query would
be as follows:



SELECT price FROM fruitsforsale WHERE fruit='Peach';




To satisfy this query, SQLite reads every row out of the
table, checks to see if the "fruit" column has the value of "Peach" and if
so, outputs the "price" column from that row.  The process is illustrated
by figure 2 below.
This is algorithm is called a full table scan 
since the entire content of the
table must be read and examined in order to find the one row of interest.
With a table of only 7 rows, a full table scan is acceptable, 
but if the table contained 7 million rows, a full table scan might read 
megabytes of content in order to find a single 8-byte number.  
For that reason, one normally tries to avoid full table scans.





Figure 2: Full Table Scan



1.2.  Lookup By Rowid


One technique for avoiding a full table scan is to do lookups by
rowid (or by the equivalent INTEGER PRIMARY KEY).   To lookup the
price of peaches, one would query for the entry with a rowid of 4:



SELECT price FROM fruitsforsale WHERE rowid=4;




Since the information is stored in the table in rowid order, SQLite
can find the correct row using a binary search.
If the table contains N elements, the time required to look up the
desired row is proportional to logN rather than being proportional
to N as in a full table scan.  If the table contains 10 million elements,
that means the query will be on the order of N/logN or about 1 million
times faster.





Figure 3: Lookup By Rowid



1.3.  Lookup By Index

The problem with looking up information by rowid is that you probably
do not care what the price of "item 4" is - you want to know the price
of peaches.  And so a rowid lookup is not helpful.



To make the original query more efficient, we can add an index on the
"fruit" column of the "fruitsforsale" table like this:



CREATE INDEX Idx1 ON fruitsforsale(fruit);




An index is another table similar to the original "fruitsforsale" table
but with the content (the fruit column in this case) stored in front of the
rowid and with all rows in content order.
Figure 4 gives a logical view of the Idx1 index.
The "fruit" column is the primary key used to order the elements of the
table and the "rowid" is the secondary key used to break the tie when
two or more rows have the same "fruit".  In the example, the rowid
has to be used as a tie-breaker for the "Orange" rows.
Notice that since the rowid
is always unique over all elements of the original table, the composite key
of "fruit" followed by "rowid" will be unique over all elements of the index.





Figure 4: An Index On The Fruit Column




This new index can be used to implement a faster algorithm for the
original "Price of Peaches" query.



SELECT price FROM fruitsforsale WHERE fruit='Peach';




The query starts by doing a binary search on the Idx1 index for entries
that have fruit='Peach'.  SQLite can do this binary search on the Idx1 index
but not on the original FruitsForSale table because the rows in Idx1 are sorted
by the "fruit" column.  Having found a row in the Idx1 index that has
fruit='Peach', the database engine can extract the rowid for that row.
Then the database engines does a second binary search
on the original FruitsForSale table to find the
original row that contains fruit='Peach'.  
From the row in the FruitsForSale table,
SQLite can then extract the value of the price column.
This procedure is illustrated by figure 5.





Figure 5: Indexed Lookup For The Price Of Peaches




SQLite has to do two binary searches to find the price of peaches using
the method show above.  But for a table with a large number of rows, this
is still much faster than doing a full table scan.


1.4.  Multiple Result Rows


In the previous query the fruit='Peach' constraint narrowed the result
down to a single row.  But the same technique works even if multiple
rows are obtained.  Suppose we looked up the price of Oranges instead of
Peaches:



SELECT price FROM fruitsforsale WHERE fruit='Orange'




Figure 6: Indexed Lookup For The Price Of Oranges




In this case, SQLite still does a single binary search to find the first
entry of the index where fruit='Orange'.  Then it extracts the rowid from
the index and uses that rowid to lookup the original table entry via
binary search and output the price from the original table.  But instead
of quitting, the database engine then advances to the next row of index
to repeat the process for next fruit='Orange' entry.  Advancing to the
next row of an index (or table) is much less costly than doing a binary
search since the next row is often located on the same database page as
the current row.  In fact, the cost of advancing to the next row is so
cheap in comparison to a binary search that we usually ignore it.  So
our estimate for the total cost of this query is 3 binary searches.
If the number of rows of output is K and the number of rows in the table
is N, then in general the cost of doing the query is proportional
to (K+1)*logN.


1.5.  Multiple AND-Connected WHERE-Clause Terms


Next, suppose that you want to look up the price of not just any orange,
but specifically California-grown oranges.  The appropriate query would
be as follows:



SELECT price FROM fruitsforsale WHERE fruit='Orange' AND state='CA'




Figure 7: Indexed Lookup Of California Oranges




One approach to this query is to use the fruit='Orange' term of the WHERE
clause to find all rows dealing with oranges, then filter those rows
by rejecting any that are from states other than California.  This
process is shown by figure 7 above.  This is a perfectly
reasonable approach in most cases.  Yes, the database engine did have
to do an extra binary search for the Florida orange row that was
later rejected, so it was not as efficient as we might hope, though
for many applications it is efficient enough.  



Suppose that in addition to the index on "fruit" there was also
an index on "state".



CREATE INDEX Idx2 ON fruitsforsale(state);




Figure 8: Index On The State Column




The "state" index works just like the "fruit" index in that it is a
new table with an extra column in front of the rowid and sorted by
that extra column as the primary key.  The only difference is that
in Idx2, the first column is "state" instead of "fruit" as it is with
Idx1.  In our example data set, there is more redundancy in the "state"
column and so they are more duplicate entries.  The ties are still
resolved using the rowid.



Using the new Idx2 index on "state", SQLite has another option for
lookup up the price of California oranges:  it can look up every row
that contains fruit from California and filter out those rows that
are not oranges.





Figure 9: Indexed Lookup Of California Oranges




Using Idx2 instead of Idx1 causes SQLite to examine a different set of
rows, but it gets the same answer in the end (which is very important -
remember that indices should never change the answer, only help SQLite to
get to the answer more quickly) and it does the same amount of work.
So the Idx2 index did not help performance in this case.



The last two queries take the same amount of time, in our example.
So which index, Idx1 or Idx2, will SQLite choose?  If the
ANALYZE command has been run on the database, so that SQLite has
had an opportunity to gather statistics about the available indices,
then SQLite will know that the Idx1 index usually narrows the search
down to a single item (our example of fruit='Orange' is the exception
to this rule) whereas the Idx2 index will normally only narrow the 
search down to two rows.  So, if all else is equal, SQLite will
choose Idx1 with the hope of narrowing the search to as small
a number of rows as possible.  This choice is only possible because
of the statistics provided by ANALYZE.  If ANALYZE has not been
run then the choice of which index to use is arbitrary.


1.6.  Multi-Column Indices


To get the maximum performance out of a query with multiple AND-connected
terms in the WHERE clause, you really want a multi-column index with
columns for each of the AND terms.  In this case we create a new index
on the "fruit" and "state" columns of FruitsForSale:



CREATE INDEX Idx3 ON FruitsForSale(fruit, state);




Figure 1: A Two-Column Index




A multi-column index follows the same pattern as a single-column index;
the indexed columns are added in front of the rowid.  The only difference
is that now multiple columns are added.  The left-most column is the
primary key used for ordering the rows in the index.  The second column is
used to break ties in the left-most column.  If there were a third column,
it would be used to break ties for the first two columns.  And so forth for
all columns in the index.  Because rowid is guaranteed
to be unique, every row of the index will be unique even if all of the
content columns for two rows are the same.  That case does not happen
in our sample data, but there is one case (fruit='Orange') where there
is a tie on the first column which must be broken by the second column.



Given the new multi-column Idx3 index, it is now possible for SQLite
to find the price of California oranges using only 2 binary searches:



SELECT price FROM fruitsforsale WHERE fruit='Orange' AND state='CA'




Figure 11: Lookup Using A Two-Column Index




With the Idx3 index on both columns that are constrained by the WHERE clause,
SQLite can do a single binary search against Idx3 to find the one rowid
for California oranges, then do a single binary search to find the price
for that item in the original table.  There are no dead-ends and no
wasted binary searches.  This is a more efficient query.



Note that Idx3 contains all the same information as the original 
Idx1.  And so if we have Idx3, we do not really need Idx1
any more.  The "price of peaches" query can be satisfied using Idx3
by simply ignoring the "state" column of Idx3:



SELECT price FROM fruitsforsale WHERE fruit='Peach'




Figure 12: Single-Column Lookup On A Multi-Column Index




Hence, a good rule of thumb is that your database schema should never
contain two indices where one index is a prefix of the other.  Drop the
index with fewer columns.  SQLite will still be able to do efficient
lookups with the longer index.




1.7.  Covering Indices


The "price of California oranges" query was made more efficient through
the use of a two-column index.  But SQLite can do even better with a
three-column index that also includes the "price" column:



CREATE INDEX Idx4 ON FruitsForSale(fruit, state, price);




Figure 13: A Covering Index




This new index contains all the columns of the original FruitsForSale table that
are used by the query - both the search terms and the output.  We call
this a "covering index".  Because all of the information needed is in
the covering index, SQLite never needs to consult the original table
in order to find the price.



SELECT price FROM fruitsforsale WHERE fruit='Orange' AND state='CA';




Figure 14: Query Using A Covering Index




Hence, by adding extra "output" columns onto the end of an index, one
can avoid having to reference the original table and thereby
cut the number of binary searches for a query in half.  This is a
constant-factor improvement in performance (roughly a doubling of
the speed).  But on the other hand, it is also just a refinement;
A two-fold performance increase is not nearly as dramatic as the
one-million-fold increase seen when the table was first indexed.
And for most queries, the difference between 1 microsecond and
2 microseconds is unlikely to be noticed.




1.8.  OR-Connected Terms In The WHERE Clause


Multi-column indices only work if the constraint terms in the WHERE
clause of the query are connected by AND.
So Idx3 and Idx4 are helpful when the search is for items that
are both Oranges and grown in California, but neither index would
be that useful if we wanted all items that were either oranges
or are grown in California.



SELECT price FROM FruitsForSale WHERE fruit='Orange' OR state='CA';




When confronted with OR-connected terms in a WHERE clause, SQLite 
examines each OR term separately and tries to use an index to
find the rowids associated with each term.
It then takes the union of the resulting rowid sets to find
the end result.  The following figure illustrates this process:





Figure 15: Query With OR Constraints




The diagram above implies that SQLite computes all of the rowids first
and then combines them with a union operation before starting to do
rowid lookups on the original table.  In reality, the rowid lookups
are interspersed with rowid computations.  SQLite uses one index at
a time to find rowids while remembering which rowids it has seen
before so as to avoid duplicates.  That is just an implementation
detail, though.  The diagram, while not 100% accurate, provides a good
overview of what is happening.



In order for the OR-by-UNION technique shown above to be useful, there
must be an index available that helps resolve every OR-connected term
in the WHERE clause.  If even a single OR-connected term is not indexed,
then a full table scan would have to be done in order to find the rowids
generated by the one term, and if SQLite has to do a full table scan, it
might as well do it on the original table and get all of the results in
a single pass without having to mess with union operations and follow-on
binary searches.



One can see how the OR-by-UNION technique could also be leveraged to
use multiple indices on queries where the WHERE clause has terms connected
by AND, by using an intersect operator in place of union.  Many SQL
database engines will do just that.  But the performance gain over using
just a single index is slight and so SQLite does not implement that technique
at this time.  However, a future version SQLite might be enhanced to support
AND-by-INTERSECT.




2.  Sorting


SQLite (like all other SQL database engines) can also use indices to
satisfy the ORDER BY clauses in a query, in addition to expediting
lookup.  In other words, indices can be used to speed up sorting as
well as searching.



When no appropriate indices are available, a query with an ORDER BY
clause must be sorted as a separate step.  Consider this query:



SELECT * FROM fruitsforsale ORDER BY fruit;




SQLite processes this by gathering all the output of query and then
running that output through a sorter.





Figure 16: Sorting Without An Index




If the number of output rows is K, then the time needed to sort is
proportional to KlogK.  If K is small, the sorting time is usually
not a factor, but in a query such as the above where K==N, the time
needed to sort can be much greater than the time needed to do a
full table scan.  Furthermore, the entire output is accumulated in
temporary storage (which might be either in main memory or on disk,
depending on various compile-time and run-time settings)
which can mean that a lot of temporary storage is required to complete
the query.


2.1.  Sorting By Rowid


Because sorting can be expensive, SQLite works hard to convert ORDER BY
clauses into no-ops.  If SQLite determines that output will
naturally appear in the order specified, then no sorting is done.
So, for example, if you request the output in rowid order, no sorting
will be done:



SELECT * FROM fruitsforsale ORDER BY rowid;




Figure 17: Sorting By Rowid




You can also request a reverse-order sort like this:



SELECT * FROM fruitsforsale ORDER BY rowid DESC;




SQLite will still omit the sorting step.  But in order for output to
appear in the correct order, SQLite will do the table scan starting at
the end and working toward the beginning, rather than starting at the
beginning and working toward the end as shown in 
figure 17.


2.2.  Sorting By Index


Of course, ordering the output of a query by rowid is seldom useful.
Usually one wants to order the output by some other column.



If an index is available on the ORDER BY column, that index can be used
for sorting.  Consider the request for all items sorted by "fruit":



SELECT * FROM fruitsforsale ORDER BY fruit;






Figure 18: Sorting With An Index




The Idx1 index is scanned from top to bottom (or from bottom to top if
"ORDER BY fruit DESC" is used) in order to find the rowids for each item
in order by fruit.  Then for each rowid, a binary search is done to lookup
and output that row.  In this way, the output appears in the requested order
without the need to gather the entire output and sort it using a separate step.



But does this really save time?  The number of steps in the 
original indexless sort is proportional to NlogN since
that is how much time it takes to sort N rows.  But when we use Idx1 as
shown here, we have to do N rowid lookups which take logN time each, so
the total time of NlogN is the same!



SQLite uses a cost-based query planner.  When there are two or more ways
of solving the same query, SQLite tries to estimate the total amount of
time needed to run the query using each plan, and then uses the plan with
the lowest estimated cost.  A cost is computed mostly from the estimated
time, and so this case could go either way depending on the table size and
what WHERE clause constraints were available, and so forth.  But generally
speaking, the indexed sort would probably be chosen, if for no other
reason, because it does not need to accumulate the entire result set in
temporary storage before sorting and thus uses much less temporary storage.


2.3.  Sorting By Covering Index


If a covering index can be used for a query, then the multiple rowid lookups
can be avoided and the cost of the query drops dramatically.





Figure 19: Sorting With A Covering Index




With a covering index, SQLite can simply walk the index from one end to the
other and deliver the output in time proportional to N and without having
allocate a large buffer to hold the result set.


3.  Searching And Sorting At The Same Time


The previous discussion has treated searching and sorting as separate
topics.  But in practice, it is often the case that one wants to search
and sort at the same time.  Fortunately, it is possible to do this
using a single index.


3.1.  Searching And Sorting With A Multi-Column Index


Suppose we want to find the prices of all kinds of oranges sorted in
order of the state where they are grown.  The query is this:



SELECT price FROM fruitforsale WHERE fruit='Orange' ORDER BY state




The query contains both a search restriction in the WHERE clause
and a sort order in the ORDER BY clause.  Both the search and the sort
can be accomplished at the same time using the two-column index Idx3.





Figure 20: Search And Sort By Multi-Column Index




The query does a binary search on the index to find the subset of rows
that have fruit='Orange'.  (Because the fruit column is the left-most column
of the index and the rows of the index are in sorted order, all such 
rows will be adjacent.)  Then it scans the matching index rows from top to
bottom to get the rowids for the original table, and for each rowid does
a binary search on the original table to find the price.



You will notice that there is no "sort" box anywhere in the above diagram.
The ORDER BY clause of the query has become a no-op.  No sorting has to be
done here because the output order is by the state column and the state
column also happens to be the first column after the fruit column in the
index.  So, if we scan entries of the index that have the same value for
the fruit column from top to bottom, those index entries are guaranteed to
be ordered by the state column.




3.2.  Searching And Sorting With A Covering Index


A covering index can also be used to search and sort at the same time.
Consider the following:



SELECT * FROM fruitforsale WHERE fruit='Orange' ORDER BY state




Figure 21: Search And Sort By Covering Index




As before, SQLite does single binary search
for the range of rows in the covering
index that satisfy the WHERE clause, the scans that range from top to 
bottom to get the desired results.  
The rows that satisfy the WHERE clause are guaranteed to be adjacent
since the WHERE clause is an equality constraint on the left-most
column of the index.  And by scanning the matching index rows from
top to bottom, the output is guaranteed to be ordered by state since the
state column is the very next column to the right of the fruit column.
And so the resulting query is very efficient.



SQLite can pull a similar trick for a descending ORDER BY:



SELECT * FROM fruitforsale WHERE fruit='Orange' ORDER BY state DESC




The same basic algorithm is followed, except this time the matching rows
of the index are scanned from bottom to top instead of from top to bottom,
so that the states will appear in descending order.




3.3.  Partial Sorting Using An Index (a.k.a. Block Sorting)


Sometimes only part of an ORDER BY clause can be satisfied using indexes.
Consider, for example, the following query:



SELECT * FROM fruitforsale ORDER BY fruit, price




If the covering index is used for the scan, the "fruit" column will appear
naturally in the correct order, but when there are two or more rows with
the same fruit, the price might be out of order.  When this occurs, SQLite
does many small sorts, one sort for each distinct value of fruit, rather
than one large sort.  Figure 22 below illustrates the concept.





Figure 22: Partial Sort By Index




In the example, instead of a single sort of 7 elements, there
are 5 sorts of one-element each and 1 sort of 2 elements for the
case of fruit=='Orange'.


The advantages of doing many smaller sorts instead of a single large sort
are:

Multiple small sorts collectively use fewer CPU cycles than a single
    large sort.
Each small sort is run independently, meaning that much less information
    needs to be kept in temporary storage at any one time.
Those columns of the ORDER BY that are already in the correct order
    due to indexes can be omitted from the sort key, further reducing
    storage requirements and CPU time.
Output rows can be returned to the application as each small sort
    completes, and well before the table scan is complete.
If a LIMIT clause is present, it might be possible to avoid scanning
    the entire table.


Because of these advantages, SQLite always tries to do a partial sort using an
index even if a complete sort by index is not possible.

4.  WITHOUT ROWID tables


The basic principals described above apply to both ordinary rowid tables
and WITHOUT ROWID tables.
The only difference is that the rowid column that serves as the key for
tables and that appears as the right-most term in indexes is replaced by
the PRIMARY KEY.


c2: Recent SQLite News
c5: news.html
c4:
Recent News

2022-09-05 - Version 3.39.3
Version 3.39.3 is a patch release that fixes a few obscure problems
in the 3.39.0 release.  See the release notes for details.
2022-07-21 - Version 3.39.2
Version 3.39.2 is a security release that addresses multiple long-standing
issues in SQLite.  The most severe problem is identified by CVE-2022-35737.
That issue is associated with the auxiliary C-language APIs
of SQLite and cannot be reached using SQL or database inputs, and hence is
unlikely to affect your application.  Nevertheless, upgrading is recommended.
The other issues are comparatively minor.
This release also fixes a performance regression that appeared in version
3.39.0 that might affect some multi-way joins that use LEFT JOIN.
2022-07-13 - Version 3.39.1
Version 3.39.1 is a patch release that fixes a few minor problem in version
3.39.0.  Upgrading is optional.
2022-06-25 - Version 3.39.0
Version 3.39.0 is regular maintenance release of SQLite.  The key enhancement
in this release is added support for RIGHT and FULL JOIN.  There are other
language and performance enhancements as well &mdash; see the
release notes for details.
2022-05-06 - Version 3.38.5
The 3.38.4 patch release included a minor change to the CLI source code
that did not work.  The release manager only ran a subset of the normal
release tests, and hence did not catch the problem. As a result, the CLI
will segfault when using columnar output modes in version 3.38.4.  This
blunder did not affect the core SQLite library.  It only affected the CLI.

Take-away lesson:  Always run all of your tests prior to
a release - even a trival patch release.  Always.

The 3.38.5 patch release fixes the 3.38.4 blunder.
2022-05-04 - Version 3.38.4
Another user-discovered problem in the new Bloom filter optimization
is fixed in this patch release.  Without the fix, it is possible for
a multi-way join that uses a Bloom filters for two or more tables in
the join to enter an infinite loop if the key constraint on one of those
tables contains a NULL value.
2022-04-27 - Version 3.38.3
Version 3.38.3 fixes a bug in the automatic-index and Bloom filter
construction logic that might cause SQLite to be overly aggressive
in the use of ON clause constraints, resulting in a incorret
automatic-index or Bloom filter that excludes some valid rows from
output.  The bug was introduced in version 3.38.0.  Other minor
changes were tossed in to complete the patch.
2022-03-26 - Version 3.38.2
Version 3.38.2 fixes another bug in the new Bloom filter
optimization that might cause incorrect answers for a
LEFT JOIN that has an IS NULL constraint on the right-hand
table.
2022-03-12 - Version 3.38.1
Version 3.38.1 fixes a pair of bugs in the Bloom filter
optimization that was introduced in version 3.38.0.  These
bugs might cause incorrect answers for some obscure queries.
Various other minor problems and documentation typos were
fixed at the same time.
2022-02-22 - Version 3.38.0
Version 3.38.0 is a routine maintenance release of
SQLite.  There are various minor enhancements and
about a 0.5% reduction in the number of CPU cycles
used. See the
release notes for
more detail.
2022-01-06 - Version 3.37.2
Version 3.37.2 fixes a
database corruption bug.  You are
encouraged to upgrade, especially if you are using SAVEPOINT.

The problem first appeared in version 3.35.0 (2021-03-12) and
affects all subsequent releases through 3.37.1.
If temporary files are store in memory (which is not the default
behavior, but is sometimes selected by applications using either
-DSQLITE_TEMP_STORE or PRAGMA temp_store) and
if a SAVEPOINT is rolled back and then subsequent changes
within the same transaction are committed, the database file might
(with low but non-zero probability) go corrupt.
2021-12-30 - Version 3.37.1
Version 3.37.1 fixes a bug in the UPSERT logic, introduced by
the UPSERT enhancements of version 3.35.0, that can cause
incorrect byte-code to be generated in some cases, resulting
in an infinite loop in the byte code, or a NULL-pointer dereference.
This patch release also fixes some other minor problems with
assert() statements and in the CLI.
2021-11-27 - Version 3.37.0
Version 3.37.0 is a routine maintenance release of SQLite.
The biggest new feature in this release is support for
STRICT tables.  Other enhancements are described in
the release notes.
2021-06-18 - Version 3.36.0
Version 3.36.0 is a routine maintenance release of SQLite.
There are no new major features, only incremental improvements
to existing features and small performance improvements.
2021-04-19 - Patch release 3.35.5
The new ALTER TABLE DROP COLUMN capability that was added
in the 3.35.0 release contained a bug that might cause the
table content to go corrupt when the table was rewritten
to remove the dropped column.  Fixed by this patch.
2021-04-02 - Patch release 3.35.4
Version 3.35.4 is yet another patch release to fix
obscure problems in features associated with the 3.35.0.
2021-03-26 - Patch release 3.35.3
Version 3.35.3 contains patches for a handful of minor
problems discovered in prior releases.
2021-03-17 - Patch release 3.35.2
Version 3.35.2 is a small patch release to fix some minor problems
that were discovered shortly after the 3.35.1 release.
2021-03-15 - Patch release 3.35.1
A user discovered an issue with the new DROP COLUMN capability
in version 3.35.0, and so version 3.35.1 was created to fix it.
No need to upgrade if you are not using DROP COLUMN.
2021-03-12 - Release 3.35.0
SQLite version 3.35.0 is a routine maintenance release.  This
release adds a number of new language features, including
support for ALTER TABLE DROP COLUMN, built-in math functions,
generalized UPSERT, and the MATERIALIZED hint on common table
expressions.  There are also query planner optimizations and
incremental CLI improvements.
2020-01-20 - Release 3.34.1
SQLite version 3.34.1 is a patch releases that fixes a possible
use-after-free bug that can be provoked by malicious SQL.  Other
minor issues in extensions and documentation are also fixed.
2020-12-01 - Release 3.34.0
SQLite version 3.34.0 is a routine maintenance release.  This
release adds incremental improvements to performance and features,
including enhancements to the query planner, multiple recursive
SELECTS in recursive common table expressions, and better error messages
from CHECK constraint failures.  See the change log for details.
2020-08-14 - Release 3.33.0
SQLite version 3.33.0 is a routine maintenance release.  This
release features added support for "UPDATE FROM" following the
PostgreSQL syntax, and a doubling of the maximum database size
to 281 TB, as well as many other improvements.  See the change
log for details.
2020-06-18 - Release 3.32.3
The 3.32.3 release is a patch release that contains fixes for
various issues discovered by fuzzers.  None of the issues fixed
are likely to be encountered by applications that use SQLite in
ordinary ways, though upgrading never hurts.

Map of all changes since the 3.32.0 release:
https://www.sqlite.org/src/timeline?p=version-3.32.3&bt=version-3.32.0
2020-06-04 - Release 3.32.2
The 3.32.2 release is a one-line change relative to 3.32.1
that fixes a long-standing bug in the COMMIT command.  Since
version 3.17.0, if you were to retry a COMMIT command over
and over after it returns SQLITE_BUSY, it might eventually
report success, even though it was still blocked.  This patch
fixes the problem.
2020-05-25 - Release 3.32.1
Grey-hats published
information about two SQLite bugs approximately 24 hours after
the release of version 3.32.0.  These bugs enable maliciously
crafted SQL to crash the process that is running SQLite.  Both
bugs are long-standing problems that affect releases prior to
3.32.0. The 3.32.1 release fixes both problems.
2020-05-22 - Release 3.32.0
Version 3.32.0 is an ordinary maintenance release of SQLite.
This release features the ability to run an
approximate ANALYZE to gather database statistics for
use by the query planner, without having to scan every row
of every index.
See the change log for additional enhancements
and improvements.
2020-01-27 - Release 3.31.1
Applications that use SQLite should only interface with SQLite
through the officially published APIs.  Applications should not
depend upon or use the internal data structures of SQLite as those
structures might change from one release to another.  However, there
is a popular application that does depend on the details of the
internal layout of data in an internal SQLite data structure, and
those details changed in version 3.31.0, breaking the application.
This is, technically, a bug in the application, not in SQLite.
But it is within the power of SQLite to fix it, by reverting the
internal data structure change, and so that is what we have done
for the 3.31.1 release.
2020-01-22 - Release 3.31.0
Version 3.31.0 is an ordinary maintenance release of SQLite.
This release features the ability to define
generated columns for tables as well as many other enhancements.
See the change log for additional information.
2019-10-11 - Release 3.30.1
Version 3.30.1 is a bug-fix release that addresses a problem
that can occur when an aggregate function in a nested query
makes use of the new FILTER clause capability.  Some addition
patches for various obscure issues are also included, for
completeness.
2019-10-04 - Release 3.30.0
Version 3.30.0 is a regularly scheduled maintenance release
of SQLite containing miscellaneous performance and feature
enhancements.  This release adds support fo the NULLS FIRST
and NULLS LAST clauses on ORDER BY statements and the 
addition of FILTER clauses on all aggregate functions.
See the change log for details.
2019-07-10 - Release 3.29.0
Version 3.29.0 is a regularly scheduled maintenance release
of SQLite containing miscellaneous performance and feature
enhancements.  See the change log for
details.

Beginning with this release, the
double-quoted string literal misfeature is deprecated.
The misfeature is still enabled by default, for legacy
compatibility, however developers are encouraged to
disable it at compile-time using the
-DSQLITE_DQS=0 option, or at run-time using
the SQLITE_DBCONFIG_DQS_DML and SQLITE_DBCONFIG_DQS_DDL
actions to the sqlite3_db_config() interface.  This is
especially true for double-quoted string literals in
CREATE TABLE and CREATE INDEX statements, as those elements
can cause unexpected problems following an ALTER TABLE.
See ticket 9b78184be266fd70
for an example.
2019-04-16 - Release 3.28.0
Version 3.28.0 is a regularly scheduled maintenance release
of SQLite containing miscellaneous performance and feature
enhancements.  See the change log for
details.

This release fixes an obscure security issue. Applications
using older versions of SQLite may be vulnerable if

 SQLite is compiled with certain optional extensions
     enabled, and
 the attacker is able to inject arbitrary SQL.

We are not aware of any applications that are vulnerable to
this problem. On the other hand, we do not know about
every application that uses SQLite.  
If your application allows unauthenticated users on the internet
(and hence potential attackers) to run arbitrary SQL and if
you build SQLite with any of the optional extensions enabled,
then you should take this upgrade at your earliest opportunity.

For further information about improving SQLite's robustness
in internet-facing applications, see the our
security recommendations.
2019-02-25 - Release 3.27.2
Version 3.27.2 is a patch release that
fixes a two bugs and various documentation
errors found in the version 3.27.1.  The
changes from version 3.27.1 and 3.27.0 are
minimal.
2019-02-08 - Release 3.27.1
After release 3.27.0 was tagged, but before the build could
be completed and uploaded, a 
long-standing bug
in the query optimizer was reported via System.Data.SQLite.
Since people will be upgrading anyhow, we decided 
publish the fix for this newly discovered problems right
away.  Hence, 3.27.1 was released less than 24 hours after
3.27.0.

It would have been better if the query optimizer bug had come to
our attention one day earlier, so that we could have incorporated
a fix into 3.27.0, but sometimes that's the way things go.
2019-02-07 - Release 3.27.0
SQLite version 3.27.0 is a routine maintenance release with various
performance and feature enhancements.  See the
release notes for details.
2018-12-01 - Release 3.26.0
SQLite version 3.26.0 is a routine maintenance release with various
performance and feature enhancements.  See the
release notes for details.
2018-11-05 - Release 3.25.3
SQLite version 3.25.3 is a third patch against 3.25.0 that fixes various
problems that have come to light and which seem serious enough to 
justify a patch.
2018-09-25 - Release 3.25.2
SQLite version 3.25.2 is another patch against 3.25.0 that fixes still
more problems associated with the new window function feature and the
ALTER TABLE enhancements.  Of particular note is the new
PRAGMA legacy_alter_table=ON command, which causes the ALTER TABLE RENAME
command to behave in the same goofy way that it did before the enhancements
found in version 3.25.0 &rarr; references to renamed tables that are inside
the bodies of triggers and views are not updated.  The legacy behavior is
arguably a bug, but some programs depend on the older buggy behavior.  The
3.25.2 release also contains a fix to window function processing for
VIEWs.  There also a slew of other minor fixes that affect obscure
compile-time options.  See the
Fossil Timeline for
details.
2018-09-18 - Release 3.25.1
SQLite version 3.25.1 is a patch against version 3.25.0 that contains
two one-line fixes for bug that were introduced in version 3.25.0.
See the change log for details.  Upgrading from 3.25.0 is recommended.
2018-09-15 - Release 3.25.0
SQLite version 3.25.0 is a regularly scheduled maintenance release.
Two big enhancements in this release:


Support for window functions was added, using the PostgreSQL documentation
as the baseline.

The ALTER TABLE command was enhanced to support renaming of columns, and
so that column and table renames are propagated into trigger bodies and views.

In addition, there are various performance enhancements and minor fixes.
One bug of note is 
ticket 9936b2fa443fec
which describes a hard-to-reach condition where the ORDER BY LIMIT
optimization might cause an infinite loop during query evaluation.  
This ticket raised a lot of
concern on 
HackerNews and
Reddit,
probably due to my choice of the ticket
title.  "Infinite Loop" sounds scary.  But I argue that the bug isn't really
all that bad in that it is very difficult to reach, will show up during
testing (rather than magically appearing after a product is
deployed), does not cause any data loss, and does not return an
incorrect result.  It was an important error, but not nearly as dire
as many people interpreted it to be.  And, in any event, it is fixed now.
2018-06-04 - Release 3.24.0
SQLite version 3.24.0 is a regularly scheduled maintenance release.
Highlights of this release include support for PostgreSQL-style
UPSERT and improved performance, especially for ORDER BY LIMIT queries.
2018-04-10 - Release 3.23.1
The version 3.23.1 release fixes a bug in the new
LEFT JOIN strength reduction optimization added to version 3.23.0.
A few other minor and obscure fixes were also inserted, as well as
a small performance optimization.  Code changes relative to
version 3.23.0 are minimal.
2018-04-02 - Release 3.23.0
The version 3.23.0 release is a regularly scheduled maintenance release.
See the change log for a list of enhancements and bug
fixes.
2018-01-22 - Release 3.22.0
The version 3.22.0 release is a regularly scheduled maintenance release.
There are many minor, though interesting, enhancements in this release.
See the change log for details.
2017-10-24 - Release 3.21.0
The version 3.21.0 release is a regularly scheduled maintenance release.
There are lots of enhancements in this release.
See the change log for details.
2017-08-24 - Release 3.20.1
The version 3.20.1 patch release changes two lines of code in
the sqlite3_result_pointer() interface in order to fix a rare
memory leak.  There are no other changes relative to version 3.20.0.
2017-08-01 - Release 3.20.0
SQLite version 3.20.0 is a regularly scheduled maintenance release
of SQLite.

This release contains many minor enhancements, including:

 Several new extensions
 Enhancements to the "sqlite3.exe" command-line shell
 Query planner enhancements
 Miscellaneous code optimizations for improved performance
 Fixes for some obscure bugs


See the release notes for more information.
2017-06-17 - Release 3.18.2
SQLite version 3.18.2 is another backport of a bug fix found
in SQLite version 3.19.0, specifically the fix for
ticket 61fe9745.  Changes
against version 3.18.0 are minimal.
2017-06-16 - Release 3.18.1
SQLite version 3.18.1 is a bug-fix release against version 3.18.0
that fixes the auto_vacuum corruption bug described in ticket
fda22108.  This release was
created for users who need that bug fix but do not yet want to upgrade 
to version 3.19.3.  
2017-06-08 - Release 3.19.3
Version 3.19.3 is an emergency patch release to fix a 
bug in 
auto_vacuum logic that can lead to database corruption.
The bug was introduced in version 3.16.0 
(2017-01-02).  Though the bug is obscure and rarely
encountered, upgrading is recommended for all users, and
especially for users who turn on auto_vacuum.
2017-05-25 - Release 3.19.2
Still more problems have been found in the LEFT JOIN
flattening optimization
that was added in the 3.19.0 release.  This patch release fixes all known
issues with that optimization and adds new test cases.  Hopefully this
will be the last patch.
2017-05-24 - Release 3.19.1
One of the new query planner optimizations in the 3.19.0 release contained
bugs.  The 3.19.1 patch release fixes them.

Beginning with 3.19.0, subqueries and views on the right-hand side of
a LEFT JOIN operator could sometimes be
flattened into the
main query.  The new optimization worked well for all of the test cases
that the developers devised, and for millions of legacy test cases, but
once 3.19.0 was released, users found some other cases where the optimization
failed.  Ticket
cad1ab4cb7b0fc344 contains
examples.

These problems exist only in 3.19.0.  Users of SQLite 3.19.0 should
upgrade, but users of all prior versions of SQLite are safe.
2017-05-22 - Release 3.19.0
SQLite version 3.19.0 is a regularly scheduled maintenance release.

The emphasis on this release is improvements to the query planner.
There are also some obscure bug fixes.  There is no reason to upgrade
unless you are having problems with a prior release.
2017-03-30 - Release 3.18.0
SQLite version 3.18.0 is a regularly scheduled maintenance release.

This release features an initial implementation the 
"PRAGMA optimize" command.  This command can now be used to cause
ANALYZE to be run on an as-needed basis.  Applications should invoke
"PRAGMA optimize" just before closing the database connection.
The "PRAGMA optimize" statement will likely be enhanced to do other
kinds of automated database maintenance in future releases.

The Fossil version control system that is
used to manage the SQLite project has been upgraded to use SHA3-256 hashes
instead of SHA1.  Therefore, the version identifications for SQLite now
show a 64-hex-digit SHA3-256 hash rather than the 40-hex-digit SHA1 hash.

See the change log for other enhancements and optimizations
in this release.
2017-02-13 - Release 3.17.0
SQLite version 3.17.0 is a regularly scheduled maintenance release.

Most of the changes in this release are performance optimizations.
Optimizations to the R-Tree extension are especially noticeable.

In this release, the default size of the 
lookaside buffer allocated for each database connection
is increased from 64,000 to 120,000 bytes.  This provides improved
performance on many common workloads in exchange for a small increase
in memory usage.
Applications that value a small memory footprint over raw speed
can change the lookaside buffer size back to its old value (or to zero)
using the SQLITE_DEFAULT_LOOKASIDE compile-time option, or the
sqlite3_config(SQLITE_CONFIG_LOOKASIDE)
start-time setting, or the
sqlite3_db_config(SQLITE_DBCONFIG_LOOKASIDE)
run-time setting.
2017-01-06 - Release 3.16.2
One of the performance optimizations added in 3.16.0 caused triggers
and foreign keys to malfunction for the REPLACE statement on
WITHOUT ROWID tables that lack secondary indexes.  This patch
release fixes the problem.  See ticket 
30027b613b4 for details.
2017-01-03 - Release 3.16.1
SQLite version 3.16.1 fixes a bug in the row-value logic for UPDATE
statements inside of triggers.  The bug has been there since row-values
were added by release 3.15.0, but was not discovered until just a few
minutes after the 3.16.0 release was published, and so it was not fixed
by 3.16.0.  This patch release is version 3.16.0 with the row-value bug fix.
2017-01-02 - Release 3.16.0
SQLite version 3.16.0 is a regularly schedule maintenance release.


This release includes many microoptimizations that collectively reduce
the CPU cycle count by about 9%,
add there have been important enhancements to the command-line shell.

Support for PRAGMA functions is added,
so that many pragma statements can be used as part of a larger SQL query.
This is considered an experimental feature.
We do not anticipate any changes to the PRAGMA function interface, but
will keep continue to call this interface "experimental" for a few release
cycles in case unforeseen issues arise.

See the change log for other enhancements.
2016-11-28 - Release 3.15.2
SQLite version 3.15.2 is a bug-fix patch release that fixes several minor
issues in the 3.15.0 and 3.15.1 releases.
2016-11-04 - Release 3.15.1
SQLite version 3.15.1 is a bug-fix patch release that fixes some minor
issues in the 3.15.0 release.
2016-10-14 - Release 3.15.0
SQLite version 3.15.0 is a regularly scheduled maintenance release.
The key feature in this release is the added support for
row values.  There are also other enhancements and
fixes for a number of obscure bugs.

The 3.15.0 release uses about 7% fewer CPU cycles than 3.14.2.
Most of the improvement in this release
is in the SQL parser, query planner, and
byte-code generator (the front-end) corresponding
to the sqlite3_prepare_v2() interface.  Overall,
version 3.15.0 uses about half as much CPU time as
version 3.8.1 (2013-10-17).  These
performance measurements are made using the "speedtest1.c"
workload on x64 compiled with gcc and -Os. Performance
improvements may vary with different platforms and
workloads.
2016-09-12 - Release 3.14.2
SQLite version 3.14.2 fixes several obscure bugs and adds 
improved support for building SQLite using the STDCALL calling
convention on 32-bit windows systems.  Upgrading from versions
3.14 and 3.14.1 is optional.
2016-08-11 - Release 3.14.1
SQLite version 3.14.1 adds a small patch to improve the performance
of the pcache1TruncateUnsafe() routine for cases when the only a few
pages on the end of the cache are being removed.  This causes COMMITs
to run faster when there is a very large page cache.  Upgrading from
version 3.14 is optional.
2016-08-08 - Release 3.14
SQLite version 3.14 (the "&#960;" release)
is a regularly scheduled maintenance
release containing performance enhancements, new features, and fixes for
obscure bugs.
2016-05-18 - Release 3.13.0
SQLite version 3.13.0 is a regularly schedule maintenance release containing
performance enhancements and fixes for obscure bugs.
2016-04-18 - Release 3.12.2
Yikes!  The 3.12.0 and 3.12.1 releases contain a backwards compatibility bug!
   Tables that declare a column with type "INTEGER" PRIMARY KEY
   (where the datatype name INTEGER is quoted) generate an incompatible
   database file.  The mistake came about because the developers have never
   thought to put a typename in quotes before, and so there was no documentation 
   of that capability nor any tests.  (There are tests now, though, of course.)
   Instances of quoting the datatype name are probably infrequent in the wild,
   so we do not expect the impact of this bug to be too severe.
   Upgrading is still strongly recommended.
Fixes for three other minor issues were included in this patch release.
   The other issues would have normally been deferred until the next scheduled
   release, but since a patch release is being issued anyhow, they might as
   well be included.
2016-04-08 - Release 3.12.1
SQLite version 3.12.1 is an emergency patch release to address a 
   crash bug that snuck
   into version 3.12.0.  Upgrading from version 3.12.0 is highly
   recommended.
Another minor problem involving datatypes on view columns, and
   a query planner deficiency are fixed at the same time.  These two
   issues did not justify a new release on their own, but since a release
   is being issued to deal with the crash bug, we included these other
   fixes for good measure.
2016-03-29 - Release 3.12.0
SQLite version 3.12.0 is a regularly scheduled maintenance release.
   A notable change in this release is an
   increase in the default page size for newly created database files.
   There are also various performance improvements.
   See the change log for details.
2016-03-03 - Release 3.11.1
SQLite version 3.11.1 is a patch release that fixes problems in the
   new FTS5 extension and increases a default setting in the spellfix1
   extension, and implements enhancements to some of the Windows makefiles.
   The SQLite core is unchanged from 3.11.0. Upgrading is optional.
2016-02-15 - Release 3.11.0
SQLite version 3.11.0 is a regularly scheduled maintenance release.
2016-01-20 - Release 3.10.2
Yikes!  An optimization attempt gone bad resulted in a 
bug in the LIKE operator
which is fixed by this patch release.
Three other minor but low-risk fixes are also included in the patch.
2016-01-14 - Release 3.10.1
SQLite version 3.10.1 is a bug-fix release primarily targeting the
fix for the query planner bug
cb3aa0641d9a4 discovered by
Mapscape.  Also included is a minor API enhancement requested by
the Firefox developers at Mozilla.  The differences from version 
3.10.0 are minimal.
2016-01-06 - Release 3.10.0
SQLite version 3.10.0 is a regularly scheduled maintenance release.


Old news...


c2: SQLite Changes From Version 3.5.9 To 3.6.0
c5: 35to36.html
c4:
Moving From SQLite 3.5.9 to 3.6.0
  SQLite version 3.6.0 (2008-07-16)
  contains many changes.  As is the custom with
  the SQLite project, most changes are fully backwards compatible.
  However, a few of the changes in version 3.6.0 are incompatible and
  might require modifications to application code and/or makefiles.
  This document is a briefing on the changes in SQLite 3.6.0
  with special attention to the incompatible changes.


  Key Points:
  
   The database file format is unchanged. 
   All incompatibilities are on obscure interfaces and hence should
       have zero impact on most applications. 
  

1.0 Incompatible Changes
  Incompatible changes are covered first since they are the most
  important to maintainers and programmers.

1.1 Overview Of Incompatible Changes
  
  Changes to the sqlite3_vfs object

      
      The signature of the xAccess method has been modified to
      return an error code and to store its output into an integer pointed
      to by a parameter, rather than returning the output directly.
      This change allows the xAccess() method to report failures.
      In association with this signature change, a new 
      extended error code SQLITE_IOERR_ACCESS has been added.
      

      The xGetTempname method has been removed from sqlite3_vfs.
      In its place, the xOpen method is enhanced to open a temporary file
      of its own invention when the filename parameter is NULL.

      Added the xGetLastError() method to sqlite3_vfs for returning
      filesystem-specific error messages and error codes back to
      SQLite.
      
  

  The signature of the xCheckReservedLock method on sqlite3_io_methods
      has been modified so that it returns an error code and stores its
      boolean result into an integer pointed to by a parameter.  In
      association with this change, a new extended error code
      SQLITE_IOERR_CHECKRESERVEDLOCK has been added.

  When SQLite is ported to new operating systems (operating systems 
      other than Unix, Windows, and OS/2 for which ports are provided together
      with the core)
      two new functions, sqlite3_os_init() and sqlite3_os_end(), must
      be provided as part of the port.

  The way in which the IN and NOT IN operators handle NULL values
      in their right-hand expressions has been brought into compliance with
      the SQL standard and with other SQL database engines.

  The column names for the result sets of SELECT statements have
      been tweaked in some cases to work more like other SQL database
      engines.      

  Changes to compile-time options:

      
      The SQLITE_MUTEX_APPDEF compile-time parameter is no longer
      recognized.  As a replacement, alternative 
      mutex implementations may be created
      at runtime using sqlite3_config() with the SQLITE_CONFIG_MUTEX
      operator and the sqlite3_mutex_methods object.

      Compile-time options OS_UNIX, OS_WIN, OS_OS2, OS_OTHER, and
      TEMP_STORE have been renamed to include an "SQLITE_" prefix in order
      to help avoid namespace collisions with application software.  The
      new names of these options are respectively:
      SQLITE_OS_UNIX, SQLITE_OS_WIN, SQLITE_OS_OS2, SQLITE_OS_OTHER,
      and SQLITE_TEMP_STORE.
      
  
  

1.2 Changes To The VFS Layer
  SQLite version 3.5.0 introduced a new OS interface layer that
  provided an abstraction of the underlying operating system.
  This was an important innovation and has proven to be helpful
  in porting and maintaining SQLite.
  However, the developers have discovered some minor flaws in the
  original "virtual file system" design introduced in version 3.5.0
  and so SQLite 3.6.0 includes some small incompatible changes
  to address these flaws.


  Key Point:  The incompatible
  changes in the SQLite operating-system interface for version 3.6.0
  only affect the rare applications that make use of the 
  virtual file system interface or that
  supply an application-defined mutex implementation
  or that make use of other obscure compile-time options.  The
  changes introduced by SQLite version 3.6.0 will have zero impact on the
  vast majority of SQLite applications that use the built-in interfaces
  to Unix, Windows, and OS/2 and that use the standard build configuration.

1.3 Changes In The Way The IN Operator Handles NULLs
  All versions of SQLite up to and including version 3.5.9 have mishandled
  NULL values on the right-hand side of IN and NOT IN operators.
  Specifically, SQLite has previously ignored NULLs on the right-hand side
  of IN and NOT IN.


  Suppose we have a table X1 defined as follows:


  CREATE TABLE x1(x INTEGER);
  INSERT INTO x1 VALUES(1);
  INSERT INTO x1 VALUES(2);
  INSERT INTO x1 VALUES(NULL);

  Given the definition of X1 above, the following expressions have
  historically evaluated to FALSE in SQLite, though the correct
  answer is actually NULL:


  3 IN (1,2,NULL)
  3 IN (SELECT * FROM x1)

  Similarly, the following expressions have historically evaluated to
  TRUE when in fact NULL is also the correct answer here:


  3 NOT IN (1,2,NULL)
  3 NOT IN (SELECT * FROM x1)

  The historical behavior of SQLite is incorrect according to the SQL:1999
  standard and it is inconsistent with the behavior of MySQL and
  PostgreSQL.  Version 3.6.0 changes the behavior of the IN and
  NOT IN operators to conform to the standard and to give the same
  results as other SQL database engines.


  Key Point: The change to the way NULL values are handled
  by the IN and NOT IN operators is technically a bug fix, not a design
  change.  However, maintainers should check to ensure that applications
  do not depend on the older, buggy behavior prior to upgrading to
  version 3.6.0.

1.4 Changes To Column Naming Rules
  The column names reported by join subqueries have been modified slightly
  in order to work more like other database engines.  Consider the following
  query:


  CREATE TABLE t1(a);
  CREATE TABLE t2(x);
  SELECT * FROM (SELECT t1.a FROM t1 JOIN t2 ORDER BY t2.x LIMIT 1) ORDER BY 1;

  In version 3.5.9 the query above would return a single column named "t1.a".
  In version 3.6.0 the column name is just "a".


  SQLite has never made any promises about the names of columns in the
  result set of SELECT statement unless the column contains an AS clause.
  So this change to column name is technically not an incompatibility.
  SQLite is merely changing from one undefined behavior to another.
  Nevertheless, many applications depend on the unspecified column naming
  behavior of SQLite and so this change is discussed under the
  incompatible changes subheading.

1.5 Changes To Compile-Time Options
  Compile-time options to SQLite are controlled by C-preprocessor 
  macros.  SQLite version 3.6.0 changes the names of some of these
  macros so that all C-preprocessor macros that are specific to
  SQLite begin with the "SQLITE_" prefix.  This is done to reduce the
  risk of name collisions with other software modules.


  Key Point:  Changes to compile-time options have the
  potential to affect makefiles in projects that do customized builds of
  SQLite. These changes should have zero impact on application code and for
  most projects which use a standard, default build of SQLite.

2.0 Fully Backwards-Compatible Enhancements
  In addition to the incompatible changes listed above, SQLite
  version 3.6.0 adds the following backwards compatible changes and
  enhancements:


  

  The new sqlite3_config() interface allows an application
  to customize the behavior of SQLite at run-time.  Customizations possible
  using sqlite3_config() include the following:
  
  Specify an alternative mutex implementation using the
  SQLITE_CONFIG_MUTEX verb with the sqlite3_mutex_methods object.
  Specify an alternative malloc implementation using the
  SQLITE_CONFIG_MALLOC verb with the sqlite3_mem_methods object.
  Partially or fully disable the use of mutexes using
  SQLITE_CONFIG_SINGLETHREAD, SQLITE_CONFIG_MULTITHREAD and
  SQLITE_CONFIG_SERIALIZED.
  
  

  A new flag SQLITE_OPEN_NOMUTEX is made available to the
  sqlite3_open_v2() interface.

  The new sqlite3_status() interface allows an application to query
  the performance status of SQLite at runtime.
  

  The sqlite3_memory_used() and sqlite3_memory_highwater()
  interfaces are deprecated.  The equivalent functionality is now available
  through sqlite3_status().

  The sqlite3_initialize() interface can be called to explicitly
  initialize the SQLite subsystem.  The sqlite3_initialize() interface is
  called automatically when invoking certain interfaces so the use of
  sqlite3_initialize() is not required, but it is recommended.

  The sqlite3_shutdown() interface causes SQLite to release any
  system resources (memory allocations, mutexes, open file handles)
  that might have been allocated by sqlite3_initialize().

  The sqlite3_next_stmt() interface allows an application to discover
  all prepared statements associated with a database connection.

  Added the page_count PRAGMA for returning the size of the underlying
  database file in pages.

  Added a new R*Tree index extension.


c2: Defense Against The Dark Arts
c5: security.html
c4:
1. SQLite Always Validates Its Inputs


SQLite should never crash, overflow a buffer, leak memory,
or exhibit any other harmful behavior, even when presented with
maliciously malformed SQL inputs or database files.  SQLite should
always detect erroneous inputs and raise an error, not crash or
corrupt memory.
Any malfunction caused by an SQL input or database file
is considered a serious bug and will be promptly addressed when
brought to the attention of the SQLite developers.  SQLite is
extensively fuzz-tested to help ensure that it is resistant
to these kinds of errors.


Nevertheless, bugs happen.
If you are writing an application that sends untrusted SQL inputs
or database files to SQLite, there are additional steps you can take
to help reduce the attack surface and
prevent zero-day exploits caused by undetected bugs.

1.1. Untrusted SQL Inputs

Applications that accept untrusted SQL inputs should take the following
precautions:



Set the SQLITE_DBCONFIG_DEFENSIVE flag.
This prevents ordinary SQL statements from deliberately corrupting the 
database file.  SQLite should be proof against attacks that involve both
malicious SQL inputs and a maliciously corrupted database file at the
same time.  Nevertheless, denying a script-only attacker access to 
corrupt database inputs provides an extra layer of defense.


Reduce the limits that SQLite imposes on inputs.  This can help prevent
denial of service attacks and other kinds of mischief that can occur
as a result of unusually large inputs.  You can do this either at compile-time
using -DSQLITE_MAX_... options, or at run-time using the
sqlite3_limit() interface.  Most applications can reduce limits
dramatically without impacting functionality.  The table below
provides some suggestions, though exact values will vary depending
on the application:


Limit SettingDefault ValueHigh-security Value
LIMIT_LENGTH1,000,000,0001,000,000
LIMIT_SQL_LENGTH1,000,000,000100,000
LIMIT_COLUMN2,000100
LIMIT_EXPR_DEPTH1,00010
LIMIT_COMPOUND_SELECT5003
LIMIT_VDBE_OP250,000,00025,000
LIMIT_FUNCTION_ARG1278
LIMIT_ATTACH100
LIMIT_LIKE_PATTERN_LENGTH50,00050
LIMIT_VARIABLE_NUMBER99910
LIMIT_TRIGGER_DEPTH1,00010



Consider using the sqlite3_set_authorizer() interface to limit
the scope of SQL that will be processed.  For example, an application
that does not need to change the database schema might add an
sqlite3_set_authorizer() callback that causes any CREATE or DROP
statement to fail.


The SQL language is very powerful, and so it is always possible for
malicious SQL inputs (or erroneous SQL inputs caused by an application
bug) to submit SQL that runs for a very long time.  To prevent this
from becoming a denial-of-service attack, consider using the
sqlite3_progress_handler() interface to invoke a callback periodically
as each SQL statement runs, and have that callback return non-zero to
abort the statement if the statement runs for too long.  Alternatively,
set a timer in a separate thread and invoke sqlite3_interrupt() when
the timer goes off to prevent the SQL statement from running forever.


Limit the maximum amount of memory that SQLite will allocate using
the sqlite3_hard_heap_limit64() interface.  This helps prevent
denial-of-service attacks.  To find out how much heap space an
application actually needs, run the it against typical inputs and
then measure the maximum instantaneous memory usage with the 
sqlite3_memory_highwater() interface.  Set the hard heap limit
to the maximum observed instantaneous memory usage plus some margin.


For embedded systems, consider compiling SQLite with the
-DSQLITE_ENABLE_MEMSYS5 option and then providing SQLite with
a fixed chunk of memory to use as its heap via the
sqlite3_config(SQLITE_CONFIG_HEAP) interface. This will
prevent malicious SQL from executing a denial-of-service attack
by using an excessive amount of memory. If (say) 5 MB of memory
is provided for SQLite to use, once that much has been consumed,
SQLite will start returning SQLITE_NOMEM errors rather than
soaking up memory needed by other parts of the application.
This also sandboxes SQLite's memory so that a write-after-free
error in some other part of the application will not cause
problems for SQLite, or vice versa.



To control memory usage in the printf() SQL function, compile
with "-DSQLITE_PRINTF_PRECISION_LIMIT=100000" or some similarly
reasonable value.
This #define limits the width and precision for %-substitutions in the
printf() function, and thus prevents a hostile SQL statement from
consuming large amounts of RAM via constructs such as
"printf('%1000000000s','hi')".

Note that SQLite uses its built-in printf() internally to help it
format the sql column in the sqlite_schema table.  For that reason,
no table, index, view, or trigger definition can be much larger than the
precision limit.  You can set a precision limit of less than 100000,
but be careful that whatever precision limit you use is at least as
long as the longest CREATE statement in your schema.






1.2. Untrusted SQLite Database Files

Applications that read or write SQLite database files of uncertain
provenance should take precautions enumerated below.

Even if the application does not deliberately accept database files 
from untrusted sources, beware of attacks in which a local 
database file is altered.  For best security, any database file which 
might have ever been writable by an agent in a different security domain
should be treated as suspect.



If the application includes any custom SQL functions or 
custom virtual tables that have side effects or that might leak
privileged information, then the application should use one or more
of the techniques below to prevent a maliciously crafted database
schema from surreptitiously running those SQL functions and/or
virtual tables for nefarious purposes:

 Invoke sqlite3_db_config(db,SQLITE_DBCONFIG_TRUSTED_SCHEMA,0,0)
     on each database connection as soon as it is opened.
 Run the PRAGMA trusted_schema=OFF statement on each database connection
     as soon as it is opened.
 Compile SQLite using the -DSQLITE_TRUSTED_SCHEMA=0 compile-time option.
 Disable the surreptitious use of custom SQL functions and virtual tables
     by setting the SQLITE_DIRECTONLY flag on all custom SQL functions and
     the SQLITE_VTAB_DIRECTONLY flag on all custom virtual tables.



If the application does not use triggers or views, consider disabling the
unused capabilities with:

sqlite3_db_config(db,SQLITE_DBCONFIG_ENABLE_TRIGGER,0,0);
sqlite3_db_config(db,SQLITE_DBCONFIG_ENABLE_VIEW,0,0);





For reading database files that are unusually high-risk, such as database
files that are received from remote machines, and possibly from anonymous
contributors, the following extra precautions
might be justified.  These added defenses come with performance costs,
however, and so may not be appropriate in every situation:



Run PRAGMA integrity_check or PRAGMA quick_check on the database
as the first SQL statement after opening the database files and
prior to running any other SQL statements.  Reject and refuse to
process any database file containing errors.


Enable the PRAGMA cell_size_check=ON setting.



Do not enable memory-mapped I/O.
In other words, make sure that PRAGMA mmap_size=0.



2. Summary


The precautions above are not required in order to use SQLite safely
with potentially hostile inputs.
However, they do provide an extra layer of defense against zero-day
exploits and are encouraged for applications that pass data from
untrusted sources into SQLite.


c2: The Advantages Of Flexible Typing
c5: flextypegood.html
c4:
1. Introduction

SQLite provides developers with the freedom to store content in
any desired format, regardless of the declared datatype of the column.
Some people find this feature troublesome.  Some developers are shocked
to discover that it is possible to insert text into a column marked INTEGER.

This article advocates in favor of the flexible type rules
in SQLite.

2. About Flexible Typing

Details regarding the flexible type system of SQLite are found
in the separate Datatypes In SQLite document.  Here is a quick
summary:


 Datatype names on column definitions are optional.  A column definition
        can consist of just the column name and nothing else.
 When datatype names are provided, they can be just about any text.
        SQLite attempts to deduce the preferred datatype for the column based
        on the datatype name in the column definition, but that preferred
        datatype is advisory, not mandatory.  The preferred datatype is
        call the "column affinity".
 An attempt is made to transform incoming data into the preferred
        datatype of the column.  (All SQL database engines do this, not
        just SQLite.)  If this transformation is successful, all is well.
        But if unsuccessful, instead of raising an error, SQLite just stores
        the content using its original datatype.
 The above can lead to situations that advocates of rigid typing
        find incommodious:
        
        Column DatatypeTypes Allowed In That Column
        INTEGERINTEGER, REAL, TEXT, BLOB
        REALREAL, TEXT, BLOB
        TEXTTEXT, BLOB
        BLOBINTEGER, REAL, TEXT, BLOB
        
 Note that an INTEGER or REAL value will never end up being stored
        in a TEXT column, since an INTEGER or REAL value can and always will
        be converted into its equivalent TEXT representation.  Similarly,
        and INTEGER will never be stored in a REAL column because it will
        always be converted into a REAL.  But TEXT does not always look like
        and INTEGER or REAL value and so cannot always be converted.  And a
        BLOB cannot be converted into anything and nothing else can be
        converted into a BLOB.



3. Cases Where Flexible Typing Is Useful

Some readers, upon first encountering flexible typing in SQLite, ask
themselves "how could this ever be useful?"  Here is an
attempt to answer that question:

3.1. Attribute tables

 Many applications, especially those that use SQLite as an
application file format, need a place to store miscellaneous attributes
such as thumbnail images (as BLOB values), short pieces of text (such
as the user's name), as well as numeric, date, and JSON values.  It is
convenient to create a single table to handle this storage:

CREATE TABLE attribute(name TEXT PRIMARY KEY, value) WITHOUT ROWID;


Without flexible typing, such a table would need to be more complex,
with separate columns for each possible type of data.  Flexible typing
of the "value" column makes the table conceptually simpler, more
space-efficient, and easier to access and update.

In the Fossil version control system, each
repository has a CONFIG table that is used to store all kinds of settings
with every possible datatype.  The user-specific configuration file
for Fossil (the ~/.fossil file) is a separate SQLite database that contains
a single attribute table hold the user-specific state across all
repositories.

 Some applications use an SQLite database as a pure key-value store
The database schema contains a single table that looks something like this:

CREATE TABLE storage(name TEXT PRIMARY KEY, value ANYTHING);


3.2. The "value" column output from the json_tree virtual tables

The json_tree and json_each table-valued functions that are
built into SQLite both have a "value" column that can hold values of
type INTEGER, REAL, or TEXT depending on the type of the corresponding
JSON field.  For example:

SELECT typeof(value) FROM json_each('{"a":1,"b":2.5,"c":"hello"}');


The query above returns three rows of one column with values
"integer", "real", and "text", respectively.

3.3. Storage for dirty data

 Analysts sometimes encounter CSV files where some columns contain
a mixture of integer, real, and text data.  CSV file that are obtained
from Excel spreadsheet exports commonly have this trait, for example.
When importing such "dirty data" into an SQL database, it is convenient
to have flexible columns to import into.

 Dirty data is not restricted to CSV files coming out of Excel, of
course.  There are many data sources in which a single field might
contain a mix of types.  For example, a data column might the number
of seconds since 1970 sometimes, or as a text date string in other cases.
It is desirable to clean up these inconsistent representations,
but at the same time it is convenient to be able to store all the different
representations in the same column of the intermediate database while the
cleanup is underway.

3.4. Dynamic programming languages

SQLite began as a TCL extension that later escaped into the wild.
TCL is a dynamic language in the sense that the programmer does not need
to be aware of datatypes.  Under the hood, TCL keeps careful track of the
datatype of every value, but to the developer and user of a TCL program,
everything looks like a string.  Flexible typing is a natural fit for
use with dynamic programming languages like TCL and others, since with
a dynamic programming language, you can not always predict in advance what
datatype a variable will hold.  So when you need to store the value of that
variable into the database, having a database that is supports flexible
typing makes storage much easier.

3.5. Data typename cross-compatibility

Every SQL database engine seems to have its own unique set of supported
datatype names:


 BIGINT
 UNSIGNED SMALL INT
 TEXT
 VARCHAR
 VARYING CHARACTER
 NATIONAL VARYING CHARACTER
 NVARCHAR
 JSON
 REAL
 FLOAT
 DOUBLE PRECISION
  ... and so forth ...


The fact that SQLite will accept any of these names as a valid typename,
and let you store any kind of content into the column, increases the chances
that a script written to run on some other SQL database engine will also
work in SQLite.

3.6. Repurposing unused or disused columns in legacy databases

 Because an SQLite database file is a single file on disk, some
applications use SQLite as an application file format.  This means
that a single instance of the application might, over the course of its
life, talk to hundreds or thousands of separate databases, each in a separate
file.  When such applications evolve over years, some columns in the
underlying database will have their meanings altered subtly.  Or, it might
be desirable to repurpose an existing column to serve two or more purposes.
This is much easier to do if the column has a flexible datatype.

4. Perceived Disadvantages of Flexible Typing (With Rebuttals)

 The following perceived disadvantages of flexible typing were
    gleaned and compiled from countless posts on Hacker News and
    Reddit and similar forums where developers discuss these sorts of
    things.  If you can think of other reasons why flexible typing
    is a bad idea, please contact the SQLite developers or leave a
    post on the SQLite Forum so
    that your idea can be added to the list.


4.1. We've never done it that way before

 Many skeptics of flexible typing simply express
shock and disbelief, without offering any rationale for why they think
flexible typing is a bad idea.  Without supporting arguments, one must
assume their reason for not liking flexible typing is that it is different
from what they are used to.

 Presumably, many developers who are aghast at SQLite's flexible
typing feel this way because they have just never encountered anything
like it before.  All prior exposure to databases and especially SQL
databases has involved rigid typing, and the readers mental model of
SQL includes rigid typing as a fundamental feature.  Flexible typing
upsets their world-view.

 Yes, flexible typing is a new way of thinking about data in an
SQL database.  But new is not necessary bad.  Sometimes, and I think
especially in the case of flexible typing, innovation leads to improvement.

4.2. Rigid type enforcement helps prevent application bugs

 It has become a point of doctrine among many programmers that the
best way to prevent application bugs is strict type enforcement.  But I
find no evidence in support of this.

 To be sure, strict type enforcement does help prevent some kinds of
bugs in lower-level languages like C and C++ that present a model that is
close to machine hardware.  But this does not seem to
be the case for higher-abstraction languages in which all data is 
passed around in a "Value" superclass of some kind which is subclassed
for the various lower-level data types.  When everything is a Value
object, specific datatypes cease to be important.

 This technical note is being writing by the original author of SQLite.
I having been writing TCL programs for 27 years.  TCL has no type enforcement
whatsoever.  The "Value" class in TCL (called Tcl_Obj) can hold many
different datatypes, but it presents the content to the program and to
the application user as a string.  And I've had a lot of bugs in
those TCL programs over the years.  But I do not recall a single instance
where the bugs might have been caught by a rigid type system.  I have
also written a lot of C code over a span of 35 years, not the
least of which is SQLite itself.  I have found the type system in C
to be very helpful at finding and preventing problems.  For the
Fossil Version Control System, which is written
in C, I have even implemented supplemental static analysis programs that
scan the Fossil source code prior to compilation, looking for problems
that compilers miss.  This works well for compiled programs.

 The SQL language model is a higher-level abstraction that C/C++.
In SQLite, every data item is stored in memory as an "sqlite3_value" object.
There are subclasses of this object for strings, integer, floating-point
numbers, blobs, and other representations.  Everything is passed around
inside the SQL language implemented by SQLite as "sqlite3_value" objects
so the underlying datatype does not really matter.  I have never found
rigid type enforcement to be helpful in languages like TCL and SQLite
that have a single "Value" superclass used to represent any data element.
Fossil makes extensive use of SQLite in its implementation.  There have
been many bugs in Fossil over its 14-years history, but I cannot recall
a single bug that might have been prevented by rigid type enforcement in
the SQLite.  C-language bugs might have been caught by better type enforcement
(which is why I wrote the supplemental source code scanners), but no
SQL bugs.

 Based on decades of experience, I reject the thesis that rigid
type enforcement helps prevent application bugs.  I will accept and
believe a slightly modified thesis:  Rigid type enforcement helps to
prevent applications bugs in languages that lack a single
top-level "Value" superclass.  But SQLite does have the
single "sqlite3_value" superclass, so that proverb does not apply.

4.3. Rigid type enforcement prevents data pollution

 Some people contend that if you have rigorous constraints on the
schema, and especially strict enforcement of column datatypes, this
will help prevent incorrect data from being added to the database.
This is not true.  It is true that type enforcement might help prevent
egregiously incorrect data from getting into the system.
But type enforcement is no help in prevent subtly incorrect data
from being recorded.

 So, for example, rigid type enforcement can successfully prevent
the customer name (text) from being inserted into integer
Customer.creditScore column.  On the other hand, if that mistake occurs,
it is very easy to spot the problem and find all affected rows.
But type enforcement is no help in preventing a bug where the customer
family name and given name are reversed, since both are text fields.

 (Aside:  Decades ago, I worked on a team where there was a woman
named "Merritt Tracy".  "Merritt" was her given name and "Tracy" was
her family name.  She reported that she spend an inordinate amount of
time and energy trying to correct databases that had "Tracy" has her 
given name and "Merritt" as her family name.)

 By suppressing easy-to-detect errors and passing through only the
hard-to-detect errors, rigid type enforcement can actually make it more
difficult to find and fix bugs.  Data errors tend to cluster.  If you have
20 different data sources, most of the data errors will usually come
from just 2 or 3 of those sources.  The presence of egregious
errors (such as text in an integer column) is a convenient early warning
signal that something is amiss.  The source of the problem can be 
tracked quickly and extra scrutiny applied to the source of the
egregious errors, thus hopefully also fixing the subtle errors too.
When egregious errors are suppressed, you lose an important signal
that helps you to detect and fix the subtle errors.

 Data errors are inevitable.  They will happen regardless of how
much type checking is done.  Rigid type enforcement can catch only
a small subset of those cases - the most obvious cases.  It does
nothing to help find and fix the more subtle cases.  And, by suppressing
the signal of which data sources are problematic, it can sometimes
make the subtle errors more difficult to locate.

4.4. Other SQL database engines don't work this way

 Because SQLite is less restrictive and allows you to do more things,
SQL scripts that work on other database engines will also usually work
on SQLite, but script written initially for SQLite might not work
on more restrictive database engines.  This can cause problems when
developers user SQLite for prototyping and testing and then migrate their
application to a more restrictive SQL engine for deployment.  If the
application was (unintentionally) taking advantage of the flexible
typing available in SQLite, then it will fail when migrated.

 People use this problem to argue that SQLite should be more
restrictive about datatypes.  But you could just as easily turn
that argument around and say that other database engines should be
more flexible with regard to datatypes.  The application was working
correctly under SQLite, prior to be migrated, after all.  If rigid
type enforcement is really all that useful, why did it break an
application that was previously working?

5. If You Insist On Rigid Type Enforcement...

As of SQLite version 3.37.0 (2021-11-27), SQLite supports this
development style using STRICT tables.

If you find a real-world case where STRICT tables prevented or
would have prevented a bug in an application, please post a message to the
SQLite Forum so that we can add your story
to this document.

6. Embrace Freedom

If flexible typing in an SQL database is a new concept to you,
I encourage you to give it a try.  It probably will not cause you
any problems and it might make your program simpler and easier to
write and maintain.  I think that even if you are skeptical at first,
if you will just give flexible typing a try, you will eventually
come to realize that it is a better approach and will start
encouraging other database vendors to support at least an ANY
datatype if not complete SQLite-style type flexibility.

Most of the time, flexible typing does not matter because a column
stores a single well-defined type.  But occasionally you will run
across situations where having a flexible type system makes the
solution to your problem cleaner and easier.


c2: Invalid UTF Policy
c5: invalidutf.html
c4:
1. Garbage In, Garbage Out


With regard to invalid UTF, SQLite follows a policy of
Garbage-In, Garbage-Out (GIGO). If you insert invalid UTF
into an SQLite database, then try to query that data, what you get back out
might not be exactly what you put in.  If you put garbage in, then you
may not complain if you get different garbage back out.


For the purposes of this discussion, "invalid UTF" can mean any of
the following circumstances:



Invalid surrogate pairs in UTF-16.


Invalid multi-byte sequences in UTF-8.


Using more bytes of UTF-8 than necessary to represent a single
code point.  (Example:  encoding 'A' as the two-byte sequence
0xc1, 0x01 instead of just a single 0x41 byte.)


NUL characters (U+0000) embedded in strings.


Invalid sequences of combining characters.


UTF-8 or UTF-16 bytes sequences that encode numbers that are not
defined Unicode characters.



1.1. Invalid UTF will never cause memory errors


If you insert invalid UTF into an SQLite database, then SQLite makes
no guarantees about what text you might get back out.  But it does
promise that invalid UTF will never cause memory errors
(array overruns, reads or writes of uninitialized memory, etc), at
least for the built-in processing of SQLite.
In other words, invalid UTF will not cause SQLite to crash.


This promise only applies to the core SQLite components, not
application-provided extensions, of course.
If an application adds new application-defined SQL functions or
virtual tables or collating sequences or other extensions, and a
database contains invalid UTF, then invalid UTF might get passed
into those extensions.  If the invalid UTF causes one of those
extensions to crash, then that is a problem with the extension,
not with SQLite.

2. No enforcement of text formatting rules


SQLite does not try to enforce UTF formatting rules.  You can
insert invalid UTF into a TEXT field and SQLite will not complain
about this.  It stores the invalid TEXT as best it can.  SQLite
sees its role in the world as a storage engine, not a text format
validation engine.

3. Best effort to preserve text


SQLite does not promise to always preserve invalid UTF, but it does
make an effort.  Generally speaking, if you insert invalid UTF into
SQLite, you will get the exact same byte sequence back out, as long
as you do not ask SQLite to transform the text in any way.


For example, if you insert some UTF-16LE with invalid surrogates into
a TEXT column of a table of a database that has PRAGMA encoding=UTF16LE,
then later query that column using sqlite3_column_text16(), you will 
probably get back the same exact invalid UTF-16.  But if you insert the
same invalid UTF-16LE content in a PRAGMA encoding=UTF8 database,
the content must be converted into UTF8 when it is stored, which could
cause irreversible changes to the content.  Or if you insert that
same invalid UTF-16LE content into a PRAGMA encoding=UTF16LE database
but then read it out using sqlite3_column_text(), then a UTF16 to
UTF8 conversion must occur during the read-out and that conversion might
introduce irreversible changes.


Or, suppose you are doing everything using UTF-8 (the most common case).
Invalid UTF-8 will normally pass through the database without any change
in its byte sequence.  However, if you try to transform the invalid
UTF-8 with SQL function like substr() or replace()
or if you try to do string matching with the LIKE operator, then
you might get unexpected results.


So, in other words, SQLite does not actively try to subvert your
invalid text.  But when you ask SQLite to make transformations of invalid
UTF, there are no guarantees that those transformations will be reversible
or even sensible.


c2: The Lemon LALR(1) Parser Generator
c5: lemon.html
c4:
1. Overview

The SQL language parser for SQLite is generated using a code-generator
program called "Lemon".  The Lemon program reads a grammar of the input
language and emits C-code to implement a parser for that language.


1.1. Lemon Source Files And Documentation

Lemon does not have its own source repository.  Rather, Lemon consists
of a few files in the SQLite source tree:



     lemon.html &rarr;
     The original detailed usage documentation and programmers reference
     for Lemon.

     lemon.c &rarr; The source code
     for the utility program that reads a grammar file and generates 
     corresponding parser C-code.

     lempar.c &rarr; A template
     for the generated parser C-code.  The "lemon" utility program reads this
     template and inserts additional code in order to generate a parser.


2. Advantages of Lemon

Lemon generates an LALR(1) parser.  Its operation is similar to the
more familiar tools Yacc and
Bison, but Lemon adds important
improvements, including:



     The grammar syntax is less error prone - using symbolic names for
     semantic values rather that the "$1"-style positional notation
     of Yacc.

     In Lemon, the tokenizer calls the parser.  Yacc operates the other
     way around, with the parser calling the tokenizer.  The Lemon
     approach is reentrant and threadsafe, whereas Yacc uses global 
     variables and is therefore neither.  Reentrancy is especially
     important for SQLite since some SQL statements make recursive calls
     to the parser.  For example, when parsing a CREATE TABLE statement,
     SQLite invokes the parser recursively to generate an INSERT statement
     to make a new entry in the sqlite_schema table.

     Lemon has the concept of a non-terminal destructor that can be
     used to reclaim memory or other resources following a syntax error
     or other aborted parse.


2.1. Use of Lemon Within SQLite

Lemon is used in two places in SQLite.

The primary use of Lemon is to create the SQL language parser.
A grammar file (parse.y) is
compiled by Lemon into parse.c and parse.h.  The parse.c file is
incorporated into the amalgamation without further modification.

Lemon is also used to generate the parser for the query pattern
expressions in the FTS5 extension.  In this case, the input grammar
file is fts5parse.y.

2.2. Lemon Customizations Especially For SQLite

One of the advantages of hosting code generator tools as part of
the project is that the tools can be optimized to serve specific needs of
the overall project.  Lemon has benefited from this effect. Over the years,
the Lemon parser generator has been extended and enhanced to provide
new capabilities and improved performance to SQLite.  A few of the
specific enhancements to Lemon that are specifically designed for use
by SQLite include:



Lemon has the concept of a "fallback" token.
The SQL language contains a large number of keywords and these keywords
have the potential to collide with identifier names.
Lemon has the ability to designate some keywords as being able to
"fallback" to an identifier.  If the keyword appears in the input token
stream in a context that would otherwise be a syntax error, the token
is automatically transformed into its fallback before the syntax error
is raised.  This feature allows the parser to be very forgiving of
reserved words used as identifiers, which is a problem that comes up
frequently in the SQL language.


In support of the 100% MC/DC testing goal for SQLite, 
the parser code generated by Lemon has no unreachable branches,
and contains extra (compile-time selected) instrumentation useful
for measuring test coverage.


Lemon supports conditional compilation of grammar file rules, so that
a different parser can be generated depending on compile-time options.


As a performance optimization, reduce actions in the Lemon input grammar
are allowed to contain comments of the form "/*A-overwrites-Z*/" to indicate
that the semantic value "A" on the right-hand side of the rule is allowed
to directly overwrite the semantic value "Z" on the left-hand side.
This simple optimization reduces the number of stack operations in the
push-down automaton used to parse the input grammar, and thus improve
performance of the parser.  It also makes the generated code a little smaller.


The parsing of SQL statements is a significant consumer of CPU cycles 
in any SQL database engine.  On-going efforts to optimize SQLite have caused
the developers to spend a lot of time tweaking Lemon to generate faster
parsers.  These efforts have benefited all users of the Lemon parser generator,
not just SQLite.  But if Lemon had been a separately maintained tool, it
would have been more difficult to make coordinated changes to both SQLite
and Lemon, and as a result not as much optimization would have been
accomplished.  Hence, the fact that the parser generator tool is included
in the source tree for SQLite has turned out to be a net benefit for both
the tool itself and for SQLite.

3. History Of Lemon

Lemon was originally written by D. Richard Hipp (also the creator of SQLite)
while he was in graduate school at Duke University between 1987 and 1992.
The original creation date of Lemon has been lost, but was probably sometime
around 1990.  Lemon generates an LALR(1) parser.  There was a companion 
LL(1) parser generator tool named "Lime", but the source code for Lime
has been lost.

The Lemon source code was originally written as separate source files,
and only later merged into a single "lemon.c" source file.

The author of Lemon and SQLite (Hipp) reports that his C programming
skills were greatly enhanced by studying John Ousterhout's original
source code to Tcl.  Hipp discovered and studied Tcl in 1993.  Lemon
was written before then, and SQLite afterwards.  There is a clear
difference in the coding styles of these two products, with SQLite seeming
to be cleaner, more readable, and easier to maintain.


c2: SQLite Unlock-Notify API
c5: unlock_notify.html
c4:
Using the sqlite3_unlock_notify() API


/* This example uses the pthreads API */
#include &lt;pthread.h&gt;

/*
** A pointer to an instance of this structure is passed as the user-context
** pointer when registering for an unlock-notify callback.
*/
typedef struct UnlockNotification UnlockNotification;
struct UnlockNotification {
  int fired;                         /* True after unlock event has occurred */
  pthread_cond_t cond;               /* Condition variable to wait on */
  pthread_mutex_t mutex;             /* Mutex to protect structure */
};

/*
** This function is an unlock-notify callback registered with SQLite.
*/
static void unlock_notify_cb(void **apArg, int nArg){
  int i;
  for(i=0; i&lt;nArg; i++){
    UnlockNotification *p = (UnlockNotification *)apArg[i];
    pthread_mutex_lock(&p-&gt;mutex);
    p-&gt;fired = 1;
    pthread_cond_signal(&p-&gt;cond);
    pthread_mutex_unlock(&p-&gt;mutex);
  }
}

/*
** This function assumes that an SQLite API call (either sqlite3_prepare_v2() 
** or sqlite3_step()) has just returned SQLITE_LOCKED. The argument is the
** associated database connection.
**
** This function calls sqlite3_unlock_notify() to register for an 
** unlock-notify callback, then blocks until that callback is delivered 
** and returns SQLITE_OK. The caller should then retry the failed operation.
**
** Or, if sqlite3_unlock_notify() indicates that to block would deadlock 
** the system, then this function returns SQLITE_LOCKED immediately. In 
** this case the caller should not retry the operation and should roll 
** back the current transaction (if any).
*/
static int wait_for_unlock_notify(sqlite3 *db){
  int rc;
  UnlockNotification un;

  /* Initialize the UnlockNotification structure. */
  un.fired = 0;
  pthread_mutex_init(&un.mutex, 0);
  pthread_cond_init(&un.cond, 0);

  /* Register for an unlock-notify callback. */
  rc = sqlite3_unlock_notify(db, unlock_notify_cb, (void *)&un);
  assert( rc==SQLITE_LOCKED || rc==SQLITE_OK );

  /* The call to sqlite3_unlock_notify() always returns either SQLITE_LOCKED 
  ** or SQLITE_OK. 
  **
  ** If SQLITE_LOCKED was returned, then the system is deadlocked. In this
  ** case this function needs to return SQLITE_LOCKED to the caller so 
  ** that the current transaction can be rolled back. Otherwise, block
  ** until the unlock-notify callback is invoked, then return SQLITE_OK.
  */
  if( rc==SQLITE_OK ){
    pthread_mutex_lock(&un.mutex);
    if( !un.fired ){
      pthread_cond_wait(&un.cond, &un.mutex);
    }
    pthread_mutex_unlock(&un.mutex);
  }

  /* Destroy the mutex and condition variables. */
  pthread_cond_destroy(&un.cond);
  pthread_mutex_destroy(&un.mutex);

  return rc;
}

/*
** This function is a wrapper around the SQLite function sqlite3_step().
** It functions in the same way as step(), except that if a required
** shared-cache lock cannot be obtained, this function may block waiting for
** the lock to become available. In this scenario the normal API step()
** function always returns SQLITE_LOCKED.
**
** If this function returns SQLITE_LOCKED, the caller should rollback
** the current transaction (if any) and try again later. Otherwise, the
** system may become deadlocked.
*/
int sqlite3_blocking_step(sqlite3_stmt *pStmt){
  int rc;
  while( SQLITE_LOCKED==(rc = sqlite3_step(pStmt)) ){
    rc = wait_for_unlock_notify(sqlite3_db_handle(pStmt));
    if( rc!=SQLITE_OK ) break;
    sqlite3_reset(pStmt);
  }
  return rc;
}

/*
** This function is a wrapper around the SQLite function sqlite3_prepare_v2().
** It functions in the same way as prepare_v2(), except that if a required
** shared-cache lock cannot be obtained, this function may block waiting for
** the lock to become available. In this scenario the normal API prepare_v2()
** function always returns SQLITE_LOCKED.
**
** If this function returns SQLITE_LOCKED, the caller should rollback
** the current transaction (if any) and try again later. Otherwise, the
** system may become deadlocked.
*/
int sqlite3_blocking_prepare_v2(
  sqlite3 *db,              /* Database handle. */
  const char *zSql,         /* UTF-8 encoded SQL statement. */
  int nSql,                 /* Length of zSql in bytes. */
  sqlite3_stmt **ppStmt,    /* OUT: A pointer to the prepared statement */
  const char **pz           /* OUT: End of parsed string */
){
  int rc;
  while( SQLITE_LOCKED==(rc = sqlite3_prepare_v2(db, zSql, nSql, ppStmt, pz)) ){
    rc = wait_for_unlock_notify(db);
    if( rc!=SQLITE_OK ) break;
  }
  return rc;
}




  When two or more connections access the same database in shared-cache 
  mode, read and write (shared and exclusive) locks on individual tables
  are used to ensure that concurrently executing transactions are kept 
  isolated. Before writing to a table, a write (exclusive) lock must be 
  obtained on that table. Before reading, a read (shared) lock must be
  obtained. A connection releases all held table locks when it concludes
  its transaction. If a connection cannot obtain a required lock, then
  the call to sqlite3_step() returns SQLITE_LOCKED.


  Although it is less common, a call to sqlite3_prepare() or
  sqlite3_prepare_v2() may also return SQLITE_LOCKED if it cannot obtain
  a read-lock on the sqlite_schema table of each attached database. These
  APIs need to read the schema data contained in the sqlite_schema table
  in order to compile SQL statements to sqlite3_stmt* objects.


  This article presents a technique using the SQLite sqlite3_unlock_notify()
  interface such that calls to sqlite3_step() and sqlite3_prepare_v2()
  block until the required locks are available instead of returning 
  SQLITE_LOCKED immediately. If the
  sqlite3_blocking_step() or sqlite3_blocking_prepare_v2() functions presented 
  to the left return SQLITE_LOCKED, this indicates that to block would 
  deadlock the system.


  The sqlite3_unlock_notify() API, which is only available if the library is
  compiled with the pre-processor symbol SQLITE_ENABLE_UNLOCK_NOTIFY defined,
  is documented here. This article is not a
  substitute for reading the full API documentation!


  The sqlite3_unlock_notify() interface is designed for use in systems
  that have a separate thread assigned to each database connection.  There
  is nothing in the implementation that prevents a single thread from running
  multiple database connections.  However, the sqlite3_unlock_notify()
  interface only works on a single connection at a time, so the lock
  resolution logic presented here will only work for a single 
  database connection per thread.

The sqlite3_unlock_notify() API


  After a call to sqlite3_step() or sqlite3_prepare_v2() returns 
  SQLITE_LOCKED, the sqlite3_unlock_notify() API may be invoked to register 
  for an unlock-notify callback. The unlock-notify callback is invoked by 
  SQLite after the database connection holding the table-lock that prevented
  the call to sqlite3_step() or sqlite3_prepare_v2() from succeeding has
  finished its transaction and released all locks. For example, if a call to
  sqlite3_step() is an attempt to read from table X, and some other connection
  Y is holding a write-lock on table X, then sqlite3_step() will return
  SQLITE_LOCKED. If sqlite3_unlock_notify() is then called, the unlock-notify
  callback will be invoked after connection Y's transaction is concluded. The
  connection that the unlock-notify callback is waiting on, in this case
  connection Y, is known as the "blocking connection".


  If a call to sqlite3_step() that attempts to write to a database table
  returns SQLITE_LOCKED, then more than one other connection may be holding 
  a read-lock on the database table in question. In this case SQLite simply
  selects one of those other connections arbitrarily and issues the 
  unlock-notify callback when that connection's transaction is finished.
  Whether the call to sqlite3_step() was blocked by one or many connections, 
  when the corresponding unlock-notify callback is issued it is not 
  guaranteed that the required lock is available, only that it may be.


  When the unlock-notify callback is issued, it is issued from within a
  call to sqlite3_step() (or sqlite3_close()) associated with the blocking 
  connection. It is illegal to invoke any sqlite3_XXX() API functions from
  within an unlock-notify callback. The expected use is that the unlock-notify
  callback will signal some other waiting thread or schedule some action
  to take place later.


  The algorithm used by the sqlite3_blocking_step() function is as follows:


   Call sqlite3_step() on the supplied statement handle. If the call
       returns anything other than SQLITE_LOCKED, then return this value
       to the caller. Otherwise, continue.

   Invoke sqlite3_unlock_notify() on the database connection handle
       associated with the supplied statement handle to register for an
       unlock-notify callback. If the call to unlock_notify() returns
       SQLITE_LOCKED, then return this value to the caller.

   Block until the unlock-notify callback is invoked by another thread.

   Call sqlite3_reset() on the statement handle. Since an 
       SQLITE_LOCKED error may only occur on the first call to sqlite3_step()
       (it is not possible for one call to sqlite3_step() to return 
       SQLITE_ROW and then the next SQLITE_LOCKED), the statement handle may 
       be reset at this point without affecting the results of the query
       from the point of view of the caller. If sqlite3_reset() were not
       called at this point, the next call to sqlite3_step() would return
       SQLITE_MISUSE.

   Return to step 1.



  The algorithm used by the sqlite3_blocking_prepare_v2() function is similar,
  except that step 4 (resetting the statement handle) is omitted.


Writer Starvation


  Multiple connections may hold a read-lock simultaneously.  If many threads
  are acquiring overlapping read-locks, it might be the case that at least
  one thread is always holding a read lock.
  Then a table waiting for a write-lock will wait forever.
  This scenario is called "writer starvation."


  SQLite helps applications avoid writer starvation. After any attempt to
  obtain a write-lock on a table fails (because one or more other 
  connections are holding read-locks), all attempts to open new transactions
  on the shared-cache fail until one of the following is true:


   The current writer concludes its transaction, OR
   The number of open read-transactions on the shared-cache drops to zero.



  Failed attempts to open new read-transactions return SQLITE_LOCKED to the
  caller. If the caller then calls sqlite3_unlock_notify() to register for
  an unlock-notify callback, the blocking connection is the connection that
  currently has an open write-transaction on the shared-cache. This prevents
  writer-starvation since if no new read-transactions may be opened and 
  assuming all existing read-transactions are eventually concluded, the 
  writer will eventually have an opportunity to obtain the required
  write-lock.

The pthreads API

   By the time sqlite3_unlock_notify() is invoked by
      wait_for_unlock_notify(), it is possible that the blocking connection
      that prevented the sqlite3_step() or sqlite3_prepare_v2() call from 
      succeeding has already finished its transaction. In this case, the 
      unlock-notify callback is invoked immediately, before 
      sqlite3_unlock_notify() returns. Or, it is possible that the 
      unlock-notify callback is invoked by a second thread after 
      sqlite3_unlock_notify() is called but before the thread starts waiting 
      to be asynchronously signaled. 

   Exactly how such a potential race-condition is handled depends on the
      threads and synchronization primitives interface used by the application.
      This example uses pthreads, the interface provided by modern UNIX-like 
      systems, including Linux.

   The pthreads interface provides the pthread_cond_wait() function. 
      This function allows the caller to simultaneously release a mutex
      and start waiting for an asynchronous signal. Using this function,
      a "fired" flag and a mutex, the race-condition described above may
      be eliminated as follows:

   When the unlock-notify callback is invoked, which may be before the
      thread that called sqlite3_unlock_notify() begins waiting for the 
      asynchronous signal, it does the following:

    
       Obtains the mutex.
       Sets the "fired" flag to true.
       Attempts to signal a waiting thread.
       Releases the mutex.
    

   When the wait_for_unlock_notify() thread is ready to begin waiting for
      the unlock-notify callback to arrive, it:

    
       Obtains the mutex.
       Checks if the "fired" flag has been set. If so, the unlock-notify
           callback has already been invoked. Release the mutex and continue.
       Atomically releases the mutex and begins waiting for the
           asynchronous signal. When the signal arrives, continue.
    

   This way, it doesn't matter if the unlock-notify callback has already
      been invoked, or is being invoked, when the wait_for_unlock_notify() 
      thread begins blocking.

Possible Enhancements

   The code in this article could be improved in at least two ways:

  
     It could manage thread priorities.
     It could handle a special case of SQLITE_LOCKED that can occur
         when dropping a table or index.
  

  
    Even though the sqlite3_unlock_notify() function only allows the caller
    to specify a single user-context pointer, an unlock-notify callback 
    is passed an array of such context pointers. This is because if when
    a blocking connection concludes its transaction, if there is more
    than one unlock-notify registered to call the same C function, the
    context-pointers are marshaled into an array and a single callback
    issued. If each thread were assigned a priority, then instead of just
    signaling the threads in arbitrary order as this implementation does,
    higher priority threads could be signaled before lower priority threads.

  
    If a "DROP TABLE" or "DROP INDEX" SQL command is executed, and the
    same database connection currently has one or more actively
    executing SELECT statements, then SQLITE_LOCKED is returned. If
    sqlite3_unlock_notify() is called in this case, then the specified
    callback will be invoked immediately. Re-attempting the "DROP
    TABLE" or "DROP INDEX" statement will return another SQLITE_LOCKED 
    error. In the implementation of sqlite3_blocking_step() shown to the
    left, this could cause an infinite loop.

  
    The caller could distinguish between this special "DROP TABLE|INDEX" 
    case and other cases by using extended error codes. When it is appropriate
    to call sqlite3_unlock_notify(), the extended error code is
    SQLITE_LOCKED_SHAREDCACHE. Otherwise, in the "DROP TABLE|INDEX" case,
    it is just plain SQLITE_LOCKED. Another solution might be to limit
    the number of times that any single query could be reattempted (to say 
    100). Although this might be less efficient than one might wish, the 
    situation in question is not likely to occur often.


c2: Syntax Diagrams For SQLite
c5: syntaxdiagrams.html
c4:
Syntax Diagrams For SQLite

aggregate-function-invocation:



References:&nbsp;&nbsp; expr&nbsp;&nbsp; filter-clause
See also:&nbsp;&nbsp; lang_aggfunc.html&nbsp;&nbsp; lang_expr.html#booleanexpr


alter-table-stmt:



Used by:&nbsp;&nbsp; sql-stmt
References:&nbsp;&nbsp; column-def
See also:&nbsp;&nbsp; lang_altertable.html


analyze-stmt:



Used by:&nbsp;&nbsp; sql-stmt
See also:&nbsp;&nbsp; lang_analyze.html


attach-stmt:



Used by:&nbsp;&nbsp; sql-stmt
References:&nbsp;&nbsp; expr
See also:&nbsp;&nbsp; lang_attach.html


begin-stmt:



Used by:&nbsp;&nbsp; sql-stmt
See also:&nbsp;&nbsp; lang_transaction.html


column-constraint:



Used by:&nbsp;&nbsp; column-def
References:&nbsp;&nbsp; conflict-clause&nbsp;&nbsp; expr&nbsp;&nbsp; foreign-key-clause&nbsp;&nbsp; literal-value&nbsp;&nbsp; signed-number
See also:&nbsp;&nbsp; gencol.html&nbsp;&nbsp; lang_altertable.html&nbsp;&nbsp; lang_createtable.html&nbsp;&nbsp; lang_createtable.html#tablecoldef


column-def:



Used by:&nbsp;&nbsp; alter-table-stmt&nbsp;&nbsp; create-table-stmt
References:&nbsp;&nbsp; column-constraint&nbsp;&nbsp; type-name
See also:&nbsp;&nbsp; lang_altertable.html&nbsp;&nbsp; lang_altertable.html#altertabaddcol&nbsp;&nbsp; lang_createtable.html&nbsp;&nbsp; lang_createtable.html#tablecoldef


column-name-list:



Used by:&nbsp;&nbsp; update-stmt&nbsp;&nbsp; update-stmt-limited&nbsp;&nbsp; upsert-clause
See also:&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_insert.html&nbsp;&nbsp; lang_update.html&nbsp;&nbsp; lang_upsert.html


comment-syntax:



See also:&nbsp;&nbsp; lang_comment.html


commit-stmt:



Used by:&nbsp;&nbsp; sql-stmt
See also:&nbsp;&nbsp; lang_transaction.html


common-table-expression:



Used by:&nbsp;&nbsp; compound-select-stmt&nbsp;&nbsp; delete-stmt&nbsp;&nbsp; delete-stmt-limited&nbsp;&nbsp; factored-select-stmt&nbsp;&nbsp; insert-stmt&nbsp;&nbsp; select-stmt&nbsp;&nbsp; simple-select-stmt&nbsp;&nbsp; update-stmt&nbsp;&nbsp; update-stmt-limited
References:&nbsp;&nbsp; select-stmt
See also:&nbsp;&nbsp; lang_aggfunc.html&nbsp;&nbsp; lang_altertable.html&nbsp;&nbsp; lang_attach.html&nbsp;&nbsp; lang_createindex.html&nbsp;&nbsp; lang_createtable.html&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_createview.html&nbsp;&nbsp; lang_delete.html&nbsp;&nbsp; lang_expr.html&nbsp;&nbsp; lang_insert.html&nbsp;&nbsp; lang_returning.html&nbsp;&nbsp; lang_select.html&nbsp;&nbsp; lang_select.html#compound&nbsp;&nbsp; lang_select.html#simpleselect&nbsp;&nbsp; lang_update.html&nbsp;&nbsp; lang_upsert.html&nbsp;&nbsp; lang_with.html&nbsp;&nbsp; partialindex.html


compound-operator:



Used by:&nbsp;&nbsp; factored-select-stmt&nbsp;&nbsp; select-stmt
See also:&nbsp;&nbsp; lang_aggfunc.html&nbsp;&nbsp; lang_altertable.html&nbsp;&nbsp; lang_attach.html&nbsp;&nbsp; lang_createindex.html&nbsp;&nbsp; lang_createtable.html&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_createview.html&nbsp;&nbsp; lang_delete.html&nbsp;&nbsp; lang_expr.html&nbsp;&nbsp; lang_insert.html&nbsp;&nbsp; lang_returning.html&nbsp;&nbsp; lang_select.html&nbsp;&nbsp; lang_select.html#compound&nbsp;&nbsp; lang_select.html#simpleselect&nbsp;&nbsp; lang_update.html&nbsp;&nbsp; lang_upsert.html&nbsp;&nbsp; lang_with.html&nbsp;&nbsp; partialindex.html


compound-select-stmt:



References:&nbsp;&nbsp; common-table-expression&nbsp;&nbsp; expr&nbsp;&nbsp; ordering-term&nbsp;&nbsp; select-core
See also:&nbsp;&nbsp; lang_select.html#compound


conflict-clause:



Used by:&nbsp;&nbsp; column-constraint&nbsp;&nbsp; table-constraint
See also:&nbsp;&nbsp; lang_altertable.html&nbsp;&nbsp; lang_conflict.html&nbsp;&nbsp; lang_createtable.html&nbsp;&nbsp; lang_createtable.html#notnullconst


create-index-stmt:



Used by:&nbsp;&nbsp; sql-stmt
References:&nbsp;&nbsp; expr&nbsp;&nbsp; indexed-column
See also:&nbsp;&nbsp; lang_createindex.html&nbsp;&nbsp; partialindex.html


create-table-stmt:



Used by:&nbsp;&nbsp; sql-stmt
References:&nbsp;&nbsp; column-def&nbsp;&nbsp; select-stmt&nbsp;&nbsp; table-constraint&nbsp;&nbsp; table-options
See also:&nbsp;&nbsp; lang_createtable.html


create-trigger-stmt:



Used by:&nbsp;&nbsp; sql-stmt
References:&nbsp;&nbsp; delete-stmt&nbsp;&nbsp; expr&nbsp;&nbsp; insert-stmt&nbsp;&nbsp; select-stmt&nbsp;&nbsp; update-stmt
See also:&nbsp;&nbsp; lang_createtrigger.html


create-view-stmt:



Used by:&nbsp;&nbsp; sql-stmt
References:&nbsp;&nbsp; select-stmt
See also:&nbsp;&nbsp; lang_createview.html


create-virtual-table-stmt:



Used by:&nbsp;&nbsp; sql-stmt
See also:&nbsp;&nbsp; lang_createvtab.html


cte-table-name:



Used by:&nbsp;&nbsp; recursive-cte&nbsp;&nbsp; with-clause
See also:&nbsp;&nbsp; lang_with.html&nbsp;&nbsp; lang_with.html#recursivecte


delete-stmt:



Used by:&nbsp;&nbsp; create-trigger-stmt&nbsp;&nbsp; sql-stmt
References:&nbsp;&nbsp; common-table-expression&nbsp;&nbsp; expr&nbsp;&nbsp; qualified-table-name&nbsp;&nbsp; returning-clause
See also:&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_delete.html


delete-stmt-limited:



Used by:&nbsp;&nbsp; sql-stmt
References:&nbsp;&nbsp; common-table-expression&nbsp;&nbsp; expr&nbsp;&nbsp; ordering-term&nbsp;&nbsp; qualified-table-name&nbsp;&nbsp; returning-clause
See also:&nbsp;&nbsp; lang_delete.html


detach-stmt:



Used by:&nbsp;&nbsp; sql-stmt
See also:&nbsp;&nbsp; lang_detach.html


drop-index-stmt:



Used by:&nbsp;&nbsp; sql-stmt
See also:&nbsp;&nbsp; lang_dropindex.html


drop-table-stmt:



Used by:&nbsp;&nbsp; sql-stmt
See also:&nbsp;&nbsp; lang_droptable.html


drop-trigger-stmt:



Used by:&nbsp;&nbsp; sql-stmt
See also:&nbsp;&nbsp; lang_droptrigger.html


drop-view-stmt:



Used by:&nbsp;&nbsp; sql-stmt
See also:&nbsp;&nbsp; lang_dropview.html


expr:



Used by:&nbsp;&nbsp; aggregate-function-invocation&nbsp;&nbsp; attach-stmt&nbsp;&nbsp; column-constraint&nbsp;&nbsp; compound-select-stmt&nbsp;&nbsp; create-index-stmt&nbsp;&nbsp; create-trigger-stmt&nbsp;&nbsp; delete-stmt&nbsp;&nbsp; delete-stmt-limited&nbsp;&nbsp; factored-select-stmt&nbsp;&nbsp; filter-clause&nbsp;&nbsp; frame-spec&nbsp;&nbsp; indexed-column&nbsp;&nbsp; insert-stmt&nbsp;&nbsp; join-constraint&nbsp;&nbsp; ordering-term&nbsp;&nbsp; over-clause&nbsp;&nbsp; result-column&nbsp;&nbsp; returning-clause&nbsp;&nbsp; select-core&nbsp;&nbsp; select-stmt&nbsp;&nbsp; simple-function-invocation&nbsp;&nbsp; simple-select-stmt&nbsp;&nbsp; table-constraint&nbsp;&nbsp; table-or-subquery&nbsp;&nbsp; update-stmt&nbsp;&nbsp; update-stmt-limited&nbsp;&nbsp; upsert-clause&nbsp;&nbsp; window-defn&nbsp;&nbsp; window-function-invocation
References:&nbsp;&nbsp; filter-clause&nbsp;&nbsp; literal-value&nbsp;&nbsp; over-clause&nbsp;&nbsp; raise-function&nbsp;&nbsp; select-stmt&nbsp;&nbsp; type-name
See also:&nbsp;&nbsp; lang_aggfunc.html&nbsp;&nbsp; lang_altertable.html&nbsp;&nbsp; lang_attach.html&nbsp;&nbsp; lang_createindex.html&nbsp;&nbsp; lang_createtable.html&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_createview.html&nbsp;&nbsp; lang_delete.html&nbsp;&nbsp; lang_expr.html&nbsp;&nbsp; lang_insert.html&nbsp;&nbsp; lang_returning.html&nbsp;&nbsp; lang_select.html&nbsp;&nbsp; lang_select.html#compound&nbsp;&nbsp; lang_select.html#simpleselect&nbsp;&nbsp; lang_update.html&nbsp;&nbsp; lang_upsert.html&nbsp;&nbsp; lang_with.html&nbsp;&nbsp; partialindex.html


factored-select-stmt:



References:&nbsp;&nbsp; common-table-expression&nbsp;&nbsp; compound-operator&nbsp;&nbsp; expr&nbsp;&nbsp; ordering-term&nbsp;&nbsp; select-core
See also:&nbsp;&nbsp; lang_select.html


filter-clause:



Used by:&nbsp;&nbsp; aggregate-function-invocation&nbsp;&nbsp; expr&nbsp;&nbsp; window-function-invocation
References:&nbsp;&nbsp; expr
See also:&nbsp;&nbsp; lang_aggfunc.html&nbsp;&nbsp; lang_altertable.html&nbsp;&nbsp; lang_attach.html&nbsp;&nbsp; lang_createindex.html&nbsp;&nbsp; lang_createtable.html&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_createview.html&nbsp;&nbsp; lang_delete.html&nbsp;&nbsp; lang_expr.html&nbsp;&nbsp; lang_insert.html&nbsp;&nbsp; lang_returning.html&nbsp;&nbsp; lang_select.html&nbsp;&nbsp; lang_select.html#compound&nbsp;&nbsp; lang_select.html#simpleselect&nbsp;&nbsp; lang_update.html&nbsp;&nbsp; lang_upsert.html&nbsp;&nbsp; lang_with.html&nbsp;&nbsp; partialindex.html


foreign-key-clause:



Used by:&nbsp;&nbsp; column-constraint&nbsp;&nbsp; table-constraint
See also:&nbsp;&nbsp; lang_altertable.html&nbsp;&nbsp; lang_altertable.html#altertabaddcol&nbsp;&nbsp; lang_createtable.html


frame-spec:



Used by:&nbsp;&nbsp; over-clause&nbsp;&nbsp; window-defn
References:&nbsp;&nbsp; expr
See also:&nbsp;&nbsp; lang_aggfunc.html&nbsp;&nbsp; lang_altertable.html&nbsp;&nbsp; lang_attach.html&nbsp;&nbsp; lang_createindex.html&nbsp;&nbsp; lang_createtable.html&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_createview.html&nbsp;&nbsp; lang_delete.html&nbsp;&nbsp; lang_expr.html&nbsp;&nbsp; lang_insert.html&nbsp;&nbsp; lang_returning.html&nbsp;&nbsp; lang_select.html&nbsp;&nbsp; lang_select.html#compound&nbsp;&nbsp; lang_select.html#simpleselect&nbsp;&nbsp; lang_update.html&nbsp;&nbsp; lang_upsert.html&nbsp;&nbsp; lang_with.html&nbsp;&nbsp; partialindex.html


indexed-column:



Used by:&nbsp;&nbsp; create-index-stmt&nbsp;&nbsp; table-constraint&nbsp;&nbsp; upsert-clause
References:&nbsp;&nbsp; expr
See also:&nbsp;&nbsp; lang_createindex.html&nbsp;&nbsp; lang_createtable.html&nbsp;&nbsp; lang_createtable.html#primkeyconst&nbsp;&nbsp; lang_createtable.html#uniqueconst&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_insert.html&nbsp;&nbsp; lang_upsert.html&nbsp;&nbsp; partialindex.html


insert-stmt:



Used by:&nbsp;&nbsp; create-trigger-stmt&nbsp;&nbsp; sql-stmt
References:&nbsp;&nbsp; common-table-expression&nbsp;&nbsp; expr&nbsp;&nbsp; returning-clause&nbsp;&nbsp; select-stmt&nbsp;&nbsp; upsert-clause
See also:&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_insert.html


join-clause:



Used by:&nbsp;&nbsp; select-core&nbsp;&nbsp; select-stmt&nbsp;&nbsp; table-or-subquery&nbsp;&nbsp; update-stmt&nbsp;&nbsp; update-stmt-limited
References:&nbsp;&nbsp; join-constraint&nbsp;&nbsp; join-operator&nbsp;&nbsp; table-or-subquery
See also:&nbsp;&nbsp; lang_aggfunc.html&nbsp;&nbsp; lang_altertable.html&nbsp;&nbsp; lang_attach.html&nbsp;&nbsp; lang_createindex.html&nbsp;&nbsp; lang_createtable.html&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_createview.html&nbsp;&nbsp; lang_delete.html&nbsp;&nbsp; lang_expr.html&nbsp;&nbsp; lang_insert.html&nbsp;&nbsp; lang_returning.html&nbsp;&nbsp; lang_select.html&nbsp;&nbsp; lang_select.html#compound&nbsp;&nbsp; lang_select.html#simpleselect&nbsp;&nbsp; lang_update.html&nbsp;&nbsp; lang_upsert.html&nbsp;&nbsp; lang_with.html&nbsp;&nbsp; partialindex.html


join-constraint:



Used by:&nbsp;&nbsp; join-clause
References:&nbsp;&nbsp; expr
See also:&nbsp;&nbsp; lang_aggfunc.html&nbsp;&nbsp; lang_altertable.html&nbsp;&nbsp; lang_attach.html&nbsp;&nbsp; lang_createindex.html&nbsp;&nbsp; lang_createtable.html&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_createview.html&nbsp;&nbsp; lang_delete.html&nbsp;&nbsp; lang_expr.html&nbsp;&nbsp; lang_insert.html&nbsp;&nbsp; lang_returning.html&nbsp;&nbsp; lang_select.html&nbsp;&nbsp; lang_select.html#compound&nbsp;&nbsp; lang_select.html#fromclause&nbsp;&nbsp; lang_select.html#simpleselect&nbsp;&nbsp; lang_update.html&nbsp;&nbsp; lang_upsert.html&nbsp;&nbsp; lang_with.html&nbsp;&nbsp; partialindex.html


join-operator:



Used by:&nbsp;&nbsp; join-clause
See also:&nbsp;&nbsp; lang_aggfunc.html&nbsp;&nbsp; lang_altertable.html&nbsp;&nbsp; lang_attach.html&nbsp;&nbsp; lang_createindex.html&nbsp;&nbsp; lang_createtable.html&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_createview.html&nbsp;&nbsp; lang_delete.html&nbsp;&nbsp; lang_expr.html&nbsp;&nbsp; lang_insert.html&nbsp;&nbsp; lang_returning.html&nbsp;&nbsp; lang_select.html&nbsp;&nbsp; lang_select.html#compound&nbsp;&nbsp; lang_select.html#fromclause&nbsp;&nbsp; lang_select.html#nonstd&nbsp;&nbsp; lang_select.html#simpleselect&nbsp;&nbsp; lang_update.html&nbsp;&nbsp; lang_upsert.html&nbsp;&nbsp; lang_with.html&nbsp;&nbsp; partialindex.html


literal-value:



Used by:&nbsp;&nbsp; column-constraint&nbsp;&nbsp; expr
See also:&nbsp;&nbsp; lang_aggfunc.html&nbsp;&nbsp; lang_altertable.html&nbsp;&nbsp; lang_attach.html&nbsp;&nbsp; lang_createindex.html&nbsp;&nbsp; lang_createtable.html&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_createview.html&nbsp;&nbsp; lang_delete.html&nbsp;&nbsp; lang_expr.html&nbsp;&nbsp; lang_insert.html&nbsp;&nbsp; lang_returning.html&nbsp;&nbsp; lang_select.html&nbsp;&nbsp; lang_select.html#compound&nbsp;&nbsp; lang_select.html#simpleselect&nbsp;&nbsp; lang_update.html&nbsp;&nbsp; lang_upsert.html&nbsp;&nbsp; lang_with.html&nbsp;&nbsp; partialindex.html


numeric-literal:



See also:&nbsp;&nbsp; lang_expr.html#litvalue


ordering-term:



Used by:&nbsp;&nbsp; compound-select-stmt&nbsp;&nbsp; delete-stmt-limited&nbsp;&nbsp; factored-select-stmt&nbsp;&nbsp; over-clause&nbsp;&nbsp; select-stmt&nbsp;&nbsp; simple-select-stmt&nbsp;&nbsp; update-stmt-limited&nbsp;&nbsp; window-defn
References:&nbsp;&nbsp; expr
See also:&nbsp;&nbsp; lang_aggfunc.html&nbsp;&nbsp; lang_altertable.html&nbsp;&nbsp; lang_attach.html&nbsp;&nbsp; lang_createindex.html&nbsp;&nbsp; lang_createtable.html&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_createview.html&nbsp;&nbsp; lang_delete.html&nbsp;&nbsp; lang_expr.html&nbsp;&nbsp; lang_insert.html&nbsp;&nbsp; lang_returning.html&nbsp;&nbsp; lang_select.html&nbsp;&nbsp; lang_select.html#compound&nbsp;&nbsp; lang_select.html#simpleselect&nbsp;&nbsp; lang_update.html&nbsp;&nbsp; lang_upsert.html&nbsp;&nbsp; lang_with.html&nbsp;&nbsp; partialindex.html


over-clause:



Used by:&nbsp;&nbsp; expr
References:&nbsp;&nbsp; expr&nbsp;&nbsp; frame-spec&nbsp;&nbsp; ordering-term
See also:&nbsp;&nbsp; lang_aggfunc.html&nbsp;&nbsp; lang_altertable.html&nbsp;&nbsp; lang_attach.html&nbsp;&nbsp; lang_createindex.html&nbsp;&nbsp; lang_createtable.html&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_createview.html&nbsp;&nbsp; lang_delete.html&nbsp;&nbsp; lang_expr.html&nbsp;&nbsp; lang_insert.html&nbsp;&nbsp; lang_returning.html&nbsp;&nbsp; lang_select.html&nbsp;&nbsp; lang_select.html#compound&nbsp;&nbsp; lang_select.html#simpleselect&nbsp;&nbsp; lang_update.html&nbsp;&nbsp; lang_upsert.html&nbsp;&nbsp; lang_with.html&nbsp;&nbsp; partialindex.html


pragma-stmt:



Used by:&nbsp;&nbsp; sql-stmt
References:&nbsp;&nbsp; pragma-value
See also:&nbsp;&nbsp; pragma.html#syntax


pragma-value:



Used by:&nbsp;&nbsp; pragma-stmt
References:&nbsp;&nbsp; signed-number
See also:&nbsp;&nbsp; pragma.html#syntax


qualified-table-name:



Used by:&nbsp;&nbsp; delete-stmt&nbsp;&nbsp; delete-stmt-limited&nbsp;&nbsp; update-stmt&nbsp;&nbsp; update-stmt-limited
See also:&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_delete.html&nbsp;&nbsp; lang_indexedby.html&nbsp;&nbsp; lang_update.html


raise-function:



Used by:&nbsp;&nbsp; expr
See also:&nbsp;&nbsp; lang_aggfunc.html&nbsp;&nbsp; lang_altertable.html&nbsp;&nbsp; lang_attach.html&nbsp;&nbsp; lang_createindex.html&nbsp;&nbsp; lang_createtable.html&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_createtrigger.html#raise&nbsp;&nbsp; lang_createview.html&nbsp;&nbsp; lang_delete.html&nbsp;&nbsp; lang_expr.html&nbsp;&nbsp; lang_insert.html&nbsp;&nbsp; lang_returning.html&nbsp;&nbsp; lang_select.html&nbsp;&nbsp; lang_select.html#compound&nbsp;&nbsp; lang_select.html#simpleselect&nbsp;&nbsp; lang_update.html&nbsp;&nbsp; lang_upsert.html&nbsp;&nbsp; lang_with.html&nbsp;&nbsp; partialindex.html


recursive-cte:



References:&nbsp;&nbsp; cte-table-name
See also:&nbsp;&nbsp; lang_with.html#recursivecte


reindex-stmt:



Used by:&nbsp;&nbsp; sql-stmt
See also:&nbsp;&nbsp; lang_reindex.html


release-stmt:



Used by:&nbsp;&nbsp; sql-stmt
See also:&nbsp;&nbsp; lang_savepoint.html


result-column:



Used by:&nbsp;&nbsp; select-core&nbsp;&nbsp; select-stmt
References:&nbsp;&nbsp; expr
See also:&nbsp;&nbsp; lang_aggfunc.html&nbsp;&nbsp; lang_altertable.html&nbsp;&nbsp; lang_attach.html&nbsp;&nbsp; lang_createindex.html&nbsp;&nbsp; lang_createtable.html&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_createview.html&nbsp;&nbsp; lang_delete.html&nbsp;&nbsp; lang_expr.html&nbsp;&nbsp; lang_insert.html&nbsp;&nbsp; lang_returning.html&nbsp;&nbsp; lang_select.html&nbsp;&nbsp; lang_select.html#compound&nbsp;&nbsp; lang_select.html#simpleselect&nbsp;&nbsp; lang_update.html&nbsp;&nbsp; lang_upsert.html&nbsp;&nbsp; lang_with.html&nbsp;&nbsp; partialindex.html


returning-clause:



Used by:&nbsp;&nbsp; delete-stmt&nbsp;&nbsp; delete-stmt-limited&nbsp;&nbsp; insert-stmt&nbsp;&nbsp; update-stmt&nbsp;&nbsp; update-stmt-limited
References:&nbsp;&nbsp; expr
See also:&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_delete.html&nbsp;&nbsp; lang_insert.html&nbsp;&nbsp; lang_returning.html&nbsp;&nbsp; lang_update.html


rollback-stmt:



Used by:&nbsp;&nbsp; sql-stmt
See also:&nbsp;&nbsp; lang_savepoint.html&nbsp;&nbsp; lang_transaction.html


savepoint-stmt:



Used by:&nbsp;&nbsp; sql-stmt
See also:&nbsp;&nbsp; lang_savepoint.html


select-core:



Used by:&nbsp;&nbsp; compound-select-stmt&nbsp;&nbsp; factored-select-stmt&nbsp;&nbsp; simple-select-stmt
References:&nbsp;&nbsp; expr&nbsp;&nbsp; join-clause&nbsp;&nbsp; result-column&nbsp;&nbsp; table-or-subquery&nbsp;&nbsp; window-defn
See also:&nbsp;&nbsp; lang_select.html&nbsp;&nbsp; lang_select.html#compound&nbsp;&nbsp; lang_select.html#simpleselect


select-stmt:



Used by:&nbsp;&nbsp; common-table-expression&nbsp;&nbsp; create-table-stmt&nbsp;&nbsp; create-trigger-stmt&nbsp;&nbsp; create-view-stmt&nbsp;&nbsp; expr&nbsp;&nbsp; insert-stmt&nbsp;&nbsp; sql-stmt&nbsp;&nbsp; table-or-subquery&nbsp;&nbsp; with-clause
References:&nbsp;&nbsp; common-table-expression&nbsp;&nbsp; compound-operator&nbsp;&nbsp; expr&nbsp;&nbsp; join-clause&nbsp;&nbsp; ordering-term&nbsp;&nbsp; result-column&nbsp;&nbsp; table-or-subquery&nbsp;&nbsp; window-defn
See also:&nbsp;&nbsp; changes.html#version_3_35_3&nbsp;&nbsp; lang_aggfunc.html&nbsp;&nbsp; lang_altertable.html&nbsp;&nbsp; lang_attach.html&nbsp;&nbsp; lang_createindex.html&nbsp;&nbsp; lang_createtable.html&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_createview.html&nbsp;&nbsp; lang_delete.html&nbsp;&nbsp; lang_expr.html&nbsp;&nbsp; lang_insert.html&nbsp;&nbsp; lang_returning.html&nbsp;&nbsp; lang_select.html&nbsp;&nbsp; lang_select.html#compound&nbsp;&nbsp; lang_select.html#simpleselect&nbsp;&nbsp; lang_update.html&nbsp;&nbsp; lang_upsert.html&nbsp;&nbsp; lang_with.html&nbsp;&nbsp; lang_with.html#recursivecte&nbsp;&nbsp; partialindex.html&nbsp;&nbsp; releaselog/3_35_3.html&nbsp;&nbsp; releaselog/3_35_4.html&nbsp;&nbsp; releaselog/3_35_5.html


signed-number:



Used by:&nbsp;&nbsp; column-constraint&nbsp;&nbsp; pragma-value&nbsp;&nbsp; type-name
See also:&nbsp;&nbsp; lang_aggfunc.html&nbsp;&nbsp; lang_altertable.html&nbsp;&nbsp; lang_attach.html&nbsp;&nbsp; lang_createindex.html&nbsp;&nbsp; lang_createtable.html&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_createview.html&nbsp;&nbsp; lang_delete.html&nbsp;&nbsp; lang_expr.html&nbsp;&nbsp; lang_insert.html&nbsp;&nbsp; lang_returning.html&nbsp;&nbsp; lang_select.html&nbsp;&nbsp; lang_select.html#compound&nbsp;&nbsp; lang_select.html#simpleselect&nbsp;&nbsp; lang_update.html&nbsp;&nbsp; lang_upsert.html&nbsp;&nbsp; lang_with.html&nbsp;&nbsp; partialindex.html&nbsp;&nbsp; pragma.html#syntax


simple-function-invocation:



References:&nbsp;&nbsp; expr
See also:&nbsp;&nbsp; lang_expr.html#booleanexpr


simple-select-stmt:



References:&nbsp;&nbsp; common-table-expression&nbsp;&nbsp; expr&nbsp;&nbsp; ordering-term&nbsp;&nbsp; select-core
See also:&nbsp;&nbsp; lang_select.html#simpleselect


sql-stmt:



Used by:&nbsp;&nbsp; sql-stmt-list
References:&nbsp;&nbsp; alter-table-stmt&nbsp;&nbsp; analyze-stmt&nbsp;&nbsp; attach-stmt&nbsp;&nbsp; begin-stmt&nbsp;&nbsp; commit-stmt&nbsp;&nbsp; create-index-stmt&nbsp;&nbsp; create-table-stmt&nbsp;&nbsp; create-trigger-stmt&nbsp;&nbsp; create-view-stmt&nbsp;&nbsp; create-virtual-table-stmt&nbsp;&nbsp; delete-stmt&nbsp;&nbsp; delete-stmt-limited&nbsp;&nbsp; detach-stmt&nbsp;&nbsp; drop-index-stmt&nbsp;&nbsp; drop-table-stmt&nbsp;&nbsp; drop-trigger-stmt&nbsp;&nbsp; drop-view-stmt&nbsp;&nbsp; insert-stmt&nbsp;&nbsp; pragma-stmt&nbsp;&nbsp; reindex-stmt&nbsp;&nbsp; release-stmt&nbsp;&nbsp; rollback-stmt&nbsp;&nbsp; savepoint-stmt&nbsp;&nbsp; select-stmt&nbsp;&nbsp; update-stmt&nbsp;&nbsp; update-stmt-limited&nbsp;&nbsp; vacuum-stmt
See also:&nbsp;&nbsp; lang.html&nbsp;&nbsp; lang_explain.html


sql-stmt-list:



References:&nbsp;&nbsp; sql-stmt
See also:&nbsp;&nbsp; lang.html


table-constraint:



Used by:&nbsp;&nbsp; create-table-stmt
References:&nbsp;&nbsp; conflict-clause&nbsp;&nbsp; expr&nbsp;&nbsp; foreign-key-clause&nbsp;&nbsp; indexed-column
See also:&nbsp;&nbsp; lang_createtable.html&nbsp;&nbsp; lang_createtable.html#primkeyconst&nbsp;&nbsp; lang_createtable.html#tablecoldef&nbsp;&nbsp; lang_createtable.html#uniqueconst


table-options:



Used by:&nbsp;&nbsp; create-table-stmt
See also:&nbsp;&nbsp; lang_createtable.html


table-or-subquery:



Used by:&nbsp;&nbsp; join-clause&nbsp;&nbsp; select-core&nbsp;&nbsp; select-stmt&nbsp;&nbsp; update-stmt&nbsp;&nbsp; update-stmt-limited
References:&nbsp;&nbsp; expr&nbsp;&nbsp; join-clause&nbsp;&nbsp; select-stmt
See also:&nbsp;&nbsp; lang_aggfunc.html&nbsp;&nbsp; lang_altertable.html&nbsp;&nbsp; lang_attach.html&nbsp;&nbsp; lang_createindex.html&nbsp;&nbsp; lang_createtable.html&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_createview.html&nbsp;&nbsp; lang_delete.html&nbsp;&nbsp; lang_expr.html&nbsp;&nbsp; lang_insert.html&nbsp;&nbsp; lang_returning.html&nbsp;&nbsp; lang_select.html&nbsp;&nbsp; lang_select.html#compound&nbsp;&nbsp; lang_select.html#simpleselect&nbsp;&nbsp; lang_update.html&nbsp;&nbsp; lang_upsert.html&nbsp;&nbsp; lang_with.html&nbsp;&nbsp; partialindex.html


type-name:



Used by:&nbsp;&nbsp; column-def&nbsp;&nbsp; expr
References:&nbsp;&nbsp; signed-number
See also:&nbsp;&nbsp; lang_aggfunc.html&nbsp;&nbsp; lang_altertable.html&nbsp;&nbsp; lang_attach.html&nbsp;&nbsp; lang_createindex.html&nbsp;&nbsp; lang_createtable.html&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_createview.html&nbsp;&nbsp; lang_delete.html&nbsp;&nbsp; lang_expr.html&nbsp;&nbsp; lang_insert.html&nbsp;&nbsp; lang_returning.html&nbsp;&nbsp; lang_select.html&nbsp;&nbsp; lang_select.html#compound&nbsp;&nbsp; lang_select.html#simpleselect&nbsp;&nbsp; lang_update.html&nbsp;&nbsp; lang_upsert.html&nbsp;&nbsp; lang_with.html&nbsp;&nbsp; partialindex.html


update-stmt:



Used by:&nbsp;&nbsp; create-trigger-stmt&nbsp;&nbsp; sql-stmt
References:&nbsp;&nbsp; column-name-list&nbsp;&nbsp; common-table-expression&nbsp;&nbsp; expr&nbsp;&nbsp; join-clause&nbsp;&nbsp; qualified-table-name&nbsp;&nbsp; returning-clause&nbsp;&nbsp; table-or-subquery
See also:&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_update.html


update-stmt-limited:



Used by:&nbsp;&nbsp; sql-stmt
References:&nbsp;&nbsp; column-name-list&nbsp;&nbsp; common-table-expression&nbsp;&nbsp; expr&nbsp;&nbsp; join-clause&nbsp;&nbsp; ordering-term&nbsp;&nbsp; qualified-table-name&nbsp;&nbsp; returning-clause&nbsp;&nbsp; table-or-subquery
See also:&nbsp;&nbsp; lang_update.html#upfrom


upsert-clause:



Used by:&nbsp;&nbsp; insert-stmt
References:&nbsp;&nbsp; column-name-list&nbsp;&nbsp; expr&nbsp;&nbsp; indexed-column
See also:&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_insert.html&nbsp;&nbsp; lang_upsert.html


vacuum-stmt:



Used by:&nbsp;&nbsp; sql-stmt
See also:&nbsp;&nbsp; lang_vacuum.html


window-defn:



Used by:&nbsp;&nbsp; select-core&nbsp;&nbsp; select-stmt&nbsp;&nbsp; window-function-invocation
References:&nbsp;&nbsp; expr&nbsp;&nbsp; frame-spec&nbsp;&nbsp; ordering-term
See also:&nbsp;&nbsp; lang_aggfunc.html&nbsp;&nbsp; lang_altertable.html&nbsp;&nbsp; lang_attach.html&nbsp;&nbsp; lang_createindex.html&nbsp;&nbsp; lang_createtable.html&nbsp;&nbsp; lang_createtrigger.html&nbsp;&nbsp; lang_createview.html&nbsp;&nbsp; lang_delete.html&nbsp;&nbsp; lang_expr.html&nbsp;&nbsp; lang_insert.html&nbsp;&nbsp; lang_returning.html&nbsp;&nbsp; lang_select.html&nbsp;&nbsp; lang_select.html#compound&nbsp;&nbsp; lang_select.html#simpleselect&nbsp;&nbsp; lang_update.html&nbsp;&nbsp; lang_upsert.html&nbsp;&nbsp; lang_with.html&nbsp;&nbsp; partialindex.html


window-function-invocation:



References:&nbsp;&nbsp; expr&nbsp;&nbsp; filter-clause&nbsp;&nbsp; window-defn
See also:&nbsp;&nbsp; lang_expr.html#booleanexpr


with-clause:



References:&nbsp;&nbsp; cte-table-name&nbsp;&nbsp; select-stmt
See also:&nbsp;&nbsp; lang_with.html


c2: The Error And Warning Log
c5: errlog.html
c4:
Overview 

SQLite can be configured to invoke a callback function containing
an error code and a terse error message whenever anomalies occur.
This mechanism is very helpful in tracking obscure problems that
occur rarely and in the field.  Application developers are encouraged
to take advantage of the error logging facility of SQLite in their
products, as it is very low CPU and memory cost but can be a
huge aid for debugging.

1. Setting Up The Error Logging Callback

There can only be a single error logging callback per process.
The error logging callback is registered at start-time using C-code
similar to the following:


sqlite3_config(SQLITE_CONFIG_LOG, errorLogCallback, pData);


The error logger callback function might look something like this:


void errorLogCallback(void *pArg, int iErrCode, const char *zMsg){
  fprintf(stderr, "(%d) %s\n", iErrCode, zMsg);
}


The example above illustrates the signature of the error logger callback.
However, in an embedded application, one usually does not print
messages on stderr.  Instead, one might store the messages in a
preallocated circular buffer where they can be accessed when diagnostic
information is needed during debugging.  Or perhaps the messages can be
sent to Syslog.  Somehow, the
messages need to be stored where they are accessible to developers,
not displayed to end users.

Do not misunderstand: There is nothing technically wrong with displaying 
the error logger messages to end users.  The messages do not contain
sensitive or private information that must be protected from unauthorized
viewing.  Rather the messages are technical in nature and are not useful
or meaningful to the typical end user.  The messages coming from the
error logger are intended for database geeks.  Display them accordingly.

2. Interface Details

The third argument to the sqlite3_config(SQLITE_CONFIG_LOG,...) 
interface (the "pData" argument in the example above) is a pointer to arbitrary
data.  SQLite passes this pointer through to the first argument of the
error logger callback.  The pointer can be used to pass application-specific 
setup or state information, if desired.  Or it can simply be a NULL 
pointer which is ignored by the callback.

The second argument to the error logger callback is an integer
extended error code.  The third argument to the error logger is the
text of the error message.  The error message text is stored in a fixed-length
stack buffer in the calling function and so will only be valid for the
duration of the error logger callback function.  The error logger should
make a copy of this message into persistent storage if retention of the
message is needed.

The error logger callback should be treated like a signal handler.
The application should save off or otherwise process the error, then return
as soon as possible.  No other SQLite APIs should be invoked, directly or
indirectly, from the error logger.  SQLite is not reentrant through
the error logger callback.  In particular, the error logger callback
is invoked when a memory allocation fails, so it is generally a bad idea
to try to allocate memory inside the error logger.  Do not even think
about trying to store the error message in another SQLite database.

Applications can use the sqlite3_log(E,F,..) API to send new messages
to the log, if desired, but this is discouraged.  The sqlite3_log()
interface is intended for use by extensions only, not by applications.

3. Variety of Error Messages

The error messages that might be sent to the error logger and their
exact format is subject to changes from one release to the next.  So
applications should not depend on any particular error message text formats or
error codes.  Things do not change capriciously, but they do sometimes
changes.

The following is a partial list of the kinds of messages that might
appear in the error logger callback.



Any time there is an error either compiling an SQL statement 
(using sqlite3_prepare_v2() or its siblings) or running an SQL
statement (using sqlite3_step()) that error is logged.



When a schema change occurs that requires a prepared statement to be reparsed
and reprepared, that event is logged with the error code SQLITE_SCHEMA.
The reparse and reprepare is normally automatic (assuming that
sqlite3_prepare_v2() has been used to prepare the statements originally,
which is recommended) and so these logging events are normally the only
way to know that reprepares are taking place.


SQLITE_NOTICE messages are logged whenever a database has to be recovered
because the previous writer crashed without completing its transaction.
The error code is SQLITE_NOTICE_RECOVER_ROLLBACK when recovering a
rollback journal and SQLITE_NOTICE_RECOVER_WAL when recovering a 
write-ahead log.



SQLITE_WARNING messages are logged when database files are renamed or
aliased in ways that can lead to database corruption.
(See 1 and 2 for
additional information.)



Out of memory (OOM) error conditions generate error logging events
with the SQLITE_NOMEM error code and a message that says how many bytes
of memory were requested by the failed allocation.


I/O errors in the OS-interface generate error logging events.
The message to these events gives the line number in the source code where
the error originated and the filename associated with the event when
there is a corresponding file. 

When database corruption is detected, an SQLITE_CORRUPT error
logger callback is invoked.  As with I/O errors, the error message text
contains the line number in the original source code where the error
was first detected.


An error logger callback is invoked on SQLITE_MISUSE errors.
This is useful in detecting application design issues when return codes
are not consistently checked in the application code.


SQLite strives to keep error logger traffic low and only send messages
to the error logger when there really is something wrong.  Applications
might further cull the error message traffic 
by deliberately ignoring certain classes of error
messages that they do not care about.  For example, an application that
makes frequent database schema changes might want to ignore all
SQLITE_SCHEMA errors.

4. Summary

The use of the error logger callback is highly recommended.
The debugging information that the error logger provides has proven
very useful in tracking down obscure problems that occur with applications
after they get into the field.  The error logger callback has also 
proven useful in catching occasional errors that the application
misses because of inconsistent checking of API return codes.
Developers are encouraged to implement an error logger callback early
in the development cycle in order to spot unexpected behavior quickly,
and to leave the error logger callback turned on through deployment.
If the error logger never finds a problem, then no harm is done.  
But failure to set up an appropriate error logger might compromise
diagnostic capabilities later on.


c2: SQLite is a Self Contained System
c5: selfcontained.html
c4:
SQLite is "stand-alone" or "self-contained" in the sense that it
has very few dependencies.  It runs on any operating system, even
stripped-down bare-bones embedded operating systems.  SQLite uses
no external libraries or interfaces (other than a few standard C-library 
calls described below).  The entire SQLite library is
encapsulated in a single source code file that requires
no special facilities or tools to build.


A minimal build of SQLite requires just these routines from the
standard C library:

memcmp()
memcpy()
memmove()
memset()
strcmp()
strlen()
strncmp()



Most builds also use the system memory allocation routines:

malloc()
realloc()
free()

But those routines are optional and can be omitted
using a compile-time option.


Default builds of SQLite contain appropriate VFS objects for talking
to underlying operating system, and those VFS objects will contain
operating system calls such as open(), read(), write(), fsync(), and
so forth.  All of these interfaces are readily available on most platforms,
and custom VFSes can be designed to run SQLite on even the most
austere embedded devices.


c2: The SQLITE_DBPAGE Virtual Table
c5: dbpage.html
c4:
1. Overview


The SQLITE_DBPAGE extension implements an eponymous-only virtual table that
provides direct access to the underlying database file by interacting
with the pager.  SQLITE_DBPAGE is capable of both reading and writing any
page of the database.  Because interaction is through the pager layer, all
changes are transactional.



Warning: writing to the SQLITE_DBPAGE virtual table can very easily
cause unrecoverably database corruption.  Do not allow untrusted components
to access the SQLITE_DBPAGE table.  Use appropriate care while using the
SQLITE_DBPAGE table.  Back up important data prior to experimenting with the
SQLITE_DBPAGE table.  Writes to the SQLITE_DBPAGE virtual table are
disabled when the SQLITE_DBCONFIG_DEFENSIVE flag is set.


The SQLITE_DBPAGE extension is included in the amalgamation though 
it is disabled
by default.  Use the SQLITE_ENABLE_DBPAGE_VTAB compile-time option to enable
the SQLITE_DBPAGE extension.  The SQLITE_DBPAGE extension makes use of
unpublished internal interfaces and is not run-time loadable.  The only way
to add SQLITE_DBPAGE to an application is to compile it in using the
SQLITE_ENABLE_DBPAGE_VTAB compile-time option.



The SQLITE_DBPAGE extension is enabled in default builds
of the command-line shell.

2. Usage


The SQLITE_DBPAGE virtual table read/write table that provides direct
access to the underlying disk file on a page-by-page basis.  The
virtual table appears to have a schema like this:

CREATE TABLE sqlite_dbpage(
  pgno INTEGER PRIMARY KEY,
  data BLOB
);



An SQLite database file is divided into pages.
The first page is 1, the second page is 2, and so forth.
There is no page 0.
Every page is the same size.
The size of every page is a power of 2 between 512 and 65536.
See the file format documentation for further details.


The SQLITE_DBPAGE table allows an application to view or replace the
raw binary content of each page of the database file.
No attempt is made to interpret the content of the page.
Content is returned byte-for-byte as it appears on disk.


The SQLITE_DBPAGE table has one row for each page in the database file.
SQLITE_DBPAGE allows pages to be read or to be overwritten.
However the size of the database file cannot be changed.  It is not
possible to change the number of rows in the SQLITE_DBPAGE table by
running DELETE or INSERT operations against that table.

2.1. Using SQLITE_DBPAGE On ATTACH-ed Databases


The SQLITE_DBPAGE table schema shown above is incomplete.  There is
a third hidden column named "schema" that determines which
ATTACH-ed database should be read or written.  Because
the "schema" column is hidden, it can be used as a parameter when
SQLITE_DBPAGE is invoked as a table-valued function.


For example, suppose an additional database is attached to the 
database connection using a statement like this:

ATTACH 'auxdata1.db' AS aux1;



Then to read the first page of that database file, one merely runs:

SELECT data FROM sqlite_dbpage('aux1') WHERE pgno=1;



If the "schema" is omitted, it defaults to the primary database
(usually called 'main', unless renamed using SQLITE_DBCONFIG_MAINDBNAME).
Hence, the following two queries are normally equivalent:

SELECT data FROM sqlite_dbpage('main') WHERE pgno=1;
SELECT data FROM sqlite_dbpage WHERE pgno=1;



The SQLITE_DBPAGE table can participate in a join just like any other
table.  Hence, to see the content of the first page to all connected
database files, one might run a statement like this:

SELECT dbpage.data, dblist.name
  FROM pragma_database_list AS dblist
  JOIN sqlite_dbpage(dblist.name) AS dbpage
 WHERE dbpage.pgno=1;


c2: Row Values
c5: rowvalue.html
c4:
1. Definitions

A "value" is a single number, string, BLOB or NULL.
Sometimes the qualified name "scalar value" is used to emphasize that
only a single quantity is involved.

A "row value" is an ordered list of two or more scalar values.
In other words, a "row value" is a vector or tuple.

The "size" of a row value is the number of scalar values the row value contains.
The size of a row value is always at least 2.
A row value with a single column is just a scalar value.
A row value with no columns is a syntax error.

2. Syntax

SQLite allows row values to be expressed in two ways:

A parenthesized, comma-separated list of scalar values.
A subquery expression with two or more result columns.


SQLite can use row values in two contexts:

Two row values of the same size 
can be compared using operators &lt;, &lt;=, &gt;, &gt;=,
=, &lt;&gt;, IS, IS NOT, IN, NOT IN, BETWEEN, or CASE.
In an UPDATE statement, a list of column names can be set to a row value of
the same size.


The syntax for row values and the circumstances in which row values
can be used are illustrated in examples below.

2.1. Row Value Comparisons

Two row values are compared by looking at the constituent scalar
values from left to right.
A NULL means of "unknown".  
The overall result of comparison is NULL if it is possible to make the
result either true or false by substituting alternative values in place
of the constituent NULLs.
The following query demonstrates some row value comparisons:

SELECT
  (1,2,3) = (1,2,3),          -- 1
  (1,2,3) = (1,NULL,3),       -- NULL
  (1,2,3) = (1,NULL,4),       -- 0
  (1,2,3) &lt; (2,3,4),          -- 1
  (1,2,3) &lt; (1,2,4),          -- 1
  (1,2,3) &lt; (1,3,NULL),       -- 1
  (1,2,3) &lt; (1,2,NULL),       -- NULL
  (1,3,5) &lt; (1,2,NULL),       -- 0
  (1,2,NULL) IS (1,2,NULL);   -- 1


The result of "(1,2,3)=(1,NULL,3)" is NULL because the result might be
true if we replaced NULL&rarr;2 or false if we replaced NULL&rarr;9.
The result of "(1,2,3)=(1,NULL,4)" is not NULL because there is no
substitutions of the constituent NULL that will make the expression true,
since 3 will never equal 4 in the third column.

Any of the row values in the previous example could be replace by a
subquery that returns three columns and the same answer would result.
For example:

CREATE TABLE t1(a,b,c);
INSERT INTO t1(a,b,c) VALUES(1,2,3);
SELECT (1,2,3)=(SELECT * FROM t1); -- 1




2.2. Row Value IN Operators

For a row-value IN operator, the left-hand side (hereafter "LHS") can be either
a parenthesized list of values or a subquery with multiple columns.  But the
right-hand side (hereafter "RHS") must be a subquery expression.

CREATE TABLE t2(x,y,z);
INSERT INTO t2(x,y,z) VALUES(1,2,3),(2,3,4),(1,NULL,5);
SELECT
   (1,2,3) IN (SELECT * FROM t2),  -- 1
   (7,8,9) IN (SELECT * FROM t2),  -- 0
   (1,3,5) IN (SELECT * FROM t2);  -- NULL


2.3. Row Values In UPDATE Statements

Row values can also be used in the SET clause of an UPDATE statement.
The LHS must be a list of column names.  The RHS can be any row value.
For example:

UPDATE tab3 
   SET (a,b,c) = (SELECT x,y,z
                    FROM tab4
                   WHERE tab4.w=tab3.d)
 WHERE tab3.e BETWEEN 55 AND 66;


3. Example Uses Of Row Values

3.1. Scrolling Window Queries

Suppose an application wants to display a list of contacts
in alphabetical order by lastname, firstname, in a scrolling window
that can only show 7 contacts at a time.  Initialize the scrolling
window to the first 7 entries is easy:

SELECT * FROM contacts
 ORDER BY lastname, firstname
 LIMIT 7;


When the user scrolls down, the application needs to find the
second set of 7 entries.  One way to do this is to use the OFFSET clause:

SELECT * FROM contacts
 ORDER BY lastname, firstname
 LIMIT 7 OFFSET 7;


OFFSET gives the correct answer.  However, OFFSET requires time
proportional to the offset value.  What really happens
with "LIMIT x OFFSET y" is that SQLite computes the query as
"LIMIT x+y" and discards the first y values without returning them
to the application.  So as the window scrolls down toward
the bottom of a long list, and the y value becomes larger and larger,
successive offset computations take more and more time.

A more efficient approach is to remember the last entry currently
displayed and then use a row value comparison in the WHERE
clause:

SELECT * FROM contacts
 WHERE (lastname,firstname) &gt; (?1,?2)
 ORDER BY lastname, firstname
 LIMIT 7;


If the lastname and firstname on the bottom row of the previous
screen are bound to ?1 and ?2, then the query above computes the next
7 rows.  And, assuming there is an appropriate index, it does so
very efficiently &mdash; much more efficiently than OFFSET.

3.2. Comparison of dates stored as separate fields

The usual way of storing a date in a database table is as a single
field, as either a unix timestamp, a julian day number, or an ISO-8601
dates string.  But some application store dates as three separate
fields for the year, month, and day.  

CREATE TABLE info(
  year INT,          -- 4 digit year
  month INT,         -- 1 through 12
  day INT,           -- 1 through 31
  other_stuff BLOB   -- blah blah blah
);


When dates are stored this way, row value comparisons provide a
convenient way to compare dates:

SELECT * FROM info
 WHERE (year,month,day) BETWEEN (2015,9,12) AND (2016,9,12);


3.3. Search against multi-column keys

Suppose we want to know the order number, product number, and quantity
for any item in which the product number and quantity match the product
number and quantity of any item in order number 365:

SELECT ordid, prodid, qty
  FROM item
 WHERE (prodid, qty) IN (SELECT prodid, qty
                           FROM item
                          WHERE ordid = 365);


The query above could be rewritten as a join and without the use
of row values:

SELECT t1.ordid, t1.prodid, t1.qty
  FROM item AS t1, item AS t2
 WHERE t1.prodid=t2.prodid
   AND t1.qty=t2.qty
   AND t2.ordid=365;


Because the same query could be written without the use of row values,
row values do not provide new capabilities.  However, many developers say
that the row value format is easier to read, write, and debug.

Even in the JOIN form, the query can be made clearer through the use of
row values:

SELECT t1.ordid, t1.prodid, t1.qty
  FROM item AS t1, item AS t2
 WHERE (t1.prodid,t1.qty) = (t2.prodid,t2.qty)
   AND t2.ordid=365;


This later query generates exactly the same bytecode as the previous
scalar formulation, but using syntax that it cleaner and
easier to read.

3.4. Update multiple columns of a table based on a query

The row-value notation is useful for updating two or more columns
of a table from the result of a single query.
An example of this is in the full-text search feature of the
Fossil version control system.

In the Fossil full-text search system,
documents that participate in the full-text search (wiki pages, tickets,
check-ins, documentation files, etc) are tracked by a table called
"ftsdocs" (full text search documents).
As new documents are added to the repository, they are not indexed right
away.  Indexing is deferred until there is a search request.  The
ftsdocs table contains an "idxed" field which is true if the document
has been indexed and false if not.

When a search request occurs and pending documents are indexed for the
first time, the ftsdocs table must be updated by setting the idxed column
to true and also filling in several other columns with information pertinent
to the search.  That other information is obtained from a join.  The
query is this:

UPDATE ftsdocs SET
  idxed=1,
  name=NULL,
  (label,url,mtime) = 
      (SELECT printf('Check-in &#91;%%.16s&#93; on %%s',blob.uuid,
                     datetime(event.mtime)),
              printf('/timeline?y=ci&amp;c=%%.20s',blob.uuid),
              event.mtime
         FROM event, blob
        WHERE event.objid=ftsdocs.rid
          AND blob.rid=ftsdocs.rid)
WHERE ftsdocs.type='c' AND NOT ftsdocs.idxed


(See the 
source code
for further detail.  Other examples
here and
here.)

Five out of nine columns in the ftsdocs table are updated.  Two of
the modified columns, "idxed" and "name", can be updated independently of
the query.  But the three columns "label", "url", and "mtime" all require
a join query against the "event" and "blob" tables.  Without row values,
the equivalent UPDATE would require that the join be repeated three times, 
once for each column to be updated.

3.5. Clarity of presentation

Sometimes the use of row values just makes the SQL easier to read
and write.  Consider the following two UPDATE statements:

UPDATE tab1 SET (a,b)=(b,a);
UPDATE tab1 SET a=b, b=a;


Both UPDATE statements do exactly the same thing.  (They generate
identical bytecode.)  But the first form, the row value form, seems
to make it clearer that the intent of the statement is to swap the
values in columns A and B.

Or consider these identical queries:

SELECT * FROM tab1 WHERE a=?1 AND b=?2;
SELECT * FROM tab1 WHERE (a,b)=(?1,?2);


Once again, the SQL statements generate identical bytecode and thus
do exactly the same job in exactly the same way.  But the second form
is made easier for humans to read by grouping the query parameters together
into a single row value rather than scattering them across the WHERE
clause.

4. Backwards Compatibility

Row values were added to SQLite
version 3.15.0 (2016-10-14).  Attempts to use row values in
prior versions of SQLite will generate syntax errors.


c2: The UINT Collating Sequence
c5: uintcseq.html
c4:
1. Overview

 The UINT collating sequences is a loadable extension for
SQLite that implements a new collating sequence that compares text
containing unsigned integers in numeric order.

 The UINT collating sequence is not a standard part of SQLite.
It must be loaded as a separate extension.  The source code to
UINT is in the uint.c source file
in the ext/misc/ folder of the
SQLite source tree.

 The UINT collating sequence is not included in standard builds of
the SQLite library, but it is loaded by default in the CLI.  This
is typical of the CLI which loads various extensions above and beyond
what are available in the standard SQLite library.

 The UINT collating sequence works just like the default
BINARY collating sequence for text, except that embedded strings
of digits compare in numeric order.



Leading zeros are handled properly, in the sense that
they do not mess of the maginitude comparison of embedded
strings of digits.  "x00123y" is equal to "x123y".


Only unsigned integers are recognized.  Plus and minus
signs are ignored.  Decimal points and exponential notation
are ignored.


Embedded integers can be of arbitrary length.  Comparison
is not limited to integers that can be expressed as a
64-bit machine integer.


2. Example:



COLLATE binaryCOLLATE uint

'0000123457'
'123456'
'abc0000000010xyz'
'abc0010xyy'
'abc10xzz'
'abc674xyz'
'abc87xyz'
'abc9xyz'

'123456'
'0000123457'
'abc9xyz'
'abc0010xyy'
'abc0000000010xyz'
'abc10xzz'
'abc87xyz'
'abc674xyz'


c2: Datatypes In SQLite version 2
c5: datatypes.html
c4:
Datatypes In SQLite Version 2

1.0 &nbsp; Typelessness

SQLite is "typeless".  This means that you can store any
kind of data you want in any column of any table, regardless of the
declared datatype of that column.  
(See the one exception to this rule in section 2.0 below.)
This behavior is a feature, not
a bug.  A database is supposed to store and retrieve data and it 
should not matter to the database what format that data is in.
The strong typing system found in most other SQL engines and
codified in the SQL language spec is a misfeature -
it is an example of the implementation showing through into the
interface.  SQLite seeks to overcome this misfeature by allowing
you to store any kind of data into any kind of column and by
allowing flexibility in the specification of datatypes.



A datatype to SQLite is any sequence of zero or more names
optionally followed by a parenthesized lists of one or two
signed integers.  Notice in particular that a datatype may
be zero or more names.  That means that an empty
string is a valid datatype as far as SQLite is concerned.
So you can declare tables where the datatype of each column
is left unspecified, like this:



CREATE TABLE ex1(a,b,c);



Even though SQLite allows the datatype to be omitted, it is
still a good idea to include it in your CREATE TABLE statements,
since the data type often serves as a good hint to other
programmers about what you intend to put in the column. And
if you ever port your code to another database engine, that
other engine will probably require a datatype of some kind.
SQLite accepts all the usual datatypes.  For example:



CREATE TABLE ex2(
  a VARCHAR(10),
  b NVARCHAR(15),
  c TEXT,
  d INTEGER,
  e FLOAT,
  f BOOLEAN,
  g CLOB,
  h BLOB,
  i TIMESTAMP,
  j NUMERIC(10,5)
  k VARYING CHARACTER (24),
  l NATIONAL VARYING CHARACTER(16)
);



And so forth.  Basically any sequence of names optionally followed by 
one or two signed integers in parentheses will do.


2.0 &nbsp; The INTEGER PRIMARY KEY


One exception to the typelessness of SQLite is a column whose type
is INTEGER PRIMARY KEY.  (And you must use "INTEGER" not "INT".
A column of type INT PRIMARY KEY is typeless just like any other.)
INTEGER PRIMARY KEY columns must contain a 32-bit signed integer.  Any
attempt to insert non-integer data will result in an error.



INTEGER PRIMARY KEY columns can be used to implement the equivalent
of AUTOINCREMENT.  If you try to insert a NULL into an INTEGER PRIMARY
KEY column, the column will actually be filled with an integer that is
one greater than the largest key already in the table.  Or if the
largest key is 2147483647, then the column will be filled with a
random integer.  Either way, the INTEGER PRIMARY KEY column will be
assigned a unique integer.  You can retrieve this integer using
the sqlite_last_insert_rowid() API function or using the
last_insert_rowid() SQL function in a subsequent SELECT statement.


3.0 &nbsp; Comparison and Sort Order


SQLite is typeless for the purpose of deciding what data is allowed
to be stored in a column.  But some notion of type comes into play
when sorting and comparing data.  For these purposes, a column or
an expression can be one of two types: numeric and text.
The sort or comparison may give different results depending on which
type of data is being sorted or compared.



If data is of type text then the comparison is determined by
the standard C data comparison functions memcmp() or
strcmp().  The comparison looks at bytes from two inputs one
by one and returns the first non-zero difference.
Strings are '\000' terminated so shorter
strings sort before longer strings, as you would expect.



For numeric data, this situation is more complex.  If both inputs
look like well-formed numbers, then they are converted
into floating point values using atof() and compared numerically.
If one input is not a well-formed number but the other is, then the
number is considered to be less than the non-number.  If neither inputs
is a well-formed number, then strcmp() is used to do the
comparison.



Do not be confused by the fact that a column might have a "numeric"
datatype.  This does not mean that the column can contain only numbers.
It merely means that if the column does contain a number, that number
will sort in numerical order.



For both text and numeric values, NULL sorts before any other value.
A comparison of any value against NULL using operators like "&lt;" or
"&gt;=" is always false.


4.0 &nbsp; How SQLite Determines Datatypes


For SQLite version 2.6.3 and earlier, all values used the numeric datatype.
The text datatype appears in version 2.7.0 and later.  In the sequel it
is assumed that you are using version 2.7.0 or later of SQLite.



For an expression, the datatype of the result is often determined by
the outermost operator.  For example, arithmetic operators ("+", "*", "%")
always return a numeric results.  The string concatenation operator
("||") returns a text result.  And so forth.  If you are ever in doubt
about the datatype of an expression you can use the special typeof()
SQL function to determine what the datatype is.  For example:



sqlite&gt; SELECT typeof('abc'+123);
numeric
sqlite&gt; SELECT typeof('abc'||123);
text



For table columns, the datatype is determined by the type declaration
of the CREATE TABLE statement.  The datatype is text if and only if
the type declaration contains one or more of the following strings:



BLOB
CHAR
CLOB
TEXT



The search for these strings in the type declaration is case insensitive,
of course.  If any of the above strings occur anywhere in the type
declaration, then the datatype of the column is text.  Notice that
the type "VARCHAR" contains "CHAR" as a substring so it is considered
text.

If none of the strings above occur anywhere in the type declaration,
then the datatype is numeric.  Note in particular that the datatype for columns
with an empty type declaration is numeric.


5.0 &nbsp; Examples


Consider the following two command sequences:



CREATE TABLE t1(a INTEGER UNIQUE);        CREATE TABLE t2(b TEXT UNIQUE);
INSERT INTO t1 VALUES('0');               INSERT INTO t2 VALUES(0);
INSERT INTO t1 VALUES('0.0');             INSERT INTO t2 VALUES(0.0);


In the sequence on the left, the second insert will fail.  In this case,
the strings '0' and '0.0' are treated as numbers since they are being 
inserted into a numeric column but 0==0.0 which violates the uniqueness
constraint.  However, the second insert in the right-hand sequence works.  In
this case, the constants 0 and 0.0 are treated a strings which means that
they are distinct.

SQLite always converts numbers into double-precision (64-bit) floats
for comparison purposes.  This means that a long sequence of digits that
differ only in insignificant digits will compare equal if they
are in a numeric column but will compare unequal if they are in a text
column.  We have:


INSERT INTO t1                            INSERT INTO t2
   VALUES('12345678901234567890');           VALUES(12345678901234567890);
INSERT INTO t1                            INSERT INTO t2
   VALUES('12345678901234567891');           VALUES(12345678901234567891);


As before, the second insert on the left will fail because the comparison
will convert both strings into floating-point number first and the only
difference in the strings is in the 20-th digit which exceeds the resolution
of a 64-bit float.  In contrast, the second insert on the right will work
because in that case, the numbers being inserted are strings and are
compared using memcmp().


Numeric and text types make a difference for the DISTINCT keyword too:



CREATE TABLE t3(a INTEGER);               CREATE TABLE t4(b TEXT);
INSERT INTO t3 VALUES('0');               INSERT INTO t4 VALUES(0);
INSERT INTO t3 VALUES('0.0');             INSERT INTO t4 VALUES(0.0);
SELECT DISTINCT * FROM t3;                SELECT DISTINCT * FROM t4;



The SELECT statement on the left returns a single row since '0' and '0.0'
are treated as numbers and are therefore indistinct.  But the SELECT 
statement on the right returns two rows since 0 and 0.0 are treated
a strings which are different.


c2: Quirks, Caveats, and Gotchas In SQLite
c5: quirks.html
c4:
1. Overview


The SQL language is a "standard".
Even so, no two SQL database engines work exactly alike.
Every SQL implementation has it own peculiarities and oddities,
and SQLite is no exception to this rule.


This document strives to highlight the principal differences
between SQLite and other SQL implementations, as an aid to developers
that are porting to or from SQLite or who are trying to build a
system that works across multiple database engines.


If you are an SQLite user whose has stumbled over some quirk of
SQLite that is not mentioned here, please send us an email so that
we can document the problem.

2. SQLite Is Embedded, Not Client-Server


Whenever comparing SQLite to other SQL database engines like
SQL Server, PostgreSQL, MySQL, or Oracle, it is important first of all
to realize that SQLite is not intended as a replacement or competitor to
any of those systems.  SQLite is serverless.  There is no separate
server process that manages the database.  An application interacts
with the database engine using function calls, not by sending messages
to a separate process or thread.


The fact that SQLite is embedded and serverless instead of being
client/server is a feature, not a bug.


Client/server databases like MySQL, PostgreSQL, SQL Server, Oracle, and
others are an important component of modern systems.  
These systems solve an important problem.  
But SQLite solves a different problem.
Both SQLite and client/server databases have their role.
Developers who are comparing SQLite against other SQL database engines need
to clearly understand this distinction.


See the Appropriate Uses For SQLite document for additional information.

3. Flexible Typing


SQLite is flexible with regard to datatypes.  Datatypes are advisory
rather than mandatory.


Some commentators say that SQLite is "weakly typed" and that other
SQL databases are "strongly typed".  We consider these terms to be
inaccurate and even pejorative.  We prefer to say that SQLite is 
"flexibly typed" and that other SQL database engines are
"rigidly typed".


See the Datatypes in SQLite document for a detailed
discussion of the type system in SQLite.


The key point is that SQLite is very forgiving of the type of data that
you put into the database.  For example, if a column has a datatype of
"INTEGER" and the application inserts a text string into that column,
SQLite will first try to convert the text string into an integer, just like
every other SQL database engine.  Thus, if one inserts '1234' into
an INTEGER column, that value is converted into an integer 1234 and stored.
But, if you insert a non-numeric string like 'wxyz' into an INTEGER
column, unlike other SQL databases, SQLite does not throw an error.  Instead,
SQLite stores the actual string value in the column.


Similarly, SQLite allows you to store a 2000-character string into a
column of type VARCHAR(50).  Other SQL implementations would either throw
an error or truncate the string.  SQLite stores the entire 2000-character
string with no loss of information and without complaint.


Where this ends up causing problems is when developers do some initial
coding work using SQLite and get their application working, but then try
to convert to another database like PostgreSQL or SQL Server for deployment.
If the application is initially taking advantage of SQLite's flexible typing,
then it will fail when moved to another database that uses a more rigid
and unforgiving type enforcement policy.


Flexible typing is a feature of SQLite, not a bug.  Flexible typing
is about freedom.
Nevertheless, we recognize that this feature does sometimes cause
confusion for developers who are accustomed to working with
other databases that are more strict with regard to data type rules.
In retrospect, perhaps it would have been less confusing if SQLite had merely
implemented an ANY datatype so that developers could explicitly state
when they wanted to use flexible typing, rather than making flexible
typing the default.
As an accommodation for those who expect rigid typing, SQLite version 3.37.0
(2021-11-27) introduced the option of STRICT tables.
These either impose
the mandatory datatype constraints found in other SQL database engines,
or allow the explicit ANY datatype to retain SQLite's flexible typing.

3.1. No Separate BOOLEAN Datatype


Unlike most other SQL implementations,
SQLite does not have a separate BOOLEAN data type.
Instead, TRUE and FALSE are (normally) represented as integers 1 and 0,
respectively.
This does not seem to cause many problems, as we seldom get complaints
about it.  But it is important to recognize.


Beginning with SQLite version 3.23.0 (2018-04-02), SQLite also
recognizes TRUE and FALSE keywords as aliases for integer values 1 and 0,
respectively.
This provides better compatibility with other SQL implementations.
But for backwards compatibility, if there are columns named TRUE or
FALSE, then the keywords are treated as identifiers referencing those
columns, rather than BOOLEAN literals.

3.2. No Separate DATETIME Datatype


SQLite has no DATETIME datatype.
Instead, dates and times can be stored in any of these ways:


 As a TEXT string in the ISO-8601 format.  Example: '2018-04-02 12:13:46'.
 As an INTEGER number of seconds since 1970 (also known as "unix time").
 As a REAL value that is the fractional 
     Julian day number.



The built-in date and time functions of SQLite understand date/times in
all of the formats above, and can freely change between them.
Which format you use, is entirely up to your application.

3.3. The datatype is optional


Because SQLite is flexible and forgiving with regard to datatypes,
table columns can be created that have no specified datatype at all.
For example:
CREATE TABLE t1(a,b,c,d);

The table "t1" has four columns "a", "b", "c", and "d" that have
no particular datatype assigned.  You can store anything you want in
any of those columns.

4. Foreign Key Enforcement Is Off By Default

SQLite has parsed foreign key constraints for time out of mind,
but added the ability to actually enforce those constraints much later,
with version 3.6.19 (2009-10-14).  By the time foreign key constraint
enforcement was added, there were already countless millions of databases
in circulation that contained foreign key constraints, some of which
were not correct.  To avoid breaking those legacy databases, foreign key
constraint enforcement is turned off by default in SQLite.

Applications can activate foreign key enforcement at run-time using
the PRAGMA foreign_keys statement.  Or, foreign key enforcement can
be activated at compile-time using the
-DSQLITE_DEFAULT_FOREIGN_KEYS=1 compile-time option.

5. PRIMARY KEYs Can Sometimes Contain NULLs


Usually (the exceptions are INTEGER PRIMARY KEY tables and
WITHOUT ROWID tables) a PRIMARY KEY in an SQLite table is really
the same as a UNIQUE constraint.  Due to an historical oversight,
the column values of such a PRIMARY KEY are allowed to be NULL.
This is a bug, but by the time the problem was discovered there
where so many databases in circulation that depended on the bug that
the decision was made to support the buggy behavior moving forward.

The value of an INTEGER PRIMARY KEY column must always be a 
non-NULL integer.  The PRIMARY KEY columns of a WITHOUT ROWID
table are also required to be non-NULL.

6. Aggregate Queries Can Contain Non-Aggregate Result Columns
That Are Not In The GROUP BY Clause


In most SQL implementations, output columns of an aggregate query
may only reference aggregate functions or columns named in the
GROUP BY clause.  It does not make good sense to reference an ordinary
column in an aggregate query because each output row might be composed
from two or more rows in the input table(s).


SQLite does not enforce this restriction.
The output columns from an aggregate query can be arbitrary expressions
that include columns not found in GROUP BY clause.
This feature has two uses:



With SQLite (but not any other SQL implementation that we know of) if
an aggregate query contains a single min() or max() function, then the
values of columns used in the output are taken from the row where
the min() or max() value was achieved.  If two or more rows have the
same min() or max() value, then the columns values will be chosen arbitrarily
from one of those rows.

For example to find the highest paid employee:
SELECT max(salary), first_name, last_name FROM employee;


In the query above, the values for the first_name and last_name columns
will correspond to the row that satisfied the max(salary) condition.


If a query contains no aggregate functions at all, then a GROUP BY
clause can be added as a substitute for the DISTINCT ON clause.  In other words,
output rows are filtered so that only one row is shown for each distinct
set of values in the GROUP BY clause.  If two or more output rows would
have otherwise had the same set of values for the GROUP BY columns, then
one of the rows is chosen arbitrarily.  (SQLite supports DISTINCT but not
DISTINCT ON, whose functionality is provided instead by GROUP BY.)


7. Does Not Do Full Unicode Case Folding By Default


SQLite does not know about the upper-case/lower-case distinction
for all unicode characters.  SQL functions like
upper() and lower() only work on ASCII characters.  There are two
reasons for this:

 Though stable now, when SQLite was first designed, the rules for
     unicode case folding were still in flux.  That means that the
     behavior might have changed with each new unicode release, disrupting
     applications and corrupting indexes in the process.
 The tables necessary to do full and proper unicode case folding are
     larger than the whole SQLite library.


Full unicode case folding is supported in SQLite if it is compiled
with the -DSQLITE_ENABLE_ICU option and linked against the
International Components for Unicode
library.



8. Double-quoted String Literals Are Accepted


The SQL standard requires double-quotes around identifiers
and single-quotes around string literals.  For example:

 "this is a legal SQL column name"
 'this is an SQL string literal'


SQLite accepts both of the above.  But, in an effort to be compatible
with MySQL 3.x (which was one of the most widely used RDBMSes
when SQLite was first being designed) SQLite will also interpret
a double-quotes string as
string literal if it does not match any valid identifier.

This misfeature means that a misspelled double-quoted
identifier will be interpreted as a string literal, rather than generating
an error.
It also lures developers who are new to the SQL language into the
bad habit of using double-quoted string literals when they
really need to learn to use the correct single-quoted string literal form.

In hindsight, we should not have tried to make SQLite accept MySQL 3.x
syntax, and should have never allowed double-quoted string literals.
However, there are countless applications that make use of
double-quoted string literals and so we continue to support
that capability to avoid breaking legacy.

As of SQLite 3.27.0 (2019-02-07) the use of a double-quoted
string literal causes a warning message to be sent to the error log.

As of SQLite 3.29.0 (2019-07-10) the use of double-quoted
string literals can be disabled at run-time using the
SQLITE_DBCONFIG_DQS_DDL and SQLITE_DBCONFIG_DQS_DML actions
to sqlite3_db_config().  The default settings can be altered
at compile-time using the -DSQLITE_DQS=N compile-time
option.  Application developers are encouraged to compile using
-DSQLITE_DQS=0 in order to disable the double-quoted string literal
misfeature by default.  If that is not possible, then disable
double-quoted string literals for individual database connections
using C-code like this:

sqlite3_db_config(db, SQLITE_DBCONFIG_DQS_DDL, 0, (void*)0);
sqlite3_db_config(db, SQLITE_DBCONFIG_DQS_DML, 0, (void*)0);

Or, if double-quoted string literals are disabled by default, but need
to be selectively enabled for some historical database connections,
that can be done using the same C-code as shown above except with the
third parameter changed from 0 to 1.

9. Keywords Can Often Be Used As Identifiers


The SQL language is rich in keywords.
Most SQL implementations do not allow keywords to be used as identifiers
(names of tables or columns) unless they are enclosed in double-quotes.
But SQLite is more flexible.  Many keywords can be used as identifiers without
needing to be quoted, as long as those keywords are used in a context where
it is clear that they are intended to be an identifier.

For example, the following statement is valid in SQLite:
CREATE TABLE union(true INT, with BOOLEAN);


The same SQL statement will fail on every other SQL implementation that
we know of due to the use of keywords "union", "true", and "with" as
identifiers.

The ability to use keywords as identifiers promotes backwards compatibility.
As new keywords are added, legacy schemas that just happen to use those
keywords as table or column names continue to work.  However, the ability
to use a keyword as an identifier sometimes leads to surprising outcomes.
For example:
CREATE TRIGGER AFTER INSERT ON tableX BEGIN
  INSERT INTO tableY(b) VALUES(new.a);
END;

The trigger created by the previous statement is named "AFTER"
and it is a "BEFORE" trigger.  The "AFTER" token is used as an identifier
instead of as a keyword, as that is the only way to parse the statement.
Another example:
CREATE TABLE tableZ(INTEGER PRIMARY KEY);

The tableZ table has a single column named "INTEGER".  That column
has no datatype specified, but it is the PRIMARY KEY.
The column is not the INTEGER PRIMARY KEY for the table because
it has no datatype.  The "INTEGER" token is used as an identifier for the
column name, not as a datatype keyword.

10. Dubious SQL Is Allowed Without Any Error Or Warning


The original implementation of SQLite sought to follow
Postel's Law which
states in part "Be liberal in what you accept".
This used to be considered good design - that a system would accept
dodgy inputs and try to do the best it could without complaining too much.
But lately, people have come to realize that it is sometimes better to
be strict in what you accept, so as to more easily find errors in the
input.


11. AUTOINCREMENT Does Not Work The Same As MySQL

The AUTOINCREMENT feature in SQLite works differently than
it does in MySQL.  This often causes confusion for people who
initially learned SQL on MySQL and then start using SQLite, and
expect the two systems to work identically.

See the SQLite AUTOINCREMENT documentation for
detailed instructions on what AUTOINCREMENT does and does not do
in SQLite.

12. NUL Characters Are Allowed In Text Strings

NUL characters (ASCII code 0x00 and Unicode \u0000) may appear in
the middle of strings in SQLite.  This can lead to unexpected behavior.
See the "NUL characters in strings" document for further information.


c2: The Spellfix1 Virtual Table
c5: spellfix1.html
c4:
1. Overview

This spellfix1 virtual table can be used to search
a large vocabulary for close matches.  For example, spellfix1
can be used to suggest corrections to misspelled words.  Or,
it could be used with FTS4 to do full-text search using potentially
misspelled words.

The implementation for the spellfix1 virtual table is held in the
SQLite source tree in the miscellaneous extensions folder and in
particular in the file 
ext/misc/spellfix1.c.
The spellfix1 virtual table is not included in the SQLite amalgamation
and is not a part of any standard SQLite build.  It is a loadable extension.

Once the spellfix1 extension is loaded, an instance of the spellfix1 
virtual table is created like this:


CREATE VIRTUAL TABLE demo USING spellfix1;


The "spellfix1" term is the name of the spellfix module and must be 
entered as shown.  The "demo" term is the
name of the virtual table you will be creating and can be altered
to suit the needs of your application.  The virtual table is initially
empty.  In order for the virtual table to be useful, you will need to
populate it with your vocabulary.  Suppose you
have a list of words in a table named "big_vocabulary".  Then do this:


INSERT INTO demo(word) SELECT word FROM big_vocabulary;


If you intend to use this virtual table in cooperation with an FTS4
table (for spelling correction of search terms) then you might extract
the vocabulary using an fts4aux table:


INSERT INTO demo(word) SELECT term FROM search_aux WHERE col='*';


You can also provide the virtual table with a "rank" for each word.
The "rank" is an estimate of how common the word is.  Larger numbers
mean the word is more common.  If you omit the rank when populating
the table, then a rank of 1 is assumed.  But if you have rank 
information, you can supply it and the virtual table will show a
slight preference for selecting more commonly used terms.  To
populate the rank from an fts4aux table "search_aux" do something
like this:


INSERT INTO demo(word,rank)
   SELECT term, documents FROM search_aux WHERE col='*';


To query the virtual table, include a MATCH operator in the WHERE
clause.  For example:


SELECT word FROM demo WHERE word MATCH 'kennasaw';


Using a dataset of American place names (derived from
http://geonames.usgs.gov/domestic/download_data.htm) the query above
returns 20 results beginning with:


kennesaw
kenosha
kenesaw
kenaga
keanak


If you append the character '*' to the end of the pattern, then
a prefix search is performed.  For example:


SELECT word FROM demo WHERE word MATCH 'kennes*';


Yields 20 results beginning with:


kennesaw
kennestone
kenneson
kenneys
keanes
keenes


2. Search Refinements

By default, the spellfix1 table returns no more than 20 results.
(It might return less than 20 if there were fewer good matches.)
You can change the upper bound on the number of returned rows by
adding a "top=N" term to the WHERE clause of your query, where N
is the new maximum.  For example, to see the 5 best matches:


SELECT word FROM demo WHERE word MATCH 'kennes*' AND top=5;


Each entry in the spellfix1 virtual table is associated with
a particular language, identified by the integer "langid" column.
The default langid is 0 and if no other actions are taken, the
entire vocabulary is a part of the 0 language.  But if your application
needs to operate in multiple languages, then you can specify different
vocabulary items for each language by specifying the langid field
when populating the table.  For example:


INSERT INTO demo(word,langid) SELECT word, 0 FROM en_vocabulary;
INSERT INTO demo(word,langid) SELECT word, 1 FROM de_vocabulary;
INSERT INTO demo(word,langid) SELECT word, 2 FROM fr_vocabulary;
INSERT INTO demo(word,langid) SELECT word, 3 FROM ru_vocabulary;
INSERT INTO demo(word,langid) SELECT word, 4 FROM cn_vocabulary;


After the virtual table has been populated with items from multiple
languages, specify the language of interest using a "langid=N" term
in the WHERE clause of the query:


SELECT word FROM demo WHERE word MATCH 'hildes*' AND langid=1;


Note that if you do not include the "langid=N" term in the WHERE clause,
the search will be against language 0 (English in the example above.)
All spellfix1 searches are against a single language id.  There is no
way to search all languages at once.
 

3. Virtual Table Details

Each row in the spellfix1 virtual table has a unique rowid 
with seven columns plus five extra hidden columns.
The columns are as follows:


rowid
A unique integer number associated with each
vocabulary item in the table.  This can be used
as a foreign key on other tables in the database.

word
The text of the word that matches the pattern.
Both word and pattern can contain unicode characters
and can be mixed case.

rank
This is the rank of the word, as specified in the
original INSERT statement.


distance
This is an edit distance or Levenshtein distance going
from the pattern to the word.

langid
This is the language-id of the word.  All queries are
against a single language-id, which defaults to 0.
For any given query this value is the same on all rows.

score
The score is a combination of rank and distance.  The
idea is that a lower score is better.  The virtual table
attempts to find words with the lowest score and 
by default (unless overridden by ORDER BY) returns
results in order of increasing score.

matchlen
In a prefix search, the matchlen is the number of characters in
the string that match against the prefix.  For a non-prefix search,
this is the same as length(word).

phonehash
This column shows the phonetic hash prefix that was used to restrict
the search.  For any given query, this column should be the same for
every row.  This information is available for diagnostic purposes and
is not normally considered useful in real applications.

top
(HIDDEN)  For any query, this value is the same on all
rows.  It is an integer which is the maximum number of
rows that will be output.  The actually number of rows
output might be less than this number, but it will never
be greater.  The default value for top is 20, but that
can be changed for each query by including a term of
the form "top=N" in the WHERE clause of the query.

scope
(HIDDEN)  For any query, this value is the same on all
rows.  The scope is a measure of how widely the virtual
table looks for matching words.  Smaller values of
scope cause a broader search.  The scope is normally
chosen automatically and is capped at 4.  Applications
can change the scope by including a term of the form
"scope=N" in the WHERE clause of the query.  Increasing
the scope will make the query run faster, but will reduce
the possible corrections.

srchcnt
(HIDDEN)  For any query, this value is the same on all
rows.  This value is an integer which is the number of
words examined using the edit-distance algorithm to
find the top matches that are ultimately displayed.  This
value is for diagnostic use only.

soundslike
(HIDDEN)  When inserting vocabulary entries, this field
can be set to a spelling that matches what the word
sounds like.  See the DEALING WITH UNUSUAL AND DIFFICULT
SPELLINGS section below for details.

command
(HIDDEN)  The value of the "command" column is always NULL.  However,
applications can insert special strings into the "command" column in order
to provoke certain behaviors in the spellfix1 virtual table.
For example, inserting the string 'reset' into the "command" column
will cause the virtual table to reread its edit distance weights
(if there are any).


4. Algorithm

The spellfix1 virtual table creates a single
shadow table named "%_vocab" (where the % is replaced by the name of
the virtual table; Ex: "demo_vocab" for the "demo" virtual table).  
the shadow table contains the following columns:


id
The unique id (INTEGER PRIMARY KEY)

rank
The rank of word.

langid
The language id for this entry.

word
The original UTF8 text of the vocabulary word

k1
The word transliterated into lower-case ASCII.  
There is a standard table of mappings from non-ASCII
characters into ASCII.  Examples: "æ" -> "ae",
"þ" -> "th", "ß" -> "ss", "á" -> "a", ...  The
accessory function spellfix1_translit(X) will do
the non-ASCII to ASCII mapping.  The built-in lower(X)
function will convert to lower-case.  Thus:
k1 = lower(spellfix1_translit(word)).

If the word is already all lower-case ASCII, then the k1 column
will contain a NULL.  This reduces the storage requirements for
the %_vocab table and helps spellfix to run a little faster.
Therefore, it is advantageous to populate as much of the spellfix
table as possible using lower-case ASCII vocabulary.

k2
This field holds a phonetic code derived from coalesce(k1,word).
Letters that have similar sounds are mapped into the same symbol.
For example, all vowels and vowel clusters become the
single symbol "A".  And the letters "p", "b", "f", and
"v" all become "B".  All nasal sounds are represented
as "N".  And so forth.  The mapping is based on
ideas found in Soundex, Metaphone, and other
long-standing phonetic matching systems.  This key can
be generated by the function spellfix1_phonehash(X).  
Hence: k2 = spellfix1_phonehash(coalesce(k1,word))


There is also a function for computing the Wagner edit distance or the
Levenshtein distance between a pattern and a word.  This function
is exposed as spellfix1_editdist(X,Y).  The edit distance function
returns the "cost" of converting X into Y.  Some transformations
cost more than others.  Changing one vowel into a different vowel,
for example is relatively cheap, as is doubling a constant, or
omitting the second character of a double-constant.  Other transformations
or more expensive.  The idea is that the edit distance function returns
a low cost for words that are similar and a higher cost for words
that are further apart.  In this implementation, the maximum cost
of any single-character edit (delete, insert, or substitute) is 100,
with lower costs for some edits (such as transforming vowels).

The "score" for a comparison is the edit distance between the pattern
and the word, adjusted down by the base-2 logarithm of the word rank.
For example, a match with distance 100 but rank 1000 would have a
score of 122 (= 100 - log2(1000) + 32) whereas a match with distance
100 with a rank of 1 would have a score of 131 (100 - log2(1) + 32).
(NB:  The constant 32 is added to each score to keep it from going
negative in case the edit distance is zero.)  In this way, frequently
used words get a slightly lower cost which tends to move them toward
the top of the list of alternative spellings.

A straightforward implementation of a spelling corrector would be
to compare the search term against every word in the vocabulary
and select the 20 with the lowest scores.  However, there will 
typically be hundreds of thousands or millions of words in the
vocabulary, and so this approach is not fast enough.

Suppose the term that is being spell-corrected is X.  To limit
the search space, X is converted to a k2-like key using the
equivalent of:


   key = spellfix1_phonehash(lower(spellfix1_translit(X)))


This key is then limited to "scope" characters.  The default scope
value is 4, but an alternative scope can be specified using the
"scope=N" term in the WHERE clause.  After the key has been truncated,
the edit distance is run against every term in the vocabulary that
has a k2 value that begins with the abbreviated key.

For example, suppose the input word is "Paskagula".  The phonetic 
key is "BACACALA" which is then truncated to 4 characters "BACA".
The edit distance is then run on the 4980 entries (out of
272,597 entries total) of the vocabulary whose k2 values begin with
BACA, yielding "Pascagoula" as the best match.

Only terms of the vocabulary with a matching langid are searched.
Hence, the same table can contain entries from multiple languages
and only the requested language will be used.  The default langid
is 0.



5. Configurable Edit Distance

The built-in Wagner edit-distance function with fixed weights can be
replaced by the editdist3() edit-distance function
with application-defined weights and support for unicode, by specifying
the "edit_cost_table=TABLENAME" parameter to the spellfix1 module
when the virtual table is created.
For example:


CREATE VIRTUAL TABLE demo2 USING spellfix1(edit_cost_table=APPCOST);


The editdist3() edit-distance function can also be selected or 
deselected at run-time by inserting an appropriate string into the
"command" column of the virtual table:


INSERT INTO demo2(command) VALUES('edit_cost_table=APPCOST');



In the examples above, the APPCOST table would be interrogated to find
the edit distance coefficients.  It is the presence of the "edit_cost_table="
parameter to the spellfix1 module name that causes editdist3() to be used
in place of the built-in edit distance function.  If APPCOST is an empty
string, then the built-in Wagner edit-distance function is used.

The edit distance coefficients are normally read from the APPCOST table
once and there after stored in memory.  Hence, run-time changes to the
APPCOST table will not normally affect the edit distance results.
However, inserting the special string 'reset' into the "command" column of the
virtual table causes the edit distance coefficients to be reread the
APPCOST table.  Hence, applications should run a SQL statement similar
to the following when changes to the APPCOST table occur:


INSERT INTO demo2(command) VALUES("reset");


6. Dealing With Unusual And Difficult Spellings

The algorithm above works quite well for most cases, but there are
exceptions.  These exceptions can be dealt with by making additional
entries in the virtual table using the "soundslike" column.

For example, many words of Greek origin begin with letters "ps" where
the "p" is silent.  Ex:  psalm, pseudonym, psoriasis, psyche.  In
another example, many Scottish surnames can be spelled with an
initial "Mac" or "Mc".  Thus, "MacKay" and "McKay" are both pronounced
the same.

Accommodation can be made for words that are not spelled as they
sound by making additional entries into the virtual table for the
same word, but adding an alternative spelling in the "soundslike"
column.  For example, the canonical entry for "psalm" would be this:


  INSERT INTO demo(word) VALUES('psalm');


To enhance the ability to correct the spelling of "salm" into
"psalm", make an addition entry like this:


  INSERT INTO demo(word,soundslike) VALUES('psalm','salm');


It is ok to make multiple entries for the same word as long as
each entry has a different soundslike value.  Note that if no
soundslike value is specified, the soundslike defaults to the word
itself.

Listed below are some cases where it might make sense to add additional
soundslike entries.  The specific entries will depend on the application
and the target language.


Silent "p" in words beginning with "ps":  psalm, psyche
Silent "p" in words beginning with "pn":  pneumonia, pneumatic
Silent "p" in words beginning with "pt":  pterodactyl, ptolemaic
Silent "d" in words beginning with "dj":  djinn, Djikarta
Silent "k" in words beginning with "kn":  knight, Knuthson
Silent "g" in words beginning with "gn":  gnarly, gnome, gnat
"Mac" versus "Mc" beginning Scottish surnames
"Tch" sounds in Slavic words:  Tchaikovsky vs. Chaykovsky
The letter "j" pronounced like "h" in Spanish:  LaJolla
Words beginning with "wr" versus "r":  write vs. rite
Miscellaneous problem words such as "debt", "tsetse",
      "Nguyen", "Van Nuyes".


7. Auxiliary Functions

The source code module that implements the spellfix1 virtual table also
implements several SQL functions that might be useful to applications
that employ spellfix1 or for testing or diagnostic work while developing
applications that use spellfix1.  The following auxiliary functions are
available:


editdist3(P,W)editdist3(P,W,L)editdist3(T)
These routines provide direct access to the version of the Wagner
edit-distance function that allows for application-defined weights
on edit operations.  The first two forms of this function compare
pattern P against word W and return the edit distance.  In the first
function, the langid is assumed to be 0 and in the second, the
langid is given by the L parameter.  The third form of this function
reloads edit distance coefficients from the table named by T.

spellfix1_editdist(P,W)
This routine provides access to the built-in Wagner edit-distance
function that uses default, fixed costs.  The value returned is
the edit distance needed to transform W into P.

spellfix1_phonehash(X)
This routine constructs a phonetic hash of the pure ascii input word X
and returns that hash.  This routine is used internally by spellfix1 in
order to transform the K1 column of the shadow table into the K2
column.

spellfix1_scriptcode(X)
Given an input string X, this routine attempts to determine the dominant
script of that input and returns the ISO-15924 numeric code for that
script.  The current implementation understands the following scripts:

 215 - Latin
 220 - Cyrillic
 200 - Greek

Additional language codes might be added in future releases.

spellfix1_translit(X)
This routine transliterates unicode text into pure ascii, returning
the pure ascii representation of the input text X.  This is the function
that is used internally to transform vocabulary words into the K1
column of the shadow table.





8. The editdist3 function

The editdist3 algorithm is a function that computes the minimum edit 
distance (a.k.a. the Levenshtein distance) between two input strings.
The editdist3 algorithm is a configurable alternative to the default
edit distance function of spellfix1.
Features of editdist3 include:


It works with unicode (UTF8) text.

A table of insertion, deletion, and substitution costs can be 
       provided by the application.

Multi-character insertions, deletions, and substitutions can be
       enumerated in the cost table.


9. The editdist3 COST table

To program the costs of editdist3, create a table such as the following:


CREATE TABLE editcost(
  iLang INT,   -- The language ID
  cFrom TEXT,  -- Convert text from this
  cTo   TEXT,  -- Convert text into this
  iCost INT    -- The cost of doing the conversion
);


The cost table can be named anything you want - it does not have to be
called "editcost".  And the table can contain additional columns.
The only requirement is that the
table must contain the four columns show above, with exactly the names shown.

The iLang column is a non-negative integer that identifies a set of costs
appropriate for a particular language.  The editdist3 function will only use
a single iLang value for any given edit-distance computation.  The default
value is 0.  It is recommended that applications that only need to use a
single language always use iLang==0 for all entries.

The iCost column is the numeric cost of transforming cFrom into cTo.  This
value should be a non-negative integer, and should probably be less than 100.
The default single-character insertion and deletion costs are 100 and the
default single-character to single-character substitution cost is 150.  A
cost of 10000 or more is considered "infinite" and causes the rule to be
ignored.

The cFrom and cTo columns show edit transformation strings.  Either or both
columns may contain more than one character.  Or either column (but not both)
may hold an empty string.  When cFrom is empty, that is the cost of inserting
cTo.  When cTo is empty, that is the cost of deleting cFrom.

In the spellfix1 algorithm, cFrom is the text as the user entered it and
cTo is the correctly spelled text as it exists in the database.  The goal
of the editdist3 algorithm is to determine how close the user-entered text is
to the dictionary text.

There are three special-case entries in the cost table:


cFromcToMeaning
'''?'The default insertion cost
'?'''The default deletion cost
'?''?'The default substitution cost


If any of the special-case entries shows above are omitted, then the
value of 100 is used for insertion and deletion and 150 is used for
substitution.  To disable the default insertion, deletion, and/or substitution
set their respective cost to 10000 or more.

Other entries in the cost table specific transforms for particular 
characters.
The cost of specific transforms should be less than the default costs, or else
the default costs will take precedence and the specific transforms will never 
be used.

Some example, cost table entries:


INSERT INTO editcost(iLang, cFrom, cTo, iCost)
VALUES(0, 'a', 'ä', 5);


The rule above says that the letter "a" in user input can be matched against
the letter "ä" in the dictionary with a penalty of 5.


INSERT INTO editcost(iLang, cFrom, cTo, iCost)
VALUES(0, 'ss', 'ß', 8);


The number of characters in cFrom and cTo do not need to be the same.  The
rule above says that "ss" on user input will match "ß" with a penalty of 8.

10. Experimenting with the editcost3() function

The spellfix1 virtual table
uses editdist3 if the "edit_cost_table=TABLE" option
is specified as an argument when the spellfix1 virtual table is created.  
But editdist3 can also be tested directly using the built-in "editdist3()"
SQL function.  The editdist3() SQL function has 3 forms:


 editdist3('TABLENAME');
 editdist3('string1', 'string2');
 editdist3('string1', 'string2', langid);


The first form loads the edit distance coefficients from a table called
'TABLENAME'.  Any prior coefficients are discarded.  So when experimenting
with weights and the weight table changes, simply rerun the single-argument
form of editdist3() to reload revised coefficients.  Note that the 
edit distance
weights used by the editdist3() SQL function are independent from the
weights used by the spellfix1 virtual table.

The second and third forms return the computed edit distance between strings
'string1' and "string2'.  In the second form, a language id of 0 is used.
The language id is specified in the third form.


c2: The DBSTAT Virtual Table
c5: dbstat.html
c4:
1. Overview


The DBSTAT virtual table is a read-only eponymous virtual table that returns
information about the amount of disk space used to store the content
of an SQLite database.
Example use cases for the
DBSTAT virtual table include the sqlite3_analyzer.exe
utility program and the
table size pie-chart in
the Fossil-implemented version control system
for SQLite.



The DBSTAT virtual table is available on all 
database connections when SQLite is built using the
SQLITE_ENABLE_DBSTAT_VTAB compile-time option.


The DBSTAT virtual table is an eponymous virtual table, meaning
that is not necessary to run CREATE VIRTUAL TABLE to create an
instance of the dbstat virtual table before using it.  The "dbstat"
module name can be used as if it were a table name to query the
dbstat virtual table directly.  For example:

SELECT * FROM dbstat;



If a named virtual table that uses the dbstat module is desired,
then the recommended way to create an instance of the dbstat
virtual table is as follows:

CREATE VIRTUAL TABLE temp.stat USING dbstat(main);



Note the "temp." qualifier before the virtual table name ("stat").  This
qualifier causes the virtual table to be temporary - to only exist for
the duration of the current database connection.  This is the
recommended approach.


The "main" argument to dbstat is default schema
for which information is to be provided.  The default is "main", and
so the use of "main" in the example above is redundant.  For any
particular query, the schema can be changed by specifying the
alternative schema as a function argument to the virtual table
name in the FROM clause of the query.  (See further discussion of
table-valued functions in the FROM clause for more details.)


The schema for the DBSTAT virtual table looks like this:
CREATE TABLE dbstat(
  name       TEXT,        -- Name of table or index
  path       TEXT,        -- Path to page from root
  pageno     INTEGER,     -- Page number, or page count
  pagetype   TEXT,        -- 'internal', 'leaf', 'overflow', or NULL
  ncell      INTEGER,     -- Cells on page (0 for overflow pages)
  payload    INTEGER,     -- Bytes of payload on this page or btree
  unused     INTEGER,     -- Bytes of unused space on this page or btree
  mx_payload INTEGER,     -- Largest payload size of all cells on this row
  pgoffset   INTEGER,     -- Byte offset of the page in the database file
  pgsize     INTEGER,     -- Size of the page, in bytes
  schema     TEXT HIDDEN, -- Database schema being analyzed
  aggregate  BOOL HIDDEN  -- True to enable aggregate mode
);



The DBSTAT table only reports on the content of btrees within the database file.
Freelist pages, pointer-map pages, and the lock page are omitted from
the analysis.


By default, there is a single row in the DBSTAT table for each
btree page the database file.  Each row provides
information about the space utilization of that one page of the
database.  However, if the hidden column "aggregate" is TRUE, then
results are aggregated and there is a single row in the DBSTAT table
for each btree in the database, providing information about space
utilization across the entire btree.


2. The "path" column of the dbstat virtual table


The "path" column describes the path taken from the 
root node of the btree structure to each page.  The
"path" of the root node itself is '/'.
The "path" is NULL when "aggregate" is TRUE.

The "path" for the left-most child page of the root of
a btree page is '/000/'. (Btrees store content ordered from left to right
so the pages to the left have smaller keys than the pages to the right.)
The next to left-most child of the root page is '/001', and so on,
each sibling page identified by a 3-digit hex  value.
The children of the 451st left-most sibling have paths such
as '/1c2/000/, '/1c2/001/' etc.

Overflow pages are specified by appending a '+' character and a 
six-digit hexadecimal value to the path to the cell they are linked
from. For example, the three overflow pages in a chain linked from 
the left-most cell of the 450th child of the root page are identified
by the paths:

'/1c2/000+000000'         // First page in overflow chain
'/1c2/000+000001'         // Second page in overflow chain
'/1c2/000+000002'         // Third page in overflow chain



If the paths are sorted using the BINARY collation sequence, then
the overflow pages associated with a cell will appear earlier in the
sort-order than its child page:

'/1c2/000/'               // Left-most child of 451st child of root




3. Aggregated Data


Beginning with SQLite version 3.31.0 (2020-01-22), the DBSTAT table
has a new hidden column named "aggregate", which if constrained to be
TRUE will cause DBSTAT to generate one row per btree in the database,
rather than one row per page.  When running in aggregated mode, the 
"path", "pagetype", and "pgoffset" columns are always NULL and the
"pageno" column holds the number of pages in the entire btree, rather
than the number of the page that corresponds to the row.


The following table shows the meanings of the (non-hidden) columns of
DBSTAT in both normal and aggregated mode:




Column
Normal meaning
Aggregate-mode meaning


name

The name of the table or index that is implemented by
the btree of the current row

path
See description above
Always NULL

pageno
The page number of the database page for the current row
The total number of pages in the btree for the current row

pagetype
'leaf' or 'interior'
Always NULL

ncell
Number of cells on the current page or btree

payload
Bytes of useful payload on the current page or btree

unused
Unused bytes of on the current page or btree

mx_payload
The largest payload found anywhere in the current page
or btree.

pgoffset
Byte offset to the start of the page
Always NULL

pgsize
Total storage space used by the current page or btree.



4. Example uses of the dbstat virtual table


To find the total number of pages used to store table "xyz" in schema "aux1",
use either of the following two queries (the first is the traditional way,
and the second shows the use of the aggregated feature):

SELECT count(*) FROM dbstat('aux1') WHERE name='xyz';
SELECT pageno FROM dbstat('aux1',1) WHERE name='xyz';



To see how efficiently the content of a table is stored on disk,
compute the amount of space used to hold actual content divided
by the total amount of disk space used.  The closer this number
is to 100%, the more efficient the packing.  (In this example, the
'xyz' table is assumed to be in the 'main' schema.  Again, there
are two different versions that show the use of DBSTAT both without
and with the new aggregated feature, respectively.)

SELECT sum(pgsize-unused)*100.0/sum(pgsize) FROM dbstat WHERE name='xyz';
SELECT (pgsize-unused)*100.0/pgsize FROM dbstat
 WHERE name='xyz' AND aggregate=TRUE;



To find the average fan-out for a table, run:

SELECT avg(ncell) FROM dbstat WHERE name='xyz' AND pagetype='internal';




Modern filesystems operate faster when disk accesses are sequential.
Hence, SQLite will run faster if the content of the database file
is on sequential pages.  To find out what fraction of the pages in
a database are sequential (and thus obtain a measurement that might
be useful in determining when to VACUUM), run a query like the following:

CREATE TEMP TABLE s(rowid INTEGER PRIMARY KEY, pageno INT);
INSERT INTO s(pageno) SELECT pageno FROM dbstat ORDER BY path;
SELECT sum(s1.pageno+1==s2.pageno)*1.0/count(*)
  FROM s AS s1, s AS s2
 WHERE s1.rowid+1=s2.rowid;
DROP TABLE s;


c2: The Checksum VFS Shim
c5: cksumvfs.html
c4:
1. Overview

The checksum VFS extension is a VFS shim that adds an 8-byte
checksum to the end of every page in an SQLite database.  The checksum
is added as each page is written and verified as each page is read.
The checksum is intended to help detect database corruption caused by
random bit-flips in the mass storage device.

The checksum VFS extension requires 
SQLite version 3.32.0 (2020-05-22) or later.  It will not
work with earlier versions of SQLite.

2. Compiling

The checksum VFS module is a loadable extension.  It is not
included in the amalgamation.  It must be added to SQLite
either at compile-time or at run-time.  The source code to
the checksum VFS module is in the
ext/misc/cksumvfs.c
source file in the
SQLite source tree.

To build the checksum VFS module into a run-time loadable
extension, use commands similar to the following:


(linux) &rarr; gcc -fPIC -shared cksumvfs.c -o cksumvfs.so
(mac) &rarr; clang -fPIC -dynamiclib cksumvfs.c -o cksumvfs.dylib
(windows) &rarr; cl cksumvfs.c -link -dll -out:cksumvfs.dll


You may want to add additional compiler options, of course,
according to the needs of your project.

To statically link this extension with your product,
compile it like any other C-language module but add the
"-DSQLITE_CKSUMVFS_STATIC" option so that this module knows that
it is being statically linked rather than dynamically linked.

3. Loading

To load this extension as a shared library, you first have to
bring up a dummy SQLite database connection to use as the argument
to the sqlite3_load_extension() API call.  Then you invoke the
sqlite3_load_extension() API and shutdown the dummy database
connection.  All subsequent database connections that are opened
will include this extension.  For example:

sqlite3 *db;
sqlite3_open(":memory:", &db);
sqlite3_load_extension(db, "./cksumvfs");
sqlite3_close(db);


If this extension is compiled with -DSQLITE_CKSUMVFS_STATIC and
statically linked against the application, initialize it using
a single API call as follows:

sqlite3_cksumvfs_init();


Cksumvfs is a VFS shim. When loaded, "cksmvfs" becomes the new
default VFS, and it uses the prior default VFS as the next VFS
down in the stack.  This is normally what you want.  However, in
complex situations where multiple VFS shims are being loaded,
it might be important to ensure that cksumvfs is loaded in the
correct order so that it sequences itself into the default VFS
Shim stack in the right order.

4. Usage

Open database connections using the sqlite3_open() or 
sqlite3_open_v2() interfaces, as normal.  Ordinary database files
(without a checksum) will operate normally.  Databases with 
checksums will return an SQLITE_IOERR_DATA error if a page is
encountered that contains an invalid checksum.

Checksumming only works on databases that have a reserve bytes
value of exactly 8.  The default value for reserve-bytes is 0.
Hence, newly created database files will omit the checksum by
default.  To create a database that includes a checksum, change
the reserve-bytes value to 8 by running code similar to this:

int n = 8;
sqlite3_file_control(db, 0, SQLITE_FCNTL_RESERVE_BYTES, &n);


If you do this immediately after creating a new database file,
before anything else has been written into the file, then that
might be all that you need to do.  Otherwise, the API call
above should be followed by:

sqlite3_exec(db, "VACUUM", 0, 0, 0);


It never hurts to run the VACUUM, even if you don't need it.
If the database is in WAL mode, you should shutdown and
reopen all database connections before continuing.

From the CLI, use the ".filectrl reserve_bytes 8" command, 
followed by "VACUUM;".

Note that SQLite allows the number of reserve-bytes to be
increased but not decreased.  So if a database file already
has a reserve-bytes value greater than 8, there is no way to
activate checksumming on that database, other than to dump
and restore the database file.  Note also that other extensions
might also make use of the reserve-bytes.  Checksumming will
be incompatible with those other extensions.

5. Verification Of Checksums

If any checksum is incorrect, the "PRAGMA quick_check" command
will find it.  To verify that checksums are actually enabled
and running, use SQL like the following:

SELECT count(*), verify_checksum(data)
  FROM sqlite_dbpage
 GROUP BY 2;


There are three possible outputs from the verify_checksum()
function: 1, 0, and NULL.  1 is returned if the checksum is
correct.  0 is returned if the checksum is incorrect.  NULL
is returned if the page is unreadable.  If checksumming is
enabled, the read will fail if the checksum is wrong, so the
usual result from verify_checksum() on a bad checksum is NULL.

If everything is OK, the query above should return a single
row where the second column is 1.  Any other result indicates
either that there is a checksum error, or checksum validation
is disabled.

6. Controlling Checksum Verification

The cksumvfs extension implements a new PRAGMA statement that can
be used to disable, re-enable, or query the status of checksum
verification:

PRAGMA checksum_verification;          -- query status
PRAGMA checksum_verification=OFF;      -- disable verification
PRAGMA checksum_verification=ON;       -- re-enable verification


The "checksum_verification" pragma will return "1" (true) or "0"
(false) if checksum verification is enabled or disabled, respectively.
"Verification" in this context means the feature that causes
SQLITE_IOERR_DATA errors if a checksum mismatch is detected while
reading.  Checksums are always kept up-to-date as long as the
reserve bytes value of the database is 8, regardless of the setting
of this pragma.  Checksum verification can be disabled (for example)
to do forensic analysis of a database that has previously reported
a checksum error.

The "checksum_verification" pragma will always respond with "0" if
the database file does not have a reserve bytes value of 8.  The
pragma will return no rows at all if the cksumvfs extension is
not loaded.


c2: Most Widely Deployed SQL Database Engine
c5: mostdeployed.html
c4:
Most Widely Deployed and Used Database Engine

SQLite is likely used more than all other database engines
combined.  Billions and billions of copies
of SQLite exist in the wild.  SQLite is found in:


Every Android device
Every iPhone and iOS device
Every Mac
Every Windows10 machine
Every Firefox, Chrome, and Safari web browser
Every instance of Skype
Every instance of iTunes
Every Dropbox client
Every TurboTax and QuickBooks
PHP and Python
Most television sets and set-top cable boxes
Most automotive multimedia systems
Countless millions of other applications


Since SQLite is used extensively in every smartphone, and there are
more than 4.0 billion (4.0e9) smartphones in active use, each holding
hundreds of
SQLite database files, it is seems likely that there are over one trillion
(1e12) SQLite databases in active use.

Most Widely Deployed Software Module of Any Type?

SQLite is probably one of the top five most deployed
software modules of any description.  
Other libraries with similar reach include:



The original zlib implementation by
Jean-loup Gailly and Mark Adler.

The original reference implementation for
libpng

Libjpeg from the Independent JPEG Group



Libc is omitted from the above list even though
it is more common than SQLite because
libc is not a single software component but rather several
competing implementations (ex: BSD vs. GNU) with similar interfaces.
There are also independent implementations of libjpeg and libpng,
though in those cases the canonical implementations are very popular.



Precise numbers are difficult to obtain and so exact rankings
are impossible.  But our best guess is that SQLite is the second
mostly widely deployed software library, after libz.
Some commentators observe that SQLite tends
to be statically linked and thus have multiple instances on
each machine, whereas libz tends to have just
a single instance per machine in the form of a shared library or DLL.
So even though the number of devices containing libz
may be greater than the number of
devices that contain SQLite, the total number of instances per device
might be higher for SQLite and so SQLite might
be the single most widely deployed and used software component.


c2: SQLite Changes From Version 3.4.2 To 3.5.0
c5: 34to35.html
c4:
Moving From SQLite 3.4.2 to 3.5.0
  SQLite version 3.5.0 (2007-09-04) introduces a new OS interface layer that
  is incompatible with all prior versions of SQLite.  In addition,
  a few existing interfaces have been generalized to work across all
  database connections within a process rather than just all
  connections within a thread.  The purpose of this article
  is to describe the changes to 3.5.0 in detail so that users
  of prior versions of SQLite can judge what, if any, effort will
  be required to upgrade to newer versions.

1.0 Overview Of Changes
  A quick enumeration of the changes in SQLite version 3.5.0
  is provided here.  Subsequent sections will describe these
  changes in more detail.


  
  The OS interface layer has been completely reworked:
  
  The undocumented sqlite3_os_switch() interface has
      been removed.
  The SQLITE_ENABLE_REDEF_IO compile-time flag no longer functions.
      I/O procedures are now always redefinable.
  Three new objects are defined for specifying I/O procedures:
      sqlite3_vfs, sqlite3_file, and sqlite3_io_methods.
  Three new interfaces are used to create alternative OS interfaces:
      sqlite3_vfs_register(), sqlite3_vfs_unregister(), and
      sqlite3_vfs_find().
  A new interface has been added to provided additional control over
      the creation of new database connections: sqlite3_open_v2().
      The legacy interfaces of sqlite3_open() and
      sqlite3_open16() continue to be fully supported.
  
  The optional shared cache and memory management features that
      were introduced in version 3.3.0 can now be used across multiple
      threads within the same process.  Formerly, these extensions only
      applied to database connections operating within a single thread.
   
   The sqlite3_enable_shared_cache() interface now applies to all
       threads within a process, not to just the one thread in which it
       was run.
   The sqlite3_soft_heap_limit() interface now applies to all threads
       within a process, not to just the one thread in which it was run.
   The sqlite3_release_memory() interface will now attempt to reduce
       the memory usages across all database connections in all threads, not
       just connections in the thread where the interface is called.
   The sqlite3_thread_cleanup() interface has become a no-op.
   
  Restrictions on the use of the same database connection by multiple
      threads have been dropped.  It is now safe for
      multiple threads to use the same database connection at the same
      time.
  There is now a compile-time option that allows an application to
      define alternative malloc()/free() implementations without having
      to modify any core SQLite code.
  There is now a compile-time option that allows an application to
      define alternative mutex implementations without having
      to modify any core SQLite code.
  


  Of these changes, only 1a and 2a through 2c are incompatibilities
  in any formal sense.
  But users who have previously made custom modifications to the
  SQLite source (for example to add a custom OS layer for embedded
  hardware) might find that these changes have a larger impact.
  On the other hand, an important goal of these changes is to make
  it much easier to customize SQLite for use on different operating
  systems. 

2.0 The OS Interface Layer
  If your system defines a custom OS interface for SQLite or if you
  were using the undocumented sqlite3_os_switch()
  interface, then you will need to make modifications in order to
  upgrade to SQLite version 3.5.0.  This may seem painful at first
  glance.  But as you look more closely, you will probably discover
  that your changes are made smaller and easier to understand and manage
  by the new SQLite interface.  It is likely that your changes will
  now also work seamlessly with the SQLite amalgamation.  You will
  no longer need to make any changes to the code SQLite source code.
  All of your changes can be effected by application code and you can
  link against a standard, unmodified version of the SQLite amalgamation.
  Furthermore, the OS interface layer, which was formerly undocumented,
  is now an officially support interface for SQLite.  So you have
  some assurance that this will be a one-time change and that your
  new backend will continue to work in future versions of SQLite.

2.1 The Virtual File System Object
  The new OS interface for SQLite is built around an object named
  sqlite3_vfs.  The "vfs" stands for "Virtual File System".
  The sqlite3_vfs object is basically a structure containing pointers
  to functions that implement the primitive disk I/O operations that
  SQLite needs to perform in order to read and write databases.  
  In this article, we will often refer to an sqlite3_vfs objects as a "VFS".


  SQLite is able to use multiple VFSes at the same time.  Each
  individual database connection is associated with just one VFS.
  But if you have multiple database connections, each connection
  can be associated with a different VFS.


  There is always a default VFS.
  The legacy interfaces sqlite3_open() and sqlite3_open16() always
  use the default VFS.
  The new interface for creating database connections,
  sqlite3_open_v2(), allows you to specify which VFS you want to
  use by name.

2.1.1 Registering New VFS Objects
  Standard builds of SQLite for Unix or Windows come with a single
  VFS named "unix" or "win32", as appropriate.  This one VFS is also 
  the default.  So if you are using the legacy open functions, everything
  will continue to operate as it has before.  The change is that an application
  now has the flexibility of adding new VFS modules to implement a
  customized OS layer.  The sqlite3_vfs_register() API can be used
  to tell SQLite about one or more application-defined VFS modules:


int sqlite3_vfs_register(sqlite3_vfs*, int makeDflt);

  Applications can call sqlite3_vfs_register() at any time, though of course
  a VFS needs to be registered before it can be used.  The first argument
  is a pointer to a customized VFS object that the application has prepared.
  The second argument is true to make the new VFS the default VFS so that
  it will be used by the legacy sqlite3_open() and sqlite3_open16() APIs.
  If the new VFS is not the default, then you will probably have to use
  the new sqlite3_open_v2() API to use it.  Note, however, that if
  a new VFS is the only VFS known to SQLite (if SQLite was compiled without
  its usual default VFS or if the precompiled default VFS was removed
  using sqlite3_vfs_unregister()) then the new VFS automatically becomes the
  default VFS regardless of the makeDflt argument to sqlite3_vfs_register().


  Standard builds include the default "unix" or "win32" VFSes.
  But if you use the -DOS_OTHER=1 compile-time option, then SQLite is
  built without a default VFS.  In that case, the application must
  register at least one VFS prior to calling sqlite3_open().
  This is the approach that embedded applications should use.
  Rather than modifying the SQLite source to insert an alternative
  OS layer as was done in prior releases of SQLite, instead compile
  an unmodified SQLite source file (preferably the amalgamation)
  with the -DOS_OTHER=1 option, then invoke sqlite3_vfs_register()
  to define the interface to the underlying filesystem prior to
  creating any database connections.

2.1.2 Additional Control Over VFS Objects
  The sqlite3_vfs_unregister() API is used to remove an existing
  VFS from the system.


int sqlite3_vfs_unregister(sqlite3_vfs*);

  The sqlite3_vfs_find() API is used to locate a particular VFS
  by name.  Its prototype is as follows:


sqlite3_vfs *sqlite3_vfs_find(const char *zVfsName);

  The argument is the symbolic name for the desired VFS.  If the
  argument is a NULL pointer, then the default VFS is returned.
  The function returns a pointer to the sqlite3_vfs object that
  implements the VFS.  Or it returns a NULL pointer if no object
  could be found that matched the search criteria.

2.1.3 Modifications Of Existing VFSes
  Once a VFS has been registered, it should never be modified.  If
  a change in behavior is required, a new VFS should be registered.
  The application could, perhaps, use sqlite3_vfs_find() to locate
  the old VFS, make a copy of the old VFS into a new sqlite3_vfs
  object, make the desired modifications to the new VFS, unregister
  the old VFS, then register the new VFS in its place.  Existing
  database connections would continue to use the old VFS even after
  it is unregistered, but new database connections would use the
  new VFS.

2.1.4 The VFS Object
  A VFS object is an instance of the following structure:


typedef struct sqlite3_vfs sqlite3_vfs;
struct sqlite3_vfs {
  int iVersion;            /* Structure version number */
  int szOsFile;            /* Size of subclassed sqlite3_file */
  int mxPathname;          /* Maximum file pathname length */
  sqlite3_vfs *pNext;      /* Next registered VFS */
  const char *zName;       /* Name of this virtual file system */
  void *pAppData;          /* Pointer to application-specific data */
  int (*xOpen)(sqlite3_vfs*, const char *zName, sqlite3_file*,
               int flags, int *pOutFlags);
  int (*xDelete)(sqlite3_vfs*, const char *zName, int syncDir);
  int (*xAccess)(sqlite3_vfs*, const char *zName, int flags);
  int (*xGetTempName)(sqlite3_vfs*, char *zOut);
  int (*xFullPathname)(sqlite3_vfs*, const char *zName, char *zOut);
  void *(*xDlOpen)(sqlite3_vfs*, const char *zFilename);
  void (*xDlError)(sqlite3_vfs*, int nByte, char *zErrMsg);
  void *(*xDlSym)(sqlite3_vfs*,void*, const char *zSymbol);
  void (*xDlClose)(sqlite3_vfs*, void*);
  int (*xRandomness)(sqlite3_vfs*, int nByte, char *zOut);
  int (*xSleep)(sqlite3_vfs*, int microseconds);
  int (*xCurrentTime)(sqlite3_vfs*, double*);
  /* New fields may be appended in figure versions.  The iVersion
  ** value will increment whenever this happens. */
};

  To create a new VFS, an application fills in an instance of this
  structure with appropriate values and then calls sqlite3_vfs_register().


  The iVersion field of sqlite3_vfs should be 1 for SQLite version 3.5.0.
  This number may increase in future versions of SQLite if we have to
  modify the VFS object in some way.  We hope that this never happens,
  but the provision is made in case it does.


  The szOsFile field is the size in bytes of the structure that defines
  an open file: the sqlite3_file object.  This object will be described
  more fully below.  The point here is that each VFS implementation can
  define its own sqlite3_file object containing whatever information
  the VFS implementation needs to store about an open file.  SQLite needs
  to know how big this object is, however, in order to preallocate enough
  space to hold it.


  The mxPathname field is the maximum length of a file pathname that
  this VFS can use.  SQLite sometimes has to preallocate buffers of
  this size, so it should be as small as reasonably possible.  Some
  filesystems permit huge pathnames, but in practice pathnames rarely
  extend beyond 100 bytes or so.  You do not have to put the longest
  pathname that the underlying filesystem can handle here.  You only
  have to put the longest pathname that you want SQLite to be able to
  handle.  A few hundred is a good value in most cases.


  The pNext field is used internally by SQLite.  Specifically, SQLite
  uses this field to form a linked list of registered VFSes.


  The zName field is the symbolic name of the VFS.  This is the name 
  that the sqlite3_vfs_find() compares against when it is looking for
  a VFS.


  The pAppData pointer is unused by the SQLite core.  The pointer is
  available to store auxiliary information that a VFS information might
  want to carry around.


  The remaining fields of the sqlite3_vfs object all store pointers
  to functions that implement primitive operations.  We call these
  "methods".  The first method, xOpen, is used to open files on
  the underlying storage media.  The result is an sqlite3_file
  object.  There are additional methods, defined by the sqlite3_file
  object itself that are used to read and write and close the file.
  The additional methods are detailed below.  The filename is in UTF-8.
  SQLite will guarantee that the zFilename string passed to
  xOpen() is a full pathname as generated by xFullPathname() and
  that the string will be valid and unchanged until xClose() is
  called.  So the sqlite3_file can store a pointer to the
   filename if it needs to remember the filename for some reason.
   The flags argument to xOpen() is a copy of the flags argument
   to sqlite3_open_v2().  If sqlite3_open() or sqlite3_open16()
   is used, then flags is SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE.
   If xOpen() opens a file read-only then it sets *pOutFlags to
   include SQLITE_OPEN_READONLY.  Other bits in *pOutFlags may be
   set.
   SQLite will also add one of the following flags to the xOpen()
   call, depending on the object being opened:
   
     SQLITE_OPEN_MAIN_DB
     SQLITE_OPEN_MAIN_JOURNAL
     SQLITE_OPEN_TEMP_DB
     SQLITE_OPEN_TEMP_JOURNAL
     SQLITE_OPEN_TRANSIENT_DB
     SQLITE_OPEN_SUBJOURNAL
     SQLITE_OPEN_SUPER_JOURNAL
   
   The file I/O implementation can use the object type flags to
   changes the way it deals with files.  For example, an application
   that does not care about crash recovery or rollback, might make
   the open of a journal file a no-op.  Writes to this journal are
   also a no-op.  Any attempt to read the journal returns SQLITE_IOERR.
   Or the implementation might recognize the a database file will
   be doing page-aligned sector reads and writes in a random order
   and set up its I/O subsystem accordingly.
   SQLite might also add one of the following flags to the xOpen
   method:
   
    SQLITE_OPEN_DELETEONCLOSE
    SQLITE_OPEN_EXCLUSIVE
   
   The SQLITE_OPEN_DELETEONCLOSE flag means the file should be
   deleted when it is closed.  This will always be set for TEMP 
   databases and journals and for subjournals.  The 
   SQLITE_OPEN_EXCLUSIVE flag means the file should be opened
   for exclusive access.  This flag is set for all files except
   for the main database file.
   The sqlite3_file structure passed as the third argument to
   xOpen is allocated by the caller.  xOpen just fills it in.  The
   caller allocates a minimum of szOsFile bytes for the sqlite3_file
   structure.


  The differences between an SQLITE_OPEN_TEMP_DB database and an
  SQLITE_OPEN_TRANSIENT_DB database is this:  The SQLITE_OPEN_TEMP_DB
  is used for explicitly declared and named TEMP tables (using the
  CREATE TEMP TABLE syntax) or for named tables in a temporary database
  that is created by opening a database with a filename that is an empty
  string.  An SQLITE_OPEN_TRANSIENT_DB holds a database table that
  SQLite creates automatically in order to evaluate a subquery or
  ORDER BY or GROUP BY clause.  Both TEMP_DB and TRANSIENT_DB databases
  are private and are deleted automatically.  TEMP_DB databases last
  for the duration of the database connection.  TRANSIENT_DB databases
  last only for the duration of a single SQL statement.


  The xDelete method is used to delete a file.  The name of the file is
  given in the second parameter.  The filename will be in UTF-8.
  The VFS must convert the filename into whatever character representation
  the underlying operating system expects.  If the syncDir parameter is
  true, then the xDelete method should not return until the change
  to the directory contents for the directory containing the
  deleted file have been synced to disk in order to ensure that the
  file does not "reappear" if a power failure occurs soon after.


  The xAccess method is used to check for access permissions on a file.
  The filename will be UTF-8 encoded.  The flags argument will be
  SQLITE_ACCESS_EXISTS to check for the existence of the file,
  SQLITE_ACCESS_READWRITE to check to see if the file is both readable
  and writable, or SQLITE_ACCESS_READ to check to see if the file is
  at least readable.  The "file" named by the second parameter might
  be a directory or folder name.


  The xGetTempName method computes the name of a temporary file that
  SQLite can use.  The name should be written into the buffer given
  by the second parameter.  SQLite will size that buffer to hold
  at least mxPathname bytes.  The generated filename should be in UTF-8.
  To avoid security problems, the generated temporary filename should
  contain enough randomness to prevent an attacker from guessing the
  temporary filename in advance.


  The xFullPathname method is used to convert a relative pathname
  into a full pathname.  The resulting full pathname is written into
  the buffer provided by the third parameter.  SQLite will size the
  output buffer to at least mxPathname bytes.  Both the input and
  output names should be in UTF-8.


  The xDlOpen, xDlError, xDlSym, and xDlClose methods are all used for
  accessing shared libraries at run-time.  These methods may be omitted
  (and their pointers set to zero) if the library is compiled with
  SQLITE_OMIT_LOAD_EXTENSION or if the sqlite3_enable_load_extension()
  interface is never used to enable dynamic extension loading.  The
  xDlOpen method opens a shared library or DLL and returns a pointer to
  a handle.  NULL is returned if the open fails.  If the open fails,
  the xDlError method can be used to obtain a text error message.
  The message is written into the zErrMsg buffer of the third parameter
  which is at least nByte bytes in length.  The xDlSym returns a pointer
  to a symbol in the shared library.  The name of the symbol is given
  by the second parameter.  UTF-8 encoding is assumed.  If the symbol
  is not found a NULL pointer is returned.  The xDlClose routine closes
  the shared library.


  The xRandomness method is used exactly once to initialize the 
  pseudo-random number generator (PRNG) inside of SQLite.  Only
  the xRandomness method on the default VFS is used.  The xRandomness
  methods on other VFSes are never accessed by SQLite.
  The xRandomness routine requests that nByte bytes of randomness
  be written into zOut.  The routine returns the actual number of
  bytes of randomness obtained.  The quality of the randomness so obtained
  will determine the quality of the randomness generated by built-in 
  SQLite functions such as random() and randomblob().  SQLite also
  uses its PRNG to generate temporary file names.  On some platforms
  (ex: Windows) SQLite assumes that temporary file names are unique
  without actually testing for collisions, so it is important to have
  good-quality randomness even if the random() and randomblob() 
  functions are never used.


  The xSleep method is used to suspend the calling thread for at
  least the number of microseconds given.  This method is used to
  implement the sqlite3_sleep() and sqlite3_busy_timeout() APIs.
  In the case of sqlite3_sleep() the xSleep method of the default
  VFS is always used.  If the underlying system does not have a
  microsecond resolution sleep capability, then the sleep time should
  be rounded up.  xSleep returns this rounded-up value.


  The xCurrentTime method finds the current time and date and writes
  the result as a double-precision floating point value into pointer
  provided by the second parameter.  The time and date is in
  coordinated universal time (UTC) and is a fractional Julian day number.

2.1.5 The Open File Object
  The result of opening a file is an instance of an sqlite3_file object.
  The sqlite3_file object is an abstract base class defined as follows:


typedef struct sqlite3_file sqlite3_file;
struct sqlite3_file {
  const struct sqlite3_io_methods *pMethods;
};

  Each VFS implementation will subclass the sqlite3_file by adding
  additional fields at the end to hold whatever information the VFS
  needs to know about an open file.  It does not matter what information
  is stored as long as the total size of the structure does not exceed
  the szOsFile value recorded in the sqlite3_vfs object.


  The sqlite3_io_methods object is a structure that contains pointers
  to methods for reading, writing, and otherwise dealing with files.
  This object is defined as follows:


typedef struct sqlite3_io_methods sqlite3_io_methods;
struct sqlite3_io_methods {
  int iVersion;
  int (*xClose)(sqlite3_file*);
  int (*xRead)(sqlite3_file*, void*, int iAmt, sqlite3_int64 iOfst);
  int (*xWrite)(sqlite3_file*, const void*, int iAmt, sqlite3_int64 iOfst);
  int (*xTruncate)(sqlite3_file*, sqlite3_int64 size);
  int (*xSync)(sqlite3_file*, int flags);
  int (*xFileSize)(sqlite3_file*, sqlite3_int64 *pSize);
  int (*xLock)(sqlite3_file*, int);
  int (*xUnlock)(sqlite3_file*, int);
  int (*xCheckReservedLock)(sqlite3_file*);
  int (*xFileControl)(sqlite3_file*, int op, void *pArg);
  int (*xSectorSize)(sqlite3_file*);
  int (*xDeviceCharacteristics)(sqlite3_file*);
  /* Additional methods may be added in future releases */
};

  The iVersion field of sqlite3_io_methods is provided as insurance
  against future enhancements.  The iVersion value should always be
  1 for SQLite version 3.5.


  The xClose method closes the file.  The space for the sqlite3_file
  structure is deallocated by the caller.  But if the sqlite3_file
  contains pointers to other allocated memory or resources, those
  allocations should be released by the xClose method.


  The xRead method reads iAmt bytes from the file beginning at a byte
  offset to iOfst.  The data read is stored in the pointer of the
  second parameter.  xRead returns the SQLITE_OK on success,
  SQLITE_IOERR_SHORT_READ if it was not able to read the full number
  of bytes because it reached end-of-file, or SQLITE_IOERR_READ for
  any other error.


  The xWrite method writes iAmt bytes of data from the second parameter
  into the file beginning at an offset of iOfst bytes.  If the size of
  the file is less than iOfst bytes prior to the write, then xWrite should
  ensure that the file is extended with zeros up to iOfst bytes prior
  to beginning its write.  xWrite continues to extends the file as
  necessary so that the size of the file is at least iAmt+iOfst bytes 
  at the conclusion of the xWrite call.  The xWrite method returns
  SQLITE_OK on success.  If the write cannot complete because the
  underlying storage medium is full, then SQLITE_FULL is returned.
  SQLITE_IOERR_WRITE should be returned for any other error.


  The xTruncate method truncates a file to be nByte bytes in length.
  If the file is already nByte bytes or less in length then this
  method is a no-op.  The xTruncate method returns SQLITE_OK on
  success and SQLITE_IOERR_TRUNCATE if anything goes wrong.


  The xSync method is used to force previously written data out of
  operating system cache and into non-volatile memory.  The second
  parameter is usually SQLITE_SYNC_NORMAL.  If the second parameter
  is SQLITE_SYNC_FULL then the xSync method should make sure that
  data has also been flushed through the disk controllers cache.
  The SQLITE_SYNC_FULL parameter is the equivalent of the F_FULLSYNC
  ioctl() on Mac OS X. The xSync method returns
  SQLITE_OK on success and SQLITE_IOERR_FSYNC if anything goes wrong.


  The xFileSize() method determines the current size of the file
  in bytes and writes that value into *pSize.  It returns SQLITE_OK
  on success and SQLITE_IOERR_FSTAT if something goes wrong.


  The xLock and xUnlock methods are used to set and clear file locks.
  SQLite supports five levels of file locks, in order:
  
   SQLITE_LOCK_NONE
   SQLITE_LOCK_SHARED
   SQLITE_LOCK_RESERVED
   SQLITE_LOCK_PENDING
   SQLITE_LOCK_EXCLUSIVE
  
  The underlying implementation can support some subset of these locking
  levels as long as it meets the other requirements of this paragraph.
  The locking level is specified as the second argument to both xLock
  and xUnlock.  The xLock method increases the locking level to the
  specified locking level or higher.  The xUnlock method decreases the
  locking level to no lower than the level specified.  
  SQLITE_LOCK_NONE means that the file is unlocked.  SQLITE_LOCK_SHARED
  gives permission to read the file.  Multiple database connections can
  hold SQLITE_LOCK_SHARED at the same time.
  SQLITE_LOCK_RESERVED is like SQLITE_LOCK_SHARED in that it is permission
  to read the file.  But only a single connection can hold a reserved lock
  at any point in time.  The SQLITE_LOCK_PENDING is also permission to
  read the file.  Other connections can continue to read the file as well,
  but no other connection is allowed to escalate a lock from none to shared.
  SQLITE_LOCK_EXCLUSIVE is permission to write on the file.  Only a single
  connection can hold an exclusive lock and no other connection can hold
  any lock (other than "none") while one connection holds an exclusive
  lock.  The xLock returns SQLITE_OK on success, SQLITE_BUSY if it
  is unable to obtain the lock, or SQLITE_IOERR_RDLOCK if something else
  goes wrong.  The xUnlock method returns SQLITE_OK on success and
  SQLITE_IOERR_UNLOCK for problems.


  The xCheckReservedLock() method checks to see if another connection or
  another process is currently holding a reserved, pending, or exclusive
  lock on the file.  It returns true or false.


  The xFileControl() method is a generic interface that allows custom
  VFS implementations to directly control an open file using the
  (new and experimental)
  sqlite3_file_control() interface.  The second "op" argument
  is an integer opcode.  The third
  argument is a generic pointer which is intended to be a pointer
  to a structure that may contain arguments or space in which to
  write return values.  Potential uses for xFileControl() might be
  functions to enable blocking locks with timeouts, to change the
  locking strategy (for example to use dot-file locks), to inquire
  about the status of a lock, or to break stale locks.  The SQLite
  core reserves opcodes less than 100 for its own use. 
  A list of opcodes less than 100 is available.
  Applications that define a custom xFileControl method should use opcodes 
  greater than 100 to avoid conflicts.


  The xSectorSize returns the "sector size" of the underlying
  non-volatile media.  A "sector" is defined as the smallest unit of
  storage that can be written without disturbing adjacent storage.
  On a disk drive the "sector size" has until recently been 512 bytes,
  though there is a push to increase this value to 4KiB.  SQLite needs
  to know the sector size so that it can write a full sector at a
  time, and thus avoid corrupting adjacent storage space if a power
  loss occurs in the middle of a write.


  The xDeviceCharacteristics method returns an integer bit vector that
  defines any special properties that the underlying storage medium might
  have that SQLite can use to increase performance.  The allowed return
  is the bit-wise OR of the following values:
  
   SQLITE_IOCAP_ATOMIC
   SQLITE_IOCAP_ATOMIC512
   SQLITE_IOCAP_ATOMIC1K
   SQLITE_IOCAP_ATOMIC2K
   SQLITE_IOCAP_ATOMIC4K
   SQLITE_IOCAP_ATOMIC8K
   SQLITE_IOCAP_ATOMIC16K
   SQLITE_IOCAP_ATOMIC32K
   SQLITE_IOCAP_ATOMIC64K
   SQLITE_IOCAP_SAFE_APPEND
   SQLITE_IOCAP_SEQUENTIAL
  
  The SQLITE_IOCAP_ATOMIC bit means that all writes to this device are
  atomic in the sense that either the entire write occurs or none of it
  occurs.  The other 
  SQLITE_IOCAP_ATOMICnnn values indicate that
  writes of aligned blocks of the indicated size are atomic.
  SQLITE_IOCAP_SAFE_APPEND means that when extending a file with new
  data, the new data is written first and then the file size is updated.
  So if a power failure occurs, there is no chance that the file might have
  been extended with randomness.  The SQLITE_IOCAP_SEQUENTIAL bit means
  that all writes occur in the order that they are issued and are not
  reordered by the underlying file system.

2.1.6 Checklist For Constructing A New VFS
  The preceding paragraphs contain a lot of information.
  To ease the task of constructing
  a new VFS for SQLite we offer the following implementation checklist:


  
   Define an appropriate subclass of the sqlite3_file object.
   Implement the methods required by the sqlite3_io_methods object.
   Create a static and 
       constant sqlite3_io_methods object containing pointers
       to the methods from the previous step.
   Implement the xOpen method that opens a file and populates an
       sqlite3_file object, including setting pMethods to
       point to the sqlite3_io_methods object from the previous step.
   Implement the other methods required by sqlite3_vfs.
   Define a static (but not constant) sqlite3_vfs structure that
       contains pointers to the xOpen method and the other methods and
       which contains the appropriate values for iVersion, szOsFile,
       mxPathname, zName, and pAppData.
   Implement a procedure that calls sqlite3_vfs_register() and
       passes it a pointer to the sqlite3_vfs structure from the previous
       step.  This procedure is probably the only exported symbol in the
       source file that implements your VFS.
  


  Within your application, call the procedure implemented in the last
  step above as part of your initialization process before any
  database connections are opened.  

3.0 The Memory Allocation Subsystem
  Beginning with version 3.5, SQLite obtains all of the heap memory it
  needs using the routines sqlite3_malloc(), sqlite3_free(), and
  sqlite3_realloc().  These routines have existed in prior versions
  of SQLite, but SQLite has previously bypassed these routines and used
  its own memory allocator.  This all changes in version 3.5.0.


  The SQLite source tree actually contains multiple versions of the
  memory allocator.  The default high-speed version found in the
  "mem1.c" source file is used for most builds.  But if the SQLITE_MEMDEBUG
  flag is enabled, a separate memory allocator the "mem2.c" source file
  is used instead.  The mem2.c allocator implements lots of hooks to
  do error checking and to simulate memory allocation failures for testing
  purposes.  Both of these allocators use the malloc()/free() implementation
  in the standard C library.


  Applications are not required to use either of these standard memory
  allocators.  If SQLite is compiled with SQLITE_OMIT_MEMORY_ALLOCATION
  then no implementation for the sqlite3_malloc(), sqlite3_realloc(),
  and sqlite3_free() functions is provided.  Instead, the application
  that links against SQLite must provide its own implementation of these
  functions.  The application provided memory allocator is not required
  to use the malloc()/free() implementation in the standard C library.
  An embedded application might provide an alternative memory allocator
  that uses memory for a fixed memory pool set aside for the exclusive
  use of SQLite, for example.


  Applications that implement their own memory allocator must provide
  implementation for the usual three allocation functions 
  sqlite3_malloc(), sqlite3_realloc(), and sqlite3_free().
  And they must also implement a fourth function:


int sqlite3_memory_alarm(
  void(*xCallback)(void *pArg, sqlite3_int64 used, int N),
  void *pArg,
  sqlite3_int64 iThreshold
);

 The sqlite3_memory_alarm routine is used to register
 a callback on memory allocation events.
 This routine registers or clears a callback that fires when
 the amount of memory allocated exceeds iThreshold.  Only
 a single callback can be registered at a time.  Each call
 to sqlite3_memory_alarm() overwrites the previous callback.
 The callback is disabled by setting xCallback to a NULL
 pointer.


 The parameters to the callback are the pArg value, the 
 amount of memory currently in use, and the size of the
 allocation that provoked the callback.  The callback will
 presumably invoke sqlite3_free() to free up memory space.
 The callback may invoke sqlite3_malloc() or sqlite3_realloc()
 but if it does, no additional callbacks will be invoked by
 the recursive calls.


 The sqlite3_soft_heap_limit() interface works by registering
 a memory alarm at the soft heap limit and invoking 
 sqlite3_release_memory() in the alarm callback.  Application
 programs should not attempt to use the sqlite3_memory_alarm()
 interface because doing so will interfere with the
 sqlite3_soft_heap_limit() module.  This interface is exposed
 only so that applications can provide their own
 alternative implementation when the SQLite core is
 compiled with SQLITE_OMIT_MEMORY_ALLOCATION.


  The built-in memory allocators in SQLite also provide the following
  additional interfaces:


sqlite3_int64 sqlite3_memory_used(void);
sqlite3_int64 sqlite3_memory_highwater(int resetFlag);

  These interfaces can be used by an application to monitor how
  much memory SQLite is using.  The sqlite3_memory_used() routine
  returns the number of bytes of memory currently in use and the
  sqlite3_memory_highwater() returns the maximum instantaneous
  memory usage.  Neither routine includes the overhead associated
  with the memory allocator.  These routines are provided for use
  by the application.  SQLite never invokes them itself.  So if
  the application is providing its own memory allocation subsystem,
  it can omit these interfaces if desired.

4.0 The Mutex Subsystem
  SQLite has always been threadsafe in the sense that it is safe to
  use different SQLite database connections in different threads at the
  same time.  The constraint was that the same database connection
  could not be used in two separate threads at once.  SQLite version 3.5.0
  relaxes this constraint. 


  In order to allow multiple threads to use the same database connection
  at the same time, SQLite must make extensive use of mutexes.  And for
  this reason a new mutex subsystem as been added.  The mutex subsystem
  as the following interface:


sqlite3_mutex *sqlite3_mutex_alloc(int);
void sqlite3_mutex_free(sqlite3_mutex*);
void sqlite3_mutex_enter(sqlite3_mutex*);
int sqlite3_mutex_try(sqlite3_mutex*);
void sqlite3_mutex_leave(sqlite3_mutex*);

  Though these routines exist for the use of the SQLite core, 
  application code is free to use these routines as well, if desired.
  A mutex is an sqlite3_mutex object.  The sqlite3_mutex_alloc()
  routine allocates a new mutex object and returns a pointer to it.
  The argument to sqlite3_mutex_alloc() should be 
  SQLITE_MUTEX_FAST or SQLITE_MUTEX_RECURSIVE for non-recursive
  and recursive mutexes, respectively.  If the underlying system does
  not provide non-recursive mutexes, then a recursive mutex can be
  substituted in that case.  The argument to sqlite3_mutex_alloc()
  can also be a constant designating one of several static mutexes:
  
    SQLITE_MUTEX_STATIC_MAIN
    SQLITE_MUTEX_STATIC_MEM
    SQLITE_MUTEX_STATIC_MEM2
    SQLITE_MUTEX_STATIC_PRNG
    SQLITE_MUTEX_STATIC_LRU
  
  These static mutexes are reserved for use internally by SQLite
  and should not be used by the application.  The static mutexes
  are all non-recursive.


  The sqlite3_mutex_free() routine should be used to deallocate
  a non-static mutex.  If a static mutex is passed to this routine
  then the behavior is undefined.


  The sqlite3_mutex_enter() attempts to enter the mutex and blocks
  if another threads is already there.  sqlite3_mutex_try() attempts
  to enter and returns SQLITE_OK on success or SQLITE_BUSY if another
  thread is already there.  sqlite3_mutex_leave() exits a mutex.
  The mutex is held until the number of exits matches the number of
  entrances.  If sqlite3_mutex_leave() is called on a mutex that 
  the thread is not currently holding, then the behavior is undefined.
  If any routine is called for a deallocated mutex, then the behavior
  is undefined.


  The SQLite source code provides multiple implementations of these
  APIs, suitable for varying environments.  If SQLite is compiled with
  the SQLITE_THREADSAFE=0 flag then a no-op mutex implementation that 
  is fast but does no real mutual exclusion is provided.  That 
  implementation is suitable for use in single-threaded applications
  or applications that only use SQLite in a single thread.  Other
  real mutex implementations are provided based on the underlying
  operating system.


  Embedded applications may wish to provide their own mutex implementation.
  If SQLite is compiled with the -DSQLITE_MUTEX_APPDEF=1 compile-time flag
  then the SQLite core provides no mutex subsystem and a mutex subsystem
  that matches the interface described above must be provided by the
  application that links against SQLite.

5.0 Other Interface Changes
  Version 3.5.0 of SQLite changes the behavior of a few APIs in ways
  that are technically incompatible.  However, these APIs are seldom
  used and even when they are used it is difficult to imagine a
  scenario where the change might break something.  The changes
  actually makes these interface much more useful and powerful.


  Prior to version 3.5.0, the sqlite3_enable_shared_cache() API
  would enable and disable the shared cache feature for all connections
  within a single thread - the same thread from which the 
  sqlite3_enable_shared_cache() routine was called.  Database connections
  that used the shared cache were restricted to running in the same
  thread in which they were opened.  Beginning with version 3.5.0,
  the sqlite3_enable_shared_cache() applies to all database connections
  in all threads within the process.  Now database connections running
  in separate threads can share a cache.  And database connections that
  use shared cache can migrate from one thread to another.


  Prior to version 3.5.0 the sqlite3_soft_heap_limit() set an upper
  bound on heap memory usage for all database connections within a
  single thread.  Each thread could have its own heap limit.  Beginning
  in version 3.5.0, there is a single heap limit for the entire process.
  This seems more restrictive (one limit as opposed to many) but in
  practice it is what most users want.


  Prior to version 3.5.0 the sqlite3_release_memory() function would
  try to reclaim memory from all database connections in the same thread
  as the sqlite3_release_memory() call.  Beginning with version 3.5.0,
  the sqlite3_release_memory() function will attempt to reclaim memory
  from all database connections in all threads.

6.0 Summary
  The transition from SQLite version 3.4.2 to 3.5.0 is a major change.
  Every source code file in the SQLite core had to be modified, some
  extensively.  And the change introduced some minor incompatibilities
  in the C interface.  But we feel that the benefits of the transition
  from 3.4.2 to 3.5.0 far outweigh the pain of porting.  The new
  VFS layer is now well-defined and stable and should simplify future
  customizations.  The VFS layer, and the separable memory allocator
  and mutex subsystems allow a standard SQLite source code amalgamation
  to be used in an embedded project without change, greatly simplifying
  configuration management.  And the resulting system is much more
  tolerant of highly threaded designs.


c2: Window Functions
c5: windowfunctions.html
c4:
1. Introduction to Window Functions

A window function is an SQL function where the input
values are taken from
a "window" of one or more rows in the results set of a SELECT statement.

Window functions are distinguished from other SQL functions by the
presence of an OVER clause. If a function has an OVER clause,
then it is a window function. If it lacks an OVER clause, then it is an
ordinary aggregate or scalar function. Window functions might also
have a FILTER clause in between the function and the OVER clause.

The syntax for a window function is like this:


window-function-invocation:
hide
 
 

expr:
show
 
 

literal-value:
show
 
 


over-clause:
show
 
 

frame-spec:
hide
 
 


ordering-term:
show
 
 



raise-function:
show
 
 


select-stmt:
show
 
 

common-table-expression:
show
 
 


compound-operator:
show
 
 


join-clause:
show
 
 

join-constraint:
show
 
 


join-operator:
show
 
 



ordering-term:
show
 
 


result-column:
show
 
 


table-or-subquery:
show
 
 



type-name:
show
 
 

signed-number:
show
 
 




filter-clause:
hide
 
 


window-defn:
hide
 
 

frame-spec:
hide
 
 


ordering-term:
show
 
 







Unlike ordinary functions, window functions
cannot use the DISTINCT keyword.
Also, Window functions may only appear in the result set and in the
ORDER BY clause of a SELECT statement.

Window functions come in two varieties: 
aggregate window functions and
built-in window functions.  Every aggregate window function
can also work as a ordinary aggregate function, simply by omitting
the OVER and FILTER clauses.  Furthermore, all of the built-in
aggregate functions of SQLite can be used as an
aggregate window function by adding an appropriate OVER clause.
Applications can register new aggregate window functions using
the sqlite3_create_window_function() interface.
The built-in window functions, however, require special-case
handling in the query planner and hence new window functions
that exhibit the exceptional properties found in the built-in
window functions cannot be added by the application.

Here is an example using the built-in row_number()
window function:

CREATE TABLE t0(x INTEGER PRIMARY KEY, y TEXT);
INSERT INTO t0 VALUES (1, 'aaa'), (2, 'ccc'), (3, 'bbb');

-- The following SELECT statement returns:
-- 
--   x | y | row_number
-----------------------
--   1 | aaa | 1         
--   2 | ccc | 3         
--   3 | bbb | 2         
-- 
SELECT x, y, row_number() OVER (ORDER BY y) AS row_number FROM t0 ORDER BY x;



The row_number() window function
assigns consecutive integers to each
row in order of the "ORDER BY" clause within the
window-defn (in this case "ORDER BY y"). Note that 
this does not affect the order in which results are returned from
the overall query.  The order of the final output is
still governed by the ORDER BY clause attached to the SELECT
statement (in this case "ORDER BY x").

Named window-defn clauses may also be added to a SELECT 
statement using a WINDOW clause and then referred to by name within window
function invocations. For example, the following SELECT statement contains
two named window-defs clauses, "win1" and "win2":

SELECT x, y, row_number() OVER win1, rank() OVER win2 
FROM t0 
WINDOW win1 AS (ORDER BY y RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW),
       win2 AS (PARTITION BY y ORDER BY x)
ORDER BY x;


The WINDOW clause, when one is present, comes after any HAVING clause and
before any ORDER BY.



2. Aggregate Window Functions

 The examples in this section all assume that the database is populated as
follows:

CREATE TABLE t1(a INTEGER PRIMARY KEY, b, c);
INSERT INTO t1 VALUES   (1, 'A', 'one'  ),
                        (2, 'B', 'two'  ),
                        (3, 'C', 'three'),
                        (4, 'D', 'one'  ),
                        (5, 'E', 'two'  ),
                        (6, 'F', 'three'),
                        (7, 'G', 'one'  );


 An aggregate window function is similar to an 
ordinary aggregate function, except
adding it to a query does not change the number of rows returned. Instead,
for each row the result of the aggregate window function is as if the
corresponding aggregate were run over all rows in the "window frame"
specified by the OVER clause.


-- The following SELECT statement returns:
-- 
--   a | b | group_concat
-------------------------
--   1 | A | A.B         
--   2 | B | A.B.C       
--   3 | C | B.C.D       
--   4 | D | C.D.E       
--   5 | E | D.E.F       
--   6 | F | E.F.G       
--   7 | G | F.G         
-- 
SELECT a, b, group_concat(b, '.') OVER (
  ORDER BY a ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING
) AS group_concat FROM t1;


 In the example above, the window frame consists of all rows between the
previous row ("1 PRECEDING") and the following row ("1 FOLLOWING"), inclusive,
where rows are sorted according to the ORDER BY clause in the
window-defn (in this case "ORDER BY a"). 
For example, the frame for the row with (a=3) consists of rows (2, 'B', 'two'),
(3, 'C', 'three') and (4, 'D', 'one'). The result of group_concat(b, '.') 
for that row is therefore 'B.C.D'.

 All of SQLite's aggregate functions may
be used as aggregate window functions. It is also possible to 
create user-defined aggregate window functions.




2.1. The PARTITION BY Clause

 For the purpose of computing window functions, the result set
of a query is divided into one or more "partitions".  A partition consists
of all rows that have the same value for all terms of the PARTITION BY clause
in the window-defn.  If there is no PARTITION BY clause,
then the entire result set of the query is a single partition.
Window-function processing is performed separately for each partition.

 For example:

-- The following SELECT statement returns:
-- 
--   c     | a | b | group_concat
---------------------------------
--   one   | 1 | A | A.D.G       
--   one   | 4 | D | D.G         
--   one   | 7 | G | G           
--   three | 3 | C | C.F         
--   three | 6 | F | F           
--   two   | 2 | B | B.E         
--   two   | 5 | E | E           
-- 
SELECT c, a, b, group_concat(b, '.') OVER (
  PARTITION BY c ORDER BY a RANGE BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
) AS group_concat
FROM t1 ORDER BY c, a;


 In the query above, the "PARTITION BY c" clause breaks the
result set up into three partitions. The first partition has
three rows with c=='one'.  The second partition has two rows with
c=='three' and the third partition has two rows with c=='two'.

 In the example above, all the rows for each partition are
grouped together in the final output.  This is because the PARTITION BY
clause is a prefix of the ORDER BY clause on the overall query.  
But that does not have
to be the case.  A partition can be composed of rows scattered
about haphazardly within the result set.  For example:

-- The following SELECT statement returns:
-- 
--   c     | a | b | group_concat
---------------------------------
--   one   | 1 | A | A.D.G       
--   two   | 2 | B | B.E         
--   three | 3 | C | C.F         
--   one   | 4 | D | D.G         
--   two   | 5 | E | E           
--   three | 6 | F | F           
--   one   | 7 | G | G           
-- 
SELECT c, a, b, group_concat(b, '.') OVER (
  PARTITION BY c ORDER BY a RANGE BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
) AS group_concat
FROM t1 ORDER BY a;





2.2. Frame Specifications

 The frame-spec determines which output rows are
read by an aggregate window function.  The
frame-spec consists of four parts:


   A frame type - either ROWS, RANGE or GROUPS,
   A starting frame boundary,
   An ending frame boundary,
   An EXCLUDE clause.


 Here are the syntax details:

frame-spec:
hide
 
 

expr:
show
 
 

filter-clause:
show
 
 


literal-value:
show
 
 


over-clause:
show
 
 

ordering-term:
show
 
 



raise-function:
show
 
 


select-stmt:
show
 
 

common-table-expression:
show
 
 


compound-operator:
show
 
 


join-clause:
show
 
 

join-constraint:
show
 
 


join-operator:
show
 
 



ordering-term:
show
 
 


result-column:
show
 
 


table-or-subquery:
show
 
 


window-defn:
show
 
 



type-name:
show
 
 

signed-number:
show
 
 








The ending frame boundary can be omitted (if the
BETWEEN and AND keywords that surround the starting frame boundary
are also omitted),
in which case the ending frame boundary defaults to CURRENT ROW.

 If the frame type is RANGE or GROUPS, then rows with the same values for
all ORDER BY expressions are considered "peers". Or, if there are no ORDER BY
terms, all rows are peers. Peers are always within the same frame.

The default frame-spec is:

RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW EXCLUDE NO OTHERS


The default means that aggregate window functions read all
rows from the beginning of the partition up to and including the
current row and its peers. This implies that rows that have the same values for
all ORDER BY expressions will also have the same value for the result of the
window function (as the window frame is the same). For example:

-- The following SELECT statement returns:
-- 
--   a | b | c | group_concat
-----------------------------
--   1 | A | one   | A.D.G       
--   2 | B | two   | A.D.G.C.F.B.E
--   3 | C | three | A.D.G.C.F   
--   4 | D | one   | A.D.G       
--   5 | E | two   | A.D.G.C.F.B.E
--   6 | F | three | A.D.G.C.F   
--   7 | G | one   | A.D.G       
-- 
SELECT a, b, c, 
       group_concat(b, '.') OVER (ORDER BY c) AS group_concat 
FROM t1 ORDER BY a;




2.2.1. Frame Type


There are three frame types:  ROWS, GROUPS, and RANGE.
The frame type determines how the starting and ending boundaries
of the frame are measured.


ROWS: 
The ROWS frame type means that the starting and ending boundaries
for the frame are determined by counting individual rows relative
to the current row.



GROUPS:
The GROUPS frame type means that the starting and ending boundaries
are determine by counting "groups" relative to the current group.
A "group" is a set of rows that all have equivalent values for all
all terms of the window ORDER BY clause.  ("Equivalent" means that
the IS operator is true when comparing the two values.)
In other words, a group consists of all peers of a row.



RANGE:
The RANGE frame type requires that the ORDER BY clause of the
window have exactly one term.  Call that term "X".  With the
RANGE frame type, the elements of the frame are determined by
computing the value of expression X for all rows in the partition
and framing those rows for which the value of X is within a certain
range of the value of X for the current row.  See the description
in the "&lt;expr&gt; PRECEDING" boundary
specification below for details.


The ROWS and GROUPS frame types are similar in that they
both determine the extent of a frame by counting relative to
the current row.  The difference is that ROWS counts individual
rows and GROUPS counts peer groups.
The RANGE frame type is different.
The RANGE frame type determines the extent of a frame by
looking for expression values that are within some band of
values relative to the current row.



2.2.2. Frame Boundaries

 There are five ways to describe starting and ending frame boundaries:


 UNBOUNDED PRECEDING
     The frame boundary is the first
     row in the partition.

 &lt;expr&gt; PRECEDING
        &lt;expr&gt; must be a non-negative constant numeric expression.
        The boundary is a row that is &lt;expr&gt; "units" prior to
        the current row.  The meaning of "units" here depends on the
        frame type:
        
         ROWS &rarr;
        The frame boundary is the row that is &lt;expr&gt;
        rows before the current row, or the first row of the
        partition if there are fewer than &lt;expr&gt; rows
        before the current row.  &lt;expr&gt; must be an integer.
         GROUPS &rarr;
        A "group" is a set of peer rows - rows that all have
        the same values for every term in the ORDER BY clause.
        The frame boundary is the group that is &lt;expr&gt;
        groups before the group containing the current row, or the
        first group of the partition if there are fewer
        than &lt;expr&gt; groups before the current row.
        For the starting boundary of a frame, the first
        row of the group is used and for the ending boundary
        of a frame, the last row of the group is used.
        &lt;expr&gt; must be an integer.
        

         RANGE &rarr;
        For this form, the ORDER BY clause of the
        window-defn must have a single
        term.  Call that ORDER BY term "X".  Let
        Xi be the value of the X
        expression for the i-th row in the partition and let
        Xc be the value of X for the
        current row.  Informally, a RANGE bound is the first row
        for which Xi is within
        the &lt;expr&gt; of Xc.
        More precisely:
        
         If either Xi or
             Xc are non-numeric, then
             the boundary is the first row for which the expression
             "Xi IS Xc"
             is true.
         Else if the ORDER BY is ASC then the boundary
             is the first row for which
             Xi&gt;=Xc-&lt;expr&gt;.
         Else if the ORDER BY is DESC then the boundary
        is the first row for which
        Xi&lt;=Xc+&lt;expr&gt;.
        
        For this form, the &lt;expr&gt; does not have to be an
        integer. It can evaluate to a real number as long as
        it is constant and non-negative.
        
    The boundary description "0 PRECEDING" always means the same
    thing as "CURRENT ROW".

CURRENT ROW
        The current row. For RANGE and GROUPS frame types,
        peers of the current row are also included in the frame,
        unless specifically excluded by the EXCLUDE clause.
        This is true regardless of whether CURRENT ROW is used
        as the starting or ending frame boundary.

&lt;expr&gt; FOLLOWING
        This is the same as "&lt;expr&gt; PRECEDING" except that
        the boundary is &lt;expr&gt; units after the current
        rather than before the current row.

 UNBOUNDED FOLLOWING
   The frame boundary is the last
   row in the partition.



 The ending frame boundary may not take a form that appears higher in
the above list than the starting frame boundary.

 In the following example, the window frame for each row consists of all
rows from the current row to the end of the set, where rows are sorted
according to "ORDER BY a".

-- The following SELECT statement returns:
-- 
--   c     | a | b | group_concat
---------------------------------
--   one   | 1 | A | A.D.G.C.F.B.E
--   one   | 4 | D | D.G.C.F.B.E 
--   one   | 7 | G | G.C.F.B.E   
--   three | 3 | C | C.F.B.E     
--   three | 6 | F | F.B.E       
--   two   | 2 | B | B.E         
--   two   | 5 | E | E           
-- 
SELECT c, a, b, group_concat(b, '.') OVER (
  ORDER BY c, a ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
) AS group_concat
FROM t1 ORDER BY c, a;



 
2.2.3. The EXCLUDE Clause

 The optional EXCLUDE clause may take any of the following four forms:


    EXCLUDE NO OTHERS: This is the default. In this case no
  rows are excluded from the window frame as defined by its starting and ending
  frame boundaries.

    EXCLUDE CURRENT ROW: In this case the current row is 
  excluded from the window frame.  Peers of the current row remain in
  the frame for the GROUPS and RANGE frame types.

    EXCLUDE GROUP: In this case the current row and all other
  rows that are peers of the current row are excluded from the frame. When
  processing an EXCLUDE clause, all rows with the same ORDER BY values, or all
  rows in the partition if there is no ORDER BY clause, are considered peers,
  even if the frame type is ROWS.

    EXCLUDE TIES: In this case the current row is part of the
  frame, but peers of the current row are excluded.


 The following example demonstrates the effect of the various
forms of the EXCLUDE clause:

-- The following SELECT statement returns:
-- 
--   c    | a | b | no_others     | current_row | grp       | ties
--  one   | 1 | A | A.D.G         | D.G         |           | A
--  one   | 4 | D | A.D.G         | A.G         |           | D
--  one   | 7 | G | A.D.G         | A.D         |           | G
--  three | 3 | C | A.D.G.C.F     | A.D.G.F     | A.D.G     | A.D.G.C
--  three | 6 | F | A.D.G.C.F     | A.D.G.C     | A.D.G     | A.D.G.F
--  two   | 2 | B | A.D.G.C.F.B.E | A.D.G.C.F.E | A.D.G.C.F | A.D.G.C.F.B
--  two   | 5 | E | A.D.G.C.F.B.E | A.D.G.C.F.B | A.D.G.C.F | A.D.G.C.F.E
-- 
SELECT c, a, b,
  group_concat(b, '.') OVER (
    ORDER BY c GROUPS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW EXCLUDE NO OTHERS
  ) AS no_others,
  group_concat(b, '.') OVER (
    ORDER BY c GROUPS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW EXCLUDE CURRENT ROW
  ) AS current_row,
  group_concat(b, '.') OVER (
    ORDER BY c GROUPS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW EXCLUDE GROUP
  ) AS grp,
  group_concat(b, '.') OVER (
    ORDER BY c GROUPS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW EXCLUDE TIES
  ) AS ties
FROM t1 ORDER BY c, a;


2.3. The FILTER Clause

filter-clause:
hide
 
 

expr:
show
 
 

literal-value:
show
 
 


over-clause:
show
 
 

frame-spec:
show
 
 


ordering-term:
show
 
 



raise-function:
show
 
 


select-stmt:
show
 
 

common-table-expression:
show
 
 


compound-operator:
show
 
 


join-clause:
show
 
 

join-constraint:
show
 
 


join-operator:
show
 
 



ordering-term:
show
 
 


result-column:
show
 
 


table-or-subquery:
show
 
 


window-defn:
show
 
 

frame-spec:
show
 
 




type-name:
show
 
 

signed-number:
show
 
 







If a FILTER clause is provided, then only rows for which the expr is
true are included in the window frame. The aggregate window still returns a
value for every row, but those for which the FILTER expression evaluates to
other than true are not included in the window frame for any row. For example:

-- The following SELECT statement returns:
-- 
--   c     | a | b | group_concat
---------------------------------
--   one   | 1 | A | A           
--   two   | 2 | B | A           
--   three | 3 | C | A.C         
--   one   | 4 | D | A.C.D       
--   two   | 5 | E | A.C.D       
--   three | 6 | F | A.C.D.F     
--   one   | 7 | G | A.C.D.F.G   
-- 
SELECT c, a, b, group_concat(b, '.') FILTER (WHERE c!='two') OVER (
  ORDER BY a
) AS group_concat
FROM t1 ORDER BY a;





3. Built-in Window Functions

 As well as aggregate window functions, SQLite features a set of built-in
window functions based on 

those supported by PostgreSQL.

 Built-in window functions honor any PARTITION BY clause in the same way
as aggregate window functions - each selected row is assigned to a partition 
and each partition is processed separately. The ways in which any ORDER BY
clause affects each built-in window function is described below. Some of
the window functions (rank(), dense_rank(), percent_rank() and ntile()) use
the concept of "peer groups" (rows within the same partition that have the
same values for all ORDER BY expressions). In these cases, it does not matter
whether the frame-spec specifies ROWS, GROUPS, or RANGE.
For the purposes of built-in window function processing, rows with the same values
for all ORDER BY expressions are considered peers regardless of the frame type.

 Most built-in window functions ignore the
frame-spec, the exceptions being first_value(),
last_value() and nth_value(). It is a syntax error to specify a FILTER 
clause as part of a built-in window function invocation.



 SQLite supports the following 11 built-in window functions:


  row_number()
   The number of the row within the current partition. Rows are
       numbered starting from 1 in the order defined by the ORDER BY clause in
       the window definition, or in arbitrary order otherwise.
  rank()
   The row_number() of the first peer in each group - the rank of the
       current row with gaps. If there is no ORDER BY clause, then all rows
       are considered peers and this function always returns 1.
  dense_rank()
   The number of the current row's peer group within its partition - the
       rank of the current row without gaps. Rows are numbered starting
       from 1 in the order defined by the ORDER BY clause in the window
       definition. If there is no ORDER BY clause, then all rows are
       considered peers and this function always returns 1.
  percent_rank()
   Despite the name, this function always returns a value between 0.0
       and 1.0 equal to (rank - 1)/(partition-rows - 1), where
       rank is the value returned by built-in window function rank()
       and partition-rows is the total number of rows in the
       partition. If the partition contains only one row, this function
       returns 0.0.
  cume_dist()
   The cumulative distribution. Calculated as
       row-number/partition-rows, where row-number is
       the value returned by row_number() for the last peer in the group
       and partition-rows the number of rows in the partition.
  ntile(N)
   Argument N is handled as an integer. This function divides the
       partition into N groups as evenly as possible and assigns an integer
       between 1 and N to each group, in the order defined by the ORDER
       BY clause, or in arbitrary order otherwise. If necessary, larger groups
       occur first. This function returns the integer value assigned to the
       group that the current row is a part of.

  lag(expr)lag(expr, offset)lag(expr, offset, default)
   The first form of the lag() function returns the result of evaluating
       expression expr against the previous row in the partition. Or, if
       there is no previous row (because the current row is the first), NULL.

       If the offset argument is provided, then it must be a
       non-negative integer. In this case the value returned is the result
       of evaluating expr against the row offset rows before the
       current row within the partition. If offset is 0, then
       expr is evaluated against the current row. If there is no row
       offset rows before the current row, NULL is returned.

       If default is also provided, then it is returned instead of
      NULL if the row identified by offset does not exist.
       
  lead(expr)lead(expr, offset)lead(expr, offset, default)
   The first form of the lead() function returns the result of evaluating
       expression expr against the next row in the partition. Or, if
       there is no next row (because the current row is the last), NULL.

       If the offset argument is provided, then it must be a
       non-negative integer. In this case the value returned is the result
       of evaluating expr against the row offset rows after the
       current row within the partition. If offset is 0, then
       expr is evaluated against the current row. If there is no row
       offset rows after the current row, NULL is returned.

       If default is also provided, then it is returned instead of
      NULL if the row identified by offset does not exist.
  first_value(expr)
   This built-in window function calculates the window frame for each
       row in the same way as an aggregate window function. It returns the
       value of expr evaluated against the first row in the window frame
       for each row.
  last_value(expr)
   This built-in window function calculates the window frame for each
       row in the same way as an aggregate window function. It returns the
       value of expr evaluated against the last row in the window frame
       for each row.
  nth_value(expr, N)
   This built-in window function calculates the window frame for each
       row in the same way as an aggregate window function. It returns the
       value of expr evaluated against the row N of the window
       frame. Rows are numbered within the window frame starting from 1 in
       the order defined by the ORDER BY clause if one is present, or in
       arbitrary order otherwise. If there is no Nth row in the
       partition, then NULL is returned.
  


The examples in this section use the
previously defined T1 table
as well as the following T2 table:

CREATE TABLE t2(a, b);
INSERT INTO t2 VALUES('a', 'one'), 
                     ('a', 'two'), 
                     ('a', 'three'), 
                     ('b', 'four'), 
                     ('c', 'five'), 
                     ('c', 'six');


The following example illustrates the behaviour of the five ranking
functions - row_number(), rank(), dense_rank(), percent_rank() and 
cume_dist().

-- The following SELECT statement returns:
-- 
--   a | row_number | rank | dense_rank | percent_rank | cume_dist
------------------------------------------------------------------
--   a |          1 |    1 |          1 |          0.0 |       0.5
--   a |          2 |    1 |          1 |          0.0 |       0.5
--   a |          3 |    1 |          1 |          0.0 |       0.5
--   b |          4 |    4 |          2 |          0.6 |       0.66
--   c |          5 |    5 |          3 |          0.8 |       1.0
--   c |          6 |    5 |          3 |          0.8 |       1.0
-- 
SELECT a                        AS a,
       row_number() OVER win    AS row_number,
       rank() OVER win          AS rank,
       dense_rank() OVER win    AS dense_rank,
       percent_rank() OVER win  AS percent_rank,
       cume_dist() OVER win     AS cume_dist
FROM t2
WINDOW win AS (ORDER BY a);


The example below uses ntile() to divide the six rows into two groups (the
ntile(2) call) and into four groups (the ntile(4) call). For ntile(2), there
are three rows assigned to each group. For ntile(4), there are two groups of
two and two groups of one. The larger groups of two appear first.

-- The following SELECT statement returns:
-- 
--   a | b     | ntile_2 | ntile_4
----------------------------------
--   a | one   |       1 |       1
--   a | two   |       1 |       1
--   a | three |       1 |       2
--   b | four  |       2 |       2
--   c | five  |       2 |       3
--   c | six   |       2 |       4
-- 
SELECT a                        AS a,
       b                        AS b,
       ntile(2) OVER win        AS ntile_2,
       ntile(4) OVER win        AS ntile_4
FROM t2
WINDOW win AS (ORDER BY a);


 The next example demonstrates lag(), lead(), first_value(), last_value()
and nth_value(). The frame-spec is ignored by 
both lag() and lead(), but respected by first_value(), last_value()
and nth_value().

-- The following SELECT statement returns:
-- 
--   b | lead | lag  | first_value | last_value | nth_value_3
-------------------------------------------------------------
--   A | C    | NULL | A           | A          | NULL       
--   B | D    | A    | A           | B          | NULL       
--   C | E    | B    | A           | C          | C          
--   D | F    | C    | A           | D          | C          
--   E | G    | D    | A           | E          | C          
--   F | n/a  | E    | A           | F          | C          
--   G | n/a  | F    | A           | G          | C          
-- 
SELECT b                          AS b,
       lead(b, 2, 'n/a') OVER win AS lead,
       lag(b) OVER win            AS lag,
       first_value(b) OVER win    AS first_value,
       last_value(b) OVER win     AS last_value,
       nth_value(b, 3) OVER win   AS nth_value_3
FROM t1
WINDOW win AS (ORDER BY b ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)





4. Window Chaining


Window chaining is a shorthand that allows one window to be defined in terms 
of another. Specifically, the shorthand allows the new window to implicitly
copy the PARTITION BY and optionally ORDER BY clauses of the base window. For
example, in the following:

SELECT group_concat(b, '.') OVER (
  win ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
)
FROM t1
WINDOW win AS (PARTITION BY a ORDER BY c)



the window used by the group_concat() function is equivalent
to "PARTITION BY a ORDER BY c ROWS BETWEEN UNBOUNDED PRECEDING
AND CURRENT ROW". In order to use window
chaining, all of the following must be true:


  The new window definition must not include a PARTITION BY clause. The
       PARTITION BY clause, if there is one, must be supplied by the base
       window specification.

  If the base window has an ORDER BY clause, it is copied into the new
       window. In this case the new window must not specify an ORDER BY clause.
       If the base window has no ORDER BY clause, one may be specified as part
       of the new window definition.
  
  The base window may not specify a frame specification.  The frame
      specification can only be given in the new window specification.


The two fragments of SQL below are similar, but not entirely equivalent, as
the latter will fail if the definition of window "win" contains a frame
specification.

SELECT group_concat(b, '.') OVER win ...
SELECT group_concat(b, '.') OVER (win) ...




5. User-Defined Aggregate Window Functions

 User-defined aggregate window functions may be created using the
sqlite3_create_window_function() API. Implementing an aggregate window
function is very similar to an ordinary aggregate function. Any user-defined
aggregate window function may also be used as an ordinary aggregate. To 
implement a user-defined aggregate window function the application must
supply four callback functions:


Callback Description
xStep 
      This method is required by both window aggregate and legacy aggregate
      function implementations. It is invoked to add a row to the current
      window. The function arguments, if any, corresponding to the row being
      added are passed to the implementation of xStep.

xFinal 
      This method is required by both window aggregate and legacy aggregate
      function implementations. It is invoked to return the current value
      of the aggregate (determined by the contents of the current window),
      and to free any resources allocated by earlier calls to xStep.

xValue 
      This method is  only required for window aggregate functions.  The presence
      of this method is what distinguishes a window aggregate function from a
      legacy  aggregate function.  This method is invoked to return the current
      value of the aggregate. Unlike xFinal, the implementation should not
      delete any context.

xInverse 
      This method is only required for window aggregate functions, not legacy
      aggregate function implementations. It is invoked to remove the oldest
      presently aggregated result of xStep from the current window.
      The function arguments, if any, are those
      passed to xStep for the row being removed.


 The C code below implements a simple window aggregate function named
sumint(). This works in the same way as the built-in sum() function, except
that it throws an exception if passed an argument that is not an integer
value.


/*
** xStep for sumint().
**
** Add the value of the argument to the aggregate context (an integer).
*/
static void sumintStep(
  sqlite3_context *ctx, 
  int nArg, 
  sqlite3_value *apArg[]
){
  sqlite3_int64 *pInt;

  assert( nArg==1 );
  if( sqlite3_value_type(apArg[0])!=SQLITE_INTEGER ){
    sqlite3_result_error(ctx, "invalid argument", -1);
    return;
  }
  pInt = (sqlite3_int64*)sqlite3_aggregate_context(ctx, sizeof(sqlite3_int64));
  if( pInt ){
    *pInt += sqlite3_value_int64(apArg[0]);
  }
}

/*
** xInverse for sumint().
**
** This does the opposite of xStep() - subtracts the value of the argument
** from the current context value. The error checking can be omitted from
** this function, as it is only ever called after xStep() (so the aggregate
** context has already been allocated) and with a value that has already
** been passed to xStep() without error (so it must be an integer).
*/
static void sumintInverse(
  sqlite3_context *ctx, 
  int nArg, 
  sqlite3_value *apArg[]
){
  sqlite3_int64 *pInt;
  assert( sqlite3_value_type(apArg[0])==SQLITE_INTEGER );
  pInt = (sqlite3_int64*)sqlite3_aggregate_context(ctx, sizeof(sqlite3_int64));
  *pInt -= sqlite3_value_int64(apArg[0]);
}

/*
** xFinal for sumint().
**
** Return the current value of the aggregate window function. Because
** this implementation does not allocate any resources beyond the buffer
** returned by sqlite3_aggregate_context, which is automatically freed
** by the system, there are no resources to free. And so this method is
** identical to xValue().
*/
static void sumintFinal(sqlite3_context *ctx){
  sqlite3_int64 res = 0;
  sqlite3_int64 *pInt;
  pInt = (sqlite3_int64*)sqlite3_aggregate_context(ctx, 0);
  if( pInt ) res = *pInt;
  sqlite3_result_int64(ctx, res);
}

/*
** xValue for sumint().
**
** Return the current value of the aggregate window function.
*/
static void sumintValue(sqlite3_context *ctx){
  sqlite3_int64 res = 0;
  sqlite3_int64 *pInt;
  pInt = (sqlite3_int64*)sqlite3_aggregate_context(ctx, 0);
  if( pInt ) res = *pInt;
  sqlite3_result_int64(ctx, res);
}

/*
** Register sumint() window aggregate with database handle db. 
*/
int register_sumint(sqlite3 *db){
  return sqlite3_create_window_function(db, "sumint", 1, SQLITE_UTF8, 0,
      sumintStep, sumintFinal, sumintValue, sumintInverse, 0
  );
}




 The following example uses the sumint() function implemented by the above
C code. For each row, the window consists of the preceding row (if any), the current row and the following row (again, if any):

CREATE TABLE t3(x, y);
INSERT INTO t3 VALUES('a', 4),
                     ('b', 5),
                     ('c', 3),
                     ('d', 8),
                     ('e', 1);

-- Assuming the database is populated using the above script, the 
-- following SELECT statement returns:
-- 
--   x | sum_y
--------------
--   a | 9    
--   b | 12   
--   c | 16   
--   d | 12   
--   e | 9    
-- 
SELECT x, sumint(y) OVER (
  ORDER BY x ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING
) AS sum_y
FROM t3 ORDER BY x;


In processing the query above, SQLite invokes the sumint callbacks as
follows:



   xStep(4) - add "4" to the current window.
   xStep(5) - add "5" to the current window.
   xValue() - invoke xValue() to obtain the value of sumint() for
       the row with (x='a'). The window currently consists of values 4 and 5, 
       and so the result is 9.
   xStep(3) - add "3" to the current window.
   xValue() - invoke xValue() to obtain the value of sumint() for
       the row with (x='b'). The window currently consists of values 4, 5 and
       3, and so the result is 12.
   xInverse(4) - remove "4" from the window. 
   xStep(8) - add "8" to the current window. The window now consists
       of values 5, 3 and 8.
   xValue() - invoked to obtain the value for the row with (x='c').
       In this case, 16.
   xInverse(5) - remove value "5" from the window.
   xStep(1) - add value "1" to the window.
   xValue() - invoked to obtain the value for row (x='d').
   xInverse(3) - remove value "3" from the window. The window now
       contains values 8 and 1 only.
   xValue() - invoked to obtain the value for row (x='e'). 9.
   xFinal() - invoked to reclaim any allocated resources.


6. History

Window function support was first added to SQLite with release
version 3.25.0 (2018-09-15). The SQLite developers used
the PostgreSQL window function
documentation as their primary reference for how window functions
ought to behave.  Many test cases have been run against PostgreSQL
to ensure that window functions operate the same way in both
SQLite and PostgreSQL.

In SQLite version 3.28.0 (2019-04-16),
windows function support was extended to include the EXCLUDE clause,
GROUPS frame types, window chaining, and support for
"&lt;expr&gt; PRECEDING" and "&lt;expr&gt; FOLLOWING" boundaries
in RANGE frames.


c2: 35% Faster Than The Filesystem
c5: fasterthanfs.html
c4:
1. Summary

SQLite reads and writes small blobs (for example, thumbnail images)
35% faster&sup1; than the same blobs
can be read from or written to individual files on disk using
fread() or fwrite().

Furthermore, a single SQLite database holding
10-kilobyte blobs uses about 20% less disk space than
storing the blobs in individual files.

The performance difference arises (we believe) because when
working from an SQLite database, the open() and close() system calls
are invoked only once, whereas
open() and close() are invoked once for each blob
when using blobs stored in individual files.  It appears that the
overhead of calling open() and close() is greater than the overhead
of using the database.  The size reduction arises from the fact that
individual files are padded out to the next multiple of the filesystem
block size, whereas the blobs are packed more tightly into an SQLite
database.


The measurements in this article were made during the week of 2017-06-05
using a version of SQLite in between 3.19.2 and 3.20.0.  You may expect
future versions of SQLite to perform even better.

1.1. Caveats



&sup1;The 35% figure above is approximate.  Actual timings vary
depending on hardware, operating system, and the
details of the experiment, and due to random performance fluctuations
on real-world hardware.  See the text below for more detail.
Try the experiments yourself.  Report significant deviations on
the SQLite forum.



The 35% figure is based on running tests on every machine
that the author has easily at hand.
Some reviewers of this article report that SQLite has higher 
latency than direct I/O on their systems.  We do not yet understand
the difference.  We also see indications that SQLite does not
perform as well as direct I/O when experiments are run using
a cold filesystem cache.


So let your take-away be this: read/write latency for
SQLite is competitive with read/write latency of individual files on
disk.  Often SQLite is faster.  Sometimes SQLite is almost
as fast.  Either way, this article disproves the common
assumption that a relational database must be slower than direct
filesystem I/O.

1.2. Related Studies


Jim Gray
and others studied the read performance of BLOBs
versus file I/O for Microsoft SQL Server and found that reading BLOBs 
out of the 
database was faster for BLOB sizes less than between 250KiB and 1MiB.
(Paper).
In that study, the database still stores the filename of the content even
if the content is held in a separate file.  So the database is consulted
for every BLOB, even if it is only to extract the filename.  In this
article, the key for the BLOB is the filename, so no preliminary database
access is required.  Because the database is never used at all when
reading content from individual files in this article, the threshold
at which direct file I/O becomes faster is smaller than it is in Gray's
paper.


The Internal Versus External BLOBs article on this website is an
earlier investigation (circa 2011) that uses the same approach as the
Jim Gray paper &mdash; storing the blob filenames as entries in the
database &mdash; but for SQLite instead of SQL Server.



2. How These Measurements Are Made

I/O performance is measured using the
kvtest.c program
from the SQLite source tree.
To compile this test program, first gather the kvtest.c source file
into a directory with the SQLite amalgamation source
files "sqlite3.c" and "sqlite3.h".  Then on unix, run a command like
the following:

gcc -Os -I. -DSQLITE_DIRECT_OVERFLOW_READ &#92;
  kvtest.c sqlite3.c -o kvtest -ldl -lpthread


Or on Windows with MSVC:

cl -I. -DSQLITE_DIRECT_OVERFLOW_READ kvtest.c sqlite3.c


Instructions for compiling for Android
are shown below.


Use the resulting "kvtest" program to
generate a test database with 100,000 random uncompressible
blobs, each with a random
size between 8,000 and 12,000 bytes
using a command like this:

./kvtest init test1.db --count 100k --size 10k --variance 2k



If desired, you can verify the new database by running this command:

./kvtest stat test1.db



Next, make copies of all the blobs into individual files in a directory
using a command like this:

./kvtest export test1.db test1.dir



At this point, you can measure the amount of disk space used by
the test1.db database and the space used by the test1.dir directory
and all of its content.  On a standard Ubuntu Linux desktop, the
database file will be 1,024,512,000 bytes in size and the test1.dir
directory will use 1,228,800,000 bytes of space (according to "du -k"),
about 20% more than the database.


The "test1.dir" directory created above puts all the blobs into a single
folder.  It was conjectured that some operating systems would perform 
poorly when a single directory contains 100,000 objects.  To test this,
the kvtest program can also store the blobs in a hierarchy of folders with no
more than 100 files and/or subdirectories per folder.  The alternative
on-disk representation of the blobs can be created using the --tree
command-line option to the "export" command, like this:

./kvtest export test1.db test1.tree --tree



The test1.dir directory will contain 100,000 files
with names like "000000", "000001", "000002" and so forth but the
test1.tree directory will contain the same files in subdirectories like
"00/00/00", "00/00/01", and so on.  The test1.dir and test1.test
directories take up approximately the same amount of space, though
test1.test is very slightly larger due to the extra directory entries.


All of the experiments that follow operate the same with either 
"test1.dir" or "test1.tree".  Very little performance difference is
measured in either case, regardless of operating system.


Measure the performance for reading blobs from the database and from
individual files using these commands:

./kvtest run test1.db --count 100k --blob-api
./kvtest run test1.dir --count 100k --blob-api
./kvtest run test1.tree --count 100k --blob-api



Depending on your hardware and operating system, you should see that reads 
from the test1.db database file are about 35% faster than reads from 
individual files in the test1.dir or test1.tree folders.  Results can vary
significantly from one run to the next due to caching, so it is advisable
to run tests multiple times and take an average or a worst case or a best
case, depending on your requirements.

The --blob-api option on the database read test causes kvtest to use
the sqlite3_blob_read() feature of SQLite to load the content of the
blobs, rather than running pure SQL statements.  This helps SQLite to run
a little faster on read tests.  You can omit that option to compare the
performance of SQLite running SQL statements.
In that case, the SQLite still out-performs direct reads, though
by not as much as when using sqlite3_blob_read().
The --blob-api option is ignored for tests that read from individual disk
files.


Measure write performance by adding the --update option.  This causes
the blobs are overwritten in place with another random blob of
exactly the same size.

./kvtest run test1.db --count 100k --update
./kvtest run test1.dir --count 100k --update
./kvtest run test1.tree --count 100k --update



The writing test above is not completely fair, since SQLite is doing
power-safe transactions whereas the direct-to-disk writing is not.
To put the tests on a more equal footing, add either the --nosync
option to the SQLite writes to disable calling fsync() or
FlushFileBuffers() to force content to disk, or using the --fsync option
for the direct-to-disk tests to force them to invoke fsync() or
FlushFileBuffers() when updating disk files.


By default, kvtest runs the database I/O measurements all within
a single transaction.  Use the --multitrans option to run each blob
read or write in a separate transaction.  The --multitrans option makes
SQLite much slower, and uncompetitive with direct disk I/O.  This
option proves, yet again, that to get the most performance out of
SQLite, you should group as much database interaction as possible within
a single transaction.


There are many other testing options, which can be seen by running
the command:

./kvtest help


2.1. Read Performance Measurements

The chart below shows data collected using 
kvtest.c on five different
systems:


Win7: A circa-2009 Dell Inspiron laptop, Pentium dual-core
    at 2.30GHz, 4GiB RAM, Windows7.
Win10: A 2016 Lenovo YOGA 910, Intel i7-7500 at 2.70GHz,
    16GiB RAM, Windows10.
Mac: A 2015 MacBook Pro, 3.1GHz intel Core i7, 16GiB RAM,
    MacOS 10.12.5
Ubuntu: Desktop built from Intel i7-4770K at 3.50GHz, 32GiB RAM,
    Ubuntu 16.04.2 LTS
Android: Galaxy S3, ARMv7, 2GiB RAM


All machines use SSD except Win7 which has a
hard-drive. The test database is 100K blobs with sizes uniformly
distributed between 8K and 12K, for a total of about 1 gigabyte
of content.  The database page size
is 4KiB.  The -DSQLITE_DIRECT_OVERFLOW_READ compile-time option was
used for all of these tests.
Tests were run multiple times.
The first run was used to warm up the cache and its timings were discarded.


The chart below shows average time to read a blob directly from the
filesystem versus the time needed to read the same blob from the SQLite 
database.
The actual timings vary considerably from one system to another 
(the Ubuntu desktop is much
faster than the Galaxy S3 phone, for example).  
This chart shows the ratio of the
times needed to read blobs from a file divided by the time needed to
from the database.  The left-most column in the chart is the normalized
time to read from the database, for reference.


In this chart, an SQL statement ("SELECT v FROM kv WHERE k=?1") 
is prepared once.  Then for each blob, the blob key value is bound 
to the ?1 parameter and the statement is evaluated to extract the
blob content.


The chart shows that on Windows10, content can be read from the SQLite
database about 5 times faster than it can be read directly from disk.
On Android, SQLite is only about 35% faster than reading from disk.






Chart 1:  SQLite read latency relative to direct filesystem reads.
100K blobs, avg 10KB each, random order using SQL



The performance can be improved slightly by bypassing the SQL layer
and reading the blob content directly using the
sqlite3_blob_read() interface, as shown in the next chart:






Chart 2:  SQLite read latency relative to direct filesystem reads.
100K blobs, avg size 10KB, random order
using sqlite3_blob_read().



Further performance improves can be made by using the
memory-mapped I/O feature of SQLite.  In the next chart, the
entire 1GB database file is memory mapped and blobs are read
(in random order) using the sqlite3_blob_read() interface.
With these optimizations, SQLite is twice as fast as Android
or MacOS-X and over 10 times faster than Windows.






Chart 3:  SQLite read latency relative to direct filesystem reads.
100K blobs, avg size 10KB, random order
using sqlite3_blob_read() from a memory-mapped database.



The third chart shows that reading blob content out of SQLite can be
twice as fast as reading from individual files on disk for Mac and
Android, and an amazing ten times faster for Windows.

2.2. Write Performance Measurements


Writes are slower.
On all systems, using both direct I/O and SQLite, write performance is
between 5 and 15 times slower than reads.


Write performance measurements were made by replacing (overwriting)
an entire blob with a different blob.  All of the blobs in these
experiment are random and incompressible.  Because writes are so much
slower than reads, only 10,000 of the 100,000 blobs in the database
are replaced.  The blobs to be replaced are selected at random and
are in no particular order.


The direct-to-disk writes are accomplished using fopen()/fwrite()/fclose().
By default, and in all the results shown below, the OS filesystem buffers are
never flushed to persistent storage using fsync() or
FlushFileBuffers().  In other words, there is no attempt to make the
direct-to-disk writes transactional or power-safe.
We found that invoking fsync() or FlushFileBuffers() on each file
written causes direct-to-disk storage
to be about 10 times or more slower than writes to SQLite.


The next chart compares SQLite database updates in WAL mode
against raw direct-to-disk overwrites of separate files on disk.
The PRAGMA synchronous setting is NORMAL.
All database writes are in a single transaction.
The timer for the database writes is stopped after the transaction
commits, but before a checkpoint is run.
Note that the SQLite writes, unlike the direct-to-disk writes,
are transactional and power-safe, though because the synchronous
setting is NORMAL instead of FULL, the transactions are not durable.






Chart 4:  SQLite write latency relative to direct filesystem writes.
10K blobs, avg size 10KB, random order,
WAL mode with synchronous NORMAL,
exclusive of checkpoint time



The android performance numbers for the write experiments are omitted
because the performance tests on the Galaxy S3 are so random.  Two
consecutive runs of the exact same experiment would give wildly different
times.  And, to be fair, the performance of SQLite on android is slightly
slower than writing directly to disk.


The next chart shows the performance of SQLite versus direct-to-disk
when transactions are disabled (PRAGMA journal_mode=OFF)
and PRAGMA synchronous is set to OFF.  These settings put SQLite on an
equal footing with direct-to-disk writes, which is to say they make the
data prone to corruption due to system crashes and power failures.






Chart 5:  SQLite write latency relative to direct filesystem writes.
10K blobs, avg size 10KB, random order,
journaling disabled, synchronous OFF.



In all of the write tests, it is important to disable anti-virus software
prior to running the direct-to-disk performance tests.  We found that
anti-virus software slows down direct-to-disk by an order of magnitude
whereas it impacts SQLite writes very little.  This is probably due to the
fact that direct-to-disk changes thousands of separate files which all need
to be checked by anti-virus, whereas SQLite writes only changes the single
database file.

2.3. Variations

The -DSQLITE_DIRECT_OVERFLOW_READ compile-time option causes SQLite
to bypass its page cache when reading content from overflow pages.  This
helps database reads of 10K blobs run a little faster, but not all that much
faster.  SQLite still holds a speed advantage over direct filesystem reads
without the SQLITE_DIRECT_OVERFLOW_READ compile-time option.

Other compile-time options such as using -O3 instead of -Os or
using -DSQLITE_THREADSAFE=0 and/or some of the other
recommended compile-time options might help SQLite to run even faster
relative to direct filesystem reads.

The size of the blobs in the test data affects performance.
The filesystem will generally be faster for larger blobs, since
the overhead of open() and close() is amortized over more bytes of I/O,
whereas the database will be more efficient in both speed and space
as the average blob size decreases.


3. General Findings



SQLite is competitive with, and usually faster than, blobs stored in
separate files on disk, for both reading and writing.


SQLite is much faster than direct writes to disk on Windows
when anti-virus protection is turned on.  Since anti-virus software
is and should be on by default in Windows, that means that SQLite
is generally much faster than direct disk writes on Windows.


Reading is about an order of magnitude faster than writing, for all
systems and for both SQLite and direct-to-disk I/O.


I/O performance varies widely depending on operating system and hardware.
Make your own measurements before drawing conclusions.


Some other SQL database engines advise developers to store blobs in separate
files and then store the filename in the database.  In that case, where
the database must first be consulted to find the filename before opening
and reading the file, simply storing the entire blob in the database
gives much faster read and write performance with SQLite.
See the Internal Versus External BLOBs article for more information.



4. Additional Notes


4.1. Compiling And Testing on Android


The kvtest program is compiled and run on Android as follows.
First install the Android SDK and NDK.  Then prepare a script
named "android-gcc" that looks approximately like this:

#!/bin/sh
#
NDK=/home/drh/Android/Sdk/ndk-bundle
SYSROOT=$NDK/platforms/android-16/arch-arm
ABIN=$NDK/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin
GCC=$ABIN/arm-linux-androideabi-gcc
$GCC --sysroot=$SYSROOT -fPIC -pie $*


Make that script executable and put it on your $PATH.  Then
compile the kvtest program as follows:

android-gcc -Os -I. kvtest.c sqlite3.c -o kvtest-android


Next, move the resulting kvtest-android executable to the Android
device:

adb push kvtest-android /data/local/tmp


Finally use "adb shell" to get a shell prompt on the Android device,
cd into the /data/local/tmp directory, and begin running the tests
as with any other unix host.


c2: Well-Known Users Of SQLite
c5: famous.html
c4:
Well-Known Users of SQLite


SQLite is used by literally millions of applications
with literally billions and billions of deployments.
SQLite is the most widely deployed database engine in the
world today.


A few of the better-known users of SQLite are shown below
in alphabetical order.
This is not a complete list.
SQLite is in the 
public domain and so most
developers use it in their projects without ever telling us.







  Adobe uses SQLite as the 
  application 
  file format for their 
  Photoshop Lightroom
  product.
  SQLite is also a standard part of the
  Adobe Integrated Runtime (AIR).
  It is reported that 
  Acrobat Reader
  also uses SQLite.






  Airbus confirms that SQLite is being
  used in the flight software for the 
  A350 XWB family
  of aircraft.






  Apple uses SQLite in many (most?) of the
  native applications running on Mac OS-X desktops and servers and
  on iOS devices such as iPhones and iPods.  SQLite is also used in
  iTunes, even on non-Apple hardware.






  Bentley Systems uses SQLite as the
  application file format for their
  Microstation
  CAD/CAM product.






  Bosch uses SQLite in the multimedia
  systems install on GM, Nissan, and Suzuki automobiles.
  (link)






  The increasingly popular Dropbox file
  archiving and synchronization service is reported to use SQLite as
  the primary data store on the client side.






  Expensify uses SQLite as a server-side
  database engine for their enterprise-scale expense reporting software.






  Facebook uses SQLite as the SQL database
  engine in their 
  osquery product.






  We believe that General Electric
  uses SQLite in some product or 
  another because they have written to the SQLite developers at 
  least four separate times requesting 
  the US Export Control Number for SQLite. So presumably GE is using
  SQLite in something that they are exporting. But nobody
  (outside of GE) seems to know what that might be.






  uses SQLite in their 
  Android cell-phone 
  operating system, and in the 
  Chrome Web Browser.






  Intuit apparently uses SQLite in 
  QuickBooks and in
  TurboTax to judge from some error
  reports from users seen 
  here and
  here.






  The United States Library of Congress recognizes SQLite as a
  recommended storage format for preservation of digital content.






  McAfee uses SQLite in its antivirus 
  programs.  
  Mentioned here and implied 
  here.






  Microsoft uses SQLite as a core component
  of Windows 10, and in other products.






  SQLite is the primary meta-data storage format for
  the Firefox Web Browser and the
  Thunderbird Email Reader from
  Mozilla.






  The Navigation Data Standard uses 
  SQLite as its application file format.






  The popular PHP
  programming language comes with both SQLite2 and SQLite3 built in.






  All Python
  distributions since Python 2.5 include SQLite.






  The RedHat Package Manager (RPM)
  uses SQLite to track its state.






  There are 
  multiple
  sightings
  of SQLite in the Skype client for Mac OS X and Windows.






  The Tcl/Tk programming language now comes with SQLite built-in.  SQLite works
  particularly well with Tcl since SQLite was originally a Tcl extension that
  subsequently "escaped" into the wild.






  SQLite comes bundled with the Xojo
  programming environment
  (formerly REALbasic)










  Adobe uses SQLite as the 
  application 
  file format for their 
  Photoshop Lightroom
  product.
  SQLite is also a standard part of the
  Adobe Integrated Runtime (AIR).
  It is reported that 
  Acrobat Reader
  also uses SQLite.






  Airbus confirms that SQLite is being
  used in the flight software for the 
  A350 XWB family
  of aircraft.






  Apple uses SQLite in many (most?) of the
  native applications running on Mac OS-X desktops and servers and
  on iOS devices such as iPhones and iPods.  SQLite is also used in
  iTunes, even on non-Apple hardware.






  Bentley Systems uses SQLite as the
  application file format for their
  Microstation
  CAD/CAM product.






  Bosch uses SQLite in the multimedia
  systems install on GM, Nissan, and Suzuki automobiles.
  (link)






  The increasingly popular Dropbox file
  archiving and synchronization service is reported to use SQLite as
  the primary data store on the client side.






  Expensify uses SQLite as a server-side
  database engine for their enterprise-scale expense reporting software.






  Facebook uses SQLite as the SQL database
  engine in their 
  osquery product.






  We believe that General Electric
  uses SQLite in some product or 
  another because they have written to the SQLite developers at 
  least four separate times requesting 
  the US Export Control Number for SQLite. So presumably GE is using
  SQLite in something that they are exporting. But nobody
  (outside of GE) seems to know what that might be.






  uses SQLite in their 
  Android cell-phone 
  operating system, and in the 
  Chrome Web Browser.






  Intuit apparently uses SQLite in 
  QuickBooks and in
  TurboTax to judge from some error
  reports from users seen 
  here and
  here.






  The United States Library of Congress recognizes SQLite as a
  recommended storage format for preservation of digital content.






  McAfee uses SQLite in its antivirus 
  programs.  
  Mentioned here and implied 
  here.






  Microsoft uses SQLite as a core component
  of Windows 10, and in other products.






  SQLite is the primary meta-data storage format for
  the Firefox Web Browser and the
  Thunderbird Email Reader from
  Mozilla.






  The Navigation Data Standard uses 
  SQLite as its application file format.






  The popular PHP
  programming language comes with both SQLite2 and SQLite3 built in.






  All Python
  distributions since Python 2.5 include SQLite.






  The RedHat Package Manager (RPM)
  uses SQLite to track its state.






  There are 
  multiple
  sightings
  of SQLite in the Skype client for Mac OS X and Windows.






  The Tcl/Tk programming language now comes with SQLite built-in.  SQLite works
  particularly well with Tcl since SQLite was originally a Tcl extension that
  subsequently "escaped" into the wild.






  SQLite comes bundled with the Xojo
  programming environment
  (formerly REALbasic)


c2: Imposter Tables
c5: imposter.html
c4:
1. Introduction


An imposter table is a table that is attached to the same b-tree
as an index.
An imposter table allows the content of an index to be queried or
modified as if the index were an ordinary table.



Imposter tables are intended for analysis and debugging only.
This is not a feature that most application developers should understand
or even know about.
Imposter tables are for experts only.



Improper use of imposter tables can cause index corruption, though
any corruption created this way can be fixed by running REINDEX.


2. Details


Each table and each index in SQLite is stored in a separate b-tree
in the database file.  Each b-tree is identified by its root page
number.  The root page number for any index or table can be found
by querying the "rootpage" column of the sqlite_schema table.
See the indexing tutorial and the file format documentation
for further background on this design.



Usually the b-trees for tables and indexes are slightly different.
A table b-tree contains a 64-bit integer key and arbitrary data.
The 64-bit integer key is the ROWID.  Index b-trees contain
an arbitrary binary key and no data.  So table b-trees and index
b-trees are not directly compatible.



However, the b-tree for a WITHOUT ROWID table is in the same format
as an index b-tree.  Thus, an index b-tree can be accessed as if it
were a WITHOUT ROWID table.


2.1. Manually Created Imposter Tables


One way to create an imposter table is to directly edit the sqlite_schema
table to insert a new row that describes the table.
For example, suppose the schema is like this:


CREATE TABLE t1(a INTEGER PRIMARY KEY,b TEXT,c INT, d INT);
CREATE INDEX t1bc ON t1(b,c);



The WITHOUT ROWID table that has the same structure as the t1bc index
would look like this:


CREATE TABLE t2(b TEXT,c INT,a INT, PRIMARY KEY(b,c,a)) WITHOUT ROWID;



To create a permanent imposter table "t2" against index "t1bc" one
should first enable editing of the sqlite_schema table by running
"PRAGMA writable_schema=ON".
(Be careful to observe the warnings that accompany this PRAGMA.
A mistake can cause severe database corruption.)
Then insert a new entry into the sqlite_schema table like this:


INSERT INTO sqlite_schema(type,name,tbl_name,rootpage,sql)
 SELECT 'table','t2','t2',rootpage,
   'CREATE TABLE t2(b,c,a,PRIMARY KEY(b,c,a))WITHOUT ROWID'
   FROM sqlite_schema
  WHERE name='t1bc';



The INSERT statement above adds a new row to the sqlite_schema table that
defines a table "t2" having the same on-disk format as index "t1bc" and 
pointing to the same b-tree.
After adding this sqlite_schema table entry, it is necessary to close and
reopen the database in order to get SQLite to reread the schema.
Then the "t2" table can be queried to see the content of the "t1bc" index.


2.1.1. Corrupted Database


A serious problem with the manual imposter table approach described above
is that after adding the new "t2" entry to the "sqlite_schema" table, the
database file will technically be corrupt.  Both the "t1bc" index and the
"t2" table will point to the same b-tree.  This will not cause
any immediate problems, though one should avoid running VACUUM.



It is possible to write into the "t2" table, thus changing the content
of the index.
But doing so will get the "t1bc" index out of synchronization with its
parent table "t1".  An out-of-sync index can result in incorrect query
results.



Since the "t2" imposter table is a form of database corruption, the
manual approach to creating imposter tables is not recommended.
Actually, any use of imposter tables is discouraged for all but
expert developers, but manually created imposter tables are
especially discouraged because they are permanent.


2.2. Transient Imposter Tables


Another (safer) approach to creating an imposter table is to add an
entry for the imposter table to SQLite's internal symbol table without
updating the "sqlite_schema" table on disk.
That way, the imposter table exists in only a single database connection
and is automatically removed whenever the schema is reloaded.



Creation of a transient imposter table involves a special
sqlite3_test_control() call.  Unlike all other SQLite APIs,
sqlite3_test_control() interface is subject to incompatible changes
from one release to the next, and so the mechanism described below
is not guaranteed to work in future releases of SQLite.  The
SQLite developers do not consider this a problem because imposter
tables should not be used in applications.  Imposter tables are for
analysis and testing use only.



To create a transient imposter table, first call sqlite3_test_control()
as follows:


sqlite3_test_control(SQLITE_TESTCTRL_IMPOSTER, db, "main", 1, tnum);



The "db" parameter is a pointer to the database connection.
The "main" argument is the name of the schema in which the imposter
table is to be created.  The "1" argument enables the imposter table
mechanism.  "tnum" is the root page of the index that the imposter
table should mirror.



After the sqlite3_test_control() call above, then run a CREATE TABLE
statement the defines the imposter table.
With the imposter mechanism enabled, this CREATE TABLE statement does
not create a real table but instead merely adds an entry in SQLite's
internal symbol table.  Note that the CREATE TABLE statement must
be in the correct format for the index.  If the imposter table has the
wrong number of columns or is not a WITHOUT ROWID table or is otherwise
incompatible with the index b-tree, then SQLITE_CORRUPT errors will result
when the imposter table is used.



After running the CREATE TABLE statement, disable the imposter mechanism
as follows:


sqlite3_test_control(SQLITE_TESTCTRL_IMPOSTER, db, "main", 0, 0);



In other words, make the same sqlite3_test_control() call except change
the last two parameters to zero.



After the imposter table is loaded into SQLite's internal schema as
described above, the imposter table can be used as any other table.
But the imposter table will only be visible to the one database
connection that created it.  No changes are made to the database file
on disk.  And the imposter table will disappear the next time the schema 
is loaded.




2.3. The .imposter Shell Command


As of SQLite 3.16.0 (2017-01-02), the command-line shell contains
a dot-command ".imposter" that does all of the work of setting up a
transient imposter table.
Instead of making multiple calls to sqlite3_test_control() and figuring
out and invoking a compatible CREATE TABLE statement, a transient
imposter table can be constructed as follows:


.imposter t1bc t2



Of course, substitute the desired index and imposter table names in
place of the "t1bc" and "t2" shown in the example.
The ".imposter" command reads the schema of the "t1bc" index, uses
that information to construct a compatible CREATE TABLE statement for
the imposter table, then makes all the necessary calls to create the
transient imposter table automatically.


3. Summary And Final Warning


The imposter table mechanism is a power analysis and debugging tool
for SQLite.  But as with all sharp tools, it can also be dangerous and
can result in corrupt database files if misused.  Do not attempt to
use imposter tables in an application.  Imposter tables are intended
for use in the laboratory by experts.


