# page_content

c2: The Advantages Of Flexible Typing
c3: 3.2. The "value" column output from the json_tree virtual tables
c5: flextypegood.html#the_value_column_output_from_the_json_tree_virtual_tables
c4:
The json_tree and json_each table-valued functions that are
built into SQLite both have a "value" column that can hold values of
type INTEGER, REAL, or TEXT depending on the type of the corresponding
JSON field.  For example:

SELECT typeof(value) FROM json_each('{"a":1,"b":2.5,"c":"hello"}');


The query above returns three rows of one column with values
"integer", "real", and "text", respectively.


c2: The Advantages Of Flexible Typing
c3: 3.3. Storage for dirty data
c5: flextypegood.html#storage_for_dirty_data
c4:
Analysts sometimes encounter CSV files where some columns contain
a mixture of integer, real, and text data.  CSV file that are obtained
from Excel spreadsheet exports commonly have this trait, for example.
When importing such "dirty data" into an SQL database, it is convenient
to have flexible columns to import into.

 Dirty data is not restricted to CSV files coming out of Excel, of
course.  There are many data sources in which a single field might
contain a mix of types.  For example, a data column might the number
of seconds since 1970 sometimes, or as a text date string in other cases.
It is desirable to clean up these inconsistent representations,
but at the same time it is convenient to be able to store all the different
representations in the same column of the intermediate database while the
cleanup is underway.


c2: The Advantages Of Flexible Typing
c3: 3.4. Dynamic programming languages
c5: flextypegood.html#dynamic_programming_languages
c4:
SQLite began as a TCL extension that later escaped into the wild.
TCL is a dynamic language in the sense that the programmer does not need
to be aware of datatypes.  Under the hood, TCL keeps careful track of the
datatype of every value, but to the developer and user of a TCL program,
everything looks like a string.  Flexible typing is a natural fit for
use with dynamic programming languages like TCL and others, since with
a dynamic programming language, you can not always predict in advance what
datatype a variable will hold.  So when you need to store the value of that
variable into the database, having a database that is supports flexible
typing makes storage much easier.


c2: The Advantages Of Flexible Typing
c3: 3.5. Data typename cross-compatibility
c5: flextypegood.html#data_typename_cross_compatibility
c4:
Every SQL database engine seems to have its own unique set of supported
datatype names:


 BIGINT
 UNSIGNED SMALL INT
 TEXT
 VARCHAR
 VARYING CHARACTER
 NATIONAL VARYING CHARACTER
 NVARCHAR
 JSON
 REAL
 FLOAT
 DOUBLE PRECISION
  ... and so forth ...


The fact that SQLite will accept any of these names as a valid typename,
and let you store any kind of content into the column, increases the chances
that a script written to run on some other SQL database engine will also
work in SQLite.


c2: The Advantages Of Flexible Typing
c3: 3.6. Repurposing unused or disused columns in legacy databases
c5: flextypegood.html#repurposing_unused_or_disused_columns_in_legacy_databases
c4:
Because an SQLite database file is a single file on disk, some
applications use SQLite as an application file format.  This means
that a single instance of the application might, over the course of its
life, talk to hundreds or thousands of separate databases, each in a separate
file.  When such applications evolve over years, some columns in the
underlying database will have their meanings altered subtly.  Or, it might
be desirable to repurpose an existing column to serve two or more purposes.
This is much easier to do if the column has a flexible datatype.


c2: The Advantages Of Flexible Typing
c3: 4. Perceived Disadvantages of Flexible Typing (With Rebuttals)
c5: flextypegood.html#perceived_disadvantages_of_flexible_typing_with_rebuttals_
c4:
The following perceived disadvantages of flexible typing were
    gleaned and compiled from countless posts on Hacker News and
    Reddit and similar forums where developers discuss these sorts of
    things.  If you can think of other reasons why flexible typing
    is a bad idea, please contact the SQLite developers or leave a
    post on the SQLite Forum so
    that your idea can be added to the list.


c2: The Advantages Of Flexible Typing
c3: 4.1. We've never done it that way before
c5: flextypegood.html#we_ve_never_done_it_that_way_before
c4:
Many skeptics of flexible typing simply express
shock and disbelief, without offering any rationale for why they think
flexible typing is a bad idea.  Without supporting arguments, one must
assume their reason for not liking flexible typing is that it is different
from what they are used to.

 Presumably, many developers who are aghast at SQLite's flexible
typing feel this way because they have just never encountered anything
like it before.  All prior exposure to databases and especially SQL
databases has involved rigid typing, and the readers mental model of
SQL includes rigid typing as a fundamental feature.  Flexible typing
upsets their world-view.

 Yes, flexible typing is a new way of thinking about data in an
SQL database.  But new is not necessary bad.  Sometimes, and I think
especially in the case of flexible typing, innovation leads to improvement.


c2: The Advantages Of Flexible Typing
c3: 4.2. Rigid type enforcement helps prevent application bugs
c5: flextypegood.html#rigid_type_enforcement_helps_prevent_application_bugs
c4:
It has become a point of doctrine among many programmers that the
best way to prevent application bugs is strict type enforcement.  But I
find no evidence in support of this.

 To be sure, strict type enforcement does help prevent some kinds of
bugs in lower-level languages like C and C++ that present a model that is
close to machine hardware.  But this does not seem to
be the case for higher-abstraction languages in which all data is 
passed around in a "Value" superclass of some kind which is subclassed
for the various lower-level data types.  When everything is a Value
object, specific datatypes cease to be important.

 This technical note is being writing by the original author of SQLite.
I having been writing TCL programs for 27 years.  TCL has no type enforcement
whatsoever.  The "Value" class in TCL (called Tcl_Obj) can hold many
different datatypes, but it presents the content to the program and to
the application user as a string.  And I've had a lot of bugs in
those TCL programs over the years.  But I do not recall a single instance
where the bugs might have been caught by a rigid type system.  I have
also written a lot of C code over a span of 35 years, not the
least of which is SQLite itself.  I have found the type system in C
to be very helpful at finding and preventing problems.  For the
Fossil Version Control System, which is written
in C, I have even implemented supplemental static analysis programs that
scan the Fossil source code prior to compilation, looking for problems
that compilers miss.  This works well for compiled programs.

 The SQL language model is a higher-level abstraction that C/C++.
In SQLite, every data item is stored in memory as an "sqlite3_value" object.
There are subclasses of this object for strings, integer, floating-point
numbers, blobs, and other representations.  Everything is passed around
inside the SQL language implemented by SQLite as "sqlite3_value" objects
so the underlying datatype does not really matter.  I have never found
rigid type enforcement to be helpful in languages like TCL and SQLite
that have a single "Value" superclass used to represent any data element.
Fossil makes extensive use of SQLite in its implementation.  There have
been many bugs in Fossil over its 14-years history, but I cannot recall
a single bug that might have been prevented by rigid type enforcement in
the SQLite.  C-language bugs might have been caught by better type enforcement
(which is why I wrote the supplemental source code scanners), but no
SQL bugs.

 Based on decades of experience, I reject the thesis that rigid
type enforcement helps prevent application bugs.  I will accept and
believe a slightly modified thesis:  Rigid type enforcement helps to
prevent applications bugs in languages that lack a single
top-level "Value" superclass.  But SQLite does have the
single "sqlite3_value" superclass, so that proverb does not apply.


c2: The Advantages Of Flexible Typing
c3: 4.3. Rigid type enforcement prevents data pollution
c5: flextypegood.html#rigid_type_enforcement_prevents_data_pollution
c4:
Some people contend that if you have rigorous constraints on the
schema, and especially strict enforcement of column datatypes, this
will help prevent incorrect data from being added to the database.
This is not true.  It is true that type enforcement might help prevent
egregiously incorrect data from getting into the system.
But type enforcement is no help in prevent subtly incorrect data
from being recorded.

 So, for example, rigid type enforcement can successfully prevent
the customer name (text) from being inserted into integer
Customer.creditScore column.  On the other hand, if that mistake occurs,
it is very easy to spot the problem and find all affected rows.
But type enforcement is no help in preventing a bug where the customer
family name and given name are reversed, since both are text fields.

 (Aside:  Decades ago, I worked on a team where there was a woman
named "Merritt Tracy".  "Merritt" was her given name and "Tracy" was
her family name.  She reported that she spend an inordinate amount of
time and energy trying to correct databases that had "Tracy" has her 
given name and "Merritt" as her family name.)

 By suppressing easy-to-detect errors and passing through only the
hard-to-detect errors, rigid type enforcement can actually make it more
difficult to find and fix bugs.  Data errors tend to cluster.  If you have
20 different data sources, most of the data errors will usually come
from just 2 or 3 of those sources.  The presence of egregious
errors (such as text in an integer column) is a convenient early warning
signal that something is amiss.  The source of the problem can be 
tracked quickly and extra scrutiny applied to the source of the
egregious errors, thus hopefully also fixing the subtle errors too.
When egregious errors are suppressed, you lose an important signal
that helps you to detect and fix the subtle errors.

 Data errors are inevitable.  They will happen regardless of how
much type checking is done.  Rigid type enforcement can catch only
a small subset of those cases - the most obvious cases.  It does
nothing to help find and fix the more subtle cases.  And, by suppressing
the signal of which data sources are problematic, it can sometimes
make the subtle errors more difficult to locate.


c2: The Advantages Of Flexible Typing
c3: 4.4. Other SQL database engines don't work this way
c5: flextypegood.html#other_sql_database_engines_don_t_work_this_way
c4:
Because SQLite is less restrictive and allows you to do more things,
SQL scripts that work on other database engines will also usually work
on SQLite, but script written initially for SQLite might not work
on more restrictive database engines.  This can cause problems when
developers user SQLite for prototyping and testing and then migrate their
application to a more restrictive SQL engine for deployment.  If the
application was (unintentionally) taking advantage of the flexible
typing available in SQLite, then it will fail when migrated.

 People use this problem to argue that SQLite should be more
restrictive about datatypes.  But you could just as easily turn
that argument around and say that other database engines should be
more flexible with regard to datatypes.  The application was working
correctly under SQLite, prior to be migrated, after all.  If rigid
type enforcement is really all that useful, why did it break an
application that was previously working?


c2: The Advantages Of Flexible Typing
c3: 5. If You Insist On Rigid Type Enforcement...
c5: flextypegood.html#if_you_insist_on_rigid_type_enforcement_
c4:
As of SQLite version 3.37.0 (2021-11-27), SQLite supports this
development style using STRICT tables.

If you find a real-world case where STRICT tables prevented or
would have prevented a bug in an application, please post a message to the
SQLite Forum so that we can add your story
to this document.


c2: The Advantages Of Flexible Typing
c3: 6. Embrace Freedom
c5: flextypegood.html#embrace_freedom
c4:
If flexible typing in an SQL database is a new concept to you,
I encourage you to give it a try.  It probably will not cause you
any problems and it might make your program simpler and easier to
write and maintain.  I think that even if you are skeptical at first,
if you will just give flexible typing a try, you will eventually
come to realize that it is a better approach and will start
encouraging other database vendors to support at least an ANY
datatype if not complete SQLite-style type flexibility.

Most of the time, flexible typing does not matter because a column
stores a single well-defined type.  But occasionally you will run
across situations where having a flexible type system makes the
solution to your problem cleaner and easier.


c2: Invalid UTF Policy
c3: 1. Garbage In, Garbage Out
c5: invalidutf.html#garbage_in_garbage_out
c4:
With regard to invalid UTF, SQLite follows a policy of
Garbage-In, Garbage-Out (GIGO). If you insert invalid UTF
into an SQLite database, then try to query that data, what you get back out
might not be exactly what you put in.  If you put garbage in, then you
may not complain if you get different garbage back out.


For the purposes of this discussion, "invalid UTF" can mean any of
the following circumstances:



Invalid surrogate pairs in UTF-16.


Invalid multi-byte sequences in UTF-8.


Using more bytes of UTF-8 than necessary to represent a single
code point.  (Example:  encoding 'A' as the two-byte sequence
0xc1, 0x01 instead of just a single 0x41 byte.)


NUL characters (U+0000) embedded in strings.


Invalid sequences of combining characters.


UTF-8 or UTF-16 bytes sequences that encode numbers that are not
defined Unicode characters.


c2: Invalid UTF Policy
c3: 1.1. Invalid UTF will never cause memory errors
c5: invalidutf.html#invalid_utf_will_never_cause_memory_errors
c4:
If you insert invalid UTF into an SQLite database, then SQLite makes
no guarantees about what text you might get back out.  But it does
promise that invalid UTF will never cause memory errors
(array overruns, reads or writes of uninitialized memory, etc), at
least for the built-in processing of SQLite.
In other words, invalid UTF will not cause SQLite to crash.


This promise only applies to the core SQLite components, not
application-provided extensions, of course.
If an application adds new application-defined SQL functions or
virtual tables or collating sequences or other extensions, and a
database contains invalid UTF, then invalid UTF might get passed
into those extensions.  If the invalid UTF causes one of those
extensions to crash, then that is a problem with the extension,
not with SQLite.


c2: Invalid UTF Policy
c3: 2. No enforcement of text formatting rules
c5: invalidutf.html#no_enforcement_of_text_formatting_rules
c4:
SQLite does not try to enforce UTF formatting rules.  You can
insert invalid UTF into a TEXT field and SQLite will not complain
about this.  It stores the invalid TEXT as best it can.  SQLite
sees its role in the world as a storage engine, not a text format
validation engine.


c2: Invalid UTF Policy
c3: 3. Best effort to preserve text
c5: invalidutf.html#best_effort_to_preserve_text
c4:
SQLite does not promise to always preserve invalid UTF, but it does
make an effort.  Generally speaking, if you insert invalid UTF into
SQLite, you will get the exact same byte sequence back out, as long
as you do not ask SQLite to transform the text in any way.


For example, if you insert some UTF-16LE with invalid surrogates into
a TEXT column of a table of a database that has PRAGMA encoding=UTF16LE,
then later query that column using sqlite3_column_text16(), you will 
probably get back the same exact invalid UTF-16.  But if you insert the
same invalid UTF-16LE content in a PRAGMA encoding=UTF8 database,
the content must be converted into UTF8 when it is stored, which could
cause irreversible changes to the content.  Or if you insert that
same invalid UTF-16LE content into a PRAGMA encoding=UTF16LE database
but then read it out using sqlite3_column_text(), then a UTF16 to
UTF8 conversion must occur during the read-out and that conversion might
introduce irreversible changes.


Or, suppose you are doing everything using UTF-8 (the most common case).
Invalid UTF-8 will normally pass through the database without any change
in its byte sequence.  However, if you try to transform the invalid
UTF-8 with SQL function like substr() or replace()
or if you try to do string matching with the LIKE operator, then
you might get unexpected results.


So, in other words, SQLite does not actively try to subvert your
invalid text.  But when you ask SQLite to make transformations of invalid
UTF, there are no guarantees that those transformations will be reversible
or even sensible.


c2: The Lemon LALR(1) Parser Generator
c3: 1. Overview
c5: lemon.html#overview
c4:
The SQL language parser for SQLite is generated using a code-generator
program called "Lemon".  The Lemon program reads a grammar of the input
language and emits C-code to implement a parser for that language.


c2: The Lemon LALR(1) Parser Generator
c3: 1.1. Lemon Source Files And Documentation
c5: lemon.html#lemon_source_files_and_documentation
c4:
Lemon does not have its own source repository.  Rather, Lemon consists
of a few files in the SQLite source tree:



     lemon.html &rarr;
     The original detailed usage documentation and programmers reference
     for Lemon.

     lemon.c &rarr; The source code
     for the utility program that reads a grammar file and generates 
     corresponding parser C-code.

     lempar.c &rarr; A template
     for the generated parser C-code.  The "lemon" utility program reads this
     template and inserts additional code in order to generate a parser.


c2: The Lemon LALR(1) Parser Generator
c3: 2. Advantages of Lemon
c5: lemon.html#advantages_of_lemon
c4:
Lemon generates an LALR(1) parser.  Its operation is similar to the
more familiar tools Yacc and
Bison, but Lemon adds important
improvements, including:



     The grammar syntax is less error prone - using symbolic names for
     semantic values rather that the "$1"-style positional notation
     of Yacc.

     In Lemon, the tokenizer calls the parser.  Yacc operates the other
     way around, with the parser calling the tokenizer.  The Lemon
     approach is reentrant and threadsafe, whereas Yacc uses global 
     variables and is therefore neither.  Reentrancy is especially
     important for SQLite since some SQL statements make recursive calls
     to the parser.  For example, when parsing a CREATE TABLE statement,
     SQLite invokes the parser recursively to generate an INSERT statement
     to make a new entry in the sqlite_schema table.

     Lemon has the concept of a non-terminal destructor that can be
     used to reclaim memory or other resources following a syntax error
     or other aborted parse.


c2: The Lemon LALR(1) Parser Generator
c3: 2.1. Use of Lemon Within SQLite
c5: lemon.html#use_of_lemon_within_sqlite
c4:
Lemon is used in two places in SQLite.

The primary use of Lemon is to create the SQL language parser.
A grammar file (parse.y) is
compiled by Lemon into parse.c and parse.h.  The parse.c file is
incorporated into the amalgamation without further modification.

Lemon is also used to generate the parser for the query pattern
expressions in the FTS5 extension.  In this case, the input grammar
file is fts5parse.y.


c2: The Lemon LALR(1) Parser Generator
c3: 2.2. Lemon Customizations Especially For SQLite
c5: lemon.html#lemon_customizations_especially_for_sqlite
c4:
One of the advantages of hosting code generator tools as part of
the project is that the tools can be optimized to serve specific needs of
the overall project.  Lemon has benefited from this effect. Over the years,
the Lemon parser generator has been extended and enhanced to provide
new capabilities and improved performance to SQLite.  A few of the
specific enhancements to Lemon that are specifically designed for use
by SQLite include:



Lemon has the concept of a "fallback" token.
The SQL language contains a large number of keywords and these keywords
have the potential to collide with identifier names.
Lemon has the ability to designate some keywords as being able to
"fallback" to an identifier.  If the keyword appears in the input token
stream in a context that would otherwise be a syntax error, the token
is automatically transformed into its fallback before the syntax error
is raised.  This feature allows the parser to be very forgiving of
reserved words used as identifiers, which is a problem that comes up
frequently in the SQL language.


In support of the 100% MC/DC testing goal for SQLite, 
the parser code generated by Lemon has no unreachable branches,
and contains extra (compile-time selected) instrumentation useful
for measuring test coverage.


Lemon supports conditional compilation of grammar file rules, so that
a different parser can be generated depending on compile-time options.


As a performance optimization, reduce actions in the Lemon input grammar
are allowed to contain comments of the form "/*A-overwrites-Z*/" to indicate
that the semantic value "A" on the right-hand side of the rule is allowed
to directly overwrite the semantic value "Z" on the left-hand side.
This simple optimization reduces the number of stack operations in the
push-down automaton used to parse the input grammar, and thus improve
performance of the parser.  It also makes the generated code a little smaller.


The parsing of SQL statements is a significant consumer of CPU cycles 
in any SQL database engine.  On-going efforts to optimize SQLite have caused
the developers to spend a lot of time tweaking Lemon to generate faster
parsers.  These efforts have benefited all users of the Lemon parser generator,
not just SQLite.  But if Lemon had been a separately maintained tool, it
would have been more difficult to make coordinated changes to both SQLite
and Lemon, and as a result not as much optimization would have been
accomplished.  Hence, the fact that the parser generator tool is included
in the source tree for SQLite has turned out to be a net benefit for both
the tool itself and for SQLite.


c2: The Lemon LALR(1) Parser Generator
c3: 3. History Of Lemon
c5: lemon.html#history_of_lemon
c4:
Lemon was originally written by D. Richard Hipp (also the creator of SQLite)
while he was in graduate school at Duke University between 1987 and 1992.
The original creation date of Lemon has been lost, but was probably sometime
around 1990.  Lemon generates an LALR(1) parser.  There was a companion 
LL(1) parser generator tool named "Lime", but the source code for Lime
has been lost.

The Lemon source code was originally written as separate source files,
and only later merged into a single "lemon.c" source file.

The author of Lemon and SQLite (Hipp) reports that his C programming
skills were greatly enhanced by studying John Ousterhout's original
source code to Tcl.  Hipp discovered and studied Tcl in 1993.  Lemon
was written before then, and SQLite afterwards.  There is a clear
difference in the coding styles of these two products, with SQLite seeming
to be cleaner, more readable, and easier to maintain.


c2: The Error And Warning Log
c3: Overview
c5: errlog.html#overview
c4:
SQLite can be configured to invoke a callback function containing
an error code and a terse error message whenever anomalies occur.
This mechanism is very helpful in tracking obscure problems that
occur rarely and in the field.  Application developers are encouraged
to take advantage of the error logging facility of SQLite in their
products, as it is very low CPU and memory cost but can be a
huge aid for debugging.


c2: The Error And Warning Log
c3: 1. Setting Up The Error Logging Callback
c5: errlog.html#setting_up_the_error_logging_callback
c4:
There can only be a single error logging callback per process.
The error logging callback is registered at start-time using C-code
similar to the following:


sqlite3_config(SQLITE_CONFIG_LOG, errorLogCallback, pData);


The error logger callback function might look something like this:


void errorLogCallback(void *pArg, int iErrCode, const char *zMsg){
  fprintf(stderr, "(%d) %s\n", iErrCode, zMsg);
}


The example above illustrates the signature of the error logger callback.
However, in an embedded application, one usually does not print
messages on stderr.  Instead, one might store the messages in a
preallocated circular buffer where they can be accessed when diagnostic
information is needed during debugging.  Or perhaps the messages can be
sent to Syslog.  Somehow, the
messages need to be stored where they are accessible to developers,
not displayed to end users.

Do not misunderstand: There is nothing technically wrong with displaying 
the error logger messages to end users.  The messages do not contain
sensitive or private information that must be protected from unauthorized
viewing.  Rather the messages are technical in nature and are not useful
or meaningful to the typical end user.  The messages coming from the
error logger are intended for database geeks.  Display them accordingly.


c2: The Error And Warning Log
c3: 2. Interface Details
c5: errlog.html#interface_details
c4:
The third argument to the sqlite3_config(SQLITE_CONFIG_LOG,...) 
interface (the "pData" argument in the example above) is a pointer to arbitrary
data.  SQLite passes this pointer through to the first argument of the
error logger callback.  The pointer can be used to pass application-specific 
setup or state information, if desired.  Or it can simply be a NULL 
pointer which is ignored by the callback.

The second argument to the error logger callback is an integer
extended error code.  The third argument to the error logger is the
text of the error message.  The error message text is stored in a fixed-length
stack buffer in the calling function and so will only be valid for the
duration of the error logger callback function.  The error logger should
make a copy of this message into persistent storage if retention of the
message is needed.

The error logger callback should be treated like a signal handler.
The application should save off or otherwise process the error, then return
as soon as possible.  No other SQLite APIs should be invoked, directly or
indirectly, from the error logger.  SQLite is not reentrant through
the error logger callback.  In particular, the error logger callback
is invoked when a memory allocation fails, so it is generally a bad idea
to try to allocate memory inside the error logger.  Do not even think
about trying to store the error message in another SQLite database.

Applications can use the sqlite3_log(E,F,..) API to send new messages
to the log, if desired, but this is discouraged.  The sqlite3_log()
interface is intended for use by extensions only, not by applications.


c2: The Error And Warning Log
c3: 3. Variety of Error Messages
c5: errlog.html#variety_of_error_messages
c4:
The error messages that might be sent to the error logger and their
exact format is subject to changes from one release to the next.  So
applications should not depend on any particular error message text formats or
error codes.  Things do not change capriciously, but they do sometimes
changes.

The following is a partial list of the kinds of messages that might
appear in the error logger callback.



Any time there is an error either compiling an SQL statement 
(using sqlite3_prepare_v2() or its siblings) or running an SQL
statement (using sqlite3_step()) that error is logged.



When a schema change occurs that requires a prepared statement to be reparsed
and reprepared, that event is logged with the error code SQLITE_SCHEMA.
The reparse and reprepare is normally automatic (assuming that
sqlite3_prepare_v2() has been used to prepare the statements originally,
which is recommended) and so these logging events are normally the only
way to know that reprepares are taking place.


SQLITE_NOTICE messages are logged whenever a database has to be recovered
because the previous writer crashed without completing its transaction.
The error code is SQLITE_NOTICE_RECOVER_ROLLBACK when recovering a
rollback journal and SQLITE_NOTICE_RECOVER_WAL when recovering a 
write-ahead log.



SQLITE_WARNING messages are logged when database files are renamed or
aliased in ways that can lead to database corruption.
(See 1 and 2 for
additional information.)



Out of memory (OOM) error conditions generate error logging events
with the SQLITE_NOMEM error code and a message that says how many bytes
of memory were requested by the failed allocation.


I/O errors in the OS-interface generate error logging events.
The message to these events gives the line number in the source code where
the error originated and the filename associated with the event when
there is a corresponding file. 

When database corruption is detected, an SQLITE_CORRUPT error
logger callback is invoked.  As with I/O errors, the error message text
contains the line number in the original source code where the error
was first detected.


An error logger callback is invoked on SQLITE_MISUSE errors.
This is useful in detecting application design issues when return codes
are not consistently checked in the application code.


SQLite strives to keep error logger traffic low and only send messages
to the error logger when there really is something wrong.  Applications
might further cull the error message traffic 
by deliberately ignoring certain classes of error
messages that they do not care about.  For example, an application that
makes frequent database schema changes might want to ignore all
SQLITE_SCHEMA errors.


c2: The Error And Warning Log
c3: 4. Summary
c5: errlog.html#summary
c4:
The use of the error logger callback is highly recommended.
The debugging information that the error logger provides has proven
very useful in tracking down obscure problems that occur with applications
after they get into the field.  The error logger callback has also 
proven useful in catching occasional errors that the application
misses because of inconsistent checking of API return codes.
Developers are encouraged to implement an error logger callback early
in the development cycle in order to spot unexpected behavior quickly,
and to leave the error logger callback turned on through deployment.
If the error logger never finds a problem, then no harm is done.  
But failure to set up an appropriate error logger might compromise
diagnostic capabilities later on.


c2: The SQLITE_DBPAGE Virtual Table
c3: 1. Overview
c5: dbpage.html#overview
c4:
The SQLITE_DBPAGE extension implements an eponymous-only virtual table that
provides direct access to the underlying database file by interacting
with the pager.  SQLITE_DBPAGE is capable of both reading and writing any
page of the database.  Because interaction is through the pager layer, all
changes are transactional.



Warning: writing to the SQLITE_DBPAGE virtual table can very easily
cause unrecoverably database corruption.  Do not allow untrusted components
to access the SQLITE_DBPAGE table.  Use appropriate care while using the
SQLITE_DBPAGE table.  Back up important data prior to experimenting with the
SQLITE_DBPAGE table.  Writes to the SQLITE_DBPAGE virtual table are
disabled when the SQLITE_DBCONFIG_DEFENSIVE flag is set.


The SQLITE_DBPAGE extension is included in the amalgamation though 
it is disabled
by default.  Use the SQLITE_ENABLE_DBPAGE_VTAB compile-time option to enable
the SQLITE_DBPAGE extension.  The SQLITE_DBPAGE extension makes use of
unpublished internal interfaces and is not run-time loadable.  The only way
to add SQLITE_DBPAGE to an application is to compile it in using the
SQLITE_ENABLE_DBPAGE_VTAB compile-time option.



The SQLITE_DBPAGE extension is enabled in default builds
of the command-line shell.


c2: The SQLITE_DBPAGE Virtual Table
c3: 2. Usage
c5: dbpage.html#usage
c4:
The SQLITE_DBPAGE virtual table read/write table that provides direct
access to the underlying disk file on a page-by-page basis.  The
virtual table appears to have a schema like this:

CREATE TABLE sqlite_dbpage(
  pgno INTEGER PRIMARY KEY,
  data BLOB
);



An SQLite database file is divided into pages.
The first page is 1, the second page is 2, and so forth.
There is no page 0.
Every page is the same size.
The size of every page is a power of 2 between 512 and 65536.
See the file format documentation for further details.


The SQLITE_DBPAGE table allows an application to view or replace the
raw binary content of each page of the database file.
No attempt is made to interpret the content of the page.
Content is returned byte-for-byte as it appears on disk.


The SQLITE_DBPAGE table has one row for each page in the database file.
SQLITE_DBPAGE allows pages to be read or to be overwritten.
However the size of the database file cannot be changed.  It is not
possible to change the number of rows in the SQLITE_DBPAGE table by
running DELETE or INSERT operations against that table.


c2: The SQLITE_DBPAGE Virtual Table
c3: 2.1. Using SQLITE_DBPAGE On ATTACH-ed Databases
c5: dbpage.html#using_sqlite_dbpage_on_attach_ed_databases
c4:
The SQLITE_DBPAGE table schema shown above is incomplete.  There is
a third hidden column named "schema" that determines which
ATTACH-ed database should be read or written.  Because
the "schema" column is hidden, it can be used as a parameter when
SQLITE_DBPAGE is invoked as a table-valued function.


For example, suppose an additional database is attached to the 
database connection using a statement like this:

ATTACH 'auxdata1.db' AS aux1;



Then to read the first page of that database file, one merely runs:

SELECT data FROM sqlite_dbpage('aux1') WHERE pgno=1;



If the "schema" is omitted, it defaults to the primary database
(usually called 'main', unless renamed using SQLITE_DBCONFIG_MAINDBNAME).
Hence, the following two queries are normally equivalent:

SELECT data FROM sqlite_dbpage('main') WHERE pgno=1;
SELECT data FROM sqlite_dbpage WHERE pgno=1;



The SQLITE_DBPAGE table can participate in a join just like any other
table.  Hence, to see the content of the first page to all connected
database files, one might run a statement like this:

SELECT dbpage.data, dblist.name
  FROM pragma_database_list AS dblist
  JOIN sqlite_dbpage(dblist.name) AS dbpage
 WHERE dbpage.pgno=1;


c2: Row Values
c3: 1. Definitions
c5: rowvalue.html#definitions
c4:
A "value" is a single number, string, BLOB or NULL.
Sometimes the qualified name "scalar value" is used to emphasize that
only a single quantity is involved.

A "row value" is an ordered list of two or more scalar values.
In other words, a "row value" is a vector or tuple.

The "size" of a row value is the number of scalar values the row value contains.
The size of a row value is always at least 2.
A row value with a single column is just a scalar value.
A row value with no columns is a syntax error.


c2: Row Values
c3: 2. Syntax
c5: rowvalue.html#syntax
c4:
SQLite allows row values to be expressed in two ways:

A parenthesized, comma-separated list of scalar values.
A subquery expression with two or more result columns.


SQLite can use row values in two contexts:

Two row values of the same size 
can be compared using operators &lt;, &lt;=, &gt;, &gt;=,
=, &lt;&gt;, IS, IS NOT, IN, NOT IN, BETWEEN, or CASE.
In an UPDATE statement, a list of column names can be set to a row value of
the same size.


The syntax for row values and the circumstances in which row values
can be used are illustrated in examples below.


c2: Row Values
c3: 2.1. Row Value Comparisons
c5: rowvalue.html#row_value_comparisons
c4:
Two row values are compared by looking at the constituent scalar
values from left to right.
A NULL means of "unknown".  
The overall result of comparison is NULL if it is possible to make the
result either true or false by substituting alternative values in place
of the constituent NULLs.
The following query demonstrates some row value comparisons:

SELECT
  (1,2,3) = (1,2,3),          -- 1
  (1,2,3) = (1,NULL,3),       -- NULL
  (1,2,3) = (1,NULL,4),       -- 0
  (1,2,3) &lt; (2,3,4),          -- 1
  (1,2,3) &lt; (1,2,4),          -- 1
  (1,2,3) &lt; (1,3,NULL),       -- 1
  (1,2,3) &lt; (1,2,NULL),       -- NULL
  (1,3,5) &lt; (1,2,NULL),       -- 0
  (1,2,NULL) IS (1,2,NULL);   -- 1


The result of "(1,2,3)=(1,NULL,3)" is NULL because the result might be
true if we replaced NULL&rarr;2 or false if we replaced NULL&rarr;9.
The result of "(1,2,3)=(1,NULL,4)" is not NULL because there is no
substitutions of the constituent NULL that will make the expression true,
since 3 will never equal 4 in the third column.

Any of the row values in the previous example could be replace by a
subquery that returns three columns and the same answer would result.
For example:

CREATE TABLE t1(a,b,c);
INSERT INTO t1(a,b,c) VALUES(1,2,3);
SELECT (1,2,3)=(SELECT * FROM t1); -- 1


c2: Row Values
c3: 2.2. Row Value IN Operators
c5: rowvalue.html#row_value_in_operators
c4:
For a row-value IN operator, the left-hand side (hereafter "LHS") can be either
a parenthesized list of values or a subquery with multiple columns.  But the
right-hand side (hereafter "RHS") must be a subquery expression.

CREATE TABLE t2(x,y,z);
INSERT INTO t2(x,y,z) VALUES(1,2,3),(2,3,4),(1,NULL,5);
SELECT
   (1,2,3) IN (SELECT * FROM t2),  -- 1
   (7,8,9) IN (SELECT * FROM t2),  -- 0
   (1,3,5) IN (SELECT * FROM t2);  -- NULL


c2: Row Values
c3: 2.3. Row Values In UPDATE Statements
c5: rowvalue.html#row_values_in_update_statements
c4:
Row values can also be used in the SET clause of an UPDATE statement.
The LHS must be a list of column names.  The RHS can be any row value.
For example:

UPDATE tab3 
   SET (a,b,c) = (SELECT x,y,z
                    FROM tab4
                   WHERE tab4.w=tab3.d)
 WHERE tab3.e BETWEEN 55 AND 66;


c2: Row Values
c3: 3.1. Scrolling Window Queries
c5: rowvalue.html#scrolling_window_queries
c4:
Suppose an application wants to display a list of contacts
in alphabetical order by lastname, firstname, in a scrolling window
that can only show 7 contacts at a time.  Initialize the scrolling
window to the first 7 entries is easy:

SELECT * FROM contacts
 ORDER BY lastname, firstname
 LIMIT 7;


When the user scrolls down, the application needs to find the
second set of 7 entries.  One way to do this is to use the OFFSET clause:

SELECT * FROM contacts
 ORDER BY lastname, firstname
 LIMIT 7 OFFSET 7;


OFFSET gives the correct answer.  However, OFFSET requires time
proportional to the offset value.  What really happens
with "LIMIT x OFFSET y" is that SQLite computes the query as
"LIMIT x+y" and discards the first y values without returning them
to the application.  So as the window scrolls down toward
the bottom of a long list, and the y value becomes larger and larger,
successive offset computations take more and more time.

A more efficient approach is to remember the last entry currently
displayed and then use a row value comparison in the WHERE
clause:

SELECT * FROM contacts
 WHERE (lastname,firstname) &gt; (?1,?2)
 ORDER BY lastname, firstname
 LIMIT 7;


If the lastname and firstname on the bottom row of the previous
screen are bound to ?1 and ?2, then the query above computes the next
7 rows.  And, assuming there is an appropriate index, it does so
very efficiently &mdash; much more efficiently than OFFSET.


c2: Row Values
c3: 3.2. Comparison of dates stored as separate fields
c5: rowvalue.html#comparison_of_dates_stored_as_separate_fields
c4:
The usual way of storing a date in a database table is as a single
field, as either a unix timestamp, a julian day number, or an ISO-8601
dates string.  But some application store dates as three separate
fields for the year, month, and day.  

CREATE TABLE info(
  year INT,          -- 4 digit year
  month INT,         -- 1 through 12
  day INT,           -- 1 through 31
  other_stuff BLOB   -- blah blah blah
);


When dates are stored this way, row value comparisons provide a
convenient way to compare dates:

SELECT * FROM info
 WHERE (year,month,day) BETWEEN (2015,9,12) AND (2016,9,12);


c2: Row Values
c3: 3.3. Search against multi-column keys
c5: rowvalue.html#search_against_multi_column_keys
c4:
Suppose we want to know the order number, product number, and quantity
for any item in which the product number and quantity match the product
number and quantity of any item in order number 365:

SELECT ordid, prodid, qty
  FROM item
 WHERE (prodid, qty) IN (SELECT prodid, qty
                           FROM item
                          WHERE ordid = 365);


The query above could be rewritten as a join and without the use
of row values:

SELECT t1.ordid, t1.prodid, t1.qty
  FROM item AS t1, item AS t2
 WHERE t1.prodid=t2.prodid
   AND t1.qty=t2.qty
   AND t2.ordid=365;


Because the same query could be written without the use of row values,
row values do not provide new capabilities.  However, many developers say
that the row value format is easier to read, write, and debug.

Even in the JOIN form, the query can be made clearer through the use of
row values:

SELECT t1.ordid, t1.prodid, t1.qty
  FROM item AS t1, item AS t2
 WHERE (t1.prodid,t1.qty) = (t2.prodid,t2.qty)
   AND t2.ordid=365;


This later query generates exactly the same bytecode as the previous
scalar formulation, but using syntax that it cleaner and
easier to read.


c2: Row Values
c3: 3.4. Update multiple columns of a table based on a query
c5: rowvalue.html#update_multiple_columns_of_a_table_based_on_a_query
c4:
The row-value notation is useful for updating two or more columns
of a table from the result of a single query.
An example of this is in the full-text search feature of the
Fossil version control system.

In the Fossil full-text search system,
documents that participate in the full-text search (wiki pages, tickets,
check-ins, documentation files, etc) are tracked by a table called
"ftsdocs" (full text search documents).
As new documents are added to the repository, they are not indexed right
away.  Indexing is deferred until there is a search request.  The
ftsdocs table contains an "idxed" field which is true if the document
has been indexed and false if not.

When a search request occurs and pending documents are indexed for the
first time, the ftsdocs table must be updated by setting the idxed column
to true and also filling in several other columns with information pertinent
to the search.  That other information is obtained from a join.  The
query is this:

UPDATE ftsdocs SET
  idxed=1,
  name=NULL,
  (label,url,mtime) = 
      (SELECT printf('Check-in &#91;%%.16s&#93; on %%s',blob.uuid,
                     datetime(event.mtime)),
              printf('/timeline?y=ci&amp;c=%%.20s',blob.uuid),
              event.mtime
         FROM event, blob
        WHERE event.objid=ftsdocs.rid
          AND blob.rid=ftsdocs.rid)
WHERE ftsdocs.type='c' AND NOT ftsdocs.idxed


(See the 
source code
for further detail.  Other examples
here and
here.)

Five out of nine columns in the ftsdocs table are updated.  Two of
the modified columns, "idxed" and "name", can be updated independently of
the query.  But the three columns "label", "url", and "mtime" all require
a join query against the "event" and "blob" tables.  Without row values,
the equivalent UPDATE would require that the join be repeated three times, 
once for each column to be updated.


c2: Row Values
c3: 3.5. Clarity of presentation
c5: rowvalue.html#clarity_of_presentation
c4:
Sometimes the use of row values just makes the SQL easier to read
and write.  Consider the following two UPDATE statements:

UPDATE tab1 SET (a,b)=(b,a);
UPDATE tab1 SET a=b, b=a;


Both UPDATE statements do exactly the same thing.  (They generate
identical bytecode.)  But the first form, the row value form, seems
to make it clearer that the intent of the statement is to swap the
values in columns A and B.

Or consider these identical queries:

SELECT * FROM tab1 WHERE a=?1 AND b=?2;
SELECT * FROM tab1 WHERE (a,b)=(?1,?2);


Once again, the SQL statements generate identical bytecode and thus
do exactly the same job in exactly the same way.  But the second form
is made easier for humans to read by grouping the query parameters together
into a single row value rather than scattering them across the WHERE
clause.


c2: Row Values
c3: 4. Backwards Compatibility
c5: rowvalue.html#backwards_compatibility
c4:
Row values were added to SQLite
version 3.15.0 (2016-10-14).  Attempts to use row values in
prior versions of SQLite will generate syntax errors.


c2: The UINT Collating Sequence
c3: 1. Overview
c5: uintcseq.html#overview
c4:
The UINT collating sequences is a loadable extension for
SQLite that implements a new collating sequence that compares text
containing unsigned integers in numeric order.

 The UINT collating sequence is not a standard part of SQLite.
It must be loaded as a separate extension.  The source code to
UINT is in the uint.c source file
in the ext/misc/ folder of the
SQLite source tree.

 The UINT collating sequence is not included in standard builds of
the SQLite library, but it is loaded by default in the CLI.  This
is typical of the CLI which loads various extensions above and beyond
what are available in the standard SQLite library.

 The UINT collating sequence works just like the default
BINARY collating sequence for text, except that embedded strings
of digits compare in numeric order.



Leading zeros are handled properly, in the sense that
they do not mess of the maginitude comparison of embedded
strings of digits.  "x00123y" is equal to "x123y".


Only unsigned integers are recognized.  Plus and minus
signs are ignored.  Decimal points and exponential notation
are ignored.


Embedded integers can be of arbitrary length.  Comparison
is not limited to integers that can be expressed as a
64-bit machine integer.


c2: The UINT Collating Sequence
c3: 2. Example:
c5: uintcseq.html#example_
c4:
COLLATE binaryCOLLATE uint

'0000123457'
'123456'
'abc0000000010xyz'
'abc0010xyy'
'abc10xzz'
'abc674xyz'
'abc87xyz'
'abc9xyz'

'123456'
'0000123457'
'abc9xyz'
'abc0010xyy'
'abc0000000010xyz'
'abc10xzz'
'abc87xyz'
'abc674xyz'


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 1. Overview
c5: quirks.html#overview
c4:
The SQL language is a "standard".
Even so, no two SQL database engines work exactly alike.
Every SQL implementation has it own peculiarities and oddities,
and SQLite is no exception to this rule.


This document strives to highlight the principal differences
between SQLite and other SQL implementations, as an aid to developers
that are porting to or from SQLite or who are trying to build a
system that works across multiple database engines.


If you are an SQLite user whose has stumbled over some quirk of
SQLite that is not mentioned here, please send us an email so that
we can document the problem.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 2. SQLite Is Embedded, Not Client-Server
c5: quirks.html#sqlite_is_embedded_not_client_server
c4:
Whenever comparing SQLite to other SQL database engines like
SQL Server, PostgreSQL, MySQL, or Oracle, it is important first of all
to realize that SQLite is not intended as a replacement or competitor to
any of those systems.  SQLite is serverless.  There is no separate
server process that manages the database.  An application interacts
with the database engine using function calls, not by sending messages
to a separate process or thread.


The fact that SQLite is embedded and serverless instead of being
client/server is a feature, not a bug.


Client/server databases like MySQL, PostgreSQL, SQL Server, Oracle, and
others are an important component of modern systems.  
These systems solve an important problem.  
But SQLite solves a different problem.
Both SQLite and client/server databases have their role.
Developers who are comparing SQLite against other SQL database engines need
to clearly understand this distinction.


See the Appropriate Uses For SQLite document for additional information.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 3. Flexible Typing
c5: quirks.html#flexible_typing
c4:
SQLite is flexible with regard to datatypes.  Datatypes are advisory
rather than mandatory.


Some commentators say that SQLite is "weakly typed" and that other
SQL databases are "strongly typed".  We consider these terms to be
inaccurate and even pejorative.  We prefer to say that SQLite is 
"flexibly typed" and that other SQL database engines are
"rigidly typed".


See the Datatypes in SQLite document for a detailed
discussion of the type system in SQLite.


The key point is that SQLite is very forgiving of the type of data that
you put into the database.  For example, if a column has a datatype of
"INTEGER" and the application inserts a text string into that column,
SQLite will first try to convert the text string into an integer, just like
every other SQL database engine.  Thus, if one inserts '1234' into
an INTEGER column, that value is converted into an integer 1234 and stored.
But, if you insert a non-numeric string like 'wxyz' into an INTEGER
column, unlike other SQL databases, SQLite does not throw an error.  Instead,
SQLite stores the actual string value in the column.


Similarly, SQLite allows you to store a 2000-character string into a
column of type VARCHAR(50).  Other SQL implementations would either throw
an error or truncate the string.  SQLite stores the entire 2000-character
string with no loss of information and without complaint.


Where this ends up causing problems is when developers do some initial
coding work using SQLite and get their application working, but then try
to convert to another database like PostgreSQL or SQL Server for deployment.
If the application is initially taking advantage of SQLite's flexible typing,
then it will fail when moved to another database that uses a more rigid
and unforgiving type enforcement policy.


Flexible typing is a feature of SQLite, not a bug.  Flexible typing
is about freedom.
Nevertheless, we recognize that this feature does sometimes cause
confusion for developers who are accustomed to working with
other databases that are more strict with regard to data type rules.
In retrospect, perhaps it would have been less confusing if SQLite had merely
implemented an ANY datatype so that developers could explicitly state
when they wanted to use flexible typing, rather than making flexible
typing the default.
As an accommodation for those who expect rigid typing, SQLite version 3.37.0
(2021-11-27) introduced the option of STRICT tables.
These either impose
the mandatory datatype constraints found in other SQL database engines,
or allow the explicit ANY datatype to retain SQLite's flexible typing.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 3.1. No Separate BOOLEAN Datatype
c5: quirks.html#no_separate_boolean_datatype
c4:
Unlike most other SQL implementations,
SQLite does not have a separate BOOLEAN data type.
Instead, TRUE and FALSE are (normally) represented as integers 1 and 0,
respectively.
This does not seem to cause many problems, as we seldom get complaints
about it.  But it is important to recognize.


Beginning with SQLite version 3.23.0 (2018-04-02), SQLite also
recognizes TRUE and FALSE keywords as aliases for integer values 1 and 0,
respectively.
This provides better compatibility with other SQL implementations.
But for backwards compatibility, if there are columns named TRUE or
FALSE, then the keywords are treated as identifiers referencing those
columns, rather than BOOLEAN literals.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 3.2. No Separate DATETIME Datatype
c5: quirks.html#no_separate_datetime_datatype
c4:
SQLite has no DATETIME datatype.
Instead, dates and times can be stored in any of these ways:


 As a TEXT string in the ISO-8601 format.  Example: '2018-04-02 12:13:46'.
 As an INTEGER number of seconds since 1970 (also known as "unix time").
 As a REAL value that is the fractional 
     Julian day number.



The built-in date and time functions of SQLite understand date/times in
all of the formats above, and can freely change between them.
Which format you use, is entirely up to your application.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 3.3. The datatype is optional
c5: quirks.html#the_datatype_is_optional
c4:
Because SQLite is flexible and forgiving with regard to datatypes,
table columns can be created that have no specified datatype at all.
For example:
CREATE TABLE t1(a,b,c,d);

The table "t1" has four columns "a", "b", "c", and "d" that have
no particular datatype assigned.  You can store anything you want in
any of those columns.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 4. Foreign Key Enforcement Is Off By Default
c5: quirks.html#foreign_key_enforcement_is_off_by_default
c4:
SQLite has parsed foreign key constraints for time out of mind,
but added the ability to actually enforce those constraints much later,
with version 3.6.19 (2009-10-14).  By the time foreign key constraint
enforcement was added, there were already countless millions of databases
in circulation that contained foreign key constraints, some of which
were not correct.  To avoid breaking those legacy databases, foreign key
constraint enforcement is turned off by default in SQLite.

Applications can activate foreign key enforcement at run-time using
the PRAGMA foreign_keys statement.  Or, foreign key enforcement can
be activated at compile-time using the
-DSQLITE_DEFAULT_FOREIGN_KEYS=1 compile-time option.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 5. PRIMARY KEYs Can Sometimes Contain NULLs
c5: quirks.html#primary_keys_can_sometimes_contain_nulls
c4:
Usually (the exceptions are INTEGER PRIMARY KEY tables and
WITHOUT ROWID tables) a PRIMARY KEY in an SQLite table is really
the same as a UNIQUE constraint.  Due to an historical oversight,
the column values of such a PRIMARY KEY are allowed to be NULL.
This is a bug, but by the time the problem was discovered there
where so many databases in circulation that depended on the bug that
the decision was made to support the buggy behavior moving forward.

The value of an INTEGER PRIMARY KEY column must always be a 
non-NULL integer.  The PRIMARY KEY columns of a WITHOUT ROWID
table are also required to be non-NULL.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 6. Aggregate Queries Can Contain Non-Aggregate Result Columns That Are Not In The GROUP BY Clause
c4:
In most SQL implementations, output columns of an aggregate query
may only reference aggregate functions or columns named in the
GROUP BY clause.  It does not make good sense to reference an ordinary
column in an aggregate query because each output row might be composed
from two or more rows in the input table(s).


SQLite does not enforce this restriction.
The output columns from an aggregate query can be arbitrary expressions
that include columns not found in GROUP BY clause.
This feature has two uses:



With SQLite (but not any other SQL implementation that we know of) if
an aggregate query contains a single min() or max() function, then the
values of columns used in the output are taken from the row where
the min() or max() value was achieved.  If two or more rows have the
same min() or max() value, then the columns values will be chosen arbitrarily
from one of those rows.

For example to find the highest paid employee:
SELECT max(salary), first_name, last_name FROM employee;


In the query above, the values for the first_name and last_name columns
will correspond to the row that satisfied the max(salary) condition.


If a query contains no aggregate functions at all, then a GROUP BY
clause can be added as a substitute for the DISTINCT ON clause.  In other words,
output rows are filtered so that only one row is shown for each distinct
set of values in the GROUP BY clause.  If two or more output rows would
have otherwise had the same set of values for the GROUP BY columns, then
one of the rows is chosen arbitrarily.  (SQLite supports DISTINCT but not
DISTINCT ON, whose functionality is provided instead by GROUP BY.)

c5:
quirks.html#aggregate_queries_can_contain_non_aggregate_result_columns_that_are_not_in_the_group_by_clause


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 7. Does Not Do Full Unicode Case Folding By Default
c5: quirks.html#does_not_do_full_unicode_case_folding_by_default
c4:
SQLite does not know about the upper-case/lower-case distinction
for all unicode characters.  SQL functions like
upper() and lower() only work on ASCII characters.  There are two
reasons for this:

 Though stable now, when SQLite was first designed, the rules for
     unicode case folding were still in flux.  That means that the
     behavior might have changed with each new unicode release, disrupting
     applications and corrupting indexes in the process.
 The tables necessary to do full and proper unicode case folding are
     larger than the whole SQLite library.


Full unicode case folding is supported in SQLite if it is compiled
with the -DSQLITE_ENABLE_ICU option and linked against the
International Components for Unicode
library.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 8. Double-quoted String Literals Are Accepted
c5: quirks.html#double_quoted_string_literals_are_accepted
c4:
The SQL standard requires double-quotes around identifiers
and single-quotes around string literals.  For example:

 "this is a legal SQL column name"
 'this is an SQL string literal'


SQLite accepts both of the above.  But, in an effort to be compatible
with MySQL 3.x (which was one of the most widely used RDBMSes
when SQLite was first being designed) SQLite will also interpret
a double-quotes string as
string literal if it does not match any valid identifier.

This misfeature means that a misspelled double-quoted
identifier will be interpreted as a string literal, rather than generating
an error.
It also lures developers who are new to the SQL language into the
bad habit of using double-quoted string literals when they
really need to learn to use the correct single-quoted string literal form.

In hindsight, we should not have tried to make SQLite accept MySQL 3.x
syntax, and should have never allowed double-quoted string literals.
However, there are countless applications that make use of
double-quoted string literals and so we continue to support
that capability to avoid breaking legacy.

As of SQLite 3.27.0 (2019-02-07) the use of a double-quoted
string literal causes a warning message to be sent to the error log.

As of SQLite 3.29.0 (2019-07-10) the use of double-quoted
string literals can be disabled at run-time using the
SQLITE_DBCONFIG_DQS_DDL and SQLITE_DBCONFIG_DQS_DML actions
to sqlite3_db_config().  The default settings can be altered
at compile-time using the -DSQLITE_DQS=N compile-time
option.  Application developers are encouraged to compile using
-DSQLITE_DQS=0 in order to disable the double-quoted string literal
misfeature by default.  If that is not possible, then disable
double-quoted string literals for individual database connections
using C-code like this:

sqlite3_db_config(db, SQLITE_DBCONFIG_DQS_DDL, 0, (void*)0);
sqlite3_db_config(db, SQLITE_DBCONFIG_DQS_DML, 0, (void*)0);

Or, if double-quoted string literals are disabled by default, but need
to be selectively enabled for some historical database connections,
that can be done using the same C-code as shown above except with the
third parameter changed from 0 to 1.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 9. Keywords Can Often Be Used As Identifiers
c5: quirks.html#keywords_can_often_be_used_as_identifiers
c4:
The SQL language is rich in keywords.
Most SQL implementations do not allow keywords to be used as identifiers
(names of tables or columns) unless they are enclosed in double-quotes.
But SQLite is more flexible.  Many keywords can be used as identifiers without
needing to be quoted, as long as those keywords are used in a context where
it is clear that they are intended to be an identifier.

For example, the following statement is valid in SQLite:
CREATE TABLE union(true INT, with BOOLEAN);


The same SQL statement will fail on every other SQL implementation that
we know of due to the use of keywords "union", "true", and "with" as
identifiers.

The ability to use keywords as identifiers promotes backwards compatibility.
As new keywords are added, legacy schemas that just happen to use those
keywords as table or column names continue to work.  However, the ability
to use a keyword as an identifier sometimes leads to surprising outcomes.
For example:
CREATE TRIGGER AFTER INSERT ON tableX BEGIN
  INSERT INTO tableY(b) VALUES(new.a);
END;

The trigger created by the previous statement is named "AFTER"
and it is a "BEFORE" trigger.  The "AFTER" token is used as an identifier
instead of as a keyword, as that is the only way to parse the statement.
Another example:
CREATE TABLE tableZ(INTEGER PRIMARY KEY);

The tableZ table has a single column named "INTEGER".  That column
has no datatype specified, but it is the PRIMARY KEY.
The column is not the INTEGER PRIMARY KEY for the table because
it has no datatype.  The "INTEGER" token is used as an identifier for the
column name, not as a datatype keyword.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 10. Dubious SQL Is Allowed Without Any Error Or Warning
c5: quirks.html#dubious_sql_is_allowed_without_any_error_or_warning
c4:
The original implementation of SQLite sought to follow
Postel's Law which
states in part "Be liberal in what you accept".
This used to be considered good design - that a system would accept
dodgy inputs and try to do the best it could without complaining too much.
But lately, people have come to realize that it is sometimes better to
be strict in what you accept, so as to more easily find errors in the
input.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 11. AUTOINCREMENT Does Not Work The Same As MySQL
c5: quirks.html#autoincrement_does_not_work_the_same_as_mysql
c4:
The AUTOINCREMENT feature in SQLite works differently than
it does in MySQL.  This often causes confusion for people who
initially learned SQL on MySQL and then start using SQLite, and
expect the two systems to work identically.

See the SQLite AUTOINCREMENT documentation for
detailed instructions on what AUTOINCREMENT does and does not do
in SQLite.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 12. NUL Characters Are Allowed In Text Strings
c5: quirks.html#nul_characters_are_allowed_in_text_strings
c4:
NUL characters (ASCII code 0x00 and Unicode \u0000) may appear in
the middle of strings in SQLite.  This can lead to unexpected behavior.
See the "NUL characters in strings" document for further information.


c2: The Spellfix1 Virtual Table
c3: 1. Overview
c5: spellfix1.html#overview
c4:
This spellfix1 virtual table can be used to search
a large vocabulary for close matches.  For example, spellfix1
can be used to suggest corrections to misspelled words.  Or,
it could be used with FTS4 to do full-text search using potentially
misspelled words.

The implementation for the spellfix1 virtual table is held in the
SQLite source tree in the miscellaneous extensions folder and in
particular in the file 
ext/misc/spellfix1.c.
The spellfix1 virtual table is not included in the SQLite amalgamation
and is not a part of any standard SQLite build.  It is a loadable extension.

Once the spellfix1 extension is loaded, an instance of the spellfix1 
virtual table is created like this:


CREATE VIRTUAL TABLE demo USING spellfix1;


The "spellfix1" term is the name of the spellfix module and must be 
entered as shown.  The "demo" term is the
name of the virtual table you will be creating and can be altered
to suit the needs of your application.  The virtual table is initially
empty.  In order for the virtual table to be useful, you will need to
populate it with your vocabulary.  Suppose you
have a list of words in a table named "big_vocabulary".  Then do this:


INSERT INTO demo(word) SELECT word FROM big_vocabulary;


If you intend to use this virtual table in cooperation with an FTS4
table (for spelling correction of search terms) then you might extract
the vocabulary using an fts4aux table:


INSERT INTO demo(word) SELECT term FROM search_aux WHERE col='*';


You can also provide the virtual table with a "rank" for each word.
The "rank" is an estimate of how common the word is.  Larger numbers
mean the word is more common.  If you omit the rank when populating
the table, then a rank of 1 is assumed.  But if you have rank 
information, you can supply it and the virtual table will show a
slight preference for selecting more commonly used terms.  To
populate the rank from an fts4aux table "search_aux" do something
like this:


INSERT INTO demo(word,rank)
   SELECT term, documents FROM search_aux WHERE col='*';


To query the virtual table, include a MATCH operator in the WHERE
clause.  For example:


SELECT word FROM demo WHERE word MATCH 'kennasaw';


Using a dataset of American place names (derived from
http://geonames.usgs.gov/domestic/download_data.htm) the query above
returns 20 results beginning with:


kennesaw
kenosha
kenesaw
kenaga
keanak


If you append the character '*' to the end of the pattern, then
a prefix search is performed.  For example:


SELECT word FROM demo WHERE word MATCH 'kennes*';


Yields 20 results beginning with:


kennesaw
kennestone
kenneson
kenneys
keanes
keenes


c2: The Spellfix1 Virtual Table
c3: 2. Search Refinements
c5: spellfix1.html#search_refinements
c4:
By default, the spellfix1 table returns no more than 20 results.
(It might return less than 20 if there were fewer good matches.)
You can change the upper bound on the number of returned rows by
adding a "top=N" term to the WHERE clause of your query, where N
is the new maximum.  For example, to see the 5 best matches:


SELECT word FROM demo WHERE word MATCH 'kennes*' AND top=5;


Each entry in the spellfix1 virtual table is associated with
a particular language, identified by the integer "langid" column.
The default langid is 0 and if no other actions are taken, the
entire vocabulary is a part of the 0 language.  But if your application
needs to operate in multiple languages, then you can specify different
vocabulary items for each language by specifying the langid field
when populating the table.  For example:


INSERT INTO demo(word,langid) SELECT word, 0 FROM en_vocabulary;
INSERT INTO demo(word,langid) SELECT word, 1 FROM de_vocabulary;
INSERT INTO demo(word,langid) SELECT word, 2 FROM fr_vocabulary;
INSERT INTO demo(word,langid) SELECT word, 3 FROM ru_vocabulary;
INSERT INTO demo(word,langid) SELECT word, 4 FROM cn_vocabulary;


After the virtual table has been populated with items from multiple
languages, specify the language of interest using a "langid=N" term
in the WHERE clause of the query:


SELECT word FROM demo WHERE word MATCH 'hildes*' AND langid=1;


Note that if you do not include the "langid=N" term in the WHERE clause,
the search will be against language 0 (English in the example above.)
All spellfix1 searches are against a single language id.  There is no
way to search all languages at once.


c2: The Spellfix1 Virtual Table
c3: 3. Virtual Table Details
c5: spellfix1.html#virtual_table_details
c4:
Each row in the spellfix1 virtual table has a unique rowid 
with seven columns plus five extra hidden columns.
The columns are as follows:


rowid
A unique integer number associated with each
vocabulary item in the table.  This can be used
as a foreign key on other tables in the database.

word
The text of the word that matches the pattern.
Both word and pattern can contain unicode characters
and can be mixed case.

rank
This is the rank of the word, as specified in the
original INSERT statement.


distance
This is an edit distance or Levenshtein distance going
from the pattern to the word.

langid
This is the language-id of the word.  All queries are
against a single language-id, which defaults to 0.
For any given query this value is the same on all rows.

score
The score is a combination of rank and distance.  The
idea is that a lower score is better.  The virtual table
attempts to find words with the lowest score and 
by default (unless overridden by ORDER BY) returns
results in order of increasing score.

matchlen
In a prefix search, the matchlen is the number of characters in
the string that match against the prefix.  For a non-prefix search,
this is the same as length(word).

phonehash
This column shows the phonetic hash prefix that was used to restrict
the search.  For any given query, this column should be the same for
every row.  This information is available for diagnostic purposes and
is not normally considered useful in real applications.

top
(HIDDEN)  For any query, this value is the same on all
rows.  It is an integer which is the maximum number of
rows that will be output.  The actually number of rows
output might be less than this number, but it will never
be greater.  The default value for top is 20, but that
can be changed for each query by including a term of
the form "top=N" in the WHERE clause of the query.

scope
(HIDDEN)  For any query, this value is the same on all
rows.  The scope is a measure of how widely the virtual
table looks for matching words.  Smaller values of
scope cause a broader search.  The scope is normally
chosen automatically and is capped at 4.  Applications
can change the scope by including a term of the form
"scope=N" in the WHERE clause of the query.  Increasing
the scope will make the query run faster, but will reduce
the possible corrections.

srchcnt
(HIDDEN)  For any query, this value is the same on all
rows.  This value is an integer which is the number of
words examined using the edit-distance algorithm to
find the top matches that are ultimately displayed.  This
value is for diagnostic use only.

soundslike
(HIDDEN)  When inserting vocabulary entries, this field
can be set to a spelling that matches what the word
sounds like.  See the DEALING WITH UNUSUAL AND DIFFICULT
SPELLINGS section below for details.

command
(HIDDEN)  The value of the "command" column is always NULL.  However,
applications can insert special strings into the "command" column in order
to provoke certain behaviors in the spellfix1 virtual table.
For example, inserting the string 'reset' into the "command" column
will cause the virtual table to reread its edit distance weights
(if there are any).


c2: The Spellfix1 Virtual Table
c3: 4. Algorithm
c5: spellfix1.html#algorithm
c4:
The spellfix1 virtual table creates a single
shadow table named "%_vocab" (where the % is replaced by the name of
the virtual table; Ex: "demo_vocab" for the "demo" virtual table).  
the shadow table contains the following columns:


id
The unique id (INTEGER PRIMARY KEY)

rank
The rank of word.

langid
The language id for this entry.

word
The original UTF8 text of the vocabulary word

k1
The word transliterated into lower-case ASCII.  
There is a standard table of mappings from non-ASCII
characters into ASCII.  Examples: "æ" -> "ae",
"þ" -> "th", "ß" -> "ss", "á" -> "a", ...  The
accessory function spellfix1_translit(X) will do
the non-ASCII to ASCII mapping.  The built-in lower(X)
function will convert to lower-case.  Thus:
k1 = lower(spellfix1_translit(word)).

If the word is already all lower-case ASCII, then the k1 column
will contain a NULL.  This reduces the storage requirements for
the %_vocab table and helps spellfix to run a little faster.
Therefore, it is advantageous to populate as much of the spellfix
table as possible using lower-case ASCII vocabulary.

k2
This field holds a phonetic code derived from coalesce(k1,word).
Letters that have similar sounds are mapped into the same symbol.
For example, all vowels and vowel clusters become the
single symbol "A".  And the letters "p", "b", "f", and
"v" all become "B".  All nasal sounds are represented
as "N".  And so forth.  The mapping is based on
ideas found in Soundex, Metaphone, and other
long-standing phonetic matching systems.  This key can
be generated by the function spellfix1_phonehash(X).  
Hence: k2 = spellfix1_phonehash(coalesce(k1,word))


There is also a function for computing the Wagner edit distance or the
Levenshtein distance between a pattern and a word.  This function
is exposed as spellfix1_editdist(X,Y).  The edit distance function
returns the "cost" of converting X into Y.  Some transformations
cost more than others.  Changing one vowel into a different vowel,
for example is relatively cheap, as is doubling a constant, or
omitting the second character of a double-constant.  Other transformations
or more expensive.  The idea is that the edit distance function returns
a low cost for words that are similar and a higher cost for words
that are further apart.  In this implementation, the maximum cost
of any single-character edit (delete, insert, or substitute) is 100,
with lower costs for some edits (such as transforming vowels).

The "score" for a comparison is the edit distance between the pattern
and the word, adjusted down by the base-2 logarithm of the word rank.
For example, a match with distance 100 but rank 1000 would have a
score of 122 (= 100 - log2(1000) + 32) whereas a match with distance
100 with a rank of 1 would have a score of 131 (100 - log2(1) + 32).
(NB:  The constant 32 is added to each score to keep it from going
negative in case the edit distance is zero.)  In this way, frequently
used words get a slightly lower cost which tends to move them toward
the top of the list of alternative spellings.

A straightforward implementation of a spelling corrector would be
to compare the search term against every word in the vocabulary
and select the 20 with the lowest scores.  However, there will 
typically be hundreds of thousands or millions of words in the
vocabulary, and so this approach is not fast enough.

Suppose the term that is being spell-corrected is X.  To limit
the search space, X is converted to a k2-like key using the
equivalent of:


   key = spellfix1_phonehash(lower(spellfix1_translit(X)))


This key is then limited to "scope" characters.  The default scope
value is 4, but an alternative scope can be specified using the
"scope=N" term in the WHERE clause.  After the key has been truncated,
the edit distance is run against every term in the vocabulary that
has a k2 value that begins with the abbreviated key.

For example, suppose the input word is "Paskagula".  The phonetic 
key is "BACACALA" which is then truncated to 4 characters "BACA".
The edit distance is then run on the 4980 entries (out of
272,597 entries total) of the vocabulary whose k2 values begin with
BACA, yielding "Pascagoula" as the best match.

Only terms of the vocabulary with a matching langid are searched.
Hence, the same table can contain entries from multiple languages
and only the requested language will be used.  The default langid
is 0.


c2: The Spellfix1 Virtual Table
c3: 5. Configurable Edit Distance
c5: spellfix1.html#configurable_edit_distance
c4:
The built-in Wagner edit-distance function with fixed weights can be
replaced by the editdist3() edit-distance function
with application-defined weights and support for unicode, by specifying
the "edit_cost_table=TABLENAME" parameter to the spellfix1 module
when the virtual table is created.
For example:


CREATE VIRTUAL TABLE demo2 USING spellfix1(edit_cost_table=APPCOST);


The editdist3() edit-distance function can also be selected or 
deselected at run-time by inserting an appropriate string into the
"command" column of the virtual table:


INSERT INTO demo2(command) VALUES('edit_cost_table=APPCOST');



In the examples above, the APPCOST table would be interrogated to find
the edit distance coefficients.  It is the presence of the "edit_cost_table="
parameter to the spellfix1 module name that causes editdist3() to be used
in place of the built-in edit distance function.  If APPCOST is an empty
string, then the built-in Wagner edit-distance function is used.

The edit distance coefficients are normally read from the APPCOST table
once and there after stored in memory.  Hence, run-time changes to the
APPCOST table will not normally affect the edit distance results.
However, inserting the special string 'reset' into the "command" column of the
virtual table causes the edit distance coefficients to be reread the
APPCOST table.  Hence, applications should run a SQL statement similar
to the following when changes to the APPCOST table occur:


INSERT INTO demo2(command) VALUES("reset");


c2: The Spellfix1 Virtual Table
c3: 6. Dealing With Unusual And Difficult Spellings
c5: spellfix1.html#dealing_with_unusual_and_difficult_spellings
c4:
The algorithm above works quite well for most cases, but there are
exceptions.  These exceptions can be dealt with by making additional
entries in the virtual table using the "soundslike" column.

For example, many words of Greek origin begin with letters "ps" where
the "p" is silent.  Ex:  psalm, pseudonym, psoriasis, psyche.  In
another example, many Scottish surnames can be spelled with an
initial "Mac" or "Mc".  Thus, "MacKay" and "McKay" are both pronounced
the same.

Accommodation can be made for words that are not spelled as they
sound by making additional entries into the virtual table for the
same word, but adding an alternative spelling in the "soundslike"
column.  For example, the canonical entry for "psalm" would be this:


  INSERT INTO demo(word) VALUES('psalm');


To enhance the ability to correct the spelling of "salm" into
"psalm", make an addition entry like this:


  INSERT INTO demo(word,soundslike) VALUES('psalm','salm');


It is ok to make multiple entries for the same word as long as
each entry has a different soundslike value.  Note that if no
soundslike value is specified, the soundslike defaults to the word
itself.

Listed below are some cases where it might make sense to add additional
soundslike entries.  The specific entries will depend on the application
and the target language.


Silent "p" in words beginning with "ps":  psalm, psyche
Silent "p" in words beginning with "pn":  pneumonia, pneumatic
Silent "p" in words beginning with "pt":  pterodactyl, ptolemaic
Silent "d" in words beginning with "dj":  djinn, Djikarta
Silent "k" in words beginning with "kn":  knight, Knuthson
Silent "g" in words beginning with "gn":  gnarly, gnome, gnat
"Mac" versus "Mc" beginning Scottish surnames
"Tch" sounds in Slavic words:  Tchaikovsky vs. Chaykovsky
The letter "j" pronounced like "h" in Spanish:  LaJolla
Words beginning with "wr" versus "r":  write vs. rite
Miscellaneous problem words such as "debt", "tsetse",
      "Nguyen", "Van Nuyes".


c2: The Spellfix1 Virtual Table
c3: 7. Auxiliary Functions
c5: spellfix1.html#auxiliary_functions
c4:
The source code module that implements the spellfix1 virtual table also
implements several SQL functions that might be useful to applications
that employ spellfix1 or for testing or diagnostic work while developing
applications that use spellfix1.  The following auxiliary functions are
available:


editdist3(P,W)editdist3(P,W,L)editdist3(T)
These routines provide direct access to the version of the Wagner
edit-distance function that allows for application-defined weights
on edit operations.  The first two forms of this function compare
pattern P against word W and return the edit distance.  In the first
function, the langid is assumed to be 0 and in the second, the
langid is given by the L parameter.  The third form of this function
reloads edit distance coefficients from the table named by T.

spellfix1_editdist(P,W)
This routine provides access to the built-in Wagner edit-distance
function that uses default, fixed costs.  The value returned is
the edit distance needed to transform W into P.

spellfix1_phonehash(X)
This routine constructs a phonetic hash of the pure ascii input word X
and returns that hash.  This routine is used internally by spellfix1 in
order to transform the K1 column of the shadow table into the K2
column.

spellfix1_scriptcode(X)
Given an input string X, this routine attempts to determine the dominant
script of that input and returns the ISO-15924 numeric code for that
script.  The current implementation understands the following scripts:

 215 - Latin
 220 - Cyrillic
 200 - Greek

Additional language codes might be added in future releases.

spellfix1_translit(X)
This routine transliterates unicode text into pure ascii, returning
the pure ascii representation of the input text X.  This is the function
that is used internally to transform vocabulary words into the K1
column of the shadow table.


c2: The Spellfix1 Virtual Table
c3: 8. The editdist3 function
c5: spellfix1.html#the_editdist3_function
c4:
The editdist3 algorithm is a function that computes the minimum edit 
distance (a.k.a. the Levenshtein distance) between two input strings.
The editdist3 algorithm is a configurable alternative to the default
edit distance function of spellfix1.
Features of editdist3 include:


It works with unicode (UTF8) text.

A table of insertion, deletion, and substitution costs can be 
       provided by the application.

Multi-character insertions, deletions, and substitutions can be
       enumerated in the cost table.


c2: The Spellfix1 Virtual Table
c3: 9. The editdist3 COST table
c5: spellfix1.html#the_editdist3_cost_table
c4:
To program the costs of editdist3, create a table such as the following:


CREATE TABLE editcost(
  iLang INT,   -- The language ID
  cFrom TEXT,  -- Convert text from this
  cTo   TEXT,  -- Convert text into this
  iCost INT    -- The cost of doing the conversion
);


The cost table can be named anything you want - it does not have to be
called "editcost".  And the table can contain additional columns.
The only requirement is that the
table must contain the four columns show above, with exactly the names shown.

The iLang column is a non-negative integer that identifies a set of costs
appropriate for a particular language.  The editdist3 function will only use
a single iLang value for any given edit-distance computation.  The default
value is 0.  It is recommended that applications that only need to use a
single language always use iLang==0 for all entries.

The iCost column is the numeric cost of transforming cFrom into cTo.  This
value should be a non-negative integer, and should probably be less than 100.
The default single-character insertion and deletion costs are 100 and the
default single-character to single-character substitution cost is 150.  A
cost of 10000 or more is considered "infinite" and causes the rule to be
ignored.

The cFrom and cTo columns show edit transformation strings.  Either or both
columns may contain more than one character.  Or either column (but not both)
may hold an empty string.  When cFrom is empty, that is the cost of inserting
cTo.  When cTo is empty, that is the cost of deleting cFrom.

In the spellfix1 algorithm, cFrom is the text as the user entered it and
cTo is the correctly spelled text as it exists in the database.  The goal
of the editdist3 algorithm is to determine how close the user-entered text is
to the dictionary text.

There are three special-case entries in the cost table:


cFromcToMeaning
'''?'The default insertion cost
'?'''The default deletion cost
'?''?'The default substitution cost


If any of the special-case entries shows above are omitted, then the
value of 100 is used for insertion and deletion and 150 is used for
substitution.  To disable the default insertion, deletion, and/or substitution
set their respective cost to 10000 or more.

Other entries in the cost table specific transforms for particular 
characters.
The cost of specific transforms should be less than the default costs, or else
the default costs will take precedence and the specific transforms will never 
be used.

Some example, cost table entries:


INSERT INTO editcost(iLang, cFrom, cTo, iCost)
VALUES(0, 'a', 'ä', 5);


The rule above says that the letter "a" in user input can be matched against
the letter "ä" in the dictionary with a penalty of 5.


INSERT INTO editcost(iLang, cFrom, cTo, iCost)
VALUES(0, 'ss', 'ß', 8);


The number of characters in cFrom and cTo do not need to be the same.  The
rule above says that "ss" on user input will match "ß" with a penalty of 8.


c2: The Spellfix1 Virtual Table
c3: 10. Experimenting with the editcost3() function
c5: spellfix1.html#experimenting_with_the_editcost3_function
c4:
The spellfix1 virtual table
uses editdist3 if the "edit_cost_table=TABLE" option
is specified as an argument when the spellfix1 virtual table is created.  
But editdist3 can also be tested directly using the built-in "editdist3()"
SQL function.  The editdist3() SQL function has 3 forms:


 editdist3('TABLENAME');
 editdist3('string1', 'string2');
 editdist3('string1', 'string2', langid);


The first form loads the edit distance coefficients from a table called
'TABLENAME'.  Any prior coefficients are discarded.  So when experimenting
with weights and the weight table changes, simply rerun the single-argument
form of editdist3() to reload revised coefficients.  Note that the 
edit distance
weights used by the editdist3() SQL function are independent from the
weights used by the spellfix1 virtual table.

The second and third forms return the computed edit distance between strings
'string1' and "string2'.  In the second form, a language id of 0 is used.
The language id is specified in the third form.


c2: The DBSTAT Virtual Table
c3: 1. Overview
c5: dbstat.html#overview
c4:
The DBSTAT virtual table is a read-only eponymous virtual table that returns
information about the amount of disk space used to store the content
of an SQLite database.
Example use cases for the
DBSTAT virtual table include the sqlite3_analyzer.exe
utility program and the
table size pie-chart in
the Fossil-implemented version control system
for SQLite.



The DBSTAT virtual table is available on all 
database connections when SQLite is built using the
SQLITE_ENABLE_DBSTAT_VTAB compile-time option.


The DBSTAT virtual table is an eponymous virtual table, meaning
that is not necessary to run CREATE VIRTUAL TABLE to create an
instance of the dbstat virtual table before using it.  The "dbstat"
module name can be used as if it were a table name to query the
dbstat virtual table directly.  For example:

SELECT * FROM dbstat;



If a named virtual table that uses the dbstat module is desired,
then the recommended way to create an instance of the dbstat
virtual table is as follows:

CREATE VIRTUAL TABLE temp.stat USING dbstat(main);



Note the "temp." qualifier before the virtual table name ("stat").  This
qualifier causes the virtual table to be temporary - to only exist for
the duration of the current database connection.  This is the
recommended approach.


The "main" argument to dbstat is default schema
for which information is to be provided.  The default is "main", and
so the use of "main" in the example above is redundant.  For any
particular query, the schema can be changed by specifying the
alternative schema as a function argument to the virtual table
name in the FROM clause of the query.  (See further discussion of
table-valued functions in the FROM clause for more details.)


The schema for the DBSTAT virtual table looks like this:
CREATE TABLE dbstat(
  name       TEXT,        -- Name of table or index
  path       TEXT,        -- Path to page from root
  pageno     INTEGER,     -- Page number, or page count
  pagetype   TEXT,        -- 'internal', 'leaf', 'overflow', or NULL
  ncell      INTEGER,     -- Cells on page (0 for overflow pages)
  payload    INTEGER,     -- Bytes of payload on this page or btree
  unused     INTEGER,     -- Bytes of unused space on this page or btree
  mx_payload INTEGER,     -- Largest payload size of all cells on this row
  pgoffset   INTEGER,     -- Byte offset of the page in the database file
  pgsize     INTEGER,     -- Size of the page, in bytes
  schema     TEXT HIDDEN, -- Database schema being analyzed
  aggregate  BOOL HIDDEN  -- True to enable aggregate mode
);



The DBSTAT table only reports on the content of btrees within the database file.
Freelist pages, pointer-map pages, and the lock page are omitted from
the analysis.


By default, there is a single row in the DBSTAT table for each
btree page the database file.  Each row provides
information about the space utilization of that one page of the
database.  However, if the hidden column "aggregate" is TRUE, then
results are aggregated and there is a single row in the DBSTAT table
for each btree in the database, providing information about space
utilization across the entire btree.


c2: The DBSTAT Virtual Table
c3: 2. The "path" column of the dbstat virtual table
c5: dbstat.html#the_path_column_of_the_dbstat_virtual_table
c4:
The "path" column describes the path taken from the 
root node of the btree structure to each page.  The
"path" of the root node itself is '/'.
The "path" is NULL when "aggregate" is TRUE.

The "path" for the left-most child page of the root of
a btree page is '/000/'. (Btrees store content ordered from left to right
so the pages to the left have smaller keys than the pages to the right.)
The next to left-most child of the root page is '/001', and so on,
each sibling page identified by a 3-digit hex  value.
The children of the 451st left-most sibling have paths such
as '/1c2/000/, '/1c2/001/' etc.

Overflow pages are specified by appending a '+' character and a 
six-digit hexadecimal value to the path to the cell they are linked
from. For example, the three overflow pages in a chain linked from 
the left-most cell of the 450th child of the root page are identified
by the paths:

'/1c2/000+000000'         // First page in overflow chain
'/1c2/000+000001'         // Second page in overflow chain
'/1c2/000+000002'         // Third page in overflow chain



If the paths are sorted using the BINARY collation sequence, then
the overflow pages associated with a cell will appear earlier in the
sort-order than its child page:

'/1c2/000/'               // Left-most child of 451st child of root


c2: The DBSTAT Virtual Table
c3: 3. Aggregated Data
c5: dbstat.html#aggregated_data
c4:
Beginning with SQLite version 3.31.0 (2020-01-22), the DBSTAT table
has a new hidden column named "aggregate", which if constrained to be
TRUE will cause DBSTAT to generate one row per btree in the database,
rather than one row per page.  When running in aggregated mode, the 
"path", "pagetype", and "pgoffset" columns are always NULL and the
"pageno" column holds the number of pages in the entire btree, rather
than the number of the page that corresponds to the row.


The following table shows the meanings of the (non-hidden) columns of
DBSTAT in both normal and aggregated mode:




Column
Normal meaning
Aggregate-mode meaning


name

The name of the table or index that is implemented by
the btree of the current row

path
See description above
Always NULL

pageno
The page number of the database page for the current row
The total number of pages in the btree for the current row

pagetype
'leaf' or 'interior'
Always NULL

ncell
Number of cells on the current page or btree

payload
Bytes of useful payload on the current page or btree

unused
Unused bytes of on the current page or btree

mx_payload
The largest payload found anywhere in the current page
or btree.

pgoffset
Byte offset to the start of the page
Always NULL

pgsize
Total storage space used by the current page or btree.


c2: The DBSTAT Virtual Table
c3: 4. Example uses of the dbstat virtual table
c5: dbstat.html#example_uses_of_the_dbstat_virtual_table
c4:
To find the total number of pages used to store table "xyz" in schema "aux1",
use either of the following two queries (the first is the traditional way,
and the second shows the use of the aggregated feature):

SELECT count(*) FROM dbstat('aux1') WHERE name='xyz';
SELECT pageno FROM dbstat('aux1',1) WHERE name='xyz';



To see how efficiently the content of a table is stored on disk,
compute the amount of space used to hold actual content divided
by the total amount of disk space used.  The closer this number
is to 100%, the more efficient the packing.  (In this example, the
'xyz' table is assumed to be in the 'main' schema.  Again, there
are two different versions that show the use of DBSTAT both without
and with the new aggregated feature, respectively.)

SELECT sum(pgsize-unused)*100.0/sum(pgsize) FROM dbstat WHERE name='xyz';
SELECT (pgsize-unused)*100.0/pgsize FROM dbstat
 WHERE name='xyz' AND aggregate=TRUE;



To find the average fan-out for a table, run:

SELECT avg(ncell) FROM dbstat WHERE name='xyz' AND pagetype='internal';




Modern filesystems operate faster when disk accesses are sequential.
Hence, SQLite will run faster if the content of the database file
is on sequential pages.  To find out what fraction of the pages in
a database are sequential (and thus obtain a measurement that might
be useful in determining when to VACUUM), run a query like the following:

CREATE TEMP TABLE s(rowid INTEGER PRIMARY KEY, pageno INT);
INSERT INTO s(pageno) SELECT pageno FROM dbstat ORDER BY path;
SELECT sum(s1.pageno+1==s2.pageno)*1.0/count(*)
  FROM s AS s1, s AS s2
 WHERE s1.rowid+1=s2.rowid;
DROP TABLE s;


c2: The Checksum VFS Shim
c3: 1. Overview
c5: cksumvfs.html#overview
c4:
The checksum VFS extension is a VFS shim that adds an 8-byte
checksum to the end of every page in an SQLite database.  The checksum
is added as each page is written and verified as each page is read.
The checksum is intended to help detect database corruption caused by
random bit-flips in the mass storage device.

The checksum VFS extension requires 
SQLite version 3.32.0 (2020-05-22) or later.  It will not
work with earlier versions of SQLite.


c2: The Checksum VFS Shim
c3: 2. Compiling
c5: cksumvfs.html#compiling
c4:
The checksum VFS module is a loadable extension.  It is not
included in the amalgamation.  It must be added to SQLite
either at compile-time or at run-time.  The source code to
the checksum VFS module is in the
ext/misc/cksumvfs.c
source file in the
SQLite source tree.

To build the checksum VFS module into a run-time loadable
extension, use commands similar to the following:


(linux) &rarr; gcc -fPIC -shared cksumvfs.c -o cksumvfs.so
(mac) &rarr; clang -fPIC -dynamiclib cksumvfs.c -o cksumvfs.dylib
(windows) &rarr; cl cksumvfs.c -link -dll -out:cksumvfs.dll


You may want to add additional compiler options, of course,
according to the needs of your project.

To statically link this extension with your product,
compile it like any other C-language module but add the
"-DSQLITE_CKSUMVFS_STATIC" option so that this module knows that
it is being statically linked rather than dynamically linked.


c2: The Checksum VFS Shim
c3: 3. Loading
c5: cksumvfs.html#loading
c4:
To load this extension as a shared library, you first have to
bring up a dummy SQLite database connection to use as the argument
to the sqlite3_load_extension() API call.  Then you invoke the
sqlite3_load_extension() API and shutdown the dummy database
connection.  All subsequent database connections that are opened
will include this extension.  For example:

sqlite3 *db;
sqlite3_open(":memory:", &db);
sqlite3_load_extension(db, "./cksumvfs");
sqlite3_close(db);


If this extension is compiled with -DSQLITE_CKSUMVFS_STATIC and
statically linked against the application, initialize it using
a single API call as follows:

sqlite3_cksumvfs_init();


Cksumvfs is a VFS shim. When loaded, "cksmvfs" becomes the new
default VFS, and it uses the prior default VFS as the next VFS
down in the stack.  This is normally what you want.  However, in
complex situations where multiple VFS shims are being loaded,
it might be important to ensure that cksumvfs is loaded in the
correct order so that it sequences itself into the default VFS
Shim stack in the right order.


c2: The Checksum VFS Shim
c3: 4. Usage
c5: cksumvfs.html#usage
c4:
Open database connections using the sqlite3_open() or 
sqlite3_open_v2() interfaces, as normal.  Ordinary database files
(without a checksum) will operate normally.  Databases with 
checksums will return an SQLITE_IOERR_DATA error if a page is
encountered that contains an invalid checksum.

Checksumming only works on databases that have a reserve bytes
value of exactly 8.  The default value for reserve-bytes is 0.
Hence, newly created database files will omit the checksum by
default.  To create a database that includes a checksum, change
the reserve-bytes value to 8 by running code similar to this:

int n = 8;
sqlite3_file_control(db, 0, SQLITE_FCNTL_RESERVE_BYTES, &n);


If you do this immediately after creating a new database file,
before anything else has been written into the file, then that
might be all that you need to do.  Otherwise, the API call
above should be followed by:

sqlite3_exec(db, "VACUUM", 0, 0, 0);


It never hurts to run the VACUUM, even if you don't need it.
If the database is in WAL mode, you should shutdown and
reopen all database connections before continuing.

From the CLI, use the ".filectrl reserve_bytes 8" command, 
followed by "VACUUM;".

Note that SQLite allows the number of reserve-bytes to be
increased but not decreased.  So if a database file already
has a reserve-bytes value greater than 8, there is no way to
activate checksumming on that database, other than to dump
and restore the database file.  Note also that other extensions
might also make use of the reserve-bytes.  Checksumming will
be incompatible with those other extensions.


c2: The Checksum VFS Shim
c3: 5. Verification Of Checksums
c5: cksumvfs.html#verification_of_checksums
c4:
If any checksum is incorrect, the "PRAGMA quick_check" command
will find it.  To verify that checksums are actually enabled
and running, use SQL like the following:

SELECT count(*), verify_checksum(data)
  FROM sqlite_dbpage
 GROUP BY 2;


There are three possible outputs from the verify_checksum()
function: 1, 0, and NULL.  1 is returned if the checksum is
correct.  0 is returned if the checksum is incorrect.  NULL
is returned if the page is unreadable.  If checksumming is
enabled, the read will fail if the checksum is wrong, so the
usual result from verify_checksum() on a bad checksum is NULL.

If everything is OK, the query above should return a single
row where the second column is 1.  Any other result indicates
either that there is a checksum error, or checksum validation
is disabled.


c2: The Checksum VFS Shim
c3: 6. Controlling Checksum Verification
c5: cksumvfs.html#controlling_checksum_verification
c4:
The cksumvfs extension implements a new PRAGMA statement that can
be used to disable, re-enable, or query the status of checksum
verification:

PRAGMA checksum_verification;          -- query status
PRAGMA checksum_verification=OFF;      -- disable verification
PRAGMA checksum_verification=ON;       -- re-enable verification


The "checksum_verification" pragma will return "1" (true) or "0"
(false) if checksum verification is enabled or disabled, respectively.
"Verification" in this context means the feature that causes
SQLITE_IOERR_DATA errors if a checksum mismatch is detected while
reading.  Checksums are always kept up-to-date as long as the
reserve bytes value of the database is 8, regardless of the setting
of this pragma.  Checksum verification can be disabled (for example)
to do forensic analysis of a database that has previously reported
a checksum error.

The "checksum_verification" pragma will always respond with "0" if
the database file does not have a reserve bytes value of 8.  The
pragma will return no rows at all if the cksumvfs extension is
not loaded.


c2: Window Functions
c3: 1. Introduction to Window Functions
c5: windowfunctions.html#introduction_to_window_functions
c4:
A window function is an SQL function where the input
values are taken from
a "window" of one or more rows in the results set of a SELECT statement.

Window functions are distinguished from other SQL functions by the
presence of an OVER clause. If a function has an OVER clause,
then it is a window function. If it lacks an OVER clause, then it is an
ordinary aggregate or scalar function. Window functions might also
have a FILTER clause in between the function and the OVER clause.

The syntax for a window function is like this:


window-function-invocation:
hide
 
 

expr:
show
 
 

literal-value:
show
 
 


over-clause:
show
 
 

frame-spec:
hide
 
 


ordering-term:
show
 
 



raise-function:
show
 
 


select-stmt:
show
 
 

common-table-expression:
show
 
 


compound-operator:
show
 
 


join-clause:
show
 
 

join-constraint:
show
 
 


join-operator:
show
 
 



ordering-term:
show
 
 


result-column:
show
 
 


table-or-subquery:
show
 
 



type-name:
show
 
 

signed-number:
show
 
 




filter-clause:
hide
 
 


window-defn:
hide
 
 

frame-spec:
hide
 
 


ordering-term:
show
 
 







Unlike ordinary functions, window functions
cannot use the DISTINCT keyword.
Also, Window functions may only appear in the result set and in the
ORDER BY clause of a SELECT statement.

Window functions come in two varieties: 
aggregate window functions and
built-in window functions.  Every aggregate window function
can also work as a ordinary aggregate function, simply by omitting
the OVER and FILTER clauses.  Furthermore, all of the built-in
aggregate functions of SQLite can be used as an
aggregate window function by adding an appropriate OVER clause.
Applications can register new aggregate window functions using
the sqlite3_create_window_function() interface.
The built-in window functions, however, require special-case
handling in the query planner and hence new window functions
that exhibit the exceptional properties found in the built-in
window functions cannot be added by the application.

Here is an example using the built-in row_number()
window function:

CREATE TABLE t0(x INTEGER PRIMARY KEY, y TEXT);
INSERT INTO t0 VALUES (1, 'aaa'), (2, 'ccc'), (3, 'bbb');

-- The following SELECT statement returns:
-- 
--   x | y | row_number
-----------------------
--   1 | aaa | 1         
--   2 | ccc | 3         
--   3 | bbb | 2         
-- 
SELECT x, y, row_number() OVER (ORDER BY y) AS row_number FROM t0 ORDER BY x;



The row_number() window function
assigns consecutive integers to each
row in order of the "ORDER BY" clause within the
window-defn (in this case "ORDER BY y"). Note that 
this does not affect the order in which results are returned from
the overall query.  The order of the final output is
still governed by the ORDER BY clause attached to the SELECT
statement (in this case "ORDER BY x").

Named window-defn clauses may also be added to a SELECT 
statement using a WINDOW clause and then referred to by name within window
function invocations. For example, the following SELECT statement contains
two named window-defs clauses, "win1" and "win2":

SELECT x, y, row_number() OVER win1, rank() OVER win2 
FROM t0 
WINDOW win1 AS (ORDER BY y RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW),
       win2 AS (PARTITION BY y ORDER BY x)
ORDER BY x;


The WINDOW clause, when one is present, comes after any HAVING clause and
before any ORDER BY.


c2: Window Functions
c3: 2. Aggregate Window Functions
c5: windowfunctions.html#aggregate_window_functions
c4:
The examples in this section all assume that the database is populated as
follows:

CREATE TABLE t1(a INTEGER PRIMARY KEY, b, c);
INSERT INTO t1 VALUES   (1, 'A', 'one'  ),
                        (2, 'B', 'two'  ),
                        (3, 'C', 'three'),
                        (4, 'D', 'one'  ),
                        (5, 'E', 'two'  ),
                        (6, 'F', 'three'),
                        (7, 'G', 'one'  );


 An aggregate window function is similar to an 
ordinary aggregate function, except
adding it to a query does not change the number of rows returned. Instead,
for each row the result of the aggregate window function is as if the
corresponding aggregate were run over all rows in the "window frame"
specified by the OVER clause.


-- The following SELECT statement returns:
-- 
--   a | b | group_concat
-------------------------
--   1 | A | A.B         
--   2 | B | A.B.C       
--   3 | C | B.C.D       
--   4 | D | C.D.E       
--   5 | E | D.E.F       
--   6 | F | E.F.G       
--   7 | G | F.G         
-- 
SELECT a, b, group_concat(b, '.') OVER (
  ORDER BY a ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING
) AS group_concat FROM t1;


 In the example above, the window frame consists of all rows between the
previous row ("1 PRECEDING") and the following row ("1 FOLLOWING"), inclusive,
where rows are sorted according to the ORDER BY clause in the
window-defn (in this case "ORDER BY a"). 
For example, the frame for the row with (a=3) consists of rows (2, 'B', 'two'),
(3, 'C', 'three') and (4, 'D', 'one'). The result of group_concat(b, '.') 
for that row is therefore 'B.C.D'.

 All of SQLite's aggregate functions may
be used as aggregate window functions. It is also possible to 
create user-defined aggregate window functions.


c2: Window Functions
c3: 2.1. The PARTITION BY Clause
c5: windowfunctions.html#the_partition_by_clause
c4:
For the purpose of computing window functions, the result set
of a query is divided into one or more "partitions".  A partition consists
of all rows that have the same value for all terms of the PARTITION BY clause
in the window-defn.  If there is no PARTITION BY clause,
then the entire result set of the query is a single partition.
Window-function processing is performed separately for each partition.

 For example:

-- The following SELECT statement returns:
-- 
--   c     | a | b | group_concat
---------------------------------
--   one   | 1 | A | A.D.G       
--   one   | 4 | D | D.G         
--   one   | 7 | G | G           
--   three | 3 | C | C.F         
--   three | 6 | F | F           
--   two   | 2 | B | B.E         
--   two   | 5 | E | E           
-- 
SELECT c, a, b, group_concat(b, '.') OVER (
  PARTITION BY c ORDER BY a RANGE BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
) AS group_concat
FROM t1 ORDER BY c, a;


 In the query above, the "PARTITION BY c" clause breaks the
result set up into three partitions. The first partition has
three rows with c=='one'.  The second partition has two rows with
c=='three' and the third partition has two rows with c=='two'.

 In the example above, all the rows for each partition are
grouped together in the final output.  This is because the PARTITION BY
clause is a prefix of the ORDER BY clause on the overall query.  
But that does not have
to be the case.  A partition can be composed of rows scattered
about haphazardly within the result set.  For example:

-- The following SELECT statement returns:
-- 
--   c     | a | b | group_concat
---------------------------------
--   one   | 1 | A | A.D.G       
--   two   | 2 | B | B.E         
--   three | 3 | C | C.F         
--   one   | 4 | D | D.G         
--   two   | 5 | E | E           
--   three | 6 | F | F           
--   one   | 7 | G | G           
-- 
SELECT c, a, b, group_concat(b, '.') OVER (
  PARTITION BY c ORDER BY a RANGE BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
) AS group_concat
FROM t1 ORDER BY a;


c2: Window Functions
c3: 2.2. Frame Specifications
c5: windowfunctions.html#frame_specifications
c4:
The frame-spec determines which output rows are
read by an aggregate window function.  The
frame-spec consists of four parts:


   A frame type - either ROWS, RANGE or GROUPS,
   A starting frame boundary,
   An ending frame boundary,
   An EXCLUDE clause.


 Here are the syntax details:

frame-spec:
hide
 
 

expr:
show
 
 

filter-clause:
show
 
 


literal-value:
show
 
 


over-clause:
show
 
 

ordering-term:
show
 
 



raise-function:
show
 
 


select-stmt:
show
 
 

common-table-expression:
show
 
 


compound-operator:
show
 
 


join-clause:
show
 
 

join-constraint:
show
 
 


join-operator:
show
 
 



ordering-term:
show
 
 


result-column:
show
 
 


table-or-subquery:
show
 
 


window-defn:
show
 
 



type-name:
show
 
 

signed-number:
show
 
 








The ending frame boundary can be omitted (if the
BETWEEN and AND keywords that surround the starting frame boundary
are also omitted),
in which case the ending frame boundary defaults to CURRENT ROW.

 If the frame type is RANGE or GROUPS, then rows with the same values for
all ORDER BY expressions are considered "peers". Or, if there are no ORDER BY
terms, all rows are peers. Peers are always within the same frame.

The default frame-spec is:

RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW EXCLUDE NO OTHERS


The default means that aggregate window functions read all
rows from the beginning of the partition up to and including the
current row and its peers. This implies that rows that have the same values for
all ORDER BY expressions will also have the same value for the result of the
window function (as the window frame is the same). For example:

-- The following SELECT statement returns:
-- 
--   a | b | c | group_concat
-----------------------------
--   1 | A | one   | A.D.G       
--   2 | B | two   | A.D.G.C.F.B.E
--   3 | C | three | A.D.G.C.F   
--   4 | D | one   | A.D.G       
--   5 | E | two   | A.D.G.C.F.B.E
--   6 | F | three | A.D.G.C.F   
--   7 | G | one   | A.D.G       
-- 
SELECT a, b, c, 
       group_concat(b, '.') OVER (ORDER BY c) AS group_concat 
FROM t1 ORDER BY a;


c2: Window Functions
c3: 2.2.1. Frame Type
c5: windowfunctions.html#frame_type
c4:
There are three frame types:  ROWS, GROUPS, and RANGE.
The frame type determines how the starting and ending boundaries
of the frame are measured.


ROWS: 
The ROWS frame type means that the starting and ending boundaries
for the frame are determined by counting individual rows relative
to the current row.



GROUPS:
The GROUPS frame type means that the starting and ending boundaries
are determine by counting "groups" relative to the current group.
A "group" is a set of rows that all have equivalent values for all
all terms of the window ORDER BY clause.  ("Equivalent" means that
the IS operator is true when comparing the two values.)
In other words, a group consists of all peers of a row.



RANGE:
The RANGE frame type requires that the ORDER BY clause of the
window have exactly one term.  Call that term "X".  With the
RANGE frame type, the elements of the frame are determined by
computing the value of expression X for all rows in the partition
and framing those rows for which the value of X is within a certain
range of the value of X for the current row.  See the description
in the "&lt;expr&gt; PRECEDING" boundary
specification below for details.


The ROWS and GROUPS frame types are similar in that they
both determine the extent of a frame by counting relative to
the current row.  The difference is that ROWS counts individual
rows and GROUPS counts peer groups.
The RANGE frame type is different.
The RANGE frame type determines the extent of a frame by
looking for expression values that are within some band of
values relative to the current row.


c2: Window Functions
c3: 2.2.2. Frame Boundaries
c5: windowfunctions.html#frame_boundaries
c4:
There are five ways to describe starting and ending frame boundaries:


 UNBOUNDED PRECEDING
     The frame boundary is the first
     row in the partition.

 &lt;expr&gt; PRECEDING
        &lt;expr&gt; must be a non-negative constant numeric expression.
        The boundary is a row that is &lt;expr&gt; "units" prior to
        the current row.  The meaning of "units" here depends on the
        frame type:
        
         ROWS &rarr;
        The frame boundary is the row that is &lt;expr&gt;
        rows before the current row, or the first row of the
        partition if there are fewer than &lt;expr&gt; rows
        before the current row.  &lt;expr&gt; must be an integer.
         GROUPS &rarr;
        A "group" is a set of peer rows - rows that all have
        the same values for every term in the ORDER BY clause.
        The frame boundary is the group that is &lt;expr&gt;
        groups before the group containing the current row, or the
        first group of the partition if there are fewer
        than &lt;expr&gt; groups before the current row.
        For the starting boundary of a frame, the first
        row of the group is used and for the ending boundary
        of a frame, the last row of the group is used.
        &lt;expr&gt; must be an integer.
        

         RANGE &rarr;
        For this form, the ORDER BY clause of the
        window-defn must have a single
        term.  Call that ORDER BY term "X".  Let
        Xi be the value of the X
        expression for the i-th row in the partition and let
        Xc be the value of X for the
        current row.  Informally, a RANGE bound is the first row
        for which Xi is within
        the &lt;expr&gt; of Xc.
        More precisely:
        
         If either Xi or
             Xc are non-numeric, then
             the boundary is the first row for which the expression
             "Xi IS Xc"
             is true.
         Else if the ORDER BY is ASC then the boundary
             is the first row for which
             Xi&gt;=Xc-&lt;expr&gt;.
         Else if the ORDER BY is DESC then the boundary
        is the first row for which
        Xi&lt;=Xc+&lt;expr&gt;.
        
        For this form, the &lt;expr&gt; does not have to be an
        integer. It can evaluate to a real number as long as
        it is constant and non-negative.
        
    The boundary description "0 PRECEDING" always means the same
    thing as "CURRENT ROW".

CURRENT ROW
        The current row. For RANGE and GROUPS frame types,
        peers of the current row are also included in the frame,
        unless specifically excluded by the EXCLUDE clause.
        This is true regardless of whether CURRENT ROW is used
        as the starting or ending frame boundary.

&lt;expr&gt; FOLLOWING
        This is the same as "&lt;expr&gt; PRECEDING" except that
        the boundary is &lt;expr&gt; units after the current
        rather than before the current row.

 UNBOUNDED FOLLOWING
   The frame boundary is the last
   row in the partition.



 The ending frame boundary may not take a form that appears higher in
the above list than the starting frame boundary.

 In the following example, the window frame for each row consists of all
rows from the current row to the end of the set, where rows are sorted
according to "ORDER BY a".

-- The following SELECT statement returns:
-- 
--   c     | a | b | group_concat
---------------------------------
--   one   | 1 | A | A.D.G.C.F.B.E
--   one   | 4 | D | D.G.C.F.B.E 
--   one   | 7 | G | G.C.F.B.E   
--   three | 3 | C | C.F.B.E     
--   three | 6 | F | F.B.E       
--   two   | 2 | B | B.E         
--   two   | 5 | E | E           
-- 
SELECT c, a, b, group_concat(b, '.') OVER (
  ORDER BY c, a ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
) AS group_concat
FROM t1 ORDER BY c, a;


c2: Window Functions
c3: 2.2.3. The EXCLUDE Clause
c5: windowfunctions.html#the_exclude_clause
c4:
The optional EXCLUDE clause may take any of the following four forms:


    EXCLUDE NO OTHERS: This is the default. In this case no
  rows are excluded from the window frame as defined by its starting and ending
  frame boundaries.

    EXCLUDE CURRENT ROW: In this case the current row is 
  excluded from the window frame.  Peers of the current row remain in
  the frame for the GROUPS and RANGE frame types.

    EXCLUDE GROUP: In this case the current row and all other
  rows that are peers of the current row are excluded from the frame. When
  processing an EXCLUDE clause, all rows with the same ORDER BY values, or all
  rows in the partition if there is no ORDER BY clause, are considered peers,
  even if the frame type is ROWS.

    EXCLUDE TIES: In this case the current row is part of the
  frame, but peers of the current row are excluded.


 The following example demonstrates the effect of the various
forms of the EXCLUDE clause:

-- The following SELECT statement returns:
-- 
--   c    | a | b | no_others     | current_row | grp       | ties
--  one   | 1 | A | A.D.G         | D.G         |           | A
--  one   | 4 | D | A.D.G         | A.G         |           | D
--  one   | 7 | G | A.D.G         | A.D         |           | G
--  three | 3 | C | A.D.G.C.F     | A.D.G.F     | A.D.G     | A.D.G.C
--  three | 6 | F | A.D.G.C.F     | A.D.G.C     | A.D.G     | A.D.G.F
--  two   | 2 | B | A.D.G.C.F.B.E | A.D.G.C.F.E | A.D.G.C.F | A.D.G.C.F.B
--  two   | 5 | E | A.D.G.C.F.B.E | A.D.G.C.F.B | A.D.G.C.F | A.D.G.C.F.E
-- 
SELECT c, a, b,
  group_concat(b, '.') OVER (
    ORDER BY c GROUPS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW EXCLUDE NO OTHERS
  ) AS no_others,
  group_concat(b, '.') OVER (
    ORDER BY c GROUPS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW EXCLUDE CURRENT ROW
  ) AS current_row,
  group_concat(b, '.') OVER (
    ORDER BY c GROUPS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW EXCLUDE GROUP
  ) AS grp,
  group_concat(b, '.') OVER (
    ORDER BY c GROUPS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW EXCLUDE TIES
  ) AS ties
FROM t1 ORDER BY c, a;


c2: Window Functions
c3: 2.3. The FILTER Clause
c5: windowfunctions.html#the_filter_clause
c4:
filter-clause:
hide
 
 

expr:
show
 
 

literal-value:
show
 
 


over-clause:
show
 
 

frame-spec:
show
 
 


ordering-term:
show
 
 



raise-function:
show
 
 


select-stmt:
show
 
 

common-table-expression:
show
 
 


compound-operator:
show
 
 


join-clause:
show
 
 

join-constraint:
show
 
 


join-operator:
show
 
 



ordering-term:
show
 
 


result-column:
show
 
 


table-or-subquery:
show
 
 


window-defn:
show
 
 

frame-spec:
show
 
 




type-name:
show
 
 

signed-number:
show
 
 







If a FILTER clause is provided, then only rows for which the expr is
true are included in the window frame. The aggregate window still returns a
value for every row, but those for which the FILTER expression evaluates to
other than true are not included in the window frame for any row. For example:

-- The following SELECT statement returns:
-- 
--   c     | a | b | group_concat
---------------------------------
--   one   | 1 | A | A           
--   two   | 2 | B | A           
--   three | 3 | C | A.C         
--   one   | 4 | D | A.C.D       
--   two   | 5 | E | A.C.D       
--   three | 6 | F | A.C.D.F     
--   one   | 7 | G | A.C.D.F.G   
-- 
SELECT c, a, b, group_concat(b, '.') FILTER (WHERE c!='two') OVER (
  ORDER BY a
) AS group_concat
FROM t1 ORDER BY a;


c2: Window Functions
c3: 3. Built-in Window Functions
c5: windowfunctions.html#built_in_window_functions
c4:
As well as aggregate window functions, SQLite features a set of built-in
window functions based on 

those supported by PostgreSQL.

 Built-in window functions honor any PARTITION BY clause in the same way
as aggregate window functions - each selected row is assigned to a partition 
and each partition is processed separately. The ways in which any ORDER BY
clause affects each built-in window function is described below. Some of
the window functions (rank(), dense_rank(), percent_rank() and ntile()) use
the concept of "peer groups" (rows within the same partition that have the
same values for all ORDER BY expressions). In these cases, it does not matter
whether the frame-spec specifies ROWS, GROUPS, or RANGE.
For the purposes of built-in window function processing, rows with the same values
for all ORDER BY expressions are considered peers regardless of the frame type.

 Most built-in window functions ignore the
frame-spec, the exceptions being first_value(),
last_value() and nth_value(). It is a syntax error to specify a FILTER 
clause as part of a built-in window function invocation.



 SQLite supports the following 11 built-in window functions:


  row_number()
   The number of the row within the current partition. Rows are
       numbered starting from 1 in the order defined by the ORDER BY clause in
       the window definition, or in arbitrary order otherwise.
  rank()
   The row_number() of the first peer in each group - the rank of the
       current row with gaps. If there is no ORDER BY clause, then all rows
       are considered peers and this function always returns 1.
  dense_rank()
   The number of the current row's peer group within its partition - the
       rank of the current row without gaps. Rows are numbered starting
       from 1 in the order defined by the ORDER BY clause in the window
       definition. If there is no ORDER BY clause, then all rows are
       considered peers and this function always returns 1.
  percent_rank()
   Despite the name, this function always returns a value between 0.0
       and 1.0 equal to (rank - 1)/(partition-rows - 1), where
       rank is the value returned by built-in window function rank()
       and partition-rows is the total number of rows in the
       partition. If the partition contains only one row, this function
       returns 0.0.
  cume_dist()
   The cumulative distribution. Calculated as
       row-number/partition-rows, where row-number is
       the value returned by row_number() for the last peer in the group
       and partition-rows the number of rows in the partition.
  ntile(N)
   Argument N is handled as an integer. This function divides the
       partition into N groups as evenly as possible and assigns an integer
       between 1 and N to each group, in the order defined by the ORDER
       BY clause, or in arbitrary order otherwise. If necessary, larger groups
       occur first. This function returns the integer value assigned to the
       group that the current row is a part of.

  lag(expr)lag(expr, offset)lag(expr, offset, default)
   The first form of the lag() function returns the result of evaluating
       expression expr against the previous row in the partition. Or, if
       there is no previous row (because the current row is the first), NULL.

       If the offset argument is provided, then it must be a
       non-negative integer. In this case the value returned is the result
       of evaluating expr against the row offset rows before the
       current row within the partition. If offset is 0, then
       expr is evaluated against the current row. If there is no row
       offset rows before the current row, NULL is returned.

       If default is also provided, then it is returned instead of
      NULL if the row identified by offset does not exist.
       
  lead(expr)lead(expr, offset)lead(expr, offset, default)
   The first form of the lead() function returns the result of evaluating
       expression expr against the next row in the partition. Or, if
       there is no next row (because the current row is the last), NULL.

       If the offset argument is provided, then it must be a
       non-negative integer. In this case the value returned is the result
       of evaluating expr against the row offset rows after the
       current row within the partition. If offset is 0, then
       expr is evaluated against the current row. If there is no row
       offset rows after the current row, NULL is returned.

       If default is also provided, then it is returned instead of
      NULL if the row identified by offset does not exist.
  first_value(expr)
   This built-in window function calculates the window frame for each
       row in the same way as an aggregate window function. It returns the
       value of expr evaluated against the first row in the window frame
       for each row.
  last_value(expr)
   This built-in window function calculates the window frame for each
       row in the same way as an aggregate window function. It returns the
       value of expr evaluated against the last row in the window frame
       for each row.
  nth_value(expr, N)
   This built-in window function calculates the window frame for each
       row in the same way as an aggregate window function. It returns the
       value of expr evaluated against the row N of the window
       frame. Rows are numbered within the window frame starting from 1 in
       the order defined by the ORDER BY clause if one is present, or in
       arbitrary order otherwise. If there is no Nth row in the
       partition, then NULL is returned.
  


The examples in this section use the
previously defined T1 table
as well as the following T2 table:

CREATE TABLE t2(a, b);
INSERT INTO t2 VALUES('a', 'one'), 
                     ('a', 'two'), 
                     ('a', 'three'), 
                     ('b', 'four'), 
                     ('c', 'five'), 
                     ('c', 'six');


The following example illustrates the behaviour of the five ranking
functions - row_number(), rank(), dense_rank(), percent_rank() and 
cume_dist().

-- The following SELECT statement returns:
-- 
--   a | row_number | rank | dense_rank | percent_rank | cume_dist
------------------------------------------------------------------
--   a |          1 |    1 |          1 |          0.0 |       0.5
--   a |          2 |    1 |          1 |          0.0 |       0.5
--   a |          3 |    1 |          1 |          0.0 |       0.5
--   b |          4 |    4 |          2 |          0.6 |       0.66
--   c |          5 |    5 |          3 |          0.8 |       1.0
--   c |          6 |    5 |          3 |          0.8 |       1.0
-- 
SELECT a                        AS a,
       row_number() OVER win    AS row_number,
       rank() OVER win          AS rank,
       dense_rank() OVER win    AS dense_rank,
       percent_rank() OVER win  AS percent_rank,
       cume_dist() OVER win     AS cume_dist
FROM t2
WINDOW win AS (ORDER BY a);


The example below uses ntile() to divide the six rows into two groups (the
ntile(2) call) and into four groups (the ntile(4) call). For ntile(2), there
are three rows assigned to each group. For ntile(4), there are two groups of
two and two groups of one. The larger groups of two appear first.

-- The following SELECT statement returns:
-- 
--   a | b     | ntile_2 | ntile_4
----------------------------------
--   a | one   |       1 |       1
--   a | two   |       1 |       1
--   a | three |       1 |       2
--   b | four  |       2 |       2
--   c | five  |       2 |       3
--   c | six   |       2 |       4
-- 
SELECT a                        AS a,
       b                        AS b,
       ntile(2) OVER win        AS ntile_2,
       ntile(4) OVER win        AS ntile_4
FROM t2
WINDOW win AS (ORDER BY a);


 The next example demonstrates lag(), lead(), first_value(), last_value()
and nth_value(). The frame-spec is ignored by 
both lag() and lead(), but respected by first_value(), last_value()
and nth_value().

-- The following SELECT statement returns:
-- 
--   b | lead | lag  | first_value | last_value | nth_value_3
-------------------------------------------------------------
--   A | C    | NULL | A           | A          | NULL       
--   B | D    | A    | A           | B          | NULL       
--   C | E    | B    | A           | C          | C          
--   D | F    | C    | A           | D          | C          
--   E | G    | D    | A           | E          | C          
--   F | n/a  | E    | A           | F          | C          
--   G | n/a  | F    | A           | G          | C          
-- 
SELECT b                          AS b,
       lead(b, 2, 'n/a') OVER win AS lead,
       lag(b) OVER win            AS lag,
       first_value(b) OVER win    AS first_value,
       last_value(b) OVER win     AS last_value,
       nth_value(b, 3) OVER win   AS nth_value_3
FROM t1
WINDOW win AS (ORDER BY b ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)


c2: Window Functions
c3: 4. Window Chaining
c5: windowfunctions.html#window_chaining
c4:
Window chaining is a shorthand that allows one window to be defined in terms 
of another. Specifically, the shorthand allows the new window to implicitly
copy the PARTITION BY and optionally ORDER BY clauses of the base window. For
example, in the following:

SELECT group_concat(b, '.') OVER (
  win ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
)
FROM t1
WINDOW win AS (PARTITION BY a ORDER BY c)



the window used by the group_concat() function is equivalent
to "PARTITION BY a ORDER BY c ROWS BETWEEN UNBOUNDED PRECEDING
AND CURRENT ROW". In order to use window
chaining, all of the following must be true:


  The new window definition must not include a PARTITION BY clause. The
       PARTITION BY clause, if there is one, must be supplied by the base
       window specification.

  If the base window has an ORDER BY clause, it is copied into the new
       window. In this case the new window must not specify an ORDER BY clause.
       If the base window has no ORDER BY clause, one may be specified as part
       of the new window definition.
  
  The base window may not specify a frame specification.  The frame
      specification can only be given in the new window specification.


The two fragments of SQL below are similar, but not entirely equivalent, as
the latter will fail if the definition of window "win" contains a frame
specification.

SELECT group_concat(b, '.') OVER win ...
SELECT group_concat(b, '.') OVER (win) ...


c2: Window Functions
c3: 5. User-Defined Aggregate Window Functions
c5: windowfunctions.html#user_defined_aggregate_window_functions
c4:
User-defined aggregate window functions may be created using the
sqlite3_create_window_function() API. Implementing an aggregate window
function is very similar to an ordinary aggregate function. Any user-defined
aggregate window function may also be used as an ordinary aggregate. To 
implement a user-defined aggregate window function the application must
supply four callback functions:


Callback Description
xStep 
      This method is required by both window aggregate and legacy aggregate
      function implementations. It is invoked to add a row to the current
      window. The function arguments, if any, corresponding to the row being
      added are passed to the implementation of xStep.

xFinal 
      This method is required by both window aggregate and legacy aggregate
      function implementations. It is invoked to return the current value
      of the aggregate (determined by the contents of the current window),
      and to free any resources allocated by earlier calls to xStep.

xValue 
      This method is  only required for window aggregate functions.  The presence
      of this method is what distinguishes a window aggregate function from a
      legacy  aggregate function.  This method is invoked to return the current
      value of the aggregate. Unlike xFinal, the implementation should not
      delete any context.

xInverse 
      This method is only required for window aggregate functions, not legacy
      aggregate function implementations. It is invoked to remove the oldest
      presently aggregated result of xStep from the current window.
      The function arguments, if any, are those
      passed to xStep for the row being removed.


 The C code below implements a simple window aggregate function named
sumint(). This works in the same way as the built-in sum() function, except
that it throws an exception if passed an argument that is not an integer
value.


/*
** xStep for sumint().
**
** Add the value of the argument to the aggregate context (an integer).
*/
static void sumintStep(
  sqlite3_context *ctx, 
  int nArg, 
  sqlite3_value *apArg[]
){
  sqlite3_int64 *pInt;

  assert( nArg==1 );
  if( sqlite3_value_type(apArg[0])!=SQLITE_INTEGER ){
    sqlite3_result_error(ctx, "invalid argument", -1);
    return;
  }
  pInt = (sqlite3_int64*)sqlite3_aggregate_context(ctx, sizeof(sqlite3_int64));
  if( pInt ){
    *pInt += sqlite3_value_int64(apArg[0]);
  }
}

/*
** xInverse for sumint().
**
** This does the opposite of xStep() - subtracts the value of the argument
** from the current context value. The error checking can be omitted from
** this function, as it is only ever called after xStep() (so the aggregate
** context has already been allocated) and with a value that has already
** been passed to xStep() without error (so it must be an integer).
*/
static void sumintInverse(
  sqlite3_context *ctx, 
  int nArg, 
  sqlite3_value *apArg[]
){
  sqlite3_int64 *pInt;
  assert( sqlite3_value_type(apArg[0])==SQLITE_INTEGER );
  pInt = (sqlite3_int64*)sqlite3_aggregate_context(ctx, sizeof(sqlite3_int64));
  *pInt -= sqlite3_value_int64(apArg[0]);
}

/*
** xFinal for sumint().
**
** Return the current value of the aggregate window function. Because
** this implementation does not allocate any resources beyond the buffer
** returned by sqlite3_aggregate_context, which is automatically freed
** by the system, there are no resources to free. And so this method is
** identical to xValue().
*/
static void sumintFinal(sqlite3_context *ctx){
  sqlite3_int64 res = 0;
  sqlite3_int64 *pInt;
  pInt = (sqlite3_int64*)sqlite3_aggregate_context(ctx, 0);
  if( pInt ) res = *pInt;
  sqlite3_result_int64(ctx, res);
}

/*
** xValue for sumint().
**
** Return the current value of the aggregate window function.
*/
static void sumintValue(sqlite3_context *ctx){
  sqlite3_int64 res = 0;
  sqlite3_int64 *pInt;
  pInt = (sqlite3_int64*)sqlite3_aggregate_context(ctx, 0);
  if( pInt ) res = *pInt;
  sqlite3_result_int64(ctx, res);
}

/*
** Register sumint() window aggregate with database handle db. 
*/
int register_sumint(sqlite3 *db){
  return sqlite3_create_window_function(db, "sumint", 1, SQLITE_UTF8, 0,
      sumintStep, sumintFinal, sumintValue, sumintInverse, 0
  );
}




 The following example uses the sumint() function implemented by the above
C code. For each row, the window consists of the preceding row (if any), the current row and the following row (again, if any):

CREATE TABLE t3(x, y);
INSERT INTO t3 VALUES('a', 4),
                     ('b', 5),
                     ('c', 3),
                     ('d', 8),
                     ('e', 1);

-- Assuming the database is populated using the above script, the 
-- following SELECT statement returns:
-- 
--   x | sum_y
--------------
--   a | 9    
--   b | 12   
--   c | 16   
--   d | 12   
--   e | 9    
-- 
SELECT x, sumint(y) OVER (
  ORDER BY x ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING
) AS sum_y
FROM t3 ORDER BY x;


In processing the query above, SQLite invokes the sumint callbacks as
follows:



   xStep(4) - add "4" to the current window.
   xStep(5) - add "5" to the current window.
   xValue() - invoke xValue() to obtain the value of sumint() for
       the row with (x='a'). The window currently consists of values 4 and 5, 
       and so the result is 9.
   xStep(3) - add "3" to the current window.
   xValue() - invoke xValue() to obtain the value of sumint() for
       the row with (x='b'). The window currently consists of values 4, 5 and
       3, and so the result is 12.
   xInverse(4) - remove "4" from the window. 
   xStep(8) - add "8" to the current window. The window now consists
       of values 5, 3 and 8.
   xValue() - invoked to obtain the value for the row with (x='c').
       In this case, 16.
   xInverse(5) - remove value "5" from the window.
   xStep(1) - add value "1" to the window.
   xValue() - invoked to obtain the value for row (x='d').
   xInverse(3) - remove value "3" from the window. The window now
       contains values 8 and 1 only.
   xValue() - invoked to obtain the value for row (x='e'). 9.
   xFinal() - invoked to reclaim any allocated resources.


c2: Window Functions
c3: 6. History
c5: windowfunctions.html#history
c4:
Window function support was first added to SQLite with release
version 3.25.0 (2018-09-15). The SQLite developers used
the PostgreSQL window function
documentation as their primary reference for how window functions
ought to behave.  Many test cases have been run against PostgreSQL
to ensure that window functions operate the same way in both
SQLite and PostgreSQL.

In SQLite version 3.28.0 (2019-04-16),
windows function support was extended to include the EXCLUDE clause,
GROUPS frame types, window chaining, and support for
"&lt;expr&gt; PRECEDING" and "&lt;expr&gt; FOLLOWING" boundaries
in RANGE frames.


c2: 35% Faster Than The Filesystem
c3: 1. Summary
c5: fasterthanfs.html#summary
c4:
SQLite reads and writes small blobs (for example, thumbnail images)
35% faster&sup1; than the same blobs
can be read from or written to individual files on disk using
fread() or fwrite().

Furthermore, a single SQLite database holding
10-kilobyte blobs uses about 20% less disk space than
storing the blobs in individual files.

The performance difference arises (we believe) because when
working from an SQLite database, the open() and close() system calls
are invoked only once, whereas
open() and close() are invoked once for each blob
when using blobs stored in individual files.  It appears that the
overhead of calling open() and close() is greater than the overhead
of using the database.  The size reduction arises from the fact that
individual files are padded out to the next multiple of the filesystem
block size, whereas the blobs are packed more tightly into an SQLite
database.


The measurements in this article were made during the week of 2017-06-05
using a version of SQLite in between 3.19.2 and 3.20.0.  You may expect
future versions of SQLite to perform even better.


c2: 35% Faster Than The Filesystem
c3: 1.1. Caveats
c5: fasterthanfs.html#caveats
c4:
&sup1;The 35% figure above is approximate.  Actual timings vary
depending on hardware, operating system, and the
details of the experiment, and due to random performance fluctuations
on real-world hardware.  See the text below for more detail.
Try the experiments yourself.  Report significant deviations on
the SQLite forum.



The 35% figure is based on running tests on every machine
that the author has easily at hand.
Some reviewers of this article report that SQLite has higher 
latency than direct I/O on their systems.  We do not yet understand
the difference.  We also see indications that SQLite does not
perform as well as direct I/O when experiments are run using
a cold filesystem cache.


So let your take-away be this: read/write latency for
SQLite is competitive with read/write latency of individual files on
disk.  Often SQLite is faster.  Sometimes SQLite is almost
as fast.  Either way, this article disproves the common
assumption that a relational database must be slower than direct
filesystem I/O.


c2: 35% Faster Than The Filesystem
c3: 1.2. Related Studies
c5: fasterthanfs.html#related_studies
c4:
Jim Gray
and others studied the read performance of BLOBs
versus file I/O for Microsoft SQL Server and found that reading BLOBs 
out of the 
database was faster for BLOB sizes less than between 250KiB and 1MiB.
(Paper).
In that study, the database still stores the filename of the content even
if the content is held in a separate file.  So the database is consulted
for every BLOB, even if it is only to extract the filename.  In this
article, the key for the BLOB is the filename, so no preliminary database
access is required.  Because the database is never used at all when
reading content from individual files in this article, the threshold
at which direct file I/O becomes faster is smaller than it is in Gray's
paper.


The Internal Versus External BLOBs article on this website is an
earlier investigation (circa 2011) that uses the same approach as the
Jim Gray paper &mdash; storing the blob filenames as entries in the
database &mdash; but for SQLite instead of SQL Server.


c2: 35% Faster Than The Filesystem
c3: 2. How These Measurements Are Made
c5: fasterthanfs.html#how_these_measurements_are_made
c4:
I/O performance is measured using the
kvtest.c program
from the SQLite source tree.
To compile this test program, first gather the kvtest.c source file
into a directory with the SQLite amalgamation source
files "sqlite3.c" and "sqlite3.h".  Then on unix, run a command like
the following:

gcc -Os -I. -DSQLITE_DIRECT_OVERFLOW_READ &#92;
  kvtest.c sqlite3.c -o kvtest -ldl -lpthread


Or on Windows with MSVC:

cl -I. -DSQLITE_DIRECT_OVERFLOW_READ kvtest.c sqlite3.c


Instructions for compiling for Android
are shown below.


Use the resulting "kvtest" program to
generate a test database with 100,000 random uncompressible
blobs, each with a random
size between 8,000 and 12,000 bytes
using a command like this:

./kvtest init test1.db --count 100k --size 10k --variance 2k



If desired, you can verify the new database by running this command:

./kvtest stat test1.db



Next, make copies of all the blobs into individual files in a directory
using a command like this:

./kvtest export test1.db test1.dir



At this point, you can measure the amount of disk space used by
the test1.db database and the space used by the test1.dir directory
and all of its content.  On a standard Ubuntu Linux desktop, the
database file will be 1,024,512,000 bytes in size and the test1.dir
directory will use 1,228,800,000 bytes of space (according to "du -k"),
about 20% more than the database.


The "test1.dir" directory created above puts all the blobs into a single
folder.  It was conjectured that some operating systems would perform 
poorly when a single directory contains 100,000 objects.  To test this,
the kvtest program can also store the blobs in a hierarchy of folders with no
more than 100 files and/or subdirectories per folder.  The alternative
on-disk representation of the blobs can be created using the --tree
command-line option to the "export" command, like this:

./kvtest export test1.db test1.tree --tree



The test1.dir directory will contain 100,000 files
with names like "000000", "000001", "000002" and so forth but the
test1.tree directory will contain the same files in subdirectories like
"00/00/00", "00/00/01", and so on.  The test1.dir and test1.test
directories take up approximately the same amount of space, though
test1.test is very slightly larger due to the extra directory entries.


All of the experiments that follow operate the same with either 
"test1.dir" or "test1.tree".  Very little performance difference is
measured in either case, regardless of operating system.


Measure the performance for reading blobs from the database and from
individual files using these commands:

./kvtest run test1.db --count 100k --blob-api
./kvtest run test1.dir --count 100k --blob-api
./kvtest run test1.tree --count 100k --blob-api



Depending on your hardware and operating system, you should see that reads 
from the test1.db database file are about 35% faster than reads from 
individual files in the test1.dir or test1.tree folders.  Results can vary
significantly from one run to the next due to caching, so it is advisable
to run tests multiple times and take an average or a worst case or a best
case, depending on your requirements.

The --blob-api option on the database read test causes kvtest to use
the sqlite3_blob_read() feature of SQLite to load the content of the
blobs, rather than running pure SQL statements.  This helps SQLite to run
a little faster on read tests.  You can omit that option to compare the
performance of SQLite running SQL statements.
In that case, the SQLite still out-performs direct reads, though
by not as much as when using sqlite3_blob_read().
The --blob-api option is ignored for tests that read from individual disk
files.


Measure write performance by adding the --update option.  This causes
the blobs are overwritten in place with another random blob of
exactly the same size.

./kvtest run test1.db --count 100k --update
./kvtest run test1.dir --count 100k --update
./kvtest run test1.tree --count 100k --update



The writing test above is not completely fair, since SQLite is doing
power-safe transactions whereas the direct-to-disk writing is not.
To put the tests on a more equal footing, add either the --nosync
option to the SQLite writes to disable calling fsync() or
FlushFileBuffers() to force content to disk, or using the --fsync option
for the direct-to-disk tests to force them to invoke fsync() or
FlushFileBuffers() when updating disk files.


By default, kvtest runs the database I/O measurements all within
a single transaction.  Use the --multitrans option to run each blob
read or write in a separate transaction.  The --multitrans option makes
SQLite much slower, and uncompetitive with direct disk I/O.  This
option proves, yet again, that to get the most performance out of
SQLite, you should group as much database interaction as possible within
a single transaction.


There are many other testing options, which can be seen by running
the command:

./kvtest help


c2: 35% Faster Than The Filesystem
c3: 2.1. Read Performance Measurements
c5: fasterthanfs.html#read_performance_measurements
c4:
The chart below shows data collected using 
kvtest.c on five different
systems:


Win7: A circa-2009 Dell Inspiron laptop, Pentium dual-core
    at 2.30GHz, 4GiB RAM, Windows7.
Win10: A 2016 Lenovo YOGA 910, Intel i7-7500 at 2.70GHz,
    16GiB RAM, Windows10.
Mac: A 2015 MacBook Pro, 3.1GHz intel Core i7, 16GiB RAM,
    MacOS 10.12.5
Ubuntu: Desktop built from Intel i7-4770K at 3.50GHz, 32GiB RAM,
    Ubuntu 16.04.2 LTS
Android: Galaxy S3, ARMv7, 2GiB RAM


All machines use SSD except Win7 which has a
hard-drive. The test database is 100K blobs with sizes uniformly
distributed between 8K and 12K, for a total of about 1 gigabyte
of content.  The database page size
is 4KiB.  The -DSQLITE_DIRECT_OVERFLOW_READ compile-time option was
used for all of these tests.
Tests were run multiple times.
The first run was used to warm up the cache and its timings were discarded.


The chart below shows average time to read a blob directly from the
filesystem versus the time needed to read the same blob from the SQLite 
database.
The actual timings vary considerably from one system to another 
(the Ubuntu desktop is much
faster than the Galaxy S3 phone, for example).  
This chart shows the ratio of the
times needed to read blobs from a file divided by the time needed to
from the database.  The left-most column in the chart is the normalized
time to read from the database, for reference.


In this chart, an SQL statement ("SELECT v FROM kv WHERE k=?1") 
is prepared once.  Then for each blob, the blob key value is bound 
to the ?1 parameter and the statement is evaluated to extract the
blob content.


The chart shows that on Windows10, content can be read from the SQLite
database about 5 times faster than it can be read directly from disk.
On Android, SQLite is only about 35% faster than reading from disk.






Chart 1:  SQLite read latency relative to direct filesystem reads.
100K blobs, avg 10KB each, random order using SQL



The performance can be improved slightly by bypassing the SQL layer
and reading the blob content directly using the
sqlite3_blob_read() interface, as shown in the next chart:






Chart 2:  SQLite read latency relative to direct filesystem reads.
100K blobs, avg size 10KB, random order
using sqlite3_blob_read().



Further performance improves can be made by using the
memory-mapped I/O feature of SQLite.  In the next chart, the
entire 1GB database file is memory mapped and blobs are read
(in random order) using the sqlite3_blob_read() interface.
With these optimizations, SQLite is twice as fast as Android
or MacOS-X and over 10 times faster than Windows.






Chart 3:  SQLite read latency relative to direct filesystem reads.
100K blobs, avg size 10KB, random order
using sqlite3_blob_read() from a memory-mapped database.



The third chart shows that reading blob content out of SQLite can be
twice as fast as reading from individual files on disk for Mac and
Android, and an amazing ten times faster for Windows.


c2: 35% Faster Than The Filesystem
c3: 2.2. Write Performance Measurements
c5: fasterthanfs.html#write_performance_measurements
c4:
Writes are slower.
On all systems, using both direct I/O and SQLite, write performance is
between 5 and 15 times slower than reads.


Write performance measurements were made by replacing (overwriting)
an entire blob with a different blob.  All of the blobs in these
experiment are random and incompressible.  Because writes are so much
slower than reads, only 10,000 of the 100,000 blobs in the database
are replaced.  The blobs to be replaced are selected at random and
are in no particular order.


The direct-to-disk writes are accomplished using fopen()/fwrite()/fclose().
By default, and in all the results shown below, the OS filesystem buffers are
never flushed to persistent storage using fsync() or
FlushFileBuffers().  In other words, there is no attempt to make the
direct-to-disk writes transactional or power-safe.
We found that invoking fsync() or FlushFileBuffers() on each file
written causes direct-to-disk storage
to be about 10 times or more slower than writes to SQLite.


The next chart compares SQLite database updates in WAL mode
against raw direct-to-disk overwrites of separate files on disk.
The PRAGMA synchronous setting is NORMAL.
All database writes are in a single transaction.
The timer for the database writes is stopped after the transaction
commits, but before a checkpoint is run.
Note that the SQLite writes, unlike the direct-to-disk writes,
are transactional and power-safe, though because the synchronous
setting is NORMAL instead of FULL, the transactions are not durable.






Chart 4:  SQLite write latency relative to direct filesystem writes.
10K blobs, avg size 10KB, random order,
WAL mode with synchronous NORMAL,
exclusive of checkpoint time



The android performance numbers for the write experiments are omitted
because the performance tests on the Galaxy S3 are so random.  Two
consecutive runs of the exact same experiment would give wildly different
times.  And, to be fair, the performance of SQLite on android is slightly
slower than writing directly to disk.


The next chart shows the performance of SQLite versus direct-to-disk
when transactions are disabled (PRAGMA journal_mode=OFF)
and PRAGMA synchronous is set to OFF.  These settings put SQLite on an
equal footing with direct-to-disk writes, which is to say they make the
data prone to corruption due to system crashes and power failures.






Chart 5:  SQLite write latency relative to direct filesystem writes.
10K blobs, avg size 10KB, random order,
journaling disabled, synchronous OFF.



In all of the write tests, it is important to disable anti-virus software
prior to running the direct-to-disk performance tests.  We found that
anti-virus software slows down direct-to-disk by an order of magnitude
whereas it impacts SQLite writes very little.  This is probably due to the
fact that direct-to-disk changes thousands of separate files which all need
to be checked by anti-virus, whereas SQLite writes only changes the single
database file.


c2: 35% Faster Than The Filesystem
c3: 2.3. Variations
c5: fasterthanfs.html#variations
c4:
The -DSQLITE_DIRECT_OVERFLOW_READ compile-time option causes SQLite
to bypass its page cache when reading content from overflow pages.  This
helps database reads of 10K blobs run a little faster, but not all that much
faster.  SQLite still holds a speed advantage over direct filesystem reads
without the SQLITE_DIRECT_OVERFLOW_READ compile-time option.

Other compile-time options such as using -O3 instead of -Os or
using -DSQLITE_THREADSAFE=0 and/or some of the other
recommended compile-time options might help SQLite to run even faster
relative to direct filesystem reads.

The size of the blobs in the test data affects performance.
The filesystem will generally be faster for larger blobs, since
the overhead of open() and close() is amortized over more bytes of I/O,
whereas the database will be more efficient in both speed and space
as the average blob size decreases.


c2: 35% Faster Than The Filesystem
c3: 3. General Findings
c5: fasterthanfs.html#general_findings
c4:
SQLite is competitive with, and usually faster than, blobs stored in
separate files on disk, for both reading and writing.


SQLite is much faster than direct writes to disk on Windows
when anti-virus protection is turned on.  Since anti-virus software
is and should be on by default in Windows, that means that SQLite
is generally much faster than direct disk writes on Windows.


Reading is about an order of magnitude faster than writing, for all
systems and for both SQLite and direct-to-disk I/O.


I/O performance varies widely depending on operating system and hardware.
Make your own measurements before drawing conclusions.


Some other SQL database engines advise developers to store blobs in separate
files and then store the filename in the database.  In that case, where
the database must first be consulted to find the filename before opening
and reading the file, simply storing the entire blob in the database
gives much faster read and write performance with SQLite.
See the Internal Versus External BLOBs article for more information.


c2: 35% Faster Than The Filesystem
c3: 4.1. Compiling And Testing on Android
c5: fasterthanfs.html#compiling_and_testing_on_android
c4:
The kvtest program is compiled and run on Android as follows.
First install the Android SDK and NDK.  Then prepare a script
named "android-gcc" that looks approximately like this:

#!/bin/sh
#
NDK=/home/drh/Android/Sdk/ndk-bundle
SYSROOT=$NDK/platforms/android-16/arch-arm
ABIN=$NDK/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin
GCC=$ABIN/arm-linux-androideabi-gcc
$GCC --sysroot=$SYSROOT -fPIC -pie $*


Make that script executable and put it on your $PATH.  Then
compile the kvtest program as follows:

android-gcc -Os -I. kvtest.c sqlite3.c -o kvtest-android


Next, move the resulting kvtest-android executable to the Android
device:

adb push kvtest-android /data/local/tmp


Finally use "adb shell" to get a shell prompt on the Android device,
cd into the /data/local/tmp directory, and begin running the tests
as with any other unix host.


c2: Imposter Tables
c3: 1. Introduction
c5: imposter.html#introduction
c4:
An imposter table is a table that is attached to the same b-tree
as an index.
An imposter table allows the content of an index to be queried or
modified as if the index were an ordinary table.



Imposter tables are intended for analysis and debugging only.
This is not a feature that most application developers should understand
or even know about.
Imposter tables are for experts only.



Improper use of imposter tables can cause index corruption, though
any corruption created this way can be fixed by running REINDEX.


c2: Imposter Tables
c3: 2. Details
c5: imposter.html#details
c4:
Each table and each index in SQLite is stored in a separate b-tree
in the database file.  Each b-tree is identified by its root page
number.  The root page number for any index or table can be found
by querying the "rootpage" column of the sqlite_schema table.
See the indexing tutorial and the file format documentation
for further background on this design.



Usually the b-trees for tables and indexes are slightly different.
A table b-tree contains a 64-bit integer key and arbitrary data.
The 64-bit integer key is the ROWID.  Index b-trees contain
an arbitrary binary key and no data.  So table b-trees and index
b-trees are not directly compatible.



However, the b-tree for a WITHOUT ROWID table is in the same format
as an index b-tree.  Thus, an index b-tree can be accessed as if it
were a WITHOUT ROWID table.


c2: Imposter Tables
c3: 2.1. Manually Created Imposter Tables
c5: imposter.html#manually_created_imposter_tables
c4:
One way to create an imposter table is to directly edit the sqlite_schema
table to insert a new row that describes the table.
For example, suppose the schema is like this:


CREATE TABLE t1(a INTEGER PRIMARY KEY,b TEXT,c INT, d INT);
CREATE INDEX t1bc ON t1(b,c);



The WITHOUT ROWID table that has the same structure as the t1bc index
would look like this:


CREATE TABLE t2(b TEXT,c INT,a INT, PRIMARY KEY(b,c,a)) WITHOUT ROWID;



To create a permanent imposter table "t2" against index "t1bc" one
should first enable editing of the sqlite_schema table by running
"PRAGMA writable_schema=ON".
(Be careful to observe the warnings that accompany this PRAGMA.
A mistake can cause severe database corruption.)
Then insert a new entry into the sqlite_schema table like this:


INSERT INTO sqlite_schema(type,name,tbl_name,rootpage,sql)
 SELECT 'table','t2','t2',rootpage,
   'CREATE TABLE t2(b,c,a,PRIMARY KEY(b,c,a))WITHOUT ROWID'
   FROM sqlite_schema
  WHERE name='t1bc';



The INSERT statement above adds a new row to the sqlite_schema table that
defines a table "t2" having the same on-disk format as index "t1bc" and 
pointing to the same b-tree.
After adding this sqlite_schema table entry, it is necessary to close and
reopen the database in order to get SQLite to reread the schema.
Then the "t2" table can be queried to see the content of the "t1bc" index.


c2: Imposter Tables
c3: 2.1.1. Corrupted Database
c5: imposter.html#corrupted_database
c4:
A serious problem with the manual imposter table approach described above
is that after adding the new "t2" entry to the "sqlite_schema" table, the
database file will technically be corrupt.  Both the "t1bc" index and the
"t2" table will point to the same b-tree.  This will not cause
any immediate problems, though one should avoid running VACUUM.



It is possible to write into the "t2" table, thus changing the content
of the index.
But doing so will get the "t1bc" index out of synchronization with its
parent table "t1".  An out-of-sync index can result in incorrect query
results.



Since the "t2" imposter table is a form of database corruption, the
manual approach to creating imposter tables is not recommended.
Actually, any use of imposter tables is discouraged for all but
expert developers, but manually created imposter tables are
especially discouraged because they are permanent.


c2: Imposter Tables
c3: 2.2. Transient Imposter Tables
c5: imposter.html#transient_imposter_tables
c4:
Another (safer) approach to creating an imposter table is to add an
entry for the imposter table to SQLite's internal symbol table without
updating the "sqlite_schema" table on disk.
That way, the imposter table exists in only a single database connection
and is automatically removed whenever the schema is reloaded.



Creation of a transient imposter table involves a special
sqlite3_test_control() call.  Unlike all other SQLite APIs,
sqlite3_test_control() interface is subject to incompatible changes
from one release to the next, and so the mechanism described below
is not guaranteed to work in future releases of SQLite.  The
SQLite developers do not consider this a problem because imposter
tables should not be used in applications.  Imposter tables are for
analysis and testing use only.



To create a transient imposter table, first call sqlite3_test_control()
as follows:


sqlite3_test_control(SQLITE_TESTCTRL_IMPOSTER, db, "main", 1, tnum);



The "db" parameter is a pointer to the database connection.
The "main" argument is the name of the schema in which the imposter
table is to be created.  The "1" argument enables the imposter table
mechanism.  "tnum" is the root page of the index that the imposter
table should mirror.



After the sqlite3_test_control() call above, then run a CREATE TABLE
statement the defines the imposter table.
With the imposter mechanism enabled, this CREATE TABLE statement does
not create a real table but instead merely adds an entry in SQLite's
internal symbol table.  Note that the CREATE TABLE statement must
be in the correct format for the index.  If the imposter table has the
wrong number of columns or is not a WITHOUT ROWID table or is otherwise
incompatible with the index b-tree, then SQLITE_CORRUPT errors will result
when the imposter table is used.



After running the CREATE TABLE statement, disable the imposter mechanism
as follows:


sqlite3_test_control(SQLITE_TESTCTRL_IMPOSTER, db, "main", 0, 0);



In other words, make the same sqlite3_test_control() call except change
the last two parameters to zero.



After the imposter table is loaded into SQLite's internal schema as
described above, the imposter table can be used as any other table.
But the imposter table will only be visible to the one database
connection that created it.  No changes are made to the database file
on disk.  And the imposter table will disappear the next time the schema 
is loaded.


c2: Imposter Tables
c3: 2.3. The .imposter Shell Command
c5: imposter.html#the_imposter_shell_command
c4:
As of SQLite 3.16.0 (2017-01-02), the command-line shell contains
a dot-command ".imposter" that does all of the work of setting up a
transient imposter table.
Instead of making multiple calls to sqlite3_test_control() and figuring
out and invoking a compatible CREATE TABLE statement, a transient
imposter table can be constructed as follows:


.imposter t1bc t2



Of course, substitute the desired index and imposter table names in
place of the "t1bc" and "t2" shown in the example.
The ".imposter" command reads the schema of the "t1bc" index, uses
that information to construct a compatible CREATE TABLE statement for
the imposter table, then makes all the necessary calls to create the
transient imposter table automatically.


c2: Imposter Tables
c3: 3. Summary And Final Warning
c5: imposter.html#summary_and_final_warning
c4:
The imposter table mechanism is a power analysis and debugging tool
for SQLite.  But as with all sharp tools, it can also be dangerous and
can result in corrupt database files if misused.  Do not attempt to
use imposter tables in an application.  Imposter tables are intended
for use in the laboratory by experts.


c2: Maintaining Private Branches Of SQLite
c3: 1. Introduction
c5: privatebranch.html#_introduction
c4:
SQLite is designed to meet most developer's needs without any 
changes or customization.  When changes are needed, they can normally
be accomplished using start-time (1)
or runtime
(2)
(3)
(4) configuration methods
or via compile-time options.  It is very rare that an application 
developer will need to edit the SQLite source code in order to 
incorporate SQLite into a product.

We call custom modifications to the SQLite source code that are held
for the use of a single application a "private branch".  When a private
branch becomes necessary, the application developer must take on the
task of keeping the private branch in synchronization with the public
SQLite sources.  This is tedious.  It can also be tricky, since while
the SQLite file format and published interfaces are very stable, the
internal implementation of SQLite changes quite rapidly.  Hundreds or
thousands of lines of code might change for any given SQLite point release.


This article outlines one possible method for keeping a private branch
of SQLite in sync with the public SQLite source code.
There are many ways of maintaining a private branch, of course.
Nobody is compelled to use the method describe here.
This article is not trying to impose a particular procedure on 
maintainers of private branches.  The point of this article is to offer
an example of one process for maintaining a private branch which can
be used as a template for designing processes best suited for the
circumstances of each individual project.


c2: Maintaining Private Branches Of SQLite
c3: 2. The Basic Idea
c5: privatebranch.html#_the_basic_idea
c4:
We propose to use the
fossil software configuration management
system to set up two branches.  One branch (the "public branch" or "trunk")
contains the published SQLite sources and the other branch is the 
private branch which contains the code that is customized for the project.
Whenever a new public release of SQLite is made, that release is added to
the public branch and then the changes are merged into the private branch.

This document proposes to use
fossil,
but any other distributed software configuration management system such as
monotone or
mercurial (a.k.a. "hg"), or
git could serve just as well.
The concept will be the same, 
though the specifics of the procedure will vary.

The diagram at the right illustrates the concept.
One begins with a standard SQLite release.  For the
sake of example, suppose that one intends to create a
private branch off of SQLite version 3.6.15.  In the
diagram this is version (1).  The
maintainer makes an exact copy of the baseline
SQLite into the branch space, shown as version (2).
Note that (1) and (2) are exactly the same.  Then
the maintainer applies the private changes to 
version (2) resulting in version (3).  In other words,
version (3) is SQLite version 3.6.15 plus edits.

Later, SQLite version 3.6.16 is released, as shown 
by circle (4) in the diagram.  At the point, the private
branch maintainer does a merge which takes all of the
changes going from (1) to (4) and applies those changes to
(3).  The result is version (5), which is SQLite 3.6.16
plus edits.

There might be merge conflicts.  In other words, it might
be that the changes from (2) to (3) are incompatible with the
changes from (1) to (4).  In that case, the maintainer will
have to manually resolve the conflicts.  Hopefully conflicts
will not come up that often.  Conflicts are less likely to
occur when the private edits are kept to a minimum.

The cycle above can be repeated many times.  The
diagram shows a third SQLite release, 3.6.17 in
circle (6).  The private branch maintainer can do
another merge in order to incorporate the changes
moving from (4) to (6) into the private branch, resulting
in version (7).


c2: Maintaining Private Branches Of SQLite
c3: 3. The Procedure
c5: privatebranch.html#_the_procedure
c4:
The remainder of this document will guide the reader through
the steps needed to maintain a private branch.  The general idea
is the same as outlined above.  This section merely provides more
detail.

We emphasize again that these steps are not intended to be the only
acceptable method for maintaining private branch.  This approach
is one of many.  Use this document as a baseline for preparing
project-specific procedures.  Do not be afraid to experiment.


c2: Maintaining Private Branches Of SQLite
c3: 3.1. Obtain The Software
c5: privatebranch.html#_obtain_the_software
c4:
Fossil is a computer program
that must be installed on your machine before you use it.
Fortunately, installing fossil is very easy.  Fossil is a single
"*.exe" file that you simply download and run.  To uninstall fossil,
simply delete the exe file.  
Detailed instructions for installing and getting started with
fossil are available on the 
fossil website.


c2: Maintaining Private Branches Of SQLite
c3: 3.2. Create A Project Repository
c5: privatebranch.html#_create_a_project_repository
c4:
Create a fossil repository to host the private branch using the
following command:


fossil new private-project.fossil


You can call your project anything you like.  The ".fossil"
suffix is optional.  For this document, we will continue to call the
project "private-project.fossil".  Note that
private-project.fossil is an ordinary disk file (actually an
SQLite database) that will contain your complete project history.
You can make a backup of the project simply by making a copy of that
one file.

If you want to configure the new project, type:


fossil ui private-project.fossil


The "ui" command will cause fossil to run a miniature built-in webserver
and to launch your web-browser pointing
at that webserver.  You can use your web-browser to configure your project
in various ways.  See the instructions on the fossil website for additional
information.

Once the project repository is created, create an open checkout of the
project by moving to the directory where you want to keep all of the
project source code and typing:


fossil open private-project.fossil


You can have multiple checkouts of the same project if you want. 
And you can "clone" the repository to different machines so that multiple
developers can use it.  See the fossil website for further information.


c2: Maintaining Private Branches Of SQLite
c3: 3.3. Installing The SQLite Baseline In Fossil
c5: privatebranch.html#_installing_the_sqlite_baseline_in_fossil
c4:
The repository created in the previous step is initially empty.  The
next step is to load the baseline SQLite release - circle (1) in the diagram
above.

Begin by obtaining a copy of SQLite in whatever form you use it.
The public SQLite you obtain should be as close to your private edited
copy as possible.  If your project uses the SQLite amalgamation, then
get a copy of the amalgamation.  If you use the preprocessed separate
source files, get those instead.  Put all the source files in the
checkout directory created in the previous step.

The source code in public SQLite releases uses unix line endings
(ASCII code 10: "newline" only, NL) and spaces instead of tabs.  If you will
be changing the line ending to windows-style line endings
(ASCII codes 13, 10: "carriage-return" and "newline"; CR-NL) or if you will be
changing space indents into tab indents, make that change now
before you check in the baseline.  The merging process will only work
well if the differences between the public and the private branches are
minimal.  If every single line of the source file is changed in the
private branch because you changed from NL to CR-NL line endings, then
the merge steps will not work correctly.

Let us assume that you are using the amalgamation source code.
Add the baseline to your project as follows:


fossil add sqlite3.c sqlite3.h


If you are using separate source files, name all of the source files instead
of just the two amalgamation source files.  Once this is done, commit your
changes as follows:


fossil commit


You will be prompted for a check-in comment.  Say whatever you like.
After the commit completes, your baseline will be part of the repository.
The following command, if you like, to see this on the "timeline":



fossil ui


That last command is the same "ui" command that we ran before.  It
starts a mini-webserver running and points your web browser at it.  But
this time we didn't have to specify the repository file because we are
located inside a checkout and so fossil can figure out the repository for
itself.  If you want to type in the repository filename as the second
argument, you can.  But it is optional.

If you do not want to use your web browser to view the new check-in,
you can get some information from the command-line using commands like
these:


fossil timeline
fossil info
fossil status


c2: Maintaining Private Branches Of SQLite
c3: 3.4. Creating The Private Branch
c5: privatebranch.html#_creating_the_private_branch
c4:
The previous step created circle (1) in the diagram above.
This step will create circle (2).  Run the following command:

 
fossil branch new private trunk -bgcolor "#add8e8"


This command will create a new branch named "private" (you can use
a different name if you like) and assign it a background color
of light blue ("#add8e8").  You can omit the background color if you want,
though having a distinct background does make it easier to tell the
branch from the "trunk" (the public branch) on timeline displays.  You
can change the background color of the private branch or of the public
branch (the "trunk") using the web interface if you like.

The command above created the new branch.  But your checkout is
still on the trunk - a fact you can see by running the command:


fossil info


To change your check-out to the private branch, type:


fossil update private


You can run the "info" command again to verify that you are on the
private branch.  To go back to the public branch, type:


fossil update trunk


Normally, fossil will modify all the files in your checkout when switching
between the private and the public branches.  But at this point, the files
are identical in both branches so no modifications need to be made.


c2: Maintaining Private Branches Of SQLite
c3: 3.5. Adding Customizations To The Code In The Private Branch
c5: privatebranch.html#_adding_customizations_to_the_code_in_the_private_branch
c4:
Now it is time to make the private, custom modifications to SQLite
which are the whole point of this exercise.  Switch to the private branch
(if you are not already there) using the "fossil update private"
command, then bring up the source files in your text editor and make
whatever changes you want to make.  Once you have finished making
changes, commit those changes using this command:


fossil commit


You will be prompted once again to enter a commit describing your
changes.  Then the commit will occur.  The commit creates a new checkin
in the repository that corresponds to circle (3) in the diagram above.

Now that the public and private branches are different, you can run
the "fossil update trunk" and "fossil update private"
commands and see that fossil really does change the files in the checkout
as you switch back and forth between branches.

Note that in the diagram above, we showed the private edits as a single
commit.  This was for clarity of presentation only.  There is nothing to stop
you from doing dozens or hundreds of separate tiny changes and committing
each separately.  In fact, making many small changes is the preferred way
to work.  The only reason for doing all the changes in a single commit
is that it makes the diagram easier to draw.


c2: Maintaining Private Branches Of SQLite
c3: 3.6. Incorporating New Public SQLite Releases
c5: privatebranch.html#_incorporating_new_public_sqlite_releases
c4:
Suppose that after a while (about a month, usually) a new version of
SQLite is released: 3.6.16.  You will want to incorporate this new
public version of SQLite into your repository in the public branch (the
trunk).  To do this, first change your repository over to the trunk:


fossil update trunk


Then download the new version of the SQLite sources and overwrite the
files that are in the checkout.

If you made NL to CR-NL line ending changes or space to tab
indentation changes in the original baseline, make the same changes
to the new source file.

Once everything is ready, run the "fossil commit" command to
check in the changes.  This creates circle (4) in the diagram above.


c2: Maintaining Private Branches Of SQLite
c3: 3.7. Merging Public SQLite Updates Into The Private Branch
c5: privatebranch.html#_merging_public_sqlite_updates_into_the_private_branch
c4:
The next step is to move the changes in the public branch over into
the private branch.  In other words, we want to create circle (5) in the
diagram above.  Begin by changing to the private branch using
"fossil update private".  Then type this command:


fossil merge trunk


The "merge" command attempts to apply all the changes between
circles (1) and (4) to the files in the local checkout.  Note that
circle (5) has not been created yet.  You will need to run the
"commit" to create circle (5).

It might be that there are conflicts in the merge.  Conflicts
occur when the same line of code was changed in different ways between
circles (1) and (4) versus circles (2) and (3).  The merge command will
announce any conflicts and will include both versions of the conflicting
lines in the output.  You will need to bring up the files that contain
conflicts and manually resolve the conflicts.

After resolving conflicts, many users like to compile and test the
new version before committing it to the repository.  Or you can commit
first and test later.  Either way, run the "fossil commit"
command to check-in the circle (5) version.


c2: Maintaining Private Branches Of SQLite
c3: 3.8. Further Updates
c5: privatebranch.html#_further_updates
c4:
As new versions of SQLite are released, repeat steps 3.6 and 3.7 to
add changes in the new release to the private branch.
Additional private changes can be
made on the private branch in between releases if desired.


c2: Maintaining Private Branches Of SQLite
c3: 4. Variations
c5: privatebranch.html#_variations
c4:
Since this document was first written, the canonical SQLite source code
has been moved from the venerable CVS system into a Fossil repository at
http://www.sqlite.org/src.  This means that if you are working with
canonical SQLite source code (as opposed to the amalgamation source code
files, sqlite3.c and sqlite3.h) then you can create a private repository
simply by cloning the official repository:


fossil clone http://www.sqlite.org/src private-project.fossil


This command both creates the new repository and populates it with
all the latest SQLite code.  You can then create a private branch as
described in section 3.4.

When the private repository is created by cloning, incorporating new
public SQLite releases becomes much easier too.  To pull in all of the
latest changes from the public SQLite repository, simply move into
the open check-out and do:


fossil update


Then continue to merge the changes in "trunk" with your "private"
changes as described in section 3.7.


c2: Why SQLite Does Not Use Git
c3: 1. Introduction
c5: whynotgit.html#introduction
c4:
SQLite does not use the
Git version control system.
SQLite uses
Fossil instead, which is a
version control system that was specifically designed
and written to support SQLite.


People often wonder why SQLite does not use the
Git version control system like everybody
else.
This article attempts to answer that question.  Also,
in section 3, 
this article provides hints to Git users
about how they can easily access the SQLite source code.


This article is not a comparison between Fossil
and Git.  See
https://fossil-scm.org/fossil/doc/trunk/www/fossil-v-git.wiki
for one comparison of the two systems.  Other third-party 
comparisons are available as well - use a search engine to find them.


This article is not advocating that you switch your projects
away from Git.  You can use whatever version control system you want.
If you are perfectly happy with Git, then by all means keep using
Git.  But, if Git is not working well for you or you are wondering 
if it can be improved or if there is something better,
then maybe try to understand the perspectives presented below.
Use the insights thus obtained to find or write a different and
better version control system, or to just make
improvements to Git itself.


c2: Why SQLite Does Not Use Git
c3: 1.1. Edits
c5: whynotgit.html#edits
c4:
This article has been revised multiple times in an attempt
to improve clarity, address concerns and misgivings,
and to fix errors.
The complete edit history for this document can be seen at
https://sqlite.org/docsrc/finfo/pages/whynotgit.in.
(Usage hint: Click on any two nodes of the graph for a diff.
BTW, are there any Git web interfaces that offers a similar
capability?)


c2: Why SQLite Does Not Use Git
c3: 2.1. Git does not provide good situational awareness
c5: whynotgit.html#git_does_not_provide_good_situational_awareness
c4:
When I want to see what has been happening on SQLite, I visit the
timeline and in a single
screen I see a summary of the latest changes, on all branches.
In a few clicks, I can drill down as much detail as I
want.  I can even do this from a phone.


GitHub and GitLab offer nothing comparable.  The closest I have
found is the network,
which is slow to render (unless it is already cached), does not 
offer nearly as much details, and scarcely works at all on mobile.
The commits view
of GitHub provides more detail, renders quickly,
and works on mobile, but only shows a single branch at a time,
so I cannot easily know if I've seen all of the recent changes.
And even if GitHub/GitLab did offer better interfaces, both are
third-party services.  They are not a core part of Git.  Hence,
using them introduces yet another dependency into the project.


I am told that Git users commonly install third-party graphical
viewers for Git, many of which do a better job of showing recent 
activity on the project.  That is great, but these are still
more third-party applications that must be installed and
managed separately.  Many are platform-specific.  (One of the
better ones, GitUp, only works on Mac, for
example.)  All require that you first sync your local repository
then bring up their graphical interface on your desktop.  And
even with all that, I still cannot see what I typically want to 
see without multiple clicks.  Checking on project status from
a phone while away from the office is not an option.


c2: Why SQLite Does Not Use Git
c3: 2.2. Git makes it difficult to find successors (descendents) of a check-in
c5: whynotgit.html#git_makes_it_difficult_to_find_successors_descendents_of_a_check_in
c4:
Git lets you look backwards in time, but not forwards.
Given some historical check-in, you can see what came before,
but it is challenging see what came next.


In contrast, Fossil offers helpful displays such as
https://sqlite.org/src/timeline?df=major-release
to show all check-ins that are derived from the most
recent major release.





All SQLite check-ins derived from the most recent major release



It is not impossible to find the descendents of a check-in
in Git.  It is merely difficult.  For example,
there is a 
stackoverflow page
showing the command sequence for finding the descendents of a check-in
in unix:

git rev-list --all --parents | grep ".\{40\}.*.*" | awk '{print $1}'




But this is not the same thing.  The command above gives
a list of descendents without showing the branching structure, which
is important for understanding what happened.  And the command only works
if you have a local clone of the repository; finding the descendents of
a check-in is not something you can do with web interfaces such
as GitHub or GitLab.


This is not really about just finding the descendents of a check-in
from time to time.  The fact that descendents are readily available in
Fossil means that the information pervades the web pages provided by
Fossil.  One example: Every Fossil check-in information page
(example) shows
a small "Context" graph of the immediate predecessor and successors 
to that check-in.  This helps the user maintain better situational
awareness, and it provides useful capabilities, such as the ability
click forward to the next check-in in sequence.  Another example:
Fossil easily shows the context around a specific check-in
(example)
which again helps to promote situational awareness and a deeper
understanding of what is happening in the code.  There is a
whole page of additional examples
in the Fossil documentation.


All of the above is theoretically possible with Git, given the right extensions
and tools and using the right commands.  But it is not easy to do,
and so it rarely gets done.  Consequently, developers have less awareness
of what is happening in the code.


c2: Why SQLite Does Not Use Git
c3: 2.3. The mental model for Git is needlessly complex
c5: whynotgit.html#the_mental_model_for_git_is_needlessly_complex
c4:
The complexity of Git
distracts attention from the software under development.  A user of Git
needs to keep all of the following in mind:

 The working directory
 The "index" or staging area
 The local head
 The local copy of the remote head
 The actual remote head


Git has commands (and/or options on commands) for moving and
comparing content between all of these locations. 

In contrast,
Fossil users only need to think about their working directory and
the check-in they are working on.  That is 60% less distraction.
Every developer has a finite number of brain-cycles.  Fossil
requires fewer brain-cycles to operate, thus freeing up 
intellectual resources to focus on the software under development.

One user of both Git and Fossil
writes in HN:


Fossil gives me peace of mind that I have everything ... synced to 
the server with a single command....
I never get this peace of mind with git.


c2: Why SQLite Does Not Use Git
c3: 2.4. Git does not track historical branch names
c5: whynotgit.html#git_does_not_track_historical_branch_names
c4:
Git keeps the complete DAG of the check-in sequence.  But branch
tags are local information that is not synced and not retained
once a branch closes.
This makes review of historical
branches tedious.


As an example, suppose a customer asks you:
"What ever became of that 'prefer-coroutine-sort-subquery' branch
from two years ago?"
You might try to answer by consulting the history in
your version control system, thusly:


GitHub: https://github.com/sqlite/sqlite/commits/prefer-coroutine-sort-subquery
Fossil: https://sqlite.org/src/timeline?r=prefer-coroutine-sort-subquery



The Fossil view clearly shows that the branch was eventually merged back into
trunk.  It shows where the branch started, and it shows two occasions where changes
on trunk were merged into the branch.  GitHub shows none of this.  In fact, the
GitHub display is mostly useless in trying to figure out what happened.


Many readers have recommended various third-party GUIs for Git that
might do a better job of showing historical development activity.  Maybe
some of them do work better than native Git and/or GitHub, though they
will all be hampered by the fact that Git does not preserve historical
branch names across syncs.  And even if those other tools are better,
the fact that it is necessary to go to a third-party tool to get the information
desired does not speak well of the core system.


c2: Why SQLite Does Not Use Git
c3: 2.5. Git requires more administrative support
c5: whynotgit.html#git_requires_more_administrative_support
c4:
Git is complex software.
One needs an installer of some kind to put Git on a developer
workstation, or to upgrade to a newer version of Git.
Standing up a Git server is non-trivial, and so most developers
use a third-party service such as GitHub or GitLab,
and thus introduce additional dependencies.


In contrast, Fossil is a single standalone binary which is
installed by putting it on $PATH.  That one binary contains all
the functionality of core Git and also GitHub and/or GitLab.  It
manages a community server with wiki, bug tracking, and forums, 
provides packaged downloads for consumers, login managements, 
and so forth, with no extra software required.  Standing up a
community server for Fossil takes minutes.  And Fossil is efficient.
A Fossil server will run fine on a $5/month VPS or a Raspberry Pi,
whereas GitLab and similar require beefier hardware.


Less administration means that programmers spend more time working
on the software (SQLite in this case) and less time fussing with
the version control system.


c2: Why SQLite Does Not Use Git
c3: 2.6. Git provides a poor user experience
c5: whynotgit.html#git_provides_a_poor_user_experience
c4:
The following https://xkcd.com/1597/ cartoon is an
exaggeration, yet hits close to home:




Let's be real.  Few people dispute that Git provides
a suboptimal user experience.  A lot of 
the underlying implementation shows through into the user
interface.  The interface is so bad that there is even a
parody site that generates
fake git man pages.

Designing software is hard.  It takes a lot of focus.
A good version control system should provide the developer with
assistance, not frustration.  Git has gotten better in this
regard over the past decade, but it still has a long way to go.


c2: Why SQLite Does Not Use Git
c3: 3. A Git-User's Guide To Accessing SQLite Source Code
c5: whynotgit.html#a_git_user_s_guide_to_accessing_sqlite_source_code
c4:
If you are a devoted Git user, you can still easily access SQLite.  
This section gives some hints on how to do so.


c2: Why SQLite Does Not Use Git
c3: 3.1. The Official GitHub Mirror
c5: whynotgit.html#the_official_github_mirror
c4:
As of 2019-03-20, there is now an 
official Git mirror of the
SQLite sources on GitHub.

The mirror is an incremental export of the 
canonical Fossil repository for
SQLite.  A cron-job updates the GitHub repository once an hour.
This is a one-way, read-only code mirror.  No pull requests or 
changes are accepted via GitHub.  The GitHub repository merely copies
the content from the Fossil repository.  All changes are input via
Fossil.


The hashes that identify check-ins and files on the Git mirror are
different from the hashes in Fossil.  There are many reasons for
this, chief among them that Fossil uses a SHA3-256 hash whereas
Git uses a SHA1 hash.  During export, the original Fossil hash for
each check-in is added as a footer to check-in comments.  To avoid
confusion, always use the original Fossil hash, not the Git hash,
when referring to SQLite check-ins.


c2: Why SQLite Does Not Use Git
c3: 3.2. Web Access
c5: whynotgit.html#web_access
c4:
The SQLite Fossil Repository contains links
for downloading  a Tarball, ZIP Archive, or SQLite Archive for any
historical version of SQLite.  The URLs for these downloads are
simple and can be incorporated easily into automated tools.  The format is:


https://sqlite.org/src/tarball/VERSION/sqlite.tar.gz



Simply replace VERSION with some description of the version to be
downloaded.  The VERSION can be a prefix of the cryptographic hash
name of a specific check-in, or the name of a branch (in which case the
most recent version of the branch is fetched) or a tag for a specific
check-in like "version-3.23.1":


https://sqlite.org/src/tarball/version-3.23.1/sqlite.tar.gz



To get the latest release, use "release"
for VERSION, like this:


https://sqlite.org/src/tarball/release/sqlite.tar.gz



To get the latest trunk check-in, us "trunk" for VERSION:


https://sqlite.org/src/tarball/trunk/sqlite.tar.gz



And so forth.
For ZIP archives and SQLite Archives, simply change the "/tarball/" element
into either "/zip/" or "/sqlar/", and maybe also change the name of the
download file to have a ".zip" or ".sqlar" suffix.


c2: Why SQLite Does Not Use Git
c3: 3.3. Fossil Access
c5: whynotgit.html#fossil_access
c4:
Fossil is easy to install and use.  Here are the steps for unix.
(Windows is similar.)



Download the self-contained Fossil executable from
https://fossil-scm.org/fossil/uv/download.html and put the executable
somewhere on your $PATH.
mkdir ~/fossils
fossil clone https://sqlite.org/src ~/fossils/sqlite.fossil
mkdir ~/sqlite; cd ~/sqlite
fossil open ~/fossils/sqlite.fossil



At this point you are ready to type "./configure; make"
(or on Windows with MSVC, "nmake /f Makefile.msc").


To change your checkout to a different version of Fossil use
the "update" command:


fossil update VERSION



Use "trunk" for VERSION to get the latest trunk version of SQLite.
Or use a prefix of a cryptographic hash name, or the name of some branch
or tag.  See
https://fossil-scm.org/fossil/doc/trunk/www/checkin_names.wiki for more
suggestions on what names can be used for VERSION.


Use the "fossil ui" command from within the ~/sqlite checkout to
bring up a local copy of the website.


Additional documentation on Fossil can be found at
https://fossil-scm.org/fossil/doc/trunk/www/permutedindex.html


Do not be afraid to explore and experiment.
Without a log-in you won't be able to
push back any changes you make, so you cannot damage the project.


c2: Why SQLite Does Not Use Git
c3: 3.4. Verifying Source Code Integrity
c5: whynotgit.html#verifying_source_code_integrity
c4:
If you need to verify that the SQLite source code that you have is
authentic and has not been modified in any way (perhaps by an adversary)
that can be done using a few simple command-line tools.  At the root 
of the SQLite source tree is a file named "manifest".  The manifest 
file contains the name of every other file in the source tree together 
with either a SHA1 or SHA3-256 hash for that file.  (SHA1 is used for
older files and SHA3-256 for newer files.)  You can write a
script to extract these hashes and verify them against the source code 
files.  The hash name for the check-in is just the SHA3-256 hash of the
"manifest" file itself, possibly with the last line omitted if the
last line begins with "# Remove this line..."


c2: Why SQLite Does Not Use Git
c3: 4. See Also
c5: whynotgit.html#see_also
c4:
Other pages that talk about Fossil and Git include:

Fossil vs. Git
What others say about Fossil and Git


c2: Database File Format
c5: fileformat.html#
c4:
This document describes and defines the on-disk database file
format used by all releases of SQLite since 
version 3.0.0 (2004-06-18).


c2: Database File Format
c3: 1. The Database File
c5: fileformat.html#the_database_file
c4:
The complete state of an SQLite database is usually
contained in a single file on disk called the "main database file".

During a transaction, SQLite stores additional information 
in a second file called the "rollback journal", or if SQLite is in
WAL mode, a write-ahead log file.


c2: Database File Format
c3: 1.1. Hot Journals
c5: fileformat.html#hot_journals
c4:
If the application or
host computer crashes before the transaction completes, then the rollback
journal or write-ahead log contains information needed 
to restore the main database file to a consistent state.  When a rollback 
journal or write-ahead log contains information necessary for recovering 
the state of the database, they are called a "hot journal" or "hot WAL file".
Hot journals and WAL files are only a factor during error recovery
scenarios and so are uncommon, but they are part of the state of an SQLite
database and so cannot be ignored.  This document defines the format
of a rollback journal and the write-ahead log file, but the focus is
on the main database file.


c2: Database File Format
c3: 1.2. Pages
c5: fileformat.html#pages
c4:
The main database file consists of one or more pages.  The size of a
page is a power of two between 512 and 65536 inclusive.  All pages within
the same database are the same size.  The page size for a database file
is determined by the 2-byte integer located at an offset of
16 bytes from the beginning of the database file.

Pages are numbered beginning with 1.  The maximum page number is
4294967294 (232 - 2).  The minimum size
SQLite database is a single 512-byte page.
The maximum size database would be 2147483646 pages at 65536 bytes per
page or 281,474,976,579,584 bytes (about 281 terabytes).  Usually SQLite will
hit the maximum file size limit of the underlying filesystem or disk
hardware long before it hits its own internal size limit.

In common use, SQLite databases tend to range in size from a few kilobytes
to a few gigabytes, though terabyte-size SQLite databases are known to exist
in production.

At any point in time, every page in the main database has a single
use which is one of the following:

The lock-byte page
A freelist page

A freelist trunk page
A freelist leaf page

A b-tree page

A table b-tree interior page
A table b-tree leaf page
An index b-tree interior page
An index b-tree leaf page

A payload overflow page
A pointer map page



All reads from and writes to the main database file begin at a page
boundary and all writes are an integer number of pages in size.  Reads
are also usually an integer number of pages in size, with the one exception
that when the database is first opened, the first 100 bytes of the
database file (the database file header) are read as a sub-page size unit.

Before any information-bearing page of the database is modified, 
the original unmodified content of that page is written into the 
rollback journal.  If a transaction is interrupted and needs to be 
rolled back, the rollback journal can then be used to restore the
database to its original state.  Freelist leaf pages bear no
information that would need to be restored on a rollback and so they
are not written to the journal prior to modification, in order to
reduce disk I/O.


c2: Database File Format
c3: 1.3. The Database Header
c5: fileformat.html#the_database_header
c4:
The first 100 bytes of the database file comprise the database file 
header.  The database file header is divided into fields as shown by
the table below.  All multibyte fields in the database file header are
stored with the most significant byte first (big-endian).


Database Header Format

OffsetSizeDescription
016
The header string: "SQLite format 3\000"
162
The database page size in bytes.  Must be a power of two between 512
and 32768 inclusive, or the value 1 representing a page size of 65536.
181
File format write version.  1 for legacy; 2 for WAL.
191
File format read version.  1 for legacy; 2 for WAL.
201
Bytes of unused "reserved" space at the end of each page.  Usually 0.
211
Maximum embedded payload fraction.  Must be 64.
221
Minimum embedded payload fraction.  Must be 32.
231
Leaf payload fraction.  Must be 32.
244
File change counter.
284
Size of the database file in pages.  The "in-header database size".
324
Page number of the first freelist trunk page.
364
Total number of freelist pages.
404
The schema cookie.
444
The schema format number.  Supported schema formats are 1, 2, 3, and 4.
484
Default page cache size.
524
The page number of the largest root b-tree page when in auto-vacuum or
incremental-vacuum modes, or zero otherwise.
564
The database text encoding.  A value of 1 means UTF-8.  A value of 2
means UTF-16le.  A value of 3 means UTF-16be.
604
The "user version" as read and set by the user_version pragma.
644
True (non-zero) for incremental-vacuum mode.  False (zero) otherwise.
684
The "Application ID" set by PRAGMA application_id.
7220
Reserved for expansion.  Must be zero.
924
The version-valid-for number.
964
SQLITE_VERSION_NUMBER


c2: Database File Format
c3: 1.3.1. Magic Header String
c5: fileformat.html#magic_header_string
c4:
Every valid SQLite database file begins with the following 16 bytes 
(in hex): 53 51 4c 69 74 65 20 66 6f 72 6d 61 74 20 33 00.  This byte sequence
corresponds to the UTF-8 string "SQLite format 3" including the nul
terminator character at the end.


c2: Database File Format
c3: 1.3.2. Page Size
c5: fileformat.html#page_size
c4:
The two-byte value beginning at offset 16 determines the page size of 
the database.  For SQLite versions 3.7.0.1 (2010-08-04)
and earlier, this value is 
interpreted as a big-endian integer and must be a power of two between
512 and 32768, inclusive.  Beginning with SQLite version 3.7.1
(2010-08-23), a page
size of 65536 bytes is supported.  The value 65536 will not fit in a
two-byte integer, so to specify a 65536-byte page size, the value
at offset 16 is 0x00 0x01.
This value can be interpreted as a big-endian
1 and thought of as a magic number to represent the 65536 page size.
Or one can view the two-byte field as a little endian number and say
that it represents the page size divided by 256.  These two 
interpretations of the page-size field are equivalent.


c2: Database File Format
c3: 1.3.3. File format version numbers
c5: fileformat.html#file_format_version_numbers
c4:
The file format write version and file format read version at offsets
18 and 19 are intended to allow for enhancements of the file format
in future versions of SQLite.  In current versions of SQLite, both of
these values are 1 for rollback journalling modes and 2 for WAL
journalling mode.  If a version of SQLite coded to the current
file format specification encounters a database file where the read
version is 1 or 2 but the write version is greater than 2, then the database
file must be treated as read-only.  If a database file with a read version
greater than 2 is encountered, then that database cannot be read or written.


c2: Database File Format
c3: 1.3.4. Reserved bytes per page
c5: fileformat.html#reserved_bytes_per_page
c4:
SQLite has the ability to set aside a small number of extra bytes at
the end of every page for use by extensions.  These extra bytes are
used, for example, by the SQLite Encryption Extension to store a nonce
and/or cryptographic checksum associated with each page.  The 
"reserved space" size in the 1-byte integer at offset 20 is the number
of bytes of space at the end of each page to reserve for extensions.
This value is usually 0.  The value can be odd.



The "usable size" of a database page is the page size specified by the
2-byte integer at offset 16 in the header less the "reserved" space size
recorded in the 1-byte integer at offset 20 in the header.  The usable
size of a page might be an odd number.  However, the usable size is not
allowed to be less than 480.  In other words, if the page size is 512,
then the reserved space size cannot exceed 32.


c2: Database File Format
c3: 1.3.5. Payload fractions
c5: fileformat.html#payload_fractions
c4:
The maximum and minimum embedded payload fractions and the leaf
payload fraction values must be 64, 32, and 32.  These values were
originally intended to be tunable parameters that could be used to
modify the storage format of the b-tree algorithm.  However, that
functionality is not supported and there are no current plans to add
support in the future.  Hence, these three bytes are fixed at the
values specified.


c2: Database File Format
c3: 1.3.6. File change counter
c5: fileformat.html#file_change_counter
c4:
The file change counter is a 4-byte big-endian integer at
offset 24 that is incremented whenever the database file is unlocked
after having been modified.
When two or more processes are reading the same database file, each 
process can detect database changes from other processes by monitoring 
the change counter.
A process will normally want to flush its database page cache when
another process modified the database, since the cache has become stale.
The file change counter facilitates this.

In WAL mode, changes to the database are detected using the wal-index
and so the change counter is not needed.  Hence, the change counter might
not be incremented on each transaction in WAL mode.


c2: Database File Format
c3: 1.3.7. In-header database size
c5: fileformat.html#in_header_database_size
c4:
The 4-byte big-endian integer at offset 28 into the header 
stores the size of the database file in pages.  If this in-header
datasize size is not valid (see the next paragraph), then the database 
size is computed by looking
at the actual size of the database file. Older versions of SQLite
ignored the in-header database size and used the actual file size
exclusively.  Newer versions of SQLite use the in-header database
size if it is available but fall back to the actual file size if
the in-header database size is not valid.

The in-header database size is only considered to be valid if
it is non-zero and if the 4-byte change counter at offset 24
exactly matches the 4-byte version-valid-for number at offset 92.
The in-header database size is always valid 
when the database is only modified using recent versions of SQLite,
versions 3.7.0 (2010-07-21) and later.
If a legacy version of SQLite writes to the database, it will not
know to update the in-header database size and so the in-header
database size could be incorrect.  But legacy versions of SQLite
will also leave the version-valid-for number at offset 92 unchanged
so it will not match the change-counter.  Hence, invalid in-header
database sizes can be detected (and ignored) by observing when
the change-counter does not match the version-valid-for number.


c2: Database File Format
c3: 1.3.8. Free page list
c5: fileformat.html#free_page_list
c4:
Unused pages in the database file are stored on a freelist.  The
4-byte big-endian integer at offset 32 stores the page number of
the first page of the freelist, or zero if the freelist is empty.
The 4-byte big-endian integer at offset 36 stores the total 
number of pages on the freelist.


c2: Database File Format
c3: 1.3.9. Schema cookie
c5: fileformat.html#schema_cookie
c4:
The schema cookie is a 4-byte big-endian integer at offset 40
that is incremented whenever the database schema changes.  A 
prepared statement is compiled against a specific version of the
database schema.  When the database schema changes, the statement
must be reprepared.  When a prepared statement runs, it first checks
the schema cookie to ensure the value is the same as when the statement
was prepared and if the schema cookie has changed, the statement either
automatically reprepares and reruns or it aborts with an SQLITE_SCHEMA 
error.


c2: Database File Format
c3: 1.3.10. Schema format number
c5: fileformat.html#schema_format_number
c4:
The schema format number is a 4-byte big-endian integer at offset 44.
The schema format number is similar to the file format read and write
version numbers at offsets 18 and 19 except that the schema format number
refers to the high-level SQL formatting rather than the low-level b-tree
formatting.  Four schema format numbers are currently defined:


Format 1 is understood by all versions of SQLite back to
version 3.0.0 (2004-06-18).
Format 2 adds the ability of rows within the same table
to have a varying number of columns, in order to support the
ALTER TABLE ... ADD COLUMN functionality.  Support for
reading and writing format 2 was added in SQLite 
version 3.1.3 on 2005-02-20.
Format 3 adds the ability of extra columns added by
ALTER TABLE ... ADD COLUMN to have non-NULL default
values.  This capability was added in SQLite version 3.1.4
on 2005-03-11.
Format 4 causes SQLite to respect the
DESC keyword on
index declarations.  (The DESC keyword is ignored in indexes for 
formats 1, 2, and 3.)
Format 4 also adds two new boolean record type values (serial types
8 and 9).  Support for format 4 was added in SQLite 3.3.0 on
2006-01-10.


New database files created by SQLite use format 4 by default.
The legacy_file_format pragma can be used to cause SQLite
to create new database files using format 1.
The format version number can be made to default to 1 instead of 4 by
setting SQLITE_DEFAULT_FILE_FORMAT=1 at compile-time.


c2: Database File Format
c3: 1.3.11. Suggested cache size
c5: fileformat.html#suggested_cache_size
c4:
The 4-byte big-endian signed integer at offset 48 is the suggested
cache size in pages for the database file.  The value is a suggestion
only and SQLite is under no obligation to honor it.  The absolute value
of the integer is used as the suggested size.  The suggested cache size
can be set using the default_cache_size pragma.


c2: Database File Format
c3: 1.3.12. Incremental vacuum settings
c5: fileformat.html#incremental_vacuum_settings
c4:
The two 4-byte big-endian integers at offsets 52 and 64 are used
to manage the auto_vacuum and incremental_vacuum modes.  If
the integer at offset 52 is zero then pointer-map (ptrmap) pages are
omitted from the database file and neither auto_vacuum nor
incremental_vacuum are supported.  If the integer at offset 52 is
non-zero then it is the page number of the largest root page in the
database file, the database file will contain ptrmap pages, and the
mode must be either auto_vacuum or incremental_vacuum.  In this latter
case, the integer at offset 64 is true for incremental_vacuum and
false for auto_vacuum.  If the integer at offset 52 is zero then
the integer at offset 64 must also be zero.


c2: Database File Format
c3: 1.3.13. Text encoding
c5: fileformat.html#text_encoding
c4:
The 4-byte big-endian integer at offset 56 determines the encoding
used for all text strings stored in the database.  
A value of 1 means UTF-8.
A value of 2 means UTF-16le.
A value of 3 means UTF-16be.
No other values are allowed.
The sqlite3.h header file defines C-preprocessor macros SQLITE_UTF8 as 1,
SQLITE_UTF16LE as 2, and SQLITE_UTF16BE as 3, to use in place of
the numeric codes for the text encoding.


c2: Database File Format
c3: 1.3.14. User version number
c5: fileformat.html#user_version_number
c4:
The 4-byte big-endian integer at offset 60 is the user version which
is set and queried by the user_version pragma.  The user version is
not used by SQLite.


c2: Database File Format
c3: 1.3.15. Application ID
c5: fileformat.html#application_id
c4:
The 4-byte big-endian integer at offset 68 is an "Application ID" that
can be set by the PRAGMA application_id command in order to identify the
database as belonging to or associated with a particular application.
The application ID is intended for database files used as an
application file-format.  The application ID can be used by utilities 
such as file(1) to determine the specific
file type rather than just reporting "SQLite3 Database".  A list of
assigned application IDs can be seen by consulting the
magic.txt
file in the SQLite source repository.


c2: Database File Format
c3: 1.3.16. Write library version number and version-valid-for number
c5: fileformat.html#write_library_version_number_and_version_valid_for_number
c4:
The 4-byte big-endian integer at offset 96 stores the 
SQLITE_VERSION_NUMBER value for the SQLite library that most
recently modified the database file.  The 4-byte big-endian integer at
offset 92 is the value of the change counter when the version number
was stored.  The integer at offset 92 indicates which transaction
the version number is valid for and is sometimes called the
"version-valid-for number".


c2: Database File Format
c3: 1.3.17. Header space reserved for expansion
c5: fileformat.html#header_space_reserved_for_expansion
c4:
All other bytes of the database file header are reserved for
future expansion and must be set to zero.


c2: Database File Format
c3: 1.4. The Lock-Byte Page
c5: fileformat.html#the_lock_byte_page
c4:
The lock-byte page is the single page of the database file
that contains the bytes at offsets between 1073741824 and 1073742335,
inclusive.  A database file that is less than or equal to 1073741824 bytes 
in size contains no lock-byte page.  A database file larger than
1073741824 contains exactly one lock-byte page.


The lock-byte page is set aside for use by the operating-system specific
VFS implementation in implementing the database file locking primitives.
SQLite does not use the lock-byte page.  The SQLite core 
will never read or write the lock-byte page,
though operating-system specific VFS 
implementations may choose to read or write bytes on the lock-byte 
page according to the 
needs and proclivities of the underlying system.  The unix and win32
VFS implementations that come built into SQLite do not write to the
lock-byte page, but third-party VFS implementations for
other operating systems might.

The lock-byte page arose from the need to support Win95 which was the
predominant operating system when this file format was designed and which 
only supported mandatory file locking.  All modern operating systems that
we know of support advisory file locking, and so the lock-byte page is
not really needed any more, but is retained for backwards compatibility.


c2: Database File Format
c3: 1.5. The Freelist
c5: fileformat.html#the_freelist
c4:
A database file might contain one or more pages that are not in
active use.  Unused pages can come about, for example, when information
is deleted from the database.  Unused pages are stored on the freelist
and are reused when additional pages are required.

The freelist is organized as a linked list of freelist trunk pages
with each trunk page containing page numbers for zero or more freelist
leaf pages.

A freelist trunk page consists of an array of 4-byte big-endian integers.
The size of the array is as many integers as will fit in the usable space
of a page.  The minimum usable space is 480 bytes so the array will always
be at least 120 entries in length.  The first integer on a freelist trunk
page is the page number of the next freelist trunk page in the list or zero 
if this is the last freelist trunk page.  The second integer on a freelist
trunk page is the number of leaf page pointers to follow.  
Call the second integer on a freelist trunk page L.
If L is greater than zero then integers with array indexes between 2 and
L+1 inclusive contain page numbers for freelist leaf pages.

Freelist leaf pages contain no information.  SQLite avoids reading or
writing freelist leaf pages in order to reduce disk I/O.

A bug in SQLite versions prior to 3.6.0 (2008-07-16)
caused the database to be
reported as corrupt if any of the last 6 entries in the freelist trunk page 
array contained non-zero values.  Newer versions of SQLite do not have
this problem.  However, newer versions of SQLite still avoid using the 
last six entries in the freelist trunk page array in order that database
files created by newer versions of SQLite can be read by older versions
of SQLite.

The number of freelist pages is stored as a 4-byte big-endian integer
in the database header at an offset of 36 from the beginning of the file.
The database header also stores the page number of the first freelist trunk
page as a 4-byte big-endian integer at an offset of 32 from the beginning
of the file.


c2: Database File Format
c3: 1.6. B-tree Pages
c5: fileformat.html#b_tree_pages
c4:
The b-tree algorithm provides key/data storage with unique and
ordered keys on page-oriented storage devices.
For background information on b-trees, see
Knuth, The Art Of Computer Programming, Volume 3 "Sorting
and Searching", pages 471-479.  Two variants of b-trees are used by
SQLite.  "Table b-trees" use a 64-bit signed integer key and store
all data in the leaves.  "Index b-trees" use arbitrary keys and store no
data at all.

A b-tree page is either an interior page or a leaf page.
A leaf page contains keys and in the case of a table b-tree each
key has associated data.  An interior page contains
K keys together with K+1 pointers to child b-tree pages.
A "pointer" in an interior b-tree page is just the 32-bit 
unsigned integer page number of the child page.

The number of keys on an interior b-tree page, K,
is almost always at least 2 and is usually much more than 2.
The only exception is when page 1 is an interior b-tree page.
Page 1 has 100 fewer bytes of storage space available,
due to the presence of the database header at the beginning of that page,
and so sometimes (rarely) if page 1 is an interior b-tree page, it can
end up holding just a a single key.  In all other cases, K is 2 or more.
The upper bound on K is as many keys as will fit on the page. Large keys
on index b-trees are split up into overflow pages so that no single key
uses more than one fourth of the available storage space on the page
and hence every internal page is able to store at least 4 keys.
The integer keys of table b-trees are never large enough to
require overflow, so key overflow only occurs on index b-trees.

Define the depth
of a leaf b-tree to be 1 and the depth of any interior b-tree to be one
more than the maximum depth of any of its children.  In a well-formed
database, all children of an interior b-tree have the same depth.

In an interior b-tree page, the pointers and keys logically alternate 
with a pointer on both ends. (The previous sentence is to be understood
conceptually - the actual layout of the keys and
pointers within the page is more complicated and will be described in
the sequel.)  All keys within the same page are unique and are logically
organized in ascending order from left to right.  (Again, this ordering
is logical, not physical.  The actual location of keys within the page
is arbitrary.) For any key X, pointers to the left
of a X refer to b-tree pages on which all keys are less than or equal to X.
Pointers to the right of X refer to pages where all keys are 
greater than X.

Within an interior b-tree page, each key and the pointer to its
immediate left are combined into a structure called a "cell".  The
right-most pointer is held separately.  A leaf b-tree page has no
pointers, but it still uses the cell structure to hold keys for
index b-trees or keys and content for table b-trees.  Data is also
contained in the cell.


Every b-tree page has at most one parent b-tree page.
A b-tree page without a parent is called a root page.  A root b-tree page
together with the closure of its children form a complete b-tree.
It is possible (and in fact rather common) to have a complete b-tree
that consists of a single page that is both a leaf and the root.
Because there are pointers from parents to children, every page of a
complete b-tree can be located if only the root page is known.  Hence,
b-trees are identified by their root page number.



A b-tree page is either a table b-tree page or an index b-tree page.
All pages within each complete b-tree are of the same type: either table
or index.  There is one table b-trees in the database file
for each rowid table in the database schema, including system tables
such as sqlite_schema.  There is one index b-tree
in the database file for each index in the schema, including implied indexes
created by uniqueness constraints.  There are no b-trees associated with
virtual tables.  Specific virtual table implementations might make use
of shadow tables for storage, but those shadow tables will have separate
entries in the database schema.  WITHOUT ROWID tables use index b-trees
rather than a table b-trees, so there is one
index b-tree in the database file for each WITHOUT ROWID table.
The b-tree corresponding to the sqlite_schema table is always a table
b-tree and always has a root page of 1.
The sqlite_schema table contains the root page number for every other 
table and index in the database file.

Each entry in a table b-tree consists of a 64-bit signed integer key
and up to 2147483647 bytes of arbitrary data.  (The key of a table b-tree
corresponds to the rowid of the SQL table that the b-tree implements.)
Interior table b-trees hold only keys and pointers to children.
All data is contained in the table b-tree leaves.

Each entry in an index b-tree consists of an arbitrary key of up
to 2147483647 bytes in length and no data.



Define the "payload" of a cell to be the arbitrary length section
of the cell.  For an index b-tree, the key is always arbitrary in length
and hence the payload is the key.  There are no arbitrary length elements
in the cells of interior table b-tree pages and so those cells have no
payload.  Table b-tree leaf pages contain arbitrary length content and
so for cells on those pages the payload is the content.
When the size of payload for a cell exceeds a certain threshold (to
be defined later) then only the first few bytes of the payload
are stored on the b-tree page and the balance is stored in a linked list
of content overflow pages.

A b-tree page is divided into regions in the following order:


The 100-byte database file header (found on page 1 only)
The 8 or 12 byte b-tree page header
The cell pointer array
Unallocated space
The cell content area
The reserved region.



The 100-byte database file header is found only on page 1, which is
always a table b-tree page.  All other b-tree pages in the database file
omit this 100-byte header.

The reserved region is an area of unused space at the end of every
page (except the locking page) that extensions can use to hold per-page
information.  The size of the reserved region is determined by the one-byte
unsigned integer found at an offset of 20 into the database file header.
The size of the reserved region is usually zero.

The b-tree page header is 8 bytes in size for leaf pages and 12
bytes for interior pages.  All multibyte values in the page header
are big-endian.
The b-tree page header is composed of the following fields:


B-tree Page Header Format

OffsetSizeDescription
01
The one-byte flag at offset 0 indicating the b-tree page type.
A value of 2 (0x02) means the page is an interior index b-tree page.
A value of 5 (0x05) means the page is an interior table b-tree page.
A value of 10 (0x0a) means the page is a leaf index b-tree page.
A value of 13 (0x0d) means the page is a leaf table b-tree page.
Any other value for the b-tree page type is an error.
12
The two-byte integer at offset 1 gives the start of the
first freeblock on the page, or is zero if there are no freeblocks.
32
The two-byte integer at offset 3 gives the number of cells on the page.
52
The two-byte integer at offset 5 designates the start of the cell content
area.  A zero value for this integer is interpreted as 65536.
71
The one-byte integer at offset 7 gives the number of fragmented free
bytes within the cell content area.
84
The four-byte page number at offset 8 is the right-most pointer.  This
value appears in the header of interior b-tree pages only and is omitted from
all other pages.


The cell pointer array of a b-tree page immediately follows the b-tree
page header.  Let K be the number of cells on the btree.  The cell pointer
array consists of K 2-byte integer offsets to the cell contents.  The
cell pointers are arranged in key order with left-most cell (the cell with the
smallest key) first and the right-most cell (the cell with the largest
key) last.

Cell content is stored in the cell content region of the b-tree page.
SQLite strives to place cells as far toward the end of the b-tree page as
it can, in order to leave space for future growth of the cell pointer array.
The area in between the last cell pointer array entry and the beginning of
the first cell is the unallocated region.


If a page contains no cells (which is only possible for a root page
of a table that contains no rows) then the offset to the
cell content area will equal the page size minus the bytes of reserved space.
If the database uses a 65536-byte page size and the reserved space is zero
(the usual value for reserved space) then the cell content offset of an
empty page wants to be 65536.  
However, that integer is too large to be stored in a
2-byte unsigned integer, so a value of 0 is used in its place.

A freeblock is a structure used to identify unallocated space within
a b-tree page.  Freeblocks are organized as a chain.  The first 2 bytes of
a freeblock are a big-endian integer which is the offset in the b-tree page
of the next freeblock in the chain, or zero if the freeblock is the last on
the chain.  The third and fourth bytes of each freeblock form
a big-endian integer which is the size of the freeblock in bytes, including
the 4-byte header.  Freeblocks are always connected in order 
of increasing offset.  The second field of the b-tree page header is the
offset of the first freeblock, or zero if there are no freeblocks on the
page.  In a well-formed b-tree page, there will always be at least one cell
before the first freeblock.

A freeblock requires at least 4 bytes of space.  If there is an isolated
group of 1, 2, or 3 unused bytes within the cell content area, those bytes
comprise a fragment.  The total number of bytes in all fragments is stored
in the fifth field of the b-tree page header.  In a well-formed b-tree page,
the total number of bytes in fragments may not exceed 60.

The total amount of free space on a b-tree page consists of the size
of the unallocated region plus the total size of all freeblocks plus the
number of fragmented free bytes.  SQLite may from time to time reorganize
a b-tree page so that there are no freeblocks or fragment bytes, all
unused bytes are contained in the unallocated space region, and all
cells are packed tightly at the end of the page.  This is called 
"defragmenting" the b-tree page.




A variable-length integer or "varint" is a static Huffman encoding
of 64-bit twos-complement integers that uses less space for small positive 
values. 
A varint is between 1 and 9 bytes in length.  The varint consists of either
zero or more bytes which have the high-order bit set followed by a single byte
with the high-order bit clear, or nine bytes, whichever is shorter.
The lower seven bits of each of the first eight bytes and all 8 bits of
the ninth byte are used to reconstruct the 64-bit twos-complement integer.
Varints are big-endian: bits taken from the earlier byte of the varint
are more significant than bits taken from the later bytes. 

The format of a cell depends on which kind of b-tree page the cell
appears on.  The following table shows the elements of a cell, in
order of appearance, for the various b-tree page types.


Table B-Tree Leaf Cell (header 0x0d):

A varint which is the total number of bytes of payload, including any
overflow
A varint which is the integer key, a.k.a. "rowid"
The initial portion of the payload that does not spill to overflow
pages.
A 4-byte big-endian integer page number for the first page of the
overflow page list - omitted if all payload fits on the b-tree page.


Table B-Tree Interior Cell (header 0x05):

A 4-byte big-endian page number which is the left child pointer.
A varint which is the integer key


Index B-Tree Leaf Cell (header 0x0a):

A varint which is the total number of bytes of key payload, including any
overflow
The initial portion of the payload that does not spill to overflow
pages.
A 4-byte big-endian integer page number for the first page of the
overflow page list - omitted if all payload fits on the b-tree page.


Index B-Tree Interior Cell (header 0x02):

A 4-byte big-endian page number which is the left child pointer.
A varint which is the total number of bytes of key payload, including any
overflow
The initial portion of the payload that does not spill to overflow
pages.
A 4-byte big-endian integer page number for the first page of the
overflow page list - omitted if all payload fits on the b-tree page.



The information above can be recast into a table format as follows:




B-tree Cell Format

Datatype
    Appears in...
    Description
Table Leaf (0x0d)
    Table Interior (0x05)
    Index Leaf (0x0a)
    Index Interior (0x02)
4-byte integer
    &nbsp;
    &#x2714;
    &nbsp;
    &#x2714;
    Page number of left child
varint
    &#x2714;
    &nbsp;
    &#x2714;
    &#x2714;
    Number of bytes of payload
varint
    &#x2714;
    &#x2714;
    &nbsp;
    &nbsp;
    Rowid
byte array
    &#x2714;
    &nbsp;
    &#x2714;
    &#x2714;
    Payload
4-byte integer
    &#x2714;
    &nbsp;
    &#x2714;
    &#x2714;
    Page number of first overflow page



The amount of payload that spills onto overflow pages also depends on
the page type.  For the following computations, let U be the usable size
of a database page, the total page size less the reserved space at the
end of each page.  And let P be the payload size.  In the following,
symbol X represents the maximum amount of payload that can be stored directly
on the b-tree page without spilling onto an overflow page and symbol M
represents the minimum amount of payload that must be stored on the btree
page before spilling is allowed.


Table B-Tree Leaf Cell:

Let X be U-35.  If the payload size P is less than or equal to X then
the entire payload is stored on the b-tree leaf page.
Let M be ((U-12)*32/255)-23 and let K be M+((P-M)%(U-4)).
If P is greater than X
then the number of bytes stored on the table b-tree leaf page is K
if K is less or equal to X or M otherwise.
The number of bytes stored on the leaf page is never less than M.


Table B-Tree Interior Cell:

Interior pages of table b-trees have no payload and so there is never
any payload to spill.


Index B-Tree Leaf Or Interior Cell:

Let X be ((U-12)*64/255)-23.  If the payload size P is less than
or equal to X then the entire payload is stored on the b-tree page.
Let M be ((U-12)*32/255)-23 and let K be M+((P-M)%(U-4)).
If P is greater than X then the number
of bytes stored on the index b-tree page is K if K is less than or
equal to X or M otherwise.
The number of bytes stored on the index page is never less than M.



Here is an alternative description of the same computation:


X is U-35 for table btree leaf pages or
    ((U-12)*64/255)-23 for index pages.
M is always ((U-12)*32/255)-23.
Let K be M+((P-M)%(U-4)).
If P&lt;=X then all P bytes of payload are stored directly on the 
    btree page without overflow.
If P&gt;X and K&lt;=X then the first K bytes of P are stored on the 
    btree page and the remaining P-K bytes are stored on overflow pages.
If P&gt;X and K&gt;X then the first M bytes of P are stored on the
    btree page and the remaining P-M bytes are stored on overflow pages.


The overflow thresholds are designed to give a minimum fanout of
4 for index b-trees and to make sure enough of the payload
is on the b-tree page that the record header can usually be accessed
without consulting an overflow page.  In hindsight, the designer of
the SQLite b-tree logic realized that these thresholds could have been
made much simpler.  However, the computations cannot be changed
without resulting in an incompatible file format.  And the current computations
work well, even if they are a little complex.


c2: Database File Format
c3: 1.7. Cell Payload Overflow Pages
c5: fileformat.html#cell_payload_overflow_pages
c4:
When the payload of a b-tree cell is too large for the b-tree page,
the surplus is spilled onto overflow pages.  Overflow pages form a linked
list.  The first four bytes of each overflow page are a big-endian
integer which is the page number of the next page in the chain, or zero
for the final page in the chain.  The fifth byte through the last usable
byte are used to hold overflow content.


c2: Database File Format
c3: 1.8. Pointer Map or Ptrmap Pages
c5: fileformat.html#pointer_map_or_ptrmap_pages
c4:
Pointer map or ptrmap pages are extra pages inserted into the database
to make the operation of auto_vacuum and incremental_vacuum modes
more efficient.  Other page types in the database typically have pointers
from parent to child.  For example, an interior b-tree page contains pointers
to its child b-tree pages and an overflow chain has a pointer
from earlier to later links in the chain.  A ptrmap page contains linkage
information going in the opposite direction, from child to parent.

Ptrmap pages must exist in any database file which has a non-zero
largest root b-tree page value at offset 52 in the database header.
If the largest root b-tree page value is zero, then the database must not
contain ptrmap pages.

In a database with ptrmap pages, the first ptrmap page is page 2.
A ptrmap page consists of an array of 5-byte entries.  Let J be the
number of 5-byte entries that will fit in the usable space of a page.
(In other words, J=U/5.)  The first ptrmap page will contain back pointer
information for pages 3 through J+2, inclusive.  The second pointer map
page will be on page J+3 and that ptrmap page will provide back pointer
information for pages J+4 through 2*J+3 inclusive.  And so forth for
the entire database file.

In a database that uses ptrmap pages, all pages at locations identified
by the computation in the previous paragraph must be ptrmap page and no
other page may be a ptrmap page.  Except, if the byte-lock page happens to
fall on the same page number as a ptrmap page, then the ptrmap is moved
to the following page for that one case.

Each 5-byte entry on a ptrmap page provides back-link information about 
one of the pages that immediately follow the pointer map.  If page B is a
ptrmap page then back-link information about page B+1 is provided by
the first entry on the pointer map.  Information about page B+2 is
provided by the second entry.  And so forth.

Each 5-byte ptrmap entry consists of one byte of "page type" information
followed by a 4-byte big-endian page number.  Five page types are recognized:



A b-tree root page.  The
page number should be zero.
A freelist page.  The page number should be
zero.
The first page of a
cell payload overflow chain.  The page number is the b-tree page that
contains the cell whose content has overflowed.
A page in an overflow chain
other than the first page.  The page number is the prior page of the
overflow chain.
A non-root b-tree page.  The
page number is the parent b-tree page.


In any database file that contains ptrmap pages, all b-tree root pages 
must come before any non-root b-tree page, cell payload overflow page, or
freelist page.  This restriction ensures that a root page will never
be moved during an auto-vacuum or incremental-vacuum.  The auto-vacuum
logic does not know how to update the root_page field of the sqlite_schema
table and so it is necessary to prevent root pages from being moved
during an auto-vacuum in order to preserve the integrity of the
sqlite_schema table.  Root pages are moved to the beginning of the
database file by the CREATE TABLE, CREATE INDEX, DROP TABLE, and
DROP INDEX operations.


c2: Database File Format
c3: 2. Schema Layer
c5: fileformat.html#schema_layer
c4:
The foregoing text describes low-level aspects of the SQLite file
format.  The b-tree mechanism provides a powerful and efficient means of
accessing a large data set.  This section will describe how the
low-level b-tree layer is used to implement higher-level SQL
capabilities.


c2: Database File Format
c3: 2.1. Record Format
c5: fileformat.html#record_format
c4:
The data for a table b-tree leaf page and the key
of an index b-tree page was characterized above
as an arbitrary sequence of bytes.
The prior discussion mentioned one key being less than another, but
did not define what "less than" meant.  The current section will address
these omissions.

Payload, either table b-tree data or index b-tree keys, 
is always in the "record format".
The record format defines a sequence of values corresponding
to columns in a table or index.  The record format specifies the number
of columns, the datatype of each column, and the content of each column.

The record format makes extensive use of the 
variable-length integer or varint
representation of 64-bit signed integers defined above.



A record contains a header and a body, in that order.  
The header begins with a single varint which determines the total number
of bytes in the header.  The varint value is the size of the header in
bytes including the size varint itself.  Following the size varint are
one or more additional varints, one per column.  These additional varints
are called "serial type" numbers and
determine the datatype of each column, according to the following chart:


Serial Type Codes Of The Record Format

Serial TypeContent SizeMeaning
00
Value is a NULL.
11
Value is an 8-bit twos-complement integer.
22
Value is a big-endian 16-bit twos-complement integer.
33
Value is a big-endian 24-bit twos-complement integer.
44
Value is a big-endian 32-bit twos-complement integer.
56
Value is a big-endian 48-bit twos-complement integer.
68
Value is a big-endian 64-bit twos-complement integer.
78
Value is a big-endian IEEE 754-2008 64-bit floating point number.
80
Value is the integer 0. (Only available for schema format 4 and higher.)
90
Value is the integer 1. (Only available for schema format 4 and higher.)
10,11
    variable
Reserved for internal use.  These serial type codes will
   never appear in a well-formed database file, but they
   might be used in transient and temporary database files
   that SQLite sometimes generates for its own use.
   The meanings of these codes can shift from one release
   of SQLite to the next.
N&#x2265;12 and even
    (N-12)/2
Value is a BLOB that is (N-12)/2 bytes in length.
N&#x2265;13 and odd
    (N-13)/2
Value is a string in the text encoding and (N-13)/2 bytes in length.
The nul terminator is not stored.


The header size varint
and serial type varints will usually consist of a single byte.  The
serial type varints for large strings and BLOBs might extend to two or three
byte varints, but that is the exception rather than the rule. 
The varint format is very efficient at coding the record header.

The values for each column in the record immediately follow the header.
For serial types 0, 8, 9, 12, and 13, the value is zero bytes in
length.  If all columns are of these types then the body section of the
record is empty.

A record might have fewer values than the number of columns in the
corresponding table.  This can happen, for example, after an
ALTER TABLE ... ADD COLUMN SQL statement has increased
the number of columns in the table schema without modifying preexisting rows
in the table.
Missing values at the end of the record are filled in using the
default value for the corresponding columns defined in the table schema.


c2: Database File Format
c3: 2.2. Record Sort Order
c5: fileformat.html#record_sort_order
c4:
The order of keys in an index b-tree is determined by the sort order of
the records that the keys represent.  Record comparison progresses column
by column.  Columns of a record are examined from left to right.  The
first pair of columns that are not equal determines the relative order
of the two records.  The sort order of individual columns is as
follows:


NULL values (serial type 0) sort first.
Numeric values (serial types 1 through 9) sort after NULLs
      and in numeric order.
Text values (odd serial types 13 and larger) sort after numeric
    values in the order determined by the columns collating function.
BLOB values (even serial types 12 and larger) sort last and in the order 
    determined by memcmp().


A collating function for each column is necessary in order to compute
the order of text fields.
SQLite defines three built-in collating functions:



BINARY
     The built-in BINARY collation compares strings byte by byte
        using the memcmp() function
        from the standard C library.
NOCASE
     The NOCASE collation is like BINARY except that uppercase
        ASCII characters ('A' through 'Z')
        are folded into their lowercase equivalents prior to running the
        comparison.  Only ASCII characters are case-folded.
        NOCASE
        does not implement a general purpose unicode caseless comparison.
RTRIM
     RTRIM is like BINARY except that extra spaces at the end of either
         string do not change the result.  In other words, strings will
         compare equal to one another as long as they
         differ only in the number of spaces at the end.


Additional application-specific collating functions can be added to
SQLite using the sqlite3_create_collation() interface.

The default collating function for all strings is BINARY.
Alternative collating functions for table columns can be specified in the
CREATE TABLE statement using the COLLATE clause on the column definition.
When a column is indexed, the same collating function specified in the
CREATE TABLE statement is used for the column in the index, by default,
though this can be overridden using a COLLATE clause in the 
CREATE INDEX statement.


c2: Database File Format
c3: 2.3. Representation Of SQL Tables
c5: fileformat.html#representation_of_sql_tables
c4:
Each ordinary SQL table in the database schema is represented on-disk
by a table b-tree.  Each entry in the table b-tree corresponds to a row
of the SQL table.  The rowid of the SQL table is the 64-bit signed
integer key for each entry in the table b-tree.

 The content of each SQL table row is stored in the database file by
first combining the values in the various columns into a byte array
in the record format, then storing that byte array as the payload in
an entry in the table b-tree.  The order of values in the record is
the same as the order of columns in the SQL table definition.
When an SQL table includes an
INTEGER PRIMARY KEY column (which aliases the rowid) then that
column appears in the record as a NULL value.  SQLite will always use
the table b-tree key rather than the NULL value when referencing the
INTEGER PRIMARY KEY column.

 If the affinity of a column is REAL and that column contains a
value that can be converted to an integer without loss of information
(if the value contains no fractional part and is not too large to be
represented as an integer) then the column may be stored in the record
as an integer.  SQLite will convert the value back to floating
point when extracting it from the record.


c2: Database File Format
c3: 2.4. Representation of WITHOUT ROWID Tables
c5: fileformat.html#representation_of_without_rowid_tables
c4:
If an SQL table is created using the "WITHOUT ROWID" clause at the
end of its CREATE TABLE statement, then that table is a WITHOUT ROWID
table and uses a different on-disk representation.  A WITHOUT ROWID
table uses an index b-tree rather than a table b-tree for storage.
The key for each entry in the WITHOUT ROWID b-tree is a record composed
of the columns of the PRIMARY KEY followed by all remaining columns of
the table.  The primary key columns appear in the order that they were
declared in the PRIMARY KEY clause and the remaining columns appear in
the order they occur in the CREATE TABLE statement.

Hence, the content encoding for a WITHOUT ROWID table is the same
as the content encoding for an ordinary rowid table, except that the
order of the columns is rearranged so that PRIMARY KEY columns come
first, and the content is used as the key in an index b-tree rather
than as the data in a table b-tree.
The special encoding rules for columns with REAL affinity
apply to WITHOUT ROWID tables the same as they do with rowid tables.


c2: Database File Format
c3: 2.4.1. Suppression of redundant columns in the PRIMARY KEY of WITHOUT ROWID tables
c5: fileformat.html#suppression_of_redundant_columns_in_the_primary_key_of_without_rowid_tables
c4:
If the PRIMARY KEY of a WITHOUT ROWID tables uses the same columns
with the same collating sequence more than once, then the second and
subsequent occurrences of that column in the PRIMARY KEY definition are
ignored.  For example, the following CREATE TABLE statements all specify
the same table, which will have the exact same representation on disk:


CREATE TABLE t1(a,b,c,d,PRIMARY KEY(a,c)) WITHOUT ROWID);
CREATE TABLE t1(a,b,c,d,PRIMARY KEY(a,c,a,c)) WITHOUT ROWID);
CREATE TABLE t1(a,b,c,d,PRIMARY KEY(a,A,a,C)) WITHOUT ROWID);
CREATE TABLE t1(a,b,c,d,PRIMARY KEY(a,a,a,a,c)) WITHOUT ROWID);


The first example above is the preferred definition of the table,
of course.  All of the examples create a WITHOUT ROWID table with
two PRIMARY KEY columns, "a" and "c", in that order, followed by
two data columns "b" and "d", also in that order.


c2: Database File Format
c3: 2.5. Representation Of SQL Indices
c5: fileformat.html#representation_of_sql_indices
c4:
Each SQL index, whether explicitly declared via a CREATE INDEX statement
or implied by a UNIQUE or PRIMARY KEY constraint, corresponds to an 
index b-tree in the database file.
Each entry in the index b-tree corresponds to a single row in the 
associated SQL table.
The key to an index b-tree is
a record composed of the columns that are being indexed followed by the
key of the corresponding table row.  For ordinary tables, the row key is
the rowid, and for WITHOUT ROWID tables the row key is the PRIMARY KEY.
Because every row in the table has a unique row key,
all keys in an index are unique.

In a normal index, there is a one-to-one mapping between rows in a 
table and entries in each index associated with that table.
However, in a partial index, the index b-tree only contains entries
corresponding to table rows for which the WHERE clause expression on the
CREATE INDEX statement is true.
Corresponding rows in the index and table b-trees share the same rowid
or primary key values and contain the same value for all indexed columns.


c2: Database File Format
c3: 2.5.1. Suppression of redundant columns in WITHOUT ROWID secondary indexes
c5: fileformat.html#suppression_of_redundant_columns_in_without_rowid_secondary_indexes_
c4:
In an index on a WITHOUT ROWID table, if a column of the PRIMARY KEY
is also a column in the index and has a matching collating sequence, then the
indexed column is not repeated in the table-key suffix on the
end of the index record.  As an example, consider the following SQL:


CREATE TABLE ex25(a,b,c,d,e,PRIMARY KEY(d,c,a)) WITHOUT rowid;
CREATE INDEX ex25ce ON ex25(c,e);
CREATE INDEX ex25acde ON ex25(a,c,d,e);
CREATE INDEX ex25ae ON ex25(a COLLATE nocase,e);


Each row in the ex25ce index is a record
with these columns: c, e, d, a.  The first two columns are
the columns being indexed, c and e.  The remaining columns are the primary
key of the corresponding table row.  Normally, the primary key would be
columns d, c, and a, but because column c already appears earlier in the
index, it is omitted from the key suffix.

In the extreme case where the columns being indexed cover all columns
of the PRIMARY KEY, the index will consist of only the columns being
indexed.  The ex25acde example above demonstrates this.  Each entry in
the ex25acde index consists of only the columns a, c, d, and e, in that
order.

Each row in ex25ae contains five columns: a, e, d, c, a.  The "a"
column is repeated since the first occurrence of "a" has a collating
function of "nocase" and the second has a collating sequence of "binary".
If the "a" column is not repeated and if the table contains two or more
entries with the same "e" value and where "a" differs only in case, then
all of those table entries would correspond to a single entry in the
index, which would break the one-to-one correspondence between the table
and the index.

 The suppression of redundant columns in the key suffix of an index
entry only occurs in WITHOUT ROWID tables.  In an ordinary rowid table,
the index entry always ends with the rowid even if the INTEGER PRIMARY KEY
column is one of the columns being indexed.


c2: Database File Format
c3: 2.6. Storage Of The SQL Database Schema
c5: fileformat.html#storage_of_the_sql_database_schema
c4:
Page 1 of a database file is the root page of a table b-tree that
holds a special table named "sqlite_schema".  This b-tree is known
as the "schema table" since it stores the complete
database schema.  The structure of the sqlite_schema table is as
if it had been created using the following SQL:


CREATE TABLE sqlite_schema(
  type text,
  name text,
  tbl_name text,
  rootpage integer,
  sql text
);


The sqlite_schema table contains one row for each table, index, view,
and trigger (collectively "objects") in the database schema, except there
is no entry for the sqlite_schema table itself.  The sqlite_schema table
contains entries for internal schema objects in addition to application-
and programmer-defined objects.


The sqlite_schema.type column will be one
of the following text strings:  'table', 'index', 'view', or 'trigger'
according to the type of object defined.  The 'table' string is used
for both ordinary and virtual tables.

The sqlite_schema.name column will hold the name of the object.
UNIQUE and PRIMARY KEY constraints on tables cause SQLite to create
internal indexes with names of the form "sqlite_autoindex_TABLE_N"
where TABLE is replaced by the name of the table that contains the
constraint and N is an integer beginning with 1 and increasing by one
with each constraint seen in the table definition.
In a WITHOUT ROWID table, there is no sqlite_schema entry for the
PRIMARY KEY, but the "sqlite_autoindex_TABLE_N" name is set aside
for the PRIMARY KEY as if the sqlite_schema entry did exist.  This
will affect the numbering of subsequent UNIQUE constraints.
The "sqlite_autoindex_TABLE_N" name is never allocated for an
INTEGER PRIMARY KEY, either in rowid tables or WITHOUT ROWID tables.


The sqlite_schema.tbl_name column holds the name of a table or view
that the object is associated with.  For a table or view, the
tbl_name column is a copy of the name column.  For an index, the tbl_name
is the name of the table that is indexed.  For a trigger, the tbl_name
column stores the name of the table or view that causes the trigger 
to fire.

The sqlite_schema.rootpage column stores the page number of the root
b-tree page for tables and indexes.  For rows that define views, triggers,
and virtual tables, the rootpage column is 0 or NULL.

The sqlite_schema.sql column stores SQL text that describes the
object.  This SQL text is a CREATE TABLE, CREATE VIRTUAL TABLE,
CREATE INDEX,
CREATE VIEW, or CREATE TRIGGER statement that if evaluated against
the database file when it is the main database of a database connection
would recreate the object.  The text is usually a copy of the original
statement used to create the object but with normalizations applied so
that the text conforms to the following rules:


The CREATE, TABLE, VIEW, TRIGGER, and INDEX keywords at the beginning
of the statement are converted to all upper case letters.
The TEMP or TEMPORARY keyword is removed if it occurs after the 
initial CREATE keyword.
Any database name qualifier that occurs prior to the name of the
object being created is removed.
Leading spaces are removed.
All spaces following the first two keywords are converted into a single
space.


The text in the sqlite_schema.sql column is a copy of the original
CREATE statement text that created the object, except normalized as
described above and as modified by subsequent ALTER TABLE statements.
The sqlite_schema.sql is NULL for the internal indexes that are
automatically created by UNIQUE or PRIMARY KEY constraints.


c2: Database File Format
c3: 2.6.1. Alternative Names For The Schema Table
c5: fileformat.html#alternative_names_for_the_schema_table
c4:
The name "sqlite_schema" does not appear anywhere in the file format.
That name is just a convention used by the database implementation.
Due to historical and operational considerations, the
"sqlite_schema" table can also sometimes be called by one of the
following aliases:


 sqlite_master
 sqlite_temp_schema
 sqlite_temp_master


Because the name of the schema table does not appear anywhere in
the file format, the meaning of the database file is not changed if
the application chooses to refer to the schema table by one of
these alternative names.


c2: Database File Format
c3: 2.6.2. Internal Schema Objects
c5: fileformat.html#internal_schema_objects
c4:
In addition to the tables, indexes, views, and triggers created by
the application and/or the developer using CREATE statements SQL, the
sqlite_schema table may contain zero or more entries for 
internal schema objects that are created by SQLite for its 
own internal use.  The names of internal schema objects
always begin with "sqlite_" and any table, index, view, or trigger
whose name begins with "sqlite_" is an internal schema object.
SQLite prohibits applications from creating objects whose names begin
with "sqlite_".  

Internal schema objects used by SQLite may include the following:


Indices with names of the form "sqlite_autoindex_TABLE_N" that
       are used to implement UNIQUE and PRIMARY KEY constraints on
       ordinary tables.

A table with the name "sqlite_sequence" that is used to keep track
       of the maximum historical INTEGER PRIMARY KEY for a table
       using AUTOINCREMENT.

Tables with names of the form "sqlite_statN" where N is an integer.
       Such tables store database statistics gathered by the ANALYZE
       command and used by the query planner to help determine the best
       algorithm to use for each query.


New internal schema objects names, always beginning with "sqlite_",
may be added to the SQLite file format in future releases.


c2: Database File Format
c3: 2.6.3. The sqlite_sequence table
c5: fileformat.html#the_sqlite_sequence_table
c4:
The sqlite_sequence table is an internal table used to help implement
AUTOINCREMENT.  The sqlite_sequence table is created automatically
whenever any ordinary table with an AUTOINCREMENT integer primary
key is created.  Once created, the sqlite_sequence table exists in the
sqlite_schema table forever; it cannot be dropped.
The schema for the sqlite_sequence table is:


CREATE TABLE sqlite_sequence(name,seq);


There is a single row in the sqlite_sequence table for each ordinary
table that uses AUTOINCREMENT.  The name of the table (as it appears in
sqlite_schema.name) is in the sqlite_sequence.name field and the largest
INTEGER PRIMARY KEY ever inserted into that table is 
in the sqlite_sequence.seq field.  
New automatically generated integer primary keys for AUTOINCREMENT
tables are guaranteed to be larger than the sqlite_sequence.seq field for
that table.
If the sqlite_sequence.seq field of an AUTOINCREMENT table is already at
the largest integer value (9223372036854775807) then attempts to add new
rows to that table with an automatically generated integer primary will fail
with an SQLITE_FULL error.
The sqlite_sequence.seq field is automatically updated if required when
new entries are inserted to an AUTOINCREMENT table.  
The sqlite_sequence row for an AUTOINCREMENT table is automatically deleted
when the table is dropped.
If the sqlite_sequence row for an AUTOINCREMENT table does not exist when
the AUTOINCREMENT table is updated, then a new sqlite_sequence row is created.
If the sqlite_sequence.seq value for an AUTOINCREMENT table is manually 
set to something other than an integer and there is a subsequent attempt to
insert the or update the AUTOINCREMENT table, then the behavior is undefined.

Application code is allowed to modify the sqlite_sequence table, to add
new rows, to delete rows, or to modify existing rows.  However, application
code cannot create the sqlite_sequence table if it does not already exist.
Application code can delete all entries from the sqlite_sequence table,
but application code cannot drop the sqlite_sequence table.


c2: Database File Format
c3: 2.6.4. The sqlite_stat1 table
c5: fileformat.html#the_sqlite_stat1_table
c4:
The sqlite_stat1 is an internal table created by the ANALYZE command
and used to hold supplemental information about tables and indexes that the
query planner can use to help it find better ways of performing queries.
Applications can update, delete from, insert into or drop the sqlite_stat1
table, but may not create or alter the sqlite_stat1 table.
The schema of the sqlite_stat1 table is as follows:


CREATE TABLE sqlite_stat1(tbl,idx,stat);


 There is normally one row per index, with the index identified by the
name in the sqlite_stat1.idx column.  The sqlite_stat1.tbl column is
the name of the table to which the index belongs.  In each such row, 
the sqlite_stat.stat column will be
a string consisting of a list of integers followed by zero or more 
arguments.  The first integer in this
list is the approximate number of rows in the index.  (The number of
rows in the index is the same as the number of rows in the table,
except for partial indexes.)
The second integer is the approximate number of rows in the index
that have the same value in the first column of the index.  The third 
integer is the number of rows in the index that have 
the same value for the first two columns.  The N-th integer (for N>1) 
is the estimated average number of rows in 
the index which have the same value for the first N-1 columns.  For
a K-column index, there will be K+1 integers in the stat column.  If
the index is unique, then the last integer will be 1.

The list of integers in the stat column can optionally be followed
by arguments, each of which is a sequence of non-space characters.
All arguments are preceded by a single space.
Unrecognized arguments are silently ignored.

If the "unordered" argument is present, then the query planner assumes
that the index is unordered and will not use the index for a range query
or for sorting.

The "sz=NNN" argument (where NNN represents a sequence of 1 or more digits)
means that the average row size over all records of the table or
index is NNN bytes per row.  The SQLite query planner might use the
estimated row size information provided by the "sz=NNN" token
to help it choose smaller tables and indexes that require less disk I/O.

The presence of the "noskipscan" token on the sqlite_stat1.stat field
of an index prevents that index from being used with the
skip-scan optimization.

New text tokens may be added to the end of the stat column in future
enhancements to SQLite.  For compatibility, unrecognized tokens at the end
of the stat column are silently ignored.

If the sqlite_stat1.idx column is NULL, then the sqlite_stat1.stat
column contains a single integer which is the approximate number of
rows in the table identified by sqlite_stat1.tbl.
If the sqlite_stat1.idx column is the same as the sqlite_stat1.tbl
column, then the table is a WITHOUT ROWID table and the sqlite_stat1.stat
field contains information about the index btree that implements the
WITHOUT ROWID table.


c2: Database File Format
c3: 2.6.5. The sqlite_stat2 table
c5: fileformat.html#the_sqlite_stat2_table
c4:
The sqlite_stat2 is only created and is only used if SQLite is compiled
with SQLITE_ENABLE_STAT2 and if the SQLite version number is between
3.6.18 (2009-09-11) and 3.7.8 (2011-09-19).
The sqlite_stat2 table is neither read nor written by any
version of SQLite before 3.6.18 nor after 3.7.8.
The sqlite_stat2 table contains additional information
about the distribution of keys within an index.
The schema of the sqlite_stat2 table is as follows:


CREATE TABLE sqlite_stat2(tbl,idx,sampleno,sample);


The sqlite_stat2.idx column and the sqlite_stat2.tbl column in each 
row of the sqlite_stat2 table identify an index described by that row.
There are usually 10 rows in the sqlite_stat2
table for each index.

The sqlite_stat2 entries for an index that have sqlite_stat2.sampleno
between 0 and 9 inclusive are samples of the left-most key value in the
index taken at evenly spaced points along the index.
Let C be the number of rows in the index.
Then the sampled rows are given by


     rownumber = (i*C*2 + C)/20


The variable i in the previous expression varies between 0 and 9.
Conceptually, the index space is divided into
10 uniform buckets and the samples are the middle row from each bucket.

The format for sqlite_stat2 is recorded here for legacy reference.  
Recent versions of SQLite no longer support sqlite_stat2 and the
sqlite_stat2 table, if is exists, is simply ignored.


c2: Database File Format
c3: 2.6.6. The sqlite_stat3 table
c5: fileformat.html#the_sqlite_stat3_table
c4:
The sqlite_stat3 is only used if SQLite is compiled
with SQLITE_ENABLE_STAT3 or SQLITE_ENABLE_STAT4
and if the SQLite version number is 3.7.9 (2011-11-01) or greater.
The sqlite_stat3 table is neither read nor written by any
version of SQLite before 3.7.9.
If the SQLITE_ENABLE_STAT4 compile-time option is used and the
SQLite version number is 3.8.1 (2013-10-17) or greater,
then sqlite_stat3 might be read but not written.
The sqlite_stat3 table contains additional information
about the distribution of keys within an index, information that the
query planner can use to devise better and faster query algorithms.
The schema of the sqlite_stat3 table is as follows:


CREATE TABLE sqlite_stat3(tbl,idx,nEq,nLt,nDLt,sample);


There are usually multiple entries in the sqlite_stat3 table for each index.
The sqlite_stat3.sample column holds the value of the left-most field of an
index identified by sqlite_stat3.idx and sqlite_stat3.tbl.
The sqlite_stat3.nEq column holds the approximate
number of entries in the index whose left-most column exactly matches
the sample.
The sqlite_stat3.nLt holds the approximate number of entries in the
index whose left-most column is less than the sample.
The sqlite_stat3.nDLt column holds the approximate
number of distinct left-most entries in the index that are less than
the sample.

There can be an arbitrary number of sqlite_stat3 entries per index.
The ANALYZE command will typically generate sqlite_stat3 tables
that contain between 10 and 40 samples that are distributed across
the key space and with large nEq values.

In a well-formed sqlite_stat3 table, the samples for any single
index must appear in the same order that they occur in the index.  
In other words, if the entry with left-most column S1 is earlier in
the index b-tree than the
entry with left-most column S2, then in the sqlite_stat3 table, 
sample S1 must have a smaller rowid than sample S2.


c2: Database File Format
c3: 2.6.7. The sqlite_stat4 table
c5: fileformat.html#the_sqlite_stat4_table
c4:
The sqlite_stat4 is only created and is only used if SQLite is compiled
with SQLITE_ENABLE_STAT4 and if the SQLite version number is
3.8.1 (2013-10-17) or greater.
The sqlite_stat4 table is neither read nor written by any
version of SQLite before 3.8.1.
The sqlite_stat4 table contains additional information
about the distribution of keys within an index or the distribution of
keys in the primary key of a WITHOUT ROWID table.
The query planner can sometimes use the additional information in
the sqlite_stat4 table to devise better and faster query algorithms.
The schema of the sqlite_stat4 table is as follows:


CREATE TABLE sqlite_stat4(tbl,idx,nEq,nLt,nDLt,sample);


There are typically between 10 to 40 entries in the sqlite_stat4 table for
each index for which statistics are available, however these limits are
not hard bounds.
The meanings of the columns in the sqlite_stat4 table are as follows:



tbl:
    The sqlite_stat4.tbl column holds name of the table that owns
    the index that the row describes

idx:
    The sqlite_stat4.idx column holds name of the index that the
    row describes, or in the case of
    an sqlite_stat4 entry for a WITHOUT ROWID table, the
    name of the table itself.

sample:
    The sqlite_stat4.sample column holds a BLOB
    in the record format that encodes the indexed columns followed by
    the rowid for a rowid table or by the columns of the primary key 
    for a WITHOUT ROWID table.
    The sqlite_stat4.sample BLOB for the WITHOUT ROWID table itself 
    contains just the columns of the primary key.
    Let the number of columns encoded by the sqlite_stat4.sample blob be N.
    For indexes on an ordinary rowid table, N will be one more than the number 
    of columns indexed.
    For indexes on WITHOUT ROWID tables, N will be the number of columns
    indexed plus the number of columns in the primary key.
    For a WITHOUT ROWID table, N will be the number of columns in the
    primary key.

nEq:
    The sqlite_stat4.nEq column holds a list of N integers where 
    the K-th integer is the approximate number of entries in the index
    whose left-most K columns exactly match the K left-most columns
    of the sample.

nLt:
    The sqlite_stat4.nLt column holds a list of N integers where
    the K-th integer is the approximate number of entries in the
    index whose K left-most columns are collectively less than the 
    K left-most columns of the sample.

nDLt:
    The sqlite_stat4.nDLt column holds a list of N integers where
    the K-th integer is the approximate
    number of entries in the index that are distinct in the first K columns and
    where the left-most K columns are collectively less than the left-most
    K columns of the sample.



The sqlite_stat4 is a generalization of the sqlite_stat3 table.  The
sqlite_stat3 table provides information about the left-most column of an
index whereas the sqlite_stat4 table provides information about all columns
of the index.

There can be an arbitrary number of sqlite_stat4 entries per index.
The ANALYZE command will typically generate sqlite_stat4 tables
that contain between 10 and 40 samples that are distributed across
the key space and with large nEq values.

In a well-formed sqlite_stat4 table, the samples for any single
index must appear in the same order that they occur in the index.  
In other words, if entry S1 is earlier in the index b-tree than 
entry S2, then in the sqlite_stat4 table, sample S1 must have a
smaller rowid than sample S2.


c2: Database File Format
c3: 3. The Rollback Journal
c5: fileformat.html#the_rollback_journal
c4:
The rollback journal is a file associated with each SQLite database
file that holds information used to restore the database file to its initial
state during the course of a transaction.
The rollback journal file is always located in the same 
directory as the database
file and has the same name as the database file but with the string
"-journal" appended.  There can only be a single rollback journal
associated with a give database and hence there can only be one write
transaction open against a single database at one time.

If a transaction is aborted due to an application crash, an operating
system crash, or a hardware power failure or crash, then the database may
be left in an inconsistent state.  The next time SQLite attempts to open
the database file, the presence of the rollback journal file will be 
detected and the journal will be automatically played back to restore the
database to its state at the start of the incomplete transaction.

A rollback journal is only considered to be valid if it exists and
contains a valid header.  Hence a transaction can be committed in one
of three ways:

The rollback journal file can be deleted,
The rollback journal file can be truncated to zero length, or
The header of the rollback journal can be overwritten with
invalid header text (for example, all zeros).


These three ways of committing a transaction correspond to the DELETE,
TRUNCATE, and PERSIST settings, respectively, of the journal_mode pragma.



A valid rollback journal begins with a header in the following format:


Rollback Journal Header Format

OffsetSizeDescription
0
    8
    Header string:  0xd9, 0xd5, 0x05, 0xf9, 0x20, 0xa1, 0x63, 0xd7
8
    4
    The "Page Count" - The number of pages in the next segment of the 
        journal, or -1 to
        mean all content to the end of the file
12
    4
    A random nonce for the checksum
16
    4
    Initial size of the database in pages
20
    4
    Size of a disk sector assumed by the process that wrote this
        journal.
24
    4
    Size of pages in this journal.



A rollback journal header is padded with zeros out to the size of a 
single sector (as defined by the sector size integer at offset 20).
The header is in a sector by itself so that if a power loss occurs while
writing the sector, information that follows the header will be
(hopefully) undamaged.

After the header and zero padding are zero or more page records.  Each
page record stores a copy of the content of a page from the database file
before it was changed.  The same page may not appear more than once
within a single rollback journal.
To rollback an incomplete transaction, a process
has merely to read the rollback journal from beginning to end and
write pages found in the journal back into the database file at the
appropriate location.

Let the database page size (the value of the integer at offset 24 
in the journal header) be N.
Then the format of a page record is as follows:


Rollback Journal Page Record Format

OffsetSizeDescription
0
    4
    The page number in the database file
4
    N
    Original content of the page prior to the start of the transaction
N+4
    4
    Checksum




The checksum is an unsigned 32-bit integer computed as follows:


Initialize the checksum to the checksum nonce value found in the
journal header at offset 12.
Initialize index X to be N-200 (where N is the size of a database page
in bytes.
Interpret the byte at offset X into the page as an 8-bit unsigned integer
 and add the value of that integer to the checksum.
Subtract 200 from X.
If X is greater than or equal to zero, go back to step 3.


The checksum value is used to guard against incomplete writes of
a journal page record following a power failure.  A different random nonce
is used each time a transaction is started in order to minimize the risk
that unwritten sectors might by chance contain data from the same page
that was a part of prior journals.  By changing the nonce for each
transaction, stale data on disk will still generate an incorrect checksum
and be detected with high probability.  The checksum only uses a sparse sample
of 32-bit words from the data record for performance reasons - design studies 
during the planning phases of SQLite 3.0.0 showed
a significant performance hit in checksumming the entire page.

Let the page count value at offset 8 in the journal header be M.
If M is greater than zero then after M page records the journal file
may be zero padded out to the next multiple of the sector size and another
journal header may be inserted.  All journal headers within the same
journal must contain the same database page size and sector size.

If M is -1 in the initial journal header, then the number of page records
that follow is computed by computing how many page records will fit in
the available space of the remainder of the journal file.


c2: Database File Format
c3: 4. The Write-Ahead Log
c5: fileformat.html#the_write_ahead_log
c4:
Beginning with version 3.7.0 (2010-07-21), 
SQLite supports a new transaction
control mechanism called "write-ahead log" or "WAL".
When a database is in WAL mode, all connections to that database must
use the WAL.  A particular database will use either a rollback journal
or a WAL, but not both at the same time.
The WAL is always located in the same directory as the database
file and has the same name as the database file but with the string
"-wal" appended.


c2: Database File Format
c3: 4.1. WAL File Format
c5: fileformat.html#wal_file_format
c4:
A WAL file consists of a header followed by zero or more "frames".
Each frame records the revised content of a single page from the
database file.  All changes to the database are recorded by writing
frames into the WAL.  Transactions commit when a frame is written that
contains a commit marker.  A single WAL can and usually does record 
multiple transactions.  Periodically, the content of the WAL is
transferred back into the database file in an operation called a
"checkpoint".

A single WAL file can be reused multiple times.  In other words, the
WAL can fill up with frames and then be checkpointed and then new
frames can overwrite the old ones.  A WAL always grows from beginning
toward the end.  Checksums and counters attached to each frame are
used to determine which frames within the WAL are valid and which
are leftovers from prior checkpoints.

The WAL header is 32 bytes in size and consists of the following eight
big-endian 32-bit unsigned integer values:


WAL Header Format

OffsetSizeDescription
04
    Magic number.  0x377f0682 or 0x377f0683
44
    File format version.  Currently 3007000.
84
    Database page size.  Example: 1024
124
    Checkpoint sequence number
164
    Salt-1: random integer incremented with each checkpoint
204
    Salt-2: a different random number for each checkpoint
244
    Checksum-1: First part of a checksum on the first 24 bytes of header
284
    Checksum-2: Second part of the checksum on the first 24 bytes of header



Immediately following the wal-header are zero or more frames. Each
frame consists of a 24-byte frame-header followed by a page-size bytes
of page data. The frame-header is six big-endian 32-bit unsigned 
integer values, as follows:


WAL Frame Header Format

OffsetSizeDescription
04
    Page number
44
    For commit records, the size of the database file in pages
        after the commit.  For all other records, zero.
84
    Salt-1 copied from the WAL header
124
    Salt-2 copied from the WAL header
164
    Checksum-1:  Cumulative checksum up through and including this page
204
    Checksum-2:  Second half of the cumulative checksum.



A frame is considered valid if and only if the following conditions are
true:


The salt-1 and salt-2 values in the frame-header match
       salt values in the wal-header

The checksum values in the final 8 bytes of the frame-header
       exactly match the checksum computed consecutively on the
       first 24 bytes of the WAL header and the first 8 bytes and
       the content of all frames
       up to and including the current frame.


c2: Database File Format
c3: 4.2. Checksum Algorithm
c5: fileformat.html#checksum_algorithm
c4:
The checksum is computed by interpreting the input as
an even number of unsigned 32-bit integers: x(0) through x(N).
The 32-bit integers are big-endian if the
magic number in the first 4 bytes of the WAL header is 0x377f0683 and
the integers are little-endian if the magic number is 0x377f0682.
The checksum values are always stored in the frame header in a
big-endian format regardless of which byte order is used to compute
the checksum.

The checksum algorithm only works for content which is a multiple of
8 bytes in length.  In other words, if the inputs are x(0) through x(N)
then N must be odd.
The checksum algorithm is as follows:

 
s0 = s1 = 0
for i from 0 to n-1 step 2:
   s0 += x(i) + s1;
   s1 += x(i+1) + s0;
endfor
# result in s0 and s1


The outputs s0 and s1 are both weighted checksums using Fibonacci weights
in reverse order.  (The largest Fibonacci weight occurs on the first element
of the sequence being summed.)  The s1 value spans all 32-bit integer
terms of the sequence whereas s0 omits the final term.


c2: Database File Format
c3: 4.3. Checkpoint Algorithm
c5: fileformat.html#checkpoint_algorithm
c4:
On a checkpoint, the WAL is first flushed to persistent storage using
the xSync method of the VFS. 
Then valid content of the WAL is transferred into the database file.
Finally, the database is flushed to persistent storage using another
xSync method call.
The xSync operations serve as write barriers - all writes launched
before the xSync must complete before any write that launches after the
xSync begins.

A checkpoint need not run to completion.  It might be that some
readers are still using older transactions with data that is contained
in the database file.  In that case, transferring content for newer
transactions from the WAL file into the database would delete the content
out from under readers still using the older transactions.  To avoid that,
checkpoints only run to completion if all reader are using the
last transaction in the WAL.


c2: Database File Format
c3: 4.4. WAL Reset
c5: fileformat.html#wal_reset
c4:
After a complete checkpoint, if no other connections are in transactions
that use the WAL, then subsequent write transactions can
overwrite the WAL file from the beginning.  This is called "resetting the
WAL".  At the start of the first new
write transaction, the WAL header salt-1 value is incremented
and the salt-2 value is randomized.  These changes to the salts invalidate
old frames in the WAL that have already been checkpointed but not yet
overwritten, and prevent them from being checkpointed again.

The WAL file can optionally be truncated on a reset, but it need not be.
Performance is usually a little better if the WAL is not truncated, since
filesystems generally will overwrite an existing file faster than they
will grow a file.


c2: Database File Format
c3: 4.5. Reader Algorithm
c5: fileformat.html#reader_algorithm
c4:
To read a page from the database (call it page number P), a reader
first checks the WAL to see if it contains page P.  If so, then the
last valid instance of page P that is followed by a commit frame
or is a commit frame itself becomes the value read.  If the WAL
contains no copies of page P that are valid and which are a commit
frame or are followed by a commit frame, then page P is read from
the database file.

To start a read transaction, the reader records the number of value
frames in the WAL as "mxFrame".  (More detail)
The reader uses this recorded mxFrame value
for all subsequent read operations.  New transactions can be appended
to the WAL, but as long as the reader uses its original mxFrame value
and ignores subsequently appended content, the reader will see a 
consistent snapshot of the database from a single point in time.  
This technique allows multiple concurrent readers to view different 
versions of the database content simultaneously.

The reader algorithm in the previous paragraphs works correctly, but 
because frames for page P can appear anywhere within the WAL, the
reader has to scan the entire WAL looking for page P frames.  If the
WAL is large (multiple megabytes is typical) that scan can be slow,
and read performance suffers.  To overcome this problem, a separate
data structure called the wal-index is maintained to expedite the
search for frames of a particular page.


c2: Database File Format
c3: 4.6. WAL-Index Format
c5: fileformat.html#wal_index_format
c4:
Conceptually, the wal-index is shared memory, though the current
VFS implementations use a memory-mapped file for operating-system
portability.  The memory-mapped
file is in the same directory as the database and has the same name
as the database with a "-shm" suffix appended.  Because
the wal-index is shared memory, SQLite does not support 
journal_mode=WAL 
on a network filesystem when clients are on different machines, as
all clients of the database must be able to share the same memory.

The purpose of the wal-index is to answer this question quickly:


Given a page number P and a maximum WAL frame index M,
return the largest WAL frame index for page P that does not exceed M, 
or return NULL if there are no frames for page P that do not exceed M.


The M value in the previous paragraph is the "mxFrame" value
defined in section 4.4 that is read at the start 
of a transaction and which defines the maximum frame from the WAL that 
the reader will use.

The wal-index is transient.  After a crash, the wal-index is
reconstructed from the original WAL file.  The VFS is required
to either truncate or zero the header of the wal-index when the last
connection to it closes.  Because the wal-index is transient, it can
use an architecture-specific format; it does not have to be cross-platform.
Hence, unlike the database and WAL file formats which store all values
as big endian, the wal-index stores multi-byte values in the native
byte order of the host computer.

This document is concerned with the persistent state of the database
file, and since the wal-index is a transient structure, no further 
information about the format of the wal-index will be provided here.
Additional details on the format of the wal-index are contained in
the separate WAL-index File Format document.


c2: TH3
c3: 1. Overview
c5: th3.html#overview
c4:
SQLite Test Harness #3 (hereafter "TH3") is one of
three test harnesses used for testing SQLite.
TH3 meets the following objectives:


 TH3 is able to run on embedded platforms that lack the support
     infrastructure of workstations.

 TH3 tests SQLite in an as-deployed configuration using only
     published and documented interfaces.
     In other words, TH3 tests the compiled object code, not
     the source code, thus verifying that no problems were introduced
     by compiler bugs.  "Test what you fly and fly what you test."

 TH3 checks SQLite's response to out-of-memory errors, disk I/O
     errors, and power loss during transaction commit. 

 TH3 exercises SQLite in a variety of run-time configurations
     (UTF8 vs UTF16, different pages sizes, varying journal modes, etc.)
     

 TH3 achieves 100% branch test coverage (and 100% 
    MC/DC)
    over the SQLite core.
    (Test coverage of extensions such as FTS and RTREE is less than 100%).
     


TH3 was originally written for validation testing only, but has
subsequently been used for development testing and debugging
as well, and has proven very helpful in those roles.  A full-coverage
test takes less than five minutes on a workstation and hence
serves as a fast regression test during day-to-day maintenance
of the SQLite code base.


c2: TH3
c3: 1.1. History
c5: th3.html#history
c4:
TH3 originated from an effort to test SQLite on 
SymbianOS.
Prior to TH3, all SQLite tests were run using the
TCL script language, but TCL would not (easily)
compile on SymbianOS which made testing difficult.  The first attempt
to remedy this problem was the "TH1" (Test Harness #1) scripting 
language - a reimplementation of parts of the TCL language in a 
more portable form that would compile and run on SymbianOS, and 
that was sufficient to run the SQLite tests.  TH1
did not survive as a standard testing tool for SQLite,
but it did find continued service as a
scripting language used to customize the 
Fossil version control system.
There was also a "Test Harness #2" which was an attempt to
create a simple scripting language using operator prefix notation
to drive tests. TH3 was the third attempt.

At about that same time, some avionics manufacturers were
expressing interest in SQLite, which prompted the SQLite developers
to design TH3 to support the rigorous testing standards of
DO-178B.

The first code for TH3 was laid down on 2008-09-25.
An intense effort over the next 10 months resulted in TH3 achieving
100% MC/DC on 2009-07-25.  The TH3 code continues to be improved and
expanded.

As of 2018-05-19, the TH3 source tree consists 
and well over 500,000 lines of source code in 1709 separate files.


c2: TH3
c3: 2. Operation
c5: th3.html#operation
c4:
TH3 is a test program generator.  The output of TH3 is a program
implemented in C-code and intended to be
linked against the SQLite library under test.  The generated test
program is compiled and run on the target platform in order to verify
correct operation of SQLite on that platform.

The inputs to TH3 are test modules written in C or SQL and
small configuration
files that determine how to initialize SQLite.  The
TH3 package includes  1,444 test
modules and more than 47 configurations (as of 2018-05-19).
New modules and configurations
can be added to customize TH3 for specialized applications.
Each time TH3 is run, it reads
a subset of the available test modules and configuration files to generate
a custom C program that performs all of the specified tests under all
specified configurations.  A complete test of SQLite normally involves running
TH3 multiple times to generate multiple test programs covering different
aspects of SQLite's operation, then linking all test programs against
a common SQLite library and running them separately on the target platform.


There are no arbitrary limits in TH3.  One could generate a
single test program that contained all test modules and all configuration files.
However, such a test program might be too large to deploy on embedded
platforms.  (As of 2018-05-19, a full-up TH3 test is over 850,000 lines and
58MB of C code.)  TH3 provides the ability to break the library of test
modules up into smaller, more easily digested pieces.

Each individual test module might contain dozens, hundreds, or thousands
of separate tests.  The test modules can be written in C or as scripts of
SQL or a mixture of the two.  About two-thirds of the existing test modules are
written in pure SQL with the remainder either in pure C or a combination 
of C and SQL.


Each test module file contains a header which describes the circumstances
under which the test is valid.  For a particular configuration, only those
modules that are compatible with the configuration are run.


c2: TH3
c3: 3. Generating A Test Program
c5: th3.html#generating_a_test_program
c4:
The TH3 program generator is a TCL script named "mkth3.tcl".
To generate a test program, one has merely to run this script and supply
the names of files containing test modules and configurations on the
command line.  Test modules are files that use the ".test" suffix
and configurations are files that use the ".cfg" suffix.  A
typical invocation of mkth3.tcl might look something like the following:

tclsh mkth3.tcl *.test *.cfg &gt;testprog1.c


The output from the mkth3.tcl script is a C program that contains
everything needed to run the tests - everything that is except for
the SQLite library itself.  The generated test program contains 
implementations for all of the support interfaces used by the test
modules and it contains the main() routine that drives the
tests.  To convert the test program into a working executable, simply
compile it against SQLite:

cc -o testprog1 testprog1.c sqlite3.c


The compilation step shown immediately above is merely representative.
In a working installation, one would normally want
to specify optimization parameters and compile-time switches on the
compiler command line.

For testing on embedded systems, the mkth3.tcl script and the compiler
steps shown above are performed on an ordinary workstation using
a cross-compiler, then the resulting test program is
transferred onto the device to be run.

Once the test program is generated, it is run with no arguments to
perform the tests.  Progress information as well as error diagnostics
appear on standard output.  (Alternative output arrangements can be made
using a compile-time option for embedded devices that lack a standard
output channel.) The program returns zero if there are no
errors and non-zero if any problems were detected.

Typical output from a single TH3 test program run looks like this:

With SQLite 3.8.11 2015-05-15 04:13:15 56ef98a04765c34c1c2f3ed7a6f03a732f3b886e
-DSQLITE_COVERAGE_TEST
-DSQLITE_NO_SYNC
-DSQLITE_SYSTEM_MALLOC
-DSQLITE_THREADSAFE=1
Config-begin c1.
Begin c1.pager08
End c1.pager08
Begin c1.build33
End c1.build33
Begin c1.orderby01
End c1.orderby01
... 15014 lines of output omitted ....
Begin 64k.syscall01
End 64k.syscall01
Begin 64k.build01
End 64k.build01
Begin 64k.auth01
End 64k.auth01
Config-end 64k. TH3 memory used: 6373738
Config-begin wal1.
Begin wal1.wal37
End wal1.wal37
Config-end wal1. TH3 memory used: 100961
All 226 VDBE coverage points reached
th3: 0 errors out of 1442264 tests in 213.741 seconds. 64-bit little-endian
th3: SQLite 3.8.11 2015-05-15 04:13:15 56ef98a04765c34c1c2f3ed7a6f03a732f3b886e


The output begins with a report of the SQLITE_SOURCE_ID
(cross-checked again sqlite3_sourceid()) for the
SQLite under test and the compile-time options used as reported
by sqlite3_compileoption_get().  The output concludes with a summary
of the test results and a repeat of the SQLITE_SOURCE_ID.  If any
errors are detected, additional lines detail the problem.  The error
reporting lines always begin with a single space character so that they
can be quickly extracted from large output files using:

grep "&#94; "


The default output shows the beginning and end of each configuration
and test module combination.  In the example above "c1" and "64k" are
configurations and "pager08", "build33", "orderby01", etc. are test modules.
Compile-time and run-time options are available to increase or decrease
the amount of output.
The output can be increased by showing each test case within each
test module.  The output can be decreased
by degrees: omitting test modules starts and stops,
omitting configuration starts and stops, and finally by omitting all output.


c2: TH3
c3: 3.1. Test Automation Scripts
c5: th3.html#test_automation_scripts
c4:
TH3 comes with additional TCL scripts that help automate the testing
process on workstations.  The "th3make" script automatically runs "mkth3.tcl"
and "gcc" and then runs the resulting test program and checks the results.
Arguments to th3make include all of the "*.test" test modules and 
"*.cfg" configurations that are to be included in the test.  Additional
options to th3make can cause the test program to be compiled using different
compilers (GCC, Clang, MSVC), to use different output verbosity levels, to
run the test program under valgrind, to check the output for coverage using
gcov, and so forth.  The th3make script also accepts "*.rc" filenames as
arguments.  These *.rc files are just collections of other arguments that
are commonly used together for a single purpose.  For example, the "quick.rc"
file contains a set of eight arguments to th3make that run a fast (3-minute)
full-coverage test.  This allows the operator to type "./th3make quick.rc" as
a short-cut to typing out all of the required command-line options.  The
following are a few of the more than 40 available *.rc files:


alignmentN.rc - 
    These files contain -D options to the compiler that are used by
    various notable downstreams.
cov.rc - Options for measuring test coverage
extensions.rc - Options to enable FTS4, R-Trees,
    and STAT4.
fast.rc - Run most tests, including those not needed for
    coverage, skipping only soak tests, using delivery compiler options
    (ex: -O3)
memdebug.rc - like test.rc but also enable
    -DSQLITE_MEMDEBUG.
min.rc - Run the minimum set of tests needed for 100% coverage.
quick.rc - Run all tests required for 100% coverage tests
    using -Os and -DSQLITE_DEBUG.
test.rc - Run the same tests as in fast.rc but without
    compiler optimization and enabling options like
    -DSQLITE_DEBUG and
    -DSQLITE_ENABLE_EXPENSIVE_ASSERT.
test-ex.rc - long-running soak tests.


The TH3 repository also includes the "multitest.tcl" script, another
TCL script used to automate TH3 testing on workstations.  Multitest.tcl
automatically compiles SQLite, then
runs ./th3make repeatedly with a variety of alignments, and captures
the output in a succinct summary screen.  A typical multitest.tcl run
generates output that looks like this:

./multitest.tcl -q --jobs 3
start-time: 2018-05-19 03:17:12 UTC
file mkdir sqlite3bld
cd sqlite3bld
exec sh /ramdisk/sqlite/configure
file copy -force config.h ../config.h
exec make clean sqlite3.c
file rename sqlite3.c ../sqlite3.c
file rename sqlite3.h ../sqlite3.h
exec make clean sqlite3.c OPTS=-DSQLITE_ENABLE_UPDATE_DELETE_LIMIT=1
file rename sqlite3.c ../sqlite3udl.c
exec make clean sqlite3.c OPTS=-DSQLITE_SMALL_STACK=1
file rename sqlite3.c ../sqlite3ss.c
cd ..
*******************************************************************************
t01: cov.rc.................................................... Ok   (00:03:42)
t02: cov.rc ++STAT4 ++DESERIALIZE -D_HAVE_SQLITE_CONFIG_H...... Ok   (00:04:45)
t03: vfs-cov.rc................................................ Ok   (00:03:59)
t04: demo.rc................................................... Ok   (00:00:05)
t07: test.rc ../th3private/*.test.............................. Ok   (00:00:21)
t08: test.rc ../th3private/*.test ++STAT4...................... Ok   (00:01:41)
t05: quick.rc.................................................. Ok   (00:04:26)
t09: quick.rc ~TEST_REALLOC_STRESS -funsigned-char............. Ok   (00:05:39)
t10: quick.rc ~THREADSAFE=0 -DLONGDOUBLE_TYPE=double........... Ok   (00:03:24)
t06: quick.rc extensions.rc -D_HAVE_SQLITE_CONFIG_H............ Ok   (00:09:03)
t11: quick.rc sqlite3ss.c ~MAX_ATTACHED=125.................... Ok   (00:04:39)
t12: quick.rc ~BYTEORDER=0 ++RTREE............................. Ok   (00:07:28)
t13: quick.rc ~DISABLE_INTRINSIC ++RTREE....................... Ok   (00:07:31)
t16: quick.rc ~TRACE_SIZE_LIMIT=15 cov1/main16.test............ Ok   (00:00:22)
t14: quick.rc ~DIRECT_OVERFLOW_READ -fsigned-char.............. Ok   (00:04:35)
t15: quick.rc ~UNTESTABLE ~EXTRA_IFNULLROW..................... Ok   (00:01:44)
t17: quick.rc ~MAX_MMAP_SIZE=0................................. Ok   (00:04:46)
t18: quick.rc ++NULL_TRIM ++OFFSET_SQL_FUNC.................... Ok   (00:04:47)
t19: quick.rc ++BATCH_ATOMIC_WRITE ++DESERIALIZE............... Ok   (00:05:41)
t20: lean1.rc quick.rc......................................... Ok   (00:03:09)
t22: test.rc alignment2.rc sqlite3udl.c........................ Ok   (00:44:22)
t21: test.rc alignment1.rc..................................... Ok   (01:02:32)
t23: memdebug1.rc extensions.rc................................ Ok   (01:49:58)
t25: valgrind1.rc -O3 extensions.rc............................ Ok   (00:56:08)
t24: memdebug2.rc extensions.rc................................ Ok   (01:43:34)
t27: test-ex1.rc............................................... Ok   (00:45:00)
t26: valgrind2.rc -O3 extensions.rc............................ Ok   (01:02:52)
t29: test-ex3.rc............................................... Ok   (00:31:48)
t28: test-ex2.rc............................................... Ok   (01:12:03)
t30: test-ex4.rc............................................... Ok   (01:09:47)
t32: test.rc alignment4.rc -m32 CC=clang....................... Ok   (00:48:31)
t31: test.rc alignment3.rc sqlite3udl.c........................ Ok   (01:22:29)
t34: test.rc alignment6.rc..................................... Ok   (00:35:31)
t33: test.rc alignment5.rc extensions.rc....................... Ok   (00:59:33)
t35: test.rc alignment7.rc..................................... Ok   (00:44:10)
t40: fast.rc alignment2.rc sqlite3udl.c........................ Ok   (00:15:46)
t39: fast.rc alignment1.rc extensions.rc -m32.................. Ok   (00:33:19)
t36: test.rc ~MUTATION_TEST.................................... Ok   (00:35:45)
t42: fast.rc alignment4.rc..................................... Ok   (00:13:03)
t43: fast.rc alignment5.rc..................................... Ok   (00:13:32)
t44: fast.rc alignment6.rc..................................... Ok   (00:11:41)
t41: fast.rc alignment3.rc sqlite3udl.c........................ Ok   (00:26:31)
t45: fast.rc alignment7.rc..................................... Ok   (00:12:57)
t46: fast.rc -fsanitize=undefined.............................. Ok   (00:38:18)
*******************************************************************************
0 failures on 44 th3makes and 198583082 tests in (07:16:01) 3 cores on bella
SQLite 3.24.0 2018-05-18 17:58:33 c6071ac99cfa4b6272ac4d739fc61a85acb544f6c1c2a


As can be seen above, a single run
of multitest.tcl invokes th3make dozens of times and takes between 12 and 24
CPU hours.  The middle section of the output shows the arguments to each
individual th3make run and the result and elapse time for that th3make.
All build products and output for the separate th3make runs are
captures in subdirectories for post-test analysis.
The two-line summary at the bottom shows the total number of errors and tests
over all th3make runs and the total elapse time, together with the 
SQLITE_SOURCE_ID information for the version of SQLite that was
tested.  This summary information is recorded in the
release
checklist during final testing.

Abbreviations are applied in the multitest.tcl output so that
each th3make invocation will fit on a single 80-column output line.
The initial "th3make" verb is omitted.
"~" is shorthand for "-DSQLITE_" and "++" is stands for
"-DSQLITE_ENABLE".  Hence, multitest.tcl output line

quick.rc ~DISABLE_INTRINSIC ++RTREE


Really means

th3make quick.rc -DSQLITE_DISABLE_INTRINSIC -DSQLITE_ENABLE_RTREE


c2: TH3
c3: 4. Test Coverage
c5: th3.html#test_coverage
c4:
Using one particular subset of the available TH3 test modules (the "cov1"
tests) SQLite obtained 
100% branch test coverage and 100% MC/DC as measured
by gcov
on Linux x86 and x86_64 hardware.  All releases of SQLite since
version 3.6.17 (2009-08-10) have been tested to this standard. 
The SQLite developers 
are committed to maintaining 100% branch coverage and MC/DC for all 
future releases of SQLite.

The cov1 test set used to obtain 100% branch test coverage are only a
subset of the tests currently implemented using TH3.  New test modules are
added on a regular basis.


c2: TH3
c3: 5. Mutation Testing
c5: th3.html#mutation_testing
c4:
The TH3 source tree contains a scripted name
"mutation-test.tcl" that automates the process of
mutation testing.

The mutation-test.tcl script takes care of all of the details for
running a mutation test:


 The script compiles the TH3 test harness into machine code ("th3.o") if
     necessary.
 The script compiles the sqlite3.c source file into assembly language
     ("sqlite3.s") if necessary.
 The script loops through instructions in the assembly language file
     to locate branch operations.
     
     The script makes a copy of the original sqlite3.s file.
     The copy is edited to change the branch instruction into either
         a no-op or an unconditional jump.
     The copy of sqlite3.s is assembled into sqlite3.o then linked
         again th3.o to generate the "th3" executable.
     The "th3" binary is run and the output checked for errors.
     
 The script shows progress for each cycle of the previous step then
     displays a summary of "survivors" at the end.  A "survivor" is a
     mutation that was not detected by TH3.


Mutation testing can be slow, since each test can take up to 5
minutes on a fast workstation, and there are two tests for each
branch instructions, and over 20,000 branch instructions.  Efforts are
made to expedite operation.  For example, TH3 is compiled in such a
way that it exits as soon as it finds the first error, and as many
of the mutations are easily detected,  many cycles happen in only
a few seconds.  Nevertheless, the mutation-test.tcl script includes
command-line options to limit the range of code lines tested so that
mutation testing only needs to be performed on blocks of code that
have recently changed.


c2: TH3
c3: 6. TH3 License
c5: th3.html#th3_license
c4:
SQLite itself is in the public domain and
can be used for any purpose.  But TH3 is proprietary and requires a license.


Even though open-source users do not have direct access to TH3, all
users of SQLite benefit from TH3 indirectly since each version of SQLite is
validated running TH3 on multiple platforms (Linux, Windows, WinRT, Mac,
OpenBSD) prior to release.  So anyone using an official release
of SQLite can deploy their application with the confidence of knowing that
it has been tested using TH3.  They simply cannot rerun those tests
themselves without purchasing a TH3 license.


c2: Temporary Files Used By SQLite
c3: 1. Introduction
c5: tempfiles.html#introduction
c4:
One of the distinctive features of
SQLite is that a database consists of a single disk file.
This simplifies the use of SQLite since moving or backing up a
database is a simple as copying a single file.  It also makes
SQLite appropriate for use as an
application file format.
But while a complete database is held in a single disk file,
SQLite does make use of many temporary files during the
course of processing a database.



This article describes the various temporary files that SQLite
creates and uses.  It describes when the files are created, when
they are deleted, what they are used for, why they are important,
and how to avoid them on systems where creating temporary files is
expensive.



The manner in which SQLite uses temporary files is not considered
part of the contract that SQLite makes with applications.  The
information in this document is a correct description of how
SQLite operates at the time that this document was written or last
updated.  But there is no guarantee that future versions of SQLite
will use temporary files in the same way.  New kinds of temporary
files might be employed  and some of
the current temporary file uses might be discontinued
in future releases of SQLite.


c2: Temporary Files Used By SQLite
c3: 2. Nine Kinds Of Temporary Files
c5: tempfiles.html#nine_kinds_of_temporary_files
c4:
SQLite currently uses nine distinct types of temporary files:



Rollback journals
Super-journals
Write-ahead Log (WAL) files
Shared-memory files
Statement journals
TEMP databases
Materializations of views and subqueries
Transient indices
Transient databases used by VACUUM



Additional information about each of these temporary file types
is in the sequel.


c2: Temporary Files Used By SQLite
c3: 2.1. Rollback Journals
c5: tempfiles.html#rollback_journals
c4:
A rollback journal is a temporary file used to implement
atomic commit and rollback capabilities in SQLite.
(For a detailed discussion of how this works, see
the separate document titled
Atomic Commit In SQLite.)
The rollback journal is always located in the same directory
as the database file and has the same name as the database
file except with the 8 characters "-journal" appended.
The rollback journal is usually created when a transaction
is first started and is usually deleted when a transaction
commits or rolls back.
The rollback journal file is essential for implementing the
atomic commit and rollback capabilities of SQLite.  Without
a rollback journal, SQLite would be unable to rollback an
incomplete transaction, and if a crash or power loss occurred
in the middle of a transaction the entire database would likely
go corrupt without a rollback journal.



The rollback journal is usually created and destroyed at the
start and end of a transaction, respectively.  But there are exceptions
to this rule.



If a crash or power loss occurs in the middle of a transaction,
then the rollback journal file is left on disk.  The next time
another application attempts to open the database file, it notices
the presence of the abandoned rollback journal (we call it a "hot
journal" in this circumstance) and uses the information in the
journal to restore the database to its state prior to the start
of the incomplete transaction.  This is how SQLite implements
atomic commit.



If an application puts SQLite in 
exclusive locking mode using
the pragma:



PRAGMA locking_mode=EXCLUSIVE;



SQLite creates a new rollback journal at the start of the first
transaction within an exclusive locking mode session.  But at the
conclusion of the transaction, it does not delete the rollback
journal.  The rollback journal might be truncated, or its header
might be zeroed (depending on what version of SQLite you are using)
but the rollback journal is not deleted.  The rollback journal is
not deleted until exclusive access mode is exited.


Rollback journal creation and deletion is also changed by the
journal_mode pragma.
The default journaling mode is DELETE, which is the default behavior
of deleting the rollback journal file at the end of each transaction,
as described above.  The PERSIST journal mode foregoes the deletion of
the journal file and instead overwrites the rollback journal header
with zeros, which prevents other processes from rolling back the
journal and thus has the same effect as deleting the journal file, though
without the expense of actually removing the file from disk.  In other
words, journal mode PERSIST exhibits the same behavior as is seen
in EXCLUSIVE locking mode. The
OFF journal mode causes SQLite to omit the rollback journal, completely.
In other words, no rollback journal is ever written if journal mode is
set to OFF.
The OFF journal mode disables the atomic
commit and rollback capabilities of SQLite.  The ROLLBACK command
is not available when OFF journal mode is set.  And if a crash or power
loss occurs in the middle of a transaction that uses the OFF journal
mode, no recovery is possible and the database file will likely
go corrupt.
The MEMORY journal mode causes the rollback journal to be stored in
memory rather than on disk.  The ROLLBACK command still works when
the journal mode is MEMORY, but because no file exists on disks for
recovery, a crash or power loss in the middle of a transaction that uses
the MEMORY journal mode will likely result in a corrupt database.


c2: Temporary Files Used By SQLite
c3: 2.2. Write-Ahead Log (WAL) Files
c5: tempfiles.html#write_ahead_log_wal_files
c4:
A write-ahead log or WAL file is used in place of a rollback journal
when SQLite is operating in WAL mode.  As with the rollback journal,
the purpose of the WAL file is to implement atomic commit and rollback.
The WAL file is always located in the same directory
as the database file and has the same name as the database
file except with the 4 characters "-wal" appended.
The WAL file is created when the first connection to the
database is opened and is normally removed when the last
connection to the database closes.  However, if the last connection
does not shutdown cleanly, the WAL file will remain in the filesystem
and will be automatically cleaned up the next time the database is
opened.


c2: Temporary Files Used By SQLite
c3: 2.3. Shared-Memory Files
c5: tempfiles.html#shared_memory_files
c4:
When operating in WAL mode, all SQLite database connections associated
with the same database file need to share some memory that is used as an
index for the WAL file.  In most implementations, this shared memory is
implemented by calling mmap() on a file created for this sole purpose:
the shared-memory file.  The shared-memory file, if it exists, is located
in the same directory as the database file and has the same name as the
database file except with the 4 characters "-shm" appended.
Shared memory files only exist while running in WAL mode.



The shared-memory file contains no persistent content.  The only purpose
of the shared-memory file is to provide a block of shared memory for use
by multiple processes all accessing the same database in WAL mode.
If the VFS is able to provide an alternative method for accessing shared
memory, then that alternative method might be used rather than the
shared-memory file.  For example, if PRAGMA locking_mode is set to
EXCLUSIVE (meaning that only one process is able to access the database
file) then the shared memory will be allocated from heap rather than out
of the shared-memory file, and the shared-memory file will never be
created.



The shared-memory file has the same lifetime as its associated WAL file.
The shared-memory file is created when the WAL file is created and is
deleted when the WAL file is deleted.  During WAL file recovery, the
shared memory file is recreated from scratch based on the contents of
the WAL file being recovered.


c2: Temporary Files Used By SQLite
c3: 2.4. Super-Journal Files
c5: tempfiles.html#super_journal_files
c4:
The super-journal file is used as part of the atomic commit
process when a single transaction makes changes to multiple
databases that have been added to a single database connection
using the ATTACH statement.  The super-journal file is always
located in the same directory as the main database file
(the main database file is the database that is identified
in the original sqlite3_open(), sqlite3_open16(), or
sqlite3_open_v2() call that created the database connection)
with a randomized suffix.  The super-journal file contains
the names of all of the various attached auxiliary databases
that were changed during the transaction.  The multi-database
transaction commits when the super-journal file is deleted.
See the documentation titled
Atomic Commit In SQLite for
additional detail.



Without the super-journal, the transaction commit on a multi-database
transaction would be atomic for each database individually, but it
would not be atomic across all databases.  In other words, if the
commit were interrupted in the middle by a crash or power loss, then
the changes to one of the databases might complete while the changes
to another database might roll back.  The super-journal causes all
changes in all databases to either rollback or commit together.



The super-journal file is only created for COMMIT operations that
involve multiple database files where at least two of the databases 
meet all of the following requirements:


The database is modified by the transaction
The PRAGMA synchronous setting is not OFF
The PRAGMA journal_mode is not OFF, MEMORY, or WAL



This means that SQLite transactions are not atomic
across multiple database files on a power-loss when the database 
files have synchronous turned off or when they are using journal 
modes of OFF, MEMORY, or WAL.  For synchronous OFF and for
journal_modes OFF and MEMORY, database will usually corrupt if
a transaction commit is interrupted by a power loss.  For 
WAL mode, individual database files are updated atomically
across a power-loss, but in the case of a multi-file transactions,
some files might rollback while others roll forward after
power is restored.


c2: Temporary Files Used By SQLite
c3: 2.5. Statement Journal Files
c5: tempfiles.html#statement_journal_files
c4:
A statement journal file is used to rollback partial results of
a single statement within a larger transaction.  For example, suppose
an UPDATE statement will attempt to modify 100 rows in the database.
But after modifying the first 50 rows, the UPDATE hits
a constraint violation which should block the entire statement.
The statement journal is used to undo the first 50 row changes
so that the database is restored to the state it was in at the start
of the statement.



A statement journal is only created for an UPDATE or INSERT statement
that might change multiple rows of a database and which might hit a
constraint or a RAISE exception within a trigger and thus need to
undo partial results.
If the UPDATE or INSERT is not contained within BEGIN...COMMIT and if
there are no other active statements on the same database connection then
no statement journal is created since the ordinary
rollback journal can be used instead.
The statement journal is also omitted if an alternative
conflict resolution algorithm is
used.  For example:



UPDATE OR FAIL ...
UPDATE OR IGNORE ...
UPDATE OR REPLACE ...
UPDATE OR ROLLBACK ...
INSERT OR FAIL ...
INSERT OR IGNORE ...
INSERT OR REPLACE ...
INSERT OR ROLLBACK ...
REPLACE INTO ....



The statement journal is given a randomized name, not necessarily
in the same directory as the main database, and is automatically
deleted at the conclusion of the transaction.  The size of the
statement journal is proportional to the size of the change implemented
by the UPDATE or INSERT statement that caused the statement journal
to be created.


c2: Temporary Files Used By SQLite
c3: 2.6. TEMP Databases
c5: tempfiles.html#temp_databases
c4:
Tables created using the "CREATE TEMP TABLE" syntax are only
visible to the database connection in which the "CREATE TEMP TABLE"
statement is originally evaluated.  These TEMP tables, together
with any associated indices, triggers, and views, are collectively
stored in a separate temporary database file that is created as
soon as the first "CREATE TEMP TABLE" statement is seen.
This separate temporary database file also has an associated
rollback journal.
The temporary database file used to store TEMP tables is deleted
automatically when the database connection is closed
using sqlite3_close().



The TEMP database file is very similar to auxiliary database
files added using the ATTACH statement, though with a few
special properties.
The TEMP database is always automatically deleted when the
database connection is closed.
The TEMP database always uses the
synchronous=OFF and journal_mode=PERSIST
PRAGMA settings.
And, the TEMP database cannot be used with DETACH nor can
another process ATTACH the TEMP database.



The temporary files associated with the TEMP database and its
rollback journal are only created if the application makes use
of the "CREATE TEMP TABLE" statement.


c2: Temporary Files Used By SQLite
c3: 2.7. Materializations Of Views And Subqueries
c5: tempfiles.html#materializations_of_views_and_subqueries
c4:
Queries that contain subqueries must sometime evaluate
the subqueries separately and store the results in a temporary
table, then use the content of the temporary table to evaluate
the outer query.
We call this "materializing" the subquery.
The query optimizer in SQLite attempts to avoid materializing,
but sometimes it is not easily avoidable.
The temporary tables created by materialization are each stored
in their own separate temporary file, which is automatically
deleted at the conclusion of the query.
The size of these temporary tables depends on the amount of
data in the materialization of the subquery, of course.



A subquery on the right-hand side of IN operator must often
be materialized.  For example:



SELECT * FROM ex1 WHERE ex1.a IN (SELECT b FROM ex2);



In the query above, the subquery "SELECT b FROM ex2" is evaluated
and its results are stored in a temporary table (actually a temporary
index) that allows one to determine whether or not a value ex2.b
exists using a simple binary search.  Once this table is constructed,
the outer query is run and for each prospective result row a check
is made to see if ex1.a is contained within the temporary table.
The row is output only if the check is true.



To avoid creating the temporary table, the query might be rewritten
as follows:



SELECT * FROM ex1 WHERE EXISTS(SELECT 1 FROM ex2 WHERE ex2.b=ex1.a);



Recent versions of SQLite (version 3.5.4 2007-12-14) and later)
will do this rewrite automatically
if an index exists on the column ex2.b.



If the right-hand side of an IN operator can be list of values
as in the following:


SELECT * FROM ex1 WHERE a IN (1,2,3);


List values on the right-hand side of IN are treated as a 
subquery that must be materialized.  In other words, the
previous statement acts as if it were:


SELECT * FROM ex1 WHERE a IN (SELECT 1 UNION ALL
                              SELECT 2 UNION ALL
                              SELECT 3);


A temporary index is always used to hold the values of the
right-hand side of an IN operator when that right-hand side
is a list of values.



Subqueries might also need to be materialized when they appear
in the FROM clause of a SELECT statement.  For example:



SELECT * FROM ex1 JOIN (SELECT b FROM ex2) AS t ON t.b=ex1.a;



Depending on the query, SQLite might need to materialize the 
"(SELECT b FROM ex2)" subquery into a temporary table, then
perform the join between ex1 and the temporary table.  The
query optimizer tries to avoid this by "flattening" the
query.  In the previous example the query can be flattened,
and SQLite will automatically transform the query into



SELECT ex1.*, ex2.b FROM ex1 JOIN ex2 ON ex2.b=ex1.a;



More complex queries may or may not be able to employ query
flattening to avoid the temporary table.  Whether or not
the query can be flattened depends on such factors as whether
or not the subquery or outer query contain aggregate functions,
ORDER BY or GROUP BY clauses, LIMIT clauses, and so forth.
The rules for when a query can and cannot be flattened are
very complex and are beyond the scope of this document.


c2: Temporary Files Used By SQLite
c3: 2.8. Transient Indices
c5: tempfiles.html#transient_indices
c4:
SQLite may make use of transient indices to
implement SQL language features such as:



An ORDER BY or GROUP BY clause
The DISTINCT keyword in an aggregate query
Compound SELECT statements joined by UNION, EXCEPT, or INTERSECT



Each transient index is stored in its own temporary file.
The temporary file for a transient index is automatically deleted
at the end of the statement that uses it.



SQLite strives to implement ORDER BY clauses using a preexisting
index.  If an appropriate index already exists, SQLite will walk
the index, rather than the underlying table, to extract the 
requested information, and thus cause the rows to come out in
the desired order.  But if SQLite cannot find an appropriate index
it will evaluate the query and store each row in a transient index
whose data is the row data and whose key is the ORDER BY terms.
After the query is evaluated, SQLite goes back and walks the
transient index from beginning to end in order to output the
rows in the desired order.



SQLite implements GROUP BY by ordering the output rows in the
order suggested by the GROUP BY terms.  Each output row is
compared to the previous to see if it starts a new "group".
The ordering by GROUP BY terms is done in exactly the same way
as the ordering by ORDER BY terms.  A preexisting index is used
if possible, but if no suitable index is available, a transient
index is created.



The DISTINCT keyword on an aggregate query is implemented by
creating a transient index in a temporary file and storing
each result row in that index.  As new result rows are computed
a check is made to see if they already exist in the transient
index and if they do the new result row is discarded.



The UNION operator for compound queries is implemented by creating
a transient index in a temporary file and storing the results
of the left and right subquery in the transient index, discarding
duplicates.  After both subqueries have been evaluated, the
transient index is walked from beginning to end to generate the final output.



The EXCEPT operator for compound queries is implemented by creating
a transient index in a temporary file, storing the results of the
left subquery in this transient index, then removing the result 
from right subquery from the transient index, and finally walking
the index from beginning to end to obtain the final output.



The INTERSECT operator for compound queries is implemented by
creating two separate transient indices, each in a separate
temporary file.  The left and right subqueries are evaluated
each into a separate transient index.  Then the two indices
are walked together and entries that appear in both indices
are output.



Note that the UNION ALL operator for compound queries does not
use transient indices by itself (though of course the right
and left subqueries of the UNION ALL might use transient indices
depending on how they are composed.)


c2: Temporary Files Used By SQLite
c3: 2.9. Transient Database Used By VACUUM
c5: tempfiles.html#transient_database_used_by_vacuum_
c4:
The VACUUM command works by creating a temporary file
and then rebuilding the entire database into that temporary
file.  Then the content of the temporary file is copied back
into the original database file and the temporary file is
deleted.



The temporary file created by the VACUUM command exists only
for the duration of the command itself.  The size of the temporary
file will be no larger than the original database.


c2: Temporary Files Used By SQLite
c3: 3. The SQLITE_TEMP_STORE Compile-Time Parameter and Pragma
c5: tempfiles.html#the_sqlite_temp_store_compile_time_parameter_and_pragma
c4:
The temporary files associated with transaction control, namely
the rollback journal, super-journal, write-ahead log (WAL) files,
and shared-memory files, are always written to disk.
But the other kinds of temporary files might be stored in memory
only and never written to disk.
Whether or not temporary files other than the rollback,
super, and statement journals are written to disk or stored only in memory
depends on the SQLITE_TEMP_STORE compile-time parameter, the
temp_store pragma,
and on the size of the temporary file.



The SQLITE_TEMP_STORE compile-time parameter is a #define whose value is
an integer between 0 and 3, inclusive.  The meaning of the
SQLITE_TEMP_STORE compile-time parameter is as follows:




Temporary files are always stored on disk regardless of the setting
of the temp_store pragma.


Temporary files are stored on disk by default but this can be
overridden by the temp_store pragma.


Temporary files are stored in memory by default but this can be
overridden by the temp_store pragma.


Temporary files are always stored in memory regardless of the setting
of the temp_store pragma.




The default value of the SQLITE_TEMP_STORE compile-time parameter is 1,
which means to store temporary files on disk but provide the option
of overriding the behavior using the temp_store pragma.



The temp_store pragma has
an integer value which also influences the decision of where to store
temporary files.  The values of the temp_store pragma have the
following meanings:




Use either disk or memory storage for temporary files as determined
by the SQLITE_TEMP_STORE compile-time parameter.


If the SQLITE_TEMP_STORE compile-time parameter specifies memory storage for
temporary files, then override that decision and use disk storage instead.
Otherwise follow the recommendation of the SQLITE_TEMP_STORE compile-time
parameter.


If the SQLITE_TEMP_STORE compile-time parameter specifies disk storage for
temporary files, then override that decision and use memory storage instead.
Otherwise follow the recommendation of the SQLITE_TEMP_STORE compile-time
parameter.




The default setting for the temp_store pragma is 0,
which means to following the recommendation of SQLITE_TEMP_STORE compile-time
parameter.



To reiterate, the SQLITE_TEMP_STORE compile-time parameter and the 
temp_store pragma only
influence the temporary files other than the rollback journal
and the super-journal.  The rollback journal and the
super-journal are always written to disk regardless of the settings of
the SQLITE_TEMP_STORE compile-time parameter and the
temp_store pragma.


c2: Temporary Files Used By SQLite
c3: 4. Other Temporary File Optimizations
c5: tempfiles.html#other_temporary_file_optimizations
c4:
SQLite uses a page cache of recently read and written database
pages.  This page cache is used not just for the main database
file but also for transient indices and tables stored in temporary
files.  If SQLite needs to use a temporary index or table and
the SQLITE_TEMP_STORE compile-time parameter and the
temp_store pragma are
set to store temporary tables and index on disk, the information
is still initially stored in memory in the page cache.  The 
temporary file is not opened and the information is not truly
written to disk until the page cache is full.



This means that for many common cases where the temporary tables
and indices are small (small enough to fit into the page cache)
no temporary files are created and no disk I/O occurs.  Only
when the temporary data becomes too large to fit in RAM does
the information spill to disk.



Each temporary table and index is given its own page cache
which can store a maximum number of database pages determined
by the SQLITE_DEFAULT_TEMP_CACHE_SIZE compile-time parameter.
(The default value is 500 pages.)
The maximum number of database pages in the page cache is the
same for every temporary table and index.  The value cannot
be changed at run-time or on a per-table or per-index basis.
Each temporary file gets its own private page cache with its
own SQLITE_DEFAULT_TEMP_CACHE_SIZE page limit.


c2: Temporary Files Used By SQLite
c3: 5. Temporary File Storage Locations
c5: tempfiles.html#temporary_file_storage_locations
c4:
The directory or folder in which temporary files are created is
determined by the OS-specific VFS.


On unix-like systems, directories are searched in the following order:

The directory set by PRAGMA temp_store_directory or by the
    sqlite3_temp_directory global variable
The SQLITE_TMPDIR environment variable
The TMPDIR environment variable
/var/tmp
/usr/tmp
/tmp
The current working directory (".")

The first of the above that is found to exist and have the write and
execute bits set is used.  The final "." fallback is important for some
applications that use SQLite inside of chroot jails that do not have
the standard temporary file locations available.


On Windows systems, folders are searched in the following order:

The folder set by PRAGMA temp_store_directory or by the
    sqlite3_temp_directory global variable
The folder returned by the GetTempPath() system interface.

SQLite itself does not pay any attention to environment variables
in this case, though presumably the GetTempPath() system call does.
The search algorithm is different for CYGWIN builds.  Check the 
source code for details.


c2: Result and Error Codes
c3: Overview
c5: rescode.html#overview
c4:
Many of the routines in the SQLite C-language Interface return
numeric result codes indicating either success or failure, and 
in the event of a failure, providing some idea of the cause of
the failure.  This document strives to explain what each
of those numeric result codes means.


c2: Result and Error Codes
c3: 1. Result Codes versus Error Codes
c5: rescode.html#result_codes_versus_error_codes
c4:
"Error codes" are a subset of "result codes" that indicate that
something has gone wrong.  There are only a few non-error result
codes:  SQLITE_OK, SQLITE_ROW, and SQLITE_DONE.  The term
"error code" means any result code other than these three.


c2: Result and Error Codes
c3: 2. Primary Result Codes versus Extended Result Codes
c5: rescode.html#primary_result_codes_versus_extended_result_codes
c4:
Result codes are signed 32-bit integers.
The least significant 8 bits of the result code define a broad category
and are called the "primary result code".  More significant bits provide
more detailed information about the error and are called the
"extended result code"


Note that the primary result code is always a part of the extended
result code.  Given a full 32-bit extended result code, the application
can always find the corresponding primary result code merely by extracting
the least significant 8 bits of the extended result code.


All extended result codes are also error codes.  Hence the terms
"extended result code" and "extended error code" are interchangeable.


For historic compatibility, the C-language interfaces return
primary result codes by default.  
The extended result code for the most recent error can be
retrieved using the sqlite3_extended_errcode() interface.
The sqlite3_extended_result_codes() interface can be used to put
a database connection into a mode where it returns the
extended result codes instead of the primary result codes.


c2: Result and Error Codes
c3: 3. Definitions
c5: rescode.html#definitions
c4:
All result codes are integers.
Symbolic names for all result codes are created using
"#define" macros in the sqlite3.h header file.
There are separate sections in the sqlite3.h header file for
the result code definitions and the extended result code definitions.


Primary result code symbolic names are of the form "SQLITE_XXXXXX" where
XXXXXX is a sequence of uppercase alphabetic characters.  Extended
result code names are of the form "SQLITE_XXXXXX_YYYYYYY" where
the XXXXXX part is the corresponding primary result code and the
YYYYYYY is an extension that further classifies the result code.


The names and numeric values for existing result codes are fixed
and unchanging.  However, new result codes, and especially new extended
result codes, might appear in future releases of SQLite.


c2: Result and Error Codes
c3: 4. Primary Result Code List
c5: rescode.html#primary_result_code_list
c4:
The 31 result codes are 
   defined in sqlite3.h and are listed in
   alphabetical order below: 


SQLITE_ABORT&nbsp;(4)
SQLITE_AUTH&nbsp;(23)
SQLITE_BUSY&nbsp;(5)
SQLITE_CANTOPEN&nbsp;(14)
SQLITE_CONSTRAINT&nbsp;(19)
SQLITE_CORRUPT&nbsp;(11)
SQLITE_DONE&nbsp;(101)
SQLITE_EMPTY&nbsp;(16)
SQLITE_ERROR&nbsp;(1)
SQLITE_FORMAT&nbsp;(24)
SQLITE_FULL&nbsp;(13)
SQLITE_INTERNAL&nbsp;(2)
SQLITE_INTERRUPT&nbsp;(9)
SQLITE_IOERR&nbsp;(10)
SQLITE_LOCKED&nbsp;(6)
SQLITE_MISMATCH&nbsp;(20)
SQLITE_MISUSE&nbsp;(21)
SQLITE_NOLFS&nbsp;(22)
SQLITE_NOMEM&nbsp;(7)
SQLITE_NOTADB&nbsp;(26)
SQLITE_NOTFOUND&nbsp;(12)
SQLITE_NOTICE&nbsp;(27)
SQLITE_OK&nbsp;(0)
SQLITE_PERM&nbsp;(3)
SQLITE_PROTOCOL&nbsp;(15)
SQLITE_RANGE&nbsp;(25)
SQLITE_READONLY&nbsp;(8)
SQLITE_ROW&nbsp;(100)
SQLITE_SCHEMA&nbsp;(17)
SQLITE_TOOBIG&nbsp;(18)
SQLITE_WARNING&nbsp;(28)


c2: Result and Error Codes
c3: 5. Extended Result Code List
c5: rescode.html#extended_result_code_list
c4:
The 74 extended result codes
  are defined in sqlite3.h and are
  listed in alphabetical order below:


SQLITE_ABORT_ROLLBACK&nbsp;(516)
SQLITE_AUTH_USER&nbsp;(279)
SQLITE_BUSY_RECOVERY&nbsp;(261)
SQLITE_BUSY_SNAPSHOT&nbsp;(517)
SQLITE_BUSY_TIMEOUT&nbsp;(773)
SQLITE_CANTOPEN_CONVPATH&nbsp;(1038)
SQLITE_CANTOPEN_DIRTYWAL&nbsp;(1294)
SQLITE_CANTOPEN_FULLPATH&nbsp;(782)
SQLITE_CANTOPEN_ISDIR&nbsp;(526)
SQLITE_CANTOPEN_NOTEMPDIR&nbsp;(270)
SQLITE_CANTOPEN_SYMLINK&nbsp;(1550)
SQLITE_CONSTRAINT_CHECK&nbsp;(275)
SQLITE_CONSTRAINT_COMMITHOOK&nbsp;(531)
SQLITE_CONSTRAINT_DATATYPE&nbsp;(3091)
SQLITE_CONSTRAINT_FOREIGNKEY&nbsp;(787)
SQLITE_CONSTRAINT_FUNCTION&nbsp;(1043)
SQLITE_CONSTRAINT_NOTNULL&nbsp;(1299)
SQLITE_CONSTRAINT_PINNED&nbsp;(2835)
SQLITE_CONSTRAINT_PRIMARYKEY&nbsp;(1555)
SQLITE_CONSTRAINT_ROWID&nbsp;(2579)
SQLITE_CONSTRAINT_TRIGGER&nbsp;(1811)
SQLITE_CONSTRAINT_UNIQUE&nbsp;(2067)
SQLITE_CONSTRAINT_VTAB&nbsp;(2323)
SQLITE_CORRUPT_INDEX&nbsp;(779)
SQLITE_CORRUPT_SEQUENCE&nbsp;(523)
SQLITE_CORRUPT_VTAB&nbsp;(267)
SQLITE_ERROR_MISSING_COLLSEQ&nbsp;(257)
SQLITE_ERROR_RETRY&nbsp;(513)
SQLITE_ERROR_SNAPSHOT&nbsp;(769)
SQLITE_IOERR_ACCESS&nbsp;(3338)
SQLITE_IOERR_AUTH&nbsp;(7178)
SQLITE_IOERR_BEGIN_ATOMIC&nbsp;(7434)
SQLITE_IOERR_BLOCKED&nbsp;(2826)
SQLITE_IOERR_CHECKRESERVEDLOCK&nbsp;(3594)
SQLITE_IOERR_CLOSE&nbsp;(4106)
SQLITE_IOERR_COMMIT_ATOMIC&nbsp;(7690)
SQLITE_IOERR_CONVPATH&nbsp;(6666)
SQLITE_IOERR_CORRUPTFS&nbsp;(8458)
SQLITE_IOERR_DATA&nbsp;(8202)
SQLITE_IOERR_DELETE&nbsp;(2570)
SQLITE_IOERR_DELETE_NOENT&nbsp;(5898)
SQLITE_IOERR_DIR_CLOSE&nbsp;(4362)
SQLITE_IOERR_DIR_FSYNC&nbsp;(1290)
SQLITE_IOERR_FSTAT&nbsp;(1802)
SQLITE_IOERR_FSYNC&nbsp;(1034)
SQLITE_IOERR_GETTEMPPATH&nbsp;(6410)
SQLITE_IOERR_LOCK&nbsp;(3850)
SQLITE_IOERR_MMAP&nbsp;(6154)
SQLITE_IOERR_NOMEM&nbsp;(3082)
SQLITE_IOERR_RDLOCK&nbsp;(2314)
SQLITE_IOERR_READ&nbsp;(266)
SQLITE_IOERR_ROLLBACK_ATOMIC&nbsp;(7946)
SQLITE_IOERR_SEEK&nbsp;(5642)
SQLITE_IOERR_SHMLOCK&nbsp;(5130)
SQLITE_IOERR_SHMMAP&nbsp;(5386)
SQLITE_IOERR_SHMOPEN&nbsp;(4618)
SQLITE_IOERR_SHMSIZE&nbsp;(4874)
SQLITE_IOERR_SHORT_READ&nbsp;(522)
SQLITE_IOERR_TRUNCATE&nbsp;(1546)
SQLITE_IOERR_UNLOCK&nbsp;(2058)
SQLITE_IOERR_VNODE&nbsp;(6922)
SQLITE_IOERR_WRITE&nbsp;(778)
SQLITE_LOCKED_SHAREDCACHE&nbsp;(262)
SQLITE_LOCKED_VTAB&nbsp;(518)
SQLITE_NOTICE_RECOVER_ROLLBACK&nbsp;(539)
SQLITE_NOTICE_RECOVER_WAL&nbsp;(283)
SQLITE_OK_LOAD_PERMANENTLY&nbsp;(256)
SQLITE_READONLY_CANTINIT&nbsp;(1288)
SQLITE_READONLY_CANTLOCK&nbsp;(520)
SQLITE_READONLY_DBMOVED&nbsp;(1032)
SQLITE_READONLY_DIRECTORY&nbsp;(1544)
SQLITE_READONLY_RECOVERY&nbsp;(264)
SQLITE_READONLY_ROLLBACK&nbsp;(776)
SQLITE_WARNING_AUTOINDEX&nbsp;(284)


c2: Result and Error Codes
c3: 6. Result Code Meanings
c5: rescode.html#result_code_meanings
c4:
The meanings for all 105
result code values are shown below,
in numeric order.



(1) SQLITE_ERROR

  The SQLITE_ERROR result code is a generic error code that is used when
  no other more specific error code is available.




(3) SQLITE_PERM

  The SQLITE_PERM result code indicates that the requested access mode
  for a newly created database could not be provided.




(5) SQLITE_BUSY

  The SQLITE_BUSY result code indicates that the database file could not
  be written (or in some cases read) because of concurrent activity by 
  some other database connection, usually a database connection in a
  separate process.
  
  For example, if process A is in the middle of a large write transaction
  and at the same time process B attempts to start a new write transaction,
  process B will get back an SQLITE_BUSY result because SQLite only supports
  one writer at a time.  Process B will need to wait for process A to finish
  its transaction before starting a new transaction.  The
  sqlite3_busy_timeout() and sqlite3_busy_handler() interfaces and
  the busy_timeout pragma are available to process B to help it deal
  with SQLITE_BUSY errors.
  
  An SQLITE_BUSY error can occur at any point in a transaction: when the
  transaction is first started, during any write or update operations, or
  when the transaction commits.
  To avoid encountering SQLITE_BUSY errors in the middle of a transaction,
  the application can use BEGIN IMMEDIATE instead of just BEGIN to
  start a transaction.  The BEGIN IMMEDIATE command might itself return
  SQLITE_BUSY, but if it succeeds, then SQLite guarantees that no 
  subsequent operations on the same database through the next COMMIT 
  will return SQLITE_BUSY.
  
  See also:  SQLITE_BUSY_RECOVERY and SQLITE_BUSY_SNAPSHOT.
  
  The SQLITE_BUSY result code differs from SQLITE_LOCKED in that
  SQLITE_BUSY indicates a conflict with a
  separate database connection, probably in a separate process,
  whereas SQLITE_LOCKED 
  indicates a conflict within the same database connection (or sometimes
  a database connection with a shared cache).




(7) SQLITE_NOMEM

  The SQLITE_NOMEM result code indicates that SQLite was unable to allocate
  all the memory it needed to complete the operation.  In other words, an
  internal call to sqlite3_malloc() or sqlite3_realloc() has failed in
  a case where the memory being allocated was required in order to continue
  the operation.




(9) SQLITE_INTERRUPT

  The SQLITE_INTERRUPT result code indicates that an operation was
  interrupted by the sqlite3_interrupt() interface.
  See also: SQLITE_ABORT




(11) SQLITE_CORRUPT

  The SQLITE_CORRUPT result code indicates that the database file has
  been corrupted.  See the How To Corrupt Your Database Files for
  further discussion on how corruption can occur.




(13) SQLITE_FULL

  The SQLITE_FULL result code indicates that a write could not complete
  because the disk is full.  Note that this error can occur when trying
  to write information into the main database file, or it can also
  occur when writing into temporary disk files.
  
  Sometimes applications encounter this error even though there is an
  abundance of primary disk space because the error occurs when writing
  into temporary disk files on a system where temporary files are stored
  on a separate partition with much less space that the primary disk.




(15) SQLITE_PROTOCOL

  The SQLITE_PROTOCOL result code indicates a problem with the file locking
  protocol used by SQLite.  The SQLITE_PROTOCOL error is currently only
  returned when using WAL mode and attempting to start a new transaction.
  There is a race condition that can occur when two separate 
  database connections both try to start a transaction at the same time
  in WAL mode.  The loser of the race backs off and tries again, after
  a brief delay.  If the same connection loses the locking race dozens
  of times over a span of multiple seconds, it will eventually give up and
  return SQLITE_PROTOCOL.  The SQLITE_PROTOCOL error should appear in practice
  very, very rarely, and only when there are many separate processes all
  competing intensely to write to the same database.




(17) SQLITE_SCHEMA

  The SQLITE_SCHEMA result code indicates that the database schema
  has changed.  This result code can be returned from sqlite3_step() for
  a prepared statement that was generated using sqlite3_prepare() or
  sqlite3_prepare16().  If the database schema was changed by some other
  process in between the time that the statement was prepared and the time
  the statement was run, this error can result.
  
  If a prepared statement is generated from sqlite3_prepare_v2() then
  the statement is automatically re-prepared if the schema changes, up to
  SQLITE_MAX_SCHEMA_RETRY times (default: 50).  The sqlite3_step()
  interface will only return SQLITE_SCHEMA back to the application if 
  the failure persists after these many retries.




(19) SQLITE_CONSTRAINT

  The SQLITE_CONSTRAINT error code means that an SQL constraint violation
  occurred while trying to process an SQL statement.  Additional information
  about the failed constraint can be found by consulting the
  accompanying error message (returned via sqlite3_errmsg() or
  sqlite3_errmsg16()) or by looking at the extended error code.
  
  The SQLITE_CONSTRAINT code can also be used as the return value from
  the xBestIndex() method of a virtual table implementation.  When
  xBestIndex() returns SQLITE_CONSTRAINT, that indicates that the particular
  combination of inputs submitted to xBestIndex() cannot result in a
  usable query plan and should not be given further consideration.




(21) SQLITE_MISUSE

  The SQLITE_MISUSE return code might be returned if the application uses
  any SQLite interface in a way that is undefined or unsupported.  For
  example, using a prepared statement after that prepared statement has
  been finalized might result in an SQLITE_MISUSE error.
  
  SQLite tries to detect misuse and report the misuse using this result code.
  However, there is no guarantee that the detection of misuse will be
  successful.  Misuse detection is probabilistic.  Applications should
  never depend on an SQLITE_MISUSE return value.
  
  If SQLite ever returns SQLITE_MISUSE from any interface, that means that
  the application is incorrectly coded and needs to be fixed.  Do not ship
  an application that sometimes returns SQLITE_MISUSE from a standard
  SQLite interface because that application contains potentially serious bugs.




(23) SQLITE_AUTH

  The SQLITE_AUTH error is returned when the
  authorizer callback indicates that an
  SQL statement being prepared is not authorized.




(25) SQLITE_RANGE

  The SQLITE_RANGE error indices that the parameter number argument
  to one of the sqlite3_bind routines or the
  column number in one of the sqlite3_column
  routines is out of range.




(27) SQLITE_NOTICE

  The SQLITE_NOTICE result code is not returned by any C/C++ interface.
  However, SQLITE_NOTICE (or rather one of its extended error codes)
  is sometimes used as the first argument in an sqlite3_log() callback
  to indicate that an unusual operation is taking place.




(100) SQLITE_ROW

  The SQLITE_ROW result code returned by
  sqlite3_step() indicates that another row of output is available.




(256) SQLITE_OK_LOAD_PERMANENTLY

  The sqlite3_load_extension() interface loads an 
  extension into a single
  database connection.  The default behavior is for that extension to be
  automatically unloaded when the database connection closes.  However,
  if the extension entry point returns SQLITE_OK_LOAD_PERMANENTLY instead
  of SQLITE_OK, then the extension remains loaded into the process address
  space after the database connection closes.  In other words, the
  xDlClose methods of the sqlite3_vfs object is not called for the
  extension when the database connection closes.
  
  The SQLITE_OK_LOAD_PERMANENTLY return code is useful to 
  loadable extensions that register new VFSes, for example.




(261) SQLITE_BUSY_RECOVERY

  The SQLITE_BUSY_RECOVERY error code is an extended error code
  for SQLITE_BUSY that indicates that an operation could not continue
  because another process is busy recovering a WAL mode database file
  following a crash.  The SQLITE_BUSY_RECOVERY error code only occurs
  on WAL mode databases.




(264) SQLITE_READONLY_RECOVERY

  The SQLITE_READONLY_RECOVERY error code is an extended error code
  for SQLITE_READONLY.  The SQLITE_READONLY_RECOVERY error code indicates
  that a WAL mode database cannot be opened because the database file
  needs to be recovered and recovery requires write access but only
  read access is available.




(267) SQLITE_CORRUPT_VTAB

  The SQLITE_CORRUPT_VTAB error code is an extended error code
  for SQLITE_CORRUPT used by virtual tables.  A virtual table might
  return SQLITE_CORRUPT_VTAB to indicate that content in the virtual table
  is corrupt.




(275) SQLITE_CONSTRAINT_CHECK

  The SQLITE_CONSTRAINT_CHECK error code is an extended error code
  for SQLITE_CONSTRAINT indicating that a CHECK constraint failed.




(283) SQLITE_NOTICE_RECOVER_WAL

  The SQLITE_NOTICE_RECOVER_WAL result code is
  passed to the callback of
  sqlite3_log() when a WAL mode database file is recovered.




(513) SQLITE_ERROR_RETRY

  The SQLITE_ERROR_RETRY is used internally to provoke sqlite3_prepare_v2()
  (or one of its sibling routines for creating prepared statements) to
  try again to prepare a statement that failed with an error on the
  previous attempt.




(517) SQLITE_BUSY_SNAPSHOT

  The SQLITE_BUSY_SNAPSHOT error code is an extended error code
  for SQLITE_BUSY that occurs on WAL mode databases when a database
  connection tries to promote a read transaction into a write transaction
  but finds that another database connection has already written to the
  database and thus invalidated prior reads.
  
  The following scenario illustrates how an SQLITE_BUSY_SNAPSHOT error
  might arise:
  
   Process A starts a read transaction on the database and does one
       or more SELECT statement.  Process A keeps the transaction open.
   Process B updates the database, changing values previous read by
       process A.
   Process A now tries to write to the database.  But process A's view
       of the database content is now obsolete because process B has
       modified the database file after process A read from it.  Hence
       process A gets an SQLITE_BUSY_SNAPSHOT error.
  




(520) SQLITE_READONLY_CANTLOCK

  The SQLITE_READONLY_CANTLOCK error code is an extended error code
  for SQLITE_READONLY.  The SQLITE_READONLY_CANTLOCK error code indicates
  that SQLite is unable to obtain a read lock on a WAL mode database
  because the shared-memory file associated with that database is read-only.




(523) SQLITE_CORRUPT_SEQUENCE

  The SQLITE_CORRUPT_SEQUENCE result code means that the schema of
  the sqlite_sequence table is corrupt.  The sqlite_sequence table
  is used to help implement the AUTOINCREMENT feature.  The
  sqlite_sequence table should have the following format:
  
  CREATE TABLE sqlite_sequence(name,seq);
  
  If SQLite discovers that the sqlite_sequence table has any other
  format, it returns the SQLITE_CORRUPT_SEQUENCE error.




(531) SQLITE_CONSTRAINT_COMMITHOOK

  The SQLITE_CONSTRAINT_COMMITHOOK error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that a
  commit hook callback returned non-zero that thus
  caused the SQL statement to be rolled back.




(769) SQLITE_ERROR_SNAPSHOT

  The SQLITE_ERROR_SNAPSHOT result code might be returned when attempting
  to start a read transaction on an historical version of the database
  by using the sqlite3_snapshot_open() interface.  If the historical
  snapshot is no longer available, then the read transaction will fail
  with the SQLITE_ERROR_SNAPSHOT.  This error code is only possible if
  SQLite is compiled with -DSQLITE_ENABLE_SNAPSHOT.




(776) SQLITE_READONLY_ROLLBACK

  The SQLITE_READONLY_ROLLBACK error code is an extended error code
  for SQLITE_READONLY.  The SQLITE_READONLY_ROLLBACK error code indicates
  that a database cannot be opened because it has a hot journal that
  needs to be rolled back but cannot because the database is readonly.




(779) SQLITE_CORRUPT_INDEX

  The SQLITE_CORRUPT_INDEX result code means that SQLite detected
  an entry is or was missing from an index.  This is a special case of
  the SQLITE_CORRUPT error code that suggests that the problem might
  be resolved by running the REINDEX command, assuming no other
  problems exist elsewhere in the database file.




(787) SQLITE_CONSTRAINT_FOREIGNKEY

  The SQLITE_CONSTRAINT_FOREIGNKEY error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that a foreign key constraint failed.




(1034) SQLITE_IOERR_FSYNC

  The SQLITE_IOERR_FSYNC error code is an extended error code
  for SQLITE_IOERR indicating an I/O error in the VFS layer
  while trying to flush previously written content out of OS and/or
  disk-control buffers and into persistent storage.  In other words,
  this code indicates a problem with the fsync() system call in unix
  or the FlushFileBuffers() system call in windows.




(1043) SQLITE_CONSTRAINT_FUNCTION

  The SQLITE_CONSTRAINT_FUNCTION error code is not currently used
  by the SQLite core.  However, this error code is available for use
  by extension functions.




(1290) SQLITE_IOERR_DIR_FSYNC

  The SQLITE_IOERR_DIR_FSYNC error code is an extended error code
  for SQLITE_IOERR indicating an I/O error in the VFS layer
  while trying to invoke fsync() on a directory.  The unix VFS attempts
  to fsync() directories after creating or deleting certain files to
  ensure that those files will still appear in the filesystem following
  a power loss or system crash.  This error code indicates a problem
  attempting to perform that fsync().




(1299) SQLITE_CONSTRAINT_NOTNULL

  The SQLITE_CONSTRAINT_NOTNULL error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that a NOT NULL constraint failed.




(1546) SQLITE_IOERR_TRUNCATE

  The SQLITE_IOERR_TRUNCATE error code is an extended error code
  for SQLITE_IOERR indicating an I/O error in the VFS layer
  while trying to truncate a file to a smaller size.




(1555) SQLITE_CONSTRAINT_PRIMARYKEY

  The SQLITE_CONSTRAINT_PRIMARYKEY error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that a PRIMARY KEY constraint failed.




(1811) SQLITE_CONSTRAINT_TRIGGER

  The SQLITE_CONSTRAINT_TRIGGER error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that a RAISE function within
  a trigger fired, causing the SQL statement to abort.




(2067) SQLITE_CONSTRAINT_UNIQUE

  The SQLITE_CONSTRAINT_UNIQUE error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that a UNIQUE constraint failed.




(2323) SQLITE_CONSTRAINT_VTAB

  The SQLITE_CONSTRAINT_VTAB error code is not currently used
  by the SQLite core.  However, this error code is available for use
  by application-defined virtual tables.




(2579) SQLITE_CONSTRAINT_ROWID

  The SQLITE_CONSTRAINT_ROWID error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that a rowid is not unique.




(2835) SQLITE_CONSTRAINT_PINNED

  The SQLITE_CONSTRAINT_PINNED error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that an UPDATE trigger attempted
  do delete the row that was being updated in the middle of the update.




(3091) SQLITE_CONSTRAINT_DATATYPE

  The SQLITE_CONSTRAINT_DATATYPE error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that an insert or update attempted
  to store a value inconsistent with the column's declared type
  in a table defined as STRICT.




(3594) SQLITE_IOERR_CHECKRESERVEDLOCK

  The SQLITE_IOERR_CHECKRESERVEDLOCK error code is
  an extended error code
  for SQLITE_IOERR indicating an I/O error
  within the xCheckReservedLock method on the sqlite3_io_methods object.




(4106) SQLITE_IOERR_CLOSE

  The SQLITE_IOERR_ACCESS error code is an extended error code
  for SQLITE_IOERR indicating an I/O error
  within the xClose method on the sqlite3_io_methods object.




(4618) SQLITE_IOERR_SHMOPEN

  The SQLITE_IOERR_SHMOPEN error code is an extended error code
  for SQLITE_IOERR indicating an I/O error
  within the xShmMap method on the sqlite3_io_methods object
  while trying to open a new shared memory segment.




(5130) SQLITE_IOERR_SHMLOCK

  The SQLITE_IOERR_SHMLOCK error code is no longer used.




(5642) SQLITE_IOERR_SEEK

  The SQLITE_IOERR_SEEK error code is an extended error code
  for SQLITE_IOERR indicating an I/O error
  within the xRead or xWrite methods on the sqlite3_io_methods object
  while trying to seek a file descriptor to the beginning point of the
  file where the read or write is to occur.




(6154) SQLITE_IOERR_MMAP

  The SQLITE_IOERR_MMAP error code is an extended error code
  for SQLITE_IOERR indicating an I/O error
  within the xFetch or xUnfetch methods on the sqlite3_io_methods object
  while trying to map or unmap part of the database file into the
  process address space.




(6666) SQLITE_IOERR_CONVPATH

  The SQLITE_IOERR_CONVPATH error code is an extended error code
  for SQLITE_IOERR used only by Cygwin VFS and indicating that
  the cygwin_conv_path() system call failed.
  See also: SQLITE_CANTOPEN_CONVPATH




(7178) SQLITE_IOERR_AUTH

  The SQLITE_IOERR_AUTH error code is a code reserved for use
  by extensions.  It is not used by the SQLite core.




(7690) SQLITE_IOERR_COMMIT_ATOMIC

  The SQLITE_IOERR_COMMIT_ATOMIC error code indicates that the
  underlying operating system reported and error on the
  SQLITE_FCNTL_COMMIT_ATOMIC_WRITE file-control.  This only comes
  up when SQLITE_ENABLE_ATOMIC_WRITE is enabled and the database
  is hosted on a filesystem that supports atomic writes.




(8202) SQLITE_IOERR_DATA

  The SQLITE_IOERR_DATA error code is an extended error code
  for SQLITE_IOERR used only by checksum VFS shim to indicate that
  the checksum on a page of the database file is incorrect.


c2: Generated Columns
c3: 1. Introduction
c5: gencol.html#introduction
c4:
Generated columns (also sometimes called "computed columns")
are columns of a table whose values are a function of other columns
in the same row.
Generated columns can be read, but their values can not be directly
written.  The only way to change the value of a generated column is to
modify the values of the other columns used to calculate
the generated column.


c2: Generated Columns
c3: 2. Syntax
c5: gencol.html#syntax
c4:
Syntactically, generated columns are designated using a
"GENERATED ALWAYS" column-constraint.  For example:

CREATE TABLE t1(
   a INTEGER PRIMARY KEY,
   b INT,
   c TEXT,
   d INT GENERATED ALWAYS AS (a*abs(b)) VIRTUAL,
   e TEXT GENERATED ALWAYS AS (substr(c,b,b+1)) STORED
);


The statement above has three ordinary columns, "a" (the PRIMARY KEY),
"b", and "c", and two generated columns "d" and "e".

The "GENERATED ALWAYS" keywords at the beginning of the constraint
and the "VIRTUAL" or "STORED" keyword at the end are all optional.
Only the "AS" keyword and the parenthesized expression are required.
If the trailing "VIRTUAL" or "STORED" keyword is omitted, then
VIRTUAL is the default.  Hence, the example statement above could
be simplified to just:

CREATE TABLE t1(
   a INTEGER PRIMARY KEY,
   b INT,
   c TEXT,
   d INT AS (a*abs(b)),
   e TEXT AS (substr(c,b,b+1)) STORED
);


c2: Generated Columns
c3: 2.1. VIRTUAL versus STORED columns
c5: gencol.html#virtual_versus_stored_columns
c4:
Generated columns can be either VIRTUAL or STORED.  The value of
a VIRTUAL column is computed when read, whereas the value of a STORED
column is computed when the row is written.  STORED columns take up space
in the database file, whereas VIRTUAL columns use more CPU cycles when
being read.

From the point of view of SQL, STORED and VIRTUAL columns are almost
exactly the same.  Queries against either class of generated column
produce the same results.  The only functional difference is that
one cannot add new STORED columns using the
ALTER TABLE ADD COLUMN command.  Only VIRTUAL columns can be added
using ALTER TABLE.


c2: Generated Columns
c3: 2.2. Capabilities
c5: gencol.html#capabilities
c4:
Generated columns can have a datatype.  SQLite attempts to transform
the result of the generating expression into that datatype using the
same affinity rules as for ordinary columns.


Generated columns may have NOT NULL, CHECK, and UNIQUE constraints,
and foreign key constraints, just like ordinary columns.


Generated columns can participate in indexes, just like ordinary
columns.


The expression of a generated column can refer to any of the
other declared columns in the table, including other generated columns,
as long as the expression does not directly or indirectly refer back
to itself.


Generated columns can occur anywhere in the table definition.  Generated
columns can be interspersed among ordinary columns.  It is not necessary
to put generated columns at the end of the list of columns in the
table definition, as is shown in the examples above.


c2: Generated Columns
c3: 2.3. Limitations
c5: gencol.html#limitations
c4:
Generated columns may not have a default value (they may not use the
"DEFAULT" clause).  The value of a generated column is always the value
specified by the expression that follows the "AS" keyword.


Generated columns may not be used as part of the PRIMARY KEY.
(Future versions of SQLite might relax this constraint for STORED columns.)


The expression of a generated column may only reference
constant literals and columns within the same row, and may only use
scalar deterministic functions.  The expression may not use subqueries,
aggregate functions, window functions, or table-valued functions.


The expression of a generated column may refer to other generated columns
in the same row, but no generated column can depend upon itself, either
directly or indirectly.  

The expression of a generated column may not directly reference
the ROWID, though it can reference the INTEGER PRIMARY KEY column,
which is often the same thing.


Every table must have at least one non-generated column.


It is not possible to ALTER TABLE ADD COLUMN a STORED column.
One can add a VIRTUAL column, however.


The datatype and collating sequence of the generated column are determined
only by the datatype and COLLATE clause on the column definition.
The datatype and collating sequence of the GENERATED ALWAYS AS expression
have no affect on the datatype and collating sequence of the column itself.


c2: Generated Columns
c3: 3. Compatibility
c5: gencol.html#compatibility
c4:
Generated column support was added with SQLite version 3.31.0
(2020-01-22).  If an earlier version of SQLite attempts to read
a database file that contains a generated column in its schema, then
that earlier version will perceive the generated column syntax as an
error and will report that the database schema is corrupt.

To clarify:  SQLite version 3.31.0 can read and write any database
created by any prior version of SQLite going back to 
SQLite 3.0.0 (2004-06-18).  And, earlier versions of SQLite,
prior to 3.31.0, can read and write databases created by SQLite
version 3.31.0 and later as long
as the database schema does not contain features, such as
generated columns, that are not understood by the earlier version.
Problems only arise if you create a new database that contains
generated columns, using SQLite version 3.31.0 or later, and then
try to read or write that database file using an earlier version of
SQLite that does not understand generated columns.


c2: Quality Management
c3: 1. Overview
c5: qmplan.html#overview
c4:
This is the Quality Management Plan for SQLite.


Quality management documents tend to expand into
binders full of incomprehensible jargon that nobody
reads.  This document strives to break that pattern by
being concise and useful.


The inspiration for this document is
DO-178B.
Among quality standards, DO-178B seems to have the highest usefulness
to paperwork ratio.  Even so, the amount of documentation needed
for a full-up DO-178B implementation is vast.  SQLite strives to be
nimble and low-ceremony, and to that end, much of the required
DO-178B documentation is omitted.  We retain only those parts that
genuinely improve quality for an open-source software project such
as SQLite.


The purpose of this document is to brief the reader on how the
SQLite development team functions on a daily basis, as they continuously
enhance the SQLite software and work to improve its already high reliability.
The document achieves its purpose if a competent developer can be
assimilated into the development team quickly after perusing this
document.


c2: Quality Management
c3: 1.1. About This Document
c5: qmplan.html#about_this_document
c4:
The quality management plan was originally composed by going through
the description of outputs in section 11 of DO-178B (pages 48 through 56)
and writing down those elements that seemed relevant to SQLite.
The text will be subsequent revised to track enhancements to the
SQLite quality process.


c2: Quality Management
c3: 2. Software Development Plan
c5: qmplan.html#software_development_plan
c4:
This section is a combination of the Plan For Software Aspects Of
Certification and the Software Development Plan sections of DO-178B.



See About SQLite for an overview of the
SQLite software and what it does and how it is different.


c2: Quality Management
c3: 2.1. Software Life Cycle
c5: qmplan.html#software_life_cycle
c4:
SQLite uses a continuous integration process.  The software
is under constant enhancement and refinement.  The latest trunk
check-ins are frequently used internally for mission-critical
operations. 


There is no pre-defined release cycle.  Releases occur
when there is a critical mass of feature enhancements and/or
bug fixes.  Historically, releases have occurred about 5 or 6
times per year.
Users of SQLite pick up new releases from the website on an
as-needed basis.


c2: Quality Management
c3: 2.1.1. Maintenance Releases
c5: qmplan.html#maintenance_releases
c4:
Routine maintenance releases of SQLite contain feature enhancements,
performance enhancements, and/or fixes for non-critical issues.
The version number for major releases are of the form "3.N.0"
for some integer N.  See the version numbering conventions document
for details.


Upcoming maintenance releases announced on the sqlite-users and
sqlite-dev mailing lists about two weeks prior to the anticipated
release.  Approximately one week prior to release, the lead developer
declares "pencils down" after which only bug-fix check-ins are
allowed on trunk.  A new 
release checklist
is created and updated as needed.  As items of the checklist are 
verified, they are checked off and turn green.  The release occurs
when all elements of the checklist are green.  That process normally
takes about a week.


c2: Quality Management
c3: 2.1.2. Patch Releases
c5: qmplan.html#patch_releases
c4:
Occasionally, a serious problem is found and a small "patch" release
must be made against a regular maintenance release.  Patches are distinct
from maintenance releases in that the number of lines of code changed
from the previous release is small.  Every effort is made to avoid
patch releases by making sure that maintenance releases are bug free.


Patch releases may or may not have a release checklist, depending on the
issue.  This is a judgement call by the project leader.


c2: Quality Management
c3: 2.2. Release History
c5: qmplan.html#release_history
c4:
The documentation system automatically maintains a
chronology of past releases, as well as a
complete list of SQLite releases with change summaries.


c2: Quality Management
c3: 2.3. Schedule
c5: qmplan.html#schedule
c4:
SQLite has a long-range vision.
Planning is done with the assumption that SQLite
will be used and supported through at least the year 2050.
All code is written with the idea that it will one day be read and
maintained by people not yet born.  The code is carefully commented
with an eye toward helping those future developers more easily 
understand the logic and the rationale behind the code.


c2: Quality Management
c3: 3. Software Development Environment
c5: qmplan.html#software_development_environment
c4:
SQLite is written in portable C code.
Development work occurs on a mix of Linux, Mac, and Windows workstations.
The developers use command-line tools and eschew integrated development
environments (IDEs) whenever possible.  All developers are expected to be
fluent with the unix command-line.


A minimum setup for compiling and testing SQLite from canonical
sources is as follows:


 A host computer with a 32-bit or 64-bit address space.
     The OS can be Linux, Mac, Windows, *BSD, Solaris, or some other.
 A C99 compiler such as GCC (including MinGW variants for Windows),
     Clang, or MSVC
 A text editor of the user's choice supporting UTF-8 text.
 Tcl version 8.6 or later.
 The "make" utility, or optionally "nmake" on Windows.



The Tcl script language is used to help translate canonical source code
into the amalgamation and to manage testing.  Tcl is not used directly
by SQLite itself (unless requested by a compile-time option).  End users
of the SQLite amalgamation sources do not need Tcl.


When building the CLI, it is helpful, but not required, to have
the following third-party libraries on hand:


 zLib
 readline
     or editline
     or linenoise for
     command-line editing.



A complete release-test of SQLite requires additional software,


 valgrind
 gcov



SQLite is expected to operate the same, and use exactly the same 
on-disk format,
on all modern operating systems, on all modern computer architectures,
and using all modern C compilers.  The developers are constantly testing
SQLite on as many diverse platforms as they can get their hands on.


c2: Quality Management
c3: 4. Software Verification Plan
c5: qmplan.html#software_verification_plan
c4:
The testing process for SQLite is described in the testing document.
Testing objectives include:


 100% MC/DC in an as-delivered configuration
 Testing of both source code and object code
 Testing on multiple platforms and with multiple compilers
 Fuzz testing
 Code change inspection
 Dynamic and static analysis of the code


The testing process is controlled by the
release testing checklists.  The checklists succinctly summarize
all steps necessary to fully validate SQLite, and they record when
and by whom each validation step was performed.

The set of checklist items for release checklist is potentially
updated for each release.  The content and complete
history of each release checklist are retained for the historical
record.


c2: Quality Management
c3: 5.1. Version Control
c5: qmplan.html#version_control
c4:
SQLite source code is managed using the Fossil
version control system.  Fossil was written specifically to support
SQLite development.  Fossil provides both distributed version control
and issue tracking.


c2: Quality Management
c3: 5.2. Survivability
c5: qmplan.html#survivability
c4:
All code is archived on three separate machines:
https://www.sqlite.org, https://www2.sqlite.org, https://www3.sqlite.org.
These machines are located in different cities (Dallas, Newark, and
San Francisco, respectively) and managed by two different hosting
companies (Linode for the first two and
Digital Ocean for the third).
This diversity is intended to avoid a single point of failure.


The main machine in Dallas https://www.sqlite.org/ is the primary
server and the one that most people use.  The other two are considered
backups.


In addition to the official repositories, the developers typically
keep complete clones of all software on their personal machines.
And there are other clones scattered about the internet.


c2: Quality Management
c3: 5.3. Repositories
c5: qmplan.html#repositories
c4:
The SQLite source is broken up into multiple repositories, each described
in a separate section below.


c2: Quality Management
c3: 5.3.1. SQLite Source Code
c5: qmplan.html#sqlite_source_code
c4:
The SQLite source code and the TCL test suite are stored together
in a single repository.  This one repository is all that is required to
build the SQLite.  The source repository is public and is
readable by anonymous passersby on the internet.


 Primary location: https://www.sqlite.org/src
 Backup A: https://www2.sqlite.org/src
 Backup B: https://www3.sqlite.org/cgi/src
 GitHub mirror: https://github.com/sqlite/sqlite/


c2: Quality Management
c3: 5.3.2. SQLite Documentation Sources
c5: qmplan.html#sqlite_documentation_sources
c4:
The documentation sources include documentation text and images with the
scripts and makefile needed to construct the SQLite website documentation.
This document is contained within the documentation sources.  The
document sources are kept in a separate repository distinct from the
source code.  The documentation sources repository is publicly readable.

The makefiles and scripts used to generate the documentation gather
text from baseline documents in the documentation source repository.
Additional text is extracted from comments in the SQLite source code.
Requirements coverage information is extracted from special comments in the
TCL test suite which is part of the source repository, and from
comments in the TH3 test suite which is in a separate private repository.


 Primary location: https://www.sqlite.org/docsrc
 Backup A: https://www2.sqlite.org/docsrc
 Backup B: https://www3.sqlite.org/cgi/docsrc


c2: Quality Management
c3: 5.3.3. SQL Logic Test
c5: qmplan.html#sql_logic_test
c4:
The SQL Logic Tests are a set of test cases designed to show that
SQLite behaves the same as other SQL database engines.  These tests
are hosted in a separate code public repository.


 Primary location: https://www.sqlite.org/sqllogictest
 Backups on private servers


c2: Quality Management
c3: 5.3.4. Test Harness #3
c5: qmplan.html#test_harness_3
c4:
The Test Harness #3 or TH3 test suite is a private set of
test cases used to test SQLite to 100% MC/DC in an as-delivered
configuration.  TH3 sources are served on the same servers as the
other SQLite repositories, but differ from the others in being
proprietary.  The TH3 code is only accessible to SQLite developers.



 Primary location: https://www.sqlite.org/th3
 Backup A: https://www3.sqlite.org/cgi/th3
 Additional backups on private servers


c2: Quality Management
c3: 5.3.5. Dbsqlfuzz
c5: qmplan.html#dbsqlfuzz
c4:
The dbsqlfuzz module is a 
libFuzzer-based fuzzer
for SQLite.  Dbsqlfuzz fuzzes both the SQL and the database file at
the same time.  Dbsqlfuzz uses a customized mutator.


Dbsqlfuzz seems to work better at finding problems than any other
fuzzer available.  For that reason, it is kept private.  We do not
want hacker gaining access to this technology.


 Primary location: https://www.sqlite.org/dbsqlfuzz
 Backup A: https://www3.sqlite.org/cgi/dbsqlfuzz
 Additional backups on private servers


c2: Quality Management
c3: 5.4. Software Verification Results
c5: qmplan.html#software_verification_results
c4:
Release testing proceeds by checklist.  The current status and
complete change history for each checklist is stored in a separate
SQLite database file.  These files are not version controlled, but
separate copies are maintained on private backup servers.

The source code to the software that runs the checklists is stored
in its own Fossil repository at https://www.sqlite.org/checklistapp.


c2: Quality Management
c3: 6. Software Requirements Standards And Data
c5: qmplan.html#software_requirements_standards_and_data
c4:
In the SQLite project, the "requirements" are the project documentation.
Special markup in the documentation text indentifies individual requirements.
The requirement numbers are based on a cryptographic hash of normalized
requirement text, so that it is impossible to change the requirement text
without also changing the requirement number.

Documentation text (and hence requirement text) is taken from the
SQLite Documentation source repository, described above, and also from
comments in the implementation.  The makefiles to build the documentation
are in the documentation source repository.

When the documentation is build, requirements are identified and labeled.
The documentation build process also scans for test cases that verify
each requirement and constructs a matrix showing which requirements have
been testing and identifying the specific test cases that test those
requirements.


c2: Quality Management
c3: 7. Software Design And Coding Standards
c5: qmplan.html#software_design_and_coding_standards
c4:
Objective coding standards for SQLite are minimal:


 2-space indentation
 No lines over 80 characters in length
 No tabs


All other design and coding rules are subjective.  The
goal here is to make the software so that it is readable
and maintainable through the year 2050.  To that end, we look
for succinct yet useful comments (no boilerplate), carefully
chosen variable names, and careful explanation of the meaning
of each data structure and the role of each code block.


c2: Quality Management
c3: 8. Problem Reports
c5: qmplan.html#problem_reports
c4:
All problems are fixed expeditiously.  There are no lingering problems
in the sQLite software.

The Fossil version control system utilized by
SQLite contains built-in support for tracking trouble-tickets.  This built-in
ticket system is used to track and document many historical problems.

The SQLite Community Forum is a place
where anybody on the internet can go to ask questions about or report bugs
against SQLite.  Bugs found by third-parties are often reported initially
on the Forum.  Forum-reported bugs will sometimes be transferred to tickets,
though recent practice as been to just deal with the bugs on the Forum.
The Forum has an excellent full-text search feature, is mirrored to
multiple machines, and is just as searchable and survivable as the ticket
system, so it seems unnecessary to duplicate Forum-originated bug reports
into the ticket system.  The public locations of the Forum are:


 Primary location: https://www.sqlite.org/forum
 Backup A: https://www2.sqlite.org/forum
 Backup B: https://www3.sqlite.org/cgi/forum



As with the source repositories, the Forum is also synced to various
private machines.
Note that because of the way Fossil works, the "backups" are more than just
read-only backups.  They can also function as data inputs.  All content
entered is synced to all repositories, regardless of which repository is
used for insertion.


c2: The sqlite3_analyzer.exe Utility Program
c3: 1. The sqlite3_analyzer.exe Utility Program
c5: sqlanalyze.html#the_sqlite3_analyzer_exe_utility_program
c4:
The sqlite3_analyzer.exe binary is a command-line utility program
that measures and displays how much and how efficiently space is used by 
individual tables and indexes with an SQLite database file.
Example usage:


sqlite3_analyzer database.sqlite



The output is a human-readable ASCII text report that provides information
on the space utilization of the database file.  The report is intended to
be self-explanatory, though there is some 
additional explanation of the
various parameters reported toward the end of the report.


The output is also valid SQL.  Most of the report text is contained within
a header comment, with various SQL statements that create and initialize
a database at the
end of the report.  The constructed database contains
the raw data from which the report was extracted.  Hence the original
report can be read into an instance of the command-line shell and then
the raw data can be queried to dig deeper into the space utilization of
a particular database file.


c2: The sqlite3_analyzer.exe Utility Program
c3: 1.1. Implementation
c5: sqlanalyze.html#implementation
c4:
The sqlite3_analyzer.exe program is a
TCL program that uses the dbstat virtual table
to gather information about the database file and then format that
information neatly.


c2: The sqlite3_analyzer.exe Utility Program
c3: 1.2. Example Output
c5: sqlanalyze.html#example_output
c4:
The following is sqlite3_analyzer output for an example
places.sqlite database used by Firefox.


/** Disk-Space Utilization Report For &#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;/places.sqlite

Page size in bytes................................ 32768     
Pages in the whole file (measured)................ 221       
Pages in the whole file (calculated).............. 221       
Pages that store data............................. 221        100.0% 
Pages on the freelist (per header)................ 0            0.0% 
Pages on the freelist (calculated)................ 0            0.0% 
Pages of auto-vacuum overhead..................... 0            0.0% 
Number of tables in the database.................. 14        
Number of indices................................. 23        
Number of defined indices......................... 17        
Number of implied indices......................... 6         
Size of the file in bytes......................... 7241728   
Bytes of user payload stored...................... 2503069     34.6% 

*** Page counts for all tables with their indices *****************************

MOZ_PLACES........................................ 142         64.3% 
MOZ_HISTORYVISITS................................. 41          18.6% 
MOZ_FAVICONS...................................... 15           6.8% 
MOZ_BOOKMARKS..................................... 5            2.3% 
MOZ_KEYWORDS...................................... 3            1.4% 
MOZ_ANNO_ATTRIBUTES............................... 2            0.90% 
MOZ_ANNOS......................................... 2            0.90% 
MOZ_BOOKMARKS_ROOTS............................... 2            0.90% 
MOZ_HOSTS......................................... 2            0.90% 
MOZ_INPUTHISTORY.................................. 2            0.90% 
MOZ_ITEMS_ANNOS................................... 2            0.90% 
SQLITE_SCHEMA..................................... 1            0.45% 
SQLITE_SEQUENCE................................... 1            0.45% 
SQLITE_STAT1...................................... 1            0.45% 

*** Page counts for all tables and indices separately *************************

MOZ_PLACES........................................ 63          28.5% 
MOZ_PLACES_URL_UNIQUEINDEX........................ 37          16.7% 
MOZ_HISTORYVISITS................................. 13           5.9% 
MOZ_FAVICONS...................................... 12           5.4% 
MOZ_HISTORYVISITS_PLACEDATEINDEX.................. 12           5.4% 
MOZ_PLACES_HOSTINDEX.............................. 11           5.0% 
MOZ_HISTORYVISITS_DATEINDEX....................... 10           4.5% 
MOZ_PLACES_GUID_UNIQUEINDEX....................... 9            4.1% 
MOZ_PLACES_LASTVISITDATEINDEX..................... 7            3.2% 
MOZ_HISTORYVISITS_FROMINDEX....................... 6            2.7% 
MOZ_PLACES_FAVICONINDEX........................... 5            2.3% 
MOZ_PLACES_FRECENCYINDEX.......................... 5            2.3% 
MOZ_PLACES_VISITCOUNT............................. 5            2.3% 
SQLITE_AUTOINDEX_MOZ_FAVICONS_1................... 3            1.4% 
MOZ_ANNO_ATTRIBUTES............................... 1            0.45% 
MOZ_ANNOS......................................... 1            0.45% 
MOZ_ANNOS_PLACEATTRIBUTEINDEX..................... 1            0.45% 
MOZ_BOOKMARKS..................................... 1            0.45% 
MOZ_BOOKMARKS_GUID_UNIQUEINDEX.................... 1            0.45% 
MOZ_BOOKMARKS_ITEMINDEX........................... 1            0.45% 
MOZ_BOOKMARKS_ITEMLASTMODIFIEDINDEX............... 1            0.45% 
MOZ_BOOKMARKS_PARENTINDEX......................... 1            0.45% 
MOZ_BOOKMARKS_ROOTS............................... 1            0.45% 
MOZ_HOSTS......................................... 1            0.45% 
MOZ_INPUTHISTORY.................................. 1            0.45% 
MOZ_ITEMS_ANNOS................................... 1            0.45% 
MOZ_ITEMS_ANNOS_ITEMATTRIBUTEINDEX................ 1            0.45% 
MOZ_KEYWORDS...................................... 1            0.45% 
MOZ_KEYWORDS_PLACEPOSTDATA_UNIQUEINDEX............ 1            0.45% 
SQLITE_AUTOINDEX_MOZ_ANNO_ATTRIBUTES_1............ 1            0.45% 
SQLITE_AUTOINDEX_MOZ_BOOKMARKS_ROOTS_1............ 1            0.45% 
SQLITE_AUTOINDEX_MOZ_HOSTS_1...................... 1            0.45% 
SQLITE_AUTOINDEX_MOZ_INPUTHISTORY_1............... 1            0.45% 
SQLITE_AUTOINDEX_MOZ_KEYWORDS_1................... 1            0.45% 
SQLITE_SCHEMA..................................... 1            0.45% 
SQLITE_SEQUENCE................................... 1            0.45% 
SQLITE_STAT1...................................... 1            0.45% 

*** All tables and indices ****************************************************

Percentage of total database...................... 100.0%    
Number of entries................................. 154969    
Bytes of storage consumed......................... 7241728   
Bytes of payload.................................. 4969404     68.6% 
Average payload per entry......................... 32.07     
Average unused bytes per entry.................... 11.15     
Average fanout.................................... 14.00     
Maximum payload per entry......................... 7640      
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 14        
Primary pages used................................ 207       
Overflow pages used............................... 0         
Total pages used.................................. 221       
Unused bytes on index pages....................... 448010      97.7% 
Unused bytes on primary pages..................... 1280642     18.9% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 1728652     23.9% 

*** All tables ****************************************************************

Percentage of total database......................  44.8%    
Number of entries................................. 28530     
Bytes of storage consumed......................... 3244032   
Bytes of payload.................................. 2508257     77.3% 
Average payload per entry......................... 87.92     
Average unused bytes per entry.................... 20.13     
Average fanout.................................... 28.00     
Maximum payload per entry......................... 7640      
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 3         
Primary pages used................................ 96        
Overflow pages used............................... 0         
Total pages used.................................. 99        
Unused bytes on index pages....................... 97551       99.23% 
Unused bytes on primary pages..................... 476741      15.2% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 574292      17.7% 

*** All indices ***************************************************************

Percentage of total database......................  55.2%    
Number of entries................................. 126439    
Bytes of storage consumed......................... 3997696   
Bytes of payload.................................. 2461147     61.6% 
Average payload per entry......................... 19.47     
Average unused bytes per entry.................... 9.13      
Average fanout.................................... 11.00     
Maximum payload per entry......................... 7259      
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 11        
Primary pages used................................ 111       
Overflow pages used............................... 0         
Total pages used.................................. 122       
Unused bytes on index pages....................... 350459      97.2% 
Unused bytes on primary pages..................... 803901      22.1% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 1154360     28.9% 

*** Table MOZ_ANNO_ATTRIBUTES and all its indices *****************************

Percentage of total database......................   0.90%   
Number of entries................................. 24        
Bytes of storage consumed......................... 65536     
Bytes of payload.................................. 721          1.1% 
Average payload per entry......................... 30.04     
Average unused bytes per entry.................... 2696.46   
Maximum payload per entry......................... 43        
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 2         
Overflow pages used............................... 0         
Total pages used.................................. 2         
Unused bytes on primary pages..................... 64715       98.7% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 64715       98.7% 

*** Table MOZ_ANNO_ATTRIBUTES w/o any indices *********************************

Percentage of total database......................   0.45%   
Number of entries................................. 12        
Bytes of storage consumed......................... 32768     
Bytes of payload.................................. 355          1.1% 
B-tree depth...................................... 1         
Average payload per entry......................... 29.58     
Average unused bytes per entry.................... 2696.42   
Maximum payload per entry......................... 42        
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 32357       98.7% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 32357       98.7% 

*** Index SQLITE_AUTOINDEX_MOZ_ANNO_ATTRIBUTES_1 of table MOZ_ANNO_ATTRIBUTES *

Percentage of total database......................   0.45%   
Number of entries................................. 12        
Bytes of storage consumed......................... 32768     
Bytes of payload.................................. 366          1.1% 
B-tree depth...................................... 1         
Average payload per entry......................... 30.50     
Average unused bytes per entry.................... 2696.50   
Maximum payload per entry......................... 43        
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 32358       98.7% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 32358       98.7% 

*** Table MOZ_ANNOS and all its indices ***************************************

Percentage of total database......................   0.90%   
Number of entries................................. 390       
Bytes of storage consumed......................... 65536     
Bytes of payload.................................. 13986       21.3% 
Average payload per entry......................... 35.86     
Average unused bytes per entry.................... 128.22    
Maximum payload per entry......................... 127       
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 2         
Overflow pages used............................... 0         
Total pages used.................................. 2         
Unused bytes on primary pages..................... 50006       76.3% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 50006       76.3% 

*** Table MOZ_ANNOS w/o any indices *******************************************

Percentage of total database......................   0.45%   
Number of entries................................. 195       
Bytes of storage consumed......................... 32768     
Bytes of payload.................................. 12115       37.0% 
B-tree depth...................................... 1         
Average payload per entry......................... 62.13     
Average unused bytes per entry.................... 101.04    
Maximum payload per entry......................... 127       
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 19702       60.1% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 19702       60.1% 

*** Index MOZ_ANNOS_PLACEATTRIBUTEINDEX of table MOZ_ANNOS ********************

Percentage of total database......................   0.45%   
Number of entries................................. 195       
Bytes of storage consumed......................... 32768     
Bytes of payload.................................. 1871         5.7% 
B-tree depth...................................... 1         
Average payload per entry......................... 9.59      
Average unused bytes per entry.................... 155.41    
Maximum payload per entry......................... 10        
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 30304       92.5% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 30304       92.5% 

*** Table MOZ_BOOKMARKS and all its indices ***********************************

Percentage of total database......................   2.3%    
Number of entries................................. 1565      
Bytes of storage consumed......................... 163840    
Bytes of payload.................................. 37104       22.6% 
Average payload per entry......................... 23.71     
Average unused bytes per entry.................... 77.62     
Maximum payload per entry......................... 518       
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 5         
Overflow pages used............................... 0         
Total pages used.................................. 5         
Unused bytes on primary pages..................... 121475      74.1% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 121475      74.1% 

*** Table MOZ_BOOKMARKS w/o any indices ***************************************

Percentage of total database......................   0.45%   
Number of entries................................. 313       
Bytes of storage consumed......................... 32768     
Bytes of payload.................................. 21937       66.9% 
B-tree depth...................................... 1         
Average payload per entry......................... 70.09     
Average unused bytes per entry.................... 29.90     
Maximum payload per entry......................... 518       
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 9358        28.6% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 9358        28.6% 

*** Indices of table MOZ_BOOKMARKS ********************************************

Percentage of total database......................   1.8%    
Number of entries................................. 1252      
Bytes of storage consumed......................... 131072    
Bytes of payload.................................. 15167       11.6% 
Average payload per entry......................... 12.11     
Average unused bytes per entry.................... 89.55     
Maximum payload per entry......................... 17        
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 4         
Overflow pages used............................... 0         
Total pages used.................................. 4         
Unused bytes on primary pages..................... 112117      85.5% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 112117      85.5% 

*** Index MOZ_BOOKMARKS_GUID_UNIQUEINDEX of table MOZ_BOOKMARKS ***************

Percentage of total database......................   0.45%   
Number of entries................................. 313       
Bytes of storage consumed......................... 32768     
Bytes of payload.................................. 5207        15.9% 
B-tree depth...................................... 1         
Average payload per entry......................... 16.64     
Average unused bytes per entry.................... 85.03     
Maximum payload per entry......................... 17        
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 26614       81.2% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 26614       81.2% 

*** Index MOZ_BOOKMARKS_ITEMINDEX of table MOZ_BOOKMARKS **********************

Percentage of total database......................   0.45%   
Number of entries................................. 313       
Bytes of storage consumed......................... 32768     
Bytes of payload.................................. 2547         7.8% 
B-tree depth...................................... 1         
Average payload per entry......................... 8.14      
Average unused bytes per entry.................... 93.53     
Maximum payload per entry......................... 9         
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 29274       89.3% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 29274       89.3% 

*** Index MOZ_BOOKMARKS_ITEMLASTMODIFIEDINDEX of table MOZ_BOOKMARKS **********

Percentage of total database......................   0.45%   
Number of entries................................. 313       
Bytes of storage consumed......................... 32768     
Bytes of payload.................................. 5020        15.3% 
B-tree depth...................................... 1         
Average payload per entry......................... 16.04     
Average unused bytes per entry.................... 85.63     
Maximum payload per entry......................... 17        
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 26801       81.8% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 26801       81.8% 

*** Index MOZ_BOOKMARKS_PARENTINDEX of table MOZ_BOOKMARKS ********************

Percentage of total database......................   0.45%   
Number of entries................................. 313       
Bytes of storage consumed......................... 32768     
Bytes of payload.................................. 2393         7.3% 
B-tree depth...................................... 1         
Average payload per entry......................... 7.65      
Average unused bytes per entry.................... 94.02     
Maximum payload per entry......................... 9         
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 29428       89.8% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 29428       89.8% 

*** Table MOZ_BOOKMARKS_ROOTS and all its indices *****************************

Percentage of total database......................   0.90%   
Number of entries................................. 10        
Bytes of storage consumed......................... 65536     
Bytes of payload.................................. 94           0.14% 
Average payload per entry......................... 9.40      
Average unused bytes per entry.................... 6539.10   
Maximum payload per entry......................... 11        
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 2         
Overflow pages used............................... 0         
Total pages used.................................. 2         
Unused bytes on primary pages..................... 65391       99.78% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 65391       99.78% 

*** Table MOZ_BOOKMARKS_ROOTS w/o any indices *********************************

Percentage of total database......................   0.45%   
Number of entries................................. 5         
Bytes of storage consumed......................... 32768     
Bytes of payload.................................. 47           0.14% 
B-tree depth...................................... 1         
Average payload per entry......................... 9.40      
Average unused bytes per entry.................... 6538.60   
Maximum payload per entry......................... 11        
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 32693       99.77% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 32693       99.77% 

*** Index SQLITE_AUTOINDEX_MOZ_BOOKMARKS_ROOTS_1 of table MOZ_BOOKMARKS_ROOTS *

Percentage of total database......................   0.45%   
Number of entries................................. 5         
Bytes of storage consumed......................... 32768     
Bytes of payload.................................. 47           0.14% 
B-tree depth...................................... 1         
Average payload per entry......................... 9.40      
Average unused bytes per entry.................... 6539.60   
Maximum payload per entry......................... 11        
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 32698       99.79% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 32698       99.79% 

*** Table MOZ_FAVICONS and all its indices ************************************

Percentage of total database......................   6.8%    
Number of entries................................. 941       
Bytes of storage consumed......................... 491520    
Bytes of payload.................................. 332765      67.7% 
Average payload per entry......................... 353.63    
Average unused bytes per entry.................... 164.00    
Average fanout.................................... 7.00      
Maximum payload per entry......................... 7640      
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 2         
Primary pages used................................ 13        
Overflow pages used............................... 0         
Total pages used.................................. 15        
Unused bytes on index pages....................... 65340       99.70% 
Unused bytes on primary pages..................... 88980       20.9% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 154320      31.4% 

*** Table MOZ_FAVICONS w/o any indices ****************************************

Percentage of total database......................   5.4%    
Number of entries................................. 471       
Bytes of storage consumed......................... 393216    
Bytes of payload.................................. 297630      75.7% 
B-tree depth...................................... 2         
Average payload per entry......................... 631.91    
Average unused bytes per entry.................... 196.60    
Average fanout.................................... 11.00     
Non-sequential pages.............................. 6           54.5% 
Maximum payload per entry......................... 7640      
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 1         
Primary pages used................................ 11        
Overflow pages used............................... 0         
Total pages used.................................. 12        
Unused bytes on index pages....................... 32676       99.72% 
Unused bytes on primary pages..................... 59923       16.6% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 92599       23.5% 

*** Index SQLITE_AUTOINDEX_MOZ_FAVICONS_1 of table MOZ_FAVICONS ***************

Percentage of total database......................   1.4%    
Number of entries................................. 470       
Bytes of storage consumed......................... 98304     
Bytes of payload.................................. 35135       35.7% 
B-tree depth...................................... 2         
Average payload per entry......................... 74.76     
Average unused bytes per entry.................... 131.32    
Average fanout.................................... 3.00      
Non-sequential pages.............................. 1           50.0% 
Maximum payload per entry......................... 7259      
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 1         
Primary pages used................................ 2         
Overflow pages used............................... 0         
Total pages used.................................. 3         
Unused bytes on index pages....................... 32664       99.68% 
Unused bytes on primary pages..................... 29057       44.3% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 61721       62.8% 

*** Table MOZ_HISTORYVISITS and all its indices *******************************

Percentage of total database......................  18.6%    
Number of entries................................. 63470     
Bytes of storage consumed......................... 1343488   
Bytes of payload.................................. 882233      65.7% 
Average payload per entry......................... 13.90     
Average unused bytes per entry.................... 3.76      
Average fanout.................................... 10.00     
Maximum payload per entry......................... 21        
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 4         
Primary pages used................................ 37        
Overflow pages used............................... 0         
Total pages used.................................. 41        
Unused bytes on index pages....................... 130482      99.55% 
Unused bytes on primary pages..................... 108158       8.9% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 238640      17.8% 

*** Table MOZ_HISTORYVISITS w/o any indices ***********************************

Percentage of total database......................   5.9%    
Number of entries................................. 15873     
Bytes of storage consumed......................... 425984    
Bytes of payload.................................. 308447      72.4% 
B-tree depth...................................... 2         
Average payload per entry......................... 19.43     
Average unused bytes per entry.................... 2.40      
Average fanout.................................... 12.00     
Non-sequential pages.............................. 8           66.7% 
Maximum payload per entry......................... 21        
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 1         
Primary pages used................................ 12        
Overflow pages used............................... 0         
Total pages used.................................. 13        
Unused bytes on index pages....................... 32668       99.69% 
Unused bytes on primary pages..................... 5435         1.4% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 38103        8.9% 

*** Indices of table MOZ_HISTORYVISITS ****************************************

Percentage of total database......................  12.7%    
Number of entries................................. 47597     
Bytes of storage consumed......................... 917504    
Bytes of payload.................................. 573786      62.5% 
Average payload per entry......................... 12.06     
Average unused bytes per entry.................... 4.21      
Average fanout.................................... 9.00      
Maximum payload per entry......................... 17        
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 3         
Primary pages used................................ 25        
Overflow pages used............................... 0         
Total pages used.................................. 28        
Unused bytes on index pages....................... 97814       99.50% 
Unused bytes on primary pages..................... 102723      12.5% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 200537      21.9% 

*** Index MOZ_HISTORYVISITS_DATEINDEX of table MOZ_HISTORYVISITS **************

Percentage of total database......................   4.5%    
Number of entries................................. 15865     
Bytes of storage consumed......................... 327680    
Bytes of payload.................................. 206221      62.9% 
B-tree depth...................................... 2         
Average payload per entry......................... 13.00     
Average unused bytes per entry.................... 4.65      
Average fanout.................................... 10.00     
Non-sequential pages.............................. 6           66.7% 
Maximum payload per entry......................... 13        
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 1         
Primary pages used................................ 9         
Overflow pages used............................... 0         
Total pages used.................................. 10        
Unused bytes on index pages....................... 32596       99.48% 
Unused bytes on primary pages..................... 41128       13.9% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 73724       22.5% 

*** Index MOZ_HISTORYVISITS_FROMINDEX of table MOZ_HISTORYVISITS **************

Percentage of total database......................   2.7%    
Number of entries................................. 15869     
Bytes of storage consumed......................... 196608    
Bytes of payload.................................. 100292      51.0% 
B-tree depth...................................... 2         
Average payload per entry......................... 6.32      
Average unused bytes per entry.................... 3.06      
Average fanout.................................... 6.00      
Non-sequential pages.............................. 4           80.0% 
Maximum payload per entry......................... 7         
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 1         
Primary pages used................................ 5         
Overflow pages used............................... 0         
Total pages used.................................. 6         
Unused bytes on index pages....................... 32702       99.80% 
Unused bytes on primary pages..................... 15927        9.7% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 48629       24.7% 

*** Index MOZ_HISTORYVISITS_PLACEDATEINDEX of table MOZ_HISTORYVISITS *********

Percentage of total database......................   5.4%    
Number of entries................................. 15863     
Bytes of storage consumed......................... 393216    
Bytes of payload.................................. 267273      68.0% 
B-tree depth...................................... 2         
Average payload per entry......................... 16.85     
Average unused bytes per entry.................... 4.93      
Average fanout.................................... 12.00     
Non-sequential pages.............................. 8           72.7% 
Maximum payload per entry......................... 17        
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 1         
Primary pages used................................ 11        
Overflow pages used............................... 0         
Total pages used.................................. 12        
Unused bytes on index pages....................... 32516       99.23% 
Unused bytes on primary pages..................... 45668       12.7% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 78184       19.9% 

*** Table MOZ_HOSTS and all its indices ***************************************

Percentage of total database......................   0.90%   
Number of entries................................. 1256      
Bytes of storage consumed......................... 65536     
Bytes of payload.................................. 27640       42.2% 
Average payload per entry......................... 22.01     
Average unused bytes per entry.................... 26.18     
Maximum payload per entry......................... 49        
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 2         
Overflow pages used............................... 0         
Total pages used.................................. 2         
Unused bytes on primary pages..................... 32888       50.2% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 32888       50.2% 

*** Table MOZ_HOSTS w/o any indices *******************************************

Percentage of total database......................   0.45%   
Number of entries................................. 628       
Bytes of storage consumed......................... 32768     
Bytes of payload.................................. 14640       44.7% 
B-tree depth...................................... 1         
Average payload per entry......................... 23.31     
Average unused bytes per entry.................... 23.90     
Maximum payload per entry......................... 49        
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 15012       45.8% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 15012       45.8% 

*** Index SQLITE_AUTOINDEX_MOZ_HOSTS_1 of table MOZ_HOSTS *********************

Percentage of total database......................   0.45%   
Number of entries................................. 628       
Bytes of storage consumed......................... 32768     
Bytes of payload.................................. 13000       39.7% 
B-tree depth...................................... 1         
Average payload per entry......................... 20.70     
Average unused bytes per entry.................... 28.46     
Maximum payload per entry......................... 47        
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 17876       54.6% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 17876       54.6% 

*** Table MOZ_INPUTHISTORY and all its indices ********************************

Percentage of total database......................   0.90%   
Number of entries................................. 16        
Bytes of storage consumed......................... 65536     
Bytes of payload.................................. 642          0.98% 
Average payload per entry......................... 40.12     
Average unused bytes per entry.................... 4050.88   
Maximum payload per entry......................... 71        
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 2         
Overflow pages used............................... 0         
Total pages used.................................. 2         
Unused bytes on primary pages..................... 64814       98.9% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 64814       98.9% 

*** Table MOZ_INPUTHISTORY w/o any indices ************************************

Percentage of total database......................   0.45%   
Number of entries................................. 8         
Bytes of storage consumed......................... 32768     
Bytes of payload.................................. 341          1.0% 
B-tree depth...................................... 1         
Average payload per entry......................... 42.62     
Average unused bytes per entry.................... 4047.38   
Maximum payload per entry......................... 71        
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 32379       98.8% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 32379       98.8% 

*** Index SQLITE_AUTOINDEX_MOZ_INPUTHISTORY_1 of table MOZ_INPUTHISTORY *******

Percentage of total database......................   0.45%   
Number of entries................................. 8         
Bytes of storage consumed......................... 32768     
Bytes of payload.................................. 301          0.92% 
B-tree depth...................................... 1         
Average payload per entry......................... 37.62     
Average unused bytes per entry.................... 4054.38   
Maximum payload per entry......................... 65        
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 32435       99.0% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 32435       99.0% 

*** Table MOZ_ITEMS_ANNOS and all its indices *********************************

Percentage of total database......................   0.90%   
Number of entries................................. 158       
Bytes of storage consumed......................... 65536     
Bytes of payload.................................. 9211        14.1% 
Average payload per entry......................... 58.30     
Average unused bytes per entry.................... 352.56    
Maximum payload per entry......................... 384       
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 2         
Overflow pages used............................... 0         
Total pages used.................................. 2         
Unused bytes on primary pages..................... 55704       85.0% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 55704       85.0% 

*** Table MOZ_ITEMS_ANNOS w/o any indices *************************************

Percentage of total database......................   0.45%   
Number of entries................................. 79        
Bytes of storage consumed......................... 32768     
Bytes of payload.................................. 8649        26.4% 
B-tree depth...................................... 1         
Average payload per entry......................... 109.48    
Average unused bytes per entry.................... 300.54    
Maximum payload per entry......................... 384       
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 23743       72.5% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 23743       72.5% 

*** Index MOZ_ITEMS_ANNOS_ITEMATTRIBUTEINDEX of table MOZ_ITEMS_ANNOS *********

Percentage of total database......................   0.45%   
Number of entries................................. 79        
Bytes of storage consumed......................... 32768     
Bytes of payload.................................. 562          1.7% 
B-tree depth...................................... 1         
Average payload per entry......................... 7.11      
Average unused bytes per entry.................... 404.57    
Maximum payload per entry......................... 9         
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 31961       97.5% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 31961       97.5% 

*** Table MOZ_KEYWORDS and all its indices ************************************

Percentage of total database......................   1.4%    
Number of entries................................. 0         
Bytes of storage consumed......................... 98304     
Bytes of payload.................................. 0            0.0% 
Average payload per entry......................... 0.0       
Average unused bytes per entry.................... 0.0       
Maximum payload per entry......................... 0         
Entries that use overflow......................... 0         
Primary pages used................................ 3         
Overflow pages used............................... 0         
Total pages used.................................. 3         
Unused bytes on primary pages..................... 98280       99.976% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 98280       99.976% 

*** Table MOZ_KEYWORDS w/o any indices ****************************************

Percentage of total database......................   0.45%   
Number of entries................................. 0         
Bytes of storage consumed......................... 32768     
Bytes of payload.................................. 0            0.0% 
B-tree depth...................................... 1         
Average payload per entry......................... 0.0       
Average unused bytes per entry.................... 0.0       
Maximum payload per entry......................... 0         
Entries that use overflow......................... 0         
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 32760       99.976% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 32760       99.976% 

*** Indices of table MOZ_KEYWORDS *********************************************

Percentage of total database......................   0.90%   
Number of entries................................. 0         
Bytes of storage consumed......................... 65536     
Bytes of payload.................................. 0            0.0% 
Average payload per entry......................... 0.0       
Average unused bytes per entry.................... 0.0       
Maximum payload per entry......................... 0         
Entries that use overflow......................... 0         
Primary pages used................................ 2         
Overflow pages used............................... 0         
Total pages used.................................. 2         
Unused bytes on primary pages..................... 65520       99.976% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 65520       99.976% 

*** Index MOZ_KEYWORDS_PLACEPOSTDATA_UNIQUEINDEX of table MOZ_KEYWORDS ********

Percentage of total database......................   0.45%   
Number of entries................................. 0         
Bytes of storage consumed......................... 32768     
Bytes of payload.................................. 0            0.0% 
B-tree depth...................................... 1         
Average payload per entry......................... 0.0       
Average unused bytes per entry.................... 0.0       
Maximum payload per entry......................... 0         
Entries that use overflow......................... 0         
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 32760       99.976% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 32760       99.976% 

*** Index SQLITE_AUTOINDEX_MOZ_KEYWORDS_1 of table MOZ_KEYWORDS ***************

Percentage of total database......................   0.45%   
Number of entries................................. 0         
Bytes of storage consumed......................... 32768     
Bytes of payload.................................. 0            0.0% 
B-tree depth...................................... 1         
Average payload per entry......................... 0.0       
Average unused bytes per entry.................... 0.0       
Maximum payload per entry......................... 0         
Entries that use overflow......................... 0         
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 32760       99.976% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 32760       99.976% 

*** Table MOZ_PLACES and all its indices **************************************

Percentage of total database......................  64.3%    
Number of entries................................. 87087     
Bytes of storage consumed......................... 4653056   
Bytes of payload.................................. 3659043     78.6% 
Average payload per entry......................... 42.02     
Average unused bytes per entry.................... 7.93      
Average fanout.................................... 17.00     
Maximum payload per entry......................... 1867      
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 8         
Primary pages used................................ 134       
Overflow pages used............................... 0         
Total pages used.................................. 142       
Unused bytes on index pages....................... 252188      96.2% 
Unused bytes on primary pages..................... 438258      10.0% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 690446      14.8% 

*** Table MOZ_PLACES w/o any indices ******************************************

Percentage of total database......................  28.5%    
Number of entries................................. 10894     
Bytes of storage consumed......................... 2064384   
Bytes of payload.................................. 1838131     89.0% 
B-tree depth...................................... 2         
Average payload per entry......................... 168.73    
Average unused bytes per entry.................... 14.10     
Average fanout.................................... 62.00     
Non-sequential pages.............................. 30          48.4% 
Maximum payload per entry......................... 1867      
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 1         
Primary pages used................................ 62        
Overflow pages used............................... 0         
Total pages used.................................. 63        
Unused bytes on index pages....................... 32207       98.3% 
Unused bytes on primary pages..................... 121406       6.0% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 153613       7.4% 

*** Indices of table MOZ_PLACES ***********************************************

Percentage of total database......................  35.7%    
Number of entries................................. 76193     
Bytes of storage consumed......................... 2588672   
Bytes of payload.................................. 1820912     70.3% 
Average payload per entry......................... 23.90     
Average unused bytes per entry.................... 7.05      
Average fanout.................................... 11.00     
Maximum payload per entry......................... 1823      
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 7         
Primary pages used................................ 72        
Overflow pages used............................... 0         
Total pages used.................................. 79        
Unused bytes on index pages....................... 219981      95.9% 
Unused bytes on primary pages..................... 316852      13.4% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 536833      20.7% 

*** Index MOZ_PLACES_FAVICONINDEX of table MOZ_PLACES *************************

Percentage of total database......................   2.3%    
Number of entries................................. 10891     
Bytes of storage consumed......................... 163840    
Bytes of payload.................................. 83178       50.8% 
B-tree depth...................................... 2         
Average payload per entry......................... 7.64      
Average unused bytes per entry.................... 4.40      
Average fanout.................................... 5.00      
Non-sequential pages.............................. 3           75.0% 
Maximum payload per entry......................... 8         
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 1         
Primary pages used................................ 4         
Overflow pages used............................... 0         
Total pages used.................................. 5         
Unused bytes on index pages....................... 32711       99.83% 
Unused bytes on primary pages..................... 15213       11.6% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 47924       29.3% 

*** Index MOZ_PLACES_FRECENCYINDEX of table MOZ_PLACES ************************

Percentage of total database......................   2.3%    
Number of entries................................. 10891     
Bytes of storage consumed......................... 163840    
Bytes of payload.................................. 76772       46.9% 
B-tree depth...................................... 2         
Average payload per entry......................... 7.05      
Average unused bytes per entry.................... 4.99      
Average fanout.................................... 5.00      
Non-sequential pages.............................. 3           75.0% 
Maximum payload per entry......................... 9         
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 1         
Primary pages used................................ 4         
Overflow pages used............................... 0         
Total pages used.................................. 5         
Unused bytes on index pages....................... 32714       99.84% 
Unused bytes on primary pages..................... 21616       16.5% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 54330       33.2% 

*** Index MOZ_PLACES_GUID_UNIQUEINDEX of table MOZ_PLACES *********************

Percentage of total database......................   4.1%    
Number of entries................................. 10887     
Bytes of storage consumed......................... 294912    
Bytes of payload.................................. 196000      66.5% 
B-tree depth...................................... 2         
Average payload per entry......................... 18.00     
Average unused bytes per entry.................... 6.07      
Average fanout.................................... 9.00      
Non-sequential pages.............................. 5           62.5% 
Maximum payload per entry......................... 18        
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 1         
Primary pages used................................ 8         
Overflow pages used............................... 0         
Total pages used.................................. 9         
Unused bytes on index pages....................... 32581       99.43% 
Unused bytes on primary pages..................... 33545       12.8% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 66126       22.4% 

*** Index MOZ_PLACES_HOSTINDEX of table MOZ_PLACES ****************************

Percentage of total database......................   5.0%    
Number of entries................................. 10885     
Bytes of storage consumed......................... 360448    
Bytes of payload.................................. 237383      65.9% 
B-tree depth...................................... 2         
Average payload per entry......................... 21.81     
Average unused bytes per entry.................... 8.29      
Average fanout.................................... 11.00     
Non-sequential pages.............................. 7           70.0% 
Maximum payload per entry......................... 49        
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 1         
Primary pages used................................ 10        
Overflow pages used............................... 0         
Total pages used.................................. 11        
Unused bytes on index pages....................... 32473       99.10% 
Unused bytes on primary pages..................... 57782       17.6% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 90255       25.0% 

*** Index MOZ_PLACES_LASTVISITDATEINDEX of table MOZ_PLACES *******************

Percentage of total database......................   3.2%    
Number of entries................................. 10889     
Bytes of storage consumed......................... 229376    
Bytes of payload.................................. 150784      65.7% 
B-tree depth...................................... 2         
Average payload per entry......................... 13.85     
Average unused bytes per entry.................... 4.21      
Average fanout.................................... 7.00      
Non-sequential pages.............................. 4           66.7% 
Maximum payload per entry......................... 14        
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 1         
Primary pages used................................ 6         
Overflow pages used............................... 0         
Total pages used.................................. 7         
Unused bytes on index pages....................... 32651       99.64% 
Unused bytes on primary pages..................... 13179        6.7% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 45830       20.0% 

*** Index MOZ_PLACES_URL_UNIQUEINDEX of table MOZ_PLACES **********************

Percentage of total database......................  16.7%    
Number of entries................................. 10859     
Bytes of storage consumed......................... 1212416   
Bytes of payload.................................. 1010666     83.4% 
B-tree depth...................................... 2         
Average payload per entry......................... 93.07     
Average unused bytes per entry.................... 15.42     
Average fanout.................................... 37.00     
Non-sequential pages.............................. 16          44.4% 
Maximum payload per entry......................... 1823      
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 1         
Primary pages used................................ 36        
Overflow pages used............................... 0         
Total pages used.................................. 37        
Unused bytes on index pages....................... 24134       73.7% 
Unused bytes on primary pages..................... 143261      12.1% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 167395      13.8% 

*** Index MOZ_PLACES_VISITCOUNT of table MOZ_PLACES ***************************

Percentage of total database......................   2.3%    
Number of entries................................. 10891     
Bytes of storage consumed......................... 163840    
Bytes of payload.................................. 66129       40.4% 
B-tree depth...................................... 2         
Average payload per entry......................... 6.07      
Average unused bytes per entry.................... 5.97      
Average fanout.................................... 5.00      
Non-sequential pages.............................. 3           75.0% 
Maximum payload per entry......................... 8         
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 1         
Primary pages used................................ 4         
Overflow pages used............................... 0         
Total pages used.................................. 5         
Unused bytes on index pages....................... 32717       99.84% 
Unused bytes on primary pages..................... 32256       24.6% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 64973       39.7% 

*** Table SQLITE_SCHEMA *******************************************************

Percentage of total database......................   0.45%   
Number of entries................................. 36        
Bytes of storage consumed......................... 32768     
Bytes of payload.................................. 5188        15.8% 
B-tree depth...................................... 1         
Average payload per entry......................... 144.11    
Average unused bytes per entry.................... 758.58    
Maximum payload per entry......................... 379       
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 27309       83.3% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 27309       83.3% 

*** Table SQLITE_SEQUENCE *****************************************************

Percentage of total database......................   0.45%   
Number of entries................................. 1         
Bytes of storage consumed......................... 32768     
Bytes of payload.................................. 15           0.046% 
B-tree depth...................................... 1         
Average payload per entry......................... 15.00     
Average unused bytes per entry.................... 32741.00  
Maximum payload per entry......................... 15        
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 32741       99.918% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 32741       99.918% 

*** Table SQLITE_STAT1 ********************************************************

Percentage of total database......................   0.45%   
Number of entries................................. 15        
Bytes of storage consumed......................... 32768     
Bytes of payload.................................. 762          2.3% 
B-tree depth...................................... 1         
Average payload per entry......................... 50.80     
Average unused bytes per entry.................... 2128.20   
Maximum payload per entry......................... 62        
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 31923       97.4% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 31923       97.4% 

*** Definitions ***************************************************************

Page size in bytes

    The number of bytes in a single page of the database file.  
    Usually 1024.

Number of pages in the whole file

    The number of 32768-byte pages that go into forming the complete
    database

Pages that store data

    The number of pages that store data, either as primary B*Tree pages or
    as overflow pages.  The number at the right is the data pages divided by
    the total number of pages in the file.

Pages on the freelist

    The number of pages that are not currently in use but are reserved for
    future use.  The percentage at the right is the number of freelist pages
    divided by the total number of pages in the file.

Pages of auto-vacuum overhead

    The number of pages that store data used by the database to facilitate
    auto-vacuum. This is zero for databases that do not support auto-vacuum.

Number of tables in the database

    The number of tables in the database, including the SQLITE_SCHEMA table
    used to store schema information.

Number of indices

    The total number of indices in the database.

Number of defined indices

    The number of indices created using an explicit CREATE INDEX statement.

Number of implied indices

    The number of indices used to implement PRIMARY KEY or UNIQUE constraints
    on tables.

Size of the file in bytes

    The total amount of disk space used by the entire database files.

Bytes of user payload stored

    The total number of bytes of user payload stored in the database. The
    schema information in the SQLITE_SCHEMA table is not counted when
    computing this number.  The percentage at the right shows the payload
    divided by the total file size.

Percentage of total database

    The amount of the complete database file that is devoted to storing
    information described by this category.

Number of entries

    The total number of B-Tree key/value pairs stored under this category.

Bytes of storage consumed

    The total amount of disk space required to store all B-Tree entries
    under this category.  The is the total number of pages used times
    the pages size.

Bytes of payload

    The amount of payload stored under this category.  Payload is the data
    part of table entries and the key part of index entries.  The percentage
    at the right is the bytes of payload divided by the bytes of storage 
    consumed.

Average payload per entry

    The average amount of payload on each entry.  This is just the bytes of
    payload divided by the number of entries.

Average unused bytes per entry

    The average amount of free space remaining on all pages under this
    category on a per-entry basis.  This is the number of unused bytes on
    all pages divided by the number of entries.

Non-sequential pages

    The number of pages in the table or index that are out of sequence.
    Many filesystems are optimized for sequential file access so a small
    number of non-sequential pages might result in faster queries,
    especially for larger database files that do not fit in the disk cache.
    Note that after running VACUUM, the root page of each table or index is
    at the beginning of the database file and all other pages are in a
    separate part of the database file, resulting in a single non-
    sequential page.

Maximum payload per entry

    The largest payload size of any entry.

Entries that use overflow

    The number of entries that user one or more overflow pages.

Total pages used

    This is the number of pages used to hold all information in the current
    category.  This is the sum of index, primary, and overflow pages.

Index pages used

    This is the number of pages in a table B-tree that hold only key (rowid)
    information and no data.

Primary pages used

    This is the number of B-tree pages that hold both key and data.

Overflow pages used

    The total number of overflow pages used for this category.

Unused bytes on index pages

    The total number of bytes of unused space on all index pages.  The
    percentage at the right is the number of unused bytes divided by the
    total number of bytes on index pages.

Unused bytes on primary pages

    The total number of bytes of unused space on all primary pages.  The
    percentage at the right is the number of unused bytes divided by the
    total number of bytes on primary pages.

Unused bytes on overflow pages

    The total number of bytes of unused space on all overflow pages.  The
    percentage at the right is the number of unused bytes divided by the
    total number of bytes on overflow pages.

Unused bytes on all pages

    The total number of bytes of unused space on all primary and overflow 
    pages.  The percentage at the right is the number of unused bytes 
    divided by the total number of bytes.

*******************************************************************************
The entire text of this report can be sourced into any SQL database
engine for further analysis.  All of the text above is an SQL comment.
The data used to generate this report follows:
*/
BEGIN;
CREATE TABLE space_used(
   name clob,        -- Name of a table or index in the database file
   tblname clob,     -- Name of associated table
   is_index boolean, -- TRUE if it is an index, false for a table
   nentry int,       -- Number of entries in the BTree
   leaf_entries int, -- Number of leaf entries
   depth int,        -- Depth of the b-tree
   payload int,      -- Total amount of data stored in this table or index
   ovfl_payload int, -- Total amount of data stored on overflow pages
   ovfl_cnt int,     -- Number of entries that use overflow
   mx_payload int,   -- Maximum payload size
   int_pages int,    -- Number of interior pages used
   leaf_pages int,   -- Number of leaf pages used
   ovfl_pages int,   -- Number of overflow pages used
   int_unused int,   -- Number of unused bytes on interior pages
   leaf_unused int,  -- Number of unused bytes on primary pages
   ovfl_unused int,  -- Number of unused bytes on overflow pages
   gap_cnt int,      -- Number of gaps in the page layout
   compressed_size int  -- Total bytes stored on disk
);
INSERT INTO space_used VALUES('sqlite_schema','sqlite_schema',0,36,36,1,5188,0,0,379,0,1,0,0,27309,0,0,32768);
INSERT INTO space_used VALUES('moz_places','moz_places',0,10955,10894,2,1838131,0,0,1867,1,62,0,32207,121406,0,30,2064384);
INSERT INTO space_used VALUES('moz_historyvisits','moz_historyvisits',0,15884,15873,2,308447,0,0,21,1,12,0,32668,5435,0,8,425984);
INSERT INTO space_used VALUES('moz_inputhistory','moz_inputhistory',0,8,8,1,341,0,0,71,0,1,0,0,32379,0,0,32768);
INSERT INTO space_used VALUES('sqlite_autoindex_moz_inputhistory_1','moz_inputhistory',1,8,8,1,301,0,0,65,0,1,0,0,32435,0,0,32768);
INSERT INTO space_used VALUES('moz_hosts','moz_hosts',0,628,628,1,14640,0,0,49,0,1,0,0,15012,0,0,32768);
INSERT INTO space_used VALUES('sqlite_autoindex_moz_hosts_1','moz_hosts',1,628,628,1,13000,0,0,47,0,1,0,0,17876,0,0,32768);
INSERT INTO space_used VALUES('moz_bookmarks','moz_bookmarks',0,313,313,1,21937,0,0,518,0,1,0,0,9358,0,0,32768);
INSERT INTO space_used VALUES('moz_bookmarks_roots','moz_bookmarks_roots',0,5,5,1,47,0,0,11,0,1,0,0,32693,0,0,32768);
INSERT INTO space_used VALUES('sqlite_autoindex_moz_bookmarks_roots_1','moz_bookmarks_roots',1,5,5,1,47,0,0,11,0,1,0,0,32698,0,0,32768);
INSERT INTO space_used VALUES('moz_keywords','moz_keywords',0,0,0,1,0,0,0,0,0,1,0,0,32760,0,0,32768);
INSERT INTO space_used VALUES('sqlite_autoindex_moz_keywords_1','moz_keywords',1,0,0,1,0,0,0,0,0,1,0,0,32760,0,0,32768);
INSERT INTO space_used VALUES('sqlite_sequence','sqlite_sequence',0,1,1,1,15,0,0,15,0,1,0,0,32741,0,0,32768);
INSERT INTO space_used VALUES('moz_favicons','moz_favicons',0,481,471,2,297630,0,0,7640,1,11,0,32676,59923,0,6,393216);
INSERT INTO space_used VALUES('sqlite_autoindex_moz_favicons_1','moz_favicons',1,471,470,2,35135,0,0,7259,1,2,0,32664,29057,0,1,98304);
INSERT INTO space_used VALUES('moz_anno_attributes','moz_anno_attributes',0,12,12,1,355,0,0,42,0,1,0,0,32357,0,0,32768);
INSERT INTO space_used VALUES('sqlite_autoindex_moz_anno_attributes_1','moz_anno_attributes',1,12,12,1,366,0,0,43,0,1,0,0,32358,0,0,32768);
INSERT INTO space_used VALUES('moz_annos','moz_annos',0,195,195,1,12115,0,0,127,0,1,0,0,19702,0,0,32768);
INSERT INTO space_used VALUES('moz_items_annos','moz_items_annos',0,79,79,1,8649,0,0,384,0,1,0,0,23743,0,0,32768);
INSERT INTO space_used VALUES('sqlite_stat1','sqlite_stat1',0,15,15,1,762,0,0,62,0,1,0,0,31923,0,0,32768);
INSERT INTO space_used VALUES('moz_places_faviconindex','moz_places',1,10894,10891,2,83178,0,0,8,1,4,0,32711,15213,0,3,163840);
INSERT INTO space_used VALUES('moz_places_hostindex','moz_places',1,10894,10885,2,237383,0,0,49,1,10,0,32473,57782,0,7,360448);
INSERT INTO space_used VALUES('moz_places_visitcount','moz_places',1,10894,10891,2,66129,0,0,8,1,4,0,32717,32256,0,3,163840);
INSERT INTO space_used VALUES('moz_places_frecencyindex','moz_places',1,10894,10891,2,76772,0,0,9,1,4,0,32714,21616,0,3,163840);
INSERT INTO space_used VALUES('moz_places_lastvisitdateindex','moz_places',1,10894,10889,2,150784,0,0,14,1,6,0,32651,13179,0,4,229376);
INSERT INTO space_used VALUES('moz_historyvisits_placedateindex','moz_historyvisits',1,15873,15863,2,267273,0,0,17,1,11,0,32516,45668,0,8,393216);
INSERT INTO space_used VALUES('moz_historyvisits_fromindex','moz_historyvisits',1,15873,15869,2,100292,0,0,7,1,5,0,32702,15927,0,4,196608);
INSERT INTO space_used VALUES('moz_historyvisits_dateindex','moz_historyvisits',1,15873,15865,2,206221,0,0,13,1,9,0,32596,41128,0,6,327680);
INSERT INTO space_used VALUES('moz_bookmarks_itemindex','moz_bookmarks',1,313,313,1,2547,0,0,9,0,1,0,0,29274,0,0,32768);
INSERT INTO space_used VALUES('moz_bookmarks_parentindex','moz_bookmarks',1,313,313,1,2393,0,0,9,0,1,0,0,29428,0,0,32768);
INSERT INTO space_used VALUES('moz_bookmarks_itemlastmodifiedindex','moz_bookmarks',1,313,313,1,5020,0,0,17,0,1,0,0,26801,0,0,32768);
INSERT INTO space_used VALUES('moz_places_url_uniqueindex','moz_places',1,10894,10859,2,1010666,0,0,1823,1,36,0,24134,143261,0,16,1212416);
INSERT INTO space_used VALUES('moz_places_guid_uniqueindex','moz_places',1,10894,10887,2,196000,0,0,18,1,8,0,32581,33545,0,5,294912);
INSERT INTO space_used VALUES('moz_bookmarks_guid_uniqueindex','moz_bookmarks',1,313,313,1,5207,0,0,17,0,1,0,0,26614,0,0,32768);
INSERT INTO space_used VALUES('moz_annos_placeattributeindex','moz_annos',1,195,195,1,1871,0,0,10,0,1,0,0,30304,0,0,32768);
INSERT INTO space_used VALUES('moz_items_annos_itemattributeindex','moz_items_annos',1,79,79,1,562,0,0,9,0,1,0,0,31961,0,0,32768);
INSERT INTO space_used VALUES('moz_keywords_placepostdata_uniqueindex','moz_keywords',1,0,0,1,0,0,0,0,0,1,0,0,32760,0,0,32768);
COMMIT;


