Handling large collections of files: Difference between revisions

The [https://www.sqlite.org SQLite software] allows for the use of a relational database which resides entirely in a single file stored on disk, without the need for a database server. The data located in the file can be accessed using standard [https://en.wikipedia.org/wiki/SQL SQL] (Structured Query Language) commands such as <tt>SELECT</tt> and there is are APIs for several common languages by means of which you can interact with your SQLite database inside of a program written in C/C++, Python, R, Java and Perl. Modern relational databases contain datatypes for handling the storage of ''binary blobs'', such as the contents of an image file, so storing a collection of 5 or 10 million small PNG or JPEG images inside of a single SQLite file may be much more practical than storing them as individual files. There is the overhead of creating the SQLite database and this approach assumes that you are familiar with SQL and how to design a simple relational database with a small number of tables. Note as well that the performance of SQLite can start to degrade for very large database files, several gigabytes or more, in which case you may need to contemplate the use of a more traditional  [[Database servers | database server]] using [https://www.mysql.com MySQL] or [https://www.postgresql.org PostgreSQL].