The Case of the Mysteriously Creeping Database

While teaching a recent ForgeRock OpenDJ class, a student of mine observed an interesting behavior that at first seemed quite odd.  While rebuilding his attribute indexes, the student found that the overall database size seemed to grow each time he performed a reindex operation.  What seems obvious to me now sure made me scratch my head as I scrambled for an answer.  I am sharing my findings here in the hopes that others will either a) find this information useful or b) find comic relief as to my misfortune.

In OpenDJ, the rebuild-index command is used to update any attribute indexes contained in the OpenDJ database. This is necessary after you make a configuration change that affects indexes (such as modifying the index entry limit). Indexes are database specific and you can elect to rebuild a single attribute index or rebuild all attribute indexes for a particular database.

The following syntax is used to rebuild ALL indexes associated with the dc=example,dc=com suffix and its use is what caused the frantic head-scratching to occur:

$ rebuild-index -h ldap.example.com -p 4444 -D "cn=Directory Manager" -w password -b dc=example,dc=com --rebuildAll

Mystery

The student observed (and questioned) that every time he rebuilt the indexes, the aggregated size of the *.jdb files actually increased by some factor. In the case of a rebuild-all, it was about 18 MB each time he ran the command; in the case of rebuilding a single index, it was only about 3 MB each time.  But the increase was consistent each time he rebuilt the index(es). This continued to occur until it reached a certain size at which time the consumption fell back to its original size (in our observations this occurred at roughly 200 MB when using the rebuild-all option).

The following details the output of the du -sh command on the userRoot database each time the rebuild-index command was run:

• 124 MB
• 142 MB (+ 18MB)
• 160 MB (+ 18MB)
• 178 MB (+ 18MB)
• 200 MB (+ 22MB)
• 123 MB (- 77MB)

This trend was consistent over several iterations.

We continued testing and observed that in addition to the increasing size, the database files on the file system (*.jdb) were changing as well. What was once 000000001.jdb and 000000002.jdb now became 000000002.jdb file and 000000003.jdb and later became 000000003.jdb file and 000000004.jdb. This occurred at the same time that we dropped back down to the 123 MB size and was the clue that unlocked the mystery.

Solution

Unlike the Berkeley Sleepycat database used in OpenDJ’s forefathers, when data is modified in the OpenDJ database, it is not immediately removed from the database. Instead it is marked for removal and the record essentially becomes inactive. Updated records are then appended to the end of the database in a log file fashion.

This process continues until OpenDJ cleaner threads detect that a database file contains less than 50% active records. Once that occurs, the cleaner threads migrate all active records from the file and append them to the end of the last file in the OpenDJ database (a new file is created if necessary). Once migrated, the cleaner threads delete the database file containing the stale entries.

During the rebuild process, old index values in each of the *.jdb files are marked as inactive and new indexes are added to the database. Simply marking these indexes as inactive does not eliminate their existence in the database and they continue to consume disk space. This process continues until the point where the cleaner threads detect that old indexes account for > 50% of the database entries.  At this point, the migration process occurs, new *.jdb files are created to store the new indexes, old stale *.jdb files are deleted (hence the *.jdb file name changes), and the disk space is returned.

Word from ForgeRock’s OpenDJ Product Manager, Ludo Poitou confirmed my suspicions in the following response:

When an index is rebuilt, the whole btree is marked as deleted. But since it actually represents specific records of the database files, they will only be collected when the file itself reaches the threshold that triggers recollection.

With small databases, you will see the behavior you’re describing. With larger databases, this will be less noticeable as the amount of index records will be larger and cleanup point may be reached faster.

So there you go, mystery solved!