YapDatabase Tutorial

ZeroDark uses a database internally. It needs atomic transactions to ensure syncing works smoothly. And for this task, it uses YapDatabase (because it's awesome, secure, and very fast).

You don't have to use YapDatabase if you don't want to. If you'd prefer to use Core Data, and continue subclassing NSManagedObject (barf), you can go right on abusing yourself.

We're biased, obviously :) But all joking aside, you're free to use any system you want. Whether that's a different database (such as Core Data), or maybe even your own custom solution. The framework doesn't force you to use its preferred database.

With that being said, you may want to consider using the same database as ZeroDark. The biggest advantage to doing so would be that you can share in the same atomic transaction that the framework uses. Plus the framework does all the work of setting up the database for you, so it's ready-to-go.

 

Introduction

YapDatabase is a collection/key/value store. The "hello world" looks like this:

let db = YapDatabase()
let connection = db.newConnection()

// We're going to store String's in the collection "test".
db.registerCodableSerialization(String.self, forCollection: "test")

connection.readWrite {(transaction) in

  transaction.setObject("quack", forKey: "duck", inCollection: "test")
}

connection.read {(transaction) in

  let str = transaction.object(forKey: "duck", inCollection: "test") as? String
  // str == "quack"
}

You can store any type of data in YapDB:

class Foo: Codable { // Just implement Codable !
    // ...
}
struct Bar: Codable { // Just implement Codable !
}

db.registerCodableSerialization(Foo.self, forCollection: "foos")
db.registerCodableSerialization(Foo.self, forCollection: "bars")

let foo = Foo()
let bar = Bar()

connection.readWrite {(transaction) in

  transaction.setObject(foo, forKey: "1", inCollection: "foos")
  transaciton.setObject(bar, forKey: "2", inCollection: "bars")
}

connection.read {(transaction) in

  let foo = transaction.object(forKey: "1", inCollection: "foos") as? Foo
  let bar = transaction.object(forKey: "2", inCollection: "bars") as? Bar
}

So it's really easy to use.

 

The ZeroDark database

When you're setting up ZeroDark, you can pass a closure that allows you to configure YapDatabase for your own custom objects:

let dbConfig = ZDCDatabaseConfig(encryptionKey: key)

dbConfig.configHook = {(db: YapDatabase) in

  db.registerCodableSerialization(Foo.self, forCollection: "foos")
  db.registerCodableSerialization(Bar.self, forCollection: "bars")

  // Further configuration can go here
}

zdc.unlockOrCreateDatabase(dbConfig)

And after that, the database is ready for you to use. You can get access to the database, or various connections through the DatabaseManager:

let dbManager = zdc.databaseManager!

// If you want access to the underlying YapDB instance
let db: YapDatabase = dbManager.database

// There's a dedicated connection you can use for
// reading from the database on the main thread.
// (More on this in another section below.)
let uiConnection: YapDatabaseConnection = dbManager.uiDatabaseConnection

// There's a dedicated connection for performing read-write transactions:
let rwConnection: YapDatabaseConnection = dbManager.rwDatabaseConnection

// Since a read-write transaction involves writing stuff to disk,
// it's recommended (by Apple) you do so on a background thread.
// This is really easy via an asyncReadWrite transaction:
// 
rwConnection.asyncReadWrite {(transaction) in

  // You're now in a background thread, as recommended by Apple.
  // So you won't block the main thread.
  transaction.setObject(foo, forKey: "1", inCollection: "foos")
}

// And ZeroDark creates a load-balanced connection pool for
// performing reads on background threads:
// 
dbManager.roDatabaseConnection.read {(transaction) in

  // Look mom, a load-balancer for my database!
}

 

YapDB Extensions

YapDatabase is both simple to use, and at the same time very powerful. This is because the database supports extensions. And it comes with a bunch of them built-in:

Extension Description
Views Sort, Group & Filter your data. Perfect for tableViews, collectionViews, and more.
Secondary Indexes Optimize your queries and find your item(s) faster.
Full Text Search Blazing fast search using SQLite's FTS module.
Relationships Create relationships between objects, and configure automatic deletion rules.
FilteredViews Quickly filter an existing view, and create view chains.
R Tree Index Fast Geospatial Queries

There's a LOT of documentation on the YapDB Wiki.

There are also some good tutorials on the web, like this this one.

 

YapDatabaseView

One of the most commonly used extensions are Views.

For example, let's say that you want to display a UITableView with all the Recipe objects that are desserts. This means you need a way to filter and sort Recipes. And this is what the "view" extension does for you:

dbConfig.configHook = {(db: YapDatabase) in

  db.registerCodableSerialization(Recipe.self, forCollection: "recipes")
  self.registerView(db)
}

func registerView(_ db: YapDatabase) {

  // GROUPING CLOSURE:
  // - Filter out items you don't want in the view.
  // - Group the items you do want in the view
  //
  let grouping = YapDatabaseViewGrouping.withObjectBlock({
    (transaction, collection, key, obj) -> String? in

    guard let recipe = obj as? Recipe else {
      return nil // exclude from view
    }
    if recipe.isSweet {
      return "sweet" // include in view, group="sweet"
    } else {
      return "savory" // incude in view, group="savory"
    }
  })

  // SORTING CLOSURE:
  // - sort recipes within each group
  //
  let sorting = YapDatabaseViewSorting.withObjectBlock({
    (transaction, group, collection1, key1, obj1,
     collection2, key2, obj2) -> ComparisonResult in

    let recipe1 = obj1 as! Recipe
      let recipe2 = obj2 as! Recipe

    return recipe1.name.compare(recipe1.name)
  })

  let version = "1"; // change me if you modify grouping or sorting closure

  let options = YapDatabaseViewOptions()
  options.allowedCollections = YapWhitelistBlacklist(whitelist: Set(["recipes"]))

  let view =
    YapDatabaseAutoView(grouping: grouping,
                         sorting: sorting,
                      versionTag: version,
                         options: options)

  let extName = "dessert_recipes_view"
    database.asyncRegister(view, withName: extName) {(ready) in

    if !ready {
      print("Error registering \(extName) !!!")
    }
  }
}

The nice thing about Views is that you get to use Swift code to perform your grouping and sorting:

  • You did NOT have to use esoteric SQL syntax
  • You did NOT have to pre-define your Recipe object, and all its various properties

But wait, it gets even better. You just plugged your own custom code directly into the database. So when you change the database in the future, YapDB will automatically invoke your grouping & sorting closures as needed to automatically update your view:

  • If you delete a Recipe object, it automatically gets deleted from the View
  • If you add a new Recipe object, it will automatically get added to the View (assuming it's a dessert)
  • If you rename a Recipe, it will automatically get re-sorted within the View

And when a view is changed, it will send you a notification that tells you exactly what changes occurred. And this information is specifically designed for animating changes to a tableView / collectionView.

More information on Views can be found on YapDB's long tutorial.