Push

It's easy to push data up to the cloud. It's just a two step process:

  • Create a node in the treesystem
  • Supply the data you want to upload (ZeroDark will encrypt it for you, and upload it for you)

 

Creating a node

Imagine we have the following treesystem:

        (home)
        /    \
    (foo)     (moo)
    /   \        |
(bar)   (buzz)  (cow)

And now we want to add a child (soap) to the node (bar). Here's what the process looks like in code.

let treesystemPath =
  ZDCTreesystemPath(pathComponents: [ "foo", "bar", "soap" ])

let db = zdc.databaseManager!
db.rwDatabaseConnection.asyncReadWrite {(transaction) in

  if let cloudTransaction = zdc.cloudTransaction(transaction, forLocalUserID: self.localUserID) {
    do {
      try cloudTransaction.createNode(withPath: treesystemPath)
    } catch {}
    // Node is now added to treesystem, and queued for upload
  }
}

Walking thru the code 1 step at-a-time:

 

ZDCTreesystemPath

To create a path you use the ZDCTreesystemPath class. Throughout the framework, this is always the class you'll use when dealing with paths. This means you don't have to worry about separator characters, or escaping illegal characters, or any of the usual baggage that comes with string-based paths.

(There aren't any illegal characters in the ZeroDark treesystem. Anything that's valid UTF-8 is acceptable. More information can be found here.)

In this example, the path is being hard-coded. But in your app you'll probably already have the path to (bar), in which case you can use something like this instead:

let treesystemPath = parentPath.appendingComponent("soap")

 

Database

ZeroDark uses a database to perform atomic transactions. When you want to alter its state (e.g. by adding a node), you do so via a read-write transaction.

There's great benefit from atomic transactions. In the code sample above, you're actually doing more than just creating a node. When you insert the node into the treesystem, the ZeroDark framework will simulataneously enqueue a push operation to upload this node to the cloud. Even if the user is currently offline, this operation is persited to disk. And it will get executed as soon as the push manager is capable of performing the operation.

Also, you're welcome to use the same database to store your own objects. You're not required to do so. But if you do, you'll have the benefit of adding your own custom objects to the database within the same atomic transaction that creates the node & enqueues the push operations. This ensures your data model is guaranteed to be in-sync with the ZeroDark framework, thanks to atomic transactions that you control.

 

cloudTransaction

The ZeroDark framework supports multiple logged-in users. So just grab the appropriate one.

The cloudTransaction is an instance of ZDCCloudTransaction, which is the primary class you'll deal with when modifying the treesystem (i.e. creating, modifying or deleting nodes).

 

createNode(withPath:)

This one-liner is all you need to create the node. Here's what happens:

  • The framework creates a ZDCNode instance, and adds it to the treesystem.
  • A ZDCNode instance contains all the metadata about the node, such as its name, parent, permissions, etc.
  • The created ZDCNode instance is returned by this method (if you need it).
  • The framework also enqueues the operations to push this node up to the cloud. (It doesn't start the network operations until after the atomic commit hits the disk.)

It will throw an error if:

  • Path components leading to the node are missing. For example, if "bar" doesn't already exist.
  • Or there is already a node at the given path.

 

ZDCNode != YourCustomObject

The ZDCNode class is just used for storing treesystem metadata. You do not subclass ZDCNode. Instead, you get to use your own custom model classes for your app's content. All you need to do is create the treesystem hierarchy in whatever manner works best for your app. It's completely up to you to decide how nodes in the treesystem are associated with objects or files in your app. (Discussed in more detail below.)

 

Uploading the node

After the node has been created, the framework will handle pushing it up to the cloud. This is an asynchronous process. The user may be offline. Or the operation may have dependencies that are still unfinished. But when the PushManager is ready to upload the node, it will query the ZeroDarkCloudDelegate to get the data that should be uploaded:

/// ZeroDark is asking us to supply the data for a node.
/// This is the data that will get uploaded to the cloud.
/// ZeroDark handles the encryption & uploading for us.
///
func data(for node: ZDCNode, at path: ZDCTreesystemPath, transaction: YapDatabaseReadTransaction) -> ZDCData {

  // Implement me !
}

Implementing this method is a 2 step process:

  1. A node is generally linked to one of your own custom objects. So determine which object the node is correlated with. (There are several utility functions which can assist with this process. More on this in the next section.)
  2. Generate the data you'd like to upload to the cloud. What you return is an instance of ZDCData, which is a simple wrapper for a data source. It supports returning data in several formats (from in-memory data, files, or via a promise).

 

Mapping from nodes to your objects

ZDCNode is a lightweight object that stores only the metadata associated with the node. This includes the name & permissions, as well as various information used for syncing, such as eTags and timestamps.

You do NOT subclass ZDCNode. Instead, you're free to create your own custom model classes. And you're free to store them however you want. There are several benefit to this system.

It's often the case that there are separate optimal designs for your data - one for local storage, and one for cloud storage.

When you're designing your data model for the local app, you're thinking about how the app itself fetches items from the database. And how best to model that data for fast retrieval, and for sorting into tableViews, etc. This is a task that we're already familiar with.

But when you're thinking about designing your data model for the cloud, the considerations are different. You're optimizing for uploads & downloads. What design will allow you to download data on demand? And how will you know what the data is without downloading it? (This is where the treesystem comes into play.)

Thus the architecture allows you to optimize in 2 dimensions: both for working with your data locally, and for storing in the cloud. And all you need is a system that bridges the gap. You'll need to be able to map back and forth, from node/treepath to your object, and vice-versa.

Sometimes this is a straight-forward task that requires little thought. And for the other times, the framework comes with several tools to help out.

One-to-one mappings

If you store your custom objects in the same database that ZeroDark is using, then you can link your objects to their corresponding node.

db.rwDatabaseConnection.asyncReadWrite {(transaction) in

  if let cloudTransaction = zdc.cloudTransaction(transaction, forLocalUserID: self.localUserID) {

    do {
      // Create the node
      let node = try cloudTransaction.createNode(withPath: treesystemPath)

      // Link it to our own object
      try cloudTransaction.linkNodeID(node.uuid, toKey: taskID, inCollection: "tasks")

    } catch {
      print("Error creating/linking node: \(error)")
    }
  }
}

Once linked, the API's make it easy to fetch the needed info on demand:

db.uiDatabaseConnection.read {(transaction) in

  if let cloudTransaction = zdc.cloudTransaction(transaction, forLocalUserID: self.localUserID) {

    // nodeID => {collection, key}
    let (collection, key) = cloudTransaction.linkedCollectionAndKey(forNodeID: node.uuid)

    // nodeID => myObject
    let task = cloudTransaction.linkedObject(forNodeID: node.uuid) as? Task

    // myObject => nodeID
    let node = cloudTransaction.linkedNode(forKey: taskID, inCollection: "tasks")
  }
}

 

Using tags for mappings

Another option is to use tags. With tags, you can associate various information with a node. Mapping information is just one such use case:

db.rwDatabaseConnection.asyncReadWrite {(transaction) in

  if let cloudTransaction = zdc.cloudTransaction(transaction, forLocalUserID: self.localUserID) {

    // Storing a tag
    cloudTransaction.setTag(myTask.uuid, forNodeID: node.uuid, withIdentifier: "taskID")

    // Reading a tag
    let taskID = cloudTransaction.tag(forNodeID: node.uuid, withIdentifier: "taskID")
  }
}

You can see the full list of functions in the API docs for ZDCCloudTransaction. And for more information on this topic, see the mappings tutorial.

 

Supplying node data

The second step in uploading a node is to supply the data you'd like to upload to the cloud. All you need to do is generate and return the data, and the framework handles:

  • encrypting the data for you
  • uploading it to the cloud
  • if the data is large, the framework automatically uses a multipart upload process, so it can quickly recover from network interruptions
  • and on iOS, it perfoms the upload as a background operation

The framework doesn't impose any kind of restrictions on the data you provide. You're free to serialize your objects however you want. Plus the framework supports everything from small files to multi-gigabyte files. In order to support this diversity of file sizes, you return an instance of ZDCData, which is a simple wrapper for a data source.

From in-memory data

To upload an object, you just need to serialize it. For example, you might convert your object to JSON, and then return the UTF-8 encoded version of the JSON string:

if let list = transaction.object(forKey: listID, inCollection: collection) as? List {
  do {
    let data = try list.serializeAsJSON()
    return ZDCData(data: data)
  }
}

The ZeroDark framework doesn't care how you serialize your objects, so you're welcome to use whatever system you prefer.

 

From a file

ZeroDark supports eveything from tiny serialized objects up to multi-gigabyte sized files. So you can upload files of any size.

To instruct ZeroDark to upload from a file, just give it a reference to the fileURL, and it will handle the rest.

if let fileURL = self.getFileURL(forNode: node) {
  return ZDCData(cleartextFileURL: fileURL)
}

 

From a crypto file

You can use the DiskManager to cache/persist files to disk. It has a bunch of useful features, such as limiting the amount of disk space used, or automatically deleting expired files. The DiskManager also automatically encrypts the files it stores to disk.

To instruct ZeroDark to upload from an encrypted file, you just give it a reference to a CryptoFile, and it will handle the rest.

let export = zdc.diskManager?.nodeData(node)
if let cryptoFile = export.cryptoFile {
  return ZDCData(cryptoFile: cryptoFile)
}

 

Via an asynchronous process

Sometimes your app requires an asynchronous process in order to supply the needed data. No problem. You can create a ZDCNode instance which uses a promise to supply the data. And ZeroDark will automatically wait for you to fullfill/reject that promise.

let promise = ZDCDataPromise()
DispatchQueue.global(qos: .default).async {
  // Async process to generate cloud data...
  let data = ZDCData(...)
  promise.fulfill(data)
}

return ZDCData(promise: promise)

The ZDCDataPromise class works like most promies. You just need to invoke either the fullfill or reject function.

 

Once you've provided the data for the node, ZeroDark will handle encrypting it and uploading it to the cloud. And after the data has been uploaded, the ZeroDarkCloudDelegate receives a notification:

/// ZeroDark just pushed our data to the cloud.
///
func didPushNodeData(_ node: ZDCNode, at path: ZDCTreesystemPath, transaction: YapDatabaseReadWriteTransaction) {
  // ...
}

In addition to the above notification, you can also monitor the upload progress via the ProgressManager.

 

Node thumbnails & metadata

All data stored in the cloud is encrypted using keys that you control. This means our servers (i.e. the servers managed by ZeroDark.cloud) are incapable of reading any of your content. So you can't do things like ask the server to provide a thumbnail of an image. The server can't read the image. The server doesn't even know the content is an image (it just sees an encrypted blob).

The security benefits are obvious. But apps still need to fetch thumbnails. So there's an easy solution for this: include the thumbnail within the file being uploaded. This way the data (full image + thumbnail) gets uploaded to the server in an atomic fashion. And this capability is built into ZeroDark. Here's how it works:

All files stored in the cloud are stored in a format we simply call "CloudFile format". The layout of this file (when decrypted) looks like this:

| header | metadata(optional) | thumbnail(optional) | data

At the very beginning of the file is a fixed size header. This header includes information such as the size of the metadata section and the size of the thumbnail section. All the sections are then appended to the header to get the cleartext version of the file. And then the cleartext version of the file is encrypted before being uploaded. (All sections are encrypted, including the header.)

In the previous section, you read about how to provide the 'data' section. There are similar API's that allow you to provide the optional metadata & thumbnail sections for the file.

func metadata(for node: ZDCNode, at path: ZDCTreesystemPath, transaction: YapDatabaseReadTransaction) -> ZDCData? {

  return nil // or add metadata
}

func thumbnail(for node: ZDCNode, at path: ZDCTreesystemPath, transaction: YapDatabaseReadTransaction) -> ZDCData? {

  return nil // or add thumbnail
}

These optional sections are useful if you're uploading large images or other large files such as movies. Their usefulness is derived from the fact that you can download the metadata and/or thumbnail sections individually. ( faster downloads! use less bandwidth! snappier user interface! )

For example, if your node is a large image, and you want to display a thumbnail of the image in your UI, then you would:

  • include a thumbnail section when uploading the node (the size of the thumbnail is up to you)
  • then you could use the ImageManager to fetch just the small thumbanil image
  • or you could use the DownloadManager to download individual sections

 

Modifying a node's data

If you change a node's data, you simply tell the framework to enqueue a push operation for it. That way it will upload the modified object or file.

For example, here's how you might update an object in the database, and enqueue the associated operation.

let newTitle = textView.text

let db = zdc.databaseManager!
db.rwDatabaseConnection.asyncReadWrite {(transaction) in

  if var task = transaction.object(forKey: taskID inCollection: kCollection_Tasks) {
    task = task.copy() as! Task
    task.title = newTitle
    transaction.setObject(task, forKey: taskID, inCollection: kCollection_Tasks)
  }

  if let cloudTransaction = zdc.cloudTransaction(transaction, forLocalUserID: self.localUserID),
     let nodeID = cloudTransaction.linkedNodeID(forKey: taskID, inCollection: kCollection_Tasks) {
    cloudTransaction.queueDataUpload(forNodeID: nodeID, withChangeset: nil)
  }
}

 

Deleting a node

Deleting a node is easy too:

db.rwDatabaseConnection.asyncReadWrite {(transaction) in

  if let cloudTransaction = zdc.cloudTransaction(transaction, forLocalUserID: self.localUserID) {

    do {
      // Removes the given node from the treesystem,
      // and enqueues a delete operation to delete it from the cloud.
      try cloudTransaction.delete(node)

    } catch {}
  }
}