Push Queue

Everytime an upload operation is requested (i.e. creating, modifying or deleting a node), an object of type ZDCCloudOperation is created and stored in the database. This queue of operations is referred to as the "pipeline".

Since the operations are safely stored to disk, it doesn't matter if the user is online or offline. When offline, ZeroDark quietly queues your operations, and waits for internet connectivity to be restored. And once the user goes online again, the system will start dequeueing operations in the pipeline.

 

ZDCCloudOperation

An instance of ZDCCloudOperation incorporates the general instructions for modifying the cloud in some way. These are the actual items that get stored in the database, and queued up in the pipeline. They are lightweight, and include minimal information. For example:

• Upload node X
• Delete node Y

(Note: Operations do NOT include the data that is to be uploaded. Just the general outline of what needs to be done. The node data is fetched on demand, at the moment it's needed.)

In most cases, these operation instances are automatically created for you. For example, if you create a node, the framework automatically enqueues operations to upload that node.

However, you can also create operations explicitly. For example, ZeroDark.cloud has an option to copy items in the cloud from one location to another. Your app may wish to make use of this for various reasons. In which case you can simply queue up the operation you'd like to perform.

 

Dependencies

Every operation can specify a list of dependencies. For example, you can specify that opC depends on both opA & opB. Then the system will ensure that opC is not started until both opA & opB have completed.

You can think of operation dependencies as HARD REQUIREMENTS

The framework will automatically create depenencies for you based on the treesystem. For example, imagine the following operations got enqueued:

Commit #1

• opA: upload: /foo/bar

Commit #2

• opB: upload: /foo/bar/buzz

In this case, the system will automatically create a dependency: opB will depend on opA. Meaning that opB will not be started until opA has completed. This dependency was created automatically because /foo/bar/buzz is a descendant of /foo/bar.

In practice, you rarely have to think about dependcies yourself. This is because the framework does the right thing by simply exploiting the treesystem hierarchy. Another example:

Commit #1

• opA: delete: /foo

Commit #2

• opB: create: /foo

Commit #3

• opC: rename: /foo => /moo

Commit #4

• opD: create: /moo/cow

Again, the framework will setup all the correct dependencies for you:

• opB depends on opA
• opC depends on opB
• opD depends on opC

 

Parallel Operations

In the absence of dependencies, the default pipeline flow allows operations to be executed in parallel. For example:

Commit #1

• opA: upload: /moo/cow  (priority == 0)

Commit #2

• opB: upload: /quack/duck  (priority == 0)

Since there are no dependencies between the 2 operations, the framework is free to upload both nodes in parallel. Strictly speaking, it will prefer opA, because it was scheduled first. However it's free to start opB before opA finishes.

 

Manually adding dependencies

There may be times when you'd like to manually add dependencies. For example, perhaps you are creating a new customer at the same time you're creating a new purchase order. You'd prefer that the customer node gets uploaded before the purchase node, because the purchase references the customerID. You can accomplish this by adding a dependecy to the operation:

let db = zdc.databaseManager!
db.rwDatabaseConnection.asyncReadWrite {(transaction) in

  guard let cloudTransaction = zdc.cloudTransaction(transaction, forLocalUserID: self.localUserID) else {
    return
  }

  transaction.setObject(customer, forKey: customer.uuid, inCollection: kCollection_Customers)
  transaction.setObject(purchase, forKey: purchase.uuid, inCollection: kCollection_Purchases)

  do {
    let customerNode = try cloudTransaction.createNode(withPath: self.treePath(customer))
    let purchaseNode = try cloudTransaction.createNode(withPath: self.treePath(purchase))

    let customerOps = cloudTransaction.addedOperations(forNodeID: customerNode.uuid),
    let purchaseOps = cloudTransaction.addedOperations(forNodeID: purchaseNode.uuid)

    for op in purchaseOps {
      op.addDependencies(customerOps ?? []) // upload customer first
      cloudTransaction.modifyOperation(op)
    }

  } catch {
    print("Error creating node: \(error)")
  }
}

 

Priorities

In contrast to dependencies, operation priorities are more like SOFT HINTS.

That is, you can give an operation a higher or lower priority as a hint to the system. The push system will take the priority into consideration (along with dependencies), when dequeueing operations.

 

YapDatabseCloudCore

If you'd like to dive deeper into the architecture of the push queue, you may wish to check out the documentation for YapDatabaseCloudCore.

YapDatabaseCloudCore is a YapDatabase extension that was created by us, and contributed to the open source community. It forms the backbone of our own push queue system. In fact, the ZDCCloud class extends YapDatabaseCloudCore. Which means you get all the functionality & power of YapDBCloudCore, should you ever need it.