Developing for CloudKit - Part 4

Read Part 1 in series on Developing for CloudKit

Read Part 2 in series on Developing for CloudKit

Read Part 3 in series on Developing for CloudKit

In the first 3 parts of Developing for CloudKit, I walk through creating a sync layer for Core Data that uses CloudKit as its backend.

I mentioned at the end of part 3 that I wanted to “abstract the sync layer so it's independent of your data model. I hope to open source it at some point in 2015”.

In the past year, I improved and made many changes to the sync layer (called SSCloudManager) and it shipped as the syncing mechanism in Screenshot++.

The main issue is SSCloudManager is still tightly coupled to the Screenshot++ data model. Creating a version of SSCloudManager that’s data model agnostic would require substantial time to invest that I simply do not have. Because of that, I began searching for a SSCloudManager replacement that didn’t involve months of my time that I would rather spend elsewhere.

Currently, I’m using Ensembles 2 as a replacement for SSCloudManager in development of a few projects. It’s everything I envisioned SSCloudManager to be and then some. I love it.

I’ll be posting in future about adopting Ensembles 2 into my projects. In the mean time, if you’ve created your own sync layer, I would love to hear about it.

Sunset for Parse

Allen Pike’s piece on the Parse shutdown is insightful, in-depth, and a great read. Marco’s response adds his thoughtful perspective.

In early 2013, I began looking for a syncing service to use in multiple iOS apps. I was one in a long list of developers who viewed sync as an add on, that sync should be as easy as adding another project dependancy. Parse added much of the functionality I was looking for. It was easy, cheap, and the company seemed to understand the needs of their customers.

It wasn’t long after I began testing Parse integration in a sample project that news broke of the Facebook acquisition. Parse promised that with Facebook’s backing, Parse would be able to grow faster and not worry about monetization. All in the name of good will for developers.

While I applauded their enthusiasm, the acquisition meant that from that day forward, Parse would be acting in Facebook’s best interests, leaving developers in second place.

Because I have zero faith in Facebook, I could not have faith in Parse as a product or a company and began exploring alternatives.

That decision paid off this week. Instead of scrambling to figure out a new backend as a service (BaaS) and migration strategy, I can instead use that time and effort to continue building new apps and making improvements to existing products.

The Parse shutdown has brought the BaaS debate back to the forefront in the developer community. That is, should developers roll their own BaaS or migrate to a Parse alternative like CloudKit?

Instead of relying on Parse, I jumped into Apple’s CloudKit over a year ago. It’s baked into Apple’s platforms and (for now) has Apple’s support. I use CloudKit as a back-end for Screenshot++ and in unannounced projects.

Allen and Marco lay out the two sides of this debate quite well. For many developers who have experience in running their own servers, this has always an easy decision. Experienced backend developers like Marco haven’t had to rely on 3rd party solutions. For them, it’s relatively easy to roll their own solution and today they’re in the clear; it’s hard not to hear the smug comments coming from atop their high horses.

For the rest of us, Parse was the solution to a problem that few app developers have the experience or resources to properly solve. I don’t view Parse as a shortcut and I don’t think the objective should be to rely on a service like Parse until you can create something better in-house.

For the 600,000 apps that used Parse, they’re in a tight spot and most of those will end up abandonware if they’re not already. To insinuate their developers should have all created custom backends is out of touch with today’s App Store economic reality.

With Parse out of the picture and the app BaaS market underserved, now is the perfect time for a new competitor to fill the void.

Developing for CloudKit - Part 3

Read Part 1 in series on Developing for CloudKit

Read Part 2 in series on Developing for CloudKit

In the 6 weeks since my last post on CloudKit Development, I’ve added the ability to make changes to a CKRecord and have those changes synced between devices. In this post, I'll cover syncing changes and data considerations.

The CKRecord from the previous posts had an image property that never changed after upload to CloudKit. Since then, I’ve added 2 properties that will need their changes synced. The first is a boolean, the second is an array.

I began by making the fatal assumption about CKRecord’s recordChangeTag property. I believed I could use it to query the server for CKRecords with changes and download changes on the result - turns out I was way off the mark.

/* Change tags are updated by the server to a unique value every time a record is modified.
   A different change tag necessarily means that the contents of the record are different. */
@property (nonatomic, readonly, copy) NSString *recordChangeTag;

That’s not what recordChangeTag was intended for so I had to look for another solution.

That solution was CloudKit’s Push Notifications. Basically, you register the app for notifications and the device will receive a notification each time CKRecord is updated (you can register for delete/creation notifs too).

That was great and all, but what do you do with that notification?

First, I don’t want to start a sync session in a background task. Downloading CKRecords in the background would be acceptable if they were small (<50KB) but they’re at least 3MB each. I also want to plan ahead and account for CKRecords with assets over 25MB each.

My goal from the start was to keep the sync engine as light-weight as possible, with minimal network transfers.

The other issue with background syncing is having to load the full sync engine as a background process, meaning far more overhead than necessary. Yes, these are artificial constraints I’ve placed on the project, and many apps on the App Store cram far more into a background task, but I digress.

The solution was to create a singleton class that keeps track of the CKRecordID.recordNames from the notification. I load the singleton, check to see if the recordID has been previously added, add it to an array if it hasn’t, then save to disk. The whole process is fast and lightweight. When the app resumes, I can download change to the records stored in the singleton, then clear the list.

Every decision comes with trade-offs. In a perfect world, I would sync changes in a background session. Of course it’s better for the user’s data to be up to date each time they open the app. But for the sake of battery life, mobile data usage, and overall performance of the OS, I think it’s the worth the user waiting 1-3 seconds for sync to complete. It’s a tradeoff I’m willing to make.

If you’ve read the previous 2 posts, updating the data for changes happens at the end of the sync session, after local offline changes are synced. In the spirit of doing the least amount of work possible, we sync inserts and deletions before changes. Consider this scenario - there are 5 total objects, 3 are on the client (synced previouslly), 2 new objects need to be downloaded from the server, one local object needs to be deleted, all objects have changes. If we synced the changes first, we would be updating the 3 local objects. Instead, we should complete the local deletion first, resulting in only updating the 2 remaining objects.

That didn’t explain the entire sync process but it illustrates the types of considerations you’ll have to make. Saving the user 5-10KB here and there doesn’t seem like much now, but after 1,000 syncs, it adds up.

What about insert/deletion notifications?

These are good for your typical CloudKit-based app where the data is online-only. For an app that allows the user to make offline changes, you need to handle this yourself.

CloudKit wasn't intended to sync Core Data. It was designed as a backend, a way for app developers to focus on native apps and let Apple handle the server implementation and cost. The sync engine I created is a small layer that sits between Core Data and CloudKit.

I'm happy with the results so far. The next step is to abstract the sync layer so it's independent of your data model. I hope to open source it at some point in 2015.

Update: Screenshot++ has shipped with the CloudKit sync engine. Give it a spin!

Developing For CloudKit- Part 2

Read Part 1 in series on Developing for CloudKit

It's been a week since my first post on developing an iOS app that uses CloudKit as a syncing solution to work with Core Data. Creating a mediation layer between Core Data and CloudKit has been no walk in the part but it's simpler than it first seemed.

My app, for now, doesn't need to make changes to its data objects. I'm constricting myself to the creation and deletion of objects in CloudKit. That decision has taken a load off my plate as I, for the first time, tackle sync totally by myself.

I started with converting NSManagedObjects to CKRecords. After the upload is complete, I use the (NSString *)recordName from CKRecord's (CKRecordID *)recordID returned from the completion handler and save that back to the NSManagedObject. This allows me to keep a local reference matching an NSManagedObject to the corresponding CKRecord stored in CloudKit. For local objects that don't have a valid recordName, I'll get to that in a bit.

Next is delete, very similar, I first delete the matching CKRecord in CloudKit (if there is one) and then delete my local copy once the server deletion is finished. CloudKit has extremely useful block callback and I highly suggest you use them!

Run that on 2 devices, add an object on the first device and... nothing happens. What?

We'll have to create a CKSubscription that will send a Push Notification to other devices signed into the same iCloud account when a CKRecord was uploaded, deleted, or updated in CloudKit. I won't go into detail about APNS here but it's not entirely complicated. Each notification will (for the most part) give you a reference to the changed object in its payload. Use that to update the corresponding NSManagedObject.

Now we have an app that syncs out locally saved data but for a device to get changes, it has to be on, open to the app. An app that isn't running will never know about changes. Next we have to add a way for our app to query the server when launched. This is a little more difficult than the steps I've covered so far.

Checking for changes that occurred while offline or in airplane mode will be done in 3 stages.

The first, to backtrack a bit, we need to keep track of objects that were deleted while offline. and delete them before the next steps. If we forget this, any offline deletions will be re-added later and create a frustrating problem for the user.

Second, we check for any CKRecords in CloudKit that we don't have a matching copy of on our local device. Start by getting a list of all CKRecordIDs, compare that to all objects we have locally, and remove the matches. Now, you're left with a list of CKRecords to download.

After that, check for local objects with a valid recordName property. Local objects with a recordName that no longer exists on the server means that object has been deleted on another device. You can safely remove those objects.

Once that's complete, you've successfully updated your device with changes made on CloudKit's server, but that's only half the story. We finish by reversing that process and updating CloudKit with changes to out local objects that happen while offline. 

This process is a little faster than the last. Start by creating an array of local objects that have a nil recordName. These have either never been synced or the upload wasn't successful. Either way, they need to be uploaded, do that now.

After that, you're done. (I say that knowing I've probably forgotten a step and will feel pretty foolish when my inbox fills up. Email Marco.)

I hope this has helped you, writing it all out helped me realize I forgot a step in my code. Bonus points if you can figure out what step I added at the last minute.

Update: Screenshot++ is now available with the CloudKit sync engine. Give it a spin!

Developing with CloudKit - First Impressions

I've recently started work on a new app that uses iOS 8's new CloudKit framework. I'm using CloudKit as the storage backend and Core Data for the local cache. So far, I'm impressed. CloudKit is a much-welcome departure and improvement over iCloud's Core Data Sync.

Before CloudKit, we relied on iCloud's Core Data sync mechanism to transfer data between devices. From the start, it was unreliable at the best of times. There was a fair bit of magic going on behind the scenes and the lack of transparency from Apple made debugging impossible. With new OS versions came significant increases in reliability but it problems persisted.

CloudKit aims to remove that magic and be as transparent as possible. In return, CloudKit is a transportation layer, sans magic. In essence, it handles sending data to iCloud, storing that data, and notifying your app where there's new data to be downloaded. The rest is up to you.

And that's actually a pretty great thing. This new app I'm working on simply takes an image, some strings and dates, and send that as a packaged object to the cloud. Setting up the sync part was actually pretty simple and took less than a day. It worked reliably, even with the network conditioner set to a poor network connection.

Next was implementing local offline storage. This is the tricky part.

I want to have the data saved onto the device for offline use and because some of the data is large and I respect my customer's data bills. When the device returns to the network, any changes are propagated via CloudKit.

Sounds a lot like sync, what's the difference?

Instead of creating a Core Data stack and then trying to get sync to work, I'm developing for CloudKit first. I'm getting Core Data to work nicely with CloudKit instead of trying to force CloudKit to play nicely with Core Data. This seemingly small distinction makes a large impact. Core Data is a robust framework and, thanks to Magical Record, is exceptionally flexible. CloudKit, on the other hand, is straight-forward and lean. 

On one side, we have CloudKit, the other the highly flexible Core Data. Now we need something to communicate between the two, like a translator. Let's call this the Intermediary.

With the ClouKit approach, Apple has left the Intermediary up to developers. This Intermediary is what I was referring to by the 'magic' of iCloud Data Sync. What happens when you try to create an Intermediary that has to cover all possible data structures and scale well? Turns out, not well.

So what I need is a simple, generic Intermediary that fits, but is not limited to, the data schemas I'm using. Data created on the device (NSManagedObject *) is saved via Core Data, the Intermediary is notified of the changes and converts the data to a CloudKit record (CKRecord *), and the record is handed off to Apple's servers via CloudKit. Any data in iCloud that's not on the device is brought down in the opposite direction converting the CKRecord back into, or creating, an NSManagedObject.

Because of the object type conversion, I have to keep a reference to later know which NSManagedObject corresponds to which CKRecord. If a new record was downloaded and I didn't know which local object to match it to, I would end up with endless duplicates. So do I save a reference to the NSManagedObject in the corresponding CKRecord or save a reference of the record in the object? 

For now, I'll try keeping a reference to the CKRecordID because UIDs on NSManagedObject change with each device. At least, that's the way it seems. More on that next time. I have my work cut out for me.

As Steve said, "The truth is in the Cloud". CloudKit is a lesson in treating cloud data as a first class citizen. 

Update: Screenshot++ is now available on the App Store with the CloudKit sync engine. Give it a spin!