MistKit

Swift Package for Server-Side and Command-Line Access to CloudKit Web Services

GitHub GitHub issues

Demonstration of MistKit via Command-Line App `mistdemoc`

Introduction
Features
Installation
Usage
Roadmap
- ~~0.1.0~~
- ~~0.2.0~~
- 0.4.0
- 0.6.0
- 0.8.0
- 0.9.0
- v1.0.0
License

Introduction

Rather than the CloudKit framework this Swift package uses CloudKit Web Services.. Why?

Building a Command Line Application
Use on Linux (or any other non-Apple OS)
Required for Server-Side Integration (via Vapor)
Access via AWS Lambda
Migrating Data from/to CloudKit

… and more

In my case, I was using this for the Vapor back-end for my Apple Watch app Heartwitch. Here’s some example code showing how to setup and use MistKit with CloudKit container.

Demo Example

CloudKit Dashboard Schema

Sample Schema for Todo List

Sample Code using MistKit

// Example for pulling a todo list from CloudKit
import MistKit
import MistKitNIOHTTP1Token

// setup your connection to CloudKit
let connection = MKDatabaseConnection(
  container: "iCloud.com.brightdigit.MistDemo", 
  apiToken: "****", 
  environment: .development
)

// setup how to manager your user's web authentication token 
let manager = MKTokenManager(storage: MKUserDefaultsStorage(), client: MKNIOHTTP1TokenClient())

// setup your database manager
let database = MKDatabase(
  connection: connection,
  tokenManager: manager
)

// create your request to CloudKit
let query = MKQuery(recordType: TodoListItem.self)

let request = FetchRecordQueryRequest(
  database: .private, 
  query: FetchRecordQuery(query: query))

// handle the result
database.query(request) { result in
  dump(result)
}

// wait for query here...

To wait for the CloudKit query to complete synchronously, you can use CFRunLoop:

...
// handle the result
database.query(request) { result in
  dump(result)

  // nessecary if you need run this synchronously
  CFRunLoopStop(CFRunLoopGetMain())
}

// nessecary if you need run this synchronously
CFRunLoopRun()

Features

Here’s what’s currently implemented with this library:

Composing Web Service Requests
Modifying Records (records/modify)
Fetching Records Using a Query (records/query)
Fetching Records by Record Name (records/lookup)
Fetching Current User Identity (users/caller)

Installation

Swift Package Manager is Apple’s decentralized dependency manager to integrate libraries to your Swift projects. It is now fully integrated with Xcode 11.

To integrate MistKit into your project using SPM, specify it in your Package.swift file:

let package = Package(
  ...
  dependencies: [
    .package(url: "https://github.com/brightdigit/MistKit", from: "0.2.0")
  ],
  targets: [
      .target(
          name: "YourTarget",
          dependencies: ["MistKit", ...]),
      ...
  ]
)

There are also products for SwiftNIO as well as Vapor if you are building server-side implmentation:

      .target(
          name: "YourTarget",
          dependencies: ["MistKit", 
            .product(name: "MistKitNIO", package: "MistKit"),  // if you are building a server-side application
            .product(name: "MistKitVapor", package: "MistKit") // if you are building a Vapor application
            ...]
      ),

Usage

Composing Web Service Requests

MistKit requires a connection be setup with the following properties:

container name in the format of iCloud.com.*.* such as iCloud.com.brightdigit.MistDemo
apiToken which can be created through the CloudKit Dashboard
environment which can be either development or production

Here’s an example of how to setup an MKDatabase:

let connection = MKDatabaseConnection(
  container: options.container, 
  apiToken: options.apiKey, 
  environment: options.environment)

// setup your database manager
let database = MKDatabase(
  connection: connection,
  tokenManager: manager
)

Before getting into make an actual request, you should probably know how to make authenticated request for private or shared databases.

Setting Up Authenticated Requests

In order to have access to private or shared databases, the Cloud Web Services API require a web authentication token. In order for the MistKit to obtain this, an http server is setup to listen to the callback from CloudKit.

Therefore when you setup your API token, make sure to setup a url for the Sign-In Callback:

CloudKit Dashboard

Once that’s setup, you can setup a MKTokenManager.

CloudKit Dashboard Callback

Managing Web Authentication Tokens

MKTokenManager requires a MKTokenStorage for storing the token for later. There are a few implementations you can use:

MKFileStorage stores the token as a simple text file
MKUserDefaultsStorage stores the token using UserDefaults
MKVaporModelStorage stores the token in a database Model object via Fluent
MKVaporSessionStorage stores the token the Vapor Session data

Optionally MistKit can setup a web server for you if needed to listen to web authentication via a MKTokenClient: There are a few implementations you can use:

MKNIOHTTP1TokenClient sets up an http server using SwiftNIO

Here’s an example of how you MKDatabase:

let connection = MKDatabaseConnection(
  container: options.container, 
  apiToken: options.apiKey, 
  environment: options.environment
 )

// setup how to manager your user's web authentication token
let manager = MKTokenManager(
  // store the token in UserDefaults
  storage: MKUserDefaultsStorage(), 
  // setup an http server at localhost for port 7000
  client: MKNIOHTTP1TokenClient(bindTo: .ipAddress(host: "127.0.0.1", port: 7000))
)

// setup your database manager
let database = MKDatabase(
  connection: connection,
  tokenManager: manager
)

Using `MKNIOHTTP1TokenClient`

If you are not building a server-side application, you can use MKNIOHTTP1TokenClient, by adding MistKitNIO to your package dependency:

let package = Package(
  ...
  dependencies: [
    .package(url: "https://github.com/brightdigit/MistKit", .branch("main")
  ],
  targets: [
      .target(
          name: "YourTarget",
          dependencies: ["MistKit", "MistKitNIOHTTP1Token", ...]),
      ...
  ]
)

When a request fails due to authentication failure, MKNIOHTTP1TokenClient will start an http server to begin listening to web authentication token. By default, MKNIOHTTP1TokenClient will simply print the url but you can override the onRequestURL:

public class MKNIOHTTP1TokenClient: MKTokenClient {
  
  public init(bindTo: BindTo, onRedirectURL : ((URL) -> Void)? = nil) {
    self.bindTo = bindTo
    self.onRedirectURL = onRedirectURL ?? {print($0)}
  }
  ...
}

CloudKit and Vapor

Static Web Authentication Tokens

If you may already have a webAuthenticationToken, you can use MKStaticTokenManager. This is a read-only implementation of MKTokenManagerProtocol which takes a read-only String? for the webAuthenticationToken.

Here’s some sample code I use in my Vapor app Heartwitch for pulling the webAuthenticationToken from my database and using that token when I create a MKDatabase instance.

import MistKit
import MistKitVapor

extension Application {
  ...
  var cloudKitConnection: MKDatabaseConnection {
    MKDatabaseConnection(
      container: configuration.cloudkitContainer,
      apiToken: configuration.cloudkitAPIKey,
      environment: environment.cloudKitEnvironment
    )
  }

  func cloudKitDatabase(using client: Client, withWebAuthenticationToken webAuthenticationToken: String? = nil) -> MKDatabase<MKVaporClient> {
    MKDatabase(
      connection: cloudKitConnection,
      client: MKVaporClient(client: client),
      tokenManager: MKStaticTokenManager(token: webAuthenticationToken, client: nil)
    )
  }
}

struct DeviceController {

  func fetch(_ request: Request) throws -> EventLoopFuture<MKServerResponse<[DeviceResponseItem]>> {
    let user = try request.auth.require(User.self)
    let userID = try user.requireID()
    let token = user.$appleUsers.query(on: request.db).field(\.$webAuthenticationToken).first().map { $0?.webAuthenticationToken }

    let cloudKitDatabase: EventLoopFuture<MKDatabase> = token.map {
      request.application.cloudKitDatabase(using: request.client, withWebAuthenticationToken: $0)
    }
    
    let cloudKitRequest = FetchRecordQueryRequest(
      database: .private,
      query: FetchRecordQuery(query: query)
    )
    
    let newEntries = cloudKitDatabase.flatMap {
      let cloudKitResult = cloudKitDatabase.query(cloudKitRequest, on: request.eventLoop)
    }

    return newEntries.mistKitResponse()
  }
  
  ...
}

Besides static strings, you can store your tokens in the session or in your database.

Storing Web Authentication Tokens in Databases and Sessions

In the mistdemod demo Vapor application, there’s an example of how to create an MKDatabase based on the request using both MKVaporModelStorage and MKVaporSessionStorage:

extension MKDatabase where HttpClient == MKVaporClient {
  init(request: Request) {
    let storage: MKTokenStorage
    if let user = request.auth.get(User.self) {
      storage = MKVaporModelStorage(model: user)
    } else {
      storage = MKVaporSessionStorage(session: request.session)
    }
    let manager = MKTokenManager(storage: storage, client: nil)

    let options = MistDemoDefaultConfiguration(apiKey: request.application.cloudKitAPIKey)
    let connection = MKDatabaseConnection(container: options.container, apiToken: options.apiKey, environment: options.environment)

    // use the webAuthenticationToken which is passed
    if let token = options.token {
      manager.webAuthenticationToken = token
    }

    self.init(connection: connection, factory: nil, client: MKVaporClient(client: request.client), tokenManager: manager)
  }
}

In this case, for the User model needs to implement MKModelStorable.

final class User: Model, Content {
  ...

  @Field(key: "cloudKitToken")
  var cloudKitToken: String?
}

extension User: MKModelStorable {
  static var tokenKey: KeyPath<User, Field<String?>> = \User.$cloudKitToken
}

The MKModelStorable protocol ensures that the Model contains the properties needed for storing the web authentication token.

While the command line tool needs a MKTokenClient to listen for the callback from CloudKit, with a server-side application you can just add a API call. Here’s an example which listens for the ckWebAuthToken and saves it to the User:

struct CloudKitController: RouteCollection {
  func token(_ request: Request) -> EventLoopFuture<HTTPStatus> {
    guard let token: String = request.query["ckWebAuthToken"] else {
      return request.eventLoop.makeSucceededFuture(.notFound)
    }

    guard let user = request.auth.get(User.self) else {
      request.cloudKitAPI.webAuthenticationToken = token
      return request.eventLoop.makeSucceededFuture(.accepted)
    }

    user.cloudKitToken = token
    return user.save(on: request.db).transform(to: .accepted)
  }

  func boot(routes: RoutesBuilder) throws {
    routes.get(["token"], use: token)
  }
}

If you have an app which already uses Apple’s existing CloudKit API, you can also save the webAuthenticationToken to your database with a CKFetchWebAuthTokenOperation.

Fetching Records Using a Query (records/query)

There are two ways to fetch records:

using an MKAnyQuery to fetch MKAnyRecord items
using a custom type which implements MKQueryRecord

Setting Up Queries

To fetch as MKAnyRecord, simply create MKAnyQuery with the matching recordType (i.e. schema name).

// create your request to CloudKit
let query = MKAnyQuery(recordType: "TodoListItem")

let request = FetchRecordQueryRequest(
  database: .private,
  query: FetchRecordQuery(query: query)
)

// handle the result
database.perform(request: request) { result in
  do {
    try print(result.get().records.information)
  } catch {
    completed(error)
    return
  }
  completed(nil)
}

This will give you MKAnyRecord items which contain a fields property with your values:

public struct MKAnyRecord: Codable {
  public let recordType: String
  public let recordName: UUID?
  public let recordChangeTag: String?
  public let fields: [String: MKValue]
  ...

The MKValue type is an enum which contains the type and value of the field.

Strong-Typed Queries

In order to use a custom type for requests, you need to implement MKQueryRecord. Here’s an example of a todo item which contains a title property:

public class TodoListItem: MKQueryRecord {
  // required property and methods for MKQueryRecord
  public static var recordType: String = "TodoItem"
  public static var desiredKeys: [String]? = ["title"]

  public let recordName: UUID?
  public let recordChangeTag: String?
  
  public required init(record: MKAnyRecord) throws {
    recordName = record.recordName
    recordChangeTag = record.recordChangeTag
    title = try record.string(fromKey: "title")
  }
  
  public var fields: [String: MKValue] {
    return ["title": .string(title)]
  }
  
  // custom fields and methods to `TodoListItem`
  public var title: String
  
  public init(title: String) {
    self.title = title
    recordName = nil
    recordChangeTag = nil
  }
}

Now you can create an MKQuery using your custom type.

// create your request to CloudKit
let query = MKQuery(recordType: TodoListItem.self)

let request = FetchRecordQueryRequest(
  database: .private,
  query: FetchRecordQuery(query: query)
)

// handle the result
database.query(request) { result in
  do {
    try print(result.get().information)
  } catch {
    completed(error)
    return
  }
  completed(nil)
}

Rather than using MKDatabase.perform(request:), use MKDatabase.query(_ query:) and MKDatabase will decode the value to your custom type.

Filters

Coming Soon

Fetching Records by Record Name (records/lookup)

let recordNames : [UUID] = [...]

let query = LookupRecordQuery(TodoListItem.self, recordNames: recordNames)

let request = LookupRecordQueryRequest(database: .private, query: query)

database.lookup(request) { result in
  try? print(result.get().count)
}

Coming Soon

Fetching Current User Identity (users/caller)

let request = GetCurrentUserIdentityRequest()
database.perform(request: request) { (result) in
  try? print(result.get().userRecordName)
}

Coming Soon

Modifying Records (records/modify)

Creating Records

let item = TodoListItem(title: title)

let operation = ModifyOperation(operationType: .create, record: item)

let query = ModifyRecordQuery(operations: [operation])

let request = ModifyRecordQueryRequest(database: .private, query: query)

database.perform(operations: request) { result in
  do {
    try print(result.get().updated.information)
  } catch {
    completed(error)
    return
  }
  completed(nil)
}

Deleting Records

In order to delete and update records, you are required to already have the object fetched from CloudKit. Therefore you’ll need to run a LookupRecordQueryRequest or FetchRecordQueryRequest to get access to the record. Once you have access to the records, simply create a delete operation with your record:

let query = LookupRecordQuery(TodoListItem.self, recordNames: recordNames)

let request = LookupRecordQueryRequest(database: .private, query: query)

database.lookup(request) { result in
  let items: [TodoListItem]
  
  do {
    items = try result.get()
  } catch {
    completed(error)
    return
  }
  
  let operations = items.map { (item) in
    ModifyOperation(operationType: .delete, record: item)
  }

  let query = ModifyRecordQuery(operations: operations)

  let request = ModifyRecordQueryRequest(database: .private, query: query)
  
  database.perform(operations: request) { result in
    do {
      try print("Deleted \(result.get().deleted.count) items.")
    } catch {
      completed(error)
      return
    }
    completed(nil)
  }
}

Updating Records

Similarly with updating records, you are required to already have the object fetched from CloudKit. Again, run a LookupRecordQueryRequest or FetchRecordQueryRequest to get access to the record. Once you have access to the records, simply create a update operation with your record:

let query = LookupRecordQuery(TodoListItem.self, recordNames: [recordName])

let request = LookupRecordQueryRequest(database: .private, query: query)

database.lookup(request) { result in
  let items: [TodoListItem]
  do {
    items = try result.get()

  } catch {
    completed(error)
    return
  }
  let operations = items.map { (item) -> ModifyOperation<TodoListItem> in
    item.title = self.newTitle
    return ModifyOperation(operationType: .update, record: item)
  }

  let query = ModifyRecordQuery(operations: operations)

  let request = ModifyRecordQueryRequest(database: .private, query: query)
  database.perform(operations: request) { result in
    do {
      try print("Updated \(result.get().updated.count) items.")
    } catch {
      completed(error)
      return
    }
    completed(nil)
  }
}

Using SwiftNIO

If you are building a server-side application and already using SwiftNIO, you might want to take advantage of some helpers which will work already existing patterns and APIs available. Primarily EventLoops from SwiftNIO and the respective HTTP clients from SwiftNIO and Vapor.

Using EventLoops

If you are building a server-side application in SwiftNIO (or Vapor), you are likely using EventLoops and EventLoopFuture for asyncronous programming. EventLoopFutures are essentially the Future/Promise implementation of SwiftNIO. Luckily there are helper methods in MistKit which provide EventLoopFutures similar to the way they implmented in SwiftNIO. These implementations augment the already existing callback:

public extension MKDatabase {
  func query<RecordType>(
    _ query: FetchRecordQueryRequest<MKQuery<RecordType>>,
    on eventLoop: EventLoop
  ) -> EventLoopFuture<[RecordType]>

  func perform<RecordType>(
    operations: ModifyRecordQueryRequest<RecordType>,
    on eventLoop: EventLoop
  ) -> EventLoopFuture<ModifiedRecordQueryResult<RecordType>>
  
  func lookup<RecordType>(
    _ lookup: LookupRecordQueryRequest<RecordType>,
    on eventLoop: EventLoop
  ) -> EventLoopFuture<[RecordType]>

  func perform<RequestType: MKRequest, ResponseType>(
    request: RequestType,
    on eventLoop: EventLoop
  ) -> EventLoopFuture<ResponseType> -> EventLoopFuture<ResponseType>
    where RequestType.Response == ResponseType
}

Also if you are using the results as Content for a Vapor HTTP response, MistKit provides a MKServerResponse enum type which distinguishes between an authentication failure (with the redirect URL) and an actual success.

public enum MKServerResponse<Success>: Codable where Success: Codable {
  public init(attemptRecoveryFrom error: Error) throws

  case failure(URL)
  case success(Success)
}

Besides EventLoopFuture, you can also use a different HTTP client for calling CloudKit Web Services.

Choosing an HTTP Client

By default, MistKit uses URLSession for making HTTP calls to the CloudKit Web Service via the MKURLSessionClient:

public struct MKURLSessionClient: MKHttpClient {
  public init(session: URLSession) {
    self.session = session
  }

  public func request(withURL url: URL, data: Data?) -> MKURLRequest
}

However if you are using SwiftNIO or Vapor, it makes more sense the use their HTTP clients for making those calls:

For SwiftNIO, there’s MKAsyncClient which uses an HTTPClient provided by the AsyncHTTPClient library
For Vapor, there’s MKVaporClient which uses an Client provided by the Vapor library

In the mistdemod example, you can see how to use a Vapor Request to create an MKDatabase with the client property of the Request:

extension MKDatabase where HttpClient == MKVaporClient {
  init(request: Request) {
    let manager: MKTokenManager    
    let connection : MKDatabaseConnection
    self.init(
      connection: connection, 
      factory: nil, 
      client: MKVaporClient(client: request.client), 
      tokenManager: manager
    )
  }
}

Examples

There are two examples on how to do basic CRUD methods in CloudKit via MistKit:

As a command line tool using Swift Argument Parser checkout the mistdemoc Swift package executable here
And a server-side Vapor application mistdemod here

Further Code Documentation

Documentation Here

Roadmap

0.1.0

Composing Web Service Requests
Modifying Records (records/modify)
Fetching Records Using a Query (records/query)
Fetching Records by Record Name (records/lookup)
Fetching Current User Identity (users/caller)

0.2.0

Vapor Token Client
Vapor Token Storage
Vapor URL Client
Swift NIO URL Client

0.4.0

Date Field Types
Location Field Types
List Field Types
System Field Integration

0.6.0

Name Component Types
Discovering User Identities (POST users/discover)
Discovering All User Identities (GET users/discover)
Support postMessage for Authentication Requests

0.8.0

Uploading Assets (assets/upload)
Referencing Existing Assets (assets/rereference)
Fetching Records Using a Query (records/query) w/ basic filtering

0.9.0

Fetching Contacts (users/lookup/contacts)
Fetching Users by Email (users/lookup/email)
Fetching Users by Record Name (users/lookup/id)

v1.0.0

Reference Field Types
Error Codes
Handle Data Size Limits

v1.x.x+

Fetching Record Changes (records/changes)
Fetching Record Information (records/resolve)
Accepting Share Records (records/accept)
Fetching Zones (zones/list)
Fetching Zones by Identifier (zones/lookup)
Modifying Zones (zones/modify)
Fetching Database Changes (changes/database)
Fetching Record Zone Changes (changes/zone)
Fetching Zone Changes (zones/changes)
Fetching Subscriptions (subscriptions/list)
Fetching Subscriptions by Identifier (subscriptions/lookup)
Modifying Subscriptions (subscriptions/modify)
Creating APNs Tokens (tokens/create)
Registering Tokens (tokens/register)

Not Planned

Fetching Current User (users/current) deprecated

License

This code is distributed under the MIT license. See the LICENSE file for more info.

MistKit

Table of Contents

Introduction

Demo Example

CloudKit Dashboard Schema

Sample Code using MistKit

Features

Installation

Usage

Composing Web Service Requests

Setting Up Authenticated Requests

Managing Web Authentication Tokens

Using MKNIOHTTP1TokenClient

CloudKit and Vapor

Static Web Authentication Tokens

Storing Web Authentication Tokens in Databases and Sessions

Fetching Records Using a Query (records/query)

Setting Up Queries

Strong-Typed Queries

Filters

Fetching Records by Record Name (records/lookup)

Fetching Current User Identity (users/caller)

Modifying Records (records/modify)

Creating Records

Deleting Records

Updating Records

Using SwiftNIO

Using EventLoops

Choosing an HTTP Client

Examples

Further Code Documentation

Roadmap

0.1.0

0.2.0

0.4.0

0.6.0

0.8.0

0.9.0

v1.0.0

v1.x.x+

Not Planned

License

Using `MKNIOHTTP1TokenClient`