Transactions - Swift Driver

Piers_Ebdon · 2020-09-02T19:06:01.997Z

Hi Everyone

I am trying to update multiple different documents within different collections using a transaction from the Swift Driver for a Vapor application. I have looked at the example transactions docs and it doesn’t seem to provide a concrete example of how to actually use them.

I am currently stuck on firstly being able to implement a transaction and secondly how to get data back from one or more of the transactions such as the objectID for a newly inserted document.

I have created example code expanding on the ComplexVaporExample to hopefully help demonstrate the problem I am having.

   func kittenTransactionExample(_ req: Request) throws -> EventLoopFuture<Response> {
       let newKitten = try req.content.decode(Kitten.self)
    
       let neighbourhoodIncDocument: BSONDocument = ["$inc": ["totalCats": 1]]
       let neighbourhoodFilter: BSONDocument = ["name": "London"]
    
       let session = req.application.mongoClient.startSession()
    
       session.startTransaction().flatMap { _ in
           req.neighbourhoodCollection.updateOne(filter: neighbourhoodFilter, update: neighbourhoodIncDocument)
       }.flatMap { _ in
          req.kittenCollection.insertOne(newKitten)
       }.flatMap { _ in
          session.commitTransaction()
       }.whenFailure { error in
          req.eventLoop.makeFailedFuture(error)
       }
       // this doesn't compile 
       // how do I pass back a Response future ????
       // how do I access insertedID for the new kitten ???
   }

struct Neighbourhood: Content  {
    let name: String
    var totalCats: Int
}

/// Possible cat food choices.
enum CatFood: String, Codable {
    case salmon,
        tuna,
        chicken,
        turkey,
        beef
 }

struct Kitten: Content {
    var _id: BSONObjectID?
    let name: String
    let color: String
    let favoriteFood: CatFood
}

Piers_Ebdon · 2020-09-04T13:53:48.638Z

Hey @Michael_Lynn

Any chance on getting some feedback on this?

So close to releasing my iOS app but this is blocking me at the moment

Thanks

Piers_Ebdon · 2020-09-04T20:46:23.178Z

Hey @Michael_Lynn

Unfortunately I haven’t come up with a solution yet for this.

I was hoping for some help as no documentation or example articles out there on how to achieve this.

Thanks

Michael_Lynn · 2020-09-04T21:27:05.060Z

Piers, sorry for delay… I’m out today - but will definitely take a look this weekend.

Piers_Ebdon · 2020-09-05T14:56:35.248Z

Hey @Michael_Lynn

This is what I have come up with so far, which at least compiles but I am not sure if it is the correct approach. The example code in the documentation uses the whenFailure call back but I can’t get that to work. I also haven’t implemented session.abortTransaction() or session.end() and I am not sure if they are required and if so, where.

Edit - BELOW DOESN’T WORK - results in a crash

func kittenTransactionExample(_ req: Request) throws -> EventLoopFuture<Response> {
    let newKitten = try req.content.decode(Kitten.self)
    
    let neighbourhoodIncDocument: BSONDocument = ["$inc": ["totalCats": 1]]
    let neighbourhoodFilter: BSONDocument = ["name": "London"]
    
    let session = req.application.mongoClient.startSession()
    
    return session.startTransaction().flatMap { _ in
        req.neighbourhoodCollection.updateOne(filter: neighbourhoodFilter, update: neighbourhoodIncDocument)
    }.flatMap { _ -> EventLoopFuture<BSONObjectID> in
        req.kittenCollection.insertOne(newKitten)
            .flatMapThrowing { insertOneResult in
                guard let insertedID = insertOneResult?.insertedID.objectIDValue else {
                    throw Abort(.notFound)
                }
                return insertedID
            }
    }.flatMap { objectID -> EventLoopFuture<BSONObjectID> in
        session.commitTransaction()
            .map { objectID }
    }.map { objectID in
        return Response(status: .ok)
    }
}

Is there any intention of expanding upon the ComplexVaporExample app? It is a solid foundation and incredibly useful but I think most projects would soon go beyond the example functionality shown in it.

Michael_Lynn · 2020-09-05T19:47:07.359Z

What version of Xcode and Swift are you using?

Piers_Ebdon · 2020-09-05T21:51:45.518Z

I’m using Xcode 12 beta 6 and Swift 5.3

Michael_Lynn · 2020-09-06T10:22:16.687Z

Ok - I’m unable to move up to 12 at the moment… and on 11.6 I can’t even get a basic vapor project to compile due to a swift-nio-transport-services bug.

/Users/mlynn/code/.../.build/checkouts/swift-nio-transport-services/Sources/NIOTransportServices/NIOTSConnectionChannel.swift:425:41: error: value of type 'NWConnection' has no member 'startDataTransferReport'

Piers_Ebdon · 2020-09-06T12:55:46.779Z

Are you able to get 11.7? (latest public release).

I can go on my work Macbook and give 11.7 a go and see if it works. It seems strange that you are unable to compile a basic Vapor project though. Probably worth doing the standard project clean stuff like deleting derived data, cleaning folder etc and trying again if you haven’t already.

Are you able to speak to the Mongo Swift Driver team if this nio problem persists?

Piers_Ebdon · 2020-09-06T15:40:24.332Z

I’ve got no issue building with Xcode 11.7 using the Mongo Swift Driver and my personal Vapor project. If that bug persists then the Vapor Discord would be a good place to ask about it as I don’t imagine it being a permanent issue that can’t be resolved.

Michael_Lynn · 2020-09-06T16:37:13.958Z

So we are in the midst of a company holiday so the swift driver team is probably afk.

I put this repo together to reproduce the error. GitHub - mrlynn/swift-nio-transport-test: Just a test

I’m thinking it’s just a problem with nio-transport.

Piers_Ebdon · 2020-09-07T18:52:03.856Z

Any luck with the above?

I’m not getting an error from the repo you created and not sure how to resolve that issue on your end.

What do you think would be the best way to tackle this?

Michael_Lynn · 2020-09-08T14:22:35.342Z

We were closed from Friday to Monday for a U.S. Holiday… Hoping to have the swift driver folks take a look today.

Piers_Ebdon · 2020-09-08T17:21:02.899Z

Awesome, thanks a lot Michael!

kmahar · 2020-09-08T21:06:43.383Z

(Sorry, pressed enter too soon on my last post!)

Hi @Piers_Ebdon, thanks for reaching out and for your patience while our team was out of office.

To address a few points of confusion that have come up here thus far -

In order to have any operation you perform be considered part of a transaction, you must pass in the session you used to start the transaction as the session parameter. For example your updateOne call should have a final session: session parameter. This is how the driver determines whether an operation is part of a particular transaction or not. This is necessary because MongoClients are intended to be global, thread-safe objects, and there could be requests occurring in parallel using the same client that are not intended to be part of the transaction.
You must always call end() when you are done using a session. Failure to do so will result in a assertionFailure (only evaluated in debug mode) when the ClientSession goes out of scope. That said, you may find it easier to use the MongoClient.withSession helper as that will handle the cleanup for you automatically by calling end() once the passed-in closure has finished executing. (Shown in example below.)
abortTransaction() will effectively “throw away” the transaction a session has in progress if it exists. A session can only have one transaction in progress at a time, so you would need to call this if you wanted to perform multiple transactions with the same session. Calling session.end() will automatically abort an in-progress transaction if one exists, so if you are only ever doing a single transaction per session it’s not strictly necessary to also call abort.
I think the source of the crash in your example method is that you are not hopping back to the request’s eventLoop before the end of the route closure. Futures returned from the driver may fire on any of the event loops in the ELG you pass in when creating the MongoClient so you need to ensure you hop back to the correct one for the request before returning. I’ve shown how to do this below. (We discussed before upcoming work to make the driver easier to use with Vapor - one goal of that project is to remove the need for you to do this kind of hopping yourself.)
I did see that you commented asking about our plans to implement the “convenient API” for transactions, which adds support for automatically retrying transactions upon certain types of failures. We do not have a timeline for completing that work at the moment but we will certainly take into account your request as we plan upcoming work.
I am glad you have found the complex example project useful. Our developer advocacy team including @Michael_Lynn is working on creating more examples now, and we could consider augmenting the existing example app as well. We very much appreciate hearing from you on what your pain points have been thus far (such as you have described in this post) as it helps us figure out what examples we should prioritize creating.

Ok, all that said, onto the code sample you’ve provided! I’m able to get it working as I would expect by doing the following:

func kittenTransactionExample(_ req: Request) throws -> EventLoopFuture<Response> {
    let newKitten = try req.content.decode(Kitten.self)

    let neighbourhoodIncDocument: BSONDocument = ["$inc": ["totalCats": 1]]
    let neighbourhoodFilter: BSONDocument = ["name": "London"]

    return req.application.mongoClient.withSession { session in
        session.startTransaction().flatMap { _ -> EventLoopFuture<UpdateResult?> in
            req.neighbourhoodCollection.updateOne(filter: neighbourhoodFilter, update: neighbourhoodIncDocument, session: session)
        }
        .flatMap { _ -> EventLoopFuture<BSONObjectID> in
            req.kittenCollection.insertOne(newKitten, session: session)
                .flatMapThrowing { insertOneResult in
                    guard let insertedID = insertOneResult?.insertedID.objectIDValue else {
                        throw Abort(.notFound)
                    }
                    return insertedID
                }
        }
        .flatMap { objectID -> EventLoopFuture<BSONObjectID> in
            session.commitTransaction()
                .map { objectID }
        }
        .map { objectID in
            Response(status: .ok)
        }
    }.hop(to: req.eventLoop)
}

Basically what I’ve changed is switched to using withSession, added a hop(to:) at the very end of the returned closure, and passed the session parameter to both updateOne and insertOne.

Let me know if that works for you or if you have any further questions or issues.

Best,
Kaitlin

Piers_Ebdon · 2020-09-11T12:51:16.025Z

Hi @kmahar

I am running out of superlatives to say thank you for your responses!

Apologies for the slow reply. Currently in the process of moving cities in the UK and so I have been unable to really look into your amazing detailed explaination and answer.

Following your response I have been able to get transactions working with the Swift driver .
I only commented on the conventient transactions api ticket because I was desperate to get the transactions working in my app , so please ignore.

In the example code you provided, I assume if updating the neighbourhood document fails then the whole transaction fails and there is no need to add a check on the updateResult response to ensure it has succeeded. Is that correct?

I will let you know if I have any other questions but I think everything else is covered

Cheers

Piers

kmahar · 2020-09-11T20:55:47.630Z

Piers_Ebdon:

Apologies for the slow reply. Currently in the process of moving cities in the UK and so I have been unable to really look into your amazing detailed explaination and answer.

Not a problem at all, good luck with the move!

I am glad you were able to get your application working!

Piers_Ebdon:

In the example code you provided, I assume if updating the neighbourhood document fails then the whole transaction fails and there is no need to add a check on the updateResult response to ensure it has succeeded. Is that correct?

Yes, that’s correct. You can test that behavior by, for example, modifying your neighbourhoodIncDocument to contain an invalid update operator the MongoDB server will reject (for ex. let neighbourhoodIncDocument: BSONDocument = ["$blah": ["totalCats": 1]]).
In that case, the response I get to a POST request to the corresponding endpoint is

{
    "error": true,
    "reason": "WriteError(writeFailure: Optional(MongoSwift.MongoError.WriteFailure(code: 9, codeName: \"\", message: \"Unknown modifier: $blah. Expected a valid update modifier or pipeline-style update specified as an array\")), writeConcernFailure: nil, errorLabels: nil)"
}

And both the kitten and neighbourhood collections remain unchanged.

system · 2020-09-16T20:55:49.091Z

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.